2.1 Billboard recognition systems

In 2018, a method for automatic video frame classification to identify frames that include outdoor advertisements was proposed (Hossari et al., 2018). This method represents the first effort to use deep neural networks to identify advertisements in video frames. It identifies billboards' presence in video frames through the binary classification task. Next, a deep learning algorithm was presented to classify the presence of the ad in the image (Rahmat, Dennis, et al., 2019). It is combined with geolocalization data from mobile phones to provide location in the wild. Another method employs machine learning and OCR to extract information from billboard images taken by phone and compare them with a database, allowing to classify whether the billboard placement is legal or not (Liu et al., 2019).

The development of deep neural networks and the growth of computational capabilities pushed the research of object detection methods forward (Zou et al., 2023). Initially, the Single Shot MultiBox Detector (SSD) and You Only Look Once (YOLOv3) were compared in the detection of ad panels task, “Both detectors have been successfully able to localise most of the test panels” was written (Morera et al., 2020). The SSD model was also used with the transfer learning technique (Chavan et al., 2021). The authors created a private dataset of billboards captured in first-person view, resulting in 1052 images, and prepared annotations of each one using bounding boxes. The proposed model reached an average precision (AP) of 59.79 on their dataset. In recent years, an in-depth study was presented (Motoki et al., 2021) where authors constructed two models: for object detection targeting billboards in images and for extracting multiple features from billboards, such as “genre,” “advertiser,” and “product name.” The authors compared two model architectures: YOLOv4 (Bochkovskiy et al., 2020) and YOLOv5 (Jocher et al., 2020) on their in-house dataset, reaching the AP of 0.53 and 0.57, respectively. The authors of (Palmer et al., 2021) proposed the procedure of detecting and analyzing urban advertising on 360° street view images. The large-scale collection of images was prepared to classify unhealthy ads. They utilized the out-of-the-box tool for semantic segmentation of street view images (Porzi et al., 2019) and classified the billboards' pixels. The validation dataset consisting of 4562 billboards with an area >2000 pixels was created to evaluate this approach. The model achieved the Intersection over Union (IoU) score of 0.458. Moreover, their analysis shows that “143 items were falsely classified as billboards, consisting of street signs, blank surfaces, traffic lights, and interestingly clock faces.”

Despite algorithms development using deep learning to detect billboards in images, none currently fulfill the criteria for automatically or semi-automatically mapping large-area advertising in drone imagery. The analysis of images acquired by drones necessitates addressing a series of challenges (Srivastava et al., 2021), particularly stemming from the diverse perspectives and varying sizes of objects. One major limitation is the spatial resolution of captured images, which is typically limited due to the UAV altitude and specific camera perspective, allowing to capture only large-size ads. This contrast is particularly noticeable compared with earlier datasets that did not involve drones (see Figure 2). Considering those challenges, we propose the first approach aimed at detecting billboards from UAVs and using both spatial and temporal information to improve the estimation of billboard locations.

2.2 Efficient instance segmentation methods

Recent developments in the computer vision field have significantly advanced the accuracy and efficiency of object detection models. Among these advances, the YOLO (You Only Look Once) family of models has garnered significant attention due to its good balance between high accuracy and inference speed (Terven & Cordova-Esparza, 2023). YOLOv7 (Wang et al., 2022) and YOLOv8 (Jocher et al., 2023) are the latest iterations of the YOLO model, and they build upon the success of their predecessors by introducing a new network architecture and the “Trainable Bag of Freebies.” These enhancements have resulted in higher accuracy and faster processing times, making them a viable choice for various applications such as autonomous operation, surveillance systems, and object tracking (Chen et al., 2023).

However, one limitation of these models is that they only provide bounding box predictions, which can lead to the imprecise computation of the object's extent, especially when its shape is not rectangular. To overcome this limitation, the extension has been proposed (Bolya et al., 2019). This model's modification involves adding an instance segmentation head to the network architecture of the previous YOLO version. This head computes pixel-wise segmentation masks that provide a more detailed understanding of the spatial distribution of the object in the image. The instance segmentation head calculates prototypes that represent the average feature vectors of all the pixels within each object and then, based on those prototypes, generates the instance segmentation masks. It is more efficient than classical pixel-wise segmentation because it significantly reduces the number of trainable parameters in the network, affecting more effective feature representations.

4.1 Camera calibration

Camera calibration is crucial for the computer vision application that measures real-world distances, especially in drone imagery, because camera distortions caused by the lens can significantly contribute to imprecise determination of spatial localization of objects. Therefore, camera calibration corrects these lens distortions and determines the camera's intrinsic parameters. These intrinsic parameters define the inherent optical characteristics, such as focal length and lens distortion coefficients, that influence how the 3D world is projected onto the camera's 2D image plane. With accurate intrinsic parameters, mapping the 2D image coordinates of the billboard back to its accurate real-world measurements becomes possible. As described in Zhang (2000), the camera calibration process typically involves capturing images of a special calibration target with a known pattern of points. The algorithm can estimate the intrinsic camera parameters that minimize the re-projection error by analyzing these images. Re-projection error refers to the average distance between the established and estimated point's location on calibration images based on the estimated intrinsic parameters and the known 3D location of the point on the calibration target. Our calibration process reached the re-projection error of 0.0456 pixels, indicating the estimated intrinsic parameters accurately model the camera's distortions.

The calibration process is required for each individual camera, even within the same drone model, as slight manufacturing variations can affect their intrinsic properties. Furthermore, the camera sensor must be autofocus-free or set to autofocus-free mode to ensure a constant focal length. Any focal length or zoom changes would require re-calibration because these parameters influence image formation and distort the computed distances.

4.2 Instance detection and surface segmentation

When the camera is calibrated and each obtained frame is rectified from distortions, the billboard detection model is applied to recognize objects in an image. However, instead of classical object detection with bounding boxes, an instance segmentation algorithm is applied to determine the billboards' surface. Aside from detecting the bounding boxes, it provides pixel-wise segmentation of an object, allowing for a more fine-grained understanding of the spatial distribution and characteristics. For example, it can be used in the future to estimate ad sizes, as it provides a precise indication of the exact border of the area covered by each billboard in an image.

4.3 Inter-frame association

Since the instance segmentation model operates on individual images, the inter-frame association becomes necessary to match detected billboards across consecutive frames. Accurate association between frames is crucial for maintaining the consistency and continuity of object tracking, avoiding duplicates. This association strategy follows the “detect to track” approach (Feichtenhofer et al., 2017). Consequently, the tracking quality heavily relies on the detection precision, significantly influencing the overall system performance. To address this challenge, the current state-of-the-art method called ByteTrack (Zhang et al., 2022) is applied. The method associates bounding boxes by measuring the mutual coverage area and calculating the visual similarity of learnable features. It differs from the previous methods by matching every detected bounding box, including those with low detection scores, which improves performance with occluded objects. Due to the similarities between low-scoring detection boxes and tracks, it recovers true objects and eliminates background false-positives.

4.4 Billboard localization estimation

When a billboard is detected, and the camera is calibrated, its relative position to the camera can be estimated. The following section describes this estimation process step by step. Additional information from drone sensors is required to calculate real-world billboard position. Various measurements, such as the coordinates of latitude (lat) and longitude (lon), as well as altitude above ground (alt) can be easily obtained from the drone GPS sensor. Furthermore, the drone software provides information about its orientation, expressed as three angles: roll, pitch, and yaw. In the same way, the information about the gimbal's tilt, indicated by the angle α, is obtained. The camera FOV on both the horizontal and vertical axes denoted as FOV_H (74.3°) and FOV_V (46.2°) are taken from drone documentation. Those values are presented in Figure 5 and are used in the following section to estimate the billboard's position in the real world.

When the relative D distance and the angle β are estimated, the GPS coordinates of the billboard are calculated. This transformation is based on the actual drone's GPS position (lat, lon) and the drone's flight path forward azimuth (az) that describes the drone's flight direction. This information is read from the drone sensors. Then, the billboard GPS coordinates are obtained by incorporating those variables and geodetic forward transformation.

5.1 Models evaluation metrics

The mask mean average precision (mask mAP) is used as the evaluation metric to assess the precision of instance segmentation models. It measures how good the predicted billboard masks are, measuring mask intersection over union (mask IoU). Unlike the classical IoU for bounding boxes (used in object detection tasks), the mask IoU metric is computed between two masks, providing a more fine-grained and accurate assessment of the model's performance. The mask IoU is calculated based on the overlap between the positive values of the ground-truth mask and the predicted one. It is mathematically defined as: IoU = TP/(TP + FP + FN), where TP, FP, and FN represent True-Positives, False-Positives, and False-Negatives, respectively. The visual representation of the differences between IoU for bounding boxes and masks can be found in Figure 6. Mask mAP is calculated by considering detections at different confidence score thresholds, typically ranging from 0.5 to 0.95. A precision-recall curve is generated for every threshold level, and the AP is calculated as the area under this curve. Finally, the mask mAP is computed as the mean AP across multiple IoU thresholds. This incorporates the trade-off between precision and recall and provides a more complete evaluation of the model's performance. In addition, one specific value reported mask [email protected], which refers to the mask mAP calculated using an IoU threshold of 0.5 and represents a balance between how precise and how well-recalled the detections are, being sufficient to consider a detection accurate for many real-world applications.

5.2 Dataset preparation

Referring to Section 3, the dataset has sequential characteristics and was collected in five separate city areas. Therefore, we perform a 5-split geolocalization-based division into five subsets for training purposes. Unlike classical random data shuffling, this approach is closer to real-world use cases, allowing one to examine the models' generalization abilities to handle overfitting issues and indicate how the models handle images from other regions. Table 2 shows the statistics for each split.

5.3 Training process details

All experiments were conducted with the identical hyperparameters. The SGD optimizer (Sutskever et al., 2013) was used with an initial learning rate of 0.01, along with a learning rate scheduler with a warm-up phase was applied. Because we consider single-stage models, the training process was performed jointly for the backbone and both heads—for bounding boxes and mask prototypes.

Additionally, several regularization techniques were employed to address the relatively small dataset size challenge and enhance the method's generalizability. All model parts, such as the backbone, the object detection head, and the instance segmentation head, were initially pre-trained to benefit from large datasets (Huh et al., 2016; Risojević & Stojnić, 2021), like the COCO dataset (Lin et al., 2014). In the data preprocessing step, the upper part of the image is cut out due to distortions from a high-perspective view to be divisible into two squares. Next, the image squares are resized to 960 × 960 pixel to match the resolution required by the model input layer. Lastly, augmentations were used to improve model performance. In particular, both single-image augmentations (rotations, translations, color jitter) and multi-image augmentations (mosaic, as introduced in Bochkovskiy et al. (2020)) were applied to enhance generalization and reduce overfitting.

5.4 Models evaluation results

Each model was evaluated in 5-split cross-validation, according to the GPS-based division described in Section 3. Each split contains three subsets for training, one for validation, and one for the testing step. Then, the average and standard deviation of the metrics were calculated.

Model performance analysis shows that free-standing billboard detection is generally more accurate with all models compared with wall-mounted billboards for the test subset. The difference may be because free-standing billboards are statistically larger and more frequent in the dataset. In addition, they are more visually distinct from their surroundings, while wall-mounted billboards can blend in with the background or be partially obscured by other objects. Free-standing billboards often have a more consistent orientation and placement, making them easier to detect and classify accurately. The detailed results are in Table 3. It is observed that YOLOv8-X reaches better test mask [email protected] (0.751 ± 0.121) and mask mAP (0.601 ± 0.121) for the free-standing class, having, at the same time, the lowest standard deviation. However, for the wall-mounted class, the results are not conclusive. Focusing on mask [email protected], YOLOv7 achieved better metrics (0.389 ± 0.096) compared with YOLOv8-X (0.376 ± 0.097). Similar conclusions can be observed for the second metric, where YOLOv7 (0.293 ± 0.096) is better than YOLOv8-X (0.291 ± 0.085).

5.5 Final model for the application

Ultimately, to train the final model for the application, the YOLOv8-X was selected because it achieved the best overall metrics for the free-standing billboard class, the most important class for our application, and exhibited the most consistent performance across all models as indicated by the lowest standard deviation. Concentrating on selected architecture, the new training, validation, and testing splits were created. This is motivated by creating a model trained in all city regions, increasing generalization ability. Considering dataset splits and their sequential characteristics, the first and last 20 samples are selected for new validation (100) and testing (100) subsets. The rest of the images are used as a training set, resulting in 1161 images.

Table 4 summarizes the results for the final model. The results are higher than those obtained during the training on splits, but direct comparison is not possible due to the use of different test sets. For the free-standing billboard, the mask [email protected] is 0.927 and mask mAP is 0.778, showing good generalization ability. For the wall-mounted billboards, the results are worse. mask [email protected] and mask mAP are 0.611 and 0.432 for this class. Referring to the confusion matrix plot (Figure 7a), these lower metrics are likely caused by the higher rate of false-positives. This could be due to cluttered backgrounds around wall-mounted billboards or limitations in the data for this category caused by their size and installation method. In this evaluation, the results for large road signs are also reported to demonstrate their influence on misclassifications with other classes, indicating the model's robust ability to distinguish between them and billboards. Analyzing the F1-Confidence plot (Figure 7b), it is observed that the confidence threshold has no significant impact on free-standing, while a higher threshold decreases the results of wall-mounted, illustrating the model's lower confidence for this billboard type.

Sample model outputs with visualized segmentation masks are presented in Figure 8. They demonstrate the correctness of predicted bounding boxes and object masks in images from the test subset. We observe that the precision of the model drops, especially in two cases: when the object is too close and is too large to fit entirely in the image and when the object is too small due to perspective and distance from the camera.

6.1 Methodology

The ground GPS coordinates of 25 free-standing billboards were estimated based on recorded videos. The reference locations of objects were inferred from the precise orthophoto (with a ground sampling distance of 3 cm/px) of the city of Poznań (SIP Poznań, 2023). Using the geodetic distance (in meters), which uses the shortest curve between those two points along the surface of the Earth model, the distances between predicted and ground-truth coordinates were calculated. Then, the mean absolute error (MAE) is employed to analyze the system's performance in the real world and provides a quantitative measure of the estimation method's performance.

6.2 Real-world coordinates evaluation

The evaluation results in the average distance error of 6.31 m for this setting, demonstrating the accuracy achieved in determining the coordinates of interest. The smallest and largest obtained errors are 2.481 and 11.923 m, highlighting the range of potential localization errors that may occur using this method. Figure 9 presents some evaluation examples, giving a visual overview of reached errors. Specifically, it presents a visual difference between ground-truth localization (red marker) and estimated (black mark) relative to the local map.

6.3 Measurement uncertainty analysis

Using consumer-grade vehicles to estimate objects' positions can result in higher errors due to the inaccuracy of on-board sensor measurements. Therefore, the impact of variations in sensor readings was addressed. Table 5 illustrates how the mean estimation error fluctuates in response to shifts. The values in the “Amount” column were chosen according to the precision of the drone's sensors. The calculations were performed for an example scenario that is a good representation of the measurements in the dataset. In this scenario, it was assumed that the drone flew at 35 m at a consistent speed of 1 m/s, with the gimbal oriented at a 30-degree downward angle. Furthermore, it' is worth acknowledging that these errors have the potential to accumulate.

7.1 Use cases

7.2 Performance

The performance benchmark was conducted on our host machine equipped with NVIDIA RTX3090 GP-GPU to evaluate the system's operation in real-world cases while processing long videos. Using floating-point precision, the host machine allows for reaching a mean level of 38 frames per second (FPS). Additionally, the profiling process showed that any memory pick does not exceed 2 GB of GPU memory during the benchmark when running inference with single-image batches.

7.3 Web-based visualization

An example of a billboard detected by the system in a recorded video is presented in Figure 10. Figure 10a displays the middle frame of the video, with the billboard located on the right side (red bounding box). Figure 10b presents a screenshot of the web application, featuring a black marker representing the estimated position of the ad and pop-up information that shows the parameters detected, like the unique index, the type of billboard, the GPS coordinates, the location information, and the appearance of the advertisement. This application utilizes JSON files generated for each video file and visualizes them on the map. The application is developed using Python's Flask framework (Grinberg, 2018) and is supported by the most popular browsers.

8.1 Limitations

While the drone's imagery usage to detect billboards with the system is promising, it is important to acknowledge some limitations of the current study.

8.2 Future work

In future work, we will examine some system limitations and propose a dedicated aerial platform for precise sensing and measuring. First, accuracy can be improved using the differential global positioning system (DGPS) instead of the classical GPS, which provides centimeter-range precision. Second, the LiDAR-based sensor can be used for accurate billboard distance measurement and 3D relative pose estimation. Additionally, we plan to create a module that enables billboard aggregation from different videos and enables content and style analysis through re-identification and OCR (optical character recognition) methods.