3.1. Addition of Shallow Feature Layer

Deep neural networks are celebrated for their expansive receptive fields, which endow them with the capacity to provide rich semantic information. However, this can also lead to the loss of detail information in small targets. Because small targets account for a considerable proportion in some scenarios, this may cause problems in road traffic detection. In contrast, shallow networks, characterized by their constricted receptive fields, are adept at preserving locational data and delivering granular detail, making them inherently more adept at discerning densely situated small targets [26]. To reconcile the need for both comprehensive semantic understanding and precise localization, the integration of a shallow feature layer within the model’s neck is proposed. This strategy leverages the path aggregation feature pyramid network (PAFPN) architecture for effective feature fusion, thereby augmenting the model’s proficiency in small target detection. The resultant enhancement in the detection capability not only ameliorates the model’s sensitivity towards small targets but also substantially elevates the overall accuracy of detection.

Most networks use FPN to implement feature fusion at three scales, which first obtain features down-sampling by eight, sixteen, and thirty-two times of the backbone network and save them as {P3, P4, P5}. Then, P5 features are up-sampled from bottom to top and fused with P4 and P3. Feature fusion at these three scales can achieve detection of some small targets, but it does not simultaneously consider the deep and rich semantic information and shallow detail information; therefore, there is still room for further improvement. This paper modifies the model neck by fusing feature information from four scales, generating four 3D tensors. The value of each tensor is equal to the product of the number of grid units and the predicted categories, anchor box positions, and confidence. Table 1 shows the resolution mapped to different feature layers.

The effective pixel information gradually decreases as the convolutional layer depth increases. In light of the specific target scale distribution characteristic of the dataset utilized in this study, images that manifest dimensions smaller than 32 × 32 pixels are categorized as small targets. The distinct information pertinent to various feature layers, in relation to these small targets, is comprehensively illustrated in Figure 2.

As depicted in Figure 2, the initial step involves dividing the output feature matrix into a series of S × S grids. This division allows for the identification of target centers within individual targets through their corresponding grid units. A grid unit assumes the responsibility for detecting a target if the target’s center resides within its boundaries. Concurrently, each grid unit is equipped to detect up to three bounding boxes. These bounding boxes encompass four coordinate offsets, a confidence score, and a classification for C distinct classes. Hence, the resultant 4D feature matrix, generated from predictions in a single-layer feature matrix, comprises parameters quantified as S × S × [3 × (4 + 1 + C)].

Considering that some areas may have dense targets, the original image grid mapping of a deep-layer feature corresponds to a large area, and the center points of different detection targets may overlap in the same unit cell, which can cause the missed detection of dense targets. Assuming that the input image pixel size is 640 × 640, after 8×, 16×, and 32× down-sampling, the numbers of feature grid cells are 80 × 80, 40 × 40, and 20 × 20, which correspond to large areas in the original image. For dense objects, there may be multiple target centers in the same unit cell. After adding a 4× down-sampling scale, the numbers of feature grid cells are 160 × 160, 80 × 80, 40 × 40, and 20 × 20, which correspond to an increased density and decreased size of the corresponding grid in the original image. The number of small dense targets in the same grid unit is reduced, thereby reducing the probability of missed detection of small objects in dense areas.

3.2. Lightweight Module Improvement

Incorporating shallow feature layers into the model significantly enhances its capacity to detect small targets but concurrently escalates both the parameter count and computational demands. To counterbalance these drawbacks, the strategic implementation of model lightweighting emerges as a crucial intervention. Our empirical investigations, detailed in Section 5.2, underscore the necessity of focusing lightweighting efforts on the network’s neck. The improved lightweight network structure is shown in Figure 3.

Figure 3 illustrates the substitution of traditional convolution and C3 module with Ghost convolution and C3Ghost module for the neck section. The modified location is indicated in the neck part on the right-hand side. The lightweight module aims to reduce the number of parameters in order to shrink the model size with minimal impact on model accuracy.

The Ghost is one of the excellent lightweight network modules proposed in 2020, which can effectively reduce the model size. The difference between the Ghost module and traditional convolution is that the Ghost module reduces the number of channels in the convolutional kernel and uses the linear transform (depth convolution) for feature fusion to obtain the same number of channels as traditional convolution. The workflow is as follows: first, feature maps are obtained through traditional convolution, then additional feature maps are generated using linear operations, and finally, the feature matrix obtained by conventional convolution and the multiple feature matrices obtained by linear mapping are concatenated [27]. The Ghost module generates the same number of feature maps with less computation, which can be used to replace standard convolutions in classical convolutional neural networks. The comparison between the conventional convolution and Ghost convolution is shown in Figure 4.

In Equation (4), the numerator represents the computation of traditional convolution parameters, and the denominator represents the computation of the Ghost convolution. n is the number of output channels of traditional convolution, c is the number of input channels, and h^′ and w^′ denote height and width of the output feature matrix, respectively. k represents the size of the convolution kernel, s is the number of grouped convolutions, and d represents the size of the linear mapping kernel.

The ratio of the computation of the Ghost module to the regular convolution is less than 1. If the number of grouped convolutions s is large, the Ghost module can reduce the computation by approximately s times of standard convolution, achieving similar feature matrices with fewer parameters and computations.

Figure 5 showcases the C3Ghost module. In addition to modifying the base convolution module to the Ghost module in the neck of the network, we also modified the C3 module of the CSP architecture to C3Ghost by changing the bottleneck to the Ghost bottleneck. The C3 module of YOLOv5s has two branches, one with one standard convolution and multiple stacked bottlenecks, and the other with traditional convolution. Finally, the results are concatenated after the two branches have been processed.

3.3. Introduction of Attention Mechanism

The attention mechanism emulates the human visual system’s capacity for selective concentration on pertinent details. This process involves an intentional focus on critical elements while disregarding extraneous information, thereby honing in on the essential data. In the context of road imagery, the background complexity is notably high, especially when it comes to distant entities such as vehicles and pedestrians, which often appear clustered. The challenge intensifies after the application of multiple convolutional layers, making it arduous to differentiate between the feature points of small targets and those of the background. Occasionally, irrelevant background objects might dominate a significant number of feature points, substantially impeding the model’s ability to accurately detect the intended targets. To mitigate the model’s parameter load, we previously incorporated Ghost and C3Ghost modules within the network’s neck. Building on this, we propose the integration of an attention mechanism into the backbone network to further refine detection capabilities.

After conducting a series of experiments, we decided to enhance the C3 module in the backbone network by integrating a global context block (GCNet). This addition aims to amplify the algorithm’s proficiency in feature learning. The specifics of this modification are depicted in Figure 6.

The global context module is an image module that can be used for global context modeling [28]. It is mainly used in detection tasks, particularly for image processing with complex backgrounds, such as natural scenes or city streets. This module has a powerful capability to model long-range dependencies through its nonlocal block, while guaranteeing a lightweight calculation model. The specific structure of the module is illustrated in Figure 7.

As depicted in Figure 7, the global attention module plays a critical role in extracting comprehensive image features, subsequently integrating these with convolutional features through either element-wise multiplication or addition. This integration significantly augments the model’s comprehensive understanding of the entire image. The process encompasses three main stages: global attention pooling, feature transformation, and feature aggregation. Initially, attention weights are determined via a 1 × 1 convolution and the softmax function, facilitating the extraction of global context features. Subsequently, feature transformation is realized through a 1 × 1 convolution operation. The process culminates in feature aggregation, where the features are amalgamated through summation. Overall, the GC module is proposed as a lightweight method to achieve global context modeling.

As the C3 module has two branches, the improvement of GC module can be attempted in both of them. Various embedding strategies were trialed, revealing that substituting the bottleneck module in the lower branch with a GCBottleNeck substantially elevates computational complexity. Conversely, incorporating the GC module prior to the conventional convolution in the upper branch enhances semantic information with only a marginal parameter increase. Consequently, we consider incorporating the GC module into the upper branch of the C3 module to improve the contextual awareness of the backbone network, as illustrated in Figure 8.

4.1. Datasets

This study utilizes two prominent datasets for the evaluation of road vehicle and pedestrian detection algorithms: the KITTI dataset [29] and the BDD100K dataset [30]. Each dataset is distinct in terms of its composition, target categories, and the scenarios it represents, thereby providing a comprehensive framework for experimental validation.

The BDD100K dataset stands as a substantial and publicly available driving dataset, comprising 80,000 labeled images captured by manually operated vehicles across a spectrum of weather conditions, scenes, and times via key frame sampling. This dataset is characterized by its diverse representations of road conditions, incorporating a wide range of weather scenarios including sunny, cloudy, overcast, rainy, snowy, and foggy conditions, along with various roadway contexts like residential zones, highways, and urban streets. On average, each image features approximately 9.7 cars and 1.2 pedestrians, rendering the BDD100K dataset an exemplary resource for exploring detection tasks across a multitude of scenarios, temporal settings, and road conditions.

The KITTI image dataset is specifically tailored for autonomous driving applications. It consists of 7481 labeled images taken by manually driven vehicles with optical lenses such as grayscale and color cameras. The dataset contains various environments and conditions, including urban areas, rural areas, and highways. The target objects in the KITTI dataset have significant differences in various aspects. For each image, up to 30 people and 15 cars are present, and some targets are small with blurring and occlusion, which adds to the dataset’s challenge.

4.2. Data Analysis

This paper analyzes raw data by selecting the labeled BDD100K and KITTI datasets. Only vehicles and pedestrians are detected in the BDD100K dataset, while vehicles, pedestrians, and bicycles are detected in the KITTI dataset. Therefore, it is necessary to merge some data categories and remove unnecessary ones. The proportional distribution of different target categories in the dataset is shown in Figure 9.

4.3. Data Preprocessing

Given that BDD100K stores annotations such as bounding boxes, categories, and weather conditions in the JSON format, while YOLOv5s training only requires category and bounding box information, it is necessary to design three processes—data filtering, format conversion, and data merging—to construct the final dataset based on the actual situation, as shown in Figure 10.

During the data filtering phase, to meet the requirements of small object detection tasks and reduce the consumption of model training, 3000 images with clear targets on sunny days were selected from the 70,000 images in the training set. First, the original label folder was traversed to determine whether each image was annotated as sunny with clear targets. If the requirements were met, the corresponding labels were stored; otherwise, they were ignored. Then, based on the saved labels, the relevant images were moved to obtain the first round of the filtered dataset. Next, the dataset was traversed to check if the number of selected images was greater than or equal to 3000 with at least three pedestrian targets. If these conditions were met, the corresponding labels and images were saved to a new folder, resulting in the second round of the filtered dataset. The distribution of categories in the original and filtered datasets is shown in Figures 11 and 12.

As shown in Figure 11, the proportions of various targets in the original dataset are significantly unbalanced, with vehicles and other categories differing by nearly an order of magnitude. By filtering images with at least three pedestrians, the two rounds of data filtering reduced the overall magnitude difference between the number of vehicles and pedestrians. Figure 12 shows the comparison of the number of targets per category in the filtered training set, validation set, and test set. From the figure, it can be seen that the filtered dataset has a higher proportion of vehicles and pedestrians, and the numbers of these categories are relatively close in the validation and test sets.

During the data consolidation phase, diverse vehicle types such as cars, buses, trucks, and trains within the BDD100K dataset were amalgamated into a unified category designated as “Vehicle”. Concurrently, pedestrians were classified into a distinct category. Subsequent to the amalgamation process, a compilation of 22,737 vehicle instances and 13,531 pedestrian instances was achieved. These instances were systematically allocated to the training, validation, and testing datasets in an 8:1:1 ratio, culminating in a dataset comprising 2400 images for training, alongside 300 images each for validation and testing purposes. Specifically, the training dataset encompasses 28,050 vehicle targets and 12,856 pedestrian targets. Within the validation dataset, composed of 300 images, there are 3504 vehicle targets and 1669 pedestrian targets. Similarly, the testing dataset, also comprising 300 images, contains 2336 vehicle targets and 1340 pedestrian targets. The outcomes of this data merging process are illustrated in Figure 13.

The KITTI dataset was processed through two steps—category merging and format conversion—to construct the final dataset, as shown in Figure 14. First, the original data labels were analyzed by opening the original label folder and traversing all the TXT files within. Each TXT file was then converted into an XML label file in VOC format. After shuffling the labels, the dataset was divided, and the XML files were converted to JSON. Finally, the dataset was constructed in the YOLO format.

The dataset merges different types of vehicles into one “Car” category, combines pedestrians and sitting persons as one “Pedestrian” category, and treats bicycles as a separate category while ignoring other categories. The merged results are shown in Figure 15. Label category merging is used to alleviate unstable training results caused by large differences in the number of different categories. It enables the effective detection of important road targets under conditions of reduced parameter quantity and provides better verification of different algorithms.

4.4. Analysis of Processed Data

The dataset, following modification, has been designated as “Partial_BDD100K”. An in-depth analysis of this curated dataset was conducted, focusing on the dimensions of the annotated bounding boxes. This examination utilized heat maps to scrutinize the width and height of the targets, alongside the relative proportions of these bounding boxes within the entire image frame. The outcomes of this analysis, particularly the distributions of both the dimensions and their corresponding proportions across the dataset, are encapsulated in the heat maps displayed in Figure 16. This visual representation aids in understanding the spatial characteristics of the targets within the Partial_BDD100K dataset, offering insights into the typical size and space occupancy of the detected objects.

As depicted in Figure 17, the concentration of green and orange-red dots in the lower left corner signifies a higher quantity of targets, while the peripheral areas marked by blue and purple dots indicate fewer targets. A significant portion of the targets within the dataset is approximately 40 × 40 pixels in size. Given this size, it is possible that features of these smaller targets may diminish or even disappear following multiple layers of convolutional pooling. Excessive down-sampling could result in the loss of most target details. In the YOLOv5 network architecture, the maximum down-sampling factor is limited to 32 times, which is a deliberate design choice to mitigate the loss of essential target information to a manageable extent. Given the substantial presence of smaller targets within the Partial_BDD100K dataset, we try to store multiple same-down-sampling feature matrices for different target sizes to retain most of the target information and ensure the extraction of valid information by the backbone network.

The analysis of the 5985 training images from the consolidated and processed KITTI dataset, as illustrated in Figure 13, reveals a significant concentration of actual targets within the dimension range of 40 × 40 to 100 × 100 pixels (width × height). Furthermore, the aspect ratio (width to height) of these targets predominantly falls within the range of 0.05 × 0.1 to 0.1 × 0.2. In addition, the number of images in the training, validation, and test sets is shown in Figure 17, and the number of targets per category is shown in Figure 18. The statistical insight underscores a notable clustering of small target sizes within a specific dimensional spectrum, highlighting the importance of tailoring detection algorithms to effectively recognize and process targets within these small size and aspect ratio ranges.

4.5. Statistics of Road Target Size

Given the average dimensions of the KITTI dataset and the Partial_BDD100K dataset at 1280 × 720 and 1242 × 375 pixels, respectively, it’s evident that the image resolutions substantially exceed those typically found in the COCO dataset. Hence, establishing the threshold for small targets in road traffic datasets at 32 × 32 pixels is a logical approach. Road traffic image datasets feature a diverse array of object sizes, allowing for categorization into small, medium, and large targets based on their scale. Meanwhile, a statistical analysis is carried out on the number and proportion of small targets in the training set category labels [31].

The number of label boxes for each scale is listed in Table 2. It can be observed from the table that the number of small objects in the KITTI training set is 5429, accounting for 14.64% of the total, in the Partial_BDD100K dataset, the number of small objects is 18,397, accounting for 31.02%, indicating that the quantity of small objects in road scene datasets is relatively high, but the overall area sizes are only 1.02% and 2.50%, respectively. The prevalence of small objects underscores the challenges faced in the road scene, where large and medium-sized objects dominate the annotations, but small objects, because of their limited pixel, undergo significant feature attenuation post-down-sampling, hard to detect. This observation emphasizes the importance of devising detection algorithms that are particularly sensitive to the nuanced features of small objects, ensuring comprehensive and accurate scene analysis in varied road environments.

5.1. Experimental Environment Framework and Setup

The application of deep learning in multiple fields is becoming increasingly common. Therefore, open-source frameworks are widely used to adapt to different environments. In this study, experiments were conducted using the Pytorch framework based on the Python language. The hardware environment consisted of an AMD Ryzen 9 5900HX with Radeon Graphics CPU and an NVIDIA GeForce GTX 3080 GPU, and the integrated development environment used was Pycharm2020.3.

The initial learning rate of the experiment was set to 0.0032 using a dynamic descent method, with a learning decay factor of 0.1. Because of hardware and memory limitations, a batch size of 16 was used, and the training iteration was set to 300. The input image size was uniformly set to 640 × 640.

For the training configuration, the pretraining weight file of YOLOv5s is selected. For hyper-parameter settings, the initial values of optimizer-related parameters such as learning rate, learning rate momentum, and weight decay coefficient are consistent with the baseline model, and the same is true for parameters related to preheat learning, loss function, and data enhancement.

In the training phase, following the training strategy of YOLOv5s algorithm, including the data enhancement strategy, the size and aspect ratio of the backbone network, and the predefined anchors, the same training strategy is adopted for different models in order to achieve fair competition among different models to a certain extent.

5.2. Detection Results on Partial_BDD100K Dataset

Experimental evaluations were conducted on the Partial_BDD100K dataset. Initially, the optimal position for the Ghost module was determined by comparing the performance of various improvement locations. Subsequently, the detection results of the C3CBAM and C3GC modules were compared to determine the optimal attention mechanism enhancement scheme. Ultimately, a comparison with classical models was performed to verify the feasibility of the proposed improvement scheme.

5.3. Detection Results on the KITTI Dataset

Given the modest differentiation in the verification performance of various attention mechanisms on the Partial_BDD100K dataset, further comparative experiments were conducted on the more extensive KITTI dataset. Subsequent comparisons of different models were undertaken to substantiate the viability of the proposed enhancements.

5.4. Visualization Test Results on the KITTI Dataset

The mAP curves of YOLOv3s, YOLOv5s, and YOLOv5s_LGC during 300 iterations of experiments on the KITTI validation set are depicted in Figure 19.

From Figure 19, the mAP values gradually converge at 150 iterations, and the proposed YOLOv5s-LGC model has partially improved in terms of convergence speed and accuracy during the iterations, which indicates that the model detects better.

Figure 20 illustrates the curves for bounding box loss, confidence loss, and class loss over 300 iterations. As the model iterates, the losses gradually converge, particularly after 200 iterations. The proposed YOLOv5s-LGC model achieves a faster convergence rate compared to the original YOLOv5s and YOLOv3s models, exhibiting lower confidence and class losses. This underscores the improved model’s effectiveness on the KITTI dataset.

To visually demonstrate the detection performance of the proposed model on traffic targets, images numbered 000073 and 000770 from the test set were selected for qualitative evaluation, presenting the target detection results of YOLOv3s, YOLOv5s, and the proposed YOLOv5s-LGC algorithm, respectively. This is shown in Figure 21.

As observed in Figure 21, on the image with label 000073, the YOLOv3 algorithm detected six pedestrians and two bicycles, YOLOv5s detected six pedestrians and three bicycles which additionally included the bicycle in the lower-right corner of the image, while the proposed YOLOv5s-LGC algorithm detected seven pedestrians and three bicycles, which included an additional pedestrian in the middle of the image. On the image with label 000770, the YOLOv3s algorithm incorrectly identified a bicycle on the left as a pedestrian, YOLOv5s algorithm failed to detect the target at all, while the proposed YOLOv5s-LGC algorithm was able to successfully detect the bicycle target and show slight improvement in detection precision for different categories. This effect can also be seen from the obtained scores of detection categories, indicating that the improved algorithm has a certain promotion effect on image target detection.

To visually display the advantages of the small target detection of the method in this paper, two images of small target objects that are far away from the camera are selected on the KITTI dataset, respectively, as shown in Figure 22.

From image 001286, it can be observed that YOLOv5s-LGC detected some pedestrian targets that were not detected by the other two models. The results of detection from image number 001466 show that YOLOv5s has mainly improved in accuracy. According to the program statistics, the improved YOLOv5s-LGC detected four pedestrians, four cars, and one bicycle, which is significantly more than the other models in terms of quantity. Thus, the YOLOv5s-LGC algorithm has improved the detection of small targets on the road to a certain extent.

To sum up, the original YOLOv5s model has a slightly poorer performance in detecting small targets in the distance, with lower accuracy and some missing target detection. However, the YOLOv5s-LGC algorithm proposed in this paper has improved the detection ability of small targets in the image and further improved the category score. This indicates that by fusing shallow features, improving the backbone network, and introducing attention mechanisms, strong semantic information is obtained from the pictures with detailed information, thus improving the detection ability of the network model and reducing the missed detection rate.

5.5. Ablation Experiments

This paper proposes three main improvements to the YOLOv5s algorithm: L: introducing shallow feature layers to tackle the issue of small target detection; G: replacing conventional convolutions and C3 modules with Ghost convolutions and C3Ghost modules, respectively; C: introducing GC attention mechanism to address background interference. To validate the necessity of different modules for the final results, ablation experiments were conducted on the KITTI dataset. Table 8 displays the model parameter and detection results in order of increasing levels of improvement: shallow features, Ghost improvements, and attention mechanism improvements.

Firstly, we added each of the three methods L/G/C to the baseline model to analyze their independent contributions. It can be seen from the table that the parameters of the YOLOv5s-L model increased by about 7.0 × 10⁵ compared to the benchmark YOLOv5s, and the model size increased by 1.5M. Under the IoU threshold of 0.5, the mAP of the validation and testing sets increased by 1.0% and 1.5%, respectively, and the overall detection accuracy improved. The detection speed was 66.7 FPS, and the detection time for a single image was 15.0 ms with the addition of shallow feature fusion, which increased by 1.1 ms compared to the original YOLOv5s detection time. The experimental results of the improved YOLOv5s-L model show that adding shallow features fusion is conducive to improve the accuracy of detection targets, and it may achieve a certain degree of improvement for some difficult-to-detect targets, but the model capacity and parameter quantity increase. For the YOLOv5s-G model, the amount of parameters in the model is reduced by about 1.4 × 10⁶, the model size is reduced by 2.5M, and the detection speed is the highest at 72.1 FPS. Yet, its accuracy is only reduced by 0.1% on the test set, which shows its excellent performance in balancing accuracy and speed. For the YOLOv5s-C model, its detection accuracy improves by 0.8% on the validation set and 0.3% on the test set, proving that the addition of this module also has a positive effect on model detection.

Secondly, for the improved YOLOv5s-LG model, the number of parameters decreased by about 1.7 × 10⁶ compared to the YOLOv5s-L model, and the model size decreased by 3.2M, which is 1.7M less than the benchmark YOLOv5s model. Under the 0.5 threshold, the mAP of the validation and testing sets decreased by 0.5% and 0.7%, respectively, compared to the YOLOv5s-L model, and increased by 0.5% and 0.8%, respectively, compared to the original YOLOv5s model. The detection speed did not change significantly. The experimental results of the YOLOv5s-LG model show that replacing the conventional convolution in the neck of the network with a lightweight Ghost convolution significantly reduces the model’s parameter quantity, which reduces the deployment difficulty of the model, although the accuracy has decreased to some extent.

Finally, for the YOLOv5s-LGC model, the model parameters increased by only 3.1 × 10⁵ and the model size increased by only 0.6M. Under the 0.5 confidence, the average precision of all categories on the validation set reached 95.2% and reached 94.4% on the testing set, an improvement of 1.6% over the benchmark YOLOv5s model. In summary, the predictive results of the improved YOLOv5s-LGC algorithm are optimal, which not only improves the overall prediction accuracy but also reduces the model’s parameter quantity.

In theory, changing the conventional convolution to the Ghost module will increase the detection speed of the model; however, experiments show that the detection speed does not show significant improvement on both datasets used. For the problem that the detection speed decreases while the model size decreases after adding the Ghost module to the model neck, it is analyzed that there are roughly the following reasons: (1) Memory bandwidth limitation. The addition of the Ghost module may lead to memory bandwidth limitation, which affects the inference speed of the model. Since the Ghost module requires channel interaction operations, this may result in more memory accesses, which increases the burden on memory bandwidth. (2) Increased data dependency. The model may need more time to wait for the computation results of the previous layer, thus reducing the overall inference speed.

5.6. Generalizability Experiments

Since the Ghost convolution layer was experimentally determined to work the best, to validate the generalization of the model, we decided to add the UA-DETRAC dataset to validate the model performance. To contrast with the previous paper, we use SSD, YOLOv3s, YOLOv5s, and YOLOv7s algorithms for comparison, as shown in Table 9.

As can be seen from Table 9, in the UA-DETRAC dataset, our method is still advanced with higher accuracy compared to other mainstream methods. Figure 23 shows the experimental comparison between YOLOv5s and our method, where the green boxes indicate that our method detects more small targets.