A Heatmap-Supplemented R-CNN Trained Using an Inflated IoU for Small Object Detection

Abstract

Object detection architectures struggle to detect small objects across applications including remote sensing and autonomous vehicles. Specifically, for unmanned aerial vehicles, poor detection of small objects directly limits this technology’s applicability. Objects both appear smaller than they are in large-scale images captured in aerial imagery and are represented by reduced information in high-altitude imagery. This paper presents a new architecture, CR-CNN, which predicts independent regions of interest from two unique prediction branches within the first stage of the network: a conventional R-CNN convolutional backbone and an hourglass backbone. Utilizing two independent sources within the first stage, our approach leads to an increase in successful predictions of regions that contain smaller objects. Anchor-based methods such as R-CNNs also utilize less than half the number of small objects compared to larger ones during training due to the poor intersection over union (IoU) scores between the generated anchors and the groundtruth—further reducing their performance on small objects. Therefore, we also propose artificially inflating the IoU of smaller objects during training using a simple, size-based Gaussian multiplier—leading to an increase in the quantity of small objects seen per training cycle based on an increase in the number of anchor–object pairs during training. This architecture and training strategy led to improved detection overall on two challenging aerial-based datasets heavily composed of small objects while predicting fewer false positives compared to Mask R-CNN. These results suggest that while new and unique architectures will continue to play a part in advancing the field of object detection, the training methodologies and strategies used will also play a valuable role.

1. Introduction

Recent work in the field of object detection has led to the ever-increasing performance of state-of-the art models. Common architectures continue to be improved upon to achieve a greater detection accuracy while attaining increased detection speeds [1,2,3] and the ability to predict segmentation masks [4]. However, modern architectures continue to detect small objects at rates significantly lower than those for larger objects [5,6,7].

Smaller objects pose multiple challenges to detection models. Represented by fewer pixels, they provide less information for both training and detection. By occupying a smaller area of an image, small objects also require models to predict their locations with a higher level of precision. Despite these challenges, the detection of small objects is an important aspect of many applications, including remote sensing [8,9,10] and pedestrian and sign detection for autonomous vehicles [11,12,13].

A prime example of an application that primarily involves the detection of smaller objects is found in unmanned aerial vehicle (UAV) imagery. Often, the objects in these applications both meet the classical definition of small objects—smaller than $32\times 32$ pixels, as defined in COCO [14]—and are relatively smaller compared to the size of the images commonly captured in these applications. The capability of a UAV to cover vast amounts of terrain while carrying high-definition imaging equipment has allowed for this technology to be adapted to applications such as wildlife tracking [15], infrastructure monitoring [16,17] and search and rescue [18]. However, the benefits UAVs provide in object detection applications are undercut if the detection architecture fails to detect smaller objects.

Numerous methodologies have been explored for improving the detection of these challenging objects, including multi-scale learning, data augmentation, generative adversarial networks, unique training strategies, context-based learning and super-resolution approaches [6,19,20,21,22,23]. The work of [24] proposed the use of a super-resolution generative adversarial network (GAN) to increase the resolution of images before detection to discern smaller objects better. Numerous forms of improvement to the Feature Pyramid Network (FPN) backbone have been proposed to increase the extraction of information related to smaller objects [25,26,27,28,29]. While the work of [30,31] proposed new training strategies for improving the learning of smaller objects on the general COCO dataset.

In this paper, we aim to improve the performance of modern architectures in the detection of small objects—demonstrated through the detection of vehicles in challenging UAV-based imagery. We explore two means of improving detection: a new architecture composed of two parallel and unique backbone networks and a novel training methodology to improve the number of small objects seen during training.

We first propose an architecture called Center R-CNN (CR-CNN) to demonstrate that the Region Proposal Network (RPN) of Mask R-CNN [4] can benefit from additional information captured from a second unique branch. This parallel branch—which predicts the center of an object’s mass—utilizes information different from that in the FPN backbone of Mask R-CNN and allows for additional unique regions of interest (RoIs) to be predicted that capture small objects better.

Secondly, to improve the learned detection of small objects, the RPNs are trained using a new strategy that inflates the intersection over union (IoU) between each object’s anchor–groundtruth pairs. This bolsters the number of small objects utilized during training and is detailed in Section 2.3.

Finally, the performance of CR-CNN is evaluated on synthetic data first and then on two UAV-based datasets to demonstrate the applicability of the two strategies to challenging real-world applications. This is followed by a discussion of the results and future directions of research in Section 4.

Small Object Detection

Generally, R-CNN architectures outperform single-stage methods in detecting small objects [5]. On the MS-COCO dataset, Mask R-CNN achieves higher AP_S scores compared to those of competitive Faster R-CNN models and single-stage models such as YOLOv3 [1]. Improved architectures that are built upon Mask R-CNN [32] have further outperformed architectures such as YOLOv4 [2] and RetinaNet [33] in terms of AP, AP₅₀ and AP_S.

However, shortcomings of the RPN have been acknowledged. The work of [34] demonstrated that the region proposals from all five RPN layers of Faster R-CNN achieve lower average precision scores as the object size decreases. As documented by [35], a driving factor behind the under-performance of modern object detection architectures on small objects is their common reliance on anchor-based regression. While training the Mask R-CNN RPN on the COCO dataset, large objects match an average of 2.54 anchors, while small objects only match an average of 1 anchor per object—the absolute minimum possible—according to the object size definitions of [14]. This suggests that smaller objects are under-represented during training, which results in the classic RPN failing to recognize smaller objects and thereby proposing fewer regions that contain small objects.

Detection limitations are further compounded in aerial-based imagery, where objects are generally smaller relative to the size of images and located within complex environments. For example, small objects in the Vehicle-Detection dataset comprise 80% of all objects, as seen in

Table 1. Object size and density in COCO and aerial datasets.

	Object Size	Number of Objects	Percent of Total Objects	Percent of Total Object Area	Percent of Total Image Area
COCO	Small	267,700	31.13	0.58	0.33
	Medium	300,129	34.90	5.99	3.39
	Large	292,172	33.97	93.44	52.82
Vehicle-Detection	Small	15,478	80.0	66.51	0.61
	Medium	3870	20.0	33.49	0.28
	Large	0	0.0	0.0	0.0
Aerial-Cars	Small	539	16.98	3.98	0.16
	Medium	2415	76.09	60.88	2.50
	Large	220	06.93	35.14	1.44

, yet occupy 0.61% of the total image area in the dataset. The Aerial-Cars dataset follows a similar pattern, with small and medium objects totaling over 90% of all objects yet occupying only 2.66% of the total pixel space. According to occupied area alone, small objects are already at a disadvantage in remote sensing data.

The lack of pixels that a small object occupies creates extra challenges when training anchor-based methods. Within the training pipeline, each of the thousands of generated anchors are utilized as a positive training example if overlapping with a groundtruth object by a certain IoU threshold [1,36]. However, large objects will produce multiple anchors that exceed the threshold and will therefore be seen multiple times during training—creating a large object bias.

When trained on COCO, YOLOv5 was seen to match an average of 1.0 anchor per object less than ${32}^2$ pixels while matching an average of 2.45 anchors per object larger than ${96}^2$ pixels. On Aerial-Cars, YOLOv5 matched an equally poor number of 1.05 anchors per small object and 2.45 anchors per large object. Mask R-CNN also faced similar challenges, matching an average of 1.0 anchor per small object on COCO. This indicates that smaller objects are utilized less often during training, creating a large object bias in anchor-based methods. An approach to bolstering the number of small objects utilized during training would likely improve their detection.

Numerous approaches to improving the performance of detection architectures on aerial-based data have been trialed. The work of [37] explored the use of a super-resolution generative adversarial network to increase the resolution of objects found within the VisDrone dataset [38] before detection by an FPN. Terrail and Jurie [39] utilized hard-mining on aerial-based images to achieve an eight-point increase in the AP compared to that of support vector machine detectors on the VeDAI dataset. Alternatively, the work of [40,41] proposed how tiling of an image could improve the detection of smaller objects. Anchor-less and orientated detectors have also received a lot of attention as a successful way to improve the detection of small objects in remote sensing imagery. This includes the General Gaussian Heatmap Label Assignment (GGHL) [42], the Appearance Sensitivity Detector (ASDet) and its unique feature extractor and dual appearance-aware loss [43] and HRDet with its approach to negating noisy and complex backgrounds and its novel regression function EOIoU [44].

Further approaches include data augmentation and a modified RPN, as proposed by [45], to pool feature maps to improve the detection of small objects in the Munich dataset. The work of [46] divides aerial-based data in the UAVDT dataset into specific domains based on imaging altitude, weather and other environmental characteristics for an increase in the AP of 2.27 points for Faster R-CNN. Additionally, the work of [47] aims to improve the performance of RoIs by incorporating ground sample distance into the detection pipeline. Finally, the work of [48] utilizes frequency-domain-based information to further extract valuable and rare information from aerial-based imagery. However, despite these and various other attempts to improve the detection of objects in UAV-based data, modern detectors continue to struggle with this challenge.

The work of [38] demonstrated that R-CNN-based detectors generally achieve higher detection accuracies on UAV-based data compared to other architectures; Ref. [49] documented the higher performance of Faster R-CNN compared to YOLOv3 and YOLOv4 on the Stanford Drone Dataset. Furthermore, Mask R-CNN demonstrates the superior performance of the R-CNN architecture on smaller objects compared to YOLOv5 [50] and the Detection Transformer (DETR) [51] on a supplemented Aerial-Cars dataset—exceeding both architectures by up to 17.3 points in AP_S. However, despite outperforming other architectures, the R-CNN architecture still fails to meet acceptable performance levels on UAV-based data. Faster R-CNN demonstrated significant under-performance on the low-altitude VisDrone dataset [52] and the Stanford Drone Dataset [49,53]. These results suggest that the R-CNN architecture can be improved upon to meet the challenges posed by small objects such as those in UAV-based imagery better.

The following work will detail the proposed new architecture and a unique training strategy that increases the number of small objects seen during training. This is followed by our results demonstrating the success of CR-CNN in detecting small objects in aerial imagery and the better results of common detection architectures when they are trained using our inflated IoU training strategy.

2. Materials and Methods

2.1. Datasets

CR-CNN was trained and evaluated on datasets comprising both real-world and synthetic data. We created the synthetic dataset as a simple and standard dataset with minimal noise and other controlled variables in order to form a baseline for the proposed methodologies. Exploratory evaluation on the synthetic data determined whether the expected outcome of both strategies was met before their application to more complex and challenging real-world data. The synthetic data consist of multiple colored shapes on different colored backgrounds, with the inclusion of some challenging and obscure objects, such as overlapping shapes. The synthetic training dataset comprises 1500 images totaling 33,520 objects—24% of which are smaller than ${32}^2$ pixels and 69.58% of which are smaller than ${96}^2$ pixels. The evaluation dataset comprises 150 images totaling 6786 objects, with 1651 shapes smaller than ${32}^2$ pixels.

For real-world aerial data, we trained and evaluated our proposed architecture on two datasets—the Aerial-Cars dataset [54] and the Vehicle-Detection dataset [55]—chosen for their challenging characteristics compared to other common UAV-based datasets [38,53,56]. Both include multiple camera angles, altitudes and complex backgrounds, which accurately capture the challenging nature of aerial-based data.

The Aerial-Cars dataset was further annotated with an additional 546 vehicles—primarily located closer to the horizon or partially obstructed—to provide a more realistic environment for stress-testing the detection architectures. The Aerial-Cars training dataset contains 3174 objects—of which 539 are smaller than ${32}^2$ pixels—within 155 images, with an average image size of $[1777,987]$. The evaluation dataset consisted of 33 images, an example of which can be seen in Figure 1.

The Vehicle-Detection training dataset (sub-datasets 1, 2 and 3) comprised 1495 images that contained 19,348 objects—only car annotations were used for training and evaluation. Of these 19,348 objects, 80% were smaller than ${32}^2$ pixels. The evaluation dataset (subset 4) comprised 190 images containing a total of 3647 objects—3325 of which were smaller than the COCO-defined $32\times 32$ pixel definition. The distribution of object size within the Vehicle-Detection dataset, with a comparison to the synthetic and Aerial-Cars data, can be seen in Figure 2.

Finally, the proposed inflated IoU training strategy was used to train and evaluated on the 2017 COCO dataset [14]. While it is not an aerial-based dataset, this comparison allowed for this novel approach to be evaluated for use in other applications. The detection results of our proposed architectures compared to those of existing methods can be seen in Section 3.

2.2. Architecture

2.2.1. Mask R-CNN

The base Mask R-CNN architecture is divided into two stages, with the first stage utilizing an FPN and an RPN to predict RoIs in the form of rectangular object proposals—each with an objectness score [4]. This first stage can be seen within the left-most red-hatched box in Figure 3. To generate these proposals, a small convolutional layer is slid across the last shared layer of the FPN backbone before two parallel $1\times 1$ convolutions are applied for box regression and box classification. Each sliding window location predicts k proposals with their coordinates predicted relative to 15 differently sized anchor boxes [4]. This first-stage architecture is heavily based off the work of [36].

Following the prediction of the RoI proposals, the second stage of Mask R-CNN performs classification, localization and segmentation. From each RoI, a small feature map is extracted by RoIAlign—allowing for each floating-number RoI to accurately represent a region from the discrete feature maps of the FPN [4]. Having extracted a small $7\times 7$ feature map from each RoI, the network head extends the work of [36] and [57] and introduces a third parallel convolution branch to predict the segmentation masks—a feature not utilized by this work due to the lack of segmented aerial-based data. This second stage, including the classification and box regression prediction branches, can be seen in the right-most red-hatched box in Figure 3.

The performance of Mask R-CNN is heavily dependent on the performance of the RPN—object locations missed during the region proposal stage will never be detected by the prediction heads. The FPN-based RPN has demonstrated successful results in detecting medium and large objects that should be retained while aiming to improve the performance of Mask R-CNN on small objects. A second parallel RPN branch can instead supplement the region proposals by predicting RoIs from convolutional information that is distinct from that from the FPN backbone. This architecture can be seen in Figure 3.

The R-CNN approach [58] sequentially predicts the region proposals before predicting objects in a two-stage fashion. Faster R-CNN [36] improved upon earlier methodologies by adopting the attention mechanism of [59] in the RPN. Mask R-CNN [4] further built upon the results of [36] through the adoption of an FPN backbone [57] and a parallel third prediction branch for the successful prediction of segmentation masks. The R-CNN architecture can also be easily improved upon: the PANet paper [32] demonstrated improved performance through using an additional bottom-up backbone pathway and the fusion of FPN feature maps.

The RPN of [36] predicts a set of region proposal bounding boxes with equivalent objectness scores from a given image. A small convolutional network acts in a sliding-window fashion across each hierarchical layer of the backbone—five layers in the case of Mask R-CNN. Each sliding-window location predicts multiple proposals as offsets of fixed-size anchors with three aspect ratios and N scales. Mask R-CNN uses $N=5$ in accordance with the five FPN scales and therefore utilizes 15 anchors at each sliding-window location. During training, the RPN is only trained on objects whose groundtruth annotations, with coordinates $t_i^{*}$, overlap with an anchor by more than 0.7 following (1):

$$\begin{array}{cc}\hfill L(p_i,t_i)& ={\displaystyle \frac{1}{N_{cls}}}\sum _i{L}_{cls}(p_i,p_i^{*})\hfill \\ \hfill & +{\displaystyle \frac{\lambda }{N_{reg}}}\sum _i{p}_i^{*}{L}_{reg}(t_i,t_i^{*})\hfill \end{array}$$

(1)

where anchor–object pairs with an IoU greater than 0.7 are assigned a positive groundtruth label $p_i^{*}=1$ and pairs with an IoU of less than 0.3 are assigned a negative label $p_i^{*}=0$ following [36]. The label $p_i^{*}$ ensures that the regression loss $L_{reg}$ is only computed for positive training labels, meaning the RPN learns to better propose regions only when objects have significant IoU overlap with any anchor. Following [36], the classification loss $L_{cls}$ is a binary log loss between the predicted objectness scores $p_i$ and the groundtruth label $p_i^{*}$. $L_{reg}$ is the robust loss function of [60] between the vector $t_i$ representing the four box coordinates of the predicted region proposal and the vector $t_i^{*}$ of the coordinates of the respective groundtruth object according to index i. The two terms are normalized by the constants $N_{cls}$ and $N_{reg}$ and are equally weighted by $\lambda$ [36].

2.2.2. CR-CNN

The work of [61,62,63] introduced object detection methods utilizing keypoints predicted from objectness heatmaps rather than bounding box anchors; all three approaches first generate heatmap feature maps using a stacked hourglass network [64]. The secondary RPN branch of CR-CNN treats the prediction of region proposals similar to these keypoint-based works—by first extracting objectness heatmaps of an image before predicting the location of the RoIs. In this manner, the center of a single predicted RoI can be treated as a central keypoint, predicted from a heatmap solely focused on the existence of an object without focusing on the object’s finer details.

CR-CNN downsamples the input image once by a stride of two, and following the work of previous keypoint-based approaches, the objectness heatmap is extracted using two consecutive stacked hourglass modules, with each module first reducing and subsequently upsampling the image five times. For an input image $I\in {\mathbb{R}}^{W\times H\times 3}$, the stacked hourglass network produces a heatmap $\widehat{Y}\in {[0,255]}^{{\displaystyle \frac{W}{R}}\times {\displaystyle \frac{H}{R}}\times C}$ where $R=4$ is the stride of the output heatmap, following other heatmap-based detectors [61,62,65,66]. The heatmap strictly predicts the existence of an object and does not take an object’s classification into consideration; therefore, the CR-CNN heatmap has an output dimension of $C=1$. This is implemented within the network as a $1\times 1$ convolution following the stacked hourglass network with an output dimension of one. Regions of the heatmap that have values near 255 are considered to be regions likely to contain an object.

During training, the stacked hourglass network is trained using a penalty-reduced pixel-wise logistic regression loss function with focal loss modified for a single category $C=1$ [61,63] as follows:

$$L_{heat}={\displaystyle \frac{-1}{N}}\sum _{xy}\left\{\begin{array}{cc}{(1-{\widehat{Y}}_{xy})}^{\alpha }log({\widehat{Y}}_{xy}),\hfill & \mathrm{if}{Y}_{xy}=255\hfill \\ \begin{array}{c}(1-Y_{xy}{)}^{\beta }{({\widehat{Y}}_{xy})}^{\alpha }\hfill \\ log(1-{\widehat{Y}}_{xy})\hfill \end{array}\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(2)

for N keypoints within the image I where $\widehat{Y}$ is the predicted heatmap and Y is the groundtruth heatmap—both of resolution $x\times y$. The focal loss values $\alpha =2$ and $\beta =4$ follow those of [33].

Each image’s equivalent groundtruth heatmap $Y\in {[0,255]}^{{\displaystyle \frac{W}{R}}\times {\displaystyle \frac{H}{R}}\times 1}$ is composed of objects splat onto the heatmap using a Gaussian kernel as follows:

$$Y_{xy}=exp({\displaystyle \frac{-{(x-{\tilde{p}}_x)}^2+{(y-{\tilde{p}}_y)}^2}{2{\sigma }_p^2}})$$

(3)

with low-resolution object center keypoints ${\tilde{p}}_x=\lfloor {\displaystyle \frac{x}{R}}\rfloor$ and ${\tilde{p}}_y=\lfloor {\displaystyle \frac{y}{R}}\rfloor$ at an $\{x,y\}$ coordinate of the heatmap, where ${\sigma }_p$ is the object-size-dependent standard deviation calculated to be ${\displaystyle \frac{1}{3}}$ the size of the object’s width or height, whichever is greater [61]. Following [65], the element-wise maximum of their kernels is taken when two objects overlap.

Two RoI prediction heads are considered in order to reinforce that the additional RPN supplementation is more influential than the specific RoI prediction head architecture itself. A prediction head closely mirroring the work of [63] treats RoI predictions as a regression problem—predicting RoI centers, offsets and sizes—and is referred to as CR-CNN-off throughout this work. A second simplified head more closely follows the RPN structure of [36] but instead predicts RoIs in absolute coordinates and is referred to as CR-CNN-abs.

CR-CNN-off: Inspired by [63], the RoI centerpoint, offset and size values are predicted from the stacked hourglass heatmap. A local offset value $L_{off}$ is predicted to offset the discretization error introduced by the output stride R in the calculation of ${\tilde{p}}_x$ and ${\tilde{p}}_y$. The offset predictions are trained using an L1 loss as follows:

$$L_{off}={\displaystyle \frac{1}{N}}\sum _p|{\widehat{O}}_{\tilde{p}}-({\displaystyle \frac{p}{R}}-\tilde{p})|$$

(4)

for the predicted local offset values $\widehat{O}$ from each object centerpoint. The $({\displaystyle \frac{p}{R}}-\tilde{p})$ term ensures the loss term is only applied when an object exists within the groundtruth heatmap Y. The offset values are predicted in the form ${\widehat{O}}_{{\widehat{x}}_k,{\widehat{y}}_k}=(\delta {\widehat{x}}_k,\delta {\widehat{y}}_k)$ for object k.

The RoI size predictions are trained following the L1 loss:

$$L_{size}={\displaystyle \frac{1}{N}}\sum _{k=1}^N|{\widehat{S}}_{pk}-s_k|$$

(5)

where the object size ${\widehat{S}}_{pk}$ is the predicted width and height—the dimensions of the region proposals in the case of CR-CNN—measured against the groundtruth object size $s_k=(x_2^{(k)}-x_1^{(k)},y_2^{(k)}-y_1^{(k)})$ for object k of N objects.

Following [63], both $L_{off}$ and $L_{size}$ are predicted alongside the centerpoint coordinates from the output of the stacked hourglass network utilizing identical $3\times 3$ convolutions, followed by a ReLU layer and additional $1\times 1$ convolution. These operations are performed by the independent RPN within the blue-hatched border in Figure 3.

During training, the supplemental RPN branch is trained as follows:

$$L_{rpn}=L_{heat}+{\lambda }_{size}{L}_{size}+{\lambda }_{off}{L}_{off}$$

(6)

where the loss function comprises the heatmap loss $L_{heat}$ defined in (2), the size loss $L_{size}$ and the offset loss $L_{off}$. The size and offset loss values are scaled by the values ${\lambda }_{size}=0.1$ and ${\lambda }_{off}=1$ following [63].

CR-CNN-abs: The supplemental RoIs are again predicted from an objectness heatmap $\widehat{Y}$—extracted with an identical stacked hourglass network to that in CR-CNN-off. The RoI prediction heads closely follow the RPN architecture of [36] for simplicity—two parallel fully connected layers predicting the bounding box coordinates and objectness scores from $\widehat{Y}$. The supplemental RPN branch is trained as follows:

$$\begin{array}{c}\hfill L_{rpn}=L_{heat}+{\displaystyle \frac{1}{N_{cls}}}\sum _i{L}_{cls}(p_i,p_i^{*})\\ \hfill +{\displaystyle \frac{\lambda }{N_{loc}}}\sum _i{p}_i^{*}{L}_{loc}(v_i,v_i^{*})\end{array}$$

(7)

where the localization loss $L_{loc}$ is the robust loss function of [60] between the predicted RoI of coordinates $v_i$ and the groundtruth objects with absolute coordinates $v_i^{*}=\{v_x^{*},v_y^{*},v_w^{*},v_h^{*}\}$. The use of absolute coordinates avoids the use of anchors by the supplemental branch. Following [36], the classification loss $L_{cls}$ is a binary log loss between the predicted objectness score $p_i$ and the binary training label $p_i^{*}$ assigned to each RoI based on the IoU overlap between each RoI–groundtruth pair. The heatmap loss $L_{heat}$ is again calculated following (2).

2.3. Inflated IoU Training

CR-CNN addresses the shortcomings of the RPN through the prediction of new region proposals using information distinct from that of the FPN. However, this method fails to improve upon the classic RPN in [4], which will still fail to recognize small objects, as documented by [34,35]. Addressing this problem through a new training strategy allows for the standard RPN architecture to detect regions containing smaller objects better in unity with the new branch of CR-CNN while minimizing increases in the computation or implementation costs.

A number of different IoU losses have been proposed, including GIoU [67], DIoU and CIoU [68]; however, regardless of the IoU-based loss utilized, smaller objects will generally be assigned positive labels at lower rates than larger objects. We trained CR-CNN using a modified training strategy that artificially inflates the IoU of object–anchor pairs for smaller objects within the dataset. The increased IoU allows for more small objects to be assigned positive training labels by surpassing the 0.7 IoU threshold and increases the number of small objects utilized while training the RPN. This achievement results in improved detection of similar smaller objects during evaluation. While data augmentation strategies such as [35,69,70] frame the problem of poor detection performance on small objects as a general lack of small objects during training, we define the problem as a lack of objects utilized during training. This approach avoids the shortcomings introduced by data augmentation methods such as the existence of objects out of context relative to the background or stark changes in illumination.

According to the $p_i^{*}{L}_{reg}(t_i,t_i^{*})$ term from (1), an object only contributes to the training of the RPN when an anchor is assigned a positive training label $p_i^{*}=1$. Every anchor is assigned a binary training label, with an anchor assigned a positive label in two scenarios: (i) the anchor has the highest IoU overlap with a groundtruth box or (ii) the anchor has an IoU overlap greater than 0.7 with any groundtruth box [36]. Our approach specifically targets rule (ii)—which allows a groundtruth box to assign a positive training label to multiple anchors—and aims to increase the number of small objects within the dataset that surpass the 0.7 IoU threshold of CR-CNN. Figure 4 illustrates the relation between the overlaid anchor proposals and the groundtruth box for a single object and the typically low IoU as a result. By inflating the IoU of smaller objects with each anchor, we effectively increase the size of the object (enlarging the red annotation box) or decrease the size of the anchor (dashed green boxes) in order to produce IoUs greater than the threshold value.

Our proposed method simply multiplies the IoU of every anchor–groundtruth pair using a scaling value $S_{IoU}$ determined by the Gaussian function

$$S_{IoU}=a\ast exp(-{\displaystyle \frac{{(x-b)}^2}{2{\sigma }^2}})$$

(8)

where a and $\sigma$ control the scaling magnitude and range of object size that the IoU inflation is applied to, while b dictates the object size that the largest scaling factor is applied to based on the object bounding box area x. Increases in $\sigma$ will allow for more objects to have their IoU artificially increased, while decreasing $\sigma$ will allow for the scaling to be focused on a more specific size of object. It was found that the overall detection began to suffer when the $\sigma$ values were too small or too large, impacting large object and small object detection, respectively. It was also found that that we could influence the detection accuracy based on object size by shifting the center point of the Gaussian using hyperparameter b. However, at a certain point, as the value of b increased, the performance gains decreased, as would be expected. This work used the values $a=0.3$, $b=1000$ and $\sigma =750$ but did not pursue a thorough optimization of these values beyond demonstrating improved detection results. The new inflated IoU is then calculated as follows:

$$Io{U}_{inflated}=Io{U}_{old}\ast (1+S_{IoU})$$

(9)

where $Io{U}_{old}$ is the base IoU of each object–anchor pair, and $Io{U}_{inflated}$ is the new IoU used to assign the binary training label $p_i^{*}$ during RPN training. In implementation, this process can be achieved with two efficient array calculations—calculating a Tensor of the scaling values from a Tensor of the object areas first before then multiplying each optimal IoU by its related scaling factor. During training, there was no notable impact on the training time; however, this would be a valuable consideration in future work.

The use of the Gaussian function was found to allow for the IoU scaling to be applied to objects within a specific size range better compared to step, square and sigmoid rules, as demonstrated in Figure 5. Each of the different curves was chosen for its ability to isolate the IoU inflation to smaller objects. The detection accuracy of Mask R-CNN on the Aerial-Cars data trained with the various rules can be seen in

Table 2. Detection accuracy of Mask R-CNN and YOLOv5 on Aerial-Cars trained utilizing various inflation curves.

(a) Mask R-CNN
	Gaussian	Step	Square	Sigmoid
AP50	76.4	72.6	73.6	73.2
AP	41.0	39.6	39.8	38.7
APS	38.7	38.5	38.0	35.3
APM	41.2	39.7	40.5	41.0
APL	57.1	52.0	53.2	51.4
(b) YOLOv5
	Gaussian	Step	Square	Sigmoid
AP50	72.0	62.3	58.5	61.1
AP	35.7	30.0	28.9	27.6
APS	32.6	28.9	22.8	26.1
APM	36.6	26.4	24.1	29.0
APL	36.5	32.8	26.8	28.4

. The results in the table represent the best performance found for each of the scaling rules. The Gaussian function is believed to outperform the other approaches due to the function allowing for objects that fall within the center of the size range to have their IoU scaled by a larger value than objects farther from the center, which permits the inflation to be applied to specific datasets better.

While the training strategy was used with CR-CNN to improve the detection of smaller objects by the classical RPN, the generality of this approach allows for the method to be applied to any anchor-based detection architecture. To demonstrate this, the training strategy was easily adapted to the YOLOv5 [50] architecture, which follows the anchor labeling rules of [36], but the overlap of the anchors and groundtruth objects was compared using a width ratio $r_w$ and a height ratio $r_h$ calculated as follows:

$$r_w={\displaystyle \frac{w_{gt}}{w_{anch}}}, r_h={\displaystyle \frac{h_{gt}}{h_{anch}}}$$

(10)

$$r_w^{max}=max(r_w,{\displaystyle \frac{1}{r_w}}), r_h^{max}=max({\displaystyle \frac{1}{r_h}})$$

(11)

$$\begin{array}{c}{r}^{max}=max(r_w^{max},r_h^{max})\end{array}$$

(12)

where the groundtruth annotation width $w_{gt}$ and height $h_{gt}$ are compared to the anchor box width $w_{anch}$ and height $h_{anch}$. The maximum among both ratios and their inverse is computed as $r^{max}$, and any anchor–groundtruth pair whose $r^{max}$ is less than $4.0$ is assigned a positive training label. This allows for a scaling value to still be calculated following (8) and easily applied following (9) where $Io{U}_{old}$ is the originally calculated $r^{max}$ value and $Io{U}_{inflated}$ is the inflated $r^{max}$, which, again, is compared to the $4.0$ threshold. The general applicability of this approach is explored in Section 3 with a comparison of CR-CNN, Mask R-CNN and YOLOv5 as trained using the inflation strategy compared to using a typical data augmentation strategy.

Throughout this work, CR-CNN was implemented using Detectron2 [71]. Both the Aerial-Cars and Vehicle-Detection datasets include only bounding box annotations of vehicles and not segmentation masks; therefore, within the scope of this work, Mask R-CNN can be considered an improved form of Faster R-CNN. The architecture only predicts bounding boxes using five FPN scales and RoIAlign without the segmentation prediction branch. Our Mask R-CNN implementation also included the additional bottom-up pathway and the adaptive feature pooling from [32] and was implemented using Detectron2. All models that incorporated an FPN backbone utilized a ResNeXt-101 model pre-trained on ImageNet. Each model was trained for the standard 300 epochs except for DETR, which was trained using 500 epochs, as in [51]. Besides DETR, all the other models were found to stabilize around the 300-epoch mark. All the other parameters followed the work of [4,50], and default anchor boxes were used.

3. Results

3.1. Architecture-Derived Improvements

CR-CNN is intended to improve the performance of modern object detection architectures on small objects, such as the challenge of detection in UAV-based imagery. The images within the COCO dataset have average dimensions of $640\times 480$ pixels and an average area of $307,200$ square pixels. Indeed, [14] defines a small object as one with an area less than 1024 square pixels—equivalent to an area $1/300$th of the size of a full image. Alternatively, the UAV-based datasets have average dimensions of $1777\times 987$ pixels and an average area of over 1.7 million square pixels. For an object to occupy the equivalent of $1/300$th of a UAV-based image, it would need an area of approximately 5800 square pixels or an average area of $76\times 76$. As shown in Figure 2, the majority of objects across both aerial datasets are less than 6000 square pixels. This suggests the AP₅₀ and AP metrics of the models evaluated are better measures of the architectures’ ability to detect small objects than the MS-COCO AP_S score on these aerial datasets.

The CR-CNN architecture shows consistent improvement over Mask R-CNN, YOLOv5 and DETR, as seen in

Table 3. CR-CNN’s results on synthetic data.

	AP	AP50	APS	APM
Mask R-CNN	55.9	76.1	53.8	72.0
YOLOv5	60.3	72.5	63.7	73.5
CR-CNN-abs	60.8	77.0	60.4	73.1
CR-CNN-off	57.2	76.6	60.1	73.5

All models are trained using the inflated IoU strategy.

and

Table 4. CR-CNN’s results on UAV-based data.

(a) Vehicle-Detection
	AP	AP50	APS	APM
Mask R-CNN	26.9	47.4	31.5	20.8
YOLOv5	18.6	39.0	20.1	8.3
DETR	19.0	50.1	19.1	23.2
CR-CNN-abs	29.8	56.9	34.0	19.4
CR-CNN-off	28.4	54.7	33.4	19.3
(b) Aerial-Cars
	AP	AP50	APS	APM	APL
Mask R-CNN	40.2	73.7	38.2	40.4	51.5
YOLOv5	23.0	55.0	20.9	24.2	6.9
DETR	38.2	74.8	35.8	38.9	33.6
CR-CNN-abs	46.6	78.9	43.8	43.6	52.7
CR-CNN-off	46.0	77.4	41.1	43.8	52.7

All models are trained using the inflated IoU strategy.

. On the synthetic data in Table 3, CR-CNN outperforms Mask R-CNN in all metrics, including more than a 6-point increase in AP_S, which demonstrates improved detection of objects smaller than 1024 square pixels—the majority of the objects in the synthetic data. CR-CNN also demonstrates an improved ability to accurately localize objects compared to YOLOv5 based on its higher AP score—a particularly challenging task when detecting primarily smaller objects.

On the two UAV-based datasets seen in Table 4 and

Table 5. Composition of CR-CNN predictions on UAV data.

	Vehicle-Detection			Aerial-Cars
	TP	FP	FN	TP	FP	FN
Mask R-CNN	1563	94	2048	850	36	286
CR-CNN-abs	1794	88	1853	881	33	255
CR-CNN-off	1790	86	1857	879	32	257

A prediction is considered to be positive at an IoU greater than 0.7 and false when the prediction’s IoU is less than 0.7.

, CR-CNN outperforms Mask R-CNN, YOLOv5 and DETR in almost every metric Both the absolute and regression prediction heads outperform the other architectures in AP₅₀, demonstrating better detection accuracy. Similarly, the higher AP metrics achieved by the two models show that CR-CNN is not only better at detecting small objects but also at accurately localizing them within large images (Figure 6 and Figure 7).

3.2. Inflation-Derived Improvements

Similar to the evaluation of CR-CNN, the inflated IoU strategy was evaluated on synthetic data, COCO and the same aerial-based datasets, with the results compared to standard Mask R-CNN and YOLOv5, along with the data augmentation strategy of [35] as seen in

Table 6. Inflated IoU training applied to standard detection architectures on synthetic data.

	AP	AP50	APS	APM
Mask R-CNN †	55.0	74.4	53.2	71.3
Mask R-CNN	55.9	76.1	53.8	72.0
Mask † + C&P	52.6	71.2	49.3	70.0
YOLOv5 †	60.1	71.2	58.9	72.9
YOLOv5 † + C&P	34.3	39.5	34.2	44.6
YOLOv5	60.3	72.5	63.7	73.5

C&P refers to the copy-and-paste method from [35], copying some of the objects within an image three times. ^† Models trained without the inflation strategy.

.

Evaluating the inflated IoU approach on the COCO dataset showed its impact on detecting objects of more varied sizes and in a different application. These results can be seen in

Table 7. Inflated IoU training applied to standard detection architectures on COCO 2017.

	AP	AP50	APS	APM	APL
Mask R-CNN †	42.0	62.5	25.2	45.6	54.6
Mask R-CNN	42.2	63.1	26.3	45.7	54.5
Mask † + C&P	42.0	62.8	25.5	45.5	54.5
YOLOv5 †	48.9	67.5	31.8	54.4	62.3
YOLOv5 † + C&P	49.2	67.4	31.9	54.2	62.4
YOLOv5	50.0	67.9	31.9	56.1	61.8

C&P refers to the copy-and-paste method from [35], copying some of the objects within an image three times. ^† Models trained without the inflation strategy.

, and while segmentation was not a focus of this work, for completeness, the inflated IoU method was evaluated on the segmentation predictions of Mask R-CNN, which can be seen in

Table 8. Inflated IoU training applied to Mask R-CNN segmentations on COCO 2017.

	AP	AP50	APS	APM	APL
Mask R-CNN †	38.0	58.3	19.8	41.2	50.1
Mask R-CNN	39.0	58.8	21.1	41.4	50.1
Mask † + C&P	38.1	58.5	19.8	41.3	50.2

C&P refers to the copy-and-paste method from [35], copying some of the objects within an image three times. ^† Models trained without the inflation strategy.

.

The evaluation of our proposed training strategy against the copy-and-paste augmentation from [35] was intended to examine two very different training approaches. The augmentation strategy produces varying results based on the architecture and dataset in question, and this is believed to be due to the variable impact on training that each copied object produces. Objects and their anchors that do not usually meet the IoU threshold to qualify as a valid training example are likely to produce copies that also fail to meet the threshold—but these copied objects then increase the clutter and complexity of the image, creating an ever more challenging detection environment. The impact of this will depend on both the dataset in question and how the prediction architecture handles complex and busy images.

While the Gaussian curve was found to improve the detection best—and was therefore the inflation rule evaluated within this section—each of the inflation rules may prove advantageous in different scenarios. The step rule would allow for a precise maximum object size to which inflation is applied to while equally inflating the IoU of all objects below this size. For datasets with very polarized object sizes and detection results, where large objects share a common size and excellent detection rate and small objects share a separate size and poor detection rate, then the step rule might be more suitable. Depending on the distribution of size, a square or sigmoid inflation rule might also be appropriate in that situation.

4. Discussion

Overall, the results of Table 4 and Table 5 support the conclusion that an objectness heatmap can successfully improve the RoI proposals of the Mask R-CNN architecture. The extraction of these additional RoIs by CR-CNN directly leads to a higher object detection accuracy in aerial imagery and fewer incorrect predictions compared to the base Mask R-CNN implementation. Additionally, CR-CNN demonstrated improved performance compared to YOLOv5 and DETR on both aerial datasets.

The advantage of CR-CNN is that object proposal prediction—the foundation of all correct or incorrect predictions—occurs from two unique approaches. While it is inevitable that the two branches will predict a large number of identical proposals, each will also produce a number of unique proposals—creating a better foundation for detecting objects that are usually missed. Additionally, the keypoint-based branch is tasked with the simplest form of detection—simply predicting whether an object does or does not exist—while providing valuable information to the second stage of the network. Meanwhile, approaches such as GANs and super-resolution approaches perform complex object prediction on enhanced, or already predicted, images—compounding incorrect predictions.

Comparison of the performance of equivalent CR-CNN architectures when trained with and without the inflated training, as in

Table 9. Inflated IoU training applied to standard detection architectures on real-world aerial data.

(a) Vehicle-Detection
	AP	AP50	APS	APM
Mask R-CNN †	25.6	46.1	28.6	19.0
Mask † + C&P	25.4	46.3	28.7	19.2
Mask R-CNN	26.9	47.4	31.5	20.8
YOLOv5 †	17.2	37.8	18.2	7.1
YOLOv5 † + C&P	16.9	37.9	18.8	7.9
YOLOv5	18.6	39.0	20.1	8.3
(b) Aerial-Cars
	AP	AP 50	APS	APM	APL
Mask R-CNN †	40.2	73.7	38.2	40.4	51.5
Mask † + C&P	40.3	74.4	37.0	40.8	48.6
Mask R-CNN	41.0	76.4	38.7	41.2	57.1
YOLOv5 †	23.0	55.0	20.9	24.2	6.9
YOLOv5 † + C&P	36.0	71.9	31.9	36.8	39.6
YOLOv5	35.7	72.0	32.6	36.6	36.5

C&P refers to the copy-and-paste method from [35], copying some of the objects within an image three times. ^† Models trained without the inflation strategy.

, reveals that the training strategy plays only a small role in improving the accurate detection of smaller objects. On the synthetic data, the training strategy leads to increases in AP_S of up to 1.2 points. On the UAV datasets, the strategy increases the AP₅₀ score by up to 1.1 points. This suggests that the additional RoI predictions from the second RPN branch are the primary drivers of improving the detection of smaller objects.

The improved detection of small objects by CR-CNN can be understood better from the composition of the architecture’s predictions on the UAV data, as shown in Table 5. The architecture introduced achieves two improvements: falsely predicting the existence of objects at a lower rate compared to Mask R-CNN—represented as fewer false positives (FPs)—and predicting the correct existence of more objects compared to Mask R-CNN, as shown by the higher number of true positive predictions (TPs). This supports the idea that the additional branch produces high-quality proposals and reduces the number of poor proposals from the original Mask R-CNN’s first stage that are utilized in the final predictions.

While CR-CNN was trained using the IoU inflation strategy in order to improve the performance of the classical RPN on smaller objects, this strategy could easily be adapted to any anchor-based architecture, such as an R-CNN or YOLO architecture. An inflated Mask R-CNN and YOLOv5 achieve an improved detection accuracy over a baseline model on simple synthetic data, as shown in Table 6, and on more complex aerial data, as shown in Table 9. On the COCO dataset, Mask R-CNN assigns an average of 1.00 anchors to each small object but assigns an average of 1.12 anchors per object when trained using our inflation strategy. YOLOv5 demonstrates similar improvements on the Aerial-Cars data by increasing the average number of small object–anchor pairs labeled as positive examples from 1.00 to 1.05. These results are summarized in Figure 8. The inflation strategy can also achieve higher average precision scores compared to data augmentation strategies such as [35]. Despite the large differences between R-CNN and YOLO detectors, the inflated training strategy produces an improved performance for both architectures. These results confirm that increasing the number of objects seen during training will lead to improved detection accuracy.

The inflated IoU approach also increased the detection of medium and large objects on both aerial datasets and increased the accurate prediction of medium but not larger objects on the COCO dataset, as shown in Table 7. This suggests that there is an upper limit on the object size for which the inflated IoU training produces improved results. While the aerial datasets have larger objects, their size is still often dwarfed by some of the largest objects within COCO. This would be an important consideration for utilizing the inflation strategy in different applications.

UAV-based imagery is a challenging environment for state-of-the-art detectors due to its high proportion of small objects, which are often dwarfed by the background and comprise only small portions of an image. We proposed an R-CNN architecture that predicted additional RoIs from an objectness heatmap which could extract the existence of small objects better. This two-branch RPN was then trained using a strategy that increased the visibility of smaller objects to improve the detection of small objects by an R-CNN architecture further. With the ever-increasing accessibility of UAVs, this work evaluated the improved performance of CR-CNN on aerial-based data as a demonstration of the real-world applicability of this architecture.

The success of the CR-CNN method suggests that the classical RPN of R-CNN architectures could be supplemented in any number of ways with further research. While CR-CNN demonstrated improved performance, it was almost certainly at the cost of increased computational complexity. While this was not a focus of this work, it would be a valuable direction of future research to explore simpler algorithms, including real-time approaches, inspired by this dual-proposal approach. Additionally, CR-CNN was tested only on aerial-based datasets, and it would be beneficial to further benchmark the proposed method on other real-world data, such as CCTV footage, to explore the general applicability of methods utilizing dual first-stage prediction branches. Additionally, IoU inflation is only one method for increasing the visibility of under-represented objects while training detection models and further work in this area would be beneficial. We did not pursue rigorous optimization of the parameters for this approach either, and the methodology could likely be optimized. However, this approach highlights that all components of the object detection pipeline should remain important aspects of future research.