AF-OSD: An Anchor-Free Oriented Ship Detector Based on Multi-Scale Dense-Point Rotation Gaussian Heatmap

Abstract

Due to the complexity of airborne remote sensing scenes, strong background and noise interference, positive and negative sample imbalance, and multiple ship scales, ship detection is a critical and challenging task in remote sensing. This work proposes an end-to-end anchor-free oriented ship detector (AF-OSD) framework based on a multi-scale dense-point rotation Gaussian heatmap (MDP-RGH) to tackle these aforementioned challenges. First, to solve the sample imbalance problem and suppress the interference of negative samples such as background and noise, the oriented ship is modeled via the proposed MDP-RGH according to its shape and direction to generate ship labels with more accurate information, while the imbalance between positive and negative samples is adaptively learned for the ships with different scales. Then, the AF-OSD based on MDP-RGH is further devised to detect the multi-scale oriented ship, which is the accurate identification and information extraction for multi-scale vessels. Finally, a multi-task object size adaptive loss function is designed to guide the training process, improving its detection quality and performance for multi-scale oriented ships. Simulation results show that extensive experiments on HRSC2016 and DOTA ship datasets reveal that the proposed method achieves significantly outperforms the compared state-of-the-art methods.

1. Introduction

Ship detection is a highly regarded topic in remote sensing. The precise identification of ships through remote sensing images is a crucial task in the field of target recognition. Typically, ships are located offshore or near coasts, leading to a lot of similarities in the imagery. Despite this, accurately determining the size, position, and orientation of ships remains a daunting challenge due to the intricate nature of remote sensing scenarios and the varying sizes of ships. With the development of imaging hardware, remote sensing images have higher resolution, so the method of ship detection based on remote sensing images has been widely studied in various fields of marine supervision [1,2,3], port traffic flow [4,5], ship reconnaissance and statistics [6,7], etc.

For traditional ship detection methods [1,8], the common problems are poor system robustness, high leakage rate in complex scenarios, and low detection accuracy. With the development of deep learning in various applications, deep learning-based methods [9,10,11] can provide automated, high-precision, and high-accuracy results for target detection from remote sensing images. Due to the specificity of remote sensing, there are still some open problems: (1) the bounding box may contain much background in the selected area, and cannot accurately represent the position and direction of ships [12]; (2) false detection because of the small and densely distributed ships [6]; (3) difficulty of recognizing multi-scale ships [13]. Aiming at these problems, some anchor-based methods have been developed [2,13,14]. However, to ensure satisfactory detection accuracy, these methods usually require the manual design of pre-selected boxes according to the actual scene, hindering their applications in practice because of a large number of hyper-parameters and high computational complexity.

To solve the problems of anchor-based detection methods, anchor-free methods have been further proposed [6,15,16,17], which do not require the anchor parameters and can directly predict the class and location information of objects. Thus, these methods can avoid hyper-parameters and reduce computational processes, gradually attracting the extensive attention of researchers. The CenterNet models [6,9,16] have been built based on the center point of the object as a positive sample point to represent an oriented object, as shown in Figure 1a. However, if a target is represented by only one point, or if the shape characteristics of the oriented target are not considered. These methods fail to balance the number of positive and negative samples well, which also causes some positive locations to be misjudged as negative samples.

Aiming at the misjudgment problem of balancing the number of positive and negative samples, dense points-based methods are further proposed to increase the number of positive samples to achieve better detection performance. Zinelli et al. [18] proposed a dense point method to represent and predict the oriented objects, thus optimizing the imbalance of positive and negative samples to some extent. Nevertheless, the number of dense points is different with different scales of objects, as shown in Figure 1b. Specifically, the small objects usually contain fewer dense points than the large ones, and the contribution of the small objects to the loss function is small and easily ignored by the optimizer during training. Moreover, different locations of positive sample points exhibit different effects on the prediction results of the same object. The richer the object features extracted from the sample points close to the object center, the greater the impact on the prediction. However, these methods can not suppress background and noise interference well, which may result in the same confidence level for the feature points at the target centroid and edges, thus generating worse prediction results after non-maximum suppression (NMS), as shown in Figure 2. Additionally, these methods do not take the shape of the target into account, which may cause some negative locations to be misallocated as positive samples, such as the two ends of a ship.

To tackle the misallocation problem, some methods shrink the object bounding box to obtain the core region and take the core region as a positive location [18,19], while the other regions of the object bounding box are the transition from the positive location to the negative location. In this way, the problem of mislabeling can be alleviated to a certain extent. However, it fails to reflect the shape and direction characteristics of the oriented ships. Gaussian rotation heatmaps [6,20] have been used as the supervisory information to distinguish the positive and negative positions of oriented ships. Ref. [20] proposed a target detection method by adopting a Gaussian rotation heatmap as prior information and an adaptive weight-adjustment mechanism (OWAM) algorithm to weight the positive and negative samples at different positions. While these methods address the issue of mislabeling and incorporate the shape and direction of oriented ships, they only assign positive and negative labels through a continuous two-dimensional function represented by a Gaussian heatmap. This approach has a crucial limitation, as the Gaussian heatmap cannot be utilized as the confidence output in the prediction stage. Additionally, the contribution of positive samples from different locations to the object is neglected, leading to the possibility of positive samples at the edge of the target location being overlooked and increasing the impact of noise on accurate target detection.

We aim to solve the imbalance problem caused by the number of positive and negative samples, better suppress background and noise interference, and simultaneously improve the robustness for remote sensing multi-scale ships and the training model’s capability. Therefore, the multi-scale dense-point rotation Gaussian heatmap (MDP-RGH) method is proposed to overcome background and noise interference while enhancing the robustness of multi-scale ship detection and the training model’s capabilities. The MDP-RGH is a discrete two-dimensional function based on a Gaussian heatmap that operates on dense points, allowing for the modeling of multi-scale oriented ships based on their shape and direction. The positive samples are weighted using the MDP-RGH to ensure rotational Gaussian distribution. The Gaussian heatmap confidence is used to predict whether a target is a positive or negative sample, thereby improving detection and reducing network computation. After that, a new anchor-free oriented ship detector network (AF-OSD) is constructed using the MDP-RGH method to detect multi-scale oriented ships. Additionally, a multi-task object size adaptive loss (OSALoss) function is designed to address the training imbalance caused by varying ship sizes. The weight of this function is determined by both the object area and the density of dense points, leading to improved ship detection accuracy. The contributions of this work can be summarized as follows.

• An oriented ship model based on MDP-RGH is proposed, which can balance the number of positive and negative samples, suppress the interference of negative samples such as background and noise in the image, and improve the training accuracy.
• An AF-OSD based on MDP-RGH is designed to achieve a better prediction for oriented ships with multi-scale attributes.
• A multi-task OSALoss function is constructed to further overcome the training imbalance problem caused by different ship sizes to improve the detection quality and performance of the whole model for multi-scale ships.

The rest of the paper is organized as follows. In Section 2, the ship model based on MDP-RGH is elaborated in detail. Section 3 describes the AF-OSD based on the MDP-RGH ship model, which shows the detail of the network. Section 4 reports the hyper-parameter settings. Section 5 shows the experimental results and analysis. Section 6 discusses the experiment. Finally, Section 7 is the conclusion.

The variables themselves indicate a certain class of meaning, e.g., $(x,y)$ for coordinate variables, $\mathbf{G}$ for ground truth, $\mathbf{F}$ for output convolutional features, and $\mathbf{M}$ for masks. The subscript of a variable indicates the qualification of the variable, indicating that the variable belongs to the range indicated by the subscript, and the next-level subscript is the qualification of the previous-level subscript.

2. The Oriented Ship Model Based on MDP-RGH

To solve the problem of an unbalanced number of positive and negative samples, predict the target using Gaussian heatmap confidence, and account for the contribution of positive samples at different locations, a new model of the ship, MDP-RGH, is proposed to suppress the background and noise interference, and effectively describe the characteristics of the shape and direction of the multi-scale ship. Specifically, there are three steps to complete the model of an oriented ship: (1) dividing the image region; (2) obtaining multi-scale dense points by down-sampling the image; (3) weighting the dense points with a rotation Gaussian Heatmap.

To address the issue of the imbalanced number of positive and negative samples, a new ship model called MDP-RGH is proposed. This model incorporates Gaussian heatmap confidence in target prediction and considers the contribution of positive samples from different locations. MDP-RGH aims to reduce background and noise interference and effectively describe the shape and direction of multi-scale ships, as shown in Figure 3. The process of creating the oriented ship model involves three steps: (1) dividing the image region; (2) obtaining multi-scale dense points through down-sampling the image; (3) weighting the dense points with a rotation Gaussian heatmap.

2.1. Dividing the Image Region

As the shape of the ship is similar to a shuttle [6], if all the areas in the object bounding box are divided into object regions, part of the background pixels will be included in the object region.

To fit the shape and direction characteristics of the ship and coordinate with the subsequent rotation Gaussian heatmap, an image region division method is designed, as shown in Figure 3. Specifically, the shrink rotation ellipse regions are delineated as the object regions. Other regions in the object-bounding box are regarded as ignorable regions, while the regions not in any object-bounding box are regarded as the background regions. In this case, we only need to determine the object regions and the ignorable regions. More specifically, here are three steps to divide the image region.

2.1.1. Transformation the Coordinate for the Oriented Ship

To facilitate the subsequent calculation, we transform the rotating ship into an upright ship. Therefore, as shown in Figure 4, coordinate transformation is performed. Specifically, we transform the points $\left[\begin{array}{c}{x}_{\mathrm{pixel}}\\ y_{\mathrm{pixel}}\end{array}\right]$ in the original pixel coordinate system $\mathbf{XOY}$ to the new coordinate system ${\mathbf{X}}^{\prime }{\mathbf{O}}^{\prime }{\mathbf{Y}}^{\prime }$ with the center point of the oriented ship being set as the origin, the long axis of the ship (the line between the center points of the bow and stern) is set as the $Y^{\prime }$ axis, and the short axis is set as the $X^{\prime }$ axis. The coordinates of pixel points in the coordinate system ${\mathbf{X}}^{\prime }{\mathbf{O}}^{\prime }{\mathbf{Y}}^{\prime }$ can be expressed as

$$\begin{array}{cc}\hfill \left[\begin{array}{c}{x}_{\mathrm{pixel}}^{\prime }\\ y_{\mathrm{pixel}}^{\prime }\end{array}\right]=\left[\begin{array}{cc}cos\alpha & sin\alpha \\ -sin\alpha & cos\alpha \end{array}\right]& \left(\left[\begin{array}{c}{x}_{\mathrm{pixel}}\\ y_{\mathrm{pixel}}\end{array}\right]-\left[\begin{array}{c}{x}_c\\ y_c\end{array}\right]\right),\hfill \end{array}$$

(1)

where $\left[\begin{array}{c}{x}_{\mathrm{c}}\\ y_{\mathrm{c}}\end{array}\right]$ means the coordinates of the center point of the oriented ship in the coordinate system $\mathbf{XOY}$. $\alpha$ represents the counterclockwise angle between the positive half-axis of the oriented ship (the direction from the center point of the ship to the center point of the bow is positive) and the positive Y axis of the pixel coordinate system $\mathbf{XOY}$.

2.1.2. Creating a Shrink Rotation Ellipse Equation for the Oriented Ship

We create a shrink rotate ellipse equation for the oriented ship to determine which region the pixels belong to. The ellipse of the shrink rotation ship bounding box in the coordinate system ${\mathbf{X}}^{\prime }{\mathbf{O}}^{\prime }{\mathbf{Y}}^{\prime }$ is a standard ellipse. Specifically, the ellipse equation is written as

$$\begin{array}{c}\hfill f_{\mathrm{ellipse}}\left(\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\end{array}\right]\right)=\frac{{\left(x^{\prime }\right)}^2}{{\left(\frac{\xi w}{2}\right)}^2}+\frac{{\left(y^{\prime }\right)}^2}{{\left(\frac{\xi h}{2}\right)}^2}=1.\end{array}$$

(2)

2.1.3. Identifying the Region to Which the Pixels in the Image Belong

After the above two steps, we can determine whether the pixel point $\left[\begin{array}{c}{x}_{\mathrm{pixel}}\\ y_{\mathrm{pixel}}\end{array}\right]$ belongs to the object region ${\mathrm{area}}_{\mathrm{ship}}$ or the ignorable region ${\mathrm{area}}_{\mathrm{ignore}}$ as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\left[\begin{array}{c}{x}_{\mathrm{pixel}}\\ y_{\mathrm{pixel}}\end{array}\right]\in {\mathrm{area}}_{\mathrm{ship}},\hfill & f_{\mathrm{ellipse}}\left(\left[\begin{array}{c}{x}_{\mathrm{pixel}}^{\prime }\\ y_{\mathrm{pixel}}^{\prime }\end{array}\right]\right)\le 1\mathrm{and}\left|{{\mathrm{x}}^{\prime }}_{\mathrm{pixel}}\right|\le \frac{\mathrm{w}}{2}\hfill \\ \left[\begin{array}{c}{x}_{\mathrm{pixel}}\\ y_{\mathrm{pixel}}\end{array}\right]\in {\mathrm{area}}_{\mathrm{ignore}},\hfill & f_{\mathrm{ellipse}}\left(\left[\begin{array}{c}{x}_{\mathrm{pixel}}^{\prime }\\ y_{\mathrm{pixel}}^{\prime }\end{array}\right]\right)>1\mathrm{and}\left|{{\mathrm{y}}^{\prime }}_{\mathrm{pixel}}\right|\le \frac{\mathrm{h}}{2},\left|{{\mathrm{x}}^{\prime }}_{\mathrm{pixel}}\right|\le \frac{\mathrm{w}}{2}.\hfill \end{array}\right.\end{array}$$

(3)

Particularly, a pixel point belongs to the background region when it does not belong to any ship’s object or ignorable region.

2.2. Multi-Scale Dense Points by Down-Sampling

Considering the multi-scale attributes of different ships, as shown in Figure 5, a multi-scale dense-point sampling method is further proposed to balance the number of positive samples of ships of different sizes. In particular, dense points can also reduce the amount of model computation since not all points need to be involved.

Specifically, according to the different scales of the ships, the image is down-sampled at different multiples to obtain different low-resolution images. We conduct s (s = 4, 8, 16) down-samplings of the image with three scales. Then, the low-resolution image is mapped to the original image to obtain a three-scale dense point matrix. The coordinate calculation formula of dense points can be described as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{x}_{\mathrm{denp}}{{}_s}_{{}_{i}j}=\left(i-1\right)\left(s\right)+\frac{s}{2}\hfill \\ y_{\mathrm{denp}}{{}_s}_{{}_{i}j}=\left(j-1\right)\left(s\right)+\frac{s}{2},\hfill \end{array}\right.\end{array}$$

(4)

where $x_{\mathrm{denp}}{{}_s}_{{}_{i}j}$ and $y_{\mathrm{denp}}{{}_s}_{{}_{i}j}$ denote the horizontal and vertical coordinates of the j-th row and i-th column sample point $\mathrm{denp}{{}_s}_{{}_{i}j}$ in the original image in the dense point matrix after s samplings, respectively.

Thus, small-scale dense points are used to represent large ships and large-scale dense points are used to represent small ships, which could balance the number of positive samples of the large, medium, and small ships to a certain extent. Among them, large-scale, medium-scale, and small-scale ships can be clustered with the k-means clustering algorithm. To better delineate the effective area of the image, dense points in the background region are defined as negative sample points, dense points in the ignorable region are ignorable dense points, and dense points in the object region are positive sample points, as shown in Figure 6a–c, respectively. Particularly, the ignorable dense points do not participate in calculating the loss function when training the model.

2.3. Weighting the Dense Points with Rotation Gaussian Heatmap

It has been proved that the closer the sample point is to the center of the ship, the richer the features of the ship are extracted during model inference, obtaining more accurate detection results. Therefore, the Gaussian weighting on the dense points with the object region is performed, where the dense points with different positions mean the various degrees of importance of the ship. Furthermore, the multi-scale dense-point rotation Gaussian heatmap is obtained, as shown in Figure 6.

Specifically, the calculation method of the rotation Gaussian heatmap value of the dense points in the ship region at each scale can be represented as

$$\begin{array}{cc}\hfill & g\left(x{{}_{\mathrm{denp}}}_{{{}_s}_{{}_{ij}}},y{{}_{\mathrm{denp}}}_{{{}_s}_{{}_{ij}}}\right)=g^{\prime }\left(x{{}_{\mathrm{denp}}^{\prime }}_{{{}_s}_{{}_{ij}}},y{{}_{\mathrm{denp}}^{\prime }}_{{{}_s}_{{}_{ij}}}\right)=\mathrm{Exp}\left(-\left(\frac{{\left(x_{{\mathrm{denp}}_{s_{ij}}}^{\prime }\right)}^2}{2{\sigma }_w^2}+\frac{{\left(y_{{\mathrm{denp}}_{s_{ij}}}^{\prime }\right)}^2}{2{\sigma }_h^2}\right)\right),\hfill \end{array}$$

(5)

where $g(·)$ means the rotation Gaussian heatmap function; $g^{\prime }(·)$ means the general Gaussian heatmap function; $x_{{\mathrm{denp}}_{s_{ij}}}^{\prime }$ and $y_{{\mathrm{denp}}_{s_{ij}}}^{\prime }$ denote the horizontal and vertical coordinates of the dense point in the coordinate system ${\mathbf{X}}^{\prime }{\mathbf{O}}^{\prime }{\mathbf{Y}}^{\prime }$, respectively; $\mathrm{Exp}(·)$ is the exponential function; ${\sigma }_w$ and ${\sigma }_h$ are the parameters related to the width and height of the ship, respectively.

To determine the values of ${\sigma }_w$ and ${\sigma }_h$, a hyper-parameter $g_{init}\in (0,1)$ is introduced, which represents the value of the rotation Gaussian heatmap when the dense point is located at the boundary of the shrink rotation ellipse. Take two points on the scaled rotating ellipse boundary, and then the values of ${\sigma }_w$ and ${\sigma }_h$ can be obtained by combining the initial Gaussian heatmap value $g_{\mathrm{init}}$, scale factor $\xi$, (1), (2), and (5). For example, we take the two vertices of this scaled rotation ellipse (e.g., the two points A and B in Figure 4) to determine the values of ${\sigma }_w$ and ${\sigma }_h$. At this point, the horizontal and vertical coordinates of these two points in the coordinate system ${\mathbf{X}}^{\prime }{\mathbf{O}}^{\prime }{\mathbf{Y}}^{\prime }$ can be calculated by (1) and (2). Bringing $g_{init}$, A, and B into (5) yields

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{g}_{init}=\mathrm{Exp}\left(-\frac{{\left(\xi h/2\right)}^2}{2{\sigma }_h^2}\right)\hfill \\ g_{init}=\mathrm{Exp}\left(-\frac{{\left(\xi w/2\right)}^2}{2{\sigma }_w^2}\right),\hfill \end{array}\right.\end{array}$$

(6)

then the values of ${\sigma }_w$ and ${\sigma }_h$ can be obtained.

In particular, the rotation Gaussian heatmap value of the dense points located in the ignorable region or background region is 0. After that, the Gaussian rotation heatmap of the ships with three scales can be generated in Figure 6d–f, respectively.

Based on the above designs, an oriented ship model based on MDP-RGH can be obtained, which can better balance the influence of positive and negative samples of the training dataset, and reduce the influence of noise and background on ship recognition. The positive samples near the edge of the ship are made to conform to the rotated Gaussian distribution by avoiding noise interference. The closer the positive sample is to the edge, the lower its value, so our positive sample is soft. A higher positive sample score means that the positive sample is more representative of the ship. Thus, the MDP-RGH improves the robustness for different size targets and the calculation speed of the proposed model. Next, we will introduce the oriented ship detection algorithm base on MDP-RGH.

3. Oriented Ship Detection Algorithm Based on MDP-RGH

Based on the proposed MDP-RGH method for training dataset improvement, an end-to-end AF-OSD is devised to detect the multi-scale oriented ship, which is accurate identification and information extraction of multi-scale ships, as shown in Figure 7. Precisely, the AF-OSD mainly consists of three parts: (1) label assignment based on MDP-RGH; (2) the AF-OSD; (3) multitask OSALoss.

3.1. Label Assignment Based on MDP-RGH

According to Section 2, based on the MDP-RGH model, the label of supervising the training process of the AF-OSD can be obtained. The label allocation strategy processes the original dataset labels into three parts: (1) multi-scale dense-point rotation Gaussian heatmap confidence label, which is the weight of dense points; (2) multi-scale dense point category label, namely, regions divided by the image; (3) multi-scale dense point ship bounding box corner offsets label, namely, the offsets of the four corner points of the ship bounding box relative to the dense points in the corresponding object region.

3.1.1. Positive Sample Confidence Labels for Dense Points Based on MDP-RGH

The confidence labels of dense points based on MDP-RGH indicate the probability that the dense points belong to a ship, which can be expressed as ${\mathbf{G}}_{{\mathrm{conf}}_s}\in {\mathbf{R}}^{\left(\frac{H}{s}\right)\times \left(\frac{W}{s}\right)\times 1}$, where $s=4,8,16$ represents the sampling scales with three values over the image and each element is a scalar.

Specifically, for the dense point ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$ falling into the object region, its Gaussian heatmap confidence label value can be calculated by (5), while its value is 0 for the dense point outside the object region, which can be represented as

$$\begin{array}{c}\hfill {\mathbf{G}}_{{\mathrm{conf}}_{s_{ij}}}=\left\{\begin{array}{cc}g\left({x_{\mathrm{denp}}}_{{{}_s}_{{}_{ij}}},{y_{\mathrm{denp}}}_{{{}_s}_{{}_{ij}}}\right),\hfill & {\mathrm{denp}}_{{{}_s}_{{}_{ij}}}\in {\mathrm{area}}_{\mathrm{ship}}\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(7)

where ${\mathbf{G}}_{{\mathrm{conf}}_{s_{ij}}}$ means the label value of the Gaussian heatmap of the dense point ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$.

3.1.2. The Classification Label for Dense Points

The classification label ${\mathbf{G}}_{{\mathrm{cls}}_s}\in {\mathbf{R}}^{\left(\frac{H}{s}\right)\times \left(\frac{W}{s}\right)\times N_c}$ of dense points is a three-dimensional matrix, which can be written as

$$\begin{array}{c}\hfill {\mathbf{G}}_{{\mathrm{cls}}_s}=\left[{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}^{\left(1\right)},\dots ,{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}^{\left(c\right)},\dots ,{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}^{\left(N_c\right)}\right],\end{array}$$

(8)

where each element is a vector with size $N_c$ (here $N_c$ represents the number of categories, including the background category). ${\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}^{\left(c\right)}$ denotes the score of dense point ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$ belonging to category c, which can be determined by

$$\begin{array}{c}\hfill {\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}^{\left(c\right)}=\left\{\begin{array}{cc}1,\hfill & {\mathrm{denp}}_{{{}_s}_{{}_{ij}}}\in c-\mathrm{th}\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(9)

where the dense point belongs to the c-th category. The c-th component of the above vector is set to 1, and the others are set to 0.

3.1.3. The Corner Points offsets Label of the Ship Bounding Box of the Dense Points

We built corner point offset labels of the ship bounding box for regression when training. The label ${\mathbf{G}}_{{\mathrm{Poffsets}}_s}\in {\mathbf{R}}^{\left(\frac{H}{s}\right)\times \left(\frac{W}{s}\right)\times 8}$ of dense points is a two-dimensional matrix, where each element is a vector with size 8. The label represents the horizontal and vertical coordinates of the four corner points offsets of the ship bounding box of dense point ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$, which can be written as

$$\begin{array}{c}\hfill {\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}=\left[{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{\left(1\right)},\dots ,{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{\left(8\right)}\right].\end{array}$$

(10)

If ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$ does not belong to the object region of any oriented ship, all components of ${\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}$ are 0. If ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$ belongs to the object region of an oriented ship, the component of ${\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}$ can be calculated as

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{(2k-1)}=\left(x_{\mathbf{P}\mathbf{k}}-x_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}}\right)/{\mathrm{scale}}_s\hfill \\ {\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{\left(2k\right)}=\left(y_{\mathbf{P}\mathbf{k}}-y_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}}\right)/{\mathrm{scale}}_s,\hfill \end{array}\right.\end{array}$$

(11)

where k ($k=1,2,3,4$) represents the k-th corner point of the ship bounding box, $x_{\mathbf{P}\mathbf{k}}$ and $y_{\mathbf{P}\mathbf{k}}$ are the horizontal and vertical coordinates of the k-th corner point of the oriented ship bounding box, respectively, and ${\mathrm{scale}}_s$ is the normalized parameter of different scales.

3.2. AF-OSD

Normally, the AF-OSD consists of two parts: (1) a multi-scale feature extraction network (MFEN) and (2) a multi-scale oriented ship detection head (MOSDH). The MFEN extracts the features of the oriented ship in the input image, while the MOSDH applies the extracted features to identify and detect the oriented ship.

3.2.1. MFEN

PANet [21] is selected for the feature extraction structure of the MFEN for the fact that the size of the ship object has a large scale range. To reduce the problem of information loss arising from long transmission distances, the bottom-up path augmentation of the FPN [22] structure is applied, which provides a shorter path for high-level information transmission and effectively reduces information loss, as shown in Figure 8.

For the backbone of MFEN, ship detection is based on remote sensing data, visual range, and complex environments to fully extract the attribute information of ships. Hence, a relatively large feature extraction backbone network is needed to design. Because the cross-stage partial network (CSPNet) [23] solves the problem of repeated gradient information in the optimization process of large CNN frameworks, CSPDarknet53 is chosen as the backbone network, which divides the feature map of the base layer into two parts and then merges them through a cross-stage hierarchy.

Based on the designed MFEN, the feature extraction of the input image can be realized. Specifically, the image $I\in {\mathbb{R}}^{H\times W\times 3}$ (H and W are the height and width, respectively) is input into the MFEN $E_{\mathrm{MFEN}}$. Then, the multi-scale deep feature ${\mathbf{F}}_s\in {\mathbb{R}}^{(H/s)\times (W/s)\times C}$ can be extracted as

$$\begin{array}{c}\hfill {\mathbf{F}}_s=E_{\mathrm{MFEN}}\left(\mathbf{I},s;{\theta }_{\mathbf{E}}\right),\end{array}$$

(12)

where C is the channel number, and ${\theta }_{\mathbf{E}}$ is the weight parameters of the designed MFEN $E_{\mathrm{MFEN}}$. The multi-scale deep feature ${\mathbf{F}}_s$ is fed into the subsequent MOSDH to identify and detect the ship.

3.2.2. MOSDH

The MOSDH is used to identify and detect the oriented ships through the features extracted by the MFEN. The output structure of MOSDH refers to the decoupling output mode of YoloX [24]. Corresponding to the three feature layers of PANet structure, the MOSDH contains three scales of oriented ship detection heads (OSDHs), where the small-scale detection head predicts large ships, the medium-scale detection head predicts medium ships, and the large-scale detection head predicts small ships.

Moreover, each OSDH has three decoupling output branches, as shown in Figure 9: (a) Gaussian heatmap confidence output branch of dense points: to predict the score that each dense point to be a positive sample (belonging to a ship); (b) Gaussian heatmap classification output branch of dense points: to predict the category of each dense point; (c) ship bounding box corner points offsets output branch of dense points: to predict the offsets of the four corner points of the ship (which ship the dense point belongs to) relative to the dense point.

(a) The Dense Point Gaussian Heatmap Confidence Output Branch

To suppress background and noise interference, the confidence output of the Gaussian heatmap of dense points is designed. The multi-scale deep feature ${\mathbf{F}}_s$ is fed into the dense point Gaussian heatmap confidence output branch of MOSDH $H_{{\mathrm{MOSDH}-\mathrm{conf}}_s}$, and the Gaussian heatmap confidence output feature ${\mathbf{F}}_{{\mathrm{conf}}_s}\in R^{(H/s)\times (W/s)\times 1}$ can be obtained as

$$\begin{array}{c}\hfill {\mathbf{F}}_{{\mathrm{conf}}_s}=H_{{\mathrm{MOSDH}-\mathrm{conf}}_s}\left({\mathbf{F}}_s,s;{\theta }_{\mathbf{H}}\right),\end{array}$$

(13)

where ${\theta }_{\mathbf{H}}$ is the weight parameters of the MOSDH. Each feature point of the Gaussian heatmap confidence output feature layer predicts the score of the dense point corresponding to this feature point belonging to the ship object region (${\mathrm{area}}_{\mathrm{ship}}$). When the score is greater than the set threshold $T_{\mathrm{conf}}\in (0,1)$, the dense point is judged to be a dense positive point (belonging to the ship object region). The judgment method can be expressed as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{\mathrm{denp}}_{s_{ij}}\in {\mathrm{area}}_{\mathrm{ship}},\hfill & sigmoid\left({\mathbf{F}}_{{\mathrm{conf}}_{s_{ij}}}\right)\ge T_{\mathrm{conf}}\hfill \\ {\mathrm{denp}}_{s_{ij}}\notin {\mathrm{area}}_{\mathrm{ship}},\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(14)

where ${\mathbf{F}}_{{\mathrm{conf}}_{s_{ij}}}$ means the eigenvalue of the j-th row and the i-th column of the Gaussian heatmap confidence output feature layer, the activation function of the confidence output uses the $sigmoid(·)$ function, and $T_{\mathrm{conf}}$ represents the confidence threshold.

Suppose that the dense point is determined to be a positive dense point. In that case, the category of the oriented object to which the dense point belongs and the coordinates of the four corner points of the oriented ship can be determined. Otherwise, it is directly judged that the dense matter belongs to the background (dense negative point), and no further discrimination and calculation will be carried out.

(b) Dense Point Classification Output Branch

We design the dense point classification output branch to determine the category of dense points. When a dense point is determined as a positive sample point, the dense point classification output branch $H_{{\mathrm{MOSDH}-\mathrm{cls}}_s}$ is used to predict the category of the dense point.

Specifically, the multi-scale deep feature ${\mathbf{F}}_s$ is input into the classification output branch of the MOSDH $H_{{\mathrm{MOSDH}-\mathrm{cls}}_s}$, and the classification output feature ${\mathbf{F}}_{{\mathrm{cls}}_s}\in R^{(H/s)\times (W/s)\times N_c}$ is obtained as

$$\begin{array}{c}\hfill {\mathbf{F}}_{{\mathrm{cls}}_s}=H_{{\mathrm{MOSDH}-\mathrm{cls}}_s}\left({\mathbf{F}}_s,s;{\theta }_{\mathbf{H}}\right).\end{array}$$

(15)

Each feature point with size $N_c$ of the feature layer of the category output predicts the score of each category that the corresponding dense point is, which can be written as

$$\begin{array}{c}\hfill {\mathrm{denpID}}_{s_{ij}}=argmax\left(softmax\left({\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}}\right)\right),\end{array}$$

(16)

where ${\mathrm{denpID}}_{s_{ij}}$ denotes the category number of the dense point ${\mathrm{denp}}_{s_{ij}}$, and ${\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}}$ represents the $N_c$ dimension feature vector of the j-th row and the i-th column of the output feature layer of the category.

Therefore, the output feature vector of the category is first converted into the score of each type by the $softmax(·)$ function, and then the category ID number of the highest category score is calculated by $argmax(·)$ function to obtain the category of the oriented object.

(c) The Corner Point offsets Output Branch

To determine the position of the bounding box, the corner points to offset the output branch are further built. When a dense matter is judged as a dense positive point, the corner point offset output branch of the dense points predicts the offsets of the four corner points of the ship bounding box relative to the dense point.

Specifically, ${\mathbf{F}}_s$ is input into the corner point offset output branch of the dense points of the MOSDH $H_{{\mathrm{MOSDH}-\mathrm{Poffsets}}_s}$, and the output of offsets feature ${\mathbf{F}}_{{\mathrm{Poffsets}}_s}\in R^{(H/s)\times (W/s)\times 8}$ is calculated as

$$\begin{array}{c}\hfill {\mathbf{F}}_{{\mathrm{Poffsets}}_s}=H_{{\mathrm{MOSDH}-\mathrm{Poffsets}}_s}\left({\mathbf{F}}_s,s,{\theta }_{\mathbf{H}}\right).\end{array}$$

(17)

Each feature point with size 8 of the output feature layer predicts the offsets of the four corner points of the ship bounding box relative to the dense point corresponding to the feature point. Therefore, the coordinates of the four corner points are the offsets plus the coordinate of the dense point, which can be expressed as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{x}_{\mathbf{P}\mathbf{k}}\hfill & =x_{\Delta \mathbf{P}\mathbf{k}}+x_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}}={\mathbf{F}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{(2k-1)}\times {\mathrm{scale}}_s+x_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}}\hfill \\ y_{\mathbf{P}\mathbf{k}}\hfill & =y_{\Delta \mathbf{P}\mathbf{k}}+y_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}}={\mathbf{F}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{\left(2k\right)}\times {\mathrm{scale}}_s+y_{{\mathrm{denp}}_{{{}_s}_{{}_{ij}}}},\hfill \end{array}\right.\end{array}$$

(18)

where $x_{\mathbf{P}\mathbf{k}}$ and $y_{\mathbf{P}\mathbf{k}}$ are the horizontal and vertical coordinates of the k-th corner point $\mathbf{P}\mathbf{k}$ of the ship bounding box, respectively, $x_{\Delta \mathbf{P}\mathbf{k}}$ and $y_{\Delta \mathbf{P}\mathbf{k}}$ represent the offsets of the k-th corner point $\mathbf{P}\mathbf{k}$ of the ship bounding box relative to the dense point ${\mathrm{denp}}_{{{}_s}_{{}_{ij}}}$, respectively, and ${\mathbf{F}}_{{\mathrm{Poffsets}}_{s_{ij}}}^{(2k-1)}$ is the $(2k-1)$-th value of the j-th row and i-th column feature vector of the output feature.

Multiple dense points may simultaneously predict the same ship object, or output layers of different scales may predict the same object. Therefore, after decoding all prediction boxes, it is necessary to use the NMS technology to filter out the redundant oriented object bounding boxes, obtaining the final detection result of the oriented ship.

3.3. Multi-Task Object Size Adaptive Loss (MOSALoss)

The multi-task OSALoss function, as the target function of the optimizer in the training stage, guides the optimization of the weight parameters of the proposed AF-OSD.

3.3.1. OSALoss Weight

The multi-scale structure of the AF-OSD network has been designed to solve the unbalanced training of different sizes of ships due to the other numbers of positive dense points. Specifically, three detection heads are designed to detect oriented ships of various sizes. However, since the object size is approximately continuous from small to large, the predicted objects of a certain output scale are large or small. The loss function model [25] focuses more on a small target with only one best anchor as a positive sample paired with it. Different from [25], we design the loss function OSALoss suitable for the MDP-RGH-based multi-scale directional ship detection in this paper.

To further solve the training imbalance caused by different object sizes, increasing the number of output characteristic layers at different scales will not only fail to completely solve this problem but also increase the amount of calculation, as shown in Figure 10. As such, an OSALoss weight (OSAWeight) is further designed, which can be adaptive to ship size.

Specifically, if the dense point ${\mathrm{denp}}_{s_{ij}}$ belongs to the object region of a ship ${\mathrm{Ships}}_{s_n}$, the weight of the sample point $W_{{\mathrm{adap}-\mathrm{denp}}_{s_{ij}}}$ is obtained as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{W}_{{\mathrm{adap}-\mathrm{denp}}_{s_{ij}}}=\frac{d_{{\mathrm{denp}}_s}}{\sqrt{A_{{\mathrm{Ships}}_{s_n}}}},\hfill & {\mathrm{denp}}_{s_{ij}}\in {\mathrm{area}}_{\mathrm{ship}}\hfill \\ W_{{\mathrm{adap}-\mathrm{denp}}_{s_{ij}}}=1,\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(19)

where ${\mathrm{Ships}}_{s_n}$ represents the n-th ship on the down-sampling scale of s times, $A_{{\mathrm{Ships}}_{s_n}}$ is the area of the ship, and $d_{{\mathrm{denp}}_s}$ is the distance between two adjacent dense points. According to (19), the weight of this dense point is inversely proportional to the square root of the area of the ship in a dense point matrix of the same scale. When the shipping area is large, the weight is small, while the weight is significant when the shipping area is small.

The difference in object size directly affects the difference in the number of positive sample points, which affects the loss related to positive sample points, leading to the problem of training imbalance. Therefore, the OSAWeight will be introduced into the loss function related to positive sample points to eliminate the influence caused by different object sizes.

3.3.2. Multi-Task OSALoss

In order to solve the situation of easy sample imbalance during training, the multi-task OSALoss ${Loss}_{\mathrm{adap}-\mathrm{multi}}$ consists of the Gaussian heatmap confidence loss ${\mathbf{GHC}Loss}_s=Loss\left({\mathbf{F}}_{{\mathrm{conf}}_s},{\mathbf{G}}_{{\mathrm{conf}}_s}\right)$ of dense points, the category prediction loss ${\mathbf{CP}Loss}_s=Loss\left({\mathbf{F}}_{{\mathrm{cls}}_s},{\mathbf{G}}_{{\mathrm{cls}}_s}\right)$ of dense points, and the corner points offsets prediction loss ${\mathbf{OP}Loss}_s=Loss\left({\mathbf{F}}_{{\mathrm{Poffsets}}_s},{\mathbf{G}}_{{\mathrm{Poffsets}}_s}\right)$ of the ship bounding box. Then, the proposed multi-task OSALoss can be written as

$$\begin{array}{cc}\hfill {Loss}_{\mathrm{adap}-\mathrm{multi}}=& \frac{1}{3}\sum _{s\in (4,8,16)}({\mathbf{GHC}Loss}_s+\gamma {\mathbf{CP}Loss}_s+\lambda {\mathbf{OP}Loss}_s),\hfill \end{array}$$

(20)

where ${\mathbf{G}}_{{\mathrm{conf}}_s}$, ${\mathbf{G}}_{{\mathrm{cls}}_s}$, and ${\mathbf{G}}_{{\mathrm{Poffsets}}_s}$ represent the confidence ground truth of Gaussian heatmap, the classification ground truth of dense points and the corner offsets ground truth of ship bounding box, respectively. $\gamma$ and $\lambda$ are the weight parameters, which are adaptive with the training.

Next, we will introduce the Gaussian heatmap confidence loss, the category prediction loss, and the corner points offset prediction loss in detail.

(a) The Gaussian Heatmap Confidence Loss:

Adaptive wing loss [26] is used for the ${\mathbf{GHC}Loss}_s$, which can be represented as

$$\begin{array}{c}\hfill {\mathbf{GHC}Loss}_s=AWingLoss\left({\mathbf{F}}_{{\mathrm{conf}}_s},{\mathbf{G}}_{{\mathrm{conf}}_s}\right).\end{array}$$

(21)

Since the ${\mathbf{GHC}Loss}_s$ adopts the relatively mature adaptive wing loss, we do not introduce the OSAWeight into this loss function.

(b) The Category Prediction Loss:

The cross-entropy function is used for the ${\mathbf{CP}Loss}_s$. Particularly, the ${\mathbf{CP}Loss}_s$ is composed of positive ${\mathbf{CP}Loss}_s$ and negative ${\mathbf{CP}Loss}_s$, which can be written as

$$\begin{array}{c}\hfill {\mathbf{CP}Loss}_s={\mathbf{CP}Loss}_s^{\mathrm{Pos}}+{\mathbf{CP}Loss}_s^{\mathrm{Neg}},\end{array}$$

(22)

where the positive category prediction loss ${\mathbf{CP}Loss}_s^{\mathrm{Pos}}$ can be given as

$$\begin{array}{cc}\hfill {\mathbf{CP}Loss}_s^{\mathrm{Pos}}& =\sum _{j=1}^{H/s}\sum _{i=1}^{W/s}{W}_{\mathrm{adap}-{{\mathrm{denp}}_s}_{ij}}\times {\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}\times \sum _{c=1}^{N_c}({\mathbf{G}}_{{{\mathrm{cls}}_s}_{ij}}^{c}log\left({\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)\hfill \\ \hfill & +\left(1-{\mathbf{G}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)log\left(1-{\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)),\hfill \end{array}$$

(23)

where ${\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}$ is the positive dense point mask. ${\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c$ is the score when the dense point ${{\mathrm{denp}}_s}_{ij}$ is the c-th category. ${\mathbf{G}}_{{{\mathrm{cls}}_s}_{ij}}^c$ is the ground truth of whether the dense point ${{\mathrm{denp}}_s}_{ij}$ is the c-th category.

${\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}$ can be written as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}=1,\hfill & {\mathrm{denp}}_{s_{ij}}\in {\mathrm{area}}_{\mathrm{ship}}\hfill \\ {\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}=0,\hfill & \mathrm{otherwise}s\hfill \end{array}\right.\end{array}$$

(24)

and ${\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c$ can be expressed as

$$\begin{array}{c}\hfill {\mathbf{S}}_{{\mathrm{cls}}_{s_{ij}}}=softmax\left({\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}}\right).\end{array}$$

(25)

For negative dense point category prediction loss, to further solve the imbalance between the number of positive and negative samples, the hard negative example mining technique is applied. Specifically, the negative dense category prediction loss for each dense point can be written as

$$\begin{array}{cc}\hfill & {Loss}_{\mathrm{neg}}\left({\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}},{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}\right)=\sum _{c=1}^{N_c}\left({\mathbf{G}}_{{{\mathrm{cls}}_s}_{ij}}^{c}log\left({\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)+\left(1-{\mathbf{G}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)log\left(1-{\mathbf{S}}_{{{\mathrm{cls}}_s}_{ij}}^c\right)\right)\times {\mathbf{M}}_{{{\mathrm{bg}}_s}_{ij}},\hfill \end{array}$$

(26)

where ${\mathbf{M}}_{{{\mathrm{bg}}_s}_{ij}}$ is a negative sample mask, which can be written as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}{\mathbf{M}}_{{{\mathrm{bg}}_s}_{ij}}=0,\hfill & {\mathrm{denp}}_{s_{ij}}\in {\mathrm{area}}_{\mathrm{ship}}\hfill \\ {\mathbf{M}}_{{{\mathrm{bg}}_s}_{ij}}=1,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.\end{array}$$

(27)

Then, the negative sample category prediction loss of dense points is sorted from the largest to the smallest, and the ranked negative sample category prediction loss ${Loss}_{{\mathrm{neg}}_{\mathrm{sorted}}}\left({\mathbf{F}}_{{\mathrm{cls}}_{s_l}},{\mathbf{G}}_{{\mathrm{cls}}_{s_l}}\right)$ can be obtained, where l denotes the serial number of the l-th dense point after the negative dense category prediction loss ranking of dense points. Finally, only the sum of the negative sample category prediction loss of the top L is taken as the negative sample category prediction loss. The category prediction loss ${\mathbf{CP}Loss}_s^{\mathrm{Neg}}$ can be represented as

$$\begin{array}{c}\hfill {\mathbf{CP}Loss}_s^{\mathrm{Neg}}=\sum _{l=1}^L{Loss}_{{\mathrm{neg}}_{\mathrm{sorted}}}\left({\mathbf{F}}_{{\mathrm{cls}}_{s_l}},{\mathbf{G}}_{{\mathrm{cls}}_{s_l}}\right).\end{array}$$

(28)

In particular, if the number of negative samples is greater than twice the number of positive samples, L takes two times the number of positive samples. Otherwise, L is equal to the number of positive samples.

(c) The Corner Points Offsets Prediction Loss:

The ship bounding box corner point offset prediction loss uses the object size adaptive weighted Smooth-L1 error loss. Specifically, we first use the Smooth-L1 loss to predict each dense point of the ship bounding box corner point offsets as

$$\begin{array}{cc}\hfill & Loss\left({\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}},{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}\right)=\left\{\begin{array}{c}\sum _{m=1}^{8}0.5\times {({\mathbf{F}}_{{{\mathrm{Poffsets}}_s}_{ij}}^m-{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^m)}^2,if\left|{\mathbf{F}}_{{{\mathrm{Poffsets}}_s}_{ij}}^m-{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^m\right|<1\hfill \\ \sum _{m=1}^8\left(\left|{\mathbf{F}}_{{{\mathrm{Poffsets}}_s}_{ij}}^m-{\mathbf{G}}_{{\mathrm{Poffsets}}_{s_{ij}}}^m\right|-0.5\right),\mathrm{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(29)

Then, the target-scale adaptive weighted summation of the predicted losses for all positive dense points ship target frame corner point offsets is performed. The total loss ${\mathbf{OP}Loss}_s$ can be written as

$$\begin{array}{cc}\hfill {\mathbf{OP}Loss}_s& =\sum _{j=1}^{H/s}\sum _{i=1}^{W/s}{W}_{\mathrm{adap}-{{\mathrm{denp}}_s}_{ij}}\times {\mathbf{M}}_{{{\mathrm{obj}}_s}_{ij}}\times Loss\left({\mathbf{F}}_{{\mathrm{cls}}_{s_{ij}}},{\mathbf{G}}_{{\mathrm{cls}}_{s_{ij}}}\right).\hfill \end{array}$$

(30)

Using the above label assignment strategy, network design, and loss function, we can train the deep learning network of the AF-OSD through the Adam optimization method [27], as summarized in Algorithm 1.

Algorithm 1: Training AF-OSD based on Adam optimization

Input: Image data set $\mathbb{I}=\left\{{\mathbf{I}}^{\left(1\right)},{\mathbf{I}}^{\left(2\right)},\dots \right\}$, and the corresponding labels $\mathbb{Y}=\left\{{\mathbf{Y}}^{\left(1\right)},{\mathbf{Y}}^{\left(2\right)},\dots \right\}$ Output: The trained AF-OSD with fixed neural network parameters ${\theta }_{\mathbf{E}}$ and ${\theta }_{\mathbf{H}}$ Construct AF-OSD Initialize ${\theta }_{\mathbf{E}}$, ${\theta }_{\mathbf{H}}$ randomly fork to $iterations$ do        $\mathbf{Select}\mathbf{randomly}minibatch{\mathcal{I}}_k\subset \mathbb{I},{\mathcal{Y}}_k\subset \mathbb{Y}$        Label assign as Section 3.1 : ${\mathcal{G}}_k\leftarrow {\mathcal{Y}}_k$        ${\mathcal{F}}_k=\mathit{AF-OSD}\left({\mathcal{I}}_k;{{\theta }_{\mathbf{E}}}_k,{{\theta }_{\mathbf{H}}}_k\right)$        ${\mathcal{L}}_k=Los{s}_{\mathit{aadap-multi}}\left({\mathcal{F}}_k,{\mathcal{G}}_k\right)$        ${{\theta }_{\mathbf{E}}}_{k+1},{{\theta }_{\mathbf{H}}}_{k+1}\leftarrow {Optimizer}_{Adam}\left({\mathcal{L}}_k;{{\theta }_{\mathbf{E}}}_k,{{\theta }_{\mathbf{H}}}_k\right)$ end for Freeze ${\theta }_{\mathbf{E}}$, ${\theta }_{\mathbf{H}}$

4. Experimental Conditions

In this section, experiments on two public-oriented ship datasets are conducted to quantitatively and qualitatively evaluate the proposed AF-OSD. A series of experiments are conducted using the DOTA ship dataset [28] and HRSC2016 dataset [29]. Next, we will introduce the experimental conditions, including experimental platforms, datasets, evaluation metrics, and implementation details.

4.1. Experimental Platforms

All the experiments are implemented on a desktop computer with an Intel Core i7-9700 CPU, 32 GB of memory, and a single NVIDIA GeForce RTX 3090 with 24 GB GPU memory.

4.2. Dataset

4.2.1. DOTA Ship Dataset

DOTA is a large-scale aerial image dataset containing 2806 images, with 15 categories labeled with oriented boxes and image sizes ranging from $800\times 800$ to $1600\times 1600$. We select 434 images containing ships (including 37,028 ships), where 90% of them are randomly selected as the training set, and the other 10% are as the validation set. The size distribution of ships is not uniform, and there are few large-scale ships. So, we enhance and expand the large ships in the dataset (rotation and flip enhancement), and the size distribution of the original dataset and the enhanced dataset is shown in Figure 11a, and the rotation angle distribution is shown in Figure 11b.

4.2.2. HRSC2016 Dataset

HRSC2016 is a remote sensing image dataset containing ship targets labeled as arbitrary orientations. It consists of 1061 images (436 images are the training set, 181 images are the validation set, and 444 images are the test set), whose spatial sizes range from $300\times 300$ to $1500\times 900$. We use the training set to train the model and the validation set to test the model. The size and rotation angle distributions of ships are illustrated in Figure 12, and the ship sizes and angles are more uniformly distributed. Therefore, we do not target special scale ships when we perform enhancement expansion (flip enhancement).

4.3. Evaluation Metrics

In this article, the widely used metrics in object detection are adopted to measure the detection performance, i.e., precision, recall, and average precision (AP) (IOU threshold set to 0.5). Moreover, we use the number of parameters (Params) and average running time to evaluate the complexity and speed of the model, respectively.

4.4. Implementation Details

Due to the large size of the original input image, we cut it before network training. Specifically, the input resolution is set to $800\times 800$. As such, the original image and the corresponding labels are first preprocessed, where the cut image is uniform $800\times 800$ in size (if the image is smaller than $800\times 800$, the padding operation is applied). To avoid the situation that an object is cut into two halves and disappears, an overlapping area of 200 pixels is set in the experiments. Of course, there will be some cases where only a part of the object is in the cropped image.

If this happens, we will judge the ratio of the area of the target on the cropped image to the area of the target itself. If this ratio is greater than the set threshold (taken as 0.6), the label information of the object will be kept. The judgment method of whether to keep or discard the target on the cropped picture is represented as

$$\begin{array}{c}\hfill \left\{\begin{array}{cc}\mathrm{Keep},\hfill & \frac{\mathbf{A}({\mathrm{box}}_{\mathrm{Object}}\cap {\mathrm{box}}_{{\mathrm{Crop}}_{\mathrm{image}}})}{\mathbf{A}\left({\mathrm{box}}_{\mathrm{Object}}\right)}>\mathrm{Threshold}\hfill \\ \mathrm{Discard},\hfill & \mathrm{otherwise},\hfill \end{array}\right.\end{array}$$

(31)

where $\mathbf{A}(·)$ denotes the function of the area and ${box}_{Object}\cap {box}_{Cro{p}_{image}}$ means the intersection of the target box and the cut picture box to be calculated. If it is determined that the target needs to be retained, its coordinates on the cut image are calculated as

$$\begin{array}{c}\hfill p^{O*}=p_i^{O*}=p_i^O-p_1^{\mathrm{Crop}},i\in 1,2,3,4\end{array}$$

(32)

where $p^{O*}$ is the new coordinate of the target frame in the cropped picture, $p_i^O$ denotes the i-th coordinate point of the target frame in the original picture, and $p_1^{\mathrm{Crop}}$ denotes the first coordinate point of the cropped picture frame in the original pictures. The image cutting schematic is shown in Figure 13.

During the training process, the Adam optimizer is used to optimize the weight parameters of our AF-OSD, and the initial learning rate is set to 0.001 with an exponential decay of 5%. The batch size is set to 10, and all networks are trained for 200 epochs.

During the testing process, since the image size of HRSC2016 is not very large, before input to the model, we resize the image and then pad its size to 800 × 800. For the test images of DOTA, we used the same processing method as the training data. The confidence threshold ${score}_t$ is set to 0.1 and the NMS threshold is set to 0.6.

5. Experimental Results

To verify the rationality and advantages of the proposed AF-OSD, some comparison experiments are implemented to study the performance differences between the proposed oriented ship detector and other state-of-the-art (SOTA) oriented-object detection methods. Moreover, some ablation experiments are conducted to analyze the effects of each part on the performance of the proposed oriented ship detector.

5.1. Ablation Experiments

To testify to the necessity of the oriented ship detector proposed, three sets of ablation comparison experiments are designed: (1) comparing the performance of a multi-scale network structure and a single-scale network structure for detecting oriented ships; (2) comparing the performance of a network without MDP-RGH confidence output branch and a network with MDP-RGH confidence output branch for detecting oriented ships; (3) comparing the performance of the AF-OSD trained with the traditional multi-task loss function and the AF-OSD trained with multi-task OSALoss function.

All the ablation comparisons of DNN networks are trained using the HRSC2016 training set, parameter settings, and training algorithms mentioned in the previous section. They are then tested in the HRSC2016 validation set for measuring their detection performance.

5.1.1. Ablation Experiment on Different Scales

To verify the effect of multi-scale detection head on model enhancement, three single-scale models and multi-scale models were compared in terms of detection accuracy on the HRSC2106 dataset. Single-scale oriented ship detection network contains different detection scales, including 50 × 50, 100 × 100, and 200 × 200, and the multi-scale oriented ship detection network has three detection scales output.

As shown in

Table 1. Ablation study of the multi-scale structure of the AF-OSD. The ✓ means network with the structure. The ✕ means network without the structure. The bold results mean the best performance.

Single Head	Multi Head	Recall	Precision	AP
50 × 50	✕	0.9020	0.8761	0.8789
100 × 100	✕	0.9020	0.7948	0.8700
200 × 200	✕	0.9039	0.8180	0.8833
✕	✓	0.9068	0.8963	0.8969

, the comparison results show that the detection accuracy of the multi-scale oriented ship detection network is better than that of any single-scale detection network. Since it is difficult for a single-scale detection head to cover the remote sensing oriented ship with a large scale range, the multi-scale detection head can solve this problem well. So, compared with the AF-OSD network that does not use the multi-scale detection head structure, the performance of the whole model can be improved by 1.31–2.69% by introducing this structure.

5.1.2. Ablation Experiment on MDP-RGH

To ensure the MDP-RGH confidence is favorable for the system, the comparison of the detection accuracy of the oriented ship detection network without the MDP-RGH confidence output branch and the oriented ship detection network with this branch is designed.

As shown in

Table 2. Ablation study of the MDP-RGH confidence output branch of the AF-OSD.

FPN	PAN	MDP-RGH	Recall	Precision	AP
✓	✓	✕	0.8854	0.8615	0.8494
✓	✓	✓	0.9068	0.8963	0.8969

, the comparison results show that the detection accuracy of the oriented ship detection network with the MDP-RGH confidence branch is better. Compared with the AF-OSD without the MDP-RGH structure, the performance of the whole framework can have achieved 4.75% AP improvement by introducing this structure on HRSC2016.

5.1.3. Ablation Experiment on Multi-Task OSALoss

To demonstrate the effectiveness of the OSAloss designed in this paper, the detection accuracy between the AF-OSD trained with the traditional multi-task loss (TMLoss) function and the network trained with the multi-task OSALoss (MOSALoss) function on HRSC2016 is compared.

As shown in

Table 3. Ablation study of the multi-task OSALoss function.

TMLoss	MOSALOSS	Recall	Precision	AP
✓	✕	0.8595	0.9082	0.8513
✕	✓	0.9068	0.8963	0.8969

, comparison results show that our AF-OSD trained with the MOSALoss function has better detection accuracy. The OSALoss weight in the MOSALoss function is proportional to the distance between the dense points and inversely proportional to the square root of the ship area. Therefore, the MOSALoss function can be adjusted adaptively with the different ship sizes. So, compared with the AF-OSD network trained with TMLoss, the performance of the model which trained through MOSAloss can be improved by 4.56%. Therefore, the AF-OSD network with the MOSALoss function can effectively solve the training imbalance caused by the different scales of oriented ships.

Based on the ablation experiments, the three parts of the AF-OSD are fully proven effective in remote sensing ship detection.

5.2. Comparisons with SOTA

To demonstrate the effectiveness of the method in this paper, the AF-RPN [18] is used as a baseline method, and the anchor-based methods RRPN [30], ROI-Transformer [31], ref. [20], and ${\mathrm{R}}^3\mathrm{Det}$ [32] are compared with our method. The anchor-free methods IENet [33], GGHL [20], and GRS-DET [6] are also used for the comparison. The baseline method shrinks the object bounding box to obtain the core region and takes all the core region points as positive. The proposed method utilized the MDP-RGH to model a ship compared to the baseline method. We can compare the benefits of MDP-RGH through experiments.

Repeatability experiments on the DOTA dataset by the proposed method in this paper yield better results on the DOTA test dataset, as shown in Figure 14. The first row in the figure is the ground truth of the dataset. From Figure 14a,b, we can find that the proposed method has a high recognition rate for small targets in remote sensing images in the DOTA dataset. From Figure 14a,c, it can be seen that our method can achieve a good result both for large and dense small-oriented ships at the same time. Moreover, AF-OSD can accurately identify the small targets in the ground truth that are not completely labeled in the ground truth, as shown in Figure 14a. However, the missed detection phenomenon of the method proposed is significantly less than the ground truth. Meanwhile, it can be seen that the bounding box accuracy of the detection algorithm in this paper is high, as shown in Figure 15, in which the green bounding box is our method and the red bounding box is the ground truth.

5.2.1. Results on the DOTA Ships Dataset

Table 4. Comparison results on the DOTA ship dataset.

Methods	Anchor-Free	Backbone	AP
RRPN	✕	ResNet-101	0.4454
ROI-Transfomer	✕	ResNet-101	0.8359
${\mathrm{R}}^3\mathrm{Det}$	✕	ResNet-152	0.8784
IENet	✓	ResNet-101	0.7161
AF-RPN	✓	CSPDarkNet53	0.8241
GRS-DET	✓	ResNet-101	0.8477
GGHL	✓	DarkNet53	0.8616
AF-OSD (Ours)	✓	CSPDarkNet53	0.8816

further shows the quantitative comparison results between the proposed AF-OSD and the SOTA-oriented ship detection algorithms on the DOTA ship dataset. Among these methods, the ${\mathrm{R}}^3\mathrm{Det}$ method has the highest accuracy among anchor-based methods. Compared with the highest accuracy of the anchor-based method ${\mathrm{R}}^3\mathrm{Det}$, our method has improved by 0.32%. Moreover, among anchor-free methods, the experimental results show that the GRS-DET achieves about 2.36% AP improvement compared to the baseline method because of the Gaussian mask and selective concatenation module (SCM) structure. Compared to the GRS-DET, the proposed MDP-RGH can better suppress the interference of negative samples such as background and noise in the image, and the OSALoss function can further overcome the training imbalance problem caused by the different ship sizes, so our method outperforms the GRS-DET with 3.39% AP improvement on the DOTA ship dataset. The positive and negative sample distribution of GGHL is based on Gaussian prior and is adjusted during the training process, making it applicable to remote sensing targets of any shape, but it will have some impact on regular symmetrical targets (the distribution of positive samples in symmetrical rules will change to some extent after adjustment), resulting in its AP value being $2\%$ lower than the proposed method on the DOTA ship targets.

5.2.2. Results on the HRSC2016 Dataset

For the HRSC2016 test dataset, we directly resize and pad the image to 800 × 800 pixels, and then input it into the model for processing. Based on the qualitative comparison of the detection effectiveness of the proposed AF-OSD and the baseline method (AF-RPN) under the HRSC2016 dataset, it can be seen that our method tends to use the point near the center of the ship to predict the ship’s bounding box. The point marked red means it had the highest score to predict the ship. This indicates that the proposed MDP-RGH can overcome the interference of negative samples such as background and noise in the image by using the center point to detect the ship. Thus, a better result can be realized by our method, as shown in Figure 16. From Figure 16b, it can be obtained that the proposed method effectively reduces the false detection rate of the ships. Moreover, AF-OSD can accurately identify the small targets in ground truth that are not completely labeled in the ground truth, as shown in Figure 16a,d.

Table 5. Comparison results on the HRSC2016 dataset.

Methods	RRPN	ROI-Transformer	${\mathbf{R}}^3\mathbf{Det}$	IENet	AF-RPN	GRS-DET	AF-OSD (Ours)
AP	0.7905	0.8620	0.8926	0.7501	0.8494	0.8957	0.8969

further shows the quantitative comparison results between the proposed AF-OSD and the SOTA-oriented ship detection algorithm on the HRSC2016 dataset. Among the anchor-based methods, the ${\mathrm{R}}^3\mathrm{Det}$ method has the highest accuracy. Compared with ${\mathrm{R}}^3\mathrm{Det}$, the accuracy can be improved by 0.43% by our method. Moreover, among anchor-free methods, the experimental results show that the GRS-DET [6] achieves about 4.63% AP improvement compared to the baseline method. Due to the scale of ships in the HRSC2016 dataset being more uniform, our method, with the help of the MDP-RGH and the OSALoss function, outperforms the GRS-DET with 0.12% AP improvement on the HRSC2016 dataset.

The results proved that the proposed AF-OSD based on MDP-RGH can achieve SOTA through qualitative and quantitative comparative experiments.

5.3. Network Complexity Analysis

Based on the experimental process analysis, considering the network model’s complexity and the computational effort, the proposed AF-OSD based on MDP-RGH in this paper is compared with the anchor-based and anchor-free models, as shown in

Table 6. Comparison of different networks on HRSC2016.

Methods	AP	Params	GPU	Speed
RRPN	0.7905	181 MB	GTX 1080TI	0.2857 s
ROI-Transformer	0.8620	273 MB	RTX 3090	0.1282 s
${\mathrm{R}}^3\mathrm{Det}$	0.8926	227 MB	RTX 3090	0.0950 s
IENet	0.7501	212 MB	GTX 1080TI	0.0592 s
AF-RPN	0.8494	189 MB	RTX 3090	0.0109 s
GRS-DET	0.8957	200 MB	GTX 1080TI	0.0729 s
AF-OSD (Ours)	0.8969	192 MB	RTX 3090	0.0162 s

. The comparison of results is based on the amount of computation and computing time within the corresponding methods’ articles.

The network parameters of our proposed AF-OSD are significantly less, and its parameters are 35 MB less than those in the ${\mathrm{R}}^3\mathrm{Det}$ method. Moreover, compared to the anchor-free method, the parameters of our method are 3 MB more than the baseline, 20 MB less than IENet, and 8 MB less than GRS-DET. Based on the analysis of the toxic network model, the computational complexity of this paper is 136 G FLOPS.

6. Discussion

In this paper, in order to solve the sample imbalance problem and suppress the interference of negative samples such as background and noise, the oriented ship is modeled via the proposed MDP-RGH according to its shape and direction to generate ship labels with more accurate information, which can predict the target for the contribution of positive samples at different positions to judge the target. Additionally, we designed an end-to-end anchor-free oriented ship detector (AF-OSD) net based on MDP-RG and validate its detection performance.

For the necessity study of network modules, the ablation experiment includes three parts: multi-scale structure, MDP-RGH-based label confidence, and MOSA loss. It has been proved that the framework with a multi-scale structure can extract the features of the oriented ship with different scales better. Multi-scale-based detection head setup enhances the robustness of the model. Therefore, the design of the multi-scale structure of the oriented ship detection network is reasonable. We use MDP-RGH to weight the positive samples so that the samples are rotationally Gaussian distributed and have values closer to the edge. The positive samples of our method are soft. As the result of the ablation experiment, the output can use the Gaussian heatmap confidence to predict whether the target is a negative sample or a positive one. A higher positive sample score means that the sample is more representative of the ship. Therefore, the MDP-RGH confidence can better suppress the interference of negative samples such as background and noise in the image, so the detection performance has improved. The improved loss function comparison experiments can prove that the weight can better solve the loss imbalance caused by the large and small ships. The larger the target, the higher the number of positive sample densities included; the smaller the target, the lower the number of positive sample densities included. The ablation experiment shows that the weight can adaptively solve the situation of sample imbalance that easily occurs during training.

Based on the test results on the DOTA ship dataset and HRSC2016 dataset, one can reach that the proposed AF-OSD has the best target recognition effect in remote sensing images with multi-scale ships, complex scenes, and positive and negative sample imbalance. The AF-OSD has features of high accuracy, few parameters of the network model, and high robustness. Due to the complexity of airborne remote sensing scenes, strong background and noise interference, positive and negative sample imbalance, and multiple ship scales, ship detection is a key and challenging task in remote sensing. The proposed method in this paper can better solve the above challenges. In summary, the proposed AF-OSD has a low number of parameters and higher model accuracy, detection accuracy, and computational speed.

As our main research object is ship targets. Most ships in the HRSC2016 and DOTA ship datasets have the characteristic of being narrow at both ends and wide in the middle, and their external contours are similar to rectangles. Therefore, the proposed method in this paper can be extended to the detection of remote sensing targets with similar external contours as rectangles (such as vehicles and stadium-like remote sensing targets) in any orientation, and even be possibly applied to symmetrical targets. However, AF-OSD is not an optimal solution for irregular and asymmetrical remote sensing targets (such as port-like targets). The application of the method in this paper to the detection of irregular targets is somewhat affected by the poor matching of the Gaussian heatmap. Therefore, the detection of full targets will be developed in the subsequent work. Moreover, to improve recognition speed and increase detection accuracy by decoupling the scale and task, the channel and spatial attention mechanism need to be considered in the design. However, the results showed that the inclusion of the attention mechanism actually led to a decrease in the detection accuracy of the network, which means that aspect requires further research and study in future work.

7. Conclusions

This paper presents the AF-OSD (anchor-free oriented ship detector) based on the MDP-RGH (multi-scale dense-point rotation Gaussian heatmap), which is designed for detecting arbitrarily oriented ships. The AF-OSD adopts a PANet structure for multi-scale feature extraction and a decoupled detection head for predicting the rotate Gaussian heatmap confidence, class, and location information of the oriented ship. The AF-OSD is capable of handling ships of different sizes, whose bounding boxes can be of any shape that fits the target. Experiments on two commonly used oriented ship datasets demonstrate the superiority of the AF-OSD over other state-of-the-art methods. These results showcase the high robustness of the AF-OSD in remote sensing ship identification applications and its ability to balance the number of positive and negative samples of various scale targets unsupervised while reducing the impact of noise and background on target identification. As a result, the proposed model has achieved state-of-the-art performance.