VHRShips: An Extensive Benchmark Dataset for Scalable Deep Learning-Based Ship Detection Applications

Abstract

The classification of maritime boats and ship targets using optical satellite imagery is a challenging subject. This research introduces a unique and rich ship dataset named Very High-Resolution Ships (VHRShips) from Google Earth images, which includes diverse ship types, different ship sizes, several inshore locations, and different data acquisition conditions to improve the scalability of ship detection and mapping applications. In addition, we proposed a deep learning-based multi-stage approach for ship type classification from very high resolution satellite images to evaluate the performance of the VHRShips dataset. Our “Hierarchical Design (HieD)” approach is an end-to-end structure that allows the optimization of the Detection, Localization, Recognition, and Identification (DLRI) stages, independently. We focused on sixteen parent ship classes for the DLR stages, and specifically considered eight child classes of the navy parent class at the identification stage. We used the Xception network in the DRI stages and implemented YOLOv4 for the localization stage. Individual optimization of each stage resulted in F1 scores of 99.17%, 94.20%, 84.08%, and 82.13% for detection, recognition, localization, and identification, respectively. The end-to-end implementation of our proposed approach resulted in F1 scores of 99.17%, 93.43%, 74.00%, and 57.05% for the same order. In comparison, end-to-end YOLOv4 yielded F1-scores of 99.17%, 86.59%, 68.87%, and 56.28% for DLRI, respectively. We achieved higher performance with HieD than YOLOv4 for localization, recognition, and identification stages, indicating the usability of the VHRShips dataset in different detection and classification models. In addition, the proposed method and dataset can be used as a benchmark for further studies to apply deep learning on large-scale geodata to boost GeoAI applications in the maritime domain.

1. Introduction

Target classification from optical satellite images is an essential task in remote sensing, which is implemented in various applications to identify geospatial objects such as airplanes, buildings, and ships [1,2,3]. Optical satellite images have been extensively used in the maritime domain owing to satellites’ capabilities to cover large areas very rapidly, provide synoptic views and high-resolution spatial details, and obtain multi-spectral information. These capabilities are used to produce valuable information for marine applications such as fishery [4], water clarity [5], shoreline extraction [6], and habitat mapping [7]. Optical satellite images are effectively used for patrolling the sea against illegal activities and monitoring maritime traffic [8]. Moreover, their broad coverage in near real-time with high spatial details makes optical satellites feasible for defense applications, such as automated detection of different targets [9,10].

Parallel to the progress in optical satellite technology and image processing algorithms, the outputs of the studies have evolved from detection through further stages [11,12]. Researchers have focused on the methods and algorithm development for monitoring the oceans since the first Landsat satellite launched in 1972. McDonnell and Lewis performed one of the preliminary studies on vessel detection using optical satellite imagery [13]. In early studies, defining a vessel from a satellite image was expressed as a classification. However, the classification of ships requires further stages like Detection, Localization, Recognition, and Identification (DLRI). Each stage has to be studied in detail according to the application requirements.

Previous research focused on different aspects of the ship detection and classification problem. Researchers applied different levels of preprocessing methodologies in previous studies, such as background and foreground segmentation (sea, lake, or river where the ship is located) using the Geographic Information System [14], Digital Elevation data [2], statistical methods [15], and deep learning methods [16]. In addition, there are studies on eliminating effects like clouds and haze [14] or minimizing the impacts of ocean waves [17]. While some studies aimed to detect ships in the open ocean [18] or inshore [2,19,20,21], some studies went further and classified the ships according to different levels [22,23].

The dataset’s quality and extent (volume) are among the most critical factors in determining the success of various studies. Some researchers used open-source datasets (Table 1), and some generated custom datasets [19,22,24] for their implementations. Furthermore, some datasets include satellite images obtained from various sensors to differentiate the objects in different levels of detail. Two previous research studies used Sentinel-2 images with 10 m spatial resolution, which are open-source data but contain some limitations in high spatial detail levels [25,26]. On the other hand, some of the studies used very high resolution satellite images with a spatial resolution of 1 m or better [27,28]. Moreover, the spectral bands used in different applications vary depending on the sensor characteristics. While many studies concentrate on Red-Green-Blue bands, or their pan-sharpened versions, some other studies benefited from NIR [29], SWIR [30], and thermal bands [31].

Different investigations have been conducted on various ship sizes regarding detectable object sizes. According to Li et al., even though some studies define the minimum size of the ship to be detected as between 2 and 150 m, most studies don’t specify any limits [10].

Aside from these considerations, the theoretical approaches for DLRI stages have evolved. McDonnell and Lewis employed fundamental statistical image processing methods in one of the first investigations in this area [13]. Today, numerous studies on different levels of the ship DLRI process are based on optical satellite images. In early studies, the outputs were limited to ship detection from the image without localization information due to the low image resolution and image processing power [11]. Parallel to the advances in optical satellite sensors, computational hardware, and software, classification of single ship images, localization of ships from the images, and multi-level classification like recognition and identification of ships have been performed using the threshold-based, salient-based, shape-and-texture-based, statistical, transform domain, anomaly detection, computer vision, or deep learning-based methods [12]. As is also the case in many other engineering disciplines, deep learning-based approaches are being increasingly used in the DLRI process of ships, especially with the success of Convolutional Neural Network (CNN) structures.

Paper Contributions

This study focused on the following requirements regarding dataset and methodology, which were not handled by the previous studies in detail.

• A database with a hierarchical level of classes which makes it suitable for all DLRI stages,
• A database that consists of a large variety of conditions in terms of geography, weather, and spatial arrangement,
• A method that follows all DLRI stages to present a proper remote sensing solution,
• A process that eliminates negative examples (images without ship), which are the largest percentage of images used in the majority of applications, before getting deeper stages,
• A method that forms a suitable base to adopt various deep learning networks and enables end-to-end and repeatable evaluation metrics which exemplify real scenarios.

This research introduces a unique and rich ship dataset named Very High-Resolution Ships (VHRShips). It proposes a unique deep learning-based Hierarchical Design (HieD) approach for multiscale and multipurpose ship detection and mapping purposes. The VHRShips dataset is specifically generated using Google Earth images obtained for different geographical regions and under various atmospheric conditions. Although we selected some images from offshore locations, most of the images in VHRShips contain complex inshore parts. Additionally, the dataset has an extensive range of ship types, including various maritime vessels and navy ship types. VHRShips enables the recognition of sixteen parent classes and a customized identification of navy ship classes for the specified eight categories matching the requirements of various maritime applications on different scales.

The proposed HieD is a multi-stage approach that follows a hierarchy of different stages. The end of each stage is tied to the beginning of the next stage, and all stages have a self-training and self-optimization process. This way, we get the flexibility to use the independent methods for each stage of DLRI, which avoids the loss function optimization for multi-stages. We also compared our proposed HieD approach with the commonly used YOLOv4 model to evaluate the performance of the hierarchical methodology. This research aims to contribute ship detection sub-domain of the maritime re-search field in these aspects: (1) providing a benchmark dataset consisting of large ge-odata, (2) introducing a novel deep learning approach to boost processing and analy-sis of object detection, and (3) automated extraction of 2D geo-information from large satellite image-based geospatial dataset.

2. Related Works

Recent research examined a variety of deep learning algorithms for different stages of DLRI. Some of these studies are summarized below.

Research by Zhang et al., was one of the initial ship detection studies conducted with deep learning techniques [27]. They built a cascade structure joining the feature extraction methods to CNN. The study claimed 99.1% precision for offshore ships and 95.9% for inshore ships. Tang et al., proposed a coarse ship location method by sea/land segmentation and ship/non-ship classification using deep learning based on wavelet coefficients [32]. They used 2000 SPOT-5 satellite images with 5 m resolution, and defined ships, ocean waves, clouds, coastline, and islands as object classes. The accuracy of the method was 97.58%. This study corresponds to the detection stage of our research. Zou and Shi developed the Singular Value Decomposition Network (SVDNet), which uses CNN structure and a SVD algorithm to detect images from optical satellite images [33]. The results are reasonably good, with a high recall rate. Liu et al., used a CNN-based algorithm to detect and classify ships among ten classes with a dataset generated from 1200 Google Earth images [23]. They achieved 99% detection-localization and 92% classification accuracy. Zhang et al., modified Faster RCNN for ship detection-localization [28]. They reached precision values between 92.95% and 97.64% according to the different confidence and intersection over union (IoU) values. Chen et al., used the Deep Residual Dense Network (DRDN) to detect and classify ships [34]. Their classification stage comprised five different non-ship (wave, island, landing stage, cloud, black background) and five different ships (big, medium, small, multiple, with wake) and achieved a 94.21% F1 score. Shi et al., used the BCCT200 dataset for their ship classification study [35]. Their architecture includes multi-features extraction, CNN, and softmax layers. In the last part of the method, they used the decision level fusion approach to vote on five different feature-based branches. The results of this approach achieved 92.5% average classification accuracy for the four classes of BCCT-200. Zhuang, Li, and Chen worked specifically on inshore ship detection. They worked with the structured sparse representation model (SSRM), Google Earth, HRSC 2016, and DOTA datasets, achieving 87.2% precision and 90.2% recall values [36]. Gao et al., emphasized the importance of negative examples in their work and proposed a method without sea–land segmentation [37]. Feng et al. handled the localization and classification separately [38]. They connected the location stage output to the classification input, the same as HieD. Zhang et al., proposed a multi-level ship classification dataset called FGSC-23 and a deep learning-based ship classification model: attribute-guided multi-level enhanced feature representation network (AMEFRN) [22]. The FGSC-23 dataset consists of 4080 images collected from Google Earth and the Gaofen-1 satellite. However, each image patch includes only one type of ship class, and the dataset is not designed for ship localization. Due to the existence of non-ship images, the dataset can be used for detection. The overall classification accuracy of AMEFRN in FGSC-23 is 93.58% in the grained level (L3). Our HieD approach and VHRShips dataset are feasible for classification, detection, and localization, unlike AMEFRN and FGSC-23, which are suitable for classification and detection. Although there have been many studies on DLRI [11,12,39], there are still plenty of gaps to be filled in developing a practical ship DLRI application.

3. Materials and Methods

3.1. Background for Ship Dataset

The dataset used strongly influences the accuracy and validity of DLRI studies. The number of total samples, the variety of the classes, the balance among the classes, the quality, the originality, the validity, the foreground-background separation format, and the accessibility of the datasets are the key specifications required for these studies [40].

This research is built explicitly for the DLRI process of ships from optical satellite images. The main features of the optical satellite images are the number of spectral bands, the spatial resolution, the incident angle of the sensor, and the radiometric resolution, which are provided with the original satellite image and metadata file. On the other hand, deep learning studies require the collection of a large and diverse dataset from satellite images for target recognition/identification; a process that is time-consuming and expensive. When using archive data, it is not always possible to obtain satellite images from the inshore and offshore regions with enough data to represent the ship types. Beyond being ship-specific, the requirement for an extensive dataset is valid for most geospatial applications where objects need to be automatically extracted from high-resolution and very high-resolution optical satellite images. However, such comprehensive datasets are not generally freely available. Nevertheless, some open-source datasets can be used in DLRI studieswith some constraints. Detailed information about the most popular open-source ship datasets is provided in Table 1.

Table 1. Open-source ship datasets for DLRI.

Dataset	Image Source	Application Purpose	Number of Classes /Number of Ship Classes	Description Assessment
VHR-10_dataset_coco [41]	Optical Satellite (Google Earth)	Object classification	10/1	Ships on the images are located yet not classified.
NWPU RESISC45 [42]	Optical Satellite	Scene classification	45/1	Ships on the images are located yet not classified.
DOTA [43]	Optical Satellite	Object classification	15/1	Ships on the images are located yet not classified.
HRSC2016 [23]	Optical Satellite (Google Earth)	Ship detection-localization & classification	19/19	Mostly in-shore images. 19 classes in Level 3. Experimental results only for Level 2 with 4 classes.
Airbus Sandbox Ship Detection ML [44]	Optical Satellite (SPOT)	Ship detection-localization	1/1	Ships on the images are located yet not classified.
xView [45]	Optical Satellite (WorldView-3)	Object classification	60/9	8 parent classes with 60 child classes (maritime vessel parent. class contains 9 different child classes).
HRRSD [46]	Optical Satellite (Google Earth and Baidu Map)	Object classification	13/1	Ships on the images are located yet not classified.
FGSD [47]	Optical Satellite (Google Earth)	Ship detection-localization & classification	44/43	Mostly in-shore images. There are two classification levels. Level 1 consists of submarine, merchant ship, aircraft carrier and warship classes.
BCCT200 [48]	Optical Satellite	Ship classification	4/4	A broad class definition with only gray-scale ship images from the RAPIER system.
MASATI v2 [49]	Optical Satellite (Microsoft Bing Maps)	Ship detection-localization	1/1	Classes: Ship, multiple ships, ship with no coast, ship(s) with coast, sea with no ship, and land with no sea.
FGSC-23 [22]	Optical Satellite (Google Earth and Gaofen-1 Satellite)	Ship classification	23/23	Three classification levels are defined which are ship-non ship (L1), coarse (L2) and grain (L3).

Table 1. Open-source ship datasets for DLRI.

Dataset	Image Source	Application Purpose	Number of Classes /Number of Ship Classes	Description Assessment
VHR-10_dataset_coco [41]	Optical Satellite (Google Earth)	Object classification	10/1	Ships on the images are located yet not classified.
NWPU RESISC45 [42]	Optical Satellite	Scene classification	45/1	Ships on the images are located yet not classified.
DOTA [43]	Optical Satellite	Object classification	15/1	Ships on the images are located yet not classified.
HRSC2016 [23]	Optical Satellite (Google Earth)	Ship detection-localization & classification	19/19	Mostly in-shore images. 19 classes in Level 3. Experimental results only for Level 2 with 4 classes.
Airbus Sandbox Ship Detection ML [44]	Optical Satellite (SPOT)	Ship detection-localization	1/1	Ships on the images are located yet not classified.
xView [45]	Optical Satellite (WorldView-3)	Object classification	60/9	8 parent classes with 60 child classes (maritime vessel parent. class contains 9 different child classes).
HRRSD [46]	Optical Satellite (Google Earth and Baidu Map)	Object classification	13/1	Ships on the images are located yet not classified.
FGSD [47]	Optical Satellite (Google Earth)	Ship detection-localization & classification	44/43	Mostly in-shore images. There are two classification levels. Level 1 consists of submarine, merchant ship, aircraft carrier and warship classes.
BCCT200 [48]	Optical Satellite	Ship classification	4/4	A broad class definition with only gray-scale ship images from the RAPIER system.
MASATI v2 [49]	Optical Satellite (Microsoft Bing Maps)	Ship detection-localization	1/1	Classes: Ship, multiple ships, ship with no coast, ship(s) with coast, sea with no ship, and land with no sea.
FGSC-23 [22]	Optical Satellite (Google Earth and Gaofen-1 Satellite)	Ship classification	23/23	Three classification levels are defined which are ship-non ship (L1), coarse (L2) and grain (L3).

However, to process DLRI in an optical satellite image, the dataset must be suitable and include the required number of images for the training, validation, and testing of all DLRI stages defined in this research. Other requirements of an effective dataset are given below.

• High definition images collected by optical satellites,
• Images with spatial resolution information to ensample a real scenario,
• Image metafile of an existing ship with location and class information,
• Defined recognition and identification levels of each ship in the dataset,
• Images from the inshore and offshore sea,
• Images with rural and urban coasts,
• Images from different locations around the world,
• Some images with clouds and wave clutter to sample a noisy background.

When we focused on the above-indicated requirements, we concluded that the existing datasets were not eligible for this research.

3.2. VHR Ship Dataset

Considering the limitations of existing ship datasets, we generated a new dataset called Very High-Resolution Ships (VHRShips) [50] by using Google Earth images. VHRShips contains 6312 images from 52 locations collected between 26 June 2005 and 1 February 2021. The VHRShips dataset includes 1000 images without a ship and 5312 images with single or multiple ships.

While forming the VHRShips dataset, the zoom scale of all images was kept near-identical by using Google Earth Professional software (Google Inc., Mountain View, CA, USA, version 7.3.4.8642)’s “eye altitude level adjustment” feature. All images were collected at 500 m eye altitude level and in the high-definition spatial resolution format (720 × 1280 pixels). Through this effort, all images in VHRShips artificially have the same spatial resolution information, corresponding to 43 cm per pixel sampling. Some images with higher or lower spatial resolution were downscaled or upscaled, respectively.

3.3. Class Definitions

The definition of each DLRI stage used in this research is justified by considering the STANAG 3769 “minimum resolved object sizes and scales for imagery interpretation” as a reference [51]. After reinterpreting these definitions according to the purposes of this research and adding a new definition for localization, the DLRI stages can be described as follows:

• Detection: The discovery of the ship’s existence (s) in the optical satellite image.
• Localization: Determination of the ship’s precise location (s) in the given optical satellite image.
• Recognition: Defining the parent class of each image among the civilian and navy ship groups.
• Identification: Defining the precise child class of each navy ship.

The parent ship classes were initially determined by Lloyd’s Register ship classification [52] and Jane’s Fighting Ship classification [53]. We re-arranged the classes of VHRShips after analyzing the obtained satellite images and ship representations. According to the class definitions, the VHRShips dataset includes 11,179 ships in 24 parent and 11 child ship categories, which are given in

Table 2. VHRShips dataset specifications.

Property	Value/Description
Number of images	6312
Number of images with ship(s)	5312
Number of images without ship	1000
No. of ships	11,179
No. of classes	24 parent, 11 child
Image spectral channel	red-green-blue
Image spatial resolution	1280 × 720
Image source	Google Earth Professional
Bounding box (bBox) type	Horizontal-rectangle
bBox statistics (min–mean–max)	Length: 7–183–991 Width: 6–47–428 bBox area: 42–13,640–282,226

and

Table 3. VHRShips dataset class distribution.

Class Distribution-Number of Ships
Parent classes (23 civilian classes and 1 navy class) and non-ship images
Lpg: 33	drill: 33	floatingDock: 51
Ferry: 99	Roro: 102	Offshore: 122
Passenger: 143	Dredging: 194	bargePontoon: 714
dredgerReclamation: 268	Coaster: 344	Undefined: 417
smallPassenger: 419	smallBoat: 891	Tug: 846
Yacht: 1581	Fishing: 37	Container: 580
oilTanker: 594	Tanker: 777	generalCargo: 655
bulkCarrier: 677	oreCarrier: 771	Navy: 831
nonShip: 1000
Child classes (navy):
Other: 27	Aircraft: 35	Landing: 34
coastGuard: 40	Submarine: 50	Cruiser: 68
Frigate: 71	patrolForce: 101	Destroyer: 90
serviceCraft: 151	Auxilary: 164

. The parent classes contain one navy and 23 civilian ship class definitions, and stand for the recognition level classification. On the other hand, the child classes detailed 11 navy ship classes of the navy parent class, representing the identification level classification.

VHRShips is a diverse and comprehensive dataset aiming to represent various real-life conditions. It includes complex backgrounds such as urban, civilian, navy shores, bridges, breakwaters, forests, canals, small and big islands/cliffs, and industrial zones. The dataset also comprises images with different ocean spectral properties, an advantage of worldwide image selection strategy. Some images contain atmospheric disturbances such as fog, haze, or clouds. Moreover, VHRShips includes some images of rough sea conditions and some images of ship wakes. These properties make it challenging to obtain successful results for all of the DLRI stages. Appendix A 

Table A1. Geographic locations of images collected to create VHRShips Dataset.

No.	Location	No.	Location
1	Port of Shanghai	22	Port of Hong Kong
2	Port of Tianjin	29	Port of Shenzhen *
3	Port of Guangzhou	30	Suez Canal
4	Port of Ningbo	31	The Panama Canal
5	Port of Rotterdam	32	Naval Station Norfolk
6	Port of Qingdao	33	Naval Station Mayport
7	Port of Dalian	34	Devonport, Plymouth
8	Port of Busan	35	Portsmouth
9	Port of Nassau	36	Port of Canaveral
10	Port of Barcelona	37	Port of Everglades
11	Port of Civitavecchia	38	Port of Cozumel
12	Port of The Balearic Islands	39	Port of New York and New Jersey
13	Port of Southampton	40	Okinawa Naval Base White Beach
14	Snezhnogorsk	41	Yokosuka Naval Base
15	Tartus, Syria	42	Sasebo Naval Base
16	Venice	43	Toulon
17	Taranto Navy Base	44	Sevastopol
18	Navy Augusta	45	San Francisco
19	Port of Los Angeles	46	Dardanelles *
20	Port of Haydarpasa	47	Bosphorus
21	Port of Izmir	48	Port of Mersin
23	England	49	Port of Antalya
24	Jordan Agaba	50	Hanau Germany
25	Kocaeli gulf	51	Port of Mersin
26	Norway	52	Pearl Harbour
27	Qatar	53	Qatar Port of Raundabout
28	San Diego	54	Port of Singapore

* These candidate locations aren’t used in VHRShips.

demonstrates the geographic locations of the collected images used to create this dataset.

After conducting initial experiments, we re-arranged the original dataset to minimize the confusion of some very similar ship classes. After this modification, this research employed the DLRI process with sixteen parent ship classes for the recognition-level classification, and eight child navy classes for the identification-level classification. This specified dataset is generated from the original dataset by performing a re-group process, which is detailed below:

• dredging and bargePontoon classes are grouped into bargePontoon (representing the steady platforms),
• smallPassenger, smallBoat, tug, yacht, and fishing classes are grouped into smallBoat (small size ships),
• oilTanker and tanker classes are grouped into tanker (common tanker group),
• generalCargo, bulkCarrier, and oreCarrier are grouped into generalCargo (cargo ships),
• cruiser, frigate, patrolForce, and destroyer are grouped into destroyer (combat navy ship group).

Some image examples from VHRShips, which are; with/without ships, clear/noisy, simple/complex, with/without land background, with calm/rough sea conditions, and with urban/rural background, are given in Appendix B Figure A1, Figure A2, Figure A3, Figure A4, Figure A5 and Figure A6.

3.4. Hardware and Software Configuration

This research does not include any computational performance analysis since it is research-oriented. Therefore, we used currently available mid-range hardware. The information related to hardware, software, and computing environments is provided in

Table 4. The hardware and software used in the study.

Hardware Parameters
Parameter	Value/Type
Processor	Intel Core i7-6700 CPU 3.4 GHz
RAM	32 GB
System	64 bit Windows 10
GPU Name	NVIDIA GeForce GTX 1060 3 GB
GPU Compute Capability	6.1
GPU Total Memory	3.22 GB (2.5 GB available for computing) 1152 CUDA cores 128 bit
Software Parameters
Software Name	MATLAB 2021a Academic Version Google Colaboratory
Software Libraries	MATLAB Computer Vision Toolbox 10.0 MATLAB GPU Coder Support Package
	MATLAB Parallel Computing Toolbox 7.4 MATLAB Deep Learning Toolbox 14.2 MATLAB Embedded Coder 7.6 MATLAB Coder 5.2 MATLAB Image Processing Toolbox 11.3 CUDA 10.1 cuDNN 9.0

.

3.5. Methodology

The proposed HieD approach is a multi-stage learning structure used for the DLRI process of the ships in optical satellite images. The main difference between HieD and the conventional multi-stage structures is its independent but hierarchical stage formation. HieD provides extensive flexibility in terms of plugging any method to each stage. The only link between the stages is the contribution of success and failure of different stages to the next stage. Xception and YOLOv4 were chosen for this study, but the HieD method can easily be used with other deep learning networks. The schematic representation of HieD is given in Figure 1, and brief descriptions of the HieD stages are provided below.

3.5.1. Detection

In almost all optical satellite image-based applications, most of the images showing a total background are labeled as negative examples. Before dealing with the ships in the positive samples, eliminating the negative examples ensures suitable inputs for further stages of DLRI. At this point, HieD’s detection stage accomplishes this task by classifying images as positives (with ship(s)) or negatives (without ship), without regard for the location of the ship(s). For this binary classification, well known deep learning classifier networks like Alexnet [54], ResNet 18/50/101 [55], DenseNet 201 [56], GoogLeNet [57], EfficientNet [58], and Xception [59] were implemented, and the highest overall accuracy was achieved with the Xception architecture.

Most images have spatial (x and y) plus channel dimensions, which are handled as a whole by the CNNs, generating a high computational load for the network. Chollet proposes applying spatial convolution to each channel separately [59]. Additionally, 1 × 1 convolution kernels are used to build the cross-channel correlations to implement the depth-wise separable convolution. Xception is a follow-up of Inception versions 1, 2, and 3 [60]. It performs the convolution kernels first and then applies channel-wise spatial convolution. There is no intermediate ReLU non-linearity layer in the Xception network. The parameters to tune Xception in HieD are given in

Table 5. The parameters used in the detection stage.

Parameter	Value/Type
Network	Xception (MATLAB version 21.1.1) pre-trained with ImageNet database
Network input	416 × 416 × 3 (original images are resized)
Network output	ship/non-ship (binary classification)
Training optimizer	Stochastic Gradient Descent with Momentum (SGDM)
Loss function	Cross-entropy
Data augmentation	Random rotation [0, 360] & random X and Y reflection
Initial learning rate (LR)	0.001
LR drop frequency	Every 10 epochs
LR drop factor	0.1
Number of epoch	50
Mini-batch size	8
Detection threshold	0.2

. In brief, the binary classification is applied to the input images, and only the images labeled as “including ship(s)” pass through the next stage, localization.

3.5.2. Localization

Some image patches of VHRShips contain multiple or very small ships that cannot be classified without extracting them from the background. This fact prevents implementing the classification algorithm directly on the image as a whole. The general approach to localizing the objects in the images is to divide the image into sub-images and propose the regions that can contain ship(s). While the sliding window is the most straightforward and least computationally efficient method to divide the images into sub-images, the selective search, the regional proposal network, and the anchor-based search are newer and more efficient approaches. In this research, YOLOv4 is chosen for the localization of ships [61]. Indeed, the YOLO architecture is an end-to-end solution that covers localization, detection, and classification. YOLO optimizes the classes and the bounding boxes (bBox) simultaneously, unlike the proposed HieD. However, in the proposed approach, only the bBox localization feature of YOLOv4 is used (Figure 2).

For the localization stage, YOLOv4 provides a black-box procedure that gets the image labeled with ship existence and exports the bBox of the ship(s) in [x, y, width, length] format.

YOLOv4 is trained and tested with different parameters during the optimization of the localization step. At first, YOLOv4 is trained with one class corresponding to the ship as the foreground and the rest as the background to force the network to focus on only the localization regression. However, the best result is achieved with two classes: the navy and the civilian ships. Additionally, to suppress multiple detections of the same ship, Non-Maximum Suppression (NMS) is applied [1]. The rest of the parameters to tune YOLOv4 for the localization stage are given in

Table 6. The parameters used in the localization stage.

Parameter	Value/Type
Network	YOLO v4 pre-trained with Coco dataset
Backbone	Cross Stage Partial Network Darknet-53
Network input	608 × 608 × 3 (original images are resized)
Network output	[x, y, width, height] bBox with the score
Loss function	Complete Intersection Over Union (CIoU)
Data augmentation	Cutmix
Initial learning rate (LR)	0.001
LR decay factor	0.0005
Number of epochs	160
Batch size	64
Mini batch size	2
bBox score threshold	0.4
Ground truth overlap threshold	0.05
Non-maximum suppression threshold	0.75

.

3.5.3. Recognition

The ship bBox with various sizes was obtained as the output of the localization stage. However, the input for the classification nets should have a fixed size. The general approach to overcome this challenge was to resize input images according to the network requirements. The VHRShips dataset includes ship bBox ranging between 42 and 283,336 square pixels in size (Table 2). If the standard image resizing is applied, some ships will be exposed to extreme up-scaling, while others will be exposed to down-scaling. As previously mentioned, this research aims to perform DLRI stages in the optical satellite images, which are scaled. Thus, to avoid losing the scale information via down-scaling or up-scaling, it was decided to keep the extracted ships at their original sizes except for the patches that include ships bigger than the input sizes of the recognition and the identification stages. Firstly, we implemented a zero-padding process around the ships to fit all the extracted images into the input sizes of the classification stages. Afterwards, we determined that the zero-padded pixels, especially in very small objects, should avoid the weights of networks when adequately trained. Finally, the Pattern Forming via Mirror Overlaying (PFMO) approach was proposed to overcome this problem. The PFMO approach follows the steps indicated below.

• Analyzing the length-width ratios of the bBox and rotating it to ensure that the long side is horizontally aligned.
• Putting the rotated bBox in the center and overlaying the rotated bBox around the center bBox until the input size of the stage is filled.
• While overlaying the bBox, the reflection in the X and Y directions is applied to prevent the unintended gradient forming.

The recognition stage uses the Xception network, the same as the detection and identification stages. Even though the same network was used in the detection and the recognition stages, few of the tuning parameters were optimized for the recognition stage, which is briefly explained as:

• The input is gathered through PFMO with 416 × 416 × 3 patch size.
• The output is the probability of 16 parent classes.
• No threshold is applied at the classification level output, the class with the highest score is selected.

Visualization of the PFMO process is provided in Figure 3.

3.5.4. Identification

This stage is an extension of the recognition stage that takes the inputs recognized as “navy ship” and identifies the child classes of “navy ship” as an output. This stage uses the same tuning parameters as the recognition stage for the Xception network training, except for using 8-child classes.

4. Results and Discussion

The results of the study are provided by three approaches. Firstly, the individual performances of the stages are provided. Secondly, the end-to-end evaluation is presented. Lastly, the performance comparison between HieD and one of the state-of-art networks, YOLOv4, is provided. We separated the dataset for training, validation, and testing. While 80% of the VHRShips dataset are kept for the training and validation (64% training and 16% validation), 20% of the dataset is used for testing. In summary, 4016 images are reserved for the training, 1021 images are reserved for validation, and 1275 images are reserved for testing. The images for the training, validation, and testing sets are selected considering the equal class distribution among the sets. There are 200 non-ship images in the testing set, and 1075 images with 2176 marked ships, 142 of which are navy ships.

4.1. Individual Stage Performances

The input of each stage is fed with entirely true samples while calculating the individual stage performances. For instance, only images featuring ship(s) are used in the localization stage. Only the navy class ships retrieved with the ground truth bBox are provided in the identification stage. The individual stage results are summarized in

Table 7. The performance of the individual stage (values in percentage).

Stage	Accuracy	Recall	Precision	F1-Score
Detection	98.59	99.81	98.53	99.17
Localization	89.04	91.04	97.60	94.20
Recognition	84.31	72.57 1	78.33 1	75.34 1
		84.30 2	83.85 2	84.08 2
Identification	80.99	83.67 1	86.63 1	85.12 1
		80.98 2	83.31 2	82.13 2

¹ Arithmetic mean value; ² Class sample weighted mean value.

.

The results showed that the accuracy decreases according to the level of the stage; therefore, details increase. The accuracy of the detection phase is very high despite the complex backgrounds. More importantly, the recall value for the next stage shows promise for avoiding missing any ship(s) images. The confusion matrix of the detection stage is given in

Table 8. The confusion matrix of the detection stage (values in percentage).

		Predicted Class
		Non-Ship	Ship
True Class	Non-ship	92	8
	Ship	0.2	99.8

.

The localization stage also presents precise outputs, but the relatively lower recall value causes some of the ships to be missed, thus resulting in lower accuracy in the following stages.

The confusion matrix of the recognition stage is given in

Table 9. The confusion matrix of the recognition stage (values in percentage).

		Predicted Class																Recall
		bargePontoon	Coaster	Container	dredgerRec	Drill	Ferry	floatingDock	generalCargo	Lpg	Navy	Offshore	Passenger	Roro	smallBoat	Tanker	Undefined
True Class	bargePontoon	81.7	0.6	1.1		0.6			3.9		5.0		0.6		4.4	1.7	0.6	82
	Coaster	3.7	31.5				1.9		7.4		1.9				29.6	13.0	11.1	31
	Container	0.9		78.4					20.7									78
	dredgerRec	3.7			94.4				1.9									94
	Drill	16.7				66.7						16.7						67
	Ferry	11.8					47.1		5.9		5.9				17.6	5.9	5.9	47
	floatingDock	25						50	8.3		16.7							50
	generalCargo	1.7	0.7	1.0					90.4		0.5	0.2			1.2	2.4	1.9	90
	Lpg									100								100
	Navy	2.1	0.7				1.4	0.7	1.4		79.6			0.7	7.0	2.1	4.2	80
	Offshore	8.0							4.0		4.0	68.0	4.0			4.0	8.0	68
	Passenger									3.7	3.7		70.4		14.8		7.4	70
	Roro			5.0					5.0		5.0			85				85
	smallBoat	1.2	1.2						0.4		3.6	0.1	0.3		91.7	0.5	0.9	92
	Tanker	0.8	1.5				0.4		3.4		0.4		0.4		2.3	91.0		91
	Undefined	15.2	2.5	1.3			1.3		7.6		6.3	3.8	1.3		22.8	2.5	35.4	35
Precision		76	46	92	100	80	62	86	86	88	69	74	76	94	91	89	46

. The barge-pontoon, the container, the general cargo, the LPG, the navy, the roro, the small boat, and the tanker classes have close accuracy values to the overall accuracy of the recognition stage. On the other hand, the coaster, the drill, the ferry, the floating dock, the offshore, the passenger, and the “undefined” classes illustrate lower accuracy values than the overall value. When the less accurate classes are analyzed, the following outcomes are deducted:

• The coaster class consists of small water tankers, ore carriers, or cargo ships, which contain small ships that can be confused with small boats, as in this study. In particular, relatively low-resolution images do not include precise details and fail to detect compelling features of these small ships.
• Even though the number of ships in the drill ship class is minimal, the accuracy is close to the overall accuracy. Additionally, the drill class features are very similar to the barge-pontoon class. The drilling instruments in a drill ship are generally located on a barge-pontoon.
• The floating dock class is another class with limited sampling, illustrating a mixture with the barge-pontoon due to the similar shape form and the navy classes since floating docks carry navy ships in the test set.
• The offshore ship class has two characteristics: the navigating bridge on the bow side of the ship and a high bow freeboard. It seems that the number of samples in this class and the level of the differential features wasn’t enough to perform over the overall value.
• Some big yachts in the small boat class can be confused with passenger ships. A decline in accuracy regarding the passenger class relative to the overall results from this variety can be noted.
• The “Undefined” class comprises ships with different features from the defined classes or having a pure resolution that makes them impossible to be classified. Most ships in the “undefined” class do not have common features to be learned by the network. Thus, the low accuracy in this class is an expected result.

The confusion matrix of the identification stage is given in

Table 10. The confusion matrix of the identification stage (values in percentage).

		Predicted Class
		Aircraft	Auxiliary	coastGuard	Destroyer	Landing	Other	serviceCraft	Submarine	Recall
True Class	Aircraft	100								100
	Auxiliary	5	60		30	5				60
	coastGuard			70	30					70
	Destroyer	1.8	3.6	1.8	92.7					93
	Landing					100				100
	Other				33.3		66.7			67
	serviceCraft		10			10		80		80
	Submarine								100	100
Precision		67	89	88	75	75	100	100	100

. The aircraft, the destroyer, the landing, the service craft, and the submarine classes have close or higher accuracy values according to the overall accuracy of the identification stage. On the other hand, the auxiliary, the coast guard, and the “other” classes are under the overall value. When the less accurate classes are analyzed, the following outcomes are deduced:

• The auxiliary class is mainly confused with the destroyer class. Both classes have similar sizes, and some auxiliary classes have similar features such as helicopter decks, sensor masts, and a navigational bridge at the bow.
• The same confusion exists between the coast guard and the destroyer classes. The coast guard ships mainly have the same form as the destroyers (especially with the frigates or the patrol ships). They differ in color (white for the coast guard class and navy gray for the destroyer class).
• The “other” class case is the same as the “undefined” class in the recognition stage.

4.2. End-to-End Performance

The HieD DLRI for optical satellite image is formed when all stages are attached end to end. The conventional metrics are insufficient for analyzing the multiple stages necessary to evaluate the HieD approach. Therefore, falseNegative2 and falsePositive2 terms are defined, and their meanings are clarified in

Table 11. The performance of the end-to-end HieD approach.

Metric	Detection	Localization	Recognition	Identification
falseNegative	2 (No. of images labeled as without ship(s) even with ship(s))	226 (No. of ships which are missed)	228 (No. of ships which couldn’t be detected & localized, so couldn’t be recognized)	29 (No. of ships couldn’t be recognized as Navy)
falseNegative2	NA (invalid metric for this stage)	2 (No. of ships couldn’t be detected, so couldn’t be localized)	NA (invalid metric for this stage)	7 (No. of ships which couldn’t be localized, so couldn’t be identified)
falsePositive	16 (No. of images labeled as with ship(s) even without ship (s))	46 (bBoxes labeled as ship even not)	405 (No. of ships wrongly recognized)	21 (No. of Navy ships wrongly identified)
falsePositive2	NA (invalid metric for this stage)	NA (invalid metric for this stage)	46 (No. of bBox labeled as ship even not, so incorrectly recognized)	50 (No. of ships recognized as Navy even not, so incorrectly identified)
truePositive	1073 (No. of images with ship(s)s)	1948 (No. of true localized ships)	1543 GTruth (No. of ships truly recognized)	85 (No. of ships truly identified)
trueNegative	184 (No. of images without ship(s))	16 (No. of images labeled as with ship(s) but none of ship localized)	NA (invalid metric for this stage)	NA (invalid metric for this stage)
Accuracy	98.59%	87.76%	69.44%	44.27%
Recall	99.81%	89.52%	70.91%	59.86%
Precision	98.53%	97.69%	77.38%	54.49%
F1-Score	99.17%	93.43%	74.00%	57.05%

. While HieD offers several flexibilities in designing the stages, it introduces the error cycle due to the serial connection. We deduced the following outcomes when we evaluated the output metrics in Table 11.

• The detection stage has the same results as the individual because it is the first stage to stay out of the error cycle. Only two images are missing, and each of them have one ship. The results of the detection phase are very satisfactory, except for the false positives.
• The localization stage is also not affected by the error cycle. Sixteen false true images detected in the previous stage are eliminated in this stage with true negative labeling. Thus, the results are very close to the individual evaluation. The number of missed ships is a bit high, which lowers the recall values of the following stages. On the other hand, a low false positive value increases the precision value of the next stages.
• The recognition starts with 274 errors from the previous two stages. For this reason, the accuracy and recall values are lower than they are in the individual evaluation.
• The identification evaluation metrics are a bit higher than 50 percent. Even though this may seem low, the relatively low number of ships belonging to navy classes, the compulsory samples in the dataset, and the below-average recognition performance (falseNegative and falsePositive2) of the navy class generated these results.

4.3. Comparison with the State of Art Network

Beyond the individual or end-to-end performance metrics of the proposed database and method, the comparison with the well-known networks is crucial. At this point, the uniqueness of both VHRShips and HieD doesn’t easily enable the adaptation of previous databases to HieD or methods to VHRShips. The main bottlenecks are the end-to-end evaluation process, four levels for databases, and four stages for methods. On the other hand, to show the position of HieD among the well-known networks, YOLOv4 is adapted to the DLRI process and end-to-end evaluation process. The results of the comparison between HieD and YOLOv4 are given here. YOLOv4 is a one-stage detector network. It processes the image and provides objectiveness with bBox and the class of the target(s). Unlike YOLOv4, HieD presents two levels of classification. For this reason, YOLOv4 is trained with twenty-three classes (fifteen parent classes plus eight child navy classes) to make a complete comparison. The classification results are divided according to the groups used in the recognition and identification stages of HieD. While the objectiveness output of YOLOv4 corresponds to the detection stage, the localization stage is a common part of the two approaches. Fifteen civilian ship classes plus eight navy ship classes on behalf of the general navy class form the recognition stage. The results of eight navy ship classifications are accepted for the identification stage. Considering these pre-assumptions, YOLOv4 was trained with the same parameters, and the results are presented in

Table 12. The performance metrics of HieD and YOLOv4.

Stage/Metric	Proposed HieD				YOLO v4
	Accuracy	Recall	Precision	F1-Score	Accuracy	Recall	Precision	F1-Score
Detection	98.59%	99.81%	98.53%	99.17%	98.59%	99.44%	98.89%	99.17%
Localization	87.76%	89.52%	97.69%	93.43%	76.35%	86.41%	86.77%	86.59%
Recognition	69.44%	70.91%	77.38%	74.00%	60.72%	68.72%	69.01%	68.87%
Identification	44.27%	59.86%	54.49%	57.05%	43.05%	45.77%	73.03%	56.28%

.

A visual representation of the comparison between the proposed HieD and YOLO v4 for inshore regions is provided in Figure 4, and provided in Figure 5 for offshore regions.

The detection performances of HieD and YOLOv4 in VHRShips are almost the same. For localization, HieD performs better than YOLOv4, even though it also uses the bBox localization feature of YOLOv4. The main reason behind this phenomenon is the number of classes used in the YOLOv4, which is twenty-three, while the YOLOv4 used in HieD deals only with two classes. In addition, YOLOv4 tries to tune classification for twenty-three classes while trying to reduce the bBox loss at the same time. This outcome is one of the reasons that HieD is proposed. The recognition performance of YOLOv4 is also less than HieD’s. The limited number of samples for navy classes negatively affects the recognition level classification. Lastly, the identification performance of HieD is better except for the precision metric. The reason for that is the high number of false positive-2 in HieD, originating from the recognition stage.

4.4. Model Limitations and Future Work

Although the positive contributions of the introduced VHRShips dataset and proposed HieD approach are explained in the previous sections, our approach has some shortcomings that need to be considered. More research can be conducted on the below captions to improve the performance further.

• An additional sub-stage can be designed to detect and eliminate false positive samples for the localization, recognition, and identification stages, if further performance is needed. This design can also reduce the effects of the error cycle created by false positive outcomes.
• The distribution of unbalanced classes can be improved by implementing different over/under-sampling approaches, and the number of samples can be increased for the classes with limited data.
• Introducing more images with different classes will strengthen the dataset for the future.
• Different CNN networks can be integrated to increase the performance of the stages.

5. Conclusions

This research introduces a scalable ship detection dataset—VHRShips—and proposes a deep learning-based HieD approach to perform an end-to-end DLRI process with optical satellite images. HieD is a dynamic and iterative approach formed with the classical CNN and YOLO networks, in which each stage can be independently reformed and tuned to maximize performance. This flexibility is not the only feature that HieD proposes; the PFMO method also helps to boost the performance of the recognition and identification stages. Additionally, the evaluation method used in the study is slightly different from the conventional methods in that we considered the effects of the error cycle and represented them in the outputs. Even though the stages of DLRI are connected with the strict error cycle phenomenon, the performance stays above a satisfactory level. Additionally, a comparison with the one-stage YOLOv4 method provides positive feedback. Another important outcome of this research is the VHRShips dataset, which was prepared from optical satellite images. We generated a complex dataset containing a large variety of ship classes with different ship sizes at various inshore and offshore background distributions to objectively represent real-life conditions. Moreover, images from this dataset reflect different acquisition conditions, such as viewing geometry and seasonal variations, so that it can be used as a base dataset for further studies related to maritime mapping and ship detection. The outputs of HieD are very promising for further studies and could be used as a benchmark.

In conclusion, the VHRShips dataset and proposed approach could be used for the transfer learning and generalization evaluation of similar tasks. The VHRShips dataset, HieD method, and end-to-end evaluation metrics adequately represent real-world applications.