Real-Time Ship Tracking under Challenges of Scale Variation and Different Visibility Weather Conditions

Abstract

Visual ship tracking provides crucial kinematic traffic information to maritime traffic participants, which helps to accurately predict ship traveling behaviors in the near future. Traditional ship tracking models obtain a satisfactory performance by exploiting distinct features from maritime images, which may fail when the ship scale varies in image sequences. Moreover, previous frameworks have not paid much attention to weather condition interferences (e.g., visibility). To address this challenge, we propose a scale-adaptive ship tracking framework with the help of a kernelized correlation filter (KCF) and a log-polar transformation operation. First, the proposed ship tracker employs a conventional KCF model to obtain the raw ship position in the current maritime image. Second, both the previous step output and ship training sample are transformed into a log-polar coordinate system, which are further processed with the correlation filter to determine ship scale factor and to suppress the negative influence of the weather conditions. We verify the proposed ship tracker performance on three typical maritime scenarios under typical navigational weather conditions (i.e., sunny, fog). The findings of the study can help traffic participants efficiently obtain maritime situation awareness information from maritime videos, in real time, under different visibility weather conditions.

1. Introduction

Smart ships are considered as fundamental facilities in the green shipping era, offering the potential of significantly reducing carbon emissions, loading larger cargos, automatically making navigation decisions, etc. [1,2,3]. Much research attention has been given to collecting on-spot traffic data from varied maritime sensors, which are further processed to fulfill maritime situation awareness tasks. Previous studies suggest that spatial-temporal ship positions provide crucial ship kinematic information, which provides elementary maritime information support for traffic situation awareness [4,5,6]. The ship position dataset can be easily obtained due to a large number of sensors (e.g., maritime surveillance cameras and automatic identification system (AIS) facilities) that are widely deployed both onboard and on off-shore buildings, and thus ship position streaming data is continuously obtained [7]. The long-range identification and tracking (LRIT) technique is mainly employed to identify ship positions when they sail in an ocean area (i.e., far away from the coastal region) [8,9,10]. However, the LRIT technique updates the ship position with a large time span (approximately every six hours), which cannot be used in real-time maritime applications.

AIS data are widely used to locate ship positions when they sail in inshore areas, and AIS-based ship positions are updated in a real-time manner. More specifically, the onboard AIS equipment broadcasts its position every 10 s. Note that the time interval can be smaller when the ship moves at a higher speed, and vice versa. AIS data have successfully fulfilled maritime situation awareness tasks (e.g., near-miss ship collision avoidance, ship trajectory exploitation and prediction, illegal maritime event identification, etc.) [11,12,13,14,15]. The following weaknesses may significantly reduce AIS performance: (1) it is not easy to collect fishery boat AIS data as it is not mandatory to be installed on the facility; (2) several ships may deliberately deactivate their AIS equipment for the purpose of avoiding being sensed by neighboring ships and maritime regulations (e.g., warships, smuggling ships, etc.). It is noted that we can employ object recognition techniques to further exploit distinct ship features from coastal maritime surveillance videos to identify warships and smuggling ships. (3) We cannot easily obtain noise-free AIS data due to fake plate ships, AIS data loss, etc. [16,17]. In that manner, many preprocessing steps (such as data quality control, anomaly detection, outlier correction, etc.) are required before employing AIS data to analyze maritime traffic situation awareness.

Computer vision-based techniques have enjoyed many successes when implemented for the ship-tracking task, showing potential in tackling the maritime traffic situation awareness problem under adverse weather constraints [18,19,20]. More specifically, visual ship tracking results provide microscopic on-spot traffic parameters which help maritime traffic participants (e.g., ship crew, maritime traffic authorities, etc.) make reasonable sailing decisions in advance. Chen et al. employed varied deep learning models (e.g., convolution neural network, you only look once (YOLO), etc.) to exploit high-level ship features, and thus maritime spatial-temporal information (e.g., trajectory, heading direction) could be extracted and further analyzed [21]. Zhang et al. introduced a discrete cosine transformation-based ship detection model for the purpose of efficient maritime traffic surveillance from non-stationary videos (e.g., cameras installed on buoys, ships, etc.) [22]. Jung et al. proposed a clustering-based optimization method to estimate ship positions from maritime video clips, which was tested in a realistic maritime rescue application in South Korea [23]. Chen et al. proposed a hybrid model via the help of a generative adversarial network and a convolutional neural network to accurately detect small ships in maritime surveillance videos [24]. Zhang et al. introduced a novel detection framework to detect ships in arbitrary orientations in the maritime images by developing a rotated regional proposed network [25]. Similar studies can be found in [26,27].

Previous studies suggest that visual ship tracking tasks are implemented by extracting visual features of the ship from maritime images (synthetic aperture radar (SAR), visible light-based videos, etc.), taken under relatively good weather conditions [28,29,30]. Much effort is put into exploiting hand-crafted features from maritime SAR images which are sensitive to sea clutter interference (i.e., they can be obviously observed under storm navigation environment) [31,32]. Deep learning relevant models have shown great success at the ship tracking task [33,34]. It is not easy to apply the deep learning-based models in the realistic maritime ship tracking tasks, due to the following two disadvantages: (1) we need to obtain sufficient ship training samples for the purpose of pre-training the model; (2) we need to deploy a powerful ship-borne computer to implement the deep learning relevant models on a ship. In our previous study, we introduced multi-view learning and sparse representation models to exploit distinct visual features of the ship from maritime video clips, which were further tested under typical maritime traffic scenarios [35]. To mitigate anomaly position oscillations, a data quality control procedure was integrated to obtain high-resolution ship positions from maritime images [36]. The above-mentioned models experienced satisfied tracking results with a large computational cost; however, they may not satisfy real-time ship tracking task challenges under different visibility weather conditions in real-world applications. Overall, previous ship tracking relevant studies can be roughly classified into three types by the manner of data sources (see

Table 1. Overview of the previous work related to ship tracking.

Data Type	Data Source	Target	Result
AIS	Ship position	Ship collision avoidance	Ship trajectory adjustment
Radar	Ship echoes	Inshore ship accident avoidance	Ship maneuvering operation
Maritime video	Maritime images	Visual traffic surveillance	Early-warning maritime traffic situation

). More specifically, relevant studies of AIS data employ ship position-related data to fulfill the ship collision-avoidance task, and ship trajectory may be adjusted to avoid potential maritime traffic accidents. The radar relevant studies employ ship echoes to avoid potential ship accidents by performing ship maneuvering operations. Currently, more attention is being paid to extracting maritime traffic situations via the help of maritime surveillance images, and thus alerting the relevant officials of potentially dangerous traffic accidents in advance.

The newly emerging kernelized correlation filter (KCF) models show satisfactory performance in many general-purpose tracking tasks (e.g., pedestrian tracking, vehicle localization, etc.) [37,38]. The conventional KCF model implements the object tracking task by exploring distinctive object features learning from both positive and negative training samples. In that manner, the KCF model can be considered as a potential method for fulfilling the real-time ship tracking task. It is noted that an intrinsic disadvantage for the relevant KCF models is their sensitivity to ship scale-variation interference (i.e., the KCF model can only tackle the scale-invariant tracking task). Ship imaging scales demonstrate the relevant position between different neighboring ships, and thus various types of ship kinematic information (e.g., speed, moving direction, etc.) can be inferred from maritime image sequences in advance. However, the classic KCF model can be easily interfered by the target ship scale variation during the ship tracking procedure, which can trigger a tracking drift phenomenon (i.e., obtain an obvious biased ship position). Moreover, ship tracking under different weather conditions (e.g., good visibility, bad visibility, etc.) is an important but on-going task. To address the above-mentioned issues, we introduce log-polar transformation to mitigate scale variation interference for the conventional KCF model (i.e., obtaining the ship scale factor). More specifically, the ship scale variation in the log-polar coordinate system can be easily observed in the form of offset in polar and angular coordinates (similar to the x and y axis in the Cartesian coordinate system), and thus it is easy to determine the ship scale factor variation in each maritime image. Based on the above analysis, we propose a scale-adaptive kernelized correlation filter (SKCF) to robustly track ships in real time under varied ship scale-interfered traffic scenarios at different weather conditions. We demonstrate our ship tracker performance in comparison with another two popular KCF-derivative trackers. We have provided a brief flowchart for the proposed framework of the study in Figure 1.

The remainder of the paper is organized as follows: Section 2 describes the proposed ship tracking model in detail. Section 3 introduces the dataset, evaluation metrics and the experimental results. Section 4 briefly concludes the research.

2. Methodology

2.1. Framework Overview

The framework for the proposed SKCF model is shown in Figure 2, which consists of ship position determination and ship scale refinement. More specifically, the SKCF model employs a conventional KCF method to obtain the ship’s position in the raw image sequences by applying cosine windows to sample ship candidates on the Fourier-transformed ship images. Then, both the previous step output (i.e., raw ship tracking position) and ship training sample are transformed into log-polar coordinate systems. In that manner, the proposed tracker exploits the intrinsic ship contour feature with a histogram of oriented gradient (HOG) descriptor. Moreover, the ship scale factor is further identified by determining the offsets in both x and y directions in the log-polar coordinates. More specifically, the SKCF tracker identifies the ship tracking position and scale by finding the maximum response between the input training sample and KCF-obtained ship positions (i.e., potential candidates).

2.2. Ship Tracking with KCF

The KCF model employs the cross-correlation criterion to identify the maximum similarity between the ship training sample and the ship candidate image [37]. More specifically, the KCF tracker identifies the ship position which shows a maximum stimulus response between the training and target ship frame in the Fourier domain. The KCF tracking output provides the initial ship tracking position in the current ship frame, which serves as the input for the ship scale refinement.

The ship training sample in frame k is presented in the form of (${\mathrm{s}}_{\mathrm{k}}$, ${\mathrm{r}}_{\mathrm{k}}$), and the ship tracker f(s) is trained in an iterative manner. More specifically, the ship tracker f(s) is the optimal solution for the Equation (1), which is indeed a linear regression problem. Previous studies suggest that the optimal solution for Equation (1) can be obtained by obtaining minimal distance between the ship sample and the candidate, which is reformulated as Equation (2). The close form solution (see Equation (3)) for Equation (2) is considered as an optimal solution. Note that the input ship training sample is transformed into feature space φ(${\mathrm{s}}_{\mathrm{k}}$) with kernel matrix b. The minimal solution for Equation (2) is considered as a linear addition of the ship training sample, and the ship tracking result (i.e., maximum response) obtained by the KCF model is formulated as Equation (4). In that manner, we re-formulate the KCF tracking model in form of Equation (5), and the ship position can be obtained via a closed form solution, as shown in Equation (6):

$$\mathrm{f}\left(\mathrm{s}\right)=\langle \mathrm{v},\mathrm{s}\rangle + \mathrm{c}$$

(1)

$$\min {\sum }_{\mathrm{k}}\mathrm{L}{\left(\mathrm{f}\left({\mathrm{s}}_{\mathrm{k}}\right),{\mathrm{r}}_{\mathrm{k}}\right)}^2+{\mathsf{\lambda }||\mathrm{v}||}^2$$

(2)

$$s={\left({\mathrm{p}}^{\mathrm{T}}\mathrm{p}+\mathsf{\lambda }\mathrm{I}\right)}^{-1}{\mathrm{p}}^{\mathrm{T}}\mathrm{q}$$

(3)

$$\mathrm{s}={\sum }_{\mathrm{k}}{\mathsf{\beta }}_{\mathrm{k}}\mathsf{\phi }\left({\mathrm{s}}_{\mathrm{k}}\right)$$

(4)

$$\mathrm{f}\left(\mathrm{s}\right)={\sum }_{\mathrm{k}}{\mathsf{\beta }}_{\mathrm{k}}\mathrm{b}\left(\mathrm{s},{\mathrm{s}}_{\mathrm{k}}\right)$$

(5)

$$\mathsf{\beta }={\left(\mathrm{B}+\mathsf{\lambda }\mathrm{I}\right)}^{-1}\mathrm{r}$$

(6)

where the symbol $\langle ·,·\rangle$ is dot production and parameter s is the linear combination of the ship training sample. The $\mathsf{\lambda }$ is the regularization magnitude of the trained ship tracker, and symbol $\mathrm{L}{\left(\mathrm{f}\left({\mathrm{s}}_{\mathrm{k}}\right),{\mathrm{r}}_{\mathrm{k}}\right)}^2$ is a loss function. The parameter B represents the kernel matrices, each of which contains a group of kernels b, and symbol $\mathrm{I}$ is the identity matrix. The parameter s consists of a group of ${\mathrm{s}}_{\mathrm{k}}$, which is applicable to the parameter r. The vector $\mathsf{\beta }$ is the optimal solution for the ship tracking model.

We employ a cyclic-shift mechanism to generate more training samples for the purpose of enhancing KCF tracker generalization capability (see Figure 3). More specifically, the cyclic-shift mechanism is iteratively applied on the ship base sample (i.e., the ground truth (GT)) to obtain ship samples, which may contain background pixels. The basic permutation matrix for acquiring the cyclic-shift ship samples is denoted as a dimensional vector $\mathrm{c}=\left[{\mathrm{c}}_1,{\mathrm{c}}_2,\cdots ,{\mathrm{c}}_{\mathrm{m}}\right]$ (see Equation (7)). We can obtain a circulant kernel matrix $\mathrm{E}\left(\mathrm{c}\right)$ by cyclically sampling on the vector c, which can generate many ship base sample derivatives (see Equation (8)). We transform the circulant kernel matrix $\mathrm{E}\left(\mathrm{c}\right)$ into a diagonal matrix under the discrete Fourier transformation rule (see Equation (9)), where parameter F is the discrete Fourier transformation matrix. In that way, the ship training sample and labels are integrated into Equation (10). The symbol $\ast$ in Equation (10) is the complex-conjugate operator and $\odot$ is the element-wise product.

$${\mathrm{n}}_s=\left[\begin{array}{ccc}\begin{array}{cc}0& 0\\ 1& 0\\ 0& 1\end{array}& \begin{array}{c}\cdots \\ \cdots \\ \cdots \end{array}& \begin{array}{cc}0& 1\\ 0& 0\\ 0& 0\end{array}\\ \begin{array}{cc}⋮& ⋮\end{array}& \ddots & \begin{array}{cc}⋮& ⋮\end{array}\\ \begin{array}{cc}0& 0\end{array}& \cdots & \begin{array}{cc}1& 0\end{array}\end{array}\right]$$

(7)

$$\mathrm{E}\left(\mathrm{c}\right)=\left[\begin{array}{ccc}\begin{array}{cc}{\mathrm{c}}_1& {\mathrm{c}}_2\\ {\mathrm{c}}_{\mathrm{m}}& {\mathrm{c}}_1\\ {\mathrm{c}}_{\mathrm{m}-1}& {\mathrm{c}}_{\mathrm{m}}\end{array}& \begin{array}{c}{\mathrm{c}}_3\\ {\mathrm{c}}_2\\ {\mathrm{c}}_1\end{array}& \begin{array}{cc}\cdots & {\mathrm{c}}_{\mathrm{m}}\\ \cdots & {\mathrm{c}}_{\mathrm{m}-1}\\ \cdots & {\mathrm{c}}_{\mathrm{m}-2}\end{array}\\ \begin{array}{cc}⋮& ⋮\end{array}& ⋮& \begin{array}{cc}\ddots & ⋮\end{array}\\ \begin{array}{cc}{ \mathrm{c}}_2& { \mathrm{c}}_3\end{array}& { \mathrm{c}}_4& \begin{array}{cc}\cdots & {\mathrm{c}}_1\end{array}\end{array}\right]$$

(8)

$$\mathrm{E}\left(\mathrm{c}\right)=\mathrm{Fdiag}\left(\hat{\mathrm{c}}\right){\mathrm{F}}^{\mathrm{B}}$$

(9)

$$\hat{\mathrm{v}}=\frac{{\hat{\mathrm{s}}}^{\ast }\odot \hat{\mathrm{r}}}{{\hat{\mathrm{s}}}^{\ast }\odot \hat{\mathrm{s}}+\mathsf{\lambda }}$$

(10)

Note that obtaining the optimal solution for Equation (6) results in a spatial-temporal domain requiring large computational cost, which may fail to satisfy the real-time ship tracking task. Motivated by the success of our previous study [37], we transformed both ship training data (i.e., ship samples and candidates) into a frequency domain with the rule of the diagonalized Discrete Fourier Transform method. In that manner, the above-mentioned operators (e.g., multiplication, transposition, inversion) were applied in pixel-wise elements on each ship frame. For each ship frame, only half of the ship image would be searched for the purpose of finding the maximal response due to the advantage of information redundancy of the diagonalized Discrete Fourier Transform (i.e., with a reduction in computational cost over 50%).

By defining a compact vector Υ (see Equation (11)), we reformulated the optimal KCF tracker solution (i.e., Equation (6)) as Equation (12), with the aim to efficiently obtain ship tracking results by suppressing the image redundancy information. Thus, the KCF tracking model considers the ship candidate area in the input maritime image by finding the maximal stimulus response between the trained KCF model and the input ship image. More specifically, the input ship image response in Equation (2) with minimal value is considered as the KCF tracking result, which is shown in Equation (13):

$${\mathsf{{\rm Y}}}_m=\mathsf{{\rm Y}}\left(\mathrm{s},\mathrm{E}{\left(\mathrm{c}\right)}^k\right)$$

(11)

$$\mathsf{\beta }={\mathrm{F}}^{-1}\left(\frac{F}{F\left(E\right)+\lambda }\right)$$

(12)

$${\mathrm{r}}^{\prime }={\sum }_k{\mathsf{\beta }}_k\mathsf{\Gamma }\left(\mathrm{p},\mathrm{C}{\left(\mathrm{b}\right)}^m\right)$$

(13)

where ${\mathsf{{\rm Y}}}_m$ is a compact form of the kernel matrix $\mathrm{E}\left(\mathrm{c}\right)$. The division operator in Equation (12) is implemented in an element-wise manner, and the ${\mathrm{r}}^{\prime }$ is the tracking result for the current ship image.

2.3. Ship Scale Refinement

The conventional KCF model may not function properly when the ship scale varies in the maritime video clips. More specifically, the KCF model may experience severe tracking outliers when the target ship length and/or width changes in each maritime image sequence, which is named as the tracking shifting phenomenon [39]. To overcome this issue, we introduced log-polar transformation to suppress the scale-variation interference considering the advantages of scale and rotation invariant properties. The log-polar image geometry theory suggests that scale and rotation variation in the Cartesian coordinate corresponds to the offset in the x and y axes in the log-polar coordinate system. We defined the polar coordinates (ρ,θ) which corresponded to radial distance and angle from the origin. Given the ship image center point $\left({\mathrm{x}}_c,{\mathrm{y}}_c\right)$, we mapped each pixel (x,y) onto polar coordinates, which are shown as Equations (14) and (15). After that, the log-polar coordinates for the pixel (x,y) (from Cartesian coordinate) was reformulated as Equation (16). A mapping example for a ship image is shown in Figure 4.

Supposing the ship size in the current image is scaled with factor α, the ship coordinates (x,y) in the Cartesian domain will be transformed into $\left(\mathsf{\alpha }x,\mathsf{\alpha }y\right)$. In that manner, the corresponding coordinates in the log-polar domain for the ship coordinates were reformulated as Equation (17). We found that the scale variation in the Cartesian domain was transformed into a displacement from the polar (and angular) coordinates in the log-polar domain. For the purposes of a robust and efficient tracking performance, the log-polar transformed ship image was transformed into the Fourier domain. Note that the common operations (e.g., translation, rotation, reflection, scale) in the Cartesian coordinate system are mapped onto their counterpart in Fourier domain [40,41]. Then, the Hanning window series were applied on the ship log-polar image transformed into the Fourier domain. The scale factor was determined with the phase correlation filter by identifying the maximal response. The pseudo code for the proposed SKCF ship tracking model is shown in Algorithm 1.

$$\mathsf{\rho }=\sqrt{{\left(\mathrm{x}-{\mathrm{x}}_c\right)}^2-{\left(\mathrm{y}-{\mathrm{y}}_c\right)}^2}$$

(14)

$$\mathsf{\theta }=\arctan \left(\frac{\mathrm{y}-{\mathrm{y}}_c}{\mathrm{x}-{\mathrm{x}}_c}\right)$$

(15)

$$\left(\mathsf{\rho },\mathsf{\theta }\right)=\left(\log \left(\mathsf{\rho }\right),\mathsf{\theta }\right)$$

(16)

$$\left(\log \left(\mathsf{\alpha }\mathrm{x}\right),\log \left(\mathsf{\alpha }\mathrm{x}\right)\right)=\left(\log \left(\mathsf{\alpha }\right)+\log \left(\mathrm{x}\right),\log \left(\mathsf{\alpha }\right)+\log \left(\mathrm{y}\right)\right)$$

(17)

Algorithm 1 The proposed SKCF ship tracker

Input: Ship images and ship position in the first frame.

Output: Estimated ship tracking position in current frame;     ifthe initial ship framethen     1. Performs parameter initialization;     2. Extracts pre-trained ship patterns and labels;     3. Trains the ship tracker in Fourier domain;     else     4. Extracts ship features from the previous image;     5. Transforms the ship image into Fourier domain;     6. Obtains maximal response and obtains raw ship position;     7. Transforms the ship (i.e., raw tracking result) and sample into the log-polar coordinate system;     8. Obtains maximal response in Fourier domain;     9. Determines ship scale factor;     end

3. Experiments

For the purpose of performance verification, we implemented our proposed SKCF ship tracker on typical maritime video clips, which contained a ship imaging scale variation challenge. The conventional KCF tracker and scale-adaptive tracker with multiple features (abbreviated as SAMF) were both tested on the same ship videos for the tracking performance comparison [37,42]. The three ship trackers were implemented on Windows 10 OS with the Intel Core CPU at 3.50 GHz processor, and the RAM was 8 G. Moreover, the GPU memory was 2 G (with NVIDIA GeForce GTX 850 M version). The ship tracking experiments were implemented on Matlab (2016 version). We carefully fine-tuned the parameters in our study to obtain a satisfactory model performance—the crucial parameter settings were as follows. The padding parameter was set to 1.3, and default patch size was set to 40. We tested the model performance with different settings for the learning rate, and it was found that the model could obtain a satisfactory performance when the learning rate was set to 0.003. Thus, we set the learning rate into 0.0003 in our study when no further specification was given in the study.

3.1. Data

We shot three maritime video clips for the purpose of evaluating ship tracker performance, denoted as video #1, #2, and #3. Video #1 aimed to test ship tracker performance when the ship scale in the image sequences showed a decreasing trend (i.e., ship imaging size became smaller in the latter frames). Video #1 contained 910 frames of images with a frame rate of 30 fps (i.e., frame per second) and an image resolution of 1280 × 720. Video #2 was to test ship tracker performance when the target ship imaging size showed an increasing tendency. Moreover, the target ship in video #2 imposed the rotation challenge on the ship tracker. Video #2 included 700 frames taken at a frame rate of 30 fps. The image resolution for video #2 and #3 was same to that of video #1. We collected video #3 under poor visibility weather conditions for the purposes of testing the ship tracker performance under adverse weather conditions. Video #3 contained 600 frames, and both the frame rate and image resolution were same to the counterparts of video #1. More detailed information for the three maritime video clips are shown in

Table 2. Detailed information for the three ship video clips.

Video No.	FRAME RATE	Resolution	Total Frame Number	Tracking Challenge
Video #1	30 fps	1280 × 720	910 frames	Ship size decreases in the video
Video #2	30 fps	1280 × 720	700 frames	Ship size increases in the video along with rotation challenge
Video #3	30 fps	1280 × 720	600 frames	Ship size decreases in the video taken in mid-foggy conditions

, and the typical ship frames are shown in Figure 5. Note that the cameras were fixed onto on-shore buildings or ship rigid areas, and thus the issue of camera movement can be ignored in our study. We did not collect video clips under severe camera vibration, which was beyond our research focus. We will publish our collected maritime images in future, considering that a few maritime benchmarks containing ship scale variation challenges could be found.

3.2. Tracking Performance Measurements

To analyze ship tracker performance, we compared the distance between the tracked and ground truth ship positions. Note that the ground truth ship positions were manually labelled in a frame-by-frame manner. We measured the distance between the tracked and ground truth positions with the indicators of root mean square error (RMSE), mean absolute deviation (MAD), mean absolute percentage error (MAPE), and intersection over union (IOU) for the purpose of tracking model performance comparisons. Moreover, we employed the fps indicator to quantify computational cost. More specifically, we employed the center point to represent ship position in each maritime frame, which was further employed to calculate RMSE, MAD and MAPE. The IOU indicator represented the overlapping level between the tracked and ground truth ship position. Moreover, the computation cost was evaluated with the fps indicator. We considered that the ship tracker obtained real-time tracking performance when the fps was larger than 20 [43].

We evaluated varied tracker performances through the Euclidean distance between the center points between tracked and ground truth ship positions (i.e., each ship was presented by center point position). Given a maritime video clip, we employed the symbol G(x,y) to represent the ground truth ship position, and T(x,y) for the counterpart obtained by the ship tracker. The Euclidean distance between G(x,y) and T(x,y) was calculated by Equations (18)–(20). We can obtain the RMSE, MAD and MAPE distributions after obtaining average Euclidean distance (see Equations (21)–(24)). The smaller values for the above-mentioned four indicators showed higher tracking accuracy with less computation complexity, and vice versa. The IOU indicator (see Equation (25)) is a popular evaluation metric used in both object detection and tracking communities. According to the rule of thumb, we considered that the ship tracker obtained a satisfactory performance when the IOU score > 0.5. The tracked ship position was well matched with the ground truth position (i.e., manually labeled ship position in image sequences) when the IOU score was 1.

$${\mathrm{N}}_{\mathrm{n}}\left(\mathrm{x}\right)={\left({\mathrm{G}}_{\mathrm{n}}\left(\mathrm{x}\right)-{\mathrm{T}}_{\mathrm{n}}\left(\mathrm{x}\right)\right)}^2$$

(18)

$${\mathrm{N}}_{\mathrm{n}}\left(\mathrm{y}\right)={\left({\mathrm{G}}_{\mathrm{n}}\left(\mathrm{y}\right)-{\mathrm{T}}_{\mathrm{t}}\left(\mathrm{y}\right)\right)}^2$$

(19)

$${\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)=\sqrt{{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{x}\right)+{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{y}\right)}$$

(20)

$$\overline{\mathrm{N}}=\frac{{\sum }_{\mathrm{n}=1}^{\mathrm{k}}{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)}{\mathrm{k}}$$

(21)

$$\mathrm{RMSE}=\sqrt{\frac{{\sum }_{\mathrm{n}=1}^{\mathrm{k}}{\left|{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)-\overline{\mathrm{N}}\right|}^2}{\mathrm{k}}}$$

(22)

$$\mathrm{MAD}=\frac{{\sum }_{\mathrm{n}=1}^{\mathrm{k}}\left|{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)-\overline{\mathrm{N}}\right|}{\mathrm{k}}$$

(23)

$$\mathrm{MAPE}=\frac{1}{\mathrm{k}}{\sum }_{\mathrm{n}=1}^{\mathrm{k}}\left|\frac{{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)-\overline{\mathrm{N}}}{{\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)}\right|$$

(24)

$$\mathrm{IOU}=\frac{\mathrm{Area} \mathrm{of} \mathrm{Overlap}}{\mathrm{Area} \mathrm{of} \mathrm{Union}}$$

(25)

where ${\mathrm{N}}_n\left(\mathrm{x}\right)$ represents the displacement between ship position $\mathrm{G}\left(\mathrm{x},\mathrm{y}\right)$ and $\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)$ on the x-coordinate at frame $\mathrm{n}$. The total frame number in the ship video is k, and the corresponding x-coordinates for the two ship positions are denoted as ${\mathrm{G}}_{\mathrm{n}}\left(\mathrm{x}\right)$ and ${\mathrm{T}}_{\mathrm{n}}\left(\mathrm{x}\right)$, respectively. The rule is applicable to parameters ${\mathrm{N}}_{\mathrm{n}}\left(\mathrm{y}\right)$, ${\mathrm{G}}_{\mathrm{n}}\left(\mathrm{y}\right)$ and ${\mathrm{T}}_{\mathrm{n}}\left(\mathrm{y}\right)$. The Euclidean distance between the two ship positions is represented as ${\mathrm{N}}_{\mathrm{n}}\left(\mathrm{G}\left(\mathrm{x},\mathrm{y}\right),\mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\right)$, and $\overline{\mathrm{N}}$ is the average Euclidean distance.

3.3. Ship Tracking Results on Video #1

For the purpose of representing the ship tracker performance, we obtained the displacement distribution between the tracked and GT positions in a frame-by-frame manner, abbreviated as D2TGT. The D2TGT distribution demonstrated the ship scale variation tendency impact on the conventional KCF model performance (see Figure 6). More specifically, the KCF model tracking error showed an increasing tendency, as the ship scale became smaller in the latter tracking image sequences. The D2TGT distribution shows obvious tracking anomaly since frame #350. After carefully checking the ship tracking result (by plotting the tracked ship positions on the video #1), we found that the KCF bounding box (i.e., tracked position) was significantly larger than that of the ground truth. The tracking outlier confirmed that the KCF tracking model could not handle the ship scale variation challenge. Compared with the KCF tracking performance, the SAMF and SKCF models showed less tracking errors (i.e., both the trackers obtained better tracking accuracy) as the D2TGT curves showed smooth variation tendency.

We employed the RMSE, MAD, MAPE, IOU and fps to further compare the three trackers’ performance, which are shown in

Table 3. Statistical metrics for varied ship trackers for video #1.

Model	RMSE	MAD	MAPE	IOU	fps
KCF	3.47	2.64	0.89	0.46	101.9
SAMF	1.22	0.92	0.11	0.73	5.4
SKCF	1.19	0.91	0.08	0.81	95

. The Euclidean distance-relevant metrics (i.e., RMSE, MAPE and MAD) showed that our proposed SKCF tracked ship position was closer to the ground truth than the other two models. More specifically, the RMSE for the KCF tracking model was 3.47 pixels, which is 2.9 times larger than that of the SCKF model. Note that the MAD obtained by the KCF model was also 2.9 times larger than the counterpart of the SKCF model. Similarly, the MAPE values for both the KCF and SAMF trackers were significantly larger than that of our model. The RMSE, MAD and MAPE for the proposed SKCF model were 1.19, 0.91 and 0.08—smaller than those of the KCF and SAMF models. From the perspective of Euclidean distance measurement, we can conclude that the proposed SKCF model obtained a satisfactory performance in video #1.

The IOU indicator presented a matched level between the tracked and ground truth ship positions, which provided a holistic evaluation on the tracker performance. The KCF model obtained an IOU score of 0.46 (smaller than the threshold 0.5), and thus we considered that the KCF tracker failed to track the ship in video #1. Note that the IOU scores for both the SAMF and SKCF models were both larger than the threshold, which were 0.73 and 0.81, respectively. The IOU indicator suggests that our proposed ship tracker could obtain a better tracking accuracy. From the perspective of computational cost, both the KCF and SKCF models could obtain real-time tracking performance, and thereby the fps value for both trackers was larger than 90. The SAMF tracker obtained an fps of 5.4, indicating that the computational complexity was obvious larger than the two trackers. The main reason for this was that SAMF exploits several ship features (e.g., color, HOG, etc.) to fulfil the ship tracking task. Consequently, it is not easy to apply the SAMF tracker to implement resource-constrained maritime tasks (i.e., tracking ships with a ship-borne computer system). Considering the above analysis on the metrics about tracking accuracy and time cost, our proposed SKCF model obtained a better tracking performance than the other two models for video #1 (i.e., the SKCF obtained satisfactory ship tracking results in real-time).

For the purpose of visualizing the tracking performance of the three trackers, typical ship tracking frames are shown in Figure 7. Note that we have cropped the region of interest from the ship tracking image for the purposes of inspecting the ship tracking performance in detail. The three trackers’ performances were very close (i.e., well-matched with the ground truth position) at the beginning of the tracking procedure (see the upper-left subplot in Figure 7). However, the KCF-tracked bounding boxes (i.e., tracked ship positions) were obviously larger than those of the ground truth in the latter three frames (where the ship scales became significantly smaller than that of frame #200). The bounding box gaps for the KCF tracker shown in Figure 7 confirmed our above analysis. Both the SAMF and SKCF trackers can automatically adjust the ship scale during the ship tracking procedure, as shown in the tracking results of frame # 400, #700 and #900 in Figure 7. The ship tracking results on the bottom-right subplot in Figure 7 show that the SAMF tracked position was larger than the ground truth, while our proposed ship tracker was much closer to the ground truth bounding box. The main reason was that the SAMF tracker may have been interfered by the color feature (mainly from the ship–wake pixels). Based on the above quantitative and qualitative analysis, we can conclude that our proposed SKCF tracker obtained a satisfactory performance during the tracking challenge for video #1.

3.4. Ship Tracking Results on Videos #2 and #3

The three tracking models were implemented on videos #2 and #3 to further test the tracker’s robustness. We have not presented the typical frames with ship tracking results due to page limitations. The D2TGT curve distributions for videos #2 and #3 are shown in Figure 8 and Figure 9, respectively. The D2TGT curve distribution indicates that there was a negative correlation between the tracking performance of the KCF model and ship scale variation. More specifically, the KCF model obtained accurate ship tracking positions, as the Euclidean distance was smaller than 20 pixels (see the blue sample points from frame #1 to #500 in Figure 8, and frame #1 to #200 in Figure 9). The KCF Euclidean distance showed an obvious increasing variation tendency when the target ship decreased its imagine size (see the blue sample points from frame #500 to #700 in Figure 8, and frame #200 to #600 in Figure 9). In comparison to the KCF tracking results, the D2TGT curve distribution for SAMF was quite similar to that of the SKCF tracker. Overall, the Euclidian distance curves for the two trackers were very smooth, and only fluctuated at small magnitudes.

The statistical metrics shown in

Table 4. Statistical metrics for varied ship trackers on video #2.

Model	RMSE	MAD	MAPE	IOU	fps
KCF	12.85	9.71	2.73	0.61	258.6
SAMF	3.81	3.19	1.38	0.63	6.2
SKCF	2.66	2.08	0.73	0.63	103.3

and

Table 5. Statistical metrics for varied ship trackers on video #3.

Model	RMSE	MAD	MAPE	IOU	fps
KCF	15.69	13.15	1.33	0.61	163.6
SAMF	6.75	4.06	0.87	0.81	13.9
SKCF	4.76	2.79	1.01	0.84	92.1

further verify the three trackers’ performance in a quantitative manner. More specifically, the RMSE metric for the SKCF model was 2.66 (4.76) pixels in video #2 (#3), which is 4.83 (3.29) times to the counterpart of the KCF model of video #2 (#3). Meanwhile, the SAMF model obtained an RMSE of approximately 1.43 times greater than that of the SKCF model in both videos #2 and #3. The MAD indicator variation in Table 4 and Table 5 showed a similar tendency as that of RMSE. The minimal MAD values for videos #2 and #3 were 2.08 and 2.79 pixels, respectively, which were both obtained by the SKCF model. Moreover, the minimal MAPE values were also obtained by the proposed ship tracker, which are 0.73 and 1.01, respectively. From the perspective of IOU, the proposed SKCF model obtained a similar performance as that of the SAMF model in both videos #2 and #3, which were both higher than the KCF model counterparts. Note that the KCF model may have failed to exploit the ship scale variation tendency considering that the model focused on obtaining an accurate ship vertex position by exploring the spatial-temporal relationship between the ship candidate and template. In that way, the KCF distance was larger than the counterparts of the other two models. The fps statistical indicator suggested that only the KCF and SKCF models demonstrated the power of satisfying the real-time ship tracking task. Therefore, we can conclude that our proposed SKCF tracker successfully tracked ships in real time during the ship scale variation challenge of videos #2 and #3.

4. Conclusions

Visual ship tracking from maritime videos provides crucial microscopic traffic parameters for maritime situation awareness; this greatly benefits automatic navigation decision making in the smart ship era. We aimed to tackle the ship scale variation challenge to fulfill a holistic maritime situation exploitation task. More specifically, the proposed framework employed the KCF module to determine the initial ship position in the frame, and the ship scale factors were further refined with a log-polar-based kernelized correlation filter. We tested the proposed ship tracker performance under three video clips with typical maritime scenarios, and under different visibility conditions (i.e., sunny, fog). The statistics indicated that the proposed ship tracking model obtained a satisfactory performance during the ship scale variation challenge (in terms of robust tracking accuracy and low computational cost). More specifically, the average RMSE, MAD and MAPE were 2.87, 1.93 and 0.61, which indicated that the ship positions obtained by the proposed SKCF model were quite close to those of the ground truth data. In addition, the average IOU was 0.76 and the fps was 96.7, which confirmed our above analysis. The study can help on-duty ship officers in the ship bridge area obtain more informative kinematic traffic parameters in an easy yet convenient manner. Moreover, the study can further enhance maritime traffic safety by integrating with additional maritime traffic data sources (e.g., AIS). In this manner, more accurate ship maneuvering decisions can be made in advance, without requiring extra cost.

Though the proposed ship tracking model obtained a satisfactory performance under the scale variation challenge, we can still enhance the ship tracker performance in following directions. Firstly, we have already tested model performance on maritime traffic scenarios under good weather conditions without obvious sea clutter interference. We can further enhance our model performance by tackling challenges of adverse weather conditions (e.g., storm, thick fog), and thus promote our study to fulfil visual navigation tasks in the real-world smart ship era. Secondly, we have verified our model performance for tracking a single-ship target; verifying the tracker performance for a multiple-ship tracking task could be another interesting direction. Thirdly, we can integrate more efficient deep learning models to address the ship scale variation challenge in terms of ship type detection. Fourth, we can enhance the performance of the ship tracking model under various coupled tracking challenges (e.g., bad visibility conditions, multi-dimensional camera movements) by combining different techniques (e.g., radar, LRIT).