Integrating Shipping Domain Knowledge into Computer Vision Models for Maritime Transportation

Abstract

Maritime transportation plays a significant role in international trade and the global supply chain. To enhance maritime safety and reduce pollution to the marine environment, various regulations and conventions are proposed by international organizations. To ensure that shipping activities comply with the relevant regulations, more and more attention has been paid to maritime surveillance. Specifically, cameras have been widely equipped on the shore and drones to capture the videos of vessels. Then, computer vision (CV) methods are adopted to recognize the specific type of ships in the videos so as to identify illegal shipping activities. However, the complex marine environments may hinder the CV models from making accurate ship recognition. Therefore, this study proposes a novel approach of integrating the domain knowledge, such as the ship features and sailing speed, in CV for ship recognition of maritime transportation, which can better support maritime surveillance. We also give two specific examples to demonstrate the great potential of this method in future research on ship recognition.

For many years, the shipping industry has played an essential role in international trade and globalization [1] and maritime transport outperforms other transportation modes in terms of the amount of transported cargoes measured in ton-kilometers [2]. Additionally, containerized trade has increased by more than 42% between 2010 and 2017 [3]. Although maritime transportation is relatively safe, losses can be great when accidents occur [4]. Meanwhile, the shipping industry produces heavy greenhouse gas emissions and pollutants to the waters and the air [5]. In particular, maritime transportation accounts for 12% of global anthropogenic ${\mathrm{SO}}_{\mathrm{x}}$ emissions [6]. In addition, the ships burn bunker fuel, which has a much higher sulfur content than that of other transportation modes. To enhance maritime safety and environmental protection, numerous international maritime policies and laws are implemented by the International Maritime Organization (IMO) [7,8] and local governments [9]. Recently, a large number of articles have spared some efforts to optimize shipping activities considering the latest ship emission reduction policies [10,11]. However, some illegal shipping activities may increase the danger of maritime transportation and bring more damage to the environment. For example, illegal river boats sailing in the sea may raise the risk of shipping accidents, and illegal fishing and excavating/dumping of sand may harm the aquatic environment seriously. Therefore, to prevent illegal shipping activities, maritime surveillance has recently attracted tremendous attention.

With the popularization of artificial intelligence (AI) and Internet of Things (IoT), maritime transportation have witnessed and benefited from the rapid development of intelligent vision techniques in video surveillance [12,13]. For instance, at important inlets or estuaries, cameras are often set up to catch illegal riverboats entering the sea. Some cameras are also mounted on unmanned or commercial aircraft to record images of shipping vessels and are used for identifying illegal shipping practices, such as illegal fishing. One critical issue in maritime surveillance is the timely, automatic, and accurate recognition and monitoring of ships. To be specific, on-shore or on-airplane cameras can provide videos and capture the visual information of ships. Then, CV methods, such as mean shift [14], deformable part-based models (DPMs) [15], support vector machine (SVM) [16], and sparse representation [17], are proposed to recognize the specific type of the ships, called ship recognition in the following context for simplicity. Over the past several years, various advanced CV models have been developed to improve the accuracy of recognition results [18].

Nevertheless, the recognition results obtained from the CV models mentioned above may easily suffer from complicated environments, including water-surface light reflections, multiple moving ships, and severe weather conditions, e.g., hazy, low-light imaging, and rainy [18]. In addition, the training errors naturally and inevitably existing in CV models may exacerbate misjudgment. Therefore, totally relying on static training data and CV models may induce incorrect ship recognition results and harm the effectiveness of maritime surveillance. In fact, we should not treat data as a static artifact divorced from the process that produced it. In other words, simply improving the CV model itself cannot guarantee an accurate recognition of ships and the shipping domain knowledge, such as the ship features and sailing speed, should be considered as auxiliary support to aid in ship recognition decisions. To be specific, we present our proposed integrated decision framework of ship recognition in Figure 1.

We propose two potential examples of integrating the domain knowledge in CV for ship recognition of maritime transportation.

Example 1: Using ship length information to support the ship recognition.

The length of a ship is a significant feature in identifying different types of ships. Some open databases, such as the World Register of Ships (WRS) database (http://www.tokyo-mou.org (accessed on 1 December 2021)), have recorded the length of each registered ship. Based on the ship size in the video recorded by the camera, the actual physical length of the ship can be calculated. Then, we can compare the calculated ship length with the recorded length of the recognized ship type by CV model in the database to verify the recognition results, that is, if the calculated ship length is consistent with the length of the recognized ship type, the recognition result is correct, otherwise, there might be a misidentification. In addition, if the CV model fails to identify the specific ship type and provides multiple potential ship types, ship length can assist in determining the particular ship type with the most similar length. Therefore, the accuracy of ship recognition can be improved with the domain knowledge of ship length.

Example 2: Calculating the ship moving speed to judge the correctness of ship recognition.

Another promising domain knowledge that can be applied to support ship recognition is the sailing speed. To be specific, this can be adopted in the case when ship images captured by multiple cameras in different locations are recognized to be the same ship by the CV model. Then, we can calculate the moving speed of the ship by its captured time and distance, which can be obtained by the location of the cameras and the relative position of the ship image in the videos. If the calculated moving speed is obviously unreasonable, i.e., much greater than the theoretical sailing speed of the ship, there may be a misidentification of the CV model. In this way, the mistakes of ship recognition can be greatly reduced.

In conclusion, we believe that employing shipping domain knowledge and embedding it into the working process of CV for maritime transportation can contribute to enhancing maritime monitoring. Future studies on this topic are expected to have a significant impact on reducing illegal shipping activities and prompting safer and more sustainable shipping industry. To be specific, the researchers can conduct relevant studies by collecting a number of actual photographs and video data captured by cameras in the shipping industry and carrying out a series of case studies to further verify the feasibility of this novel approach. Based on in-depth research, a comprehensive ship recognition framework can be developed and applied in practice. Notably, this ship recognition framework requires the industry practitioners to have both a deep understanding of the shipping industry and proficient skills in CV. Current workers in the shipping industry may not be equipped with relevant capabilities, which may hinder the promotion of this new approach. Therefore, some efforts should be made in training industry practitioners or hiring relevant technical experts in order to better apply this novel ship recognition framework in maritime surveillance.