Visual Navigation Systems for Maritime Smart Ships: A Survey

Abstract

The rapid development of artificial intelligence has greatly ensured maritime safety and made outstanding contributions to the protection of the marine environment. However, improving maritime safety still faces many challenges. In this paper, the development background and industry needs of smart ships are first studied. Then, it analyzes the development of smart ships for navigation from various fields such as the technology industry and regulation. Then, the importance of navigation technology is analyzed, and the current status of key technologies of navigation systems is deeply analyzed. Meanwhile, this paper also focuses on single perception technology and integrated perception technology based on single perception technology. As the development of artificial intelligence means that intelligent shipping is inevitably the trend for future shipping, this paper analyzes the future development trend of smart ships and visual navigation systems, providing a clear perspective on the future direction of visual navigation technology for smart ships.

1. Introduction

Research on navigation safety has a long history. With the continuous development of science and technology, the design of safer and smarter navigation systems has become mainstream. Maritime transportation has always been considered a high-risk job, and as the size and number of ships continue to increase, factors affecting ship safety need to be given high priority. The government expects maritime security organizations to ensure the smooth running of maritime operations and at the same time handle maritime accidents efficiently in order to secure the country’s economic interests. The majority of maritime accidents during the period 2002–2016 were of a navigational nature, including groundings, collisions, close quarters and contact, with a rate of 52.8 percent [1]. Figure 1 shows the components of maritime accidents, and it is clear that the majority of accidents in 2014–2019 are caused by human factors [2]. On the other hand, maritime accidents not only bring about loss of economic benefits, but also lead to air and water pollution, which has a bad impact on the environment.

Shipping is one of the most internationalized and dangerous of all large industries in the world. The history of shipping has been one of increasing safety of navigation and efficiency of transportation. Human factors such as fatigue, work stress and environmental factors are contributing factors to shipping accidents [3]. The development of smart ships effectively replaces pilots to a certain extent. The advent of smart ships and drones will reduce the number of people at risk at sea, and even if autopilot sailing does not reduce the number of accidents, this means that the safety of maritime navigation will increase. It is expected that the number of accidents will decrease with the introduction of smart ships due to the high impact of the human factor. However, quantifying what percentage of accidents can be prevented by smart ships is difficult since accidents are usually not caused by pure human error. de Vos J et al. (2021) evaluated the percentage reduction in loss of life for different scenarios where autonomous shipping is applied, assuming that the types of events affected are only related to navigation. The percentage reductions for the scenarios were 47.4% for small cargo ships being unmanned, 69.5% for all cargo ships being unmanned and 100% for all ships being unmanned. As can be seen from the data, the number of maritime shipping accidents is likely to be reduced with unmanned ships, and the development of smart ships will greatly enhance the safety of maritime navigation [4]. In previous studies on the safety of ship navigation, researchers have tended to focus mainly on how to quantify maritime risks and to focus on the relationship between humans and maritime safety and on the impact of human situation awareness on ship safety. Most of the studies only mention the development related to smart ships and the improvement of autonomous collision avoidance systems. In this paper, the development of collision avoidance technology and the importance of improving situation awareness are also studied relevantly. However, unlike the traditional approach of previous studies, this paper outlines the IMO-related guidelines and industry specifications for smart ships, which provides a background for subsequent studies on smart ships. At the same time, this paper also outlines the development and progress of smart ships, not only studying the progress of collision avoidance technology, which has mainly appeared since the development of smart ships, but also focusing on different kinds of visual perception technology, integrating different technologies into one system. By studying the organic combination of visual perception technology and collision avoidance technology, we expect to further enhance the safety of maritime navigation. In addition, this paper does not discuss a single visual perception technology, but analyzes and integrates the main visual perception technologies, which effectively deepens the depth of the research on visual perception. Through computer assistance, a visual navigation system visualizes the decisions of a visual navigation unit in order to reduce the cognitive workload of the staff and further enhance the situation awareness of the pilots. With the development of a new generation of artificial intelligence technology, autonomous systems have been widely used in the fields of unmanned vehicles, underwater vehicles and unmanned aircraft. Various technologies and advanced research are applied in navigation systems, including perception technology, motion control technology, collision avoidance technology and communication technology. Through the research on these technologies, the application of visual perception navigation technology to intelligent ships can greatly reduce the possibility of maritime accidents caused by human factors.

This article analyzes the current opportunities and challenges of navigation technology for smart ships, providing the following contributions:

• A summary of the IMO guidelines and industry codes for smart ships;
• A review of the development of navigation technology and an analysis of its advancement;
• A characterization of the combination and application of different visual navigation techniques;
• An overview of the current status and future development of visual navigation technology and smart ships.

2. Smart Ships and Navigation Technology

2.1. Smart Ships

With the development and application of smart ships, the operational pressure on staff has been reduced, the safety of maritime navigation has been greatly improved and pollution of the oceans has been drastically reduced. Section 2.1 describes and analyzes the IMO’s guidance on smart ships and the industry requirements for smart ships, as well as the progress and development of smart ships.

2.1.1. IMO Guidance and Industry Requirements for Smart Ships

The concept of an smart ship, first introduced in the 19th century, refers to a ship that is capable of carrying out a series of well-defined operations without or with a small number of crew members supervising it [5]. In order to give sustainable impetus to the development of the shipping industry in Europe, Norway was the first to develop technology for smart ships. There are three categories of more leading smart ship projects that examine the feasibility of developing viable business models in the short to medium term, reducing accident fatalities and analyzing the feasibility of smart ship scenarios [6]. Smart ships have been designed and developed for a number of reasons: to provide a better working environment for crews; to mitigate possible future crew shortages; to reduce risks in transportation; to be environmentally friendly; and to provide greater safety of navigation. The introduction of smart ships will bring about a new shift in cost efficiency, accident prevention and human resources. While smart ships are expected to be safer than conventional ships, the emergence of new technologies also brings new risks, such as the emergence of cybersecurity threats and misinformation. It is therefore important to define standards for shore-based operators of smart ships and provide relevant training. With regard to legal issues, the difference between the time it takes for technology to mature and the time it takes for regulations to be developed and put into practice may have a negative impact on the adoption of advanced technologies, and it is therefore important to recalibrate the relevant regulatory approaches. Smart ships may also raise ethical issues in emergency situations, such as problems of untimely human–machine and machine-to-machine communication, as well as when searching for and rescuing ships, crews or passengers in urgent need of help. A more comprehensive, international and harmonized approach to the development of a new regulatory framework for smart ships is important before they become fully operational. By understanding the impacts on different sectors and their associated linkages, it will be possible to prevent maritime accidents, protect the environment and maximize commercial benefits in the context of ongoing developments [7].

The International Maritime Organization (IMO) is primarily concerned with the development of international standards for the enhancement of safety and security and the protection of the marine environment [8]. The IMO therefore has a responsibility to consider the safety of international shipping for smart ships. Since the Maritime Safety Committee (MSC) placed the issue of smart ships on its agenda in January 2017, the MSC has been working to guide countries in the development and testing of autonomous navigation technologies. The development and application of smart ship technologies from 2017 to 2020 has been driven by the MSC to achieve real progress. “Autonomous” ships are not all similar, and the International Maritime Organization has classified MASSs into four categories [9], including ships with a crew and some automated systems (Class 1), ships with a crew that can be remotely controlled (Class 2), remotely controlled ships without a crew (Class 3), and ships capable of autonomous navigation without a crew and without human remote control (Class 4). The current standard for navigation is Class 1 vessels, whereas Class 2 vessels are supervised by on-board personnel who are ready to take over the operation of the system, so it is Class 3 and 4 vessels that could potentially pose a threat to the safety of shipping. The IMO, through its survey, and in order to safeguard the safe and environmentally friendly navigation of smart ships, decided in 2017 to carry out a Regulatory Scoping Exercise (RSE) aimed at assessing the extent to which the existing regulatory framework has been impacted in order to identify problems and gaps [10]. Four IMO committees—the Maritime Safety Committee (MSC), the Legal Committee (LEG), the Facilitation Committee (FAL) and the Marine Environment Protection Committee—are involved in this task, which affects many areas. At the same time, the introduction of technologically advanced Information and Communications Technology (ICT) is expected to lead to steady growth of the maritime industry [7]. The MSC has taken note of the test standards for real ship experiments on smart ship development submitted by relevant countries, provided advice on the development of standards, and contributed to the safety of smart ships and the prevention of pollution of the environment. The MSC held its 101st session in 2019, and approved interim guidelines on MASS testing, which include guidelines related to the safety, security and protection of the marine environment and list the mandatory instruments of compliance. These guidelines also cover the staffing and qualifications of participants in MASS experiments, human factors and infrastructure to ensure the safe conduction of experiments, management of cyber risks, etc.

2.1.2. Progress and Development of Smart Ships

Autonomous driving has long been utilized in many areas, and the question of autonomous driving in the shipping industry is no longer if it will happen, but when it will happen. The degree of autonomy has been categorized into three phases: In the first phase of autonomy, the crew, supported by the system, makes decisions from information or data collected during the operation of the ship. In the second stage, under the supervision and authorization of the crew, the ship can make decisions and initiate actions. And in the last stage, a fully smart ship can be realized [11]. Smart ships also have an impact on many aspects. The global maritime industry is characterized by rapid scientific and technological developments, and while manned and smart ships have some aspects in common, regulatory provisions should also take into account the characteristics of smart ships and develop specific regulatory policies. In addition, the most important thing to consider for smart ships is safety, and a range of technologies should be researched to control autonomous functions in order to improve safety. In terms of the industry, which has previously relied on the technology of the crew, autonomous technology will now re-invent the field of shipping by way of smart ships. More ship design and port facilities need to be considered to make smart ships efficient and reliable [7].

Intelligent ship autonomous technology is the integration of a variety of technologies, including autonomous navigation technology (navigation situational awareness, behavioral decision making and motion control), intelligent cabin operation and maintenance, communication technology, intelligent hulls, integrated experiments, etc. [12]. Intellectualization, greening and automation will become mainstream in global ship development. With the continuous improvement of ship intelligence, the development of unmanned cabin maintenance, assisted piloting technology and self-diagnosis of faults will all gradually reduce the labor demand of ships. In the future, the number of crew members with rich experience is likely to decrease, which will increase our concern for ship safety. The accuracy of driving-behavior decision making is directly related to the safety of water transportation, so the research on related decision-making algorithms is of great significance. The long-term navigation phase of smart ships requires comprehensive consideration of theoretical technology levels, unmanned ship navigation rules, human–machine cooperation, etc. Therefore, the study of human–machine interaction in smart ships is also a key area for the development of smart ships in the future. Human beings have advantages in understanding complex situations and reasoning, while machines have the advantages of high computational power and reliability in specific environments. Promoting the study of human–machine cooperation theories and technologies for ships can give full play to the advantages of human–machine cooperation and promote the prosperous development of smart ships. One aspect of the digital transformation of the shipping industry is the increasing digitization of shipboard systems, which is achieved by combining information technology with operating systems, and this high degree of interconnectivity and interdependence increases the likelihood of cyberattacks and cyber risks to a ship’s digital infrastructure. Therefore, although the risk of cyberattacks can hardly be reduced to zero, it is necessary to study the related cyber decision making and the design of security architectures for ships. The most common propulsion systems used in modern ships are two-stroke diesel engines, steam and gas turbines, or combined marine propulsion systems, and these require the use of different fuels, lubricants, and so on. These complex systems require adequate personnel on board to monitor and maintain them. And as ships move towards autonomy, the number of crew members will be reduced and agencies will need to consider specialized training and education for personnel involved in special functions to further develop smart ships.

Smart ships have a positive impact in terms of capital savings, lower fuel costs, energy savings, improved safety at sea and environmental protection. However, there are challenges in terms of port operations, port capacity and insurance. Putting smart ships into practical use reduces human error, which in turn reduces the costs associated with accidents and subsequent insurance and optimizes operations and processes [13]. Increased cargo handling capacity and up to a 90% reduction in labor costs will result as smart ships achieve targeted efficiencies without the need for personnel [14]. Figure 2 shows the overall percentages for the annual cost of cargo transportation. Smart ships can also increase productivity and reduce fuel consumption, with each smart ship saving more than USD 7 million over 25 years in fuel consumption, crew wages and supplies [15]. In addition, the use of smart ships can ensure more effective surveillance activities and facilitate the integration and visualization of global trade supply chains. Through more advanced and efficient automated energy management systems and improved navigation systems, smart ships can also increase the effectiveness and efficiency of other shipping activities, such as loading and unloading and lock access.

The development of smart ships cannot be separated from the progress of technology, and the development of smart ship technology is a gradual evolutionary process. Perception technology is the basis for the development of smart ships, and it has undergone an evolution from single sensors to multi-sensor fusion. Initially, perception technology relies on basic sonar systems, which can only provide limited environmental information. As technology progressed, perception technology began to integrate multiple sensors, such as radar, infrared and vision systems, to collect richer data. These data include not only static obstacle information, but also dynamic currents and weather conditions. With the continuous optimization of algorithms, perception technology has become more accurate and efficient, capable of processing a large amount of data in real time to provide strong support for ship navigation and decision making. The development of communication technology is closely linked to the progress of perception technology. Initially, ship communication mainly relied on radio waves, and although this communication method can realize basic information transmission, it is limited by its signal coverage and transmission speed. With the emergence of satellite communication and broadband network technology, the communication capability of ships has been greatly enhanced. Modern smart ships are able to transmit a large amount of information in real time through high-speed data links, including the ship’s position, status and environmental data, thus realizing more effective ship management and maritime traffic coordination. Motion control technology is the core of the autonomous navigation capability of smart ships. In the early days, the motion control of ships mainly relied on manual operations, and the crew needed to adjust a ship’s heading and speed based on experience. With the development of automation technology, motion control systems began to integrate advanced algorithms and sensor data to realize automatic navigation and path planning. The motion control system of modern smart ships can automatically optimize routes, reduce fuel consumption, and maintain the stability and safety of ships under complex sea conditions. Collision avoidance technology is the key to ensuring the safe operation of smart ships. With a deeper understanding of ship dynamics and the marine environment, collision avoidance technology has evolved from relying on manual judgment and operations to advanced systems that can automatically detect potential collision risks and take avoidance measures. The development of these technologies has significantly improved the safety of ships in complex sea conditions. With the development of single perception technologies, smart ships are beginning to explore multi-sensor integrated perception technologies. Such fusion technologies are able to integrate data from different sensors to provide a more comprehensive and accurate perception of the environment. The combined application of technologies such as AIS, radar, infrared and vision systems can provide more comprehensive environmental sensing capabilities. This integrated perception not only improves the accuracy of the perception, but also enhances the robustness of a system, which maintains an effective perception capability even when some sensors fail. Through the gradual development and mutual integration of these technological fields, smart ships are moving towards a higher level of automation and intelligence and the innovation and development of the maritime transportation industry continues to be promoted.

2.2. Navigation Technology

Navigation technology is the top priority of the whole smart ship system, and appropriate navigation technology can improve the safety of navigation, reduce operational costs and improve the efficiency of navigation at the same time. Determining collision avoidance strategies and paths by sensing and learning sea information in intelligent ships during navigation is a problem that needs to be solved through further research. Effective collision avoidance and safe navigation of smart ships on the sea rely heavily on efficient and useful smart navigation systems. Therefore, the research on smart ship navigation technology in Section 2.2 is mainly categorized into perception technology, communication technology, motion control technology and collision avoidance technology. Their relationships can be represented in Figure 3:

2.2.1. Perception Technology

The sensing technology of ship navigation needs to use radar, LIDAR, ship automatic identification system (AIS), probe, camera and other sensors to collect environmental data when ships are sailing and build intelligent algorithms such as perception enhancement, data fusion, target classification, decision recommendation, etc., to process and analyze the sensed data automatically, distinguish potential dangers and abnormalities, and at the same time formulate contingency measures, such as adjusting the ship’s sailing routes and speed, so as to improve the safety of navigation [16]. For smart ships, smart perception technology may have been developed to a more advanced stage. Ship identification, static obstacle perception, visibility impact, speed perception, distance perception, observation angle and cost are all issues that still need to be addressed [12]. Currently, radar is commonly used to detect targets, but the echo of radar cannot feedback the shape and appearance of the target, while the long distance and the motion of the ship itself will affect ship perception, so Liu et al. (2019) proposed a novel ship recognition and tracking system based on a deep learning framework, using a deep residual network and cross-layer jump connection strategy to extract advanced ship features to improve the recognition classification accuracy [17]. Leng et al. (2019) proposed a new method for ship detection based on single-channel synthetic aperture radar (SAR) image complex signal klick (CSK), which first detects the potential location of a ship based on the area proposal and then obtains the final ship target based on the target identification, which is free from the influence of false alarms caused by radio-frequency interferences (RFIs) and avoids the leakage of the detection situation under dense targets [18]. During navigation, the position of the ship to be sensed is often uncertain, and the problem of recognizing multiple types of ships is also faced, all of which undoubtedly add to the difficulty of perception. Wang et al. (2022) proposed a real-time ship collision risk perception model based on two domain proximity parameters, DDV (Degree of Domain Violation) and TDV (Time to Domain Violation), which adopts the inverse computation method and incorporates the information of the home route in the parameter computation and effectively solves the problem caused by the uncertainty of ship position prediction [19]. In order to achieve the detection and recognition of multiple types of ships, Zhou et al. (2021) improved the YOLOv5s algorithm by optimizing the loss function, regrouping the K-mean of the target initial frame at the data input side and enlarging the sensory field of view at the output frame, which greatly improved the precision and accuracy of ship image detection [20]. The variability of the maritime environment poses many challenges for the development of perception techniques; tracking a moving vessel in the context of moving ocean dynamics is one of them, and researchers overcame the challenge of tracking and recognizing moving vessels at sea by using background subtraction with real-time approximation of curve evolution based on level sets to delineate the contours of a moving vessel in the ocean [21].

At present, the research on navigation systems is still inseparable from environment perception in rain, snow, haze, low-light and other bad weather conditions; it is difficult for navigation systems to realize fast and accurate environment perception and meet the navigation and safety requirements of ships [22]. To ensure that ship detection remains accurate in low visibility, Guo et al. (2022) proposed a low-visibility enhancement network (called LVENet) based on the Retinex theory combined with depth-separable convolution and further proposed synthetic degraded image generation and hybrid loss functions to enhance visibility under low-light imaging conditions [23]. Liu et al. (2023) proposed an integrated low-visibility enhancement network (called AiOENet) that uses a unified encoder–decoder network architecture to flexibly recover low-visibility images, reducing the impact of harsh conditions on navigational safety [24]. Wang et al. (2023) constructed a foggy navigation database using a differentiated deep learning architecture and an EfficientNet neural network and a new method for foggy images and perceived visibility, which by replacing the SE module and combining the convergent block-attention module and the focal point loss function identified the best enhancement algorithm depending on the fog [25]. Li et al. (2024) proposed an inshore ship real-time detection transformer (ISRt-detr) that combines multiple coordinated attention and contrast learning to enhance the ability to perceive ships at night [26]. Detection of small targets at sea is an important area of ship navigation perception. AIS and radar systems may have the problem of missed detection; therefore, Wang et al. (2024) introduced an SSIM-based region of interest search method and an ellipse-weighted fusion method, proposed the ship panoramic vision stitching algorithm SSIM-EW, and proposed a perceptual model (YOLOv8-SGW) for sea-surface target detection, which improves the intelligence level of ships [16]. Yan et al. (2022) established relevant datasets of different complex sea states and introduced the improved Faster R-CNN for target detection in small sample datasets, which solved the problem of multiple complex sea-state data being difficult to collect simultaneously [27]. Traditional multi-feature detectors can only handle three or fewer features. Xu et al. (2020) proposed a multi-feature detector based on the Isolated Forest (iForest) algorithm in order to detect small floating targets in sea clutter, which breaks through the limitation of the number of features and reduces the cost of computation [28].

The flow of ship perception is shown in Figure 4:

2.2.2. Communication Technology

Maritime radio first appeared on board ships at the beginning of the 20th century and in the early days was mainly used for transmitting and receiving telegrams from passengers. Subsequently, the International Convention for the Safety of Life at Sea (the SOLAS Convention) was introduced, and maritime radio communications were developed in the fields of both applied technology and communication procedures, providing an effective technical means for the exchange of information and communication between ship and shore and between ships. In the 1970s, telex, telephone, and facsimile communications were gradually applied to ship communications. Narrowband direct printing telegraphy (NBDP) and radiotelephone (RT) technology were applied to terrestrial communications systems in the 1970s and 1980s. Satellite communication systems have also been used occasionally. By the end of the 1980s, satellite systems became more widely used and had a larger share of applications in ship-to-shore communications [12]. Amendments to the GMDSS on radio-communications were adopted in 1998 and fully implemented in 1992 to improve the safety of ships in the field of radio-communications [29]. At present, the information exchange between ship and shore includes not only the ship’s report and some instruction telegrams, but also image, voice and various other data about the ship. The information is digitized and transmitted, laying a certain foundation for the development of intelligent navigation in smart ships. Interference in communication networks between ships may lead to data distortion. Xie et al. (2019) proposed a reliability modeling method for ship communication networks based on the Apriori algorithm, constructed a transmission link model for ship communication networks and adjusted the reliability of the forwarding nodes of the links in ship communication networks through a fuzzy PID neural network control model, which improved the accuracy of data transmission and greatly improved the quality of communication [30]. Hoole et al. (2013) used the parameters of a communication system for ship navigation to a predetermined port terminal, utilizing electromagnetic beams to keep the ship within a marked trajectory boundary line, improving the accuracy of ship navigation [31]. Bin et al. (2024) proposed the Stackelberg Q-learning-based multi-hop cooperative routing algorithm (SQMCR) that balances packet forwarding benefits and energy costs. They also formulated collaborative communication strategies to ensure the reliability and efficiency of communication and further improve the performance of the routing algorithm [32].

The communication architecture of a smart ship is shown in Figure 5:

2.2.3. Motion Control Technology

Uncertainties in ship dynamics, random environmental perturbations and inaccuracies in measurement information are the main problems facing ship motion control [33]. The types of ship motion control are shown in Figure 6. The last three categories concern the control of heading, path, speed and stability, with the focus on which control strategy to choose. Different types of ship motion control correspond to different control algorithms. The basic algorithms used for ship heading control are proportional–integral–differential (PID) control [34], fuzzy logic control [35], predictive control [36], sliding mode control [37], autoresistant control (ADRC) [38] and artificial intelligence (AI) [39] algorithms, among others. On this basis, other algorithms are used to improve the effectiveness of heading control, such as optimizing PID controllers using Particle Swarm Optimization (PSO) [40], applying Genetic Algorithms (GA) to the optimization of the parameters of fuzzy controllers, and so on. For the study of ship stability control, the basic algorithms used are PID control [41], fuzzy control [42], predictive control [43], sliding mode control [44], ADRC [45], etc. Optimization algorithms are combined with basic algorithms on this basis, such as the combination of predictive algorithms with fuzzy control [46] and PID algorithms [47], to ensure the stability of ship navigation. For path tracking, the basic algorithms used include PID control [48], fuzzy logic control [49], predictive control [50], sliding mode control [51], ADRC [52], etc. The basic algorithms can be used in combination, with adaptivity being progressively improved as artificial intelligence is applied to path tracking. For ship trajectory tracking, the basic algorithms used include PID control [53], ADRC [54] and artificial intelligence algorithms [55]. Optimization algorithms, such as fuzzy logic control combined with PID control [56], PSO combined with ADRC [57], and neural networks combined with predictive control [58], are combined with basic algorithms, which can be used to achieve trajectory tracking and ship collision avoidance in different situations.

The structure of ship motion control is shown in Figure 7:

2.2.4. Collision Avoidance Technology

Early on, since humans were at the center of collision prevention, researchers developed navigational aids to enhance the situation awareness of human operators. More recently, with the development of autonomous driving, the focus of research has become solving collisions through machines [59]. The techniques involved in collision avoidance are called collision prevention techniques and involve three basic processes, namely, motion prediction, conflict detection and conflict resolution.

Motion prediction is the basic module of collision avoidance for ships and involves the process of predicting the trajectories of operating systems and obstacles, which involves the estimation of the future state of a ship’s motion. This is usually performed based on the ship’s current position, speed, heading and possible influence of external factors, such as wind currents. A five- to ten-second prediction of the ship’s motion can give an operator enough time to avoid a serious collision. Ding et al. (2022) proposed a new algorithm for ship motion prediction based on the modified covariance (MCOV) method and a neural network, which firstly utilizes the MCOV method for spectral analysis of a ship’s motion and then utilizes the main spectra of the ship’s motion to build a ship motion model through the neural network (NN), which advances the ship motion prediction time to ensure the consistency of the accuracy [60]. Path planning for smart ships has been a difficult problem in research. He et al. (2021) proposed an easy-to-implement multi-ship-encounter collision-avoidance path-planning method for smart ships, which combines a ship motion model and a PID controller to predict a ship’s automatic heading change process, providing an effective method for collision avoidance [61]. Kim et al. (2022) proposed an algorithm that can predict a ship’s state and plan the optimal route based on the uncertainty of the number of ships, using the Untraceable Kalman Filter (UKF) to derive the Time to the Closest Point of Approach (TCPA) and the Distance to the Closest Point of Approach (DCPA) from the geometric data of the home ship and the target ship [62]. To reduce the limitations associated with the low accuracy and high complexity of previous methods, Abebe et al. (2022) investigated a hybrid Autoregressive Integrated Moving Average (ARIMA)–Long Short-Term Memory (LSTM) model to accurately estimate near-term trajectories and assess collision risk [63].

Conflict detection refers to whether and when a ship should take evasive action and is based on motion prediction to determine whether two or more ships are at risk of collision in the future. Conflict detection in practice consists of three steps: recognizing a potential collision and alerting the operator; triggering the ship’s self-help system to find ways to avoid the collision; and evaluating the risks posed by alternative solutions or avoidance actions [59]. The first two focus on risk-informed scenarios, and the last step focuses on decision making in response to risk. Ship collision risk modeling and probabilistic and collision risk assessment can all provide effective strategies to reduce the risk of ship collisions. Liu et al. (2021) developed a dynamic ship domain model to propose a systematic approach to detect possible collision scenarios and identify the distribution of collision risk hotspots in a given area, providing support for identifying areas of high collision risk and the distribution of risk in time and space [64]. Szlapczynski et al. (2021) proposed a new ship collision risk model utilizing the ship domain concept and domain-based collision risk parameters for collision risk assessment using many auxiliary parameters derived from the ship domain concept, such as Time to Domain Violation (TDV), Time to Domain Exit (TDE), etc. [65]. Liu et al. (2022) further modeled and visualized ship collision prediction analysis by introducing the Quadratic Ship Domain (QSD) into the Vessel Conflict Ranking Operator (VCRO), visualizing risk using the Kernel Density Estimation (KDE) model and finally predicting future risk using the Convolutional Long and Short-Term Memory Network (ConvLSTM) model [66]. In terms of probability and collision risk estimation, many scholars have utilized AIS data for analysis. Liu et al. (2020) established a unified collision risk assessment framework based on an autoencoder (AE) to clean up historical ship movement data and estimate the frequency of ship collisions [67]. Xin et al. (2021), on the other hand, quantified the uncertainty distribution of trajectories using AIS data, and a two-stage Monte Carlo simulation algorithm was used to ensure the accuracy of the estimates [68]. Shi et al. (2022) established a framework for collision risk assessment at sea considering trajectory-to-trajectory collision based on a multi-factor Douglas–Peucker algorithm adapted to ship trajectory compression considering speed and turning constraints [69].

At the heart of ship collision avoidance is conflict resolution, which is the process of taking appropriate measures to avoid a collision after the risk of collision has been detected. Many technologies for ship collision avoidance have emerged today, which can be broadly categorized into six main groups. The first one is rule-based collision avoidance. Perera et al. (2011) formulated and implemented a new fuzzy inference system based on an if–then-rule-based decision-making process and integration using the MATLAB software platform; this fuzzy logic-based intelligent decision-making system greatly improves the safety of ships through collision avoidance [70]. The second one is to determine a ship’s motion using a virtual vector field. Park et al. (2021) used a Bayesian theory-based Relevance Vector Machine (RVM) to estimate a ship’s collision risk, and it improves the accuracy and efficiency of ship parameter prediction [71]. The third one is to discretize the possible scenarios and thus select the optimal one. Chen et al. (2020) proposed an improved Time-Discretized Non-Linear Velocity Obstacle (TD-NLVO) algorithm and determined hazardous scenarios based on predefined criteria, which improves the accuracy of analyzing the geometrical probability of the risk of collision of ships [72]. Tengesdal et al. (2021), on the other hand, introduced a Cross-Entropy (CE)-based estimation of ship collision probability, which is able to obtain a low variance estimate of the probability of small collisions [73]. The fourth is to represent collisions as constraints, and Cheng et al. (2024) used the Velocity Obstacle (VO) algorithm to support collision avoidance with a trajectory non-linearly predictable target vessel and reflect the effect of the remaining collision time on the outcome by being sensitive to the effect of the remaining collision time, which can provide early warning of dangerous encounters and buy more valuable time for the ship to perform collision avoidance maneuvers [74]. In order to minimize collision avoidance operation times, Li et al. (2019) proposed a multi-vessel rolling optimization method for predicting and calculating the collision risk between vessels [75]. The fifth is the replanning method, which converts the ship collision avoidance problem into a path planning problem. Zhu et al. (2023) proposed the improved route-plan-guided Artificial Potential Field (APF) method, the Fast Local Path Planning (FLPP) method and a dynamic goal-guided (APF) method, which provide technical support for ships’ autonomous navigation and collision avoidance [76]. Ren et al. (2021) proposed an autonomous collision avoidance algorithm for ships based on the improved speed barrier method that includes a path replanning algorithm by fusing the dynamics model of unmanned ships, the motion model of the encountering ship and the international maritime collision avoidance rules [77]. The last one consists of hybrid algorithms, such as the hybrid algorithm for smart ship path planning, which improves the typical Artificial Potential Field (APF) algorithm and the Velocity Obstacle (VO) method while combining them with the international rules for collision avoidance at sea [78], and the intelligent hybrid collision avoidance algorithm based on deep reinforcement learning, which improves the original sampling mechanism of DDPG, among others [79].

A comparison of the collision avoidance decision processes for manned ships and smart ships is shown in Figure 8:

3. Visual Perception Navigation Technology

Visual aid navigation is an important function of autonomous navigation in smart ships. In order to reduce the workload of smart ship pilots, a visual assistance system visualizes and processes the decisions of intelligent navigation devices for computer-aided visual navigation, which improves the situation awareness of smart ship pilots. The navigation information of the existing visual navigation system mainly comes from the traditional equipment on the ship’s bridge, such as AISs, radar, infrared sensors, electronic nautical charts, etc. Through augmented reality technology, the above navigation information will be matched to the video imagery of the ship’s video camera so as to realize the overlap of the virtual information and the real imagery and adjust the routes and sailing speeds according to the current situation of the ship, as well as make avoidance decisions so as to improve the safety of the ship [80]. In a navigation system, it is crucial to accurately detect the navigation information of the target ship. The environmental perception of ship navigation also means using AIS, radar, depth sounder, camera and other sensors to collect environmental data for ship navigation and constructing algorithms, such as data integration and perception enhancement algorithms, to automatically process and analyze perceived data, identify potential dangers, implement collision avoidance measures and safeguard the safety of navigation. The visual perception technology of ships cannot be separated from the environmental data perceived by the sensors; however, the use of any one of these methods for perception alone may have the problems of inconsistent information and insufficient data accuracy. At this stage, in addition to single-sensor perception, the main source of environmental data is multi-sensor data fusion. Generally, when designing a multi-sensor data fusion architecture, centralized fusion, distributed fusion and hybrid fusion are involved. We first analyze the technologies related to single perception, followed by a further study of integrated perception based on single perception.

3.1. Single Perception

3.1.1. AISs

AIS is the acronym for a universal shipborne automatic identification system; it is a kind of maritime mobile-band VHF broadcasting system, which is mainly used in the field of ship navigation and monitoring, realizing the data exchange between ship and ship and between ship and shore through the VHF radio band. The core function of an AIS system is to send and receive the dynamic and static information about a ship. Dynamic information includes the ship’s position, speed, heading, destination and expected arrival time, while static information involves the ship’s name, call sign, type, size, deadweight tonnage and other basic attributes [81]. The real-time updating and exchange of this information enables the ship to understand the surrounding environment and the dynamics of other ships in a timely manner so as to make more accurate navigational decisions. AISs track ship movements through the electronic exchange of navigational data between ships via shipboard transceivers and terrestrial or satellite base stations and uses the information collected for maritime security [82]. In order to detect multi-vessel encounters, Zhu et al. (2022) proposed a model to mine multi-vessel encounter situation awareness, analyze the spatial process of encounters and make collision avoidance decisions from automatic identification system (AIS) data, which overcomes the problem that the traditional collision risk assessment methods are only applicable to the differences between two ships and ship awareness [83]. Meanwhile, Rong et al. (2022) proposed a new method based on an improved sliding window algorithm to automatically identify ship collision avoidance behavior from ship trajectories, which can be used to accurately detect ship collision avoidance behavior from AIS trajectory data [84]. With continuous research and discoveries, researchers started to analyze AIS data using visual analysis tools. Öztürk et al. (2021) proposed a tangible visual analysis tool to analyze the changes in maritime traffic using the spatio-temporal basis of AISs—a novel way to advance the development of judging the safety of navigation of vessels and waters [85]. He et al. (2021) focused more on the visual model and human–computer interaction and designed and applied a visual analytics (VA) method called AIS Data Quality Visualization (ADQvis) to assess and explore the data quality of AISs [86]. Navigation technologies for smart ships utilize collected information to generate computer images, but these images also need to be converted into usable navigation data through human interpretation, increasing the likelihood of human error. To address this problem, Carter et al. (2023) designed an AIS-based visually enhanced simulation system for ship navigation that superimposes computer-generated navigation information onto real-world scenarios in real time, improving the safety and usability of the navigation system [87].

3.1.2. Radar

Radar, radio detection and ranging technology comprise an electronic system that uses electromagnetic waves for target detection and localization. It realizes precise measurement of the position, speed and shape of a target by transmitting electromagnetic waves and receiving reflected signals. The working principle of radar is based on the propagation characteristics of electromagnetic waves, and target information is obtained by analyzing the time delay, frequency change and phase difference of the echo signal. Radar technology is also being developed to be gradually applied in visual perception navigation systems. Xia et al. (2022) innovatively proposed a visual transformer framework based on contextual joint representation learning, referred to as CRTransSar, for the synthetic aperture radar (SAR) target detection task, combining the global contextual information perception capability of a transformer with the local feature representation capability of a convolutional neural network (CNN), which improves detection accuracy [88]. Chen et al. (2023) proposed a novel robust CFAR procedure for background clutter fitting, which provides a new interdisciplinary perspective for SAR image segmentation and achieves excellent detection performance [89]. Mou et al. (2019) introduced a maritime target detection method based on improved Faster R-CNN for navigation radar PPI (Plane Position Indicator) images, which improved the target detection performance of radar [90]. In addition to ordinary sea navigation, ice navigation is likewise an important research problem in path planning. Hsieh et al. (2021) used nautical radar imaging to reconstruct ice navigation scenarios, establish a sea-ice warning visualization function and select the optimal path planning scheme [91]. Naus et al. (2021), on the other hand, evaluated the accuracy of ship positioning based on the observation of navigational radar echoes from electronic nautical charts, providing the prerequisites for an autonomous navigation system that replaces the Global Navigation Satellite System (GNSS) [92]. Chen et al. (2022) proposed a maritime target detection method based on Marine-Faster R-CNN algorithm in complex environments, which utilizes convolutional neural networks (CNNs) for feature extraction and target identification of PPI images generated by radar echoes with enhanced generalization capability [93]. In order to solve the problems of the low target resolution of traditional radar and insufficient intuition of ship navigation, Li et al. (2022) proposed a multi-camera ship “video radar” enhanced navigation system and a deep learning-based feature point detection method (SRSuperpoint) to improve the navigation interface stitching effect and present a more comprehensive and intuitive navigational environment [94]. Xu et al. (2023) used radar sequence images to estimate ship dynamic features, which provides a reliable method for ship collision risk assessment in ship traffic service centers and improves operator interactivity [95].

3.1.3. Infrared

Infrared technology (IRT), or IRT for short, is an advanced navigation aid technology that detects and recognizes objects based on their thermal radiation properties. The technology uses infrared sensors to capture the thermal radiation signals emitted by the target object, and by analyzing the wavelength, intensity and distribution of these signals, it realizes the precise measurement of the target’s position, speed and shape. Its high-precision detection capability enables ships to navigate the sea more safely and efficiently. Infrared technology is applied in the form of infrared thermometers, infrared spectrometers, infrared communication devices, etc., and infrared cameras are major tools in the application of infrared technology for sensing and thus assisting navigation systems, so we mainly analyze the research on the technology related to infrared cameras. Zhou et al. (2021) proposed a robust foreground detection method based on background modeling combined with multiple features in the Fourier domain (BMMFF) for ship target detection in dynamic sea-surface backgrounds, which outperforms the relevant comparable algorithms in different challenging sea-surface scenes [96]. When the weather at sea is clear, the maritime target is located exactly between the sunlight and the infrared camera, and the gray value of the background of the infrared image is larger than that of the target. Dong et al. (2019) designed an infrared maritime target detection algorithm based on the Visual Attention Model (VAM), which improves the accuracy and efficiency of the search for maritime targets in backlit environments [97]. In order to improve the target tracking efficiency of ships, Liu et al. (2022) proposed an efficient method of SiamRPN++ based on cross-connected and spatially transformed networks of AlexNet, which effectively improved the method to 63.9 FPS [98]. Infrared thermal imaging technology has been widely used in the field of target detection. It can not only adapt to different light-intensity environments; at the same time, it has the characteristics of strong concealment, strong detection ability, long detection distance and high detection sensitivity. Cao et al. (2023) proposed an improved Canny segmentation algorithm based on the maximum interclass variance method on the basis of infrared thermal imaging, which stabilized the flow field of background light to achieve the effective segmentation of a ship’s image and obtained a high-definition operational target [99]. In addition to the above advantages, infrared technology also has an inherent advantage in small-target detection. Gao et al. (2023) took the lightweight model of infrared small-ship detection as the research object, constructed an infrared small-ship dataset and preprocessed the images using gamma transformation, and the model parameters were reduced by 83% compared with the YOLOv5m model [100]. To compensate for the limitations of radar and AISs, such as blind spots and limited detection ranges, Park et al. (2024) utilized an infrared radiometric camera to supplement conventional perception sensors which is capable of accurately detecting obstacles at sea and estimating their dynamic motions based on enhanced tracking results [101].

3.1.4. Visual Technology

Visual Technology, as a high-tech means of simulating human visual perception, gives machines the ability to “see” through image capture, processing and analysis. This technology captures images of the surrounding environment through high-resolution cameras and utilizes advanced image processing and pattern recognition algorithms to provide real-time visual information to ships. It is capable of recognizing channel markers, obstacles and other vessels, thus assisting navigation decisions. The application of vision technology not only enhances the ability of ships to navigate in complex sea conditions, but also provides additional security for maritime safety. Li et al. (2019) designed an overlapping-sector laser-propagation navigation system that utilizes red and green lasers to form a sector-shaped safety zone for safe navigation, which is easy to operate and highly sensitive [102]. Pan et al. (2020) proposed a fine-grained classification RMA (ResNet–Multi-Scale–Attention) model based on deep learning for analyzing subtle and local differences between different types of beacons for beacon recognition [103]. Electro-optical (EO) sensors such as video cameras are used to complement marine radar for accurate detection of objects on the sea surface, and Shao et al. (2022) proposed an enhanced convolutional neural network, called VarifocalNet*, that improves the detection of objects in harsh marine environments [104]. There is still a lack of low-cost and reliable sensing devices. Bi et al. (2023) designed a vision-based method to recognize ships and their microscopic features for navigational planning of ship collision avoidance, which can effectively and efficiently identify the navigation signals of the target ship [105]. There are many challenges in maritime navigation, such as narrow visibility for ship navigators, the limited view length angle of a single camera, complex maritime environments, etc. Wang et al. (2024) proposed panoramic visual perception-assisted navigation technology, introduced an SSIM-based region-of-interest search method and an ellipsoid-weighted fusion method, and put forward the ship panoramic visual stitching algorithm SSIM-EW. This technology obviously improves the ability to detect small targets at sea, which can expand the mariner’s field of view, identify the targets missed by AIS and radar systems, guarantee navigation safety, and improve the level of ship intelligence [16].

3.2. Integrated Perception

With the continuous development and advancement of technology, integrated perception technology plays an increasingly important role in the field of visual perception. It realizes comprehensive understanding and accurate mapping of complex environments by integrating data from different sensors and utilizing image processing and machine learning algorithms. The core advantage of integrated perception is its ability to overcome the limitations of a single perception method and significantly improve the accuracy and reliability of target identification through the synergistic effect of multi-source data. For the fusion of AIS and vision technologies, AIS data can provide the vessel of interest with a unique Maritime Mobile Service Identity (MMSI), position coordinates, ground heading and ground speed, while cameras can directly display the visual appearance of the vessel. Qu et al. (2023) fused AIS and vision data to improve maritime traffic surveillance and were able to obtain more accurate ship tracking results and motion characteristics, taking full advantage of multiple sources of data [106]. To address the lack of detailed information important for real-world applications in deep learning object detection methods, Gülsoylu et al. (2024) developed a technique for fusing automatic identification system (AIS) data with ships detected in images to create a dataset enriched with ship images [107]. Ding et al. (2024) also combined AIS data with visual data and showed that this technique outperformed the pure AIS data technique during the daytime and outperformed the pure visual data technique at night, providing more reliable data for collision avoidance [108].

For the fusion of visual data and radar, both electronic charts and radar are essential equipment in a ship’s navigation system, and the fusion of the two ensures that the ship is able to clearly display the dangers on the sea surface. Guo et al. (2020) proposed a deep learning-based data fusion algorithm which extracts robust features from radar images and merges marine radar and electronic chart data to provide more comprehensive information [109]. Zhang et al. (2021) proposed a radar image denoising algorithm based on the concept of the Generative Adversarial Network (GAN) with Wasserstein distance to reduce the noise interference brought by external factors and hardware and utilize the sparse theory to parallelize the high-frequency and low-frequency sub-band coefficients of detected images obtained by Fast Fourier Transformation to realize the fusion of images [110]. For the combination of visual and infrared data, Gao et al. (2022) used refined feature fusion and Sobel loss for the fusion of marine infrared and visible images [111], while Jeon et al. (2023) introduced an integrated approach to visual and infrared detection and ranging, combining deep learning-based computer vision techniques with real-time physics-based EO/IR data processing algorithms [112], all of which enhance the ability of ships to navigate autonomously in a variety of marine environments.

For the fusion of radar and AIS systems, in the maritime environment, radar signals are often blocked by islands and large ships, resulting in degradation of radar target detection and performance. Sun et al. (2023) utilized a priori environmental knowledge from electronic nautical charts (ENCs) and information from automatic identification systems (AISs) to propose an AIS-Assisted Integrated Probabilistic Data Association (A-IPDA) algorithm, which is well able to counteract long-time target occlusion in complex environments at sea and maintain tracking continuity and tracking accuracy [113]. To address the fact that false echoes in radar images and packet loss in AIS data transmission can lead to uncertainty in ship speed and heading estimation, Xu et al. (2023) proposed a ship speed and heading estimation method utilizing the fusion of radar sequential images and AIS data to quantitatively characterize the state of ship navigation [114]. In addition to the two-by-two combination of sensing technologies, some scholars have also studied the combination of multiple sensing data to enhance the navigation safety of smart ships. The use of any one ship motion sensing method alone may have problems such as inconsistent information and inaccurate data. Therefore, Wu et al. (2022) proposed a multi-sensor integrated perception system for motion detection of ships and constructed a multi-sensor integrated ship motion perception system hardware platform consisting of radar, AIS, camera and other attachments, which significantly improved the consistency of the information and the accuracy of the data for monitoring ship motion [115].

4. Future Trends

4.1. Trends in Intelligence and Automation

The relevant data for our study can be found in [4,14,16,115] as well as Rolls-Royce data, which provide us with a solid foundation for the study of smart ship navigation technology. In particular, the Rolls-Royce data provide valuable references for our research by providing information about the composition of ship cargo costs, which provides us with the advantages brought about by the development of smart ships. Through these data, we were able to gain an in-depth understanding of the pulse of the development of smart ship navigation technologies, as well as the performance and potential of each technology in practical applications. The accumulation of these research results adds persuasive power to our thesis and provides clear guidelines for our future research direction. By comprehensively reviewing the development history of smart ships, industry guidelines and current navigation technologies, this paper provides a clear perspective on the future direction of visual navigation technology for smart ships. Future ship visual navigation systems will realize the integrated processing of AIS, radar, infrared and visual data through the fusion of multi-modal perception technologies, thus improving the robustness and adaptability of the systems. The development of smart ships will benefit from the convergence of cross-domain technologies, especially the Internet of Things, big data analytics and cloud computing. The integration of these technologies will provide smart ships with powerful data processing and analysis capabilities, enabling real-time remote monitoring and decision support.

With the rapid development of artificial intelligence technology, the intelligent transformation of the future shipping industry has become an irreversible trend. The core of smart ship technology lies in the intelligence and automation of the ship’s visual navigation system. The future direction of this system will rely on the in-depth application of deep learning and machine learning algorithms to achieve accurate identification and analysis of dynamic changes in the marine environment. Through these advanced algorithms, the system will be able to provide comprehensive information support to the crew, enhance their understanding of the surrounding environment and improve their ability to anticipate potential risks, thereby significantly improving their situation awareness.

4.2. Future Challenges and Opportunities for Smart Ship Visual Navigation Systems

Smart ships are becoming more autonomous, but the challenges that come with this cannot be ignored. Human–machine collaboration still plays a crucial role in ensuring safe ship operation. Future smart ship designs will need to focus more on optimizing human–machine interaction, providing intuitive user interfaces and decision support tools that will enable crew members to effectively supervise and control ship automation systems. In addition, with the development of technology, how to ensure data security and privacy protection against cyberattacks is also a major challenge that smart ship visual navigation systems need to face. The core of ship visual navigation is security and risk management. With the advancement of data analysis technology, intelligent systems will be able to assess navigation risks more effectively, issue timely warnings to crews and help them take preventive measures, thus improving the safety of navigation. However, this also requires intelligent systems to have a high degree of reliability and stability, and any technical failure or miscalculation may lead to serious consequences. In future developments, the field of ship navigation will focus more on environmental adaptability and sustainability, adopting more environmentally friendly technologies, optimizing energy consumption and reducing the impact on the marine environment. At the same time, the application fields of ship visual navigation technology will continue to expand, extending from the traditional shipping field to a wider range of fields, such as marine scientific research, marine operations and environmental monitoring, providing more diversified technical support for the development and protection of marine resources and promoting the sustainable development of the marine economy.

The future of visual navigation systems for smart ships at sea is full of challenges and opportunities. Continuing advances in technology will drive smart ships towards higher levels of autonomy and safety. However, this will also bring new requirements, and the smart ship industry needs to work closely with governments, research institutes and international organizations to address the challenges and ensure the healthy development and widespread application of smart ship technology [116]. This includes the development of new regulations and standards to accommodate the special needs of smart ships, as well as the development of new education and training programs to ensure that crews are able to adapt to future technological changes. It is only through these combined efforts that smart ship visual navigation systems can realize their full potential and revolutionize the shipping industry.

5. Conclusions

As the scale of maritime transportation continues to expand, maritime safety issues are becoming more and more prominent, and the research and development of visual navigation systems for smart ships as a key technology to enhance navigation safety is of great significance. By comprehensively reviewing the development history of smart ships, the industry guidelines of the International Maritime Organization (IMO) and current navigation technologies, this paper provides a clear perspective on the future direction of the development of visual navigation technology for smart ships.

This paper first summarizes the IMO guidelines and industry codes for smart ships and reviews the development of smart ships. The emergence of smart ships has not only reduced the risk of maritime navigation; it has also reduced the impact on the environment and improved operational efficiency. Through the integrated application of smart ship technologies, the autonomy of ships has been significantly improved and operators’ situation awareness has been enhanced. Then, this paper discusses the latest progress in the key technology areas of perception technology, communication technology, motion control technology and collision avoidance technology, and analyzes the combination and application characteristics of different visual navigation technologies. Together, these technologies contribute to the performance enhancement of visual navigation systems for smart ships by improving ships’ environmental sensing capabilities, communication efficiency and motion control accuracy and the effectiveness of collision avoidance strategies. This paper also emphasizes the importance of integrated perception techniques. By integrating data from different sensors, such as AIS, radar, infrared and vision systems, it is possible to provide a more comprehensive and accurate understanding of the environment, significantly improving the accuracy and reliability of target recognition. Finally, this paper looks forward to the future development trend of visual navigation systems for smart ships. With the integration of technologies such as artificial intelligence, the Internet of Things, big data analysis and cloud computing, future smart ships will realize higher levels of autonomy and safety. At the same time, human–machine collaboration will continue to be a key factor in ensuring the safe operation of ships, and the design of future smart ships will pay more attention to the optimization of human–machine interaction.

The research on visual navigation technology for smart ships is not only of great significance for enhancing maritime navigation safety, but also has a profound impact in terms of promoting the sustainable development of the marine economy. Future research should continue to focus on technological innovation, cross-domain technology integration, optimization of human–machine collaboration and sustainability.