Contingency Planning of Visual Contamination for Wheeled Mobile Robots with Chameleon-Inspired Visual System

Abstract

To enable mobile robots to effectively deal with the emergency of visual contamination, contingency planning based on case-based reasoning (CBR) was performed in this paper. First, for a wheeled mobile robot (WMR) equipped with a chameleon-inspired visual system, a target search model in chameleon-inspired binocular negative correlation movement (CIBNCM) mode was established. Second, a CBR-based contingency planning model of visual contamination for WMRs was established, where the reasoning process using CBR for visual contamination was analyzed in detail. Third, through the analysis of environment perception when visual contamination occurs, a perception model in chameleon-inspired visual contamination for WMRs was built. Finally, to validate the proposed approach, a contingency planning experiment scheme for visual contamination was designed based on the robot’s general planning of target tracking, and the experimental result is discussed. The proposed CBR-based contingency planning approach for visual contamination can reason out effective solutions corresponding to the contamination situations. The rationality of the approach was verified by experiments with satisfactory results. Moreover, compared with the contingency planning method based on rule-based reasoning, the accuracy of target retracking after the robot visual system is contaminated is significantly higher for the CBR-based contingent planning method used in this paper.

1. Introduction

Visual sensors are an extremely important tool for robot environmental perception, with the characteristics of rich sensory information, etc. It is an essential requirement of mobile robots equipped with visual systems for many practical application fields, ranging from space exploration to engineering rescue [1]. Figure 1a shows the “X-RHex” six-legged robot [2] working in a muddy environment, Figure 1b shows the “Red Snake” [3,4] snake-like robot climbing in the pipeline, and the Hull BUG wheeled robot shown in Figure 1c was developed by the U.S. navy, which is mainly used for Hull cleaning [5]. Due to the complex and unknown working environments of these mobile robots, their visual systems are vulnerable to contamination, lowering the quality of images and affecting regular work. Once this emergency occurs, it is difficult for humans to clean it, and the general way is to terminate the task and command the robot to return. Moreover, the return task is also difficult to complete due to the contaminated visual system. Consequently, it is important to address unexpected events through contingency planning for the robot’s visual contamination.

Currently, the measures for visual contamination mainly depend on manual cleaning, automated cleaning systems, and contaminated image processing algorithms. The manual cleaning approach requires the robot to return to the workstation or stop and wait. This can result in wasted energy, and cleaning itself will interfere with the normal use of the equipment. It is not suitable for mobile robots. Automatic cleaning systems are often used in smart cars. An automatic cleaning mechanism is designed on the outside of the camera to restore the normal function of the sensor system. Most studies use supervised sensors to collect data to determine whether the camera has been contaminated [6]. Once the sensor is found to be contaminated, a cleaning mechanism such as a water spray pipe or blower is immediately used to clean the camera [7,8] and a heater is used to dry the remaining water droplets on the camera [9]. In addition, there is visual contamination caused by blood, debris, and other factors in medical testing. Automatic cleaning systems are common in the field of medical robots such as laparoscopic photography [10,11,12]. There are currently many cleaning methods that use the principle of ultrasound [13]. By developing specialized chips and glass [14,15], microvibration is used to quickly detect and remove dirt, ice, and water. The main focus of contaminated image processing algorithms is in the areas of image fog removal and image rain removal. In 2010, He [16] proposed a dark channel prior method for image defogging, which can effectively remove fog from images. In 2012, L.W. Kang [17] first proposed the issue of image rain removal at the IEEE conference, which led to the development of these two image processing algorithms. With the continuous development of deep learning [18,19], it has become increasingly widely used in the field of automation [20,21], and image processing-related tasks have also begun to apply deep learning technology. By utilizing machine learning [22,23] and generative adversarial networks [24,25], traditional dark channel prior methods have been improved to achieve end-to-end image defogging. The research on rain removal algorithms focuses more on improving various neural networks [26,27,28,29], ultimately achieving effective end-to-end image rain removal. In summary, the automatic cleaning system can quickly clean the camera after detecting pollution but requires a real-time control system and precise mechanical structure design. Similarly, current pollution image processing algorithms mainly focus on fog and rain, while robots face complex pollutants when working outdoors, and rely solely on image restoration cannot solve the pollution problem. The application of these two methods in the field of outdoor mobile robots with complex environments still needs improvement.

Intelligent decision control is the development trend of automation systems [30,31,32] and solving visual contamination can be transformed into decision-planning problems. Continuity planning refers to monitoring the status of autonomous robots during operation. When an unexpected event is detected, the robot autonomously generates a set of targeted response strategies based on the current emergency, maintaining the ability to continuously move with degraded functionality. NASA has conducted extensive research on emergency planning for Mars rovers. Dearden [33] proposed an incremental emergency planning method to address the needs of Mars rovers for time and resource continuity. A seed plan was constructed and emergency branches were gradually added. NASA has also conducted research on personnel decision-making and scheduling issues after emergencies and unexpected events such as suborbital re-entry of Mars probes [34,35]. In the field of intelligent vehicles, Salvado [36] proposed an emergency maneuver planner for the failure of other decision processes. This planner can use heuristic best-first search algorithms to reorganize discrete motion primitives for different scenarios, resulting in vehicle behavior that is suitable for the current situation. In robotics, Shah [37] proposed a lattice-based 5D trajectory planner. This planner can estimate collision risk and infer emergency maneuvering behavior and can integrate the collision prevention behavior of unmanned boats into the search for dynamic feasible paths. The emergency path planner proposed by Hardy [38] can generate multiple paths simultaneously to cope with dynamic obstacles. Underwater robots have high costs and a high risk of loss. Currently, they only perform conservative and simple operations and cannot fully realize their potential. Harris [39] suggested that robot uncertainty could be modeled to observe the progress of their work and the surrounding environment. If a low battery level is observed, another data collection task is scheduled, and if the battery level is sufficient, the primary task is performed.

Visual contamination occurs with the characteristics of dynamics, uncertainty, etc. In the process of robot environmental perception, conventional planning cannot properly address these problems [40,41]. In terms of contingency planning for visual contamination, robots need to automatically determine whether contamination occurs and self-learn to deduce coping strategies that are highly adaptive to the current situation [42]. Case-based reasoning (CBR) can meet the above requirements well.

CBR is used to solve new problems by searching for similar historical cases and using the existing experience or results [43,44]. It is a knowledge-based problem-solving method in the field of artificial intelligence (AI) that originated in Roger Schank’s description in Dynamic Memory. This method derives from humans’ cognitive mental activities, alleviating the bottleneck of knowledge acquisition in conventional knowledge systems. It combines quantitative analysis with qualitative analysis and has the characteristics of a dynamic knowledge base and incremental learning [45]. In this way, not only can it liberate human thought from conventional thinking, but it can also generate relatively thorough strategies by considering varieties of similar cases, making up for limited human thought. The basic steps of CBR can be summarized as case representation, case retrieval, case reuse, case revision, and case retention, known as 5R [46].

Consequently, CBR-based contingency planning was employed for the visual contamination of mobile robots in this paper to enable the robot to cope with this emergency. First, for a WMR equipped with the chameleon-inspired visual system, a target search model in CIBNCM mode was established. Second, a CBR-based contingency planning model for the visual contamination of mobile robots was established, and the process of using CBR theory for the contingency planning of visual contamination was analyzed in detail. Third, an environmental perception model with visual contamination for mobile robots was built to analyze how the robot perceives the environment with visual contamination, including the visual contamination state space and the corresponding action space. Finally, experiments were implemented to verify the effectiveness of the proposed CBR-based contingency planning strategy for visual contamination of WMRs.

2. Description of the Chameleon-Inspired Visual System for WMRs

The chameleon is called “the lion crawling on the ground” because its visual system plays an important role in defense and hunting. Its two eyes can achieve independent movement, and they are not coordinated. This kind of phenomenon is very rare in animals, as shown in Figure 2. Ott, Avni, et al. named the global scanning strategy of chameleons searching for prey in environments “negative correlation” scanning rules [47,48,49,50].

In this paper, based on the analysis of the movement mechanism of chameleon visual systems, as well as the similarity between the visual systems of WMRs and chameleons, a chameleon-inspired visual system equipped on a wheeled mobile robot was established, as shown in Figure 3a. The visual system consisted of two cameras (avenir-SSL06036M made by Seiko Japan) and two CMOS image sensors (Suntime-200 made by Sun Time Taiwan), as well as five servos to control the PTZ rotation in 5DOFs, including the horizontal rotation of the neck, the horizontal rotation, and the pitch of two cameras.

Based on the built visual system, the coordinate systems were established as shown in Figure 3b. All the coordinate systems accorded with the right-hand rule, and the corresponding parameters are shown in

Table 1. Parameters of the coordinate systems.

Parameter	Description	Value
$k_1$	Longitudinal distance between $G_b$ and $R$	165 mm
$k_2$	Vertical distance between $G_b$ and $R$	180 mm
$k_3$	Horizontal distance between $G_b$ and $G_l$	90 mm
$k_4$	Vertical distance between $G_l$ and $C_l$	79 mm
${\varphi }_b$	Rotation angle of neck around $Y$ axis of $G_b$	${\varphi }_b\in \left[-90°,90°\right]$
${\phi }_l,{\phi }_r$	Rotation angle of cameras around $Y$ axis of $G_l,G_r$	${\phi }_l\in \left[-45°,90°\right], {\phi }_r\in \left[-90°,45°\right]$
${\theta }_l,{\theta }_r$	Rotation angle of cameras around $X$ axis of $C_l,C_r$	${\theta }_l,{\theta }_r\in \left[-45°,45°\right]$
${\Phi }_X{,\Phi }_Y{,\Phi }_Z$	Rotation angle of robot around $X,Y,Z$ axis of $W$	$n/a$

where $W,R$—World and robot-body coordinate system, both at the center of the robot differential mechanism; $G_b$—Neck coordinate system, at the rotation center of the PTZ neck; $G_l,G_r$—PTZ coordinate system, at the rotation center of each PTZ; $C_l,C_r$—Camera coordinate system, at the optical center of each camera; $X_l,X_r$—Imaging coordinate system of each camera; $I_l,I_r$—Image coordinate system of each camera.

.

According to the imaging coordinate system, assuming that the cameras are distortion-free, the intrinsic parameter model of the camera is as follows:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{k}_x& 0& u_0\\ 0& k_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c/z_c\\ y_c/z_c\\ 1\end{array}\right],$$

(1)

where $(u,v)$ is the target coordinate in the image coordinate system; $(u_0,v_0)$ is the coordinate in the image coordinate system where the optical axis passes through the imaging plane; $(x_c,y_c,z_c)$ is the target coordinate in the camera coordinate system; and $k_x,k_y$ are the amplification coefficients in the $x,y$ directions.

The external parameter model can be described as follows:

$$\left[\begin{array}{c}{x}_{cl}\\ y_{cl}\\ z_{cl}\\ 1\end{array}\right]={}_R{}^{W}T\cdot {}_{C_l}{}^{R}T\cdot \left[\begin{array}{c}{x}_w\\ y_w\\ z_w\\ 1\end{array}\right],$$

(2)

where $(x_w,y_w,z_w)$ is the coordinate of the object in the world coordinate system; ${}_R{}^{W}T$ is the rigid body transformation from the robot-body coordinate system to the world coordinate system; and ${}_{C_l}{}^{R}T$ is the rigid body transformation from the camera coordinate system to the robot-body coordinate system.

$${}_R{}^{W}T=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos {\Phi }_X& -\sin {\Phi }_X\\ 0& \sin {\Phi }_X& \cos {\Phi }_X\end{array}\right]\left[\begin{array}{ccc}\cos {\Phi }_Y& 0& \sin {\Phi }_Y\\ 0& 1& 0\\ -\sin {\Phi }_Y& 0& \cos {\Phi }_Y\end{array}\right]\left[\begin{array}{ccc}\cos {\Phi }_Z& -\sin {\Phi }_Z& 0\\ \sin {\Phi }_Z& \cos {\Phi }_Z& 0\\ 0& 0& 1\end{array}\right]$$

(3)

$${}_{C_l}{}^{R}T=\left[\begin{array}{cccc}\cos {\varphi }_b& 0& \sin {\varphi }_b& 0\\ 0& 1& 0& -k_2\\ -\sin {\varphi }_b& 0& \cos {\varphi }_b& -k_1\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}\cos {\phi }_l& 0& \sin {\phi }_l& -k_3\\ 0& 1& 0& 0\\ -\sin {\phi }_l& 0& \cos {\phi }_l& 0\\ 0& 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& \cos {\theta }_l& -\sin {\theta }_l& -k_4\\ 0& \sin {\theta }_l& \cos {\theta }_l& 0\\ 0& 0& 0& 1\end{array}\right].$$

(4)

3. Target Search Model in CIBNCM Mode for WMRs

Based on the movement mechanism of the chameleon visual system, a target search model in chameleon-inspired binocular negative correlation movement (CIBNCM) mode for WMRs was established, as shown in Equation (5). With this model, the negative correlation movement behavior of two cameras in the process of target search was analyzed, and the coordinated behavior of the eyes and neck was discussed. The relationship between the rotation angles of the cameras and neck and the camera’s field of view (FOV), the robot’s region of interest (ROI), and the overlap angle is quantitatively described. To deploy the contrastive analysis, a target search model in binocular coordinated movement (BCM) mode was also built, as shown in Equation (6).

$$\begin{array}{l}{S}_{ng}\left[\left({\varphi }_{ngb},{\phi }_{ngl},{\phi }_{ngr},{\theta }_{ngl},{\theta }_{ngr},f_{ng},Mod(j=0)\right)\to \left(F_{ngv},C_{ngo}\right)\right]\\ \hspace{1em}\mathrm{s}.\mathrm{t}.\left\{\begin{array}{l}{\left({\varphi }_{ngb},{\phi }_{ngl},{\phi }_{ngr}\right)}_{\min }\le {\varphi }_{ngb},{\phi }_{ngl},{\phi }_{ngr}\le {\left({\varphi }_{ngb},{\phi }_{ngl},{\phi }_{ngr}\right)}_{\max }\\ {\left({\theta }_{ngl},{\theta }_{ngr}\right)}_{\min }\le {\theta }_{ngl},{\theta }_{ngr}\le {\left({\theta }_{ngl},{\theta }_{ngr}\right)}_{\max }\\ f_{ng\min }\le f_{ng}\le f_{ng\max }\end{array}\right.\end{array}$$

(5)

$$\begin{array}{l}{S}_{co}\left[\left({\varphi }_{cob},{\phi }_{col},{\phi }_{cor},{\theta }_{col},{\theta }_{cor},f_{co},Mod(j=1)\right)\to \left(F_{cov},C_{coo}\right)\right]\\ \hspace{1em}\mathrm{s}.\mathrm{t}.\left\{\begin{array}{l}{\left({\varphi }_{cob},{\phi }_{col},{\phi }_{cor}\right)}_{\min }\le {\varphi }_{cob},{\phi }_{col},{\phi }_{cor}\le {\left({\varphi }_{cob},{\phi }_{col},{\phi }_{cor}\right)}_{\max }\\ {\left({\theta }_{col},{\theta }_{cor}\right)}_{\min }\le {\theta }_{col},{\theta }_{cor}\le {\left({\theta }_{col},{\theta }_{cor}\right)}_{\max }\\ f_{co\min }\le f_{co}\le f_{co\max }\end{array}\right.\end{array},$$

(6)

where

• $S_{ng}\left(·\right),S_{co}\left(·\right)$—describes the target search model in CIBNCM and BCM mode for WMRs and is represented by $Mod(j=0,1)$, respectively;
• $f_{ng},f_{co}$—expresses the focal length of cameras;
• $F_{ngv},F_{cov}$—is the obtained FOV in CIBNCM and BCM mode;
• $C_{ng},C_{co}$—denotes the detected target characteristic using the selective attention algorithm.

The relationship between the rotation angles of the two cameras and neck and the camera’s FOV, the robot’s ROI, and the overlap angle is shown in Equation (7):

$$\begin{array}{c}R=\left\{{\varphi }_b,{\phi }_l,{\phi }_r,{\theta }_l,{\theta }_r,Mod(j)\right\}=R\left(\alpha ,\beta ,\delta ,r,s,\Psi \times \Phi ,Mod(j)\right),\hspace{1em}\left(j=0,1\right)\\ s.t.\left\{\begin{array}{l}{\left({\varphi }_b,{\phi }_l,{\phi }_r\right)}_{\min }\le {\varphi }_b,{\phi }_l,{\phi }_r\le {\left({\varphi }_b,{\phi }_l,{\phi }_r\right)}_{\max }\\ {\left({\theta }_l,{\theta }_r\right)}_{\min }\le {\theta }_l,{\theta }_r\le {\left({\theta }_l,{\theta }_r\right)}_{\max }\end{array}\right.\end{array},$$

(7)

where

• $\Psi \times \Phi$—represents the robot’s ROI;
• $\alpha ,\beta$—is the horizontal and vertical FOV of each camera, ${\alpha }_l={\alpha }_r=\alpha ,{\beta }_l={\beta }_r=\beta$;
• $\delta$—denotes the overlap angle, ${\delta }_h={\delta }_v=\delta$;
• $r_{ng},r_{co},s_{ng},s_{co}$—is the rotation time of cameras in the horizontal and vertical direction in CIBNCM and BCM mode, $s_{ng}=s_{co}=s$.

According to the FOV of each camera, the robot’s ROI, and the overlap angle, the rotation times of cameras in the horizontal and vertical directions can be deduced, as follows:

$$\begin{array}{ll}\left\{\begin{array}{l}2\alpha r_{ng}-2\delta \left(r_{ng}-1\right)={\Psi }_0^{+\alpha }\\ \beta s_{ng}-\delta \left(s_{ng}-1\right)={\Phi }_0^{+\beta }\end{array}\right.,& (j=0)\\ \left\{\begin{array}{l}\alpha r_{co}-\delta \left(r_{co}-1\right)={\Psi }_0^{+\alpha }\\ \beta s_{co}-\delta \left(s_{co}-1\right)={\Phi }_0^{+\beta }\end{array}\right.,& (j=1)\end{array}.$$

(8)

Subsequently, to scan the horizontal ROI, the two cameras and neck need to rotate by the following angles.

$$\left\{\begin{array}{ll}{\varphi }_{ngb}+{\phi }_{ngi}={(-1)}^c\left(\frac{\Psi }{2}-\frac{\left(2p_{ng}-1\right)\alpha }{2}+\left(p_{ng}-1\right)\delta \right), \left(p_{ng}=r_{ng},r_{ng}-1,\dots ,1\right),& (j=0)\\ {\phi }_{coi}=\frac{\Psi }{2}-\frac{\left(2p_{co}-1\right)\alpha }{2}+\left(p_{co}-1\right)\delta , \left(p_{co}=r_{co},r_{co}-1,\dots ,1\right),& (j=1)\end{array}\right.,$$

(9)

where $i\in \left\{l,r\right\}$ is the label of cameras, corresponding to $c=0,1$.

Similarly, the rotation angles of cameras in the vertical direction are obtained as follows:

$${\theta }_i=\frac{\Phi }{2}-\frac{\left(2q-1\right)\beta }{2}+\left(q-1\right)\delta , \left(q=s,s-1,\dots ,1\right), (j=0,1).$$

(10)

Through the above modeling analysis of the two search modes, a few points can be found under the condition of the same size of ROI. For the horizontal rotation angles of cameras, first, the rotation times in CIBNCM mode are reduced by half compared to those in BCM mode, which not only indicates that the power consumption of camera servos can be reduced by half, but also that the search efficiency can be doubled. Second, the processing image number of the CIBNCM mode is half that of the BCM mode, which can significantly reduce the burden of image processing and improve the processing efficiency. Finally, the introduction of the neck increases the freedom of the visual system and creates the same search area through three axis rotation, improving the search efficiency. Briefly, the combination of the neck and cameras makes the search more flexible, and the rotation angles of the cameras smaller. For the vertical rotation angles of cameras, under normal working conditions of the cameras, there is no difference between the two models; if visual contamination occurs, the CIBNCM mode can give full play to the advantage of its negative correlation movement, achieving complementary visual perception.

4. CBR-Based Contingency Planning of Chameleon-Inspired Visual Contamination for WMRs

A basic assumption of traditional robot artificial intelligence planning is that there is no uncertainty in the external environment in which the robot is located, i.e., the robot has all the a priori knowledge of the external environment and all the action outputs of the robot are predictable. Therefore, traditional robot planning is based on deterministic planning, which has disadvantages such as poor fault tolerance and the need for human monitoring at all times. Contingency planning in this paper was perception-based behavior planning for mobile robots in response to unexpected events. Given the randomness and diversity of contingencies, no deterministic prediction can be made, so a CBR-based state-behavior planning model was adopted, i.e., a responsive dynamic planning method that uses CBR to reason about the corresponding behavior strategy based on the current state of the robot. CBR is an incremental learning method that continuously updates and improves the case base to make the reasoning results more accurate to better adapt to the specific situation [51,52,53]. It is noted that the contingency planning of visual contamination for mobile robots is responsive planning, which is a planning method that generates action plans directly based on observations of the state of the world. Contingency planning is based on the general planning of robots and is triggered only when unexpected events occur. The CBR-based contingency planning model of chameleon-inspired visual contamination for WMRs was established, as shown in Equation (11):

$${ℂ}_P^{vp}={ℂ}_P^{vp}\left\{{\mathbb{Z}}_{\partial }^{vp},B_{\wp }^{vp}\left({\Theta }_{\ell }^{vp},{\Delta }_{ƛ}^{vp},{\mathsf{\Upsilon }}_{\hslash }^{vp},{\Gamma }_{\gamma }^{vp},{\Omega }_{\Xi }^{vp}\right), {\mathbb{N}}_{†}^{vp}\right\},$$

(11)

where

• ${ℂ}_P^{vp}$ is the CBR-based contingency planning space of chameleon-inspired visual contamination for WMRs;
• ${\mathbb{Z}}_{\partial }^{vp},{\mathbb{N}}_{†}^{vp}$ represents the state space and corresponding action space of visual contamination for WMRs, respectively;
• $B_{\wp }^{vp}\left({\Theta }_{\ell }^{vp},{\Delta }_{ƛ}^{vp},{\mathsf{\Upsilon }}_{\hslash }^{vp},{\Gamma }_{\gamma }^{vp},{\Omega }_{\Xi }^{vp}\right)$ denotes the CBR-based reasoning space, and ${\Theta }_{\ell }^{vp},{\Delta }_{ƛ}^{vp},{\mathsf{\Upsilon }}_{\hslash }^{vp},{\Gamma }_{\gamma }^{vp},{\Omega }_{\Xi }^{vp}$ expresses every step of the CBR-based reasoning process, respectively.

As shown in Figure 4, the visual system was monitored in the process of robots perceiving the environment. Once the contaminant was detected, the contamination detection algorithm was used to extract the required parameters of visual contamination cases. Then, the characteristic parameters were input into the CBR reasoning cycle to retrieve similar cases in the case base. Subsequently, the solutions of the retrieved base cases were reused to obtain a plan corresponding to the current contamination condition, and it was output to the robot control system, generating a series of actions. Finally, the visual contamination plan was evaluated, cases were stored selectively, and the case base was maintained to guarantee real-time.

4.1. Case Representation of Visual Contamination

Case representation is composed of case description characteristics and solution characteristics. The binocular visual contamination of WMRs is characterized by the transparency of the FOV, the centroid position of contamination, and the camera contamination topology, expressed as $D_{vp}=\left\{G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right\}$. The solution characteristic is the coping strategy of visual contamination, expressed as $S_{vp}$. Therefore, the case representation of visual contamination for WMRs can be described as Equation (12).

$$X_{vp}^k:\left\{\left(N_{vp}^k,T_{vp}^k,\stackrel{V_{Dvp}^k}{\overbrace{\left[v_{\left(G_{po}^m\right)}^k,v_{\left(P_{po}^n\right)}^k,v_{T_{ca}^{}\left(L^i,R^j\right)}^k\right]}}\right)\to v_{Svp}^k\right\},\left(k=1,2,\dots \dots m\right),$$

(12)

where $X_{vp}^k$ is the $k^{th}$ case, $m$ is the number of cases in the case base, $N_{vp}^k,T_{vp}^k$ are the name and time of the $k^{th}$ case, $V_{Dvp}^k=\left\{v_{\left(G_{po}^m\right)}^k,v_{\left(P_{po}^n\right)}^k,v_{T_{ca}^{}\left(L^i,R^j\right)}^k\right\}$ is the value of the description characteristics of the $k^{th}$ case, and $v_{Svp}^k$ is the value of the solution characteristic of the $k^{th}$ case.

Then, based on the case representation, the case base of the visual contamination for WMRs is established, as shown in Equation (13),

$$B_C^{vp}=\left\{X_{vp}^k:\left\{\left(N_{vp}^k,T_{vp}^k,\stackrel{V_{Dvp}^k}{\overbrace{\left[v_{\left(G_{po}^m\right)}^k,v_{\left(P_{po}^n\right)}^k,v_{T_{ca}^{}\left(L^i,R^j\right)}^k\right]}}\right)\to v_{Svp}^k\right\},\left(k=1,2,\dots \dots m\right)\right\}.$$

(13)

The key to case retrieval is to objectively define and quantify the similarity between cases. K-Nearest Neighbors (KNN) is the most widely used case similarity calculation method, which is easy to understand and realize. It is known as weight retrieval because the case similarity is determined by the weighted summation of every characteristic.

It is known that the base case is expressed as $X_{vp}^k,\left(k=1,2,\dots \dots m\right)$ and the target case is described as $X_{vp}^U:\left\{\left(N_{vp}^U,T_{vp}^U,V_{Dvp}^U\right)\to v_{Svp}^U\right\}$, where $V_{Dvp}^U=\left\{v_{\left(G_{po}^m\right)}^U,v_{\left(P_{po}^n\right)}^U,v_{T_{ca}^{}\left(L^i,R^j\right)}^U\right\}$. Therefore, the characteristic similarity $sim(v_{dvpi}^U,v_{dvpi}^k)$ and case similarity $SIM(X_{vp}^U,X_{vp}^k)$ can be calculated by Equations (14) and (15), respectively.

$$sim(v_{dvpi}^U,v_{dvpi}^k)=\left\{\begin{array}{l}1-\frac{\left|v_{dvpi}^U-v_{dvpi}^k\right|}{\max (v_{dvpi}^U,v_{dvpi}^k)}, \text{Real-number} \mathrm{Attr}.\\ \frac{1}{1+\sqrt{{\displaystyle \sum _{j=1}^m{\left(v_{dvpi}^U-v_{dvpi}^k\right)}^2}}}, \mathrm{Coordinate} \mathrm{Attr}.\\ \left\{\begin{array}{ll}1,& v_{dvpi}^U=v_{dvpi}^k\\ 0,& v_{dvpi}^U\ne v_{dvpi}^k\end{array}, \mathrm{Logic} \mathrm{Attr}.\right.\end{array}\right.,\hspace{1em}\left(i=1,2,3\right)$$

(14)

$$SIM(X_{vp}^U,X_{vp}^k)={\displaystyle \sum _{i=1}^n{\omega }_{i}sim(v_{dvpi}^U,v_{dvpi}^k)},\hspace{1em}\left(i=1,2,3\right),$$

(15)

where ${\omega }_i$ is the weight of every characteristic. $sim(v_{dvpi}^U,v_{dvpi}^k)\in [0,1]$ and meets the symmetry and reflexivity. The higher $SIM(X_{vp}^U,X_{vp}^k)$ is, the higher the similarity between the target case and the base case. Finally, all the matching base cases can be retrieved and ranked in descending order of similarity $SIM(X_{vp}^U,X_{vp}^k)$ and time $T_{vp}^k$.

4.2. Case Reuse of Visual Contamination

In general, the retrieved base cases are not perfectly matched with the target case. Therefore, the solution of the retrieved base cases cannot be directly used for the target case, and it needs to be modified and reused. It is assumed that the matching base cases are described as $X_{lvp}^M:\left\{\left(N_{lvp}^M,T_{lvp}^M,V_{l,Dvp}^M\right)\to v_{l,Svp}^M\right\}, \left(l=1,2,\dots \dots ,a^{(i)}\right)$, where $a^{(i)}$ is the number of matching base cases. The solution of the target case can be obtained by Equation (16) and it can be found that the solution weight of the matching cases in the target case is proportional to the similarity between the matching cases and the target case.

$$v_{Svp}^U=\left\{\begin{array}{ll}{v}_{1,Svp}^M,& SI{M}_{\max }=1\\ \frac{{\displaystyle \sum _{l=1}^{a^{(i)}}(SIM(X_{vp}^U,X_{lvp}^M)\times v_{l,Svp}^M)}}{{\displaystyle \sum _{l=1}^{a^{(i)}}SIM(X_{vp}^U,X_{lvp}^M)}},& ELSE\end{array}\right..$$

(16)

4.3. Case Evaluation and Revision of Visual Contamination

Through the above case retrieval and reuse, we can obtain the solution to the target case. However, what is the accuracy of the solution? We need to evaluate the implementation of the solution. If successful, this new case will be selectively saved in the case base according to the following rules; otherwise, the solution needs to be revised. The pseudo-code for case evaluation and revision is shown in Figure 5.

4.4. Case Retention and Case-Base Maintenance of Visual Contamination

CBR is a kind of experience-based reasoning method, and the diversity and advancement of experience play an important role in the accuracy of reasoning results. As time goes on, the number of cases is increasing; however, the continuous increase will not only lead to the overlap of cases and loss of representation but will also affect the efficiency and accuracy of case reasoning. In addition, with the development of social sciences, cases that are too old may no longer adapt to the current situation. Therefore, to increase the diversity of the case base and maintain the real-time of cases, we need to properly handle the retention and maintenance of cases. In this paper, the human supervision-based method was used to achieve the retention and maintenance of cases. The pseudo-code for case retention and case base maintenance based on human supervision is shown in Figure 6.

5. Perception Model in Chameleon-Inspired Visual Contamination of WMRs

In terms of the contamination of the chameleon-inspired visual system, an environmental perception model was established, as shown in Figure 7. First, through the analysis of the contamination condition of the binocular visual system, the state space was established based on the characteristics of visual contamination. Second, coping strategies were designed according to the contamination condition of the binocular visual system and, further, the action space of visual contamination was established corresponding to the state space.

According to different visual contamination situations described by the description characteristics, the coping scheme of visual contamination was designed. First, when the transparency of the FOV was greater than or equal to 75%, FOV mosaic perception based on camera movement was used to perceive the ROI. It can preserve the degraded visual perception and avoid stopping normal work to clean the contamination. In contrast, when the transparency of the FOV was less than 75%, the robot was seriously contaminated and had to be cleaned immediately. Human eye-inspired decontamination based on the Coanda effect was adopted, which is not described in this paper. Second, the location of the visual contamination was divided into nine typical positions based on the coordinate quadrant. Finally, the camera contamination topology of the binocular visual system was divided into four kinds of arrangements (00, 01, 10, 11).

In this paper, camera transparency was defined as the ratio of the camera non-contamination area to the entire camera area, so the equations for calculating the transparency of a single camera $G_{po}^{mo}$ and the binocular visual system $G_{po}$ were obtained as follows:

$$\left\{\begin{array}{l}{G}_{po}^{mo}=1-\frac{S_{po}^{mo}}{S_{ca}^{mo}}\times 100\%\\ G_{po}=1-\frac{\left(S_{po}^{mol}+S_{po}^{mor}\right)}{2\times S_{ca}^{mo}}\times 100\%\end{array}\right.,$$

(17)

where $S_{po}^{mo}$ denotes the contamination area of the monocular camera, $S_{ca}^{mo}$ is the area of the entire camera, and $S_{po}^{mol},S_{po}^{mor}$ represents the contamination area of the left and right cameras, respectively.

To determine the centroid position of the contamination, define the distance similarity as $S_{dt}$. That is, the similarity is expressed by the distance between the centroid of contamination and the nine reference points. The smaller the distance is, the higher the similarity between the two points.

$$S_{dt}=\frac{1}{1+\sqrt{{\left(x_p-x_{qi}\right)}^2+{\left(y_p-y_{qi}\right)}^2}},\hspace{1em}\left(i=1,2,\cdots ,9\right),$$

(18)

where $\left(x_p,y_p\right)$ denotes the coordinates of the contamination centroid, $\left(x_{qi},y_{qi}\right),\left(i=1,2,\cdots ,9\right)$ are the coordinates of the reference points, and $\left\{\left(x_q,y_q\right)\right\}=\left\{\begin{array}{l}\left(A/4,B/4\right),\left(3A/4,B/4\right),\left(A/4,3B/4\right),\left(3A/4,3B/4\right),\\ \left(A/2,B/4\right),\left(A/2,3B/4\right),\left(A/2,B/2\right),\left(A/4,B/2\right),\left(3A/4,B/2\right)\end{array}\right\}$.

Therefore, the perception model in chameleon-inspired visual contamination for WMRs can be established as follows:

$${\mathbb{Q}}_{vp}\left\{\mathbb{Z}\left[G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right], \mathbb{N}\left[{\Re }^{m=0}\left\{\aleph \left({\varphi }_{npb},{\phi }_{npl},{\phi }_{npr},{\theta }_{npl},{\theta }_{npr}\right)\right\}, {\Im }^{m=1}\left(K,Y\right)\right]\right\},$$

(19)

where

• ${\mathbb{Q}}_{vp}\left(·\right)$ is the perception space in chameleon-inspired visual contamination for WMRs, consisting of the state space and action space of visual contamination;
• $\mathbb{Z}\left(·\right)$ is the state space of chameleon-inspired visual contamination, where $G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)$ represents the transparency of the FOV, the centroid position of contamination, and the camera contamination topology, respectively,

  $$m=\left\{\begin{array}{ll}0,& G_{po}\ge 75\%\\ 1,& G_{po}<75\%\end{array}\right.$$

  $$\left(i,j\right)=\left\{\begin{array}{ll}\left(0,0\right),& T_{ca}\left(L^i,R^j\right)=\left(0,0\right)\\ \left(0,1\right),& T_{ca}\left(L^i,R^j\right)=\left(0,1\right)\\ \left(1,0\right),& T_{ca}\left(L^i,R^j\right)=\left(1,0\right)\\ \left(1,1\right),& T_{ca}\left(L^i,R^j\right)=\left(1,1\right)\end{array}\right., \left(i=j=\left\{\begin{array}{ll}0,& unpolluted\\ 1,& polluted\end{array}\right.\right)$$

  $$n=\left\{\begin{array}{ll}0,& P_{po}^n\in Q_0=\left\{\left(x,y\right)|x\approx A/2\&y\approx B/2\right\}\\ 1,& P_{po}^n\in Q_1=\left\{\left(x,y\right)|x>A/2\&y>B/2\right\}\\ 2,& P_{po}^n\in Q_2=\left\{\left(x,y\right)|x<A/2\&y>B/2\right\}\\ 3,& P_{po}^n\in Q_3=\left\{\left(x,y\right)|x<A/2\&y<B/2\right\}\\ 4,& P_{po}^n\in Q_4=\left\{\left(x,y\right)|x>A/2\&y<B/2\right\}\\ 5,& P_{po}^n\in Q_5=\left\{\left(x,y\right)|x\approx A/2\&y>B/2\right\}\\ 6,& P_{po}^n\in Q_6=\left\{\left(x,y\right)|x\approx A/2\&y>B/2\right\}\\ 7,& P_{po}^n\in Q_7=\left\{\left(x,y\right)|x<A/2\&y\approx B/2\right\}\\ 8,& P_{po}^n\in Q_8=\left\{\left(x,y\right)|x>A/2\&y\approx B/2\right\}\end{array}\right.$$

  $\mathbb{N}\left(·\right)$—is the action space of chameleon-inspired visual contamination, where ${\Re }^{m=0}\left\{\aleph \left({\varphi }_{npb},{\phi }_{npl},{\phi }_{npr},{\theta }_{npl},{\theta }_{npr}\right)\right\}$ describes the robot’s coping strategies when the transparency of the FOV satisfies $D_{po}^m\ge 75\%, (m=0)$, and ${\Im }^{m=1}\left(K,Y\right)$ is the robot’s coping strategies when the transparency of the FOV satisfies $D_{po}^m<75\%, (m=1)$.

When $D_{po}^m\ge 75\%, (m=0)$, the robot’s coping strategy is FOV mosaic perception based on camera movement. The effective horizontal and vertical FOV of each camera after contamination is defined as ${\alpha }_{ld},{\alpha }_{rd},{\beta }_{ld},{\beta }_{rd}$, so the effective FOV of the binocular visual system can be determined by Equation (20),

$$ℝ\left({\alpha }_{ld},{\alpha }_{rd},{\beta }_{ld},{\beta }_{rd}\right)=\Xi \left\{\mathbb{Z}\left[G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right]\right\}.$$

(20)

The search model in CIBNCM mode was used to perceive the environment after contamination in this paper. According to the effective FOV of each camera after contamination, the robot’s ROI, the overlap angle, and the rotation times of cameras in the horizontal and vertical directions can be deduced as follows:

$$\left\{\begin{array}{l}\left({\alpha }_{ld}+{\alpha }_{rd}\right)r_{np}-2\delta \left(r_{np}-1\right)={\Psi }_0^{+\left[\min \left({\alpha }_{ld},{\alpha }_{rd}\right)\right]}\\ {\beta }_{id}{s}_{npi}-\delta \left(s_{npi}-1\right)={\Phi }_0^{+{\beta }_{id}},\hspace{1em}i\in \left\{l,r\right\}\end{array}\right..$$

(21)

Subsequently, to scan the horizontal ROI, the two cameras and neck need to rotate by the following angles.

$${\varphi }_{npb}+{\phi }_{npi}={(-1)}^c\left(\frac{\Psi }{2}-\frac{\left(2p_{np}-1\right){\alpha }_{id}}{2}+\left(p_{np}-1\right)\delta \right), \left(p_{np}=r_{np},r_{np}-1,\dots ,1\right), i\in \left\{l,r\right\}, c=0,1.$$

(22)

Similarly, the rotation angles of cameras in the vertical direction are

$${\theta }_{npi}=\frac{\Phi }{2}-\frac{\left(2q_{npi}-1\right){\beta }_{id}}{2}+\left(q_{npi}-1\right)\delta , \left(q_{npi}=s_{npi},s_{npi}-1,\dots ,1\right), i\in \left\{l,r\right\}.$$

(23)

Finally, in terms of the different contamination situations described by the transparency of the FOV, the centroid position of contamination, and the camera contamination topology, the corresponding coping strategies of WMRs can be obtained as follows:

$${\Re }^{m=0}\left\{\aleph \left({\varphi }_{npb},{\phi }_{npl},{\phi }_{npr},{\theta }_{npl},{\theta }_{npr}\right)\right\}=\Pi \left\{\mathbb{Z}\left[G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right]\right\}.$$

(24)

6. Experiments and Comparison

To validate the proposed CBR-contingency planning method for visual contamination, an environment perception experiment in chameleon-inspired visual contamination was designed for WMRs, as shown in Figure 8. It is noted that the contingency planning of visual contamination was based on the general planning of robot target tracking. Target tracking is the basic task of the robot, and contingency planning is triggered only when visual contamination is detected.

First, the target search model in CIBNCM mode was adopted for mobile robots to scan the environment and the target was extracted using the selective attention algorithm (SAA). Second, to achieve high accuracy, rotation of the robot body and neck and a zooming model were used to realize targets. Finally, real-time target tracking was achieved using a particle filter algorithm (PFA). In the process of target tracking, real-time detection of visual contamination was carried out. Once contaminants were found, the robot was commanded to stop. Meanwhile, the description characteristics of visual contamination were extracted and input to the CBR cycle to reason out solutions using the FOV mosaic perception based on camera movement. Then, the obtained solution was output to the robot control system to track the target again.

In this experiment, the robot was located on flat ground to search and track the red object in front, as shown in Figure 9. Contaminants were monitored in the process of target tracking. At the 44th second, contaminants were detected and the robot stops. The corresponding description characteristics were $\left\{G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right\}=\left\{0.75-1,\left(99,48\right),\left(0,1\right)\right\}$. Inputting them into the CBR cycle, the reasoned-out coping strategy based on FOV mosaic perception is as follows:

$$\left({\varphi }_{npb},{\phi }_{npl},{\phi }_{npr},{\theta }_{npl},{\theta }_{npr}\right)=\left\{\begin{array}{l}\left(10,46,-46,30,30\right),q=1\\ \left(-18,34,-34,30,30\right),q=2\\ \left(-46,22,-22,30,30\right),q=3\end{array}\right..$$

Subsequently, the robot visual system performed the above CBR reasoning solution and took pictures once when each rotation stops. Through the analysis of the obtained six images, it was determined that, when the neck and right camera rotate by $\left({\varphi }_{npb},{\phi }_{npr},{\theta }_{npr}\right)=\left(10,-46,30\right)$, the area at $\left(88,126\right)$ has the highest saliency. Then, the orientation of the object was calculated with $\left(-8.9°,28.4°\right)$. Through the rotation and zooming model, the robot can aim at the target precisely.

Finally, the target was tracked again and the tracking error curve is shown in Figure 10. The tracking error was between plus or minus 15 pixels, which shows that the tracking effect is satisfactory.

The experimental results show the following: on the one hand, the contingency planning experiment scheme for visual contamination based on the robot’s general planning of target tracking is reasonable; on the other hand, the proposed CBR-based contingency planning method for visual contamination is validated, which can reason out proper solutions for the current contamination situation.

In addition, to further verify the effectiveness of the method, a comparison experiment of the RBR-based contingency planning method was designed. By comparing and analyzing the accuracy of robot target retracking after contingency planning of visual contamination based on the two reasoning methods, the coping ability of the contingency planning methods based on two reasoning methods for the visual contamination problem was compared and analyzed.

Knowledge reasoning is the thinking process of reasoning new knowledge from existing knowledge according to a certain strategy through a computer system [54,55]. That is, after inputting the problem to be solved, the existing design cases, expert experience, computational formulas, design specifications, and other knowledge in the knowledge base are called to simulate human thinking to solve the problem. Common knowledge reasoning methods are case-based reasoning (CBR) and rule-based reasoning (RBR). RBR uses a rule base and inference engine for reasoning, which relies more on the organization and form of rules and requires higher abstraction and induction ability for problem solution [56,57]. Based on rule-based reasoning, rule-based knowledge, such as expert experience, computational formulas, and design specifications, is abstracted into symbolic and normative concrete generative rules, and the rules are matched with facts to obtain conclusions through rule-based reasoning. The process of RBR-based contingency planning for robot vision contamination in this paper is shown in Figure 11.

In the process of target tracking, the robot performs image processing on the image information obtained from binocular vision, thus realizing real-time detection of visual contamination. Once the contaminant is found, the robot is ordered to stop moving and the fact of the visual contamination is extracted and input into the rule base. The rule premise is matched with the current fact until the matching is successful, and the rule conclusion, i.e., the solution, is obtained and output to the robot control system to execute it for target retargeting and tracking. For the same contamination case as the CBR-based contingency planning experiment, the fact of its visual contamination is described as $\left\{G_{po}^m,P_{po}^n,T_{ca}\left(L^i,R^j\right)\right\}=\left\{0.75-1,\left(99,48\right),\left(0,1\right)\right\}$ and the solution inferred by RBR is

$$\left({\varphi }_{npb},{\phi }_{npl},{\phi }_{npr},{\theta }_{npl},{\theta }_{npr}\right)=\left\{\begin{array}{l}\left(9,44,-44,28,28\right),q=1\\ \left(-16,33,-33,28,28\right),q=2\\ \left(-44,20,-20,28,28\right),q=3\end{array}\right..$$

The results of the RBR-based contingency planning experiment are shown in Figure 12, from which we can see that, for the same contamination situation as the CBR-based contingency planning experiment, the robot vision system cannot lock the target well due to the low accuracy of the RBR solution. The two cameras and the neck produce significant shaking during the target tracking process and the accuracy of the robot target tracking is reduced. The tracking error is basically between plus and minus 25 pixels and the tracking curve oscillates seriously. In comparison, in the CBR-based contingency planning experiment, the two cameras and the neck basically remain motionless during the target tracking process, the tracking error is basically between plus and minus 15 pixels, and the tracking accuracy is higher. Therefore, for the contingency planning of visual contamination in this paper, the CBR-based planning method has better adaptability and higher planning accuracy.

7. Conclusions

In this paper, to improve the adaptability of autonomous mobile robots to unstructured environments, a CBR-based contingency planning method for robot visual contamination was proposed to address the problem that the visual system of a robot is highly susceptible to contamination during field operations, so that it has the ability to autonomously handle the contingency of visual contamination. First, the negative-correlation movement mechanism of the chameleon visual system was introduced to the visual system of mobile robots to improve the search efficiency of targets. Second, through the analysis of visual contamination in the process of robot environment perception, a CBR-based contingency planning model was proposed, where the CBR reasoning process was described in detail. Third, a perception model in chameleon-inspired visual contamination for WMRs was established, where the action space of environment perception was analyzed corresponding to different state spaces of visual contamination. Finally, a contingency planning experiment scheme for visual contamination was designed based on the robot’s general planning of target tracking, and the proposed approach was validated by the experimental results. The tracking error was between plus or minus 15 pixels, indicating a good tracking effect. This shows that the CBR-based contingency planning method of visual contamination proposed in this paper can reason out an effective solution based on the current contamination situation, which can enable the robot to effectively cope with the problem of visual contamination in the working process and ensure the continuous perception ability of the mobile robot to continue the general planning of target tracking. Moreover, by comparing with the RBR-based contingent planning method, it was found that the accuracy of target retracking after the robot visual system is contaminated is significantly higher for the CBR-based contingent planning method used in this paper.

In view of the complexity and dynamics of the environment in which the robot is located, the visual contamination of the robot also varies. Therefore, future research will focus on analyzing and detecting different visual contamination situations of the robot in complex environments and the corresponding CBR reasoning process. Through contingency planning for the emergency of a robot’s visual contamination, the robot is equipped with autonomous problem-solving abilities to achieve maximum adaptation to the working environment.