Victim Localization in USAR Scenario Exploiting Multi-Layer Mapping Structure

Abstract

Urban search and rescue missions require rapid intervention to locate victims and survivors in the affected environments. To facilitate this activity, Unmanned Aerial Vehicles (UAVs) have been recently used to explore the environment and locate possible victims. In this paper, a UAV equipped with multiple complementary sensors is used to detect the presence of a human in an unknown environment. A novel human localization approach in unknown environments is proposed that merges information gathered from deep-learning-based human detection, wireless signal mapping, and thermal signature mapping to build an accurate global human location map. A next-best-view (NBV) approach with a proposed multi-objective utility function is used to iteratively evaluate the map to locate the presence of humans rapidly. Results demonstrate that the proposed strategy outperforms other methods in several performance measures such as the number of iterations, entropy reduction, and traveled distance.

1. Introduction

The current advanced development of aerial robotics research has enabled their deployment in a variety of tasks such as urban search and rescue (USAR) [1,2,3,4], infrastructure inspection [5], 2D/3D reconstruction for building using point cloud generated from UAVs Images [6] or 3D reconstruction of historical cities by using the laser scanning and digital photogrammetry [7], and mining [8]. Notably, the intervention of the UAVs in the USAR environment is critical in the sense that they require rapid response to locate all victims and survivors in this environment. Many sensors, such as RGB color cameras [9] and thermal cameras [10,11], were used to localize victims in the literature. Single sensors methods cannot effectively locate victims due to the harsh nature of USAR environments. Some sensors cannot operate under certain conditions such as smoke, dust, radiation, and gases. For instance, vision-based systems are incapable of locating a victim under dark lighting condition or in the presence of occlusions. Additionally, thermal-based systems can be deceived by excessive heat sources such as fires. Consequently, recent studies have been conducted to reduce victim localization time by using multiple sensors, thus improving the detection and localization performance in such unfavorable environment. The work presented in this paper tackles considers each sensor as a victim detector module. These detector modules are independent of one another, and they are used to generate victim-location maps. A probabilistic sensor fusion technique is performed on the generated maps to create a new merged map. Consequently, this map is used by the robot for environment exploration to localize the presence of victims.

In the current state of the art, a variety of contributions have been made to accomplish autonomous exploration. Most autonomous exploration approaches use frontier exploration [12,13] and, information gain theory, e.g., next-best-view(NBV) [14]. Different types of maps can be constructed to project different information such as topological occupancy maps, metric maps, and semantic maps. Human detection was widely studied by training Support Vector Machine (SVM) classifier to detect human presence from 2D images [15,16,17]. Recently, deep-learning algorithms are applied for object and human detection [18]. In a NBV exploration process, the UAV navigates in the environment using a utility function, also called a reward function, which selects the next best exploration activities that minimize the effort required to localize victims.

In this work, we present a multi-sensor-based NBV system to locate victims in a simulated urban search and rescue environment. The proposed multi-sensor system uses different sensor sources, namely vision, thermal, and wireless, to generate victim-location maps which are then fused into a merged map using probabilistic Bayesian framework.

Moreover, a multi-objective utility function is proposed to speed up the search mission of locating victims. The proposed multi-objective utility function evaluates viewpoints based on three desired objectives namely exploration, victim detection and traveled distance. Finally, an adaptive grid sampling algorithm (AGSA) is developed to resolve the local minimum issue where the robot tends to move in a repeated pattern [19], a problem commonly occurs in the Regular Grid Sampling Approach (RGSA).

In particular, our contributions are as follows:

• A multi-sensor fusion approach that uses vision, thermal, and wireless sensors to generate a probabilistic 2D victim-location maps.
• A multi-objective utility function that uses the fused 2D victim-location map for viewpoints evaluation task. The evaluation process targets three desired objectives, specifically exploration, victim detection, and traveled distance.
• An adaptive grid sampling algorithm (AGSA) to resolve the local minimum issue occurs in the Regular Grid Sampling Approach (RGSA).

The rest of the paper is organized as follows; Section 2 outlines the related work. The detailed proposed approach is presented in Section 3. Section 4 presents the simulation process and the experimental results. Finally, the conclusion and future works are given in Section 5.

2. Related Work

To autonomously locate victims inside an urban search and rescue environments with a robotic system, multiple prerequisite components should exist on the robotic system to perform its task effectively. In this section, the various robotic components that constitutes an autonomous search and rescue robot are presented, and related work to this component will be described.

2.1. Victim Detection

Victim detection is the most critical task in search and rescue missions where victims should be identified as fast as possible for the first responders to intervene and rescue them. Throughout the literature, the human detection problem has been investigated extensively from different perspectives such as thermal [17,20] and visual perspectives [15,16]. Various sensors enabled different strategies to detect humans such as the RGB cameras and thermal cameras.

From thermal detection perspective, authors in [15] used a background subtraction along with trained Support Vector Machine (SVM) classifier to detect human presence in images streamed from a thermal camera. Similarly, authors in [16] used a combination of Histogram of Oriented Gradients (HOG) and CENsus TRansform hISTogram (CENTRIST) features to train the SVM classifier for thermal detection.

However, from the visual detection scope, the detection is extensively performed using machine learning methods and currently through deep-learning methods. In the context of machine learning approaches, authors in [17] used SVM classifier along with HOG features to locate victims with different poses. They focused on diversifying their dataset with different human poses and lighting condition to boost the detection accuracy. Additionally, in [20] a rotational invariant F-HOG classifier was used to detect a lying-pose human in RGB image. They showed that the F-HOG is more discriminative and performs better compared to the traditional HOG feature descriptor.

Machine learning detection methods suffer from several limitations resulting in lower accuracy/precision performance. First, machine learning approaches consider training a set of images with a nearly uniform background, which is not reliable when performing it in a real-world disaster area. Additionally, machine learning algorithms represent human with pre-defined features. However, this assumption may not be capable of generalizing human features, which lead to detection error. Therefore, deep-learning models are used to achieve higher accurate detection without feature extraction process.

However, more accurate approaches to human detection are achieved using deep learning [18,21]. Unlike conventional methods, deep-learning models can perform higher detection accuracy by finding the most efficient features patterns existed in trained visual input [18,21,22]. An accurate approaches to human detection are achieved using deep learning is found in [18,21]. In [22] author succeeds in running the deep-learning model called Fast R-CNN at 5 fps with reasonable detection accuracy. A faster detection model was introduced in [21] where the authors designed a real-time deep-learning network called Single-Shot multi-box detector (SSD). The model involves multiple convolutional layers that perform the same detection but at different aspect ratio leading to higher overall detection accuracy. The achieved detection speed is 59 FPS on Nvidia Titan X.

However, one main drawback in deep-learning classifiers is the accuracy vs. speed trade-off. In other words, accurate CNN model tends to have more layers with a larger size to learn deep features, which consequently requires a longer time to evaluate the input images streams in real-time requirements. For examples, authors in [18] proposed a CNN structures containing 7 layers, 5 convolutional layers and 2 full-connected layers. The model is pre-trained on the ImageNet database, and then the last two fully connected layers in the CNN were fine-tuned to provide the model with distinguishable features for the victims. The main disadvantage of the system is that it possesses a high computational load and need a separate server to run the detection model. Recent work was designed to address and improve the accuracy vs. speed trade-off.

Another strategy that is used to localize humans accurately is the sensor fusion. Sensor fusion allows taking the advantages of each sensor while compensating for their limitations. In [23], authors performed sensor fusion to identify victim using rescue robots operating in cluttered USAR environment. The sensor fusion was performed in the feature level where initially an RGB-D image is captured from the environment. Then, a depth region segmentation is performed on the image to get sub-region segmentation. After that, skin feature, body part association and the geometric feature are extracted from the image. Finally, the extracted features are used to train a SVM classier to identify the victim a captured RGB-D image. In [24], the thermal image is used to obtain rough region of interest (ROI) coordinates of human location then the estimated ROI is fused with the depth image to obtain the actual human location along with occlusion handling. In [25], the depth–color joint features are extracted from the RGB-D images which used to train a human detector whose inputs are multi-channels namely LUV, HOG and depth channel. Unlike conventional methods, deep learning can find the most efficient features patterns that existed in trained visual input, in contrast to the conventional methods, which used pre-defined features leading to higher detection accuracy.

In this work, three different sensors are used to capture environment data, RGB Camera, thermal camera, and wireless sensor. Each sensor data is going to be used to generate a 2D map called victim maps. After that, the three maps are fused into one merged map that is going to be used for the exploration process. Our method is different to the state-of-the-art method because of adding the wireless sensor. The wireless sensor is used to indicate the presence of a victim from the mobile phone. Using a wireless sensor with the visual and thermal sensors can help in locating the victim accurately.

2.2. Mapping

While attempting to locate victims in an USAR environment, exploration techniques require the presence of a map that captures the information gathered by the sensors during the exploration method. Among the methods which implemented a human detector, many of them used a mapping structure to stores their detection results. The most popular mapping structure is the 2D and 3D occupancy grid where the work-space is discretized into 2D squares or 3D cubic volumes known as pixels or voxels, respectively and they are used to store the probability that a human is detected.

In occupancy grid maps, a $m_i$ is a cell with index i and this occupancy grid is simply a finite collection of all the cells modeled as $m=\{m_1,m_2,\dots ,m_n)$. Where n is the total number of cells. In a 2D occupancy grid representing victim presence, every cell contains victim detection probability. Clearly, the higher the probability, the higher indication of human presence. The cell that holds a 0.5 probability indicated as an unknown cell where it is equally likely for the victim to be presented or not. Figure 1 illustrates an occupancy cell obtained from an RGB-D camera where ray-tracing is performed to update the map based on the camera visible field of view (FOV).

To update the map, many methods found in the literature used a Bayesian approach [26,27]. They either used the Bayes rule directly or indirectly through one of its derived methods such as Kalman Filter, Particle Filter, etc. The concept behind the Bayes rule is that given some characteristic information and level of confidence about it, it is used to determine the probability a human occupies a given location. Some examples of work which used Bayes rule includes [26,27].

To capture human/victim detection, each cell in the occupancy grid map can hold one of the two distinct states $S=\left\{human,Nonhuman\right\}$. The probability that a human is observed will be denoted as $P\left(h\right)$ and probability of non-human is donated as $\overline{P}\left(h\right)$. Practically, determining the probability $P\left(h\right)$ at specific cell requires the last state on the cell, the observation from the human detection module and the robot motion. Such information is taken into account using the Bayes filter [28].

In this work, three single 2D maps are generated from three different sensors, RGB camera, thermal camera, and wireless sensor. All maps demonstrate the presence of humans and are updated using a Bayesian approach. After that, a merged map is generated by combining the weighted re-scaled maps.

2.3. Exploration

Exploration of unknown environment has been achieved using various methods throughout the literature. Frontier-based [12,13,29] and Next-Best-View (NBV) [14] approaches are the most commonly used approaches. The frontier-based methods aim to explore the frontiers (boundaries between free and unexplored space) of a given area. However, the NBV approaches are used to determine the next viewpoint or candidate position to visit to increase the chance of achieving the objective defined in the utility function. The NBV approach selects the next pose from the list of possible candidates (viewpoints) that score the highest according to the criteria defined in the utility function. The criteria may include: the amount of information that could be potentially obtained by the candidate viewpoint; the execution cost (i.e., energy, path) of the next action [30]; or the distance to be traveled to reach the next best view [14]. Consequently, a proper utility function and information gain prediction have a vital role in NBV selection.

In this research, we use the NBV as it is directly related to the work presented in this paper. The main elements of the NBV approach are viewpoint sampling, viewpoint evaluation, and a termination condition.

2.3.1. Viewpoint Sampling

To determine the next viewpoint to visit in order to maximize the chance of detecting a victim, the common methods sample the space into candidate viewpoints, and evaluate these viewpoints in order to determine which one can provide the best chance of detecting a victim. Different sampling strategies are commonly used in the literature such as (1) Regular Grid Sampling [31], (2) Frontier Sampling [32], and (3) Rapidly exploring Random Trees [33].

In the regular grid sampling, the state space around the robot is discretized based on the number of degrees of freedom (n-DOF) of the robot. Precisely, each degree of the robot is used to divide state space and the possible poses that an n-DOF robot can is visualized as points lying in a discretized n-dimensional space. In the case of 3- dimensional space, the sampled point will look like a crystal lattice; hence the name “state lattice” [31].

In frontier sampling, moving the robot to the boundaries between known and unknown space can maximize the chance of obtaining new information. This approach allows the robot to see beyond the explored map and hence expand the map. Typically, this sampling procedure is done on an occupancy grid map, as it is easy to allocate frontiers where the free and unexplored space presented clearly [13,32].

A simple form of frontier selection is to identify the frontier cell which is closest to the current robot location. Then, the robot moves to that frontier and captures data from its sensor to expand the exploration map. The robot repeats this process until no frontier is left [32,34].

Rapidly exploring Random Trees (RRTs) are another method commonly used to sample in a receding horizon manner [14]. By sampling possible future viewpoints in a geometric random tree, the defined utility function will enable the exploration of unknown volume environment. During this process, a series of random points are generated. Then, these points are connected in a tree-like expanding through explored space. Each branch represents a set of random chains and the branch which maps the unknown space is selected. Each branch of the samples RRT is then evaluated using the defined utility function, and the branch with the best possible exploration outcome is used to select the best new exploration position to travel to.

2.3.2. Viewpoint Evaluation

A utility function is usually used to select the best viewpoint from the list of available candidates generated during the sampling process. Information metric used to evaluate the viewpoints is the information gain [35]. The entropy ($\mathcal{H}$) for a specific cell x in the occupancy map is given as

$${\mathcal{H}}_i=P_{i}log\left(P_i\right)+(1-P_i)log(1-P_i)$$

(1)

From Equation (1), the entropy ${\mathcal{H}}_i$ is a measure of uncertainty associated with the cell probability. ${\mathcal{H}}_i$ is maxed when the all outcomes are equally likely while it is zero when the trial outcomes are certain.

The corresponding view entropy for a specific view denoted by subscript v is the sum on the individual cell’s entropy within the sensor FOV, and defined as:

$${\mathcal{H}}_v=\frac{1}{N}\ast \sum _{i=1}-\left(P_i\ast log\left(P_i\right)+\left(1-P_i\right)\ast log\left(1-P_i\right)\right)$$

(2)

where N is a normalization factor that represents the number of visible cells in the sampled FOV [36]

An extension of the information gain is the weighted information gain with distance penalty [12], given by:

$${\mathcal{H}}_v=e^{-\lambda \ast d}\ast \frac{1}{N}\ast \sum _{i=1}-\left(P_i\ast log\left(P_i\right)+\left(1-P_i\right)\ast log\left(1-P_i\right)\right)$$

(3)

where $\lambda$ is the penalization factor [12].

Another information gain utility function is the max probability. In this method, the viewpoints that contain cells with higher maximum probability are preferred compared to other viewpoints with cells with less maximum probability. This method works well when a victim is found as the next evaluated viewpoints will cover the victim within their FOV aiming to higher or lower the victim detection probability. The evaluation of the max probability gain of a viewpoint [2] is given by:

$${\mathcal{H}}_v=\underset{i}{max}{P}_{i}wherei=\{1,\dots ,N\}$$

(4)

2.3.3. Termination Conditions

Autonomous exploration tasks terminate once a pre-defined condition has been achieved. There are several type of termination conditions depending on the scenario such as whether a victim was found [37], whether a maximum number of iterations has been reached [38], or whether the information gain reached a max [19,31]. Additionally, for frontier-based methods, the exploration terminates when no more frontiers generated [32].

Our work is different from the state-of-the-art approaches in two elements of the NBV, viewpoint sampling, and viewpoint evaluation. The proposed Adaptive Grid Sampling Approach overcomes the limitation of one of the existing state-of-the-art approaches, which is the Regular Grid Sampling Approach [30,31]. Also, we propose a new viewpoint evaluation (utility function) method to detect a human as fast as possible and reduce the traveled distance. Our viewpoint evaluation function considers occupancy, distance, and victim detection using both the proposed merged map and the proposed sampling approach. The proposed utility function is compared with the state-of-the-art utility functions presented in Equations (3) and (4).

3. Proposed Approach

In this work, a new approach for victim localization, depicted in Figure 2, is presented. The proposed victim localization method consists of two main processes: the victim detection/mapping and the NBV exploration processes. In the first process, victim detection/mapping, the victim is detected and localized in multi-layer maps that are generated by fusing vision, thermal, and wireless data. Victim detection is accomplished by applying a detection algorithm that estimates human location with uncertainty. The victim location is stored in an occupancy grid map, which is recursively updated to increase the confidence in locating the victim. In the second process, exploration, the NBV approach is used, and a new utility function is proposed. The NBV approach includes viewpoint sampling, viewpoint evaluation, navigation, and termination conditions.

The novelty of our solution is on the integration of the wireless signal with other various classical sensors to assist in locating victims. This presents a unique solution along with the multi-sensor-based mapping approach and the multi-objective exploration strategy. Furthermore, deep-learning-based human detection is used to detect the presence of a human from a 2D image. This information is then used with a synchronized 3D point cloud to localize the human accurately. Moreover, unlike regular grid sampling approaches, our proposed adaptive viewpoint sampling method solves the local minimum problem. This approach enables rapid exploration and victims’ detection in a USAR environment. Finally, the performance of the proposed approach was compared with other approaches available in the literature, described in Section 4. Results showed that our approach performs better in various metrics such as the number of iterations, traveled distance, and entropy reduction.

3.1. Multi-Sensor Approach for Victim Localization and Mapping

3.1.1. Vision-Based Victim Localization

The procedure for the vision-based victim localization is summarized in Algorithm 1. It involves two main stages: victim detection, where the victim is found in the image frame; and mapping which register the detection confidences in a 2D occupancy grid map.

• Victim Detection: In this work, Single-Shot Multi-box detector (SSD) [21] structure was adapted in our classifier, trained it with visual object class challenge (VOC2007) dataset [39]. The SSD structure is depicted in Figure 3. The input to the detection module is a 2D RGB image and the desired output is a box overlapping the detected human with a probability reflecting the detection confidence. The victim detection aims to find a human in a received RGB image.
• Victim Mapping: Based on the analysis of the SSD detector, the minimum size of the detected box in the training-set is $7\times 10$ pixels. This box size corresponds to $0.6\times 0.8$ in meter. To estimate the distance between the detected human and the camera [40], the ratio $r_p$ of the box $\mathrm{B}$ to the image is along the y-axis is given as $r_p=\frac{{\mathrm{B}}_{\mathrm{y}}}{y_{res}}=\frac{10}{300}=0.033$ where $r_p$ is re-calculated for the tangent of the half-angle under which you see the object which given as $r_p=r_p·tan\left(\frac{vfov}{2}\right)=r·tan\left(\frac{60}{2}\right)=0.019$. The distance d to the camera is estimated as $d=\frac{l}{2\ast r}=\frac{0.8}{2\ast 0.019}=21m$. The victim will appropriately be located if its distance separation from the camera is within the camera depth operational range.

Algorithm 1 Vision-Based Victim Localization Scheme

Input: $\mathcal{I}$—Input RGB Image $\mathcal{P}$—Point Cloud synchronized with the RGB Image $Camer{a}_{FOV}$—Camera Field of View ${\mathcal{M}}_C$—Current Occupancy Map for the Vision Detection Output: ${\mathcal{M}}_U$—Updated Occupancy Map for the Vision Detection   1: $\mathcal{L}\leftarrow \varnothing$                ▹ initialize $\mathcal{L}$ as empty vector to hold victim locations   2: $\left\{\mathbf{B}\right(C),k\}\leftarrow DetectHuman\left(\mathcal{I}\right)$   ▹ where $\mathbf{B}$ is a matrix containing Box coordinates, $C=[c_1,c_2,c_3,c_4]$ represents the box coordinates,   and $\mathit{k}$ is a vector containing the human corresponding probabilities   3 for i=1:length($k$) do   4:     if ${\mathit{k}}_{\mathit{i}}>threshold$ then   5:         ${\mathcal{P}}_e\leftarrow ExtractPointCloud\left(\mathbf{B}\left(c\right)\right)$   6:         $\mathbf{C}\leftarrow EuclideanClusterExtraction\left({\mathcal{P}}_e\right)$   7:         if $\mathbf{C}$ is empty then   8:            $\mathbf{continue}$   9:         end if 10:         sort $\left\{\mathbf{C}\right\}$ in ascending order 11:         ${\mathbf{C}}_{major}\leftarrow {\mathbf{C}}_{\mathbf{0}}$ 12:         $P\leftarrow FindCentroid\left({\mathbf{C}}_{major}\right)$ 13:         $P_{2D}\leftarrow$ project P in x-y plane 14:         $\mathcal{L}\leftarrow Append(\mathcal{L},P_{2D})$          ▹ append $P_{2D}$ to the list of victim locations $\mathcal{L}$ 15:     end if 16: end for 17: for all cells $c\in {\mathcal{M}}_C\&c\in Camer{a}_{FOV}$ do 18:     for all $h_t$ do 19:         $bel\left(h_t\right)\leftarrow c$ 20:         $\overline{bel}\left(h_t\right)={\displaystyle \sum _{h_t-1}}p\left(h_t\right|u_t,h_{t-1})·bel\left(h_{t-1}\right)$ 21:         $bel\left(h_t\right)=\eta ·p\left(d_t\right|h_t)·\overline{bel}\left(h_t\right)$ 22:     end for 23:     $c\leftarrow bel\left(h_t\right)$ 24:     ${\mathcal{M}}_U\leftarrow UpdateMap({\mathcal{M}}_C,c)$ 25: end for

3.1.2. Thermal-Based Victim Localization

The procedure for thermal-based victim localization summarized in Algorithm 2 involves two stages: victim detection, where the victim is found in the thermal image based on heat signature; and thermal occupancy mapping; where detection confidences obtained from the thermal detector is stored.

• Victim Detection: Thermal detection is used to detect human heat signature presented using the infrared light which is part of the electromagnetic (EM) spectrum. This approach is useful when the human cannot be detected in RGB images, especially, in dark lighting condition. The method adopted here is a simple blob detector which locates blobs in a thermal image for a temperature range of ${34}^{\circ }$ to ${38}^{\circ }$ (normal human temperature). The procedures for thermal detection is presented in Figure 4 which is composed of three stages. In the first stage, the thermal images are converted into mono-image. Then, two thresholds, $T_{min}={34}^{\circ }$ and $T_{max}={38}^{\circ }$ are applied to the mono-image to get a binary image that correspond $T_{min}<T<T_{max}$. After that, contouring is done to extract regions which represented the human thermal location. The blob detector was implemented using OpenCV where the minimum blob area is set to 20 pixels and a minimum adjacent distance between blob is set to 25 pixels. A human face was detected as shown in Figure 4. A full-human body can also be detected using the thermal detector by relaxing the choice of $T_{min}$ and $T_{max}$.
• Victim Mapping: After a box is resolved on the human thermal image, the center of the box ${\mathcal{B}}_c$ is calculated in the image frame give as

  $${\mathcal{B}}_c={[(c_{x,max}-c_{x,min})/2,(c_{y,max}-c_{y,min})/2]}^{\mathrm{T}}$$

  (5)

  Then a ray is broadcast from the thermal camera center through the obtained ${\mathcal{B}}_c$ pixel. After identifying the corresponding 3D ray in the world frame, orthogonal projection is deployed as $(P_x,P_y,P_z)\stackrel{}{\to }(P_x,P_y)$ where the $P_z$ is removed. The final location is stored as $P_{proj}={[Px,Py]}^T$.

The thermal map is updated when using 2D rays by assigning $P\left(D\right|h)$ to 0.6 for the cells corresponding to the 2-D ray and 0.4 for the other cells within the camera FOV. Then $P\left(D\right|h)$ is obtained by finding the belief distribution $bel\left(h_t\right)$ at time t using the previous $bel\left(h_{t-1}\right)$ where $\left(h_t\right)=\eta P\left(D_t\right|h_t)\overline{bel}\left(h_t\right)$ and $\overline{bel}\left(h_t\right)=\int P\left(h_t\right|u_t,h_{t-1})bel\left(x_{t-1}\right)d_{h_t-1}$. Both control $u_t$ and the distribution of the state $h_{t1}$ are used to determine the distribution of the current state $h_t$. This step is known as the prediction. When the state space is finite, the integral in $\overline{bel}\left(h_t\right)$ results in a finite sum. The $\eta$ used here for normalization, to make sure that the resulting product is a probability function whose integral is equal to 1. The map resolution is set to 0.2 m as the thermal-based victim localization is a bearing-only localization and required multiple ray generations of different poses to sufficiently locate a victim as shown in Figure 5.

Algorithm 2 Thermal-Based Victim Localization Scheme

Input: $\mathcal{I}$—Input Thermal Image $Camer{a}_{FOV}$—Camera Field of View ${\mathcal{M}}_C$—Current Occupancy Map for the Thermal Detection Output: ${\mathcal{M}}_U$—Updated Occupancy Map for the Thermal Detection   1: $\mathcal{R}\leftarrow \varnothing$                ▹ initialize $\mathcal{R}$ as empty vector to hold the rays   2: ${\mathcal{I}}_{mono}\leftarrow ConvertToMono\left(\mathcal{I}\right)$   3: for all pixels p∈${\mathcal{I}}_{mono}$ do   4:     if ${\mathrm{T}}_{min}<p<{\mathrm{T}}_{max}$ then   5:         ${\mathcal{I}}_{th}\left(p\right)=1$   6:     else   7:         ${\mathcal{I}}_{th}\left(p\right)=0$   8:     end if   9: end for 10: $\mathbf{B}\left(C\right)\leftarrow BlobDetection\left({\mathbf{I}}_{threshold}\right)$   ▹ where $\mathbf{B}$ is a matrix containing detected Box coordinates in $\mathcal{I},C=[c_1,c_2,c_3,c_4]$ represents the   box coordinates 11: if $\mathcal{B}\left(c\right)$ is not empty then 12:     $i=0$ 13:     for all boxes c in $\mathbf{B}$ do 14:         generate 2-D line, $\mathcal{L}$, that pass through the center of box c and terminate at the end of   $Camer{a}_{FOV}$ 15:         $\mathcal{R}\leftarrow Append(\mathcal{R},\mathcal{L})$                ▹ append $\mathcal{L}$ to the list of rays $\mathcal{R}$ 16:     end for 17: end if 18: for all cells $c\in M_C\&c\in Camer{a}_{FOV}$ do 19:     for all $h_t$ do 20:         $bel\left(h_t\right)\leftarrow c$ 21:         $\overline{bel}\left(h_t\right)={\displaystyle \sum _{h_t-1}}p\left(h_t\right|u_t,h_{t-1})·bel\left(h_{t-1}\right)$ 22:         $bel\left(h_t\right)=\eta ·p\left(d_t\right|h_t)·\overline{bel}\left(h_t\right)$ 23:     end for 24:     $c\leftarrow bel\left(h_t\right)$ 25:     ${\mathcal{M}}_U\leftarrow UpdateMap({\mathcal{M}}_C,c)$ 26: end for

3.1.3. Wireless-Based Victim Localization

In the literature, many various techniques are used to find the distance or the angle to localize an unknown static node which in this work is assumed to be the cell-phone of the victim. One of the popular methods for the ranging techniques is the Received Signal Strength (RSS) which provides a distance indication to the unknown node. The robot will act as a moving anchor node and captures multiples RSS readings with the aim of locating the victim. The proposed system assumes the transmission power of victim wireless device is fixed and known. This section explains the log-normal channel model which adopted in the wireless detection module. Also, the updating procedure for the captured measurement is explained to speed up the victim search.

• Victim Detection: A victim can be detected wirelessly by monitoring a given transmitted signal from the victim phone. In this configuration, the victim phone can be treated as a transmitter while the wireless receiver can be placed on a robot. If no obstacle between the transmitter and receiver is found, the free-space propagation model is used to predict the Received Signal Strength in the line-of-sight (LOS) [41]. The RSS equation is derived from Friis Transmission formula [42]

  $$P_r=P_t{G}_t{G}_r{\displaystyle \frac{{\lambda }^2}{{(4\pi \times d_{ij})}^2}}$$

  (6)

  where $P_r$ is the Received Signal Strength from receiver node i at node transmitter node j at time t, $P_t$ is the transmitted power, $G_t$ is the transmitter gain, $G_r$ is the receiver gain, $d_{ij}$ is the distance from sensor node i at node j, and $\lambda$ is the wavelength of the signal. From Equation (6), the received power $P_r$ attenuates exponentially with the distance $d_{ij}$ [41]. The free-space path loss $P{L}_F$ is derived from the equation above by 10 log the ratio of the transmitted power $P_t$ to the received power $P_r$ and setting $G_t$ = $G_r$ = 1 because, in most of the cases, the used antennas are isotropic antennas, which radiate equally in all direction, giving constant radiation pattern [41]. Thus, the formula is the following:

  $$P{L}_F\left(d_{ij}\right)\left[dB\right]=10log\left(\frac{P_t}{P_r}\right)=20log\left(\frac{4\pi \times d_{ij}}{\lambda }\right)$$

  (7)
  When using the free-space model, the average received signal decreases with the distance between the transmitter and receiver $d_{ij}$ in all the other real environments in a logarithmic way. Therefore, the path-loss model generalized form can be obtained by changing the free-space path loss with the path-loss exponent n depending on the environment, which is known as the log-distance path-loss model which will result in the following formula [41]:

  $$P{L}_{LD}\left(d_{ij}\right)\left[dB\right]=P{L}_F\left(d_0\right)+10\eta log\left({\displaystyle \frac{d_{ij}}{d_o}}\right)$$

  (8)

  where $d_0$ is the reference distance at which the path loss inherits the characteristics of free space in (7) [41]. Every path between the transmitter and the receiver has different path loss since the environment characteristics changes as the location of the receiver changes. Moreover, the signal may not penetrate in the same way. For that reason, more realistic modeling of the transmission is assumed which is the log-normal modeling.

  $$PL\left(d_{ij}\right)\left[dB\right]=P{L}_F\left(d_0\right)+10\eta log\left({\displaystyle \frac{d_{ij}}{d_o}}\right)+X_{\sigma }$$

  (9)

  where $PL\left(d_{i}j\right)\left[dB\right]$ = $P_t\left[dB\right]$−$P_r\left[dB\right]$ is the path loss at distance $d_{ij}$ from the transmitter, $P{L}_F\left(d_0\right)$ is the path-loss model at the reference distance $d_0$ which is constant. $X_{\sigma }$ is Gaussian random variable with a zero mean and standard deviation $\sigma$ [43].
  Due to the presence of noise in the log-normal model, the path loss $PL\left(d_{ij}\right)$ will be different even the same location with distance $d_{ij}$. To obtain a relatively stable reading, $PL\left(d_{ij}\right)$ is recording for K samples, then, an averaging process is done to get the measured $PL\left(d_{ij}\right)$. The victim can be set to be identified wireless if the detected distance is less than a specific distance threshold $\mathsf{\Delta }{d}_t$. This is because RSS reading is high when the transmitter is closer to the receiver leading to a less noise variance. Hence, from a log-normal model, it is sufficient to state that the victim is found in the measured distance $d_{ij}$ satisfy the condition $d_{ij}<\mathsf{\Delta }{d}_t$
  The same idea is used in weighted least-square approach where high weights are given for low distances leading to a less RMSE in the localized node when compared to the conventional lease-square approach which assumes constant noise variance across all measured distances [44].
• Victim Mapping: When RSS is used to locate an unknown node, a single measured distance is not sufficient because the unknown node can be anywhere over a circle with radius equal to the distance. That can be solved using trilateration as shown in Figure 6. In 2D space, trilateration requires at least three-distance measurements from anchor nodes. The location of the unknown node is the intersection of the three circles as show in Figure 6 [45].
  Using trilateration can lead to a problem in case all the measured distances are large (with high noise variance), which can lead to a false located position. To alleviate this problem, the use of occupancy grid is proposed when updating the map. The measured distance is compared based on the criteria $d_{ij}<\mathsf{\Delta }{d}_t$. If this criteria it met, the measurement is trusted, and the map is updated with victim probability within a circle of radius equal to $d_{ij}$. Otherwise, the measured distance is discarded. The updating process is shown in Figure 7.

3.1.4. Multi-Sensor Occupancy Grid Map Merging

In this work, three different victim-location maps are generated using three different sensors. The probability of the victim’s presence in each map is obtained by the Bayesian-probabilistic framework independently. These maps are then fused in a Bayesian-probabilistic framework to obtain a merged map that indicates human presence probability. Following the Bayesian approach in generating the multi-sensor occupancy grid may introduce undesired errors. For example, some victim detectors may have less confidence compared to others due to sensor limitations. For this reason, multiple sensors with different weights can be used for each victim detector. Let D represents a victim detector and $C_i$ is the confidence assigned for each detector. Each of the victim detectors gives two quantities, which are the measured probability $P\left(C_i\right|Y_j)$ of finding a human given detector D and the weight $w_i\left(C\right)$ that reflects how much confidence assigned to a victim detector. The posterior probability is given as:

$$P\left(C|D_1\dots D_m\right)=\alpha \sum _{i=1}^m{w}_i\left(C\right)P\left(C\right|D_i)$$

(10)

where $\alpha ={\left[{\displaystyle \sum _{i=1}^m}{w}_i\left(C\right)\right]}^{-1}$ is a normalized factor for all the weights. Figure 8 shows the process of updating a single cell in the combined map taken into consideration the different weights for the victim modules implemented in each map. Let $Ma{p}_{merged}$ be the combined map from the three sensors. The merged map from the three sensors is used to monitor the maximum probability. The merged map is obtained by re-scaling the three occupancy maps to the smallest resolution $resolutio{n}_{min}$ which is the lowest resolutions of the three used maps. The combined map is given as

$$Ma{p}_{merged}=\alpha \sum _{i=1}^m{w}_i\left(C\right)\ast Ma{p}_{m,rescaled}$$

(11)

3.2. Exploration

We adapt the NBV approach to explore the environment. The typical pipeline for NBV includes viewpoint sampling, viewpoint evaluation, navigation and termination. In this work, two viewpoint sampling methods are used for NBV evaluation and viewpoint selection which are the Regular Grid Sampling Approach (RGSA) and a proposed Adaptive Grid Sampling Approach (AGSA). After that, the different sample viewpoint are evaluated using a utility function. Then, the robot navigates to the selected viewpoint and update the maps simultaneously. The search mission is terminated when a termination condition is met.

In this work, the search process terminates when a victim is found or when all the viewpoints provide a zero reward from the utility. The victim is assumed to be found in the maximum probability if the occupancy map went above a specific threshold or if the number of iteration exceeded a pre-defined number of iterations.

The proposed NBV approach uses a multi-objective utility function which is composed of three components. The first function $U_{exp}$ is the information gain, which focuses on the exploration objective while the function $U_{victim}$ is biased towards exploring occupancy grids with higher victim presence probability. Traveling further distances is penalized by the function $U_{dist}$ to compromise between satisfying the objectives and the traveling cost. The multi-objective utility defined as:

$$\begin{array}{cc}\hfill U& =\left(w_{exp}\times U_{exp}+w_{victim}\times U_{victim}\right)\times U_{dist}\hfill \\ & =\left(w_{exp}\times \left(\frac{H_{IG}}{\underset{}{max}\left\{H_{IG}\right\}}\right)+w_{victim}\times e^{-|\zeta {|}^{\beta }}\right)\times e^{-w_{dist}\ast d}\hfill \end{array}$$

(12)

where $\zeta =1-\frac{\underset{}{max}\left\{P_{MAX}\right\}}{\underset{}{max}\left\{P\right\}}$, P is the probability of a cell in a viewpoint, $P_{MAX}$ is the maximum probability cell within a given viewpoint, $H_{IG}$ is the information gain (IG) for a given viewpoint, d is the distance from the current location to the evaluated viewpoint, $\beta$ is the exponential penalization parameter while $w_{exp}$, $w_{victim}$ and $w_{dist}$ are the weights used to control each of the three objectives. The impact of $\beta$ on the selectivity of $U_{victim}$ is illustrated in Figure 9, used to control how much emphasis should be put on victim detection vs environment exploration.

After viewpoint evaluation, different navigation schemes are used based on the selected view generation method. For Regular Grid Sampling approach, a collision-checking is performed over the line between the vehicle and the evaluated viewpoint, to ensure that the path is safe for the drone to traverse. For adaptive sampling, initially, a straight line is performed. In case it failed, then an A* [46] planner is used.

Upon finding a victim, the search process is terminated to allow the robot to report back the victim location to the rescue team. The victim is assumed to be found in the maximum probability if the occupancy map went above a specific threshold or if the number of iteration exceeded a pre-defined number of iterations. This method can also be extended to incorporate more than one victim.

4. Experimental Results

Simulation experiments were performed on an Asus-ROG-STRIX Laptop (Intel Core i7-6700HQ @ 2.60 GHz × 8, 15.6 GB RAM). The victim localization system was implemented on Ubuntu-kinetic 16.04 using the Robot Operating System (ROS) [47], where the system consists of a group of nodes with data sharing capability. Hence, the use of ROS simply the direct implementation of the system on the hardware platform. The environment was simulated using Gazebo, and the code programmed in both C++ and Python. The GridMap library [48] was used to represent the 2-D occupancy grid, navigation was done using Octomap [49] and point cloud data (PCL) was handled using the Point Cloud Library (PCL) [50]. An open source implementation of the proposed approach can be found at [51] for further use.

4.1. Vehicle Model and Environment

The adopted vehicle model in the tests is based on a floating sensor approach, where the sensor moves virtually in space and captures data from the selected viewpoints. The floating sensor is presented as a virtual common base link among candidate sensors. Static transformations are used to refer the sensors links to the floating sensor base link. Upon moving the floating sensor, the way-point command is applied in the floating sensor local frame where the sensors take their new positions based on their respective transformations. Figure 10 shows the floating sensor along with its components. By using the floating sensor approach we skip continuous data that could be acquired during the transition between the viewpoints, resulting in a better evaluating setup where only data and information gathered from the viewpoints is used to conduct the exploration and victim detection. The system is equipped with three sensors, namely an RGB-D camera, a thermal camera, and a wireless adapter. The specifications of the RGB-D camera, thermal camera, and wirelesses transmitter specifications are presented in

Table 1. RGB camera, Thermal camera, and Wireless Sensor specifications.

Specs	RGB-D	Thermal	Specs	Wireless
HFOV	80	60	Type	isotropic
VFOV	60	43	N of samples	150
Resolution	640 × 480 px	160 × 120 px	frequency	${10}^9$ Hz
Range	0.5–5.0 m	N/A	SNR	5 dB
Position y	−0.05 m	−0.09 m	Position	0.05 m
Pitch	15	15	-	-

. Both RGB-D and thermal camera are mounted underneath to increase the unobstructed field of view (FOV). The proposed system can adapt to any UAVs.

The simulated environment is shown in Figure 11 with dimensions 13 m × 15 m. This environment has multiple rooms with dead-ends and a corridor which can demonstrate the problems that the system can experiences and how to be tackled. Due to the difficulty in simulating an actual heat signature in the environment, the output of the thermal blob detector is simulated by the detection of the victim red-shirt from the captured RGB image.

4.2. Tests Scenario and Parameters

The tests are carried out to show the effectiveness of the proposed approach, namely the combined map, viewpoint sampling method, and the multi-objective utility function. The executed tests are divided into two classes based on the sampling approaches, Regular Grid Sampling Approach (RGSA) and the proposed Adaptive Grid Sampling Approach (AGSA). The proposed AGSA is compared with the state-of-the-art RGSA [30,31]. In each sampling approach, the proposed merged map including the wireless map, is compared with the state-of-the-art single generated maps using single sensor. In addition, the proposed utility (${\mathcal{U}}_3$) function is compared with the state-of-the-art utility functions (${\mathcal{U}}_1$) and (${\mathcal{U}}_2$), weighted information gain [12] obtained using Equation (3) and max probability [2] obtained using Equation (4), respectively. The evaluation metrics used for each test are the number of iteration, distance traveled, and entropy reduction. The parameters used for RGSA and AGSA are shown in

Table 2. Regular Grid Sampling Approach Parameters.

Regular Grid Sampling Parameters	Value
x displacement between samples	0.8
y displacement between samples	0.8
yaw step-size	$\frac{\pi }{4}$
Straight-line Collision Check Box size	0.8

and

Table 3. Adaptive Grid Sampling Approach Parameters.

Adaptive Grid Sampling Parameters	Value
Adaptive scale factor base a	2
Entropy change per cell $\mathsf{\Delta }{E}_v$	0.1 m
$A\ast$ grid resolution	0.3 m
$A\ast$ circular distance to goal	0.3 m
$A\ast$ circular distance to robot in case of no-path found	0.5 m

, respectively.

In each test, the four different maps: vision, thermal, wireless, and merged map were used to test their performance. The viewpoint evaluation is executed using the three utilities: weighted information gain (${\mathcal{U}}_1$), improved max utility (${\mathcal{U}}_2$), and the proposed multi-objective utility (${\mathcal{U}}_3$).

Table 4. Parameters used in the simulation.

Vision Map Parameters	Value		Thermal Map Parameters	Value
map resolution	0.5		map resolution	0.2
$P\left(D\right|H)$	0.8		$P\left(D\right|H)$	0.7
$P\left(D\right|\overline{H})$	0.1		$P\left(D\right|\overline{H})$	0.3
$P\left(\overline{D}\right|H)$	0.2		$P\left(\overline{D}\right|H)$	0.3
$P\left(\overline{D}\right|\overline{H})$	0.9		$P\left(\overline{D}\right|\overline{H})$	0.7
SSD probability threshold	0.8
minimum cluster size	10 points		minimum cluster size	40 pixels
minimum distance between clusters	0.01 m		minimum distance between clusters	30 pixels
Wireless Map Parameters	Value		Merged Map Parameters	Value
map resolution	0.5		map resolution	0.5
$P\left(D\right|H)$	0.7		vision map weight $w_1\left(C\right)$	0.65
$P\left(D\right|\overline{H})$	0.3		thermal map weighted $w_2\left(C\right)$	0.2
$P\left(\overline{D}\right|H)$	0.3		wireless map weighted $w_3\left(C\right)$	0.15
$P\left(\overline{D}\right|\overline{H})$	0.7
wireless distance threshold	5 m

lists the parameters used in the map generation.

The Octomap resolution is set to 0.2 m. A termination condition of maximum iteration is used in all the tests where the maximum iteration $n_{max}$ is set to 120, and the victim probability threshold $P_{max}$ is set to 0.9 (the probability at which a victim is considered found). For the utilities, The distance weight $\lambda$ is set to 0.05, while for the proposed utilities, the weights are selected as $w_{\exp }=1$ and $w_{\mathrm{victim}}=2$ when evaluated in the vision and thermal map. For the wireless map, it is assigned as $w_{\exp }=1$ and $w_{\mathrm{victim}}=1$. The penalization parameter $\beta$ is set to 1. The metrics used in the evaluation are the total number of iterations before termination (It), total map entropy before the victim search begin subtracted by total map entropy after the victim is found (TER), and the total distance traveled in meters (D). Some of the parameters used during the experimentation were carefully tuned to achieve the optimum setup given the experimental environment. The benchmarking, however, was conducted with the same parameters for all the methods.

4.3. Regular Grid Sampling Approach (RGSA) Results

The obtained results of the different utility functions are shown in Figure 12a in terms of entropy and Figure 12b in terms of distance traveled. For comparison, the same results listed in

Table 5. Results of RGSA tests. VF: victim Found It: Iterations, D: Distance, TER: Total Entropy Reduction. The numbers in bold indicate better performance.

Scenario	VF	It	D (m)	TER
${\mathcal{U}}_1$-vision	No	120	101.1	368.9
${\mathcal{U}}_2$-vision	Yes	81	72.4	382.2
${\mathcal{U}}_3$-vision	Yes	77	68.0	440.9
${\mathcal{U}}_1$-thermal	Yes	100	88.1	2494.4
${\mathcal{U}}_2$-thermal	Yes	104	91.71	2067.2
${\mathcal{U}}_3$-thermal	Yes	98	83.46	2395.0
${\mathcal{U}}_1$-wireless	No	120	75.59	732
${\mathcal{U}}_2$-wireless	No	120	85.17	472.1
${\mathcal{U}}_3$-wireless	No	120	76.20	824.2
${\mathcal{U}}_1$-merged	No	120	98.46	2810
${\mathcal{U}}_2$-merged	No	120	17.24	587
${\mathcal{U}}_3$-merged	Yes	73	64.34	2344.3

.

In the vision map, the proposed utility (${\mathcal{U}}_3$) has fewer iterations (77), entropy reduction, and distance traveled when compared to the other utilities within the same map. It is also noticeable that the weighed information gain (${\mathcal{U}}_1$) failed to find the victim as it exceeds the maximum iteration 120. This because of ${\mathcal{U}}_1$ experience a local minimum indicated by the stabilized entropy curve for an extended number of iterations. The max utility (${\mathcal{U}}_2$) succeeded in locating the victim but with a relatively higher number of iterations as this utility does not have an exploration capability.

In the thermal map, ${\mathcal{U}}_3$ has fewer iterations (98), and distance traveled when compared to the other utilities within the same map. Also, ${\mathcal{U}}_1$ has more iterations but a better entropy reduction compared to ${\mathcal{U}}_3$. ${\mathcal{U}}_2$ has the worst performance with a higher number of iterations and distance travel to locate the victim.

In the wireless map, all the three methods failed in locating the victim where the total iterations reached the maximum allowable iteration. That is due to the nature of the wireless transmitter wherein the map update is done in a circular manner making it easy for the robot to fall in a local minimum.

In the merged map, ${\mathcal{U}}_3$ has the best performance with a lower number of iterations (73), entropy reduction and distance traveled compared to the other utilities in the same map as well as across all maps. In ${\mathcal{U}}_1$, the method failed to locate the victim as the vehicle experience a set of local minima which was mainly inherited from the wireless component. The merged map with ${\mathcal{U}}_2$ failed because the utility falls in an inescapable local minimum. Figure 13 shows the path trajectory of the vehicle in ${\mathcal{U}}_3$-merged test using RGSA.

4.4. Adaptive Grid Sampling Approach (AGSA) Results

Here, the Adaptive Grid Sampling Approach (AGSA) is used to test the performance of the utilities when implemented on the individual maps as well as the merged map. The settings used in the AGSA are shown in Table 3. The obtained results of the different tests shown in Figure 14a in terms of entropy and Figure 14b in terms of distance traveled. Also, the instants when the AGSA detected local minimum as shown in Figure 15. For comparison, the same results listed in

Table 6. Results of AGSA tests. VF: victim Found It: Iterations, D: Distance, TER: Total Entropy Reduction. The numbers in bold indicate better performance.

Scenario	VF	It	D (m)	TER
${\mathcal{U}}_1$-vision	No	84	78.4	645.52
${\mathcal{U}}_2$-vision	Yes	81	72.4	718.06
${\mathcal{U}}_3$-vision	Yes	80	68.4	671.1
${\mathcal{U}}_1$-thermal	Yes	89	88.7	4608.2
${\mathcal{U}}_2$-thermal	Yes	104	91.7	4932.7
${\mathcal{U}}_3$-thermal	Yes	86	75.6	4752.1
${\mathcal{U}}_1$-wireless	No	120	223	490.8
${\mathcal{U}}_2$-wireless	No	120	115.4	245.8
${\mathcal{U}}_3$-wireless	No	120	223	490.8
${\mathcal{U}}_1$-merged	Yes	76	73.6	4459.4
${\mathcal{U}}_2$-merged	No	120	96.8	4959.8
${\mathcal{U}}_3$-merged	Yes	66	61.6	4772.7

.

In the vision map, the proposed utility (${\mathcal{U}}_3$) has fewer iterations to find the victim (80). The weighed information gain (${\mathcal{U}}_1$) succeeds due to the local minimum’s problem been mitigated by the AGSA. The max utility (${\mathcal{U}}_2$) has the same performance as in the case of RGSA. That is because the ${\mathcal{U}}_2$ works by selecting a viewpoint with maximum entropy, making it hard to fall under the threshold $\mathsf{\Delta }{E}_{v,threshold}$.

In the thermal map, ${\mathcal{U}}_3$ has fewer iterations to find the victim (86). In comparison to ${\mathcal{U}}_3$, ${\mathcal{U}}_1$ has a lower entropy reduction but more iterations to locate the victim. ${\mathcal{U}}_2$ has the worst performance compared to the rest of the method. Also, the performance ${\mathcal{U}}_2$ is the same as in the case of RGSA. In the wireless map, all the utilities failed, as they require more iterations to find the victim.

In the merged map, ${\mathcal{U}}_3$ has fewer iterations to find the victim (66). Unlike in RGSA, ${\mathcal{U}}_1$ succeeds in locating the victim as the vehicle was able to escape local minima. ${\mathcal{U}}_2$ failed in locating the victim as this method does not encourage the vehicle to explore upon initial detection of the victim. That can lead to a problem if the victim is located through the wireless sensor, but an obstacle such as a wall occludes the victim. Under this scenario, it is preferable to explore and found a line-of-sight with the victim allowing the vision and the thermal module to operate, a feature the max utility is lacking.

Across all the tests, ${\mathcal{U}}_3$-merged using ARGS has the best performance in terms of iterations and distance travel with a relatively better entropy reduction.

Figure 16 shows the traveled path of the vehicle in the ${\mathcal{U}}_3$-merged test using AGSA. A video that demonstrates the efficiency of indoor environment exploration using AGSA is provided in [52]. A video that represents a real-time experiment for victim detection is shown in [53], which was conducted to demonstrate the effectiveness of the proposed work.

4.5. Discussion Results

The best performance was obtained when merging the probabilistic information in an occupancy grid map, and using it to guide the search strategy. In this case, the combined information merged in a probabilistic Bayesian framework is used instead of local optima of each system individually.

Using the Adaptive Grid Sampling Approach (AGSA), the proposed merged map with the proposed utility function outperforms both traditional single-vision and thermal maps using the proposed utility function on the number of iteration and the traveled distance. The reduction in the number of iteration was in a range of 17.5% to 23.25% while the reduction in the traveled distance was in a range of 10% to 18.5% respectively.

In addition, using the Regular Grid Sampling Approach (RGSA), the proposed merged map with the proposed utility function outperforms both proposed single-vision map and single-thermal map using proposed utility function on the number of iteration and the traveled distance. The reduction in the number of iteration was in a range of 5.2% to 25.5% while the reduction in the traveled distance was in a range of 5.4% to 23%, respectively.

Hence, the merged map results in a moderate performance over the other single maps when using the Regular Grid Sampling Approach. However, combining the merged map with the Adaptive Grid Sampling Approach presents higher performance.

5. Conclusions

In this work we proposed an USAR exploration approach based on multi-sensor fusion that uses vision, thermal, and wireless sensors to generate a probabilistic 2D victim-location maps. A multi-objective utility function exploited the fused 2D victim-location map to generate and evaluate viewpoints in a NBV fashion. The evaluation process targets three desired objectives, specifically exploration, victim detection, and traveled distance. Additionally, an adaptive grid sampling algorithm (AGSA) was proposed to resolve the local minimum issue occurs in the Regular Grid Sampling Approach (RGSA). Experimental results showed that the merged map results in a moderate performance over the other single maps when using the Regular Grid Sampling Approach. However, combining the merged map with the Adaptive Grid Sampling Approach presents higher performance. In both cases, the results demonstrated that the proposed utility function using merged map performs better than methods that rely on a single sensor in the time it takes to detect the victim, distance traveled, and information gain reduction. The tests were conducted in a monitored environment yet there are different sources of uncertainty. In future work, we will extend the system to address these uncertainty with scalability, and conduct experimental tests in real-world scenarios.