Research and Conceptual Design of Sensor Fusion for Object Detection in Dense Smoke Environments

Abstract

In this paper, we propose a conceptual framework for a sensor fusion system that can detect objects in a dense smoke environment with a visibility of less than 1 m. Based on the review of several articles, we determined that by using a single thermal IR camera, a single Frequency-Modulated Continuous-Wave (FMCW) radar, and multiple ultrasonic sensors simultaneously, the system can overcome the challenges of detecting objects in dense smoke. The four detailed methods proposed are as follows: First, a 3D ultrasonic sensor system that detects the 3D position of an object at a short distance and is not affected by temperature change/gradient; Second, detecting and classifying objects such as walls, stairs, and other obstacles using a thermal IR camera; Third, a 2D radial distance measurement method for a distant object using an FMCW radar; Fourth, sensor fusion for 3D position visualization of multiple objects using a thermal IR camera, 3D ultrasonic sensor system, and FMCW radar. Finally, a conceptual design is presented based on the proposed methodologies, and their theoretical usefulness is discussed. The framework is intended to motivate future research on the development of a sensor fusion system for object detection in dense smoke environments.

1. Introduction

According to several reports [1,2,3,4,5], about 60% of fire accidents occur in indoor environments, and more than 70% of firefighter deaths and injuries are caused by smoke inhalation, burns, and being trapped in indoor fires. In particular, extremely poor visibility from dense smoke triggers mental panic in firefighters and limits even their normal behavior [6,7,8,9]. Typically, the visibility in dense smoke is less than 1 m. Therefore, these facts motivated research on visual aid systems based on a sensor fusion that can provide useful visualization information to firefighters, as shown in Figure 1.

Various robotic and visual assistance platforms have been developed and studied to enable firefighters to successfully perform their missions in these situations [10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28]. Most of these studies on firefighting robots and firefighter visual aids have utilized visual and IR night vision cameras for perception sensors. Due to their short wavelength, these sensors are not suitable for use in dense smoke environments. In addition, the camera is not suitable for the high temperatures of dense smoke. Therefore, in those studies, a temperature sensor was used to compensate for these shortcomings.

Dense smoke contains various combustion gases such as CO₂, CO, H₂S, SO₂, NH₃, HCN, COCl₂, and HCl. In addition, because these gases combine with water vapor, the absorption, reflection, and scattering properties of radiant energy in dense smoke differ from normal environments, and they are also very different from the characteristics of white smoke. The dense smoke also creates a high temperature gradient in an indoor air space. In these conditions, a sensor with a longer wavelength is advantageous for environmental perception. Therefore, thermal IR cameras, FMCW radars, and ultrasonic sensors are more suitable for use in dense smoke environments than visual and/or IR night vision cameras because of their longer wavelengths, as shown in Figure 2 [29].

For these reasons, several studies have attempted to detect objects in dense smoke environments by combining thermal IR cameras, FMCW radars, or ultrasonic sensors, individually or in combination with other general environmental sensors, as shown in

Table 1. Research on detecting objects in dense smoke and its limitations.

Studies	Used Sensors	Limitations	Advantages
[11]	Ultrasonic sensor	- Impossible to measure 3D position - Increase of measurement error with respect to temperature variation/gradient	- Very simple H/W and S/W implementation
[12,13]	Single thermal IR camera + Lidar + UV sensor + normal camera	- Impossible to measure distance due to using only single camera (Compensated by other sensors) - Other sensors: inappropriate to use in dense smoke	- Powerful visualization in white smoke and normal environment
[14,15,21]	Stereo thermal IR camera	- Ghost effect due to stereo misalignments - Full-resolution image processing time taken is long - Object detection failure due to high-temperature thermal saturation	- Possible to use simple sensor fusion algorithm because a single type of sensor is used
[27]	Stereo thermal IR camera + Radar	- Same to above - The longer distance, the bigger the measurement error - Long-distance accuracy increased by radar - Continued short-range object detection failure	- Possible to use relatively simple sensor fusion algorithm
[28]	RGB camera + Radar + Stereo thermal IR camera	- The same problems to [14,15,21,27] - RGB camera utilized for white low-temperature smoke	- Powerful visualization in white smoke and normal environment

. However, these studies are insufficient for firefighting support due to the limitations and characteristics given in Table 1 and Figure 3 [30].

Additionally, although there are several studies on sensor fusion for firefighting robots [31,32,33,34,35], these studies are related not to object detection in dense smoke environments but to flame detection or navigation in a general fire environment. In other words, studies on sensor fusion for object detection in dense smoke environments are very rare.

As an alternative to existing studies, we propose the novel concept of a sensor fusion system that can provide 3D position information on objects with a visibility of less than 1 m. This system uses a single thermal IR camera, a single FMCW radar, and multiple ultrasonic sensors simultaneously to maximize the strengths of each sensor, as shown in Figure 2, as well as compensate for its weaknesses. Based on this sensor fusion, this paper contributes to enabling new perspectives and methodologies that can detect and visualize indistinguishable objects in dense smoke environments as follows: (1) 3D global position estimation for multiple objects through two types of 2D information and one type of 3D information, (2) application of image transformation theory to general data information transformation, (3) 3D position estimation of an object using 1D ultrasonic sensor information, (4) extension of the directional angle and detection area of the ultrasonic sensor system, (5) rejection of measurement error from ultrasonic sensors caused by temperature change in dense smoke, (6) object detection and classification through image processing from a thermal IR camera. As a result, the system presented in this paper can provide visualized information about objects in a dense smoke environment to firefighters or firefighting robots, as shown in Figure 1.

The remainder of this paper is organized as follows. In Section 2, the theoretical methodologies of proposed sensor fusion are presented for object detection using ultrasonic sensors, FMCW radar, and thermal IR cameras. In Section 3, we describe the conceptual H/W and S/W design to implement the proposed methodologies. Finally, in Section 4, we draw conclusions and give an outlook to future works.

2. Theoretical Methodology for Object Detection Using Ultrasonic Sensors, FMCW Radar, and Thermal IR Camera

2.1. Principle of 3D Position Measurement Using Ultrasonic Sensors: 3D Ultrasonic Sensor Module

Figure 4 is the configuration diagram of ultrasonic sensors, called the “3D ultrasonic sensor module” in this paper, for measuring a position of an object in a three-dimensional space. Let d₁, d₂, d₃, and d₄ be the measured distances from the sensors located at (a, 0, 0), (0, 0, 0), (0, b, 0), and (0, −b, 0) in the coordinate system (x, y, z) to an object, respectively. The distance d_* from the ultrasonic sensor with the notation number * is measured as follows

$$d_{*}=\frac{c_t\Delta t_{*}}{2}, c_t=331.3+0.606T [m/s]$$

(1)

where c_t is the speed of sound, Δt_* is the ultrasonic propagation time, called Time-of-Flight (TOF), from the sensor with the designation number * to the object, and T is the ambient temperature. Using Equation (1) and geometric information, the three-dimensional position (p_x, p_y, p_z) of the object using the ultrasonic sensor can be derived as follows

$$\begin{array}{l}{p}_x=\frac{d_2^2-d_1^2}{2a}+\frac{a}{2}\\ p_y=\frac{d_2^2-d_3^2}{2b}+\frac{b}{2}\\ p_z=\sqrt{d_2^2-p_x^2-p_y^2}\end{array}$$

(2)

2.1.1. Eliminate Measurement Errors Due to Temperature Changes

Note that the c_t is a function of T in Equation (1), and the temperature of dense smoke is generally very high and non-uniform or has a temperature gradient, as described in the introduction. Therefore, the three-dimensional position of an object derived by Equation (2) cannot be accurately measured in a dense smoke environment.

To avoid inaccuracies due to the temperature effects of ultrasonic sensors, we define the coefficient κ of the TOF ratio [36] as

$$\begin{array}{l}{\kappa }_1=\frac{\Delta t_1}{\Delta t_2}=\frac{d_1}{d_2}\\ {\kappa }_3=\frac{\Delta t_3}{\Delta t_2}=\frac{d_3}{d_2}\\ {\kappa }_4=\frac{\Delta t_4}{\Delta t_2}=\frac{d_4}{d_2}\end{array}$$

(3)

At this point, the measured κ₁, κ₂, and κ₃ are related to the unknown position (p_x, p_y, p_z) of the object, as shown below

$$\begin{array}{l}\left({\kappa }_1^2-1\right)\left(p_x^2+p_y^2+p_z^2\right)=-2a{p}_x+a^2\\ \left({\kappa }_3^2-1\right)\left(p_x^2+p_y^2+p_z^2\right)=-2b{p}_y+b^2\\ \left({\kappa }_4^2-1\right)\left(p_x^2+p_y^2+p_z^2\right)=2b{p}_y+b^2\end{array}$$

(4)

Thus, the temperature independent (p_x, p_y, p_z) can be derived as

$$\begin{array}{l}{p}_x=\frac{4b^2\alpha \beta -2\left(\beta +1\right)a^2}{-4a\left(\beta +1\right)}\\ p_y=\frac{b\left(1-\beta \right)}{2\left(\beta +1\right)}\\ p_z=\left\{\begin{array}{l}\sqrt{\frac{-2a{p}_x+a^2-\left({\kappa }_1^2-1\right)\left(p_x^2+p_y^2\right)}{{\kappa }_1^2-1}}, \mathrm{if} {\kappa }_1\ne 1\\ \sqrt{\frac{-2b{p}_y+b^2-\left({\kappa }_3^2-1\right)\left(p_x^2+p_y^2\right)}{{\kappa }_3^2-1}}, \mathrm{if} {\kappa }_3\ne 1\\ \sqrt{\frac{2b{p}_y+b^2-\left({\kappa }_4^2-1\right)\left(p_x^2+p_y^2\right)}{{\kappa }_4^2-1}}, \mathrm{if} {\kappa }_4\ne 1\end{array}\right.\end{array}$$

(5)

where α and β are defined as

$$\alpha =\frac{{\kappa }_1^2-1}{{\kappa }_3^2-1}, \beta =\frac{{\kappa }_3^2-1}{{\kappa }_4^2-1}$$

(6)

2.1.2. Extension of Directional Angle of View and Detection Area: 3D Ultrasonic Sensor System

In the 3D ultrasonic sensor module of Figure 4, the angle of view and the detection area exist in only one direction, as shown in Figure 5a,b due to the characteristics of the ultrasonic sensor. This disadvantage can be improved upon by adding additional sensor modules on top, bottom, left, and right, as shown in Figure 5c,d, called the “3D ultrasonic sensor system” in this paper.

In Figure 5, each detection area from #1 to #5 has its own local coordinate frame, from {1} to {5}, respectively. In these frames, the z-axes represent directional angles of view at the corresponding detection areas. Each sensor module measures the relative position of an object based on its own local coordinate frame. For example, the object in detection area #2 is at (p_x2, p_y2, p_z2) of frame {2} and the object in detection area #3 is at (p_x3, p_y3, p_z3) of frame {3}.

Therefore, all local positions have to be converted to global positions (p_xs, p_ys, p_zs) of global coordinate frame {S} in Figure 5. This transformation utilizes a homogeneous transformation matrix. Let i = 1, 2,⋯, 5, then the homogeneous transformation matrix $T_s^i$ from frame {S} to {i} is derived as

$$\begin{array}{l}{T}_s^1=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right],\\ T_s^2=\left[\begin{array}{cccc}-\cos \theta & 0& -\sin \theta & -\frac{l}{2}\left(1+\cos \theta \right)\\ 0& -1& 0& 0\\ -\sin \theta & 0& \cos \theta & -\frac{l}{2}\sin \theta \\ 0& 0& 0& 1\end{array}\right], T_s^3=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& \cos \theta & \sin \theta & \frac{l}{2}\left(1+\cos \theta \right)\\ 0& -\sin \theta & \cos \theta & -\frac{l}{2}\sin \theta \\ 0& 0& 0& 1\end{array}\right]\\ T_s^4=\left[\begin{array}{cccc}\cos \theta & 0& \sin \theta & \frac{l}{2}\left(1+\cos \theta \right)\\ 0& 1& 0& 0\\ -\sin \theta & 0& \cos \theta & -\frac{l}{2}\sin \theta \\ 0& 0& 0& 1\end{array}\right], T_s^5=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& -\cos \theta & -\sin \theta & -\frac{l}{2}\left(1+\cos \theta \right)\\ 0& -\sin \theta & \cos \theta & -\frac{l}{2}\sin \theta \\ 0& 0& 0& 1\end{array}\right]\end{array}$$

(7)

Using Equation (7), a relative position P_i with respect to frame {i} can be transformed into a global position P_s with respect to frame {S} as

$$P_s=T_s^i{P}_i$$

(8)

where $P_s={\left[p_{xs} p_{ys} p_{zs} 1\right]}^T$ and $P_i={\left[p_{xi} p_{yi} p_{zi} 1\right]}^T$.

2.2. Thermal IR Image Processing: Object Detection

Most indoor fire sites are composed of typical structures such as walls, stairs, and floors, as well as atypical objects such as various pieces of furniture and broken or burnt objects. Therefore, thermal IR images taken at the sites show a mixture of all objects indoors.

Appropriate image processing needs to be applied to these images (called query images) to distinguish between objects that may be obstacles and those that are not. Figure 6 shows an image processing overview for this distinction. This processing is based on the object detection method using a basic Canny algorithm and Hough transformation.

After obtaining the query image from image sequences of the thermal IR camera, calibration and rectification are applied to the image for pre-processing of edge detection. The pre-processed image is utilized to detect edges using the Canny algorithm. We chose the Canny algorithm for the following reasons [37,38]:

①

The maximum number of edges can be detected.

②

It gives very good results for detecting horizontal and vertical edges.

③

Circular and corner edges can be detected.

④

Good localization.

⑤

Only one response to a single edge.

Edge images are used to establish obstacle boundaries and planes for walls, stairs, and floors. By using Hough transformation to extract line features from these edge images, we classify vertical and horizontal lines as features of walls, stairs, and floors. After this, for object detection, we collect groups of objects such as walls, stairs, and floors using the image correlation technique of template matching.

Image correlation using template matching is detailed in ref. [39]. The correlation is processed by calculating a correlation coefficient. The correlation coefficient is obtained by moving a filter mask over the query and template images and calculating the sum of the products at each location. The maximum value of the correlation coefficient occurs when regions of the template and the query images are identical. This is called the maximum correlation, i.e., the best possible match. On the other hand, minimum correlation occurs when the regions of the template and the query images are most dissimilar.

After this template matching, everything except walls, stairs, and floors is considered an object. From these objects, small objects are discarded, and large objects are identified as obstacles because they can interfere with the heading path of robots or firefighters. Finally, a display shows the thermal IR images with 2D locations (u, v) and the size box of an object (w, h) overlay.

Note that for better computational cost, we use a relatively simple object detection algorithm (template matching algorithm) rather than a complex one. To use a template matching algorithm, we assume that we have obtained prior (or reference) data by pre-scanning the designated fire hazard area (or the other normal conditions inside buildings) with various types of walls, stairs, and floors. In this case, the template matching algorithm can work more effectively because the performance of the algorithm depends on how much data can be obtained. Nevertheless, if prior data do not fit appropriately in an environment, a traditional method such as SIFT [40] or a deep learning model such as YOLO [41] can be used instead of a template matching algorithm because these algorithms can provide better performance. However, detailed discussions about these algorithms themselves are outside the scope of this paper.

2.3. Principle of 2D Radial Position Measurement Using FMCW Radar

We use an FMCW radar that outputs a radial distance R and angle θ. According to the radar principle, the following section presents how to measure the distance and angle.

2.3.1. Range Measurement for an Object Target

Figure 7a shows the basic operation of the FMCW radar system. The transceiver of the radar system transmits a frequency-modulated signal (f_TX) to an object, and then the receiver of the radar system obtains the reflected signal (f_RX) from the object. The transmitted and received chirp signal can be expressed as

$$f_{TX}=f_0+\frac{B}{t_{chirp}}$$

(9)

$$f_{RX}=f_0+\frac{B}{t_{chirp}}(t-\Delta t), \Delta t\le t\le t_{chirp}$$

(10)

where f₀, t_chirp, and B are the starting sweep frequency, frequency sweep time, and frequency sweep bandwidth of the transmitted signal, respectively. The time delay (Δt) caused by the round-trip of the transmitted signal up to reflection by the object is measured as

$$\Delta t=\frac{2R}{c_l}$$

(11)

where c_l is the speed of light in vacuum (3 × 10⁸ m/s).

After this, the intermediate frequency (Δf) can be generated using a mixture of the transmitted and received signals. Δf can be used to estimate the range of the object. Now, the distance R of the object is derived as

$$R=\frac{c_l{t}_{chirp}\Delta f}{2B}$$

(12)

If there are n number of multiple objects in front of the FMCW radar system, Equation (12) can be expressed as

$$R_i=\frac{c_l{t}_{chirp}\Delta f_i}{2B}, i=1,2,3,\dots ,n$$

(13)

2.3.2. Angle Measurement

As shown in Figure 7b, the angle θ, called the angle of arrival (AoA) of the reflected signal in the horizontal plane, can be estimated using at least two RX antennas in the FMCW radar system. The differential distance (ΔR) from the target object to each RX antenna mathematically causes a phase shift (Δφ) as

$$\Delta \phi =\frac{2\pi \Delta R}{\lambda }$$

(14)

where λ is the wavelength of the FMCW radar. The phase shift can also be measured from the FFT peak of two received signals. Assuming the system is constructed with a basic planar wavefront geometry, and the distance between the RX antennas is L, the differential distance (ΔR) is related to the angle θ as

$$\Delta R=L\sin \theta$$

(15)

Thus, the angle θ can be estimated from the measured Δφ, using

$$\theta ={\sin }^{-1}\left(\frac{\lambda \Delta \phi }{2\pi L}\right)$$

(16)

If there are n number of multiple objects in front of the FMCW radar system, Equation (16) can be expressed as

$${\theta }_i={\sin }^{-1}\left(\frac{\lambda \Delta {\phi }_i}{2\pi L}\right), i=1,2,3,\dots ,n$$

(17)

Note that due to the nonlinear dependence of Δφ and sin(θ), the angles that can be estimated with the two antennas are limited by the maximum viewing angle of

$${\theta }_{\max }={\sin }^{-1}\left(\frac{\lambda }{2L}\right)$$

(18)

2.4. Data Fusion

2.4.1. Overview

The proposed system consists of a 3D ultrasonic sensor system, a thermal IR camera system, and an FMCW radar system. This means that each piece of information from each sensor system is measured in its own coordinate system. Therefore, this measured information must be transformed into a single global coordinate or common plane for sensor fusion. Since the information measured in each of the three types of sensor systems can be regarded as image information taken from different perspectives of the same object, such a transformation can be accomplished with the calibration methods covered by image transformation theory. Therefore, for this transformation, we estimate the homography matrix shown in Figure 8.

Adopting the calibration method proposed in ref. [42,43], we simply estimate the homography describing the transformation between the image plane and the radar plane or ultrasound plane, as shown in Figure 8. By using the calibration results and data mapping to the common plane, the proposed sensor system can visualize the information of objects, such as size and position, measured by the FMCW radar and 3D ultrasonic sensor system in an image sequence.

2.4.2. Calibration between Thermal IR Camera and 3D Ultrasonic Sensor System or FMCW Radar System

As shown in Figure 8, (x_s, y_s, z_s), (x_r, y_r, z_r), and (x_c, y_c, z_c) are 3D ultrasonic sensor system coordinates, FMCW radar coordinates, and thermal IR camera coordinates. These coordinates are denoted as the coordinate frames {S}, {R}, and {C}, respectively, and (u, v) is the image plane coordinate system. Additionally, (p_xs, p_ys, p_zs) and (r_x, r_y, r_z) are the positions of near and far objects as measured by the 3D ultrasonic sensor system and the FMCW radar system, respectively. These positions should correspond to (u_s, v_s) and (u_r, v_r) detected by image processing of the thermal IR camera image for short- and long-range objects, respectively.

In order to map (p_xs, p_ys, p_zs) to (u_s, v_s), we use the transformation equation between {S} and (u, v) as

$${\omega }_s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=H\left[\begin{array}{c}{x}_s\\ y_s\\ z_s\end{array}\right]$$

(19)

where H = [h_ij]_{i,j = 1,2,3} is a 3 × 3 homography matrix and ω_s is an unknown constant. By estimating H, we determine the transformation between the ultrasonic plane (П_s) and the image plane (П_i). The ω_s is an unknown parameter, but we can assume that the objects are equidistant in two planes, so ω_s = z_s. This means that H is the affine homography matrix with h₃₂ = h₃₂ = 0 and h₃₃ = 1. Then, we obtain two algebraic equations:

$$\begin{array}{l}u=\frac{h_{11}}{z_s}{x}_s+\frac{h_{12}}{z_s}{y}_s+h_{13}\\ v=\frac{h_{21}}{z_s}{x}_s+\frac{h_{22}}{z_s}{y}_s+h_{23}\end{array}$$

(20)

Note that z_s is not a constant, but we can treat it as a constant on each image frame, especially for static objects. Thus, for Equation (20), we get the following homogeneous linear square problem:

$$\begin{array}{l}{A}_u{H}_s=0, A_u{H}_s=0\\ A_u=\left[\begin{array}{ccccccccc}{x}_s& y_s& 1& 0& 0& 0& 0& 0& -u\end{array}\right]\\ A_v=\left[\begin{array}{ccccccccc}0& 0& 0& x_s& y_s& 1& 0& 0& -v\end{array}\right]\\ H_s={\left[\begin{array}{ccccccccc}\frac{h_{11}}{z_s}& \frac{h_{12}}{z_s}& h_{13}& \frac{h_{21}}{z_s}& \frac{h_{22}}{z_s}& h_{23}& 0& 0& 1\end{array}\right]}^T\end{array}$$

(21)

Then, using Singular Value Decomposition (SVD), we estimate the homogeneous matrix, H, in Equation (19).

To match (r_x, r_y, r_z) to (u_r, v_r), we use the same approach described in Equations (19) to (21). The corresponding radius R and angle θ are converted to Cartesian coordinates using y_r = Rsinθ and z_r = Rcosθ. Additionally, since all FMCW radar data comes from somewhere in the radar plane (П_r), we can take x_r = 0. Hence, the transformation between {R} and (u, v) becomes

$${\omega }_r\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=M\left[\begin{array}{c}{y}_r\\ z_r\\ 1\end{array}\right]$$

(22)

where M = [m_ij]_{i,j = 1,2,3} is a 3 × 3 homography matrix and ω_r is an unknown constant. By estimating the M, we determine the transformation between the radar plane (П_r) and the image plane (П_i). We simplify m₃₂ = m₃₂ = 0 and m₃₃ = 1 by assuming the affine homography. Thus, ω_r = 1, and we obtain the following two algebraic equations:

$$\begin{array}{l}u=m_{11}{y}_r+m_{12}{z}_r+m_{13}\\ v=m_{21}{y}_r+m_{22}{z}_r+m_{23}\end{array}$$

(23)

Thus, for Equation (23), we get the homogeneous linear square problem of

$$\begin{array}{l}{B}_u{M}_r=0, B_v{M}_r=0\\ B_u=\left[\begin{array}{ccccccccc}{y}_r& z_r& 1& 0& 0& 0& 0& 0& -u\end{array}\right]\\ B_v=\left[\begin{array}{ccccccccc}0& 0& 0& y_r& z_r& 1& 0& 0& -v\end{array}\right]\\ M_r={\left[\begin{array}{ccccccccc}{m}_{11}& m_{12}& m_{13}& m_{21}& m_{22}& m_{23}& 0& 0& 1\end{array}\right]}^T\end{array}$$

(24)

Then, we estimate the homogeneous matrix, M, in Equation (22) using Singular Value Decomposition (SVD).

3. Conceptual Design Based on the Theoretical Methodology

Figure 9 shows the overall block diagram of the sensor fusion method described in Section 2. Two types of homography matrices are estimated by calibrating data obtained from a thermal IR camera, an FMCW radar system, and a 3D ultrasound system. We use the estimated matrices to map data sets of (u, v) corresponding to (x_s, y_s, z_s) and (x_r, y_r, z_r) to a common plane. The object information is then visualized and displayed along with updated (u, v) sets of objects and their distances.

To implement the methodology described in Section 2 and Figure 9, a hardware sensor system, as shown in Figure 10, was conceptually designed. The conceptual system consists of five 3D ultrasonic modules (the green points of Figure 10), a single thermal IR camera (the red point of Figure 10), and a single FMCW radar system (the blue point of Figure 10). The three types of sensors are related to the sensors in Figure 9. Based on Equation (5), the distance between the ultrasonic sensors in each 3D ultrasonic module is 20 cm. This distance is related to the values of ‘a’ and ‘b’ in Figure 4. In addition, the angle θ between modules, as shown in Figure 5, was selected as 45 degrees according to Equation (7). Specifications for the selected ultrasonic sensors, thermal IR cameras, and FMCW radar systems in Figure 9 and Figure 10 are given in

Table 2. The specification of the chosen sensors.

Sensors	Category	Specification	# of Sensors Used
Ultrasonic sensor (Hargisonic, HG-M40)	Frequency (kHz)	40	20
	Input pulse	TTL or Pulse
	Output signal (V)	5
	Detection range (m)	0.3~3
FMCW radar (Infineon, BGT24MTR12)	Field of View (degree)	19 × 76	1
	Carrier frequency (GHz)	24
	Bandwidth (MHz)	200
	Sweep time (μs)	300
	Max. IF (kHz)	10
	Detection range (m)	3~10
Thermal IR camera (FLIR A35)	Focal plane (pixel)	320 × 256	1
	Pixel space (μm)	25

. The presented sensor fusion system is currently under construction.

4. Future Research Directions and Conclusions

In this paper, we have presented a conceptual sensor fusion framework to detect objects even in dense smoke environments using a thermal IR camera, ultrasonic sensors, and an FMCW radar based on the review of several research papers.

The proposed sensor fusion can provide new methodologies such as (1) 3D global position estimation for multiple objects through two types of 2D information and one type of 3D information, (2) application of image transformation theory to general data information transformation, (3) 3D position estimation of an object using 1D ultrasonic sensor information, (4) extension of the directional angle and detection area of the ultrasonic sensor system, (5) rejection of measurement error in ultrasonic sensors caused by temperature change in dense smoke, (6) object detection and classification through image processing of a thermal IR camera. The proposed system using these methodologies is expected to be able to detect objects even in environments with less than 1 m visibility, such as in dense smoke environments.

There are some limitations of the proposed methodologies as well. Because thermal IR cameras cannot provide detailed images of the environment like RGB cameras, the environmental information from the images can be ambiguous. Most of all, because the sensors will be normally used in very high-temperature environments of dense smoke, the performance of object detection can be affected by image saturation in thermal IR cameras or detection error propagation in ultrasonic sensors and FMCW radar. These unwanted phenomena are normally caused by extremely high temperatures. As a result, the proposed sensor fusion system remains most predictable when the temperature specification of used sensors is as high as possible, but this is impossible due to the implementation costs and technical limitations. Nevertheless, the proposed sensor fusion will be more useful than other studies because the selected sensors are the most appropriate for the dense smoke environment, and the proposed methodologies maximize the strengths of each sensor as well as compensate for its weaknesses.

In future work, we will complete the hardware and software shown in Figure 9 and Figure 10 and the experimental setup to validate the theoretical methodology proposed in this paper. Then, we will evaluate the performance based on some experimental conditions and find more limitations of the proposed system. The obtained results will also need to be compared to various existing methods. In addition, the computational cost will be studied because it is very important to reduce computational costs while maintaining or increasing algorithm performance simultaneously.

Theoretically, the detection error of an ultrasonic sensor and radar is none, according to the theoretical analysis in this paper. However, in reality, the detection error of an ultrasonic sensor and radar depends on the sensor’s specifications that the user chooses for their applications. Therefore, we need to study how much detection error by ultrasonic and radar sensors is available after completing the hardware setup.

Most of all, the operating temperature range of all electric devices, including the sensors used in this paper, is normally −40 to 100 degrees Celsius, and the temperature of dense smoke is more than 500 degrees Celsius. Because of that, the sensors cannot work normally at relatively high temperatures. Therefore, all applications for fire disasters, such as firefighting robots and firefighter aids, need cooling systems and/or countermeasures of heat resistance/insulation. The cooling system and the heat resistance/insulation system are other research topics in the field of firefighting platforms. The scope of this paper is not a study in any detail of cooling systems or heat resistance/insulation systems but a study of how to deal with sensors for object detection in dense smoke environments. Therefore, in this paper, note that we have assumed that the H/W platform (which will be developed in the future) will equip a system to prepare for extremely high temperatures well. Under this assumption, we propose the sensor fusion concept using ultrasonic sensors, FMCW radar, and a thermal IR camera.