3D Sensor Based Pedestrian Detection by Integrating Improved HHA Encoding and Two-Branch Feature Fusion

Abstract

Pedestrian detection is vitally important in many computer vision tasks but still suffers from some problems, such as illumination and occlusion if only the RGB image is exploited, especially in outdoor and long-range scenes. Combining RGB with depth information acquired by 3D sensors may effectively alleviate these problems. Therefore, how to utilize depth information and how to fuse RGB and depth features are the focus of the task of RGB-D pedestrian detection. This paper first improves the most commonly used HHA method for depth encoding by optimizing the gravity direction extraction and depth values mapping, which can generate a pseudo-color image from the depth information. Then, a two-branch feature fusion extraction module (TFFEM) is proposed to obtain the local and global features of both modalities. Based on TFFEM, an RGB-D pedestrian detection network is designed to locate the people. In experiments, the improved HHA encoding method is twice as fast and achieves more accurate gravity-direction extraction on four publicly-available datasets. The pedestrian detection performance of the proposed network is validated on KITTI and EPFL datasets and achieves state-of-the-art performance. Moreover, the proposed method achieved third ranking among all published works on the KITTI leaderboard. In general, the proposed method effectively fuses RGB and depth features and overcomes the effects of illumination and occlusion problems in pedestrian detection.

1. Introduction

The importance of pedestrian detection cannot be overstated. It is the basis for many computer vision tasks and is widely used in video surveillance, robotics, automatic driving and healthcare assistance. Benefitting from convolution neural network (CNN) methods and large-scale datasets, RGB image based pedestrian detection performance has made great progress with the development of RGB cameras in recent years. However, some challenges are difficult to solve if only RGB visual information is exploited, such as insufficient light, occlusion, small-scale objects and dense crowds, as shown in Figure 1a–d. Unlike RGB images, depth images generated from 3D sensors are not affected by illumination and can provide three-dimensional information, but have the problems of low resolution and high noise. Therefore, combining RGB and depth images (RGB-D data) to detect people can compensated for the other’s shortcomings and provide the possibility to solve those challenges. Many pedestrian detection methods [1,2,3,4,5,6] based on RGB-D data have been presented. They also demonstrated that the using of multi-modality RGB-D data is better than using the single modality form. However, two issues with regard to using RGB-D data are worthy of further investigaton: (1) how to use depth images; and (2) how to fuse the RGB and depth modalities.

For the issue of using depth images, it is unsuitable to directly feed it into a deep network, as the depth image is usually a single channel and 16-bit image. Therefore, the depth information is usually encoded as grayscale or a three-channel pseudo-color image (as shown in Figure 2). Among them, horizontal disparity, height above ground, and the angle of the pixel’s local surface normal makes with the inferred gravity direction (HHA) [7] contains more information in three channels. Moreover, HHA has been effectively applied to semantic segmentation [8], scene understanding [9], and salient object detection [10]. However, it performs inadequately at pedestrian detection. For example, some studies [11,12] have shown that the detection performance of converting depth information to a jet colormap is better than that of converting it to HHA. But the HHA carries more information than the jet colormap. We found that the HHA encoding only extracts the gravity direction, so it is easy to introduce noises when calculating the channel of the height above ground, which seriously affects pedestrian detection. To address this problem, we propose to improve the HHA encoding method by extracting the ground plane parameters instead of just extracting the ground direction. The improved algorithm effectively reduces the error encoding of height above ground.

For the issue of fusing multi-modal information, most existing RGB-D [2,4] or RGB-T [13,14] pedestrian detection algorithms explored the feature maps of two modalities in weighted fusion. This process possibly obtains local features of two modalities, since the additive fusion of two corresponding channels is independent of other channels. Employing $1\times 1$ convolution after concatenated two feature maps is also a common fusion method [1,3]. In contrast, this method extracts global features because each channel of the convolution output is a linear combination of all input channels. In our opinion, local features are as important as global features. Therefore, we designed a two-branch network to fuse these two modalities in both global and local views.

To this end, this paper proposes an RGB-D data based pedestrian detection network which takes RGB and improved HHA images as inputs. In the proposed model, an improved HHA method is firstly proposed to encode the depth image more accurately. We also perform depth equalization for the mapping of the depth values, which improves the color space utilization. The improved method runs twice as fast and estimates the direction of gravity more accurately. Then, we design a two-branch feature fusion extraction module (TFFEM) which uses two branches to extract the global and local features separately. In the local feature extraction, we also propose an adaptive channel cross fusion (ACCF) module to learn the weights of each channel and to become more efficient. We validate the designed network on two publicly-available datasets and achieve state-of-the-art detection performance. One dataset is KITTI [15]. It employs the LiDAR sensor to capture the point cloud, which can still be encoded as an HHA image. Another dataset is EPFL [16], whose depth images are acquired directly by the 3D sensor of Kinect v2. In summary, the main contributions of this paper are summarized as

• We design an RGB-D data based pedestrian detection network which achieves the state-of-the-art detection performance on KITTI and EPFL datasets.
• We improve the HHA encoding method, which is twice as fast and can extract the full ground parameters. Moreover, the detection performance outperforms other encoding methods.
• We propose two new modules, i.e., TFFEM and ACCF, in the deep network, which can learn rich multimodal features.

2. Related Work

2.1. Single Modal Based Pedestrian Detection

2.1.1. Depth Image Based Approaches

There are many pedestrian detection algorithms for single-depth images, and they can be classified into three main types, i.e., traditional methods, CNN-based methods, and the combination of the traditional and CNN model methods. These methods require high depth image quality, so it is not sufficient to only use the depth image outdoors or in large spaces.

Traditional methods generally require several steps. In the preprocessing stage, many studies [17,18] use background modeling algorithms to remove background regions, and some algorithms use plane detection to remove areas that contain planes. In the region of interest(ROI) extraction stage, the methods for depth images are also various. Hu et al. [17] use spatial clustering to obtain ROI. Tian et al. [19] utilize height information to remove ROI that contains points that are too high or low. And, more often, the head information is exploited because the size and shape of the person’s head is usually fixed in space, so researchers propose many head-based features [18,19,20,21] to filter the ROI. However, the problem with these methods is that they can only detect the upper body region of the people and cannot detect the people with the head obscured.

The traditional method combined with CNN [19] usually uses CNN to classify the extracted ROI at the end. Since most invalid ROIs have been removed by depth information, the speed of these methods is also significant. Meanwhile, some studies [22,23] proposed end-to-end pedestrian detection networks that only use depth images. However, these methods are still based on head features.

2.1.2. RGB Image Based Approaches

Since the RPN network was proposed in the two-stage object detection method Faster RCNN [24], the performance of CNN-based pedestrian detection algorithms [25] has greatly exceeded that of traditional methods [26,27,28]. Later, single-stage object detection methods were proposed, such as YOLO [29,30], SSD [31], and EfficientDet [32], which are faster and have high detection accuracy. Many studies of pedestrian detection are based on them. The above methods require many pre-defined anchors, which reduces the flexibility of the network in training. Meanwhile, some anchor-free object detection algorithms such as CenterNet [33] and FCOS [34] have been proposed. Correspondingly, the CenterNet-based pedestrian detection algorithm CSP [35] has achieved high detection performance. Moreover, there are also some networks designed specifically for pedestrian detection, such as TAFT [36], and W3Net [37]. RGB image based pedestrian detection algorithms have made significant progress. However, RGB images do not provide 3D information and are sensitive to illumination, which may significantly degrade the detection performance of these algorithms in changing environments.

2.2. Multi-Modal Based Pedestrian Detection

Multimodal data are more stable than single modality data, and also contain richer information. Therefore, many RGB-D pedestrian detection methods have been presented in recent years. These methods mainly address two issues: the fusion position of two modals, and the fusion method. Earlier approaches [38,39,40,41] only fused the classification results of each modality without interaction in feature extraction. They used adaptive joint probabilities to combine the scores of each modality. Based on the Faster-RCNN, Ren et al. [12] adopted two parallel VGG networks to extract features and fused them in the last layer while leaving the rest unchanged. Ophoff et al. [1,42] verified through extensive experiments that fusing two features extracted in the middle network layer is superior to fusion the early and late ones. In addition. There are also many studies that focus more on fusion methods. Zhang et al. [2,4] referred to the SENet [43] and computed weight for each channel after combining the two features.

Some studies also focused on other elements. Guo et al. [5] obtained the mask of people’s target labels by clustering depth images and combined this information to improve the stability of pedestrian detection. Some studies [44,45] proposed multitask models that enable the network to output the box and distance information to enhance the feature representation. Linder et al. [3] improved the network by adding a synthetic RGB-D dataset that is rendered with a semi-realistic game engine.

Furthermore, some different encoding strategies have been explored for using depth information in multimodal data. The pixel values of the raw depth image indicate the distance from the object to the camera. Since this value varies greatly for different devices and units, it is not directly suitable for training. The efficient way is to encode the depth image to a grayscale one (as shown in Figure 2a) according to Equation (1),

$$f\left(Z_{uv}\right)=\frac{Z_{uv}-min\left(Z\right)}{max\left(Z\right)-min\left(Z\right)}$$

(1)

where Z denotes the depth value, and $uv$ refers to the image coordinates. Existing deep learning networks are designed for three-channel color images, so many methods further encode grayscale into pseudo-color images such as jet colormap (as shown in Figure 2b) before training.

The above encoding methods, although efficient, ignore the 3D information provided by the depth. The depth image can be converted to a point cloud by camera intrinsic. Some methods extract the spatial information in the point cloud and then invert it back to the image plane to generate a new pseudo-color image, which carries adequate information. Tian et al. [19] proposed the DMH representation, which contains three channels of raw depth value, a multi-order depth template, and a height difference map. Currently, the most-commonly used depth encoding method is HHA [7,46] (as shown in Figure 2d).

3. Proposed Method

This section contains two parts. The first part introduces the improved HHA method for using depth information. We describe the original HHA encoding and its shortcomings. The improvements in gravity direction estimation and depth value mapping are also presented. The second part introduces our proposed RGB-D data-based pedestrian detection network and its submodules.

3.1. Improved HHA Encoding

The original HHA image is in three-channel, and each channel is obtained by encoding original depth information. We aim to improve the encoding procedure in two components (described in Section 3.1.3 and Section 3.1.4) while obtaining a same-size image.

3.1.1. Vanilla HHA

The first channel in an HHA image is horizontal disparity, which is inversely proportional to the depth value (denoted as z) obtained from 3D sensors. The converted value in each pixel is computed according to Equation (2).

$$d=\frac{f\ast b}{z}$$

(2)

where d is the disparity of each pixel in the first channel, and $f,b$ are the focal length and the baseline of one camera.

For the pixel values in the second and third channels, the gravity height and the angle with the gravity direction of each point are calculated respectively. The gravity direction is estimated as follows. Firstly, the depth value z is converted to a point-cloud value, and the normal is calculated [46]. Next, the gravity direction (i.e., normal of the ground plane) is estimated iteratively by point normal. The initial step is to set ${\mathbf{g}}_{\mathbf{o}}={[0,1,0]}^T$ (the default camera view is a horizontal forward view). All point normal is $\mathbf{n}$. The second step is to get the parallel normal set $N_{\Vert }$ and the horizontal normal set $N_{\perp }$ in $\mathbf{n}$ according to Equation (3)

$$\begin{array}{c}{N}_{\perp }=\left\{{\mathbf{n}}_{\mathbf{i}}\right|\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{o}})<T_d\cup \theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{o}})>180°-T_d\}\hfill \\ N_{\Vert }=\left\{{\mathbf{n}}_{\mathbf{i}}\right|\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{o}})>90°-T_d\cap \theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{o}})<90°+T_d\}\hfill \end{array}$$

(3)

where $\theta (\mathbf{a},\mathbf{b})$ denotes the angle between the three-dimensional vectors $\mathbf{a}$ and $\mathbf{b}$, and $T_d$ denotes the angle threshold. The third step is to estimate the new gravity direction using $N_{\Vert }$ and $N_{\perp }$. The estimation process is equivalent to searching the ${\mathbf{g}}_{\mathbf{n}}$ that minimizes Equation (4)

$$\underset{\mathbf{g}:{∥{\mathbf{g}}_{\mathbf{n}}∥}_{\mathbf{2}}=\mathbf{1}}{min}\sum _{{\mathbf{n}}_{\mathbf{i}}\in N_{\perp }}{cos}^2\left(\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{n}})\right)+\sum _{{\mathbf{n}}_{\mathbf{i}}\in {\mathbf{N}}_{\Vert }}{sin}^2\left(\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{n}})\right)$$

(4)

where ${\mathbf{g}}_{\mathbf{n}}$ need to be normalized to satisfy ${∥{\mathbf{g}}_{\mathbf{n}}∥}_2=1$. According to the trigonometric function, the Equation (4) can be rewritten as Equation (5).

$$\begin{array}{c}\sum _{{\mathbf{n}}_{\mathbf{i}}\in {\mathbf{N}}_{\perp }}{cos}^2\left(\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{n}})\right)+\sum _{{\mathbf{n}}_{\mathbf{i}}\in N_{\Vert }}-{cos}^2\left(\theta ({\mathbf{n}}_{\mathbf{i}},{\mathbf{g}}_{\mathbf{n}})\right)\hfill \\ ={\mathbf{g}}_{\mathbf{n}}(N_{\perp }{N}_{\perp }^T-N_{\Vert }{N}_{\Vert }^T){{\mathbf{g}}_{\mathbf{n}}}^T\hfill \end{array}$$

(5)

Then according to the Courant-Fischer theorem from linear algebra, the ${\mathbf{g}}_{\mathbf{n}}$ can be obtained by decomposing the matrix $N_{\perp }{N}_{\perp }^T-N_{\Vert }{N}_{\Vert }^T$. After that, ${\mathbf{g}}_{\mathbf{n}}$ is used as the initial direction ${\mathbf{g}}_{\mathbf{o}}$. The above steps are repeated until the stop condition is met.

The vanilla HHA [7] adopts the angle thresholds $T_d$ to ${45}^{\circ }$ and ${15}^{\circ }$ and iterates five times under each threshold. After getting the gravity direction, the angle of each point is calculated directly. And the gravity height is calculated by first rotating the whole point cloud to the horizontal direction and then subtracting the smallest y coordinate value from the y coordinate value of each point.

3.1.2. Shortcoming Analysis of Vanilla HHA

The method of gravity estimation in vanilla HHA uses horizontal elements (floor, ceiling, etc.) and vertical ones (wall, etc.) in the scene, which has high stability but suffers from some shortcomings.

First, it only estimates the gravity direction and cannot obtain the complete ground plane equation. The plane equation is as in Equation (6)

$$Ax+By+Cz+D=0$$

(6)

The gravity direction $\mathbf{g}={[A,B,C]}^T$. Hence, there is also a missing parameter D. Due to the lack of this parameter, the original method is calculating the height by subtracting the smallest y coordinate value. This calculation is sensitive to the camera pose and noise. And if the complete ground plane parameters are available, the height can be calculated directly by the point-to-plane distance equation. As shown in Figure 3b, the original method makes a difference in generating HHA images on the NYU2 and EPFL datasets.

Secondly, the original method requires ten iterations. Each iteration has to process all points, which is inefficient and susceptible to noise interference. This would cause poor accuracy of the estimated gravity direction.

3.1.3. Improved Gravity Direction Estimation

To address the shortcomings of vanilla HHA, we improve the gravity direction estimation in HHA encoding by considering the ground plane parameters and removing a part of the points after each iteration.

Our ground estimation process is designed as follows. In the first step, after obtaining the gravity direction ${\mathbf{g}}_{\mathbf{n}}$ according to the original method, we calculate the ground plane parameter D using all the parallel points $P_{\Vert }$ according to Equation (7)

$$D=\frac{1}{N}\sum _{\mathbf{p}\in P_{\Vert }}{\mathbf{p}}^T·{\mathbf{g}}_{\mathbf{n}}$$

(7)

where $\mathbf{p}={[p_x,p_y,p_z]}^T$ denotes the points, and N is the number of parallel points. In the second step, combining the obtained ground plane equation, only the points that satisfy the Equation (8) are retained

$$P_s=\left\{\mathbf{p}\right|{\mathbf{p}}^T·{\mathbf{g}}_{\mathbf{n}}+D>T_{up}\cap {\mathbf{p}}^T·{\mathbf{g}}_{\mathbf{n}}+D<T_{down}\}$$

(8)

where $T_{up}$ and $T_{down}$ are the manual thresholds. The idea of removing points is to consider that the points belonging to the ground are under the estimated plane, so threshold $T_{up}$ is set to keep the points under the plane. Meanwhile, the threshold $T_{down}$ is set to remove the points too far from the plane, preventing noise interference.

Our estimation process is iterated only three times, and the final ground plane parameters are obtained using all parallel point fits at the end of the three iterations. The final ground equation was fitted using least squares estimation, where the process is to calculate the parameters $-\frac{A}{C},-\frac{B}{C},-\frac{D}{C}$ so that Equation (9) takes the minimum value.

$$min\sum _{p\in P_{\Vert }}(-\frac{A}{C}{p}_x-\frac{B}{C}{p}_y-\frac{D}{C}-p_z{)}^2$$

(9)

The whole iterative process is shown in Figure 4a1,a2,b1,b2,c1,c2, in which the red points represent the vertical points $P_{\perp }$ and the blue points represent the parallel points $P_{\Vert }$. It can be seen in Figure 4, the number of points decreases in each iteration, and the final estimated ground is shown as the green points in Figure 4d1 d2. In addition, the threshold values for the three iterations are $T_d$ = [45°, 30°, 10°], $T_{up}$ = [−30 cm, −15 cm, −5 cm], and $T_{down}$ = [+∞, +∞, 15 cm]. The improved gravity direction extraction method reduces the number of iterations and removes redundant points each time to improve the fitting accuracy. After obtaining the ground equation, the point-to-surface distance is directly used as the gravity height channel of HHA images, which does not require manual adjustment of parameters and can be more adaptable to different data. The subsequent experimental results also prove the superiority of our method.

3.1.4. Improved Depth Value Mapping

Moreover, we update the depth value mapping to facilitate the calculation of the first channel in improved HHA. The first channel of HHA images is mapping the horizontal disparity to grayscale. Therefore, we directly use the depth values instead of the horizontal disparity. The mapping method uses average mapping in HHA encoding. The depth value is evenly arranged to the grayscale of 0–255, as shown in Figure 5c. This mapping is simple and easy to implement. However, the accuracy of most 3D sensing is proportional to the distance, and the depth error is larger when the distance is farther. Therefore, more objects are spread in the front and middle of the depth range, as shown in Figure 5a,b. As can be seen, there are already very few objects with a distance of more than 40 m in the KITTI dataset. Hence, if the depth values are equally mapped on the grayscale an imbalance will result. For example, the distant regions containing few objects occupy the same range on the grayscale as the near regions containing more objects. To address this problem, we propose a depth equalization mapping (DEM) method, as shown in Figure 5d.

The DEM refers to the histogram equalization (HE) algorithm. The mapping process can be seen as computing the mapping function $g=f\left(z\right)$ so that the depth values z are converted to grayscale values g. f is computed by Equation (10)

$$f\left(z_i\right)=L\sum _{z=0}^{z_i}H\left(z\right)$$

(10)

where L is the maximum range of the grayscale (256), and $H\left(z\right)$ is the histogram of the depth values for all objects. This histogram is calculated as Equation (11)

$$H\left(z_i\right)=p(z=z_i)=\frac{n_{z_i}}{n},z_{min}<z_i<z_{max}$$

(11)

where n is the number of all objects and $n_{z_i}$ is the number of objects with depth $z_i$.

3.2. Two-Branch Pedestrian Detection Network

3.2.1. Network Overview

The structure of the proposed RGB-D data-based pedestrian detection network is shown in Figure 6. The network inputs the RGB images and HHA images encoded by the improved method. Two parallel ResNet50 [47] networks are employed to extract features from each input separately. Since it is a detection task, the average pooling and fully connected layer of the ResNet50 are removed. Then the feature maps ${\mathbf{F}}_n^{RGB}\in {\mathbb{R}}^{H\times W\times C}$ and ${\mathbf{F}}_n^{HHA}\in {\mathbb{R}}^{H\times W\times C}$ output from the same layers are fused (excluding the first layer). Where H, W, and C are the output feature map’s height, width, and channel number. N is the index of layers. The fusion method is the multimodal fusion and attention module (MFAM) described in the next. Then the fusion result $F_n^{Fused}$ is used as the input of the feature pyramid network (FPN). Finally, existing detection head modules such as FCOS [34] and Faster RCNN [24] can be directly connected to the FPN to detect people. We next select the head modules of Faster RCNN.

The structure of MFAM is shown in Figure 7, which consists of two main parts: the TFFEM and the attention module. Each of them is described below.

3.2.2. Two-Branch Feature Fusion Extraction Module

The main idea of TFFEM is to learn the global and local features of two modalities to enhance the ability of networks for interpreting multimodal features. The structure is shown in Figure 8. Since the detection object is a single class and to improve the module efficiency, we first reduce the channel number of the input feature maps (${\mathbf{F}}_n^{RGB}$, and ${\mathbf{F}}_n^{HHA}$) by 50% using $1\times 1$ convolution to obtain the new feature maps ($({\widehat{\mathbf{F}}}_n^{RGB},{\widehat{\mathbf{F}}}_n^{HHA})\in {\mathbb{R}}^{H\times W\times \frac{C}{2}}$). Then two branches are constructed to extract the global feature map ${\widehat{\mathbf{F}}}_n^{Glob}\in {\mathbb{R}}^{H\times W\times \frac{C}{2}}$ and local feature map ${\widehat{\mathbf{F}}}_n^{Loc}\in {\mathbb{R}}^{H\times W\times \frac{C}{2}}$ from the new feature maps. Finally, the output feature map ${\mathbf{F}}_n^{TFF}\in {\mathbb{R}}^{H\times W\times C}$ of TFFEM is obtained by concatenation of the global and local feature maps as Equation (12)

$${\mathbf{F}}_n^{TFF}=Cat({\widehat{\mathbf{F}}}_n^{Glob},{\widehat{\mathbf{F}}}_n^{Loc})$$

(12)

The process of extracting global features is first to concatenate ${\widehat{\mathbf{F}}}_n^{RGB}$ and ${\widehat{\mathbf{F}}}_n^{HHA}$. Then a standard $3\times 3$ convolution is used to obtain the global feature map. Since the standard convolution is employed, each channel in the output feature map is acquired by the joint calculation of all channels in the input feature map. So it can get the global features.

The local features are extracted by our proposed ACCF block. First, each channel in feature maps ${\widehat{\mathbf{F}}}_n^{RGB}$ and ${\widehat{\mathbf{F}}}_n^{HHA}$ with scale $W\times H\times \frac{C}{2}$ is cross-arranged to generate a new feature map with size $W\times H\times C$. The new feature map is then processed by group convolution with a kernel size of $1\times 1$. The number of groups and output channels are all $\frac{C}{2}$. After group convolution, the local feature map ${\widehat{\mathbf{F}}}_n^{Loc}$ is obtained. Each channel of ${\widehat{\mathbf{F}}}_n^{Loc}$ is computed from the channels corresponding to ${\widehat{\mathbf{F}}}_n^{RGB}$ and ${\widehat{\mathbf{F}}}_n^{HHA}$ without involving other ones. So ACCF can extract the local features. Furthermore, the group convolution can be seen as assigning a weight to each channel of both feature maps, which is more flexible than the direct add. ACCF is also better than the hard threshold adding operation proposed in [48], where all channels are multiplied by the same weight. Obviously, it is more reasonable to make the network learn the weights of each channel by itself.

3.2.3. Attention Module

The fused feature map obtained by TFFEM contains both global and local features between two modalities, and this feature map can continue to be fed into the attention module. It is different from the CW or ACW fusion module proposed in [4,13]. They refer to the idea of the attention module to calculate the weights of each channel.

The convolutional block attention module (CBAM) [49] is chosen in our network, which includes channel attention and spatial attention. In addition, a standard convolution module is added before the CBAM for preprocessing the input feature maps. Finally, the output feature map ${\mathbf{F}}_n^{Fused}$ by the MFAM is computed according to Equation (13)

$${\mathbf{F}}_n^{Fused}={\mathbf{W}}_s\odot ({\mathbf{W}}_c\odot Conv\left({\mathbf{F}}_n^{TFF}\right))+{\mathbf{F}}_n^{TFF}$$

(13)

where ${\mathbf{W}}_c$, ${\mathbf{W}}_s$ represents the weights of channel and spatial attention module. ⊙ represents the element-wise multiplication. $Conv$ denotes a standard convolution, including the $3\times 3$ convolution, batch normalization (BN), and activation function ReLU. ${\mathbf{F}}_n^{TFF}$ is the feature map obtained by the TFFEM.

4. Experiments

The experiments consist of four parts. The first two parts evaluate the performance of our proposed pedestrian detection method on two publicly-available datasets. The third part compares our improved HHA encoding method with the original one from the metrics of accuracy and speed of the gravity directions estimation. The last part is the ablation study, which first verifies the superiority of improved HHA encoding in RGB-D data based pedestrian detection and then demonstrates the advantages of our proposed fusion module TFFEM.

4.1. Evaluation of Pedestrian Detection on KITTI Dataset

4.1.1. The Dataset and Evaluation Metrics

We first evaluate our pedestrian detection algorithm on the KITTI [15], a widely used and challenging dataset for autonomous driving. KITTI contains 7481 training samples and 7518 test samples. Each sample includes up to 30 pedestrians. The metrics we evaluated are consistent with the official comparision, which uses the average precision (AP) with an $IOU$ threshold of 0.5 for pedestrians. The AP also contains three difficulty levels, i.e., easy, moderate and hard levels, based on the target height, occlusion and truncation.

4.1.2. Implementation Detail

In experiments, we divide all samples into a training set (3712 samples) and a validation set (3769 samples) according to [50]. The KITTI dataset does not provide depth images, so we convert the LiDAR point cloud to the camera coordinate system and generate depth images. Afterward, depth images are recovered according to [51] and encoded to HHA images. Our network uses the FasterRCNN detection head and replaces its ROIPooling with ROIAlign [52]. Except for the TFFEM, the network is initialized with pretrained weights on the CrowdHuman [53] and Citypersons [54] datasets. Furthermore, the TFFEM cannot be pretrained only on RGB data, so its weights are initialized as [55]. The network is trained for 17.6 K iterations with a batch size of 2. The SGD gradient descent algorithm is used. The initial learning rate (LR) is set to 0.01, the momentum to 0.9, and the weight decay to 0.0001. The LR increases from 0.001 to 0.01 in the beginning of 0.5 K iterations and decreases to 10% at the 15.4 K and 16.5 K iterations. The input image resolution is kept constant while the image is padded to ensure that the width and height are multiples of 32. The entire training process takes about 5.5 h on a single GTX1080Ti GPU.

When training RGB and HHA images, the data augmentation is specially designed. Color augmentation, such as illumination, transparency and gamma, is only for RGB because pixel values of the HHA image contain geometric information. Moreover, all images are randomly translated by only −5∼5%, rotated by 0∼5 degrees, and scale transformed by 0.9∼1.1 times. Furthermore, the random horizontal flip is adopted.

We also utilize some tricks to improve the detection performance. For example, the pixel value of the DontCare region on RGB and HHA images is set to 0 according to [35]. Moreover, the width of each prediction bounding box is extended by 0.1 times. It is particularly useful for $AP$ with an $IOU$ threshold of 0.5. Besides, many samples of the KITTI dataset do not contain pedestrians, e.g., only 955 of 3712 training samples have pedestrians. So, training all the samples would reduce the network’s ability to discriminate pedestrian targets. Therefore, for each epoch, we randomly select 100 samples from the pedestrian-free samples and together with all samples containing pedestrians to training.

4.1.3. Comparison with State-of-the-Art Approaches

Table 1. Comparison with the state-of-the-art approaches on the KITTI test set for pedestrian detection.

Method	Easy	Moderate	Hard	Input	Time (s)
MM-MRFC (2017, [56])	83.79	70.76	64.81	RGB-D-F	0.05
SubCNN (2017, [57])	84.88	72.27	66.82	RGB	2
RRC (2017, [58])	85.98	76.61	71.47	RGB	3.6
ECP (2018, [59])	85.96	76.25	70.55	RGB	0.25
FRCNN+Or (2018, [60])	71.64	56.68	51.53	RGB	0.09
TAFT (2018, [36])	67.62	53.15	47.08	RGB	0.2
F-ConvNet (2019, [61])	83.63	72.91	67.18	RGB-P	0.47
VMVS (2019, [62])	82.80	71.82	66.85	RGB-P	0.25
HotSpotNet (2020, [63])	71.43	62.31	59.24	RGB	0.04
FII-CenterNet (2021, [64])	81.32	67.31	61.29	RGB	0.09
WSSN (2021, [5])	84.91	76.42	71.86	RGB-D	0.37
HHA-TFFEM (Proposed)	85.32	77.12	72.69	RGB-D	0.14

shows the pedestrian detection results of our method and other state-of-the-art methods on the KITTI test dataset. The input of these methods includes single-modal and multimodal. In Table 1, D represents the depth image, P represents the LiDAR point cloud, and F represents the optical flow. It can be seen that our method outperforms other methods on moderate and hard metrics and has reached the state-of-the-art on the easy metric. In addition, although we use two-modal data, the inference speed of our network is faster than that of many single-modal methods.

We select the model that performs best on the validation set to process the test set. The results are submitted to the official benchmark (http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d, (accessed on 18 December 2021)) for 2D pedestrian detection and ranked third among all published methods. Figure 9 shows the detection results in several challenging scenarios on the KITTI test set. These scenarios include insufficient light, occlusion, small-scale objects and dense crowds.

4.2. Evaluation of Pedestrian Detection on the EPFL Dataset

4.2.1. The Dataset and Evaluation Metrics

To further verify the robustness of our proposed pedestrian detection method, we also evaluate pedestrian detection performance on the EPFL dataset. This dataset employs Kinect V2 to capture RGB-D data in two scenes, i.e., a laboratory (EPFL-LAB) and a corridor (EPFL-CORRIDOR) in a university building. A total of 5140 samples were collected. EPFL-CORRIDOR is recognized as one of the most complex datasets [23]. EPFL contains various degrees of occlusion, and the distance between people is very close. We divide the EPFL dataset into a training set (1650 samples), a validation set (970 samples) and a test set (1570 samples) according to [23].

The metrics are adopted with the $AP$. The EPFL data set does not distinguish the difficulty level of targets, so we only calculate the $A{P}_{50}$, i.e., the IOU threshold is 0.5. In addition, we also compute the $A{P}_{75}$, i.e., the IOU threshold is 0.75 and $A{P}_{COCO}$ [65] to verify our method comprehensively. In addition, Ophoff et al. relabeled the EPFL dataset to add the bounding box of severely obscured pedestrians that are not included in original annotations. Thus, we use their annotation information.

4.2.2. Comparison with State-of-the-Art Approaches

We compared the detection performance of our proposed pedestrian detection algorithm with nine state-of-the-art approaches with different strategies. They included four classical RGB-based target detection algorithms [24,30,31,66] and five of the newest RGB-D-based pedestrian detection algorithms [1,2,3,4,5]. FasterRCNN, SSD and YOLOV3 are trained and tested utilizing the algorithms provided in the MMDetection toolbox [67], and YOLOV5 utilizes the officially-offered code. The five methods of Ophoff [1], Zhang [2], Linder [3], AAFTSNet [4], and WSSN [5], have inconsistent settings for training and testing sets (e.g., Ophoff uses 70% of the data for training and 20% for testing). Therefore, we cannot directly use the test results provided in their papers. Finally, we reimplemented these methods according to their paper and recorded the test results.

Table 2. Comparison with the state-of-the-art approaches on the EPFL dataset for pedestrian detection.

Method	AP50	AP75	APCOCO	Input
FasterRCNN (2015, [24])	78.1	59.1	50.2	RGB
SSD (2016, [31])	80.0	45.8	44.6
YOLOV3 (2018, [30])	82.3	52.7	47.8
YOLOV5 (2021, [66])	86.8	55.5	51.5
Ophoff (2019, [1])	84.0	51.6	49.0	RGB-D
Zhang (2020, [2])	86.7	54.2	51.2
Linder (2020, [3])	86.5	65.4	57.2
AAFTSNet (2021, [4])	87.7	61.4	55.3
WSSN (2021, [5])	88.4	64.1	55.8
HHA-TFFEM (Proposed)	90.2	66.0	57.4

shows the detection performance of all methods on the EPFL dataset. It can be seen that our method outperforms other methods in all three metrics. In the $A{P}_{50}$ metric, our method is the only one that exceeds 90%. In addition, the SSD-based method [2,31] has better performance on the $A{P}_{50}$ metric but is lower on $A{P}_{75}$ and $A{P}_{COCO}$ metrics, while the FasterRCNN has much lower scores in $A{P}_{50}$ than SSD, YOLOV3, and YOLOV5 but exceeds them in the $A{P}_{75}$. Figure 10 shows the Precision-Recall curves of $A{P}_{50}$, where the dashed line indicates the method using only RGB data.

4.3. Evaluation of Improved HHA Encoding

4.3.1. Datasets and Evaluation Metrics

We validated our proposed improved HHA encoding method under four commonly used RGB-D detection datasets, including UNIHALL [38,68], KTP [69,70], and two subsets of EPFL, i.e., EPFL-LAB and EPFL-COR (captured in different scenarios). To improve the validation efficiency, we selected the data captured by the first camera in UNIHALL, the first subset of EPFLCOR, and the subset of KTP named Still. Another reason for this choice is that each subset is captured with a fixed camera. Thus, the ground truth for gravity orientation is also fixed. It is worth mentioning that we did not choose the KITTI dataset for validation because the camera position of each sample on the KITTI is changed. So it is hard to obtain the ground-truth gravity orientation of all samples.

The validation metrics refer to the angle judgment proposed in [46] which judges the angle between the estimated and ground-truth gravity direction. A smaller angle indicates a better estimation. However, only the EPFL dataset provides the ground-truth gravity direction, So we manually calibrate the other two datasets to obtain the ground-truth gravity direction. The manual estimation process is first to select a depth image with the largest ground area. Then five ground points are manually marked and fitted by the least-squares algorithm to obtain the gravity direction and calculate the fitting error. Lastly, this process is repeated five times and selects the parameters with the smallest error as the ground truth.

4.3.2. Comparison of Gravity Direction Estimation

Figure 11 shows the angle of gravity direction for our improved (blue) and the original (red) methods on four datasets. In each subplot, the estimated angles of all images are listed, where the horizontal coordinates indicate the image number and the vertical coordinates indicate the angle.

As can be seen, the overall angle of our improved method is smaller than the original method, especially on the UNIHALL. The scene of UNIHALL contains a staircase whose points are miss-classified to the vertical set, thus seriously affecting the gravity estimation of the original method. In contrast, the improved method is less affected because many irrelevant points are removed at each iteration.

In addition, both methods estimate larger angles on the UNIHALL and KTP than on the EPFL. This is because these two datasets are captured by Kinect V1, and their depth information quality is lower than that of the EPFL captured with Kinect V2.

4.3.3. Comparison of Encoding Speed

Since the improved HHA encoding method reduces the number of iterations from ten to three and removes a part of the points in each iteration when estimating the gravity direction, the running speed is significantly promoted.

Table 3. HHA encoding times of our improved and original methods on five datasets.

		KTP [70]	UNIHALL [38]	EPFL-LAB [16]	EPFL-COR [16]	KITTI [15]	Avg.
Runtimes (s)	Original	0.630	0.618	0.438	0.433	1.102	0.644
	Improved	0.267	0.263	0.182	0.172	0.471	0.271
Image Resolution		$640\times 480$	$640\times 480$	$512\times 424$	$512\times 424$	$1242\times 375$

shows the HHA encoding times of our improved and original methods on five datasets. The time for each dataset is averaged over all images, and the unit is in seconds. As can be seen, our improved method is more than twice as fast as the original method. The statistics experiment uses an ordinary desktop computer with Intel Core I7-409 8700K, 16 GB of RAM and Win10. The software used is the MatLab platform, which is consistent with the official HHA encoding code. It is worth mentioning that this time it will run faster on platforms such as C++.

4.4. Ablation Study

In this section, we execute comparative experiments to demonstrate the effectiveness of these two parts.

4.4.1. Study on Different Depth Encoding Methods

Table 4. Comparison of depth encoding methods on the KITTI val dataset for RGB-D pedestrian detection.

Depth Encoding Method	Easy	Moderate	Hard	AP50	APCOCO	Input
Grayscale	85.78	78.41	71.39	66.9	31.7	RGB-D
Surface Normals	87.86	79.42	72.53	69.7	34.4
Colormap Jet	87.58	79.72	73.05	68.5	33.4
HHA Orginal	87.26	80.25	72.96	68.9	32.9
HHA + GE (Proposed)	87.44	80.84	74.00	70.6	33.5
HHA + GE + DEM (Proposed)	88.90	82.14	75.33	71.5	34.5

shows the detection performance of four types of depth information encoded images and our improved HHA images on the KITTI val dataset with RGB-D as input. These encoding methods include grayscale, surface normals, jet colormap and the original HHA. There are two main aspects to our improvement of HHA encoding. One is more efficient ground parammeter extraction (GE) and the other is DEM. Therefore, Table 4 lists two of our detection results, which are HHA + GE, i.e., only the effect after the GE is employed, and HHA + GE + DEM, when the two improvements are contained. Moreover, except for the original metrics provided in the KITTI dataset, we calculated the $A{P}_{50}$ and $A{P}_{COCO}$ results for a more comprehensive comparison. As can be seen, our encoding method outperforms other encoding methods in all metrics.

4.4.2. Study on Different Fusion Methods

We first compared our multimodal fusion module with common fusion methods on the KITTI val dataset as shown in

Table 5. Comparison of fusion methods on the KITTI val dataset for RGB-D pedestrian detection.

Fusion Method	Easy	Moderate	Hard	AP50	APCOCO	Input
Summation	86.76	79.02	72.83	68.4	31.6	RGB-D
Concatenation	87.79	80.33	73.19	69.5	33.7
TFFEM (proposed)	88.00	80.76	74.58	70.1	34.0
Summation + CBAM	87.42	79.86	73.83	70.3	33.9
Concatenation + CBAM	87.74	80.62	73.74	71.1	33.4
TFFEM + CBAM (proposed)	88.90	82.14	75.33	71.5	34.5

. The compared methods include element summation and concatenation. Since concatenation increases the channel number, we reduced the channel number using 1 × 1 convolution after concatenation. We also compare the detection results after using the attention module CBAM for each fusion method. From Table 5, we can see that the detection results of our fusion method are better than the other approaches in all metrics, both in direct fusion and followed by the attention module.

5. Discussion

The above experiments fully demonstrate the effectiveness of the proposed method. In this section, the proposed method will be further analyzed.

(1) The improved HHA encoding method obtains more accurate gravity directions, thus improving the precision of the encoded height information. At the same time, height information is a very discriminated but noise-sensitive feature to the pedestrian detection task compared to other tasks such as scene understanding [9] and salient object detection [10]. Therefore, the HHA images obtained by the original encoding method perform poorly in the pedestrian detection task but well in other tasks [11,12].

(2) The result in Table 4 shows that using the DEM is better than mean mapping. DEM allows depth regions containing more targets to be mapped to a wider color space, which increases the dissimilarity between targets and thus can improve detection performance.

(3) The proposed TFFEM extracts local and global features of two modalities, which increases the richness of the network features and thus improves the pedestrian detection performance. In addition, the attention model is also beneficial to RGB-D pedestrian detection.

(4) The improved HHA encoding method is twice as fast as the original method but still does not run in real-time. In addition, the complexity of the proposed RGB-D pedestrian detection network is high. Therefore, our subsequent work will involve improving the detection speed to satisfy the application requirements.

6. Conclusions

In this paper, we address two key issues in the RGB-D pedestrian detection task, i.e., utilizing depth information and fusing the two modalities’ information. Firstly, we analyze the reasons for the low detection performance of HHA encoded images carrying more information and improving the HHA encoding. Secondly, we propose the TFFEM and design a new RGB-D detection network based on this module.

Through the experiments, we first validate the effectiveness of the proposed RGB-D detection network on two datasets, followed by the advantages of our improved HHA encoding on four datasets. Finally, the superiority of the proposed improved HHA encoding over other encoding methods and the excellence of the proposed TFFEM are demonstrated.