AFE-RCNN: Adaptive Feature Enhancement RCNN for 3D Object Detection

Abstract

The point clouds scanned by lidar are generally sparse, which can result in fewer sampling points of objects. To perform precise and effective 3D object detection, it is necessary to improve the feature representation ability to extract more feature information of the object points. Therefore, we propose an adaptive feature enhanced 3D object detection network based on point clouds (AFE-RCNN). AFE-RCNN is a point-voxel integrated network. We first voxelize the raw point clouds and obtain the voxel features through the 3D voxel convolutional neural network. Then, the 3D feature vectors are projected to the 2D bird’s eye view (BEV), and the relationship between the features in both spatial dimension and channel dimension is learned by the proposed residual of dual attention proposal generation module. The high-quality 3D box proposals are generated based on the BEV features and anchor-based approach. Next, we sample key points from raw point clouds to summarize the information of the voxel features, and obtain the key point features by the multi-scale feature extraction module based on adaptive feature adjustment. The neighboring contextual information is integrated into each key point through this module, and the robustness of feature processing is also guaranteed. Lastly, we aggregate the features of the BEV, voxels, and point clouds as the key point features that are used for proposal refinement. In addition, to ensure the correlation among the vertices of the bounding box, we propose a refinement loss function module with vertex associativity. Our AFE-RCNN exhibits comparable performance on the KITTI dataset and Waymo open dataset to state-of-the-art methods. On the KITTI 3D detection benchmark, for the moderate difficulty level of the car and the cyclist classes, the 3D detection mean average precisions of AFE-RCNN can reach 81.53% and 67.50%, respectively.

1. Introduction

In recent years, with the rapid development of intelligent driving technology, it is necessary to improve the performance of 3D object detection, as 3D object detection is a key technology of intelligent driving. At present, the popular 3D detection datasets are the KITTI dataset [1] and the Waymo open dataset [2]. There are mainly two types of 3D object detection method based on point clouds, i.e., point-based method and voxel-based method. The point-based method uses the multi-layer perceptron (MLP) [3] and sets abstraction to process raw point clouds. This kind of method usually has high detection precision, such as in refs. [4,5,6,7,8,9,10]. However, using the same MLP to process all points cannot express the relevancy of the spatial-variant well. Voxel-based methods generally convert the unstructured point clouds into a 3D voxel or a 2D bird’s eye view grid, such as in refs. [11,12,13,14,15,16]. After voxelization, the 3D voxel or 2D bird’s eye view grid can be processed by convolutional neural networks. Compared with MLP, the convolution operation has stronger spatial-variant processing capability. However, the voxelization operation will lose structure information.

To combine the advantages of point-based methods and voxel-based methods, Shaoshuai Sh et al. proposed PV-RCNN [17] for 3D object detection. In the voxel-based branch, PV-RCNN can efficiently generate high-quality 3D box proposals. The point-based branch effectively preserves the location information of the input data by processing the raw point clouds directly. The PV-RCNN can achieve the precise fine-grained box refinement by fusing the features of bird’s eye view (BEV), raw point clouds, and voxel. Figure 1 shows the schematic diagram of these three features; the spatially sparse convolution is composed of sparse convolution [10] and submanifold convolution [18]. The submanifold convolution is an efficient convolution operation for sparse data, which can greatly reduce the convolution computation when processing sparse data. The structure of the sparse convolution is located in the lower part of the figure. The boxes with colors in the sparse convolution part represent voxels with points; the white boxes represent the voxels without points. For the sparse convolution, a matrix is firstly constructed by the convolution kernel and the index of input features. Secondly, a general matrix multiplication-based algorithm (GEMM) is used for the matrix computation. Finally, the spatial positions of the output features are recovered. More details can be obtained from ref. [10]. As shown in

Table 1. The ablation experiments on the features of raw point clouds, voxel, and BEV. Note that the results in this table are provided in rows 4–6 of Table 8 in ref. [17]. The ${\mathrm{mAP}|}_{40}$ calculated from 40 recall positions is used as the evaluation metric. Here is the example of the 3D detection ${\mathrm{mAP}|}_{40}$ for the car class with moderate-level difficulty. Note that the experiment is based on the KITTI dataset.

Voxel Features	BEV Features	Raw Point Cloud Features	$\mathbf{Moderate}{\mathbf{mAP}|}_{40}$
√			84.54
√	√		84.69
√	√	√	84.72

, the feature fusion method can improve the 3D detection performance.

However, like refs. [11,19,20], PV-RCNN processes the BEV features with the traditional convolutional neural network (CNN) in part of the region proposal network (RPN). To generate the high-quality proposals, more consideration is needed on the correlation of features in the channel branch and the spatial branch. Like refs. [4,7], Pointnet [21] or Pointnet++ [22] are used to process the raw point clouds in PV-RCNN, which cannot efficiently aggregate neighboring contextual information. Furthermore, like refs. [7,10], the smooth-L1 loss is used for box regression in PV-RCNN, which ignores the correlation among the vertices of the bounding box. In order to solve the above problems, we improve the PV-RCNN from the aspects of point cloud feature extraction, the RPN, and the loss function of the box proposal regression.

The main contributions are as follows:

• We design a residual of dual attention proposal generation module, i.e., RDA module. This module learns the correlation of features in both channel branch and spatial branch, while reducing the loss of the information transmission process. The RDA module leads to the higher quality box proposals and enhances the features of BEV. The enhanced features are beneficial to the box proposal refinement.
• We design a multi-scale feature extraction module based on feature adaptive adjustment, i.e., MSAA. The proposed module uses the multi-scale feature extraction method to enhance the robustness of sparse point cloud features. Meanwhile, to fully mine the neighboring contextual information among all points, we introduce a feature adaptive adjustment method to make the key points better describe the local neighborhood region.
• We design a loss function module based on vertex associativity, i.e., VA module. This module constructs a regression loss function based on the projection of the 3D detection box into the BEV coordinate system and the DIoU loss.

2. Related Work

Point cloud feature extraction methods: Pointnet [21] adopts 3D spatial transform net (STN) and Maxpooling to solve the disorder and rotation of point clouds. However, Pointnet simply joins point sets to obtain the global features and the features of individual points. After that, Pointnet++ [22] is proposed, which extends Pointnet with the farthest sampling layer and grouping layer to increase the information contained in each point. While the max pooling operation of Pointnet is followed in the feature extraction of ref. [21], it still cannot effectively aggregate the information in the local region. To obtain a richer semantic representation of each point, Wang Yue et al. proposed DGCNN [23] to construct a local neighborhood through EdgeConv. Although EdgeConv considers the distance between point coordinates and neighboring points, it ignores the vector direction between neighboring points. As a result, part of the local geometric information is lost. In order to process the contextual information of point clouds better, Li Yangyan et al. proposed PointCNN [24], which processes the order of features through the “X”-transformation matrix, so that the convolution operator can be used to process the disordered point clouds. Unlike ref. [24], Hengshuang Zhao et al. focused on the interaction between points in each local neighborhood, and proposed the Pointweb [25] to extract contextual features from local neighborhoods in the point clouds. Pointweb makes the points more descriptive of local regions, which is due to fully mining the contextual information among all points. For more robustness in processing sparse point clouds, we combine the multi-scale approach with the method in Pointweb.

Attention mechanism in convolutional neural networks: A large number of studies have shown that the attention mechanism is beneficial to improve the performance of a CNN. Hu Jie et al. [26] proposed a channel attention network, i.e., SENet. This attention module is composed of three parts, i.e., squeeze, excitation, and scale. Based on SENet, Sanghyun Woo et al. [27] proposed CBAM (convolutional block attention module); performance is further improved by increasing spatial attention. Moreover, a residual attention network [28] can obtain abundant key features through residual attention learning. FuJun et al. [29] proposed a double attention network, i.e., DANet, to capture the global feature dependency in the spatial and channel dimensions, which can improve segmentation results by modeling the contextual dependencies of local features. However, DANet is computationally intensive when modeling both the spatial and channel dimensions. HuajunLiu et al. [30] proposed a polarized self-attention mechanism, i.e., PSA. The PSA block uses a unique filtering approach to control the complexity of the model while maintaining a high resolution in both the spatial and channel dimensions. Generally, directly inserting the attention module into the network may not work well and can even lead to performance degradation, which may be caused by some important features being weakened, gradient disappearance, or over-fitting. To solve these problems, some tricks or methods can be used to improve the attention module. For example, the residuals can be used to correct the output of the attention block, which allows rich, simple information to propagate directly to deeper layers and ensures the integrity of the information during the transmission, e.g., ref. [31].

The loss function for object detection: Fast R-CNN [32] uses a smooth-L1 loss function to regress the coordinate values, but this loss function ignores the correlation among coordinates. IoU Loss [33] regards the location information as a whole to train the network, which considers the correlation among the vertices of the detection box. However, IoU Loss cannot optimize the two non-overlapping boxes, and cannot reflect the distance between two boxes. In order to solve the above problems, Hamid Rezatofighi et al. [34] proposed Generalized IoU Loss, i.e., GIoU. GIoU can be used as a distance measurement directly and can handle the case of non-overlapping boxes well. However, the convergence speed of GIoU is slow. To improve the convergence speed and regression accuracy, Zheng Zhaohui et al. [35] proposed Distance IoU Loss, i.e., DIoU for box regression, which takes three geometric factors into account, i.e., the overlap region, centroid distance, and aspect ratio. However, DIoU Loss is usually used for the calculation for a 2D bounding box, which is not effectively introduced into 3D object detection.

3. AFE-RCNN for Point Cloud Object Detection

The overall architecture of our AFE-RCNN is shown in Figure 2. We propose three novel modules, i.e., (a) the residual of dual attention proposal generation module (RDA module) (Figure 2a); (b) multi-scale adaptive feature extraction module based on point clouds (MSAA) (Figure 2b); (c) refinement loss function module with vertex associativity (VA module) (Figure 2c). In our AFE-RCNN, the input point clouds are processed by two branches; that is, a voxel-based branch and a point-based branch. In the voxel-based branch, we first voxelize the raw point clouds. Referring to ref. [17], we divide the raw point clouds into small voxel grids of the same scale in x,y,z directions. Then, we use the average of the features of all points inside the non-empty voxel as the features of the voxel. Secondly, we obtain the voxel features through a 3D spatially sparse convolutional network [10]. Next, we project the point clouds to the 2D BEV coordinate by eight times downsampling (i.e., stack the Z-axis features together to generate the 2D BEV feature maps), and obtain the 2D BEV features through the RDA module. Finally, the RPN generates box proposals based on the anchor box approach and BEV features. In the point-based branch, to summarize the information of the scene, we sample the key points through farthest point sampling (FPS), then the features of key points are obtained by MSAA. Next, as the rich features are beneficial to the box proposal refinement, the features of BEV, raw point clouds, and voxels are fused to build the key point features. To make the foreground points contribute more to refinement, the weights of the key points are adjusted by the weighting module (see ref. [17]). At last, the box proposal refinement is completed based on the key point features and the key point-to-grid RoI feature abstraction (see ref. [17]); the grid point is the center point of the voxel grid. We guarantee the correlation among the vertices of the box to optimize the box proposal regression by the advantages of the VA module. Details of the main operations are given in the subsections.

3.1. The Residual of Dual Attention Proposal Generation Module

Our AFE-RCNN firstly divides the raw point clouds into the several uniform scales of voxel grids, and then obtains the 3D voxel features through a 3D spatially sparse convolutional network. Secondly, the proposed network projects the information of 3D features to the 2D BEV coordinate. The network of the BEV feature extraction in ref. [17] consists of the traditional 2D CNN, which usually processes all features equivalently and cannot reflect the variability requirements in object detection tasks. Moreover, the extracted feature map only aggregates the spatial and channel information in the local receptive field, which will lead to a lack of relevance of global information.

To solve the above problems, we propose the residual of dual attention proposal generation module (RDA module). Inspired by the polarized self-attention block [30], we construct the attention block by the residual connection to ensure the integrity of information transmission for feature processing, which can process both the spatial and channel dimensions of features with very little calculation increase. The diagram of the proposed RDA module is shown in Figure 3. The input of the module is used to mainly control the direction of information processing, avoiding the situation in which background points are overly focused. The proposed module can strengthen the correlation between object features in the spatial domain and global feature channel, which can fully excavate semantic information in both spatial dimension and channel dimension.

We first assume that the input features $\mathrm{X}$ are independent; the channel attention branch ${\mathrm{B}}_c\left(\mathrm{X}\right)\in R^{C\times 1\times 1}$ is defined as Equation (1).

$${\mathrm{B}}_c{\left(\mathrm{X}\right)=F}_{SG}{[Conv}_z{\left(\right(\tau }_1{(Conv}_v{\left(\mathrm{X}\right))\times F}_{SM}{(\tau }_2{(Conv}_q\left(\mathrm{X}\right)\left)\right)]$$

(1)

where ${Cnv}_q{,Conv}_v,$ and ${Conv}_z$ are all 1 $\times$ 1 convolutional layers, ${\tau }_1$ and ${\tau }_2$ are two vector transformation symbols, $\times$ presents matrix dot product operation, $F_{SG}\left(\right)$ is a sigmoid operator, and $F_{SM}()$ is a SoftMax operator. The output of channel attention branch is:

$${\mathrm{O}}_c{=\mathrm{B}}_c\left(\mathrm{X}\right){\odot }^c\mathrm{X}\in R^{C\times H\times W}$$

(2)

where ${\odot }^c$ is the channel multiplication operator.

Here, the spatial attention branch ${\mathrm{B}}_s\left(\mathrm{X}\right)\in R^{1\times H\times W}$ is defined as Equation (3):

$${\mathrm{B}}_s{\left(\mathrm{X}\right)=F}_{SG}{[\tau }_3{(F}_{SM}{(\tau }_1{(F}_{GP}{(Conv}_q{\left(\mathrm{X}\right)\left)\right))\times \tau }_2{(Conv}_v\left(\mathrm{X}\right)\left)\right)]$$

(3)

where ${\tau }_1$, ${\tau }_2$, and ${\tau }_3$ are vector transformation symbols; $F_{GP}()$ is a global pooling operator. The output of the spatial attention branch is shown as Equation (4). ${\odot }^s$ is the spatial multiplication operator.

$${\mathrm{O}}_s{=\mathrm{B}}_s{(\mathrm{O}}_c){\odot }^s{\mathrm{O}}_c$$

(4)

The output of one residual block ${\mathrm{O}}_r$ is:

$${\mathrm{O}}_r=\mathrm{X}+{\epsilon \mathrm{O}}_s$$

(5)

where $\epsilon$ is the weighting factor of the attention module. Here, the range of $\epsilon$ is 0~1; the best performance can be achieved when $\epsilon =0.1$. After a lot of experiments, we found that the average running times before and after adding the RDA module to the RPN stage are ~0.0060 s and ~0.0071 s run on the same processor, respectively, which is a difference of milliseconds. The use of the attention network did not result in a significant decrease in the efficiency of our proposed network.

3.2. Multi-Scale Feature Extraction Module Based on Adaptive Feature Adjustment

In ref. [17], a certain number of points are sampled as the key points. That sampling approach may cause poor robustness features in the sparse region of the point clouds. To solve this problem, we propose a multi-scale feature extraction module based on adaptive feature adjustment (MSAA) as shown in Figure 4. Firstly, we sample 2048 points through FPS to summarize the information of the scene. This is followed by two times 4-fold downsampling and points grouping; the sampling layers with different scales that containing 512 and 128 points are obtained, respectively. Inspired by ref. [25], to extract richer feature information, we process the features for each grouping layer based on adaptive feature adjustment (AFA). The interaction between points can be found through the AFA, which can also integrate the neighboring contextual information into the features of each point. Moreover, the multi-scale point clouds can be constructed by different sampling and grouping layers. Thus, the features of all sampling layers are interpolated to the same feature shape and concatenated together, which can construct multi-scale features of the key points.

In the point cloud feature extraction for each sampling layer, first, the features of each point (Figure 4a) in the region Q consist of the feature set $F$, which can be expressed as ${F=\{F}_1$,$F_2$,…,$F_n$}. In order to focus on the correlation between each point pair, we connect the points in region Q into a local network, which can learn the interactions between all point pairs. We get the difference map $F_{dif}^{\left(i\right)}$ (Figure 4b) at the same time. And then, $F_{dif}^{\left(i\right)}$ learns the amount of impact among point features through the impact function to get the impact map ${\beta }_i$ Figure 4c, this is so that the points in region Q can merge the neighboring contextual information, and the features of each point have better local neighborhood representation ability. The adjusted feature $F_i^{\prime }$ is defined as Equation (6).

$$\begin{gathered} F_i^{\prime }{=F}_i+F_i^{mod}, \\ {F}_i^{mod}=f_{modual}{(F}_i,F) \end{gathered}$$

(6)

where $F_i$ is one of the features in the feature set $F$. $F_i^{mod}$ represents the amount of impact, which is affected by each feature in $F$ on $F_i$. $F_i^{mod}$ is obtained by adaptive learning of the feature modulator in the feature set $F$. The learning process of the feature modulator $f_{modual}$ is defined as Equation (7), where $·$ means product operation.

$$f_{modual}{(F}_i,F)={{\displaystyle \sum }}_{j=1}^n{f}_{impact}{(F}_i{,F}_j{)f}_{relation}{(F}_i{,F}_j)$$

(7)

The learning process of the feature modulator is mainly involved in two functions; one is the amount of impact of $F_j$ on $F_i$, which is represented by the function $f_{impact}$. The other is the relation function $f_{relation}$. The impact function $f_{impact}$ is calculated by the MLP operation. The function $f_{impact}$ is defined as Equation (8):

$$f_{impact}{(F}_i{,F}_j{)=\mathrm{MLP}(f}_{relation}{(F}_i{,F}_j\left)\right)$$

(8)

In the relation function $f_{relation}$, the amount of impact is calculated by the difference between the two feature vectors $F_i$ and $F_j$ in feature set $F$. Specifically, when $i=j$, the amount of impact on $F_i$ is the feature $F_i$ itself. $f_{relation}$ is defined as Equation (9):

$$f_{relation}{(F}_i{,F}_j)=\{\begin{array}{c}{F}_i-F_j,\hspace{1em}i\ne j\\ F_i,\hspace{1em}i=j\end{array}$$

(9)

For each feature in the local region Q, the total output of the learned feature is shown as Equation (10). Hence, $F_i^{\prime }$ incorporates contextual information from the entire region through the above densely connected local network of points.

$$\begin{gathered} F_i^{\prime }={\beta }_i^{\left(i\right)}·F_i+{\displaystyle \sum }_{j=1,j\ne i}^{\mathrm{N}}{\beta }_j^{\left(i\right)}·{(F}_j-F_i) \\ {\beta }_j^{\left(i\right)}=\{\begin{array}{c}{-f}_{impact}{(F}_i{,F}_j),i\ne j\\ f_{impact}{(F}_i{,F}_j)+1,i=j\end{array} \end{gathered}$$

(10)

In different sampling layers, we firstly use interpolation to upsample the point clouds to the same number, and then concatenate these feature vectors as multi-scale features for key points. The interpolation process can be expressed as:

$$\widehat{F_j^{lay}}=\frac{{{\displaystyle \sum }}_{j=1}^K{\omega }_j{(p}_i^{lay}{)F}_j^{lay}}{{{\displaystyle \sum }}_{j=1}^K{\omega }_j{(p}_i^{lay})}$$

(11)

$${\omega }_j{(p}_i)=\{\begin{array}{c}\frac{1}{{\left|\right|p}_i^{lay}{-p}_j^{lay}{\left|\right|}_2}{,p}_j^{lay}\in P^{lay}{(p}_i^{lay})\\ 0,other\end{array}$$

(12)

where $p_i^{lay}$ is one of the points in the sampling point set $P^{lay}$. $F_j^{lay}$ is the feature of $p_i^{lay}$; $\widehat{F_j^{lay}}$ is the feature obtained by interpolation. We calculate the distance weight ${\omega }_j$ between $p_i^{lay}$ and $p_j^{lay}$ through the K-nearest neighbors. As shown in Equation (12), the further away from $p_i^{lay}$, the smaller the contribution on $p_i^{lay}$. Here, K = 3.

In the point sampling and grouping stage, the feature set of the (k + 1)-th layer is ${ℱ}^{(k+1)lay}$, which is represented by Equation (10). To make the features of the (k + 1)-th layer consistent with the features of the previous layer in the point dimension, ${ℱ}^{(k+1)lay}$ is processed by interpolation to obtain $\widehat{{ℱ}^{(k+1)lay}}$. Finally, the previous layer $F^{\left(k\right)lay}$ is obtained by concatenating ${ℱ}^{\left(k\right)lay}$ and $\widehat{{ℱ}^{(k+1)lay}}$. The feature set of the k-th layer is:

$$\begin{gathered} F^{\left(k\right)lay}={ℱ}^{(k+1)lay}+\widehat{{ℱ}^{(k+1)lay}},\hspace{1em}\hspace{1em}k=1,2 \\ {F}^{\left(k\right)lay}={ℱ}^{\left(k\right)lay},\hspace{1em}\hspace{1em}k=3 \end{gathered}$$

(13)

In addition, to provide rich feature information for box proposal refinement, we convert the BEV features obtained in Section 3.1 to key point features by bilinear interpolation, then the set abstraction operation in ref. [22] is used to fuse the features of voxels into key points. After the above operation, the features of BEV, raw point clouds, and voxels are fused as the key point features. Finally, we sample m grid points in each box proposal. Here, m = 216. For each grid point, we aggregate the features of neighboring key points to it to help the box proposal refinement.

3.3. Refinement Loss Function Module with Vertex Associativity

The refinement network is mainly used to learn the center point, size, and direction of the detection box. Two MLP layers are used to construct the refinement network, which finally performs the two tasks of confidence prediction and box regression. Generally, for the box regression loss, the vertices are usually regarded as independent of each other. However, the vertices of the bounding box are related to each other to a certain extent. The intersection over union (IoU) [33] is used to evaluate the effectiveness of the detection boxes. For different detection boxes, there may be a situation where the smooth-L1 loss is similar but the IoU varies greatly. In order to solve this problem, we propose the refinement loss function module with vertex associativity (VA module), as shown in Equation (14). The VA module trains the bounding box as a whole and solves the problem in which the anchor cannot be regressed when the prediction box does not intersect with the ground truth box.

$${\mathrm{L}}_{reg}\left(x_{pre}{,x}_{gt}\right){=\mathrm{L}}_{smooth\_L1}\left(x_{pre}{,x}_{gt}\right){+\mathrm{L}}_{corner}{+\alpha \mathrm{L}}_{DIoU}$$

(14)

The weight coefficient $\alpha$ of ${\mathrm{L}}_{DIoU}$ is referred to the dataset and experiment. According to the statistics of multiple experiments, a better result can be achieved when the $\alpha$ value is in the range 0.1~1. The best performance can be achieved when $\alpha$ = 0.3. ${\mathrm{L}}_{smooth\_L1}$ and ${\mathrm{L}}_{corner}$ are shown as Equations (15) and (16). ${corn}_{pre}$ is the predicted corner; ${corn}_{gt}$ is the ground truth corner.

$${\mathrm{L}}_{smooth\_L1}\left(x_{pre}{,x}_{gt}\right)=\{\begin{array}{c}0.5{(x}_{pre}{-x}_{gt}{)}^{2}if\left|x_{pre}{-x}_{gt}\right|<1\\ \left|x_{pre}{-x}_{gt}\right|-0.5\hspace{1em}otherwise\end{array}$$

(15)

$${\mathrm{L}}_{corner}=mean\left({\mathrm{L}}_{smooth\_L1}\left({corn}_{pre}{,corn}_{gt}\right)\right)$$

(16)

Here, the normalized distance between the center points of the prediction box and the ground truth box is directly minimized to achieve rapid convergence. We choose the height value of the 3D bounding box within a certain angular range on the lidar coordinate. The height values are then normalized as a partial value for the length and width of the bounding box. We project the 3D bounding box to the 2D BEV coordinate through the above approach, so that the DIoU can be calculated by the 2D BEV information. L_DIoU is shown as Equation (17) [35]:

$${\mathrm{L}}_{DIoU}{=\mathrm{L}}_{IoU}+\frac{{\rho }^2{(b,b}^{gt})}{c^2}$$

(17)

where $b$ represents the center point of the prediction box; $b^{gt}$ is the center point of the ground truth box. $\rho ()$ represents the function of the Euclidean distance between two center points. c represents the diagonal distance of the smallest rectangle that can cover the predicted box and the ground truth box at the same time.

For the confidence prediction, the IoU between the ground truth boxes and the box proposals are used as the optimization object.

$${\mathrm{L}}_{IoU}=-u_k\log (\widetilde{u_k}{)-(1-u}_k\left)\log \right(1-\widetilde{u_k})$$

(18)

Here, $u_k$ is the confidence of the optimization object for the k-th box proposals. $\widetilde{u_k}$ is the predicted score.

3.4. Training Losses

Our AFE-RCNN is trained end-to-end. According to Figure 2, the training losses ${\mathrm{L}}_{train}$ consist of the region proposal loss ${\mathrm{L}}_{rpn}$, the segmentation loss ${\mathrm{L}}_{seg},$ and the refinement loss ${\mathrm{L}}_{rcnn}$.

1

The proposal generation network performs the classification and regression of anchor boxes based on the BEV features. The region proposal loss ${\mathrm{L}}_{rpn}$ is:

$${\mathrm{L}}_{rpn}{=\mathrm{L}}_{cls}+{\mathrm{L}}_{smooth\_L1}\left({\tilde{\vartheta }}^{gen},{\vartheta }^{gen}\right)$$

(19)

Among them, the anchor box regression is calculated through ${\mathrm{L}}_{smooth\_L1}$. ${\tilde{\vartheta }}^{gen}$ is the predicted residual for the anchor box; ${\vartheta }^{gen}$ is the regression target for the anchor box. ${\tilde{\vartheta }}^{gen},{\vartheta }^{gen}$ represents the position parameters (x, y, z, h, l, w) and pose parameters ($\theta$) of the 3D bounding box. ${\mathrm{L}}_{cls}$ is the anchor classification loss as shown in Equation (20).

$${\mathrm{L}}_{cls}=\{\begin{array}{c}-\beta {\left(1-\tilde{\mathrm{X}}\right)}^g\log \left(\tilde{\mathrm{X}}\right),\mathrm{X}=1\\ -\left(1-\beta \right){\tilde{\mathrm{X}}}^g\log \left(1-\tilde{\mathrm{X}}\right),\mathrm{X}=0\end{array}$$

(20)

Among them, $\mathrm{X}\in \{0,1\}$ is the ground truth label; $\tilde{\mathrm{X}}$ is the predicted input. $\beta$ is the balance factor; $g$ is the focusing parameter that is used to adjust the sample weights. Here, we set $\beta$ = 0.25, g = 2.0.

2

The key point segmentation loss ${\mathrm{L}}_{seg}$ is used to filter the foreground, where the calculation method is the same as the classification loss ${\mathrm{L}}_{cls}$.

3

The refinement network performs the confidence prediction and regression of the box proposals based on the rich feature information of key points. ${\mathrm{L}}_{IoU}$ is used for confidence prediction, while ${\mathrm{L}}_{reg}$ in Section 3.3 is used for box regression. The proposal refinement loss ${\mathrm{L}}_{rcnn}$ is:

$${\mathrm{L}}_{rcnn}{=\mathrm{L}}_{IoU}+{\mathrm{L}}_{reg}\left({\tilde{\vartheta }}^{ref},{\vartheta }^{ref}\right)$$

(21)

Among them, ${\tilde{\vartheta }}^{ref}$ is the predicted box residual; ${\vartheta }^{ref}$ is the proposal regression object. ${\tilde{\vartheta }}^{ref}$, ${\vartheta }^{ref}$represent the position parameters and pose parameters of the bounding box.

The training loss ${\mathrm{L}}_{train}$can now be expressed as the Equation (22):

$${\mathrm{L}}_{train}{=\mathrm{L}}_{rpn}+{\mathrm{L}}_{seg}+{\mathrm{L}}_{rcnn}$$

(22)

4. Experiments and Results

In the experimental part, we first test AFE-RCNN on the KITTI dataset and compare it with the state-of-art methods, then we conduct ablation experiments to verify the effectiveness of each part for the proposed AFE-RCNN. In addition, we also conduct experiments on the Waymo dataset to prove the robustness of AFE-RCNN.

4.1. Dataset and Implementation Details

KITTI dataset: The KITTI dataset is widely used in the evaluation of 3D object detection networks. The annotation file of KITTI contains 15 items, which indicate target category, truncation rate, occlusion level, observation angle (−π~π), 2D bounding box coordinates, height, width, and length of the 3D bounding box, 3D coordinates of the object position in camera coordinates, and the rotation angle relative to the y axis. The detection task is divided into three difficulty levels, i.e., easy, moderate, and hard. The objects in the easy level are fully visible, while their minimum bounding box height is 40 pixels and maximum cutoff is 15%. The objects in the moderate level are partially occluded, while their minimum bounding box height is 25 pixels and maximum cutoff is 30%. The objects in the hard level are difficult to observe, while their minimum bounding box height is 25 pixels and maximum cutoff is 50%. We compared AFE-RCNN with the state-of-the-art methods on the online test server, and conducted ablation experiments on the validation set. For voxelization, we set the point cloud range as (0, 70.4) m in the X axis, (−40, 40) m in the Y axis, and (−3, 1) m in the Z axis. We also set the size of voxel grids as (0.05, 0.05, 0.1) m.

The training and test sets contain 7481 training samples and 7518 test samples, respectively. In the evaluation of the test set, we randomly divided 80% of the training set for training and the remaining 20% for validation. In the ablation experiment, we randomly divided the training data into 3712 training split and 3769 validation split.

According to ref. [36] and the KITTI official metrics, we set the IoU threshold for the car class as 0.7 and for the cyclist class as 0.5. The ${\mathrm{mAP}|}_{40}$ and the average orientation similarity (${\mathrm{AOS}|}_{40}$) calculated from 40 recall positions were used as the evaluation indicators for the test sets and the validation sets. The mAP was calculated on the set of recall $R_n=\{1/n,2/n,\dots ,1\}$ as Equation (23):

$${\mathrm{mAP}|}_R=\frac{1}{\left|R\right|}{\displaystyle \sum }_{r\in R}{\rho }_{interp}\left(r\right)$$

(23)

where ${\mathsf{\Theta }}_{obj}$ represents the target box, ${\mathsf{\Theta }}_{det}$ represents the detected box, ${\rho }_{interp}{\left(r\right)=\max }_{r^{\prime }{:r}^{\prime }\ge r}{\rho (r}^{\prime })$. $\rho \left(r\right)$ represents the precision calculated at the recall position $r$.

The AOS is calculated as Equation (24):

$$\mathrm{AOS}=\frac{1}{\left|R\right|}{\displaystyle \sum }_{r\in R}{\max }_{r^{\prime }{:r}^{\prime }\ge r}{s(r}^{\prime })$$

(24)

where $s\left(r\right)$ represents the orientation similarity rate calculated at the recall position $r$.

Waymo open dataset: The Waymo dataset is a large-scale 3D detection dataset. The training set has 798 sequences while the validation set has 202 sequences. The objects in the Waymo dataset are divided into two difficulty levels; level one comprises objects containing at least five points, and level two comprises objects containing at least one point or objects directly marked as level two. For voxelization, we set the point cloud range as (−75.2, 75.2) m for the X and the Y axes, and (−2, 4) m for the Z axis. We set the size of voxel grids as (0.1, 0.1, 0.15) m.

We used 200 sequences for training and in total 202 sequences for validation. The mAP and the mean average precision weighted by heading (mAPH) [2] are two official evaluation indicators, which were used to evaluate the performance of our AFE-RCNN in vehicle (IoU thresh = 0.7) and cyclist (IoU thresh = 0.5) detection.

Implementation details: Our AFE-RCNN is optimized by the ADAM optimizer. On the KITTI dataset, we trained our network with the batch size 6 and initial learning rate 0.01 for 80 epochs on 3 RTX 2080 GPUs. On the Waymo dataset, we trained our network with the batch size 4 and initial learning rate 0.01 for 30 epochs on 2 RTX 2080 GPUs.

4.2. Evaluation on the KITTI Online Test Server

We evaluated the test set based on the KITTI official test server and compared the performance of AFE-RCNN with the state-of-the-art methods. The results are shown in

Table 2. The performance of 3D detection on the KITTI online test server; the ${\mathrm{mAP}|}_{40}$ is calculated with 40 recall positions. Note that the red, green, and blue represent the results of ranking first, second, and third, respectively.

Method	Modality	Car-3D Detection $({\mathbf{mAP}|}_{40})$			Cyclist-3D Detection $({\mathbf{mAP}|}_{40})$
		Easy	Moderate	Hard	Easy	Moderate	Hard
EBM3DOD [39] (CVPR Workshops 2021)	Point	91.05	80.12	72.78	-	-	-
Faraway-Frustum [37] (IEEE ITSC 2021)		87.45	79.05	76.14	77.36	62.00	55.40
3DSSD [9] (CVPR 2020 Oral)		88.36	79.57	74.55	82.48	64.10	56.90
Point-GNN [8] (CVPR 2020)		88.33	79.47	72.29	78.60	63.48	57.08
EPNet [40] (ECCV 2020)		89.81	79.28	74.59	-	-	-
PointRCNN [7] (CVPR 2019)		86.96	75.64	70.70	74.96	58.82	52.53
CIA-SSD [14] (AAAI 2021)	Voxel	89.59	80.28	72.87	78.69	61.59	55.30
MGAF-3DSSD [41] (ACMMM 2021)		88.16	79.68	72.39	80.64	63.43	55.15
Part-A2 [11] (TPAMI 2020)		87.81	78.49	73.51	79.17	63.52	56.93
TANet [42] (AAAI 2020)		84.39	75.94	68.82	75.70	59.44	52.53
Associate-3Ddet [16] (CVPR 2020)		85.99	77.40	70.53	-	-	-
PointPillars [15] (CVPR 2019)		82.58	74.31	68.99	77.10	58.65	51.92
SA-SSD [43] (CVPR 2020)	Point-Voxel combination	88.75	79.79	74.16	-	-	-
SPG [44] (ICCV 2021)		90.50	82.13	78.90	-	-	-
DVFENet [45] (Neurocomputing 2021)		86.20	79.18	74.58	79.17	63.52	56.93
H^23D R-CNN [38] (TCSVT2021)		90.43	81.55	77.22	78.67	62.74	55.78
SE-SSD [46] (CVPR 2021)		91.49	82.54	77.15	-	-	-
PV-RCNN [17] (CVPR2020)		90.25	81.43	76.82	78.60	63.71	57.65
AFE-RCNN(Ours)		88.41	81.53	77.03	82.78	67.50	61.18

and

Table 3. The performance of orientation estimation on the KITTI online test server. Note that the red, green, and blue represent the results of ranking first, second, and third, respectively.

Method	Modality	Car-Orientation $\left({\mathbf{AOS}|}_{40}\right)$			Cyclist-Orientation $\left({\mathbf{AOS}|}_{40}\right)$
		Easy	Moderate	Hard	Easy	Moderate	Hard
EBM3DOD [39] (CVPR Workshops 2021)	Point	96.39	92.88	87.58	-	-	-
Point-GNN [8] (CVPR 2020)		88.33	79.47	72.29	-	-	-
EPNet [40] (ECCV 2020)		96.13	94.22	89.68	-	-	-
PointRCNN [7] (CVPR 2019)		95.90	91.77	86.92	85.94	72.81	65.84
CIA-SSD [14] (AAAI 2021)	Voxel	96.65	93.34	85.76	-	-	-
TANet [42] (AAAI 2020)		93.52	90.11	84.61	81.15	66.37	60.10
MGAF-3DSSD [41] (ACMMM 2021)		94.45	93.77	86.25	86.28	70.16	62.99
Part-A2 [11] (TPAMI 2020)		95.00	91.73	88.86	88.70	77.52	70.41
Associate-3Ddet [16] (CVPR 2020)		0.52	1.20	1.38	-	-	-
PointPillars [15] (CVPR 2019)		93.84	90.70	87.47	83.97	68.55	61.71
SA-SSD [43] (CVPR 2020)	Point-Voxel combination	39.40	38.30	37.07	-	-	-
SPG [44] (ICCV 2021)		40.02	38.73	38.52	-	-	-
H^23D R-CNN [38] (TCSVT2021)		96.13	93.03	90.33	85.09	72.20	65.25
DVFENet [45] (Neurocomputing 2021)		95.33	94.44	91.55	85.48	73.43	66.87
SE-SSD [46] (CVPR 2021)		96.55	95.17	90.00	-	-	-
PV-RCNN [17] (CVPR2020)		98.15	94.57	91.85	86.43	79.70	72.96
AFE-RCNN(Ours)		95.84	94.63	92.07	88.89	79.18	73.65

.

According to the official information from the KITTI benchmark, we mainly compared the performance of AFE-RCNN with the state-of-the-art methods on moderate-difficulty ${\mathrm{mAP}|}_{40}$ as of 5 January 2022. The following conclusions can be observed from Table 2 and Table 3 and Figure 5. Note that refs. [9,37] do not list the performance of orientation. These papers are not focused on the 3D object orientation evaluation, which mainly provides the ${\mathrm{mAP}|}_{40}$ of cars. To ensure the fairness of the comparisons, the comparison algorithms used in Table 2 and Table 3 are different according to the results provided in the corresponding references.

For the performance of 3D detection, our AFE-RCNN has 3 evaluation indicators (total 6 indicators) ranked Top 1 of all comparison algorithms as shown in Table 2. Besides, from the results shown in Table 2, we can make the following observations.

(1) Our AFE-RCNN outperforms the voxel-based methods on moderate and hard difficulty levels of the car and all difficulty levels of the cyclist. For cars, our network improves at least 1.25% on the moderate-difficulty ${\mathrm{mAP}|}_{40}$ than the compared voxel-based methods. For cyclists, our network improves at least 3.98% on the moderate-difficulty ${\mathrm{mAP}|}_{40}$ than the compared voxel-based methods. (2) Our AFE-RCNN outperforms the point-based methods on moderate and hard difficulty levels of cars and all difficulty levels of cyclists. For cars, our network improves at least 1.41% on the moderate-difficulty ${\mathrm{mAP}|}_{40}$ than the compared point-based methods. For cyclists, our network improves at least 3.40% on the moderate-difficulty ${\mathrm{mAP}|}_{40}$ than the compared point-based methods. (3) Compared with the method combining point clouds and voxels, our AFE-RCNN reaches 81.53% on the moderate-difficulty level ${\mathrm{mAP}|}_{40}$ for cars, which is comparable to the 81.55% of ${\mathrm{mAP}|}_{40}$ in ref. [38]. In all the difficulty levels for cyclists, our AFE-RCNN outperforms the compared algorithms.

For the performance of orientation estimation, our AFE-RCNN has 5 evaluation indicators ranked Top 3 of all comparison algorithms. As shown in Table 3, 3 indicators of our AFE-RCNN ranked first, and 2 indicators of our AFE-RCNN ranked second. From the results listed in Table 3, it is easily to draw the conclusions below.

(1) Our AFE-RCNN outperforms the voxel-based methods in 5 evaluation indicators. For cars, our network improves at least 0.86% on the moderate-difficulty ${\mathrm{AOS}|}_{40}$ than the compared voxel-based methods. For cyclists, our network improves at least 1.66% on the moderate-difficulty ${\mathrm{AOS}|}_{40}$ than the compared voxel-based methods. (2) Our AFE-RCNN outperforms the point-based methods in 5 evaluation indicators. For cars, our network improves at least 0.41% on the moderate-difficulty ${\mathrm{AOS}|}_{40}$ than the compared point-based methods. For cyclists, our network improves at least 6.37% on the moderate-difficulty ${\mathrm{AOS}|}_{40}$ than the compared point-based methods. (3) Compared with the method combining point clouds and voxels, our AFE-RCNN outperforms the compared algorithms in 3 evaluation indicators. For cars, our network improves at least 0.22% on the hard-difficulty ${\mathrm{AOS}|}_{40}$ than the compared combining methods. For cyclists, our network improves at least 2.46% on the easy-difficulty ${\mathrm{AOS}|}_{40}$ than the compared combining methods.

4.3. Ablation Experiments Based on KITTI Validation Set

To evaluate the individual components of our method, we conducted ablation experiments. As AFE-RCNN is proposed based on PV-RCNN, we regarded PV-RCNN as the baseline from which to compare the performance of each module. The ablation experiment results are shown in

Table 4. Ablation experiment results. The evaluation result of the car class on the KITTI validation set. The results of PV-RCNN were obtained based on official pre-trained models.

Method		Proposed Modules			3D Detection (mAP|40)
		RDA	MSAA	VA	Easy		Moderate		Hard
					Car	Cyclist	Car	Cyclist	Car	Cyclist
PV-RCNN Baseline					92.10	89.10	84.36	70.38	82.48	66.17
Ours	Ours1	√			92.18	90.59	84.82	72.15	82.60	67.69
	Ours2	√	√		92.27	91.21	85.05	75.01	82.91	70.36
	AFE-RCNN	√	√	√	92.35	91.54	85.31	75.39	83.08	70.79
		RDA	MSAA	VA	Orientation Estimation (AOS|40)
PV-RCNN Baseline					98.25	97.04	94.26	82.11	92.07	77.88
AFE-RCNN		√	√	√	98.18	94.50	94.29	84.24	94.01	81.01

. Note that we used the same metric as the ablation experiment in PV-RCNN, i.e., the ${\mathrm{mAP}|}_{40}$ was calculated with 40 recall positions. The experiments show that each component of AFE-RCNN has a performance gain on the original model.

Effect of RDA module: The RDA module learns the relevance of features in both channel and spatial dimensions, and assigns weights to features to suppress background points by filtering high-value features. This module improves the precision of subsequent processing. As shown in Table 4, Ours1 outperforms PV-RCNN with 0.08%, 0.46%, and 0.12% of ${\mathrm{mAP}|}_{40}$ gains at each difficulty level for cars. For cyclists, Ours1 can also outperform PV-RCNN with 1.49%, 1.77%, and 1.52% of ${\mathrm{mAP}|}_{40}$ gains at each difficulty level, which proves that the RDA module can achieve obvious improvements.

Effect of MSAA: We performed three levels of feature extraction on the raw point clouds. The MSAA ensures the robustness of sampling in sparse regions of the point cloud by multi-scale grouping and sampling and finds the interaction between points in a local region base on the adaptive feature adjustment method. According to Table 4, Ours2 outperforms Ours1 with 0.09%, 0.23%, and 0.31% of ${\mathrm{mAP}|}_{40}$ gains at each difficulty level for cars, and Ours2 can also outperform Ours1 with 0.62%, 2.86%, and 2.67% of ${\mathrm{mAP}|}_{40}$ gains at each difficulty level for cyclists. Besides, compared with PV-RCNN, the 3D detection ${\mathrm{mAP}|}_{40}$ of Ours2 at each difficulty level of the car has improved by 0.17%, 0.69%, and 0.43%, respectively. At the same time, the 3D detection ${\mathrm{mAP}|}_{40}$ of Ours2 at each difficulty level of the cyclist has improved by 2.11%, 4.63%, and 4.19%, respectively. We note that the gain brought by the MSAA module is the most obvious, especially for cyclists. The possible reasons can be summarized in the following two aspects. Firstly, directly processing the raw point clouds can effectively reduce the information loss. Secondly, benefiting from the multi-scale network structure and the method of adaptive feature adjustment, the points in the local region can better aggregate the regional contextual information.

Effect of VA module: To avoid the problem in which the correlation among the vertices of the bounding box is ignored, we optimized the loss function based on DIoU Loss. Experiments show that the AFE-RCNN has a certain improvement when compared with Ours2; the 3D detection ${\mathrm{mAP}|}_{40}$ of AFE-RCNN at each difficulty level of the car has improved by 0.08%, 0.26%, and 0.17%, respectively. At the same time, the 3D detection ${\mathrm{mAP}|}_{40}$ of AFE-RCNN at each difficulty level of the cyclist has improved by 0.33%, 0.38%, and 0.43%.

Compared with PV-RCNN: According to Table 4, our AFE-RCNN has 10 evaluation indicators (total 12 indicators) better than the PV-RCNN. For the 3D detection ${\mathrm{mAP}|}_{40}$, AFE-RCNN outperforms PV-RCNN with 0.25%, 0.95%, and 0.60% of ${\mathrm{mAP}|}_{40}$ gains at each difficulty level for cars, and AFE-RCNN has improved by 2.44%, 5.01%, and 4.62% for cyclists than PV-RCNN. For the orientation estimation, AFE-RCNN outperforms PV-RCNN with 0.03% and 1.94% of ${\mathrm{AOS}|}_{40}$ gains at moderate and hard difficulty levels for cars, and outperforms PV-RCNN with 2.13% and 3.13% of ${\mathrm{AOS}|}_{40}$ gains at moderate and hard difficulty levels for cyclists.

In addition, we conducted experiments to analyze the attention mechanisms. As shown in

Table 5. Comparison results for the attention blocks. The evaluation results for the car and cyclist on the KITTI validation set. The results of PV-RCNN were obtained based on official pre-trained models.

Method	$3\mathbf{D}\mathbf{Detection}({\mathbf{mAP}|}_{40})$
	Easy		Moderate		Hard
	Car	Cyclist	Car	Cyclist	Car	Cyclist
PV-RCNN Baseline	92.10	89.10	84.36	70.38	82.48	66.17
+DA block	91.17	88.34	82.98	70.21	82.55	65.13
+RDA block	92.18	90.59	84.82	72.15	82.60	67.69

, we found that the performance of our method becomes worse after directly adding the dual attention block (DA block). For cyclists in the moderate level, the ${\mathrm{mAP}|}_{40}$ of baseline with the DA block is decreased by 0.27% compared to the baseline. However, when we used the residual module to connect the dual attention block (RDA block), the baseline with RDA block has 1.77% ${\mathrm{mAP}|}_{40}$ gains for the moderate level of cyclists. Compared to adding the DA block directly, the performance of adding the RDA block can be significantly improved, which can prove the residual connection is effective for the attention module.

From the results in Table 2, Table 3 and Table 4, we observe that the performance for cyclists has a significant improvement. As these objects are small and may have fewer sampling points, the limited number of points makes it hard to provide sufficient semantic information. We fully excavated the relevance and contextual semantic information among the key points to make the feature information in the cyclist class more representative. However, there is little improvement for the performance of the easy difficulty level, which means that our method has a limited effect with enough object points.

4.4. Evaluation on the KITTI Validation Set

For most related papers that have conducted experiments on the KITTI validation set, they only provide the ${\mathrm{mAP}|}_{11}$ (calculated with 11 recall positions) of 3D detection for cars. We followed this metric [45] and compare our AFE-RCNN with the state-of-the-art methods in

Table 6. 3D detection results on the KITTI validation set. The ${\mathrm{mAP}|}_{11}$ is calculated with 11 recall positions. Note that the red, green, and blue represent the results of ranking first, second, and third, respectively.

IOU Thresh Car = 0.7	$3\mathbf{D}\mathbf{Detection}\left({\mathbf{mAP}|}_{11}\right)$
	Car
	Easy	Moderate	Hard
PointRCNN [7] (CVPR 2019)	88.88	78.63	77.38
3DSSD [9] (CVPR 2020 Oral)	89.71	79.45	78.67
SA-SSD [43] (CVPR 2020)	90.15	79.91	78.78
Part-A2 [11] (TPAMI 2020)	89.47	79.47	78.54
CIA-SSD [14] (AAAI 2021)	90.04	79.81	78.80
TANet [42] (AAAI 2020)	87.52	76.64	73.86
DVFENet [45] (Neurocomputing 2021)	89.81	79.52	78.35
H^23D R-CNN [38] (TCSVT2021)	89.63	85.20	79.08
SE-SSD [46] (CVPR 2021)	90.21	86.25	79.22
PV-RCNN [17] (CVPR 2020)	89.34	83.69	78.70
AFE-RCNN(Ours)	89.61	83.99	79.18

.

Our AFE-RCNN has 2 evaluation indicators (total 3 indicators) ranked Top 3 for all comparison algorithms. Considering the number of indicators ranked Top 3, our method has a certain advantage. Our method does not outperform in all indicators, which may be related to the different divisions of the training and validation sets. When compared with the baseline method, our AFE-RCNN outperforms the PV-RCNN at all difficulty levels. The 3D detection ${\mathrm{mAP}|}_{11}$ has improved by 0.27%, 0.30%, and 0.48%, respectively.

4.5. Qualitative Analysis on the KITTI Dataset

We compared the visualization results of AFE-RCNN with PV-RCNN. To further demonstrate the advantages of our method, we compared PointRCNN [7] as well. As shown in Figure 6, the red boxes represent the prediction boxes, while the green and yellow represent the ground truth boxes for car and cyclist, respectively. It can be seen that PointRCNN has missing detection in the cyclist class, whereas the PV-RCNN and our AFE-RCNN have more comprehensive detection. This is because the method of PointRCNN obtains less feature information and lacks correlation as it simply processes the raw point clouds only. Our AFE-RCNN uses the feature enhancement method to extract the features of the raw point clouds and BEV. These two features are then fused with voxel features for key point features. The rich feature information leads to a better detection performance. Furthermore, the prediction boxes of AFE-RCNN are closer to the ground truth than PV-RCNN and PointRCNN.

4.6. Validation on the Waymo Open Dataset

We used our AFE-RCNN on the Waymo dataset to illustrate the effectiveness of AFE-RCNN, and the results are shown in

Table 7. Experimental results for Waymo validation set. The mean average precision (mAP) and the mean average precision weighted by heading (mAPH) [2] are two official evaluation indicators.

Method	Vehicle (LEVEL 1)		Vehicle (LEVEL 2)		Cyclist (LEVEL 1)		Cyclist (LEVEL 2)
	mAP	mAPH	mAP	mAPH	mAP	mAPH	mAP	mAPH
Centerpoint [47] (CVPR 2021)	65.98	65.40	57.98	57.47	63.05	61.68	60.72	59.39
PointPillars [15] (CVPR 2019)	65.06	64.29	57.11	56.41	49.95	43.47	48.05	41.82
SECOND [10] (Sensors 2018)	65.83	65.12	57.80	57.17	47.44	38.68	45.65	37.21
PV-RCNN [17] (CVPR 2020)	71.23	70.53	62.58	61.96	58.87	40.29	56.36	39.11
AFE-RCNN (Ours)	71.23	70.54	62.62	61.99	59.69	43.14	57.44	41.51

. We validated two classes of objects, i.e., vehicle and cyclist, and compared the results of two official evaluation indicators, i.e., mAP and mAPH [2]. According to Table 7, we can make the following conclusions.

Comparison with the state-of-the-art methods. For the vehicle, our AFE-RCNN has a relatively outstanding performance. The AFE-RCNN outperforms the compared algorithms with at least 5.14% and 4.52% mAPH gains on two 3D object detection difficulty levels. For the cyclist, the performance of our AFE-RCNN is worse than Centerpoint [47]. However, the performance of AFE-RCNN is obviously better than the other compared methods. Since our AFE-RCNN uses the anchor-based method to generate the detection boxes, when the object is tilted to a certain angle, the anchor-based method cannot coordinate this angle well, which results in a large error. The Centerpoint [47] uses a center-based method that takes advantage of the rotational invariance of the point clouds to solve this problem.

Compared with PV-RCNN. For the vehicle, the performance of our AFE-RCNN has small improvements over the PV-RCNN. However, for the cyclist, our AFE-RCNN outperforms the compared algorithms with 2.85% and 2.40% mAPH gains on two 3D object detection difficulty levels. This proves that our proposed method for the improvement of PV-RCNN is effective.

4.7. Efficiency and Robustness Analysis of the Proposed Algorithm

To verify the efficiency of the proposed algorithm, the running times of our algorithm on the validation sets of the KITTI and Waymo open datasets are shown in

Table 8. The running times of the algorithms on the validation sets of KITTI and Waymo open datasets.

Method	Running Time (s)
	KITTI	Waymo Open Dataset
PV-RCNN Baseline	0.0286	0.1033
AFE-RCNN (Ours)	$0.0345\uparrow$	$0.1102\uparrow$

. Compared to the baseline, the running time of our algorithm has a millisecond increase. However, according to the Table 2, Table 7 and Table 8, the proposed AFE-RCNN can obtain better detection accuracy with a small increase in time, which can achieve promising performance in both accuracy and efficiency for the 3D object detection task.

To analyze the class number of the objects’ impact on the efficiency and precision of our algorithm, we conducted experiments for multiple objects on the validation set of KITTI. The results are shown in

Table 9. Experimental results of multiple object detection on the validation set of KITTI.

Class Number of the Objects	$\mathbf{Moderate}{\mathbf{mAP}|}_{40}$			Running Time (s)
1 class	Car			0.0279
	85.33
2 classes	Car	Cyclist		0.0286
	85.29	75.39
3 classes	Car	Cyclist	Pedestrian	0.0289
	85.31	75.39	59.67

. Taking the car category as an example, the ${\mathrm{mAP}|}_{40}$ values in three groups of experiments are 85.33%, 85.29%, and 85.31%. The class number of the objects has a very small impact on the performance (${\mathrm{mAP}|}_{40}$) of our AFE-RCNN. Meanwhile, the running time of our AFE-RCNN for different class numbers of object detection has only a slight effect (<0.001 s). The results demonstrate that our algorithm maintains a good efficiency and robustness when performing multiple object detection.

In addition, to further verify the robustness of the proposed algorithm, we conducted the experiments for pedestrian detection on the validation set of KITTI. As shown in

Table 10. Experimental results of pedestrian object detection on the validation set of KITTI.

Method	Pedestrian
	Easy		Moderate		Hard
	mAP|40	AOS|40	mAP|40	AOS|40	mAP|40	AOS|40
PV-RCNN Baseline	62.71	67.82	54.49	62.17	49.88	58.07
AFE-RCNN	66.19	71.94	59.67	66.28	54.97	63.02

, our AFE-RCNN outperforms the baseline on all the difficulty levels. Taking the performance on the moderate difficulty level as an example, our AFE-RCNN outperforms the baseline with 5.18% ${\mathrm{mAP}|}_{40}$ gains and 4.11% ${\mathrm{AOS}|}_{40}$ gains. The results prove that our AFE-RCNN is robust and can achieve good performance in small object detection.

5. Conclusions

We have proposed a 3D object detection method known as AFE-RCNN. Firstly, to generate high-quality proposal frames, we focused on the relationship between features in both channel dimension and spatial dimension through the RDA module. Secondly, we used multi-scale sampling and adaptive methods to robustly obtain fine local information on key points. Finally, based on the rich feature information on the key points, we completed the proposal refinement through the VA module, which guarantees correlation among the vertices of the box proposals. When compared with the state-of-the-art methods, AFE-RCNN can achieve a comparable performance; especially the cyclist class with a relatively small size has a significant improvement. However, to satisfy the requirements of time-sensitivity in object detection tasks, we need to study a lightweight method that can improve the detection precision while also ensuring computational efficiency. In the future, we will try to combine point clouds with 2D image information and replace the excessive detail processing of point clouds by a lightweight image feature extraction method. The gain from lightweight image processing to replace the much-detailed processing of point clouds only is a potential research direction.