MS3D: A Multi-Scale Feature Fusion 3D Object Detection Method for Autonomous Driving Applications

Abstract

With advancements in autonomous driving, LiDAR has become central to 3D object detection due to its precision and interference resistance. However, challenges such as point cloud sparsity and unstructured data persist. This study introduces MS3D (Multi-Scale Feature Fusion 3D Object Detection Method), a novel approach to 3D object detection that leverages the architecture of a 2D Convolutional Neural Network (CNN) as its core framework. It integrates a Second Feature Pyramid Network to enhance multi-scale feature representation and contextual integration. The Adam optimizer is employed for efficient adaptive parameter tuning, significantly improving detection performance. On the KITTI dataset, MS3D achieves average precisions of 93.58%, 90.91%, and 88.46% in easy, moderate, and hard scenarios, respectively, surpassing state-of-the-art models like VoxelNet, SECOND, and PointPillars.

1. Introduction

The rapid advancement of autonomous driving technology demands precise environmental perception and reliable object detection. Traditional 2D methods, including cameras and LiDAR, are limited by lighting and occlusion, affecting their performances in complex and adverse conditions. Cameras struggle in low-light and high-contrast environments, while 2D LiDAR demonstrates excellent distance measurement precision and robustness against interference. However, due to its two-dimensional nature, it is inadequate for spatial navigation. Thus, 3D object detection using LiDAR point clouds has become a key focus in autonomous driving research.

Unlike traditional methods, 3D object detection using LiDAR point clouds offers notable advantages. It operates in all weather conditions, unaffected by lighting variations, ensuring reliable performance day and night and in adverse weather such as rain or fog. Its high spatial resolution allows for the early detection of distant objects, enhancing safety by providing more reaction time. LiDAR’s robustness to lighting changes and occlusions ensures consistent performance in complex scenarios. Additionally, the compact nature of LiDAR data facilitates efficient processing and computation, meeting the real-time demands of autonomous driving.

Overall, LiDAR-based 3D object detection is essential for autonomous vehicle perception systems, providing accurate and reliable detection crucial for safe and efficient driving. Early work, such as VoxelNet [1], introduced an end-to-end framework using voxel features extracted through 3D convolution, establishing a foundation for further research. SECOND [2] optimized efficiency with 3D sparse convolution, while PointPillars [3] enhanced detection speed by mapping voxel features to pseudo-images. SASSD [4] improved training efficiency with an auxiliary network for position and category predictions. PVRCNN [5] strengthened feature extraction by concatenating 3D sparse convolutional features. VoxelRCNN [6] increased accuracy with voxel ROI pooling. The Waymo and Google team developed the multi-view fusion algorithm [7], integrating perspective views and point clouds to boost detection accuracy. TANet [8] advanced object detection with a TA (Triple Attention) Module and CFR (Coarse-to-Fine Regression) strategy. Researchers have built large-scale LiDAR point cloud datasets, such as KITTI [9], NuScenes [10], Waymo [11], and ONCE [12], providing extensive data for training and testing, thus enhancing algorithm performance and applicability.

Despite its advantages, 3D object detection using LiDAR point clouds faces several challenges. The sparsity of point cloud data hinders traditional algorithms’ ability to delineate object boundaries and detect small, distant objects. The unordered nature of point clouds complicates feature extraction, particularly in cases of overlapping or occluded objects. Additionally, noise in point clouds can introduce misleading information, leading to increased false detections and reduced accuracy, especially in complex environments.

To address these challenges, we propose Multi-Scale Feature Fusion for 3D Object Detection (MS3D). MS3D utilizes spatial segmentation and feature extraction [3] to efficiently handle sparse LiDAR point cloud data. The method incorporates a 2D Convolutional Neural Network (CNN) to refine object boundaries, and the Second Feature Pyramid Network (SecondFPN) is utilized to augment feature discriminability. This integration enables robust cross-scale feature fusion, enhancing the model’s resilience to noise and thereby significantly boosting detection precision and stability. The Adam optimizer is used for adaptive parameter updates, accelerating convergence and boosting both training efficiency and detection precision. Moreover, a positive sample filtering strategy [13] is adopted for data augmentation, focusing on the extraction of discriminative features to minimize noise interference.

This paper is structured as follows: Section 1 introduces LiDAR-based 3D object detection. Section 2 reviews related works in 3D object detection. Section 3 explains the MS3D architecture. Section 4 compares MS3D with existing algorithms and evaluates its performance. Section 5 summarizes contributions and suggests future research directions.

2. Related Works

Three-dimensional object detection algorithms are the cornerstone of autonomous driving systems, tasked with identifying and precisely localizing road targets such as pedestrians, vehicles, and obstacles from onboard sensor data. Compared to 2D object detection, they provide more precise information, including target categories, locations (center coordinates and dimensions), and poses (rotation angles, etc.), which are crucial for helping autonomous vehicles to make precise decisions in path planning and obstacle avoidance, ensuring the safety and efficiency of driving.

In recent years, there has been a shift in 3D object detection technology from traditional computer vision methods [14,15] that rely on manual feature extraction to deep learning-driven techniques. Traditional methods, primarily based on geometric analysis and machine learning techniques such as Support Vector Machines, are limited in their ability to handle complex dynamic scenes due to environmental changes and computational efficiency. In contrast, deep learning methods, through end-to-end feature learning, have achieved single-stage direct prediction, two-stage ROI refinement strategies, and data fusion across mono-modal and cross-modal algorithms, significantly improving detection accuracy and robustness. Some 3D object detection algorithms are listed in

Table 1. List of 3D object detection algorithms.

Abbreviation	Full Name
3D-SSD [16]	3D Single Shot MultiBox Detector
IA-SSD [17]	Improved Anchor-based Single Shot MultiBox Detector
VoxelNet [1]	VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection
SECOND [2]	Sparsely Embedded Convolutional Detection
PointPillars [18]	PointPillars: Fast Encoders for Object Detection from Point Clouds
CIA-SSD [13]	Contextualized Intermediate-level Anchoring for Single Shot 3D Object Detection
SA-SSD [4]	Scale-Aware Single Shot MultiBox Detector for 3D Object Detection
Diswot [19]	Discrete Wasserstein Objectives for Learning with Limited Data
LargeKernel3D [20]	Large Kernel Matters—Improving Semantic Segmentation by Global Convolutional Network
OcTr [21]	Occlusion-aware Transformer for 3D Object Detection
VoxelNeXt [22]	VoxelNeXt: Voxels for the Next Generation of 3D Object Detection
CasA [23]	Cascaded Alignment for 3D Object Detection
BtcDet [24]	Bird’s Eye View Object Detection with Localization Refinement
MSF [25]	Multi-Scale Feature Aggregation for 3D Object Detection
ConQueR [26]	ConQueR: Monocular 3D Object Detection by Construction and Query
MV3D [27]	Multi-View 3D Object Detection Network for Autonomous Driving
EPNet [28]	Efficient PointNet for 3D Object Detection in Point Clouds
PointPainting [29]	PointPainting: Sequential Fusion for 3D Object Detection
MVP [30]	Multi-View Prediction for 3D Object Detection from Monocular Images
LoGoNet [31]	Local and Global Network for 3D Object Detection from Point Cloud
SFD [32]	Sparse Feature Detection for Point Cloud Based 3D Object Detection
Virtual Conv [33]	Virtual Convolution for Efficient Point Cloud Processing

.

In the domain of single-stage 3D object detection, algorithms such as 3D-SSD [16], IA-SSD [17], VoxelNet [1], SECOND [2], PointPillars [18], CIA-SSD [13], SA-SSD [4], Diswot [19], LargeKernel3D [20], OcTr [21], and VoxelNeXt [22] have achieved rapid and efficient target identification through the optimization of point cloud downsampling, voxelization methods, auxiliary networks, distillation techniques, innovative data structures, and sparse inspection architectures. Two-stage methods like CasA [23], BtcDet [24], MSF [25], and ConQueR [26] have enhanced detection accuracy for occluded and fast-moving targets through fine-grained feature extraction and regression optimization. Cross-modal methods including MV3D [27], EPNet [28], PointPainting [29], MVP [30], LoGoNet [31], SFD [32], and Virtual Conv [33] have further enhanced detection robustness by fusing LiDAR and visual image data.

LiDAR point cloud technology employs the echo time measurement of laser pulses to generate high-resolution point cloud models, which is vital for precise 3D spatial data acquisition in autonomous driving systems. This methodology surpasses 2D imaging by offering increased resilience against environmental lighting and weather variations, ensuring stable operation in low-visibility conditions. The technology’s high-resolution data acquisition and processing capabilities enable autonomous systems to effectively manage occlusions and detect objects across multiple scales, enhancing their perceptual abilities.

With the continuous advancement of deep learning technology, the integration of LiDAR point cloud and 3D object detection methods has significantly enhanced the accuracy and efficiency of detection in the field of autonomous driving. Methods based on voxels, such as VoxelNet [1], effectively handle large-scale point cloud data through regular three-dimensional voxel grid segmentation and the application of 3D Convolutional Neural Networks (CNNs). Meanwhile, point cloud-based methods, such as PointNet [34] and PointNet++ [35], directly process irregular point clouds and learn their local and global features, ensuring high-precision object detection. Pillar-based methods, such as PointPillars [3], enhance real-time detection by converting point cloud data to a pseudo-image plane and combining it with Convolutional Neural Networks. Additionally, the introduction of the Transformer architecture [3], leveraging its powerful attention mechanism for modeling, further captures the global relationships and detailed features within point cloud data, significantly improving the overall performance of 3D object detection.

3. Proposed Method

We introduce the 3D object detection method MS3D. Initially, we delineate the fundamental architecture of the network. Subsequently, we present the pivotal technique for voxelizing point cloud data. Following this, we elaborate on the SecondFPN feature pyramid network. Finally, we introduce the loss function.

3.1. Network Architecture

This paper introduces MS3D, a multi-scale feature fusion 3D object detection method for autonomous driving, as illustrated in Figure 1.

First, a spatial segmentation strategy divided the point cloud into uniform 3D cells, termed Pillars. Each Pillar represents a 3D cell, with each containing $\mathrm{M}$ points. Each point featured nine dimensions: 3D coordinates $\left(\mathrm{X},\mathrm{Y},\mathrm{Z}\right)$, reflection intensity $\mathrm{r}$, offset from the Pillar center $\left({\mathrm{X}}_{\mathrm{C}},{\mathrm{Y}}_{\mathrm{C}},{\mathrm{Z}}_{\mathrm{C}}\right)$, and offset from the grid center $\left({\mathrm{X}}_{\mathrm{P}},{\mathrm{Y}}_{\mathrm{P}}\right)$.

Points within each Pillar were either randomly sampled or padded with zero-value points to form a uniform data structure $\left(\mathrm{D},\mathrm{N},\mathrm{P}\right)$, where $\mathrm{D}=9$ is the nine-dimensional features, $\mathrm{N}$ is the point count threshold, and $\mathrm{P}$ is the total number of Pillars. PointNet processed these Pillar features, converting them into a $\mathrm{C}$ channel feature representation $\left(\mathrm{C},\mathrm{N},\mathrm{P}\right)$ that was then transformed into a pseudo-image with a size of $\mathrm{H}\times \mathrm{W}\times \mathrm{C}$ through max pooling, where $\mathrm{H}$ and $\mathrm{W}$ are the height and width of the grid and $\mathrm{C}$ is the number of channels.

A 2D Convolutional Neural Network backbone extracted multi-scale features from the pseudo-image. The network consisted of two sub-networks: one for top-down feature extraction at different resolutions, and another that performed feature sampling via deconvolution and concatenates features from three different-sized feature maps.

Following the backbone, a SecondFPN was integrated to refine feature extraction further and enhance multi-scale detection capabilities.

For 3D object detection, an SSD (Single Shot MultiBox Detector) was used. The SSD head defined $\mathrm{K}$ anchor boxes in the feature maps, each with varying sizes and aspect ratios to accommodate multi-scale detection needs. Regression predicted each anchor box’s $\mathrm{Z}$-axis coordinate ${\mathrm{Z}}_{\mathrm{g}}$ and height $\mathrm{h}$, constructing precise 3D bounding boxes: $\left\{\left({\mathrm{X}}_{\mathrm{g}},{\mathrm{Y}}_{\mathrm{g}},{\mathrm{Z}}_{\mathrm{g}}\right),\left(\mathrm{l},\mathrm{w},\mathrm{h}\right),\mathsf{\theta },\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e},\mathrm{c}\right\}$. Here, $\left({\mathrm{X}}_{\mathrm{g}},{\mathrm{Y}}_{\mathrm{g}},{\mathrm{Z}}_{\mathrm{g}}\right)$ is the bounding box center, $\mathrm{l}$ is the length, $\mathrm{w}$ is the width, $\mathrm{h}$ is the height, $\mathsf{\theta }$ is the rotation angle relative to the $\mathrm{x}\mathrm{y}$-plane, score is the confidence of the bounding box containing the object, and $\mathrm{c}$ is the object class (e.g., vehicle, pedestrian).

The model training was optimized using the Adam optimizer [36], which adjusted learning rates adaptively based on parameter gradients, enhancing training efficiency and model performance.

3.2. Voxelization

The voxel-based encoding strategy preprocessed point cloud data as follows:

Voxel Grid Construction: The point cloud dataset $\mathrm{P}$ was divided into a spatial grid with dimensions $\mathrm{X}\times \mathrm{Y}\times \mathrm{Z}$, with voxel height $\mathrm{H}$ defined as the difference between the maximum and minimum values in the vertical coordinate. On the $\mathrm{x}\mathrm{y}$-plane, the point cloud was segmented into $\mathrm{N}\times \mathrm{M}$ cylindrical voxels using a grid size of $\mathrm{p}\times \mathrm{q}$, where $\mathrm{N}=\mathrm{X}/\mathrm{p}$ and $\mathrm{M}=\mathrm{Y}/\mathrm{q}$.

Voxel Representation: Each point ${\mathrm{p}}_{\mathrm{i}}\in \mathrm{P}$ was characterized by its 3D coordinates $\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}}\right)$ and reflection intensity ${\mathrm{r}}_{\mathrm{i}}$. The point cloud $\mathrm{P}$ was discretized into voxels $\mathrm{v}$, with each voxel containing a set of points $\{{\mathrm{p}}_{\mathrm{i}}=\left[{\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}},{\mathrm{r}}_{\mathrm{i}}\right],\mathrm{i}=1,\dots ,\mathrm{N}\mid {\mathrm{p}}_{\mathrm{i}}\in \mathrm{v}\}$. The number of points per voxel was normalized to ${\mathrm{N}}_{\mathrm{m}\mathrm{a}\mathrm{x}}$ through sampling or padding.

Feature Vector Construction: Each cylindrical voxel ${\mathrm{P}}_{\mathrm{k}}$ in the $\mathrm{x}\mathrm{y}$-plane, with a base size of $\mathrm{d}\mathrm{x}\times \mathrm{d}\mathrm{y}$ and fixed height $\mathrm{H}$, aggregated internal points to form a feature vector ${\mathrm{f}}_{\mathrm{k}}$. This vector included global coordinates $\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}}\right)$, reflection intensity ${\mathrm{r}}_{\mathrm{i}}$, and local offsets $\left({\mathrm{x}}_{\mathrm{c}\mathrm{i}},{\mathrm{y}}_{\mathrm{c}\mathrm{i}},{\mathrm{z}}_{\mathrm{c}\mathrm{i}}\right)$ relative to the voxel center, represented as ${\mathrm{f}}_{\mathrm{k}}=[\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}},{\mathrm{z}}_{\mathrm{i}},{\mathrm{r}}_{\mathrm{i}},{\mathrm{x}}_{\mathrm{c}\mathrm{i}},{\mathrm{y}}_{\mathrm{c}\mathrm{i}},{\mathrm{z}}_{\mathrm{c}\mathrm{i}}\right)\mid \forall {\mathrm{p}}_{\mathrm{i}}\in {\mathrm{P}}_{\mathrm{k}}]$.

Pseudo-Image Reconstruction: The feature vector ${\mathrm{f}}_{\mathrm{k}}$ was then reconstructed into a pseudo-image $\mathrm{I}$ of size $\mathrm{H}\times \mathrm{W}\times \mathrm{C}$, where $\mathrm{C}$ is the number of feature channels and $\mathrm{H}$ and $\mathrm{W}$ are the grid dimensions on the $\mathrm{x}\mathrm{y}$-plane. This transformation from unstructured point cloud data to a structured feature representation enhanced the input for subsequent models, retaining spatial structure and improving environmental perception accuracy and computational efficiency in autonomous driving systems.

3.3. SecondFPN

The SecondFPN network employed a carefully designed feature fusion strategy to integrate high-level semantic information from deeper layers with high-resolution features from shallower layers, enhancing semantic richness, as shown in Figure 2.

The process was as follows:

Feature Extraction: Input feature maps $\mathrm{I}$ were processed through deep convolutional layers ${\mathcal{C}}^{\left({\mathrm{k}}_{\mathrm{d}},{\mathrm{s}}_{\mathrm{d}},{\mathrm{p}}_{\mathrm{d}}\right)}$ to extract features, where ${\mathrm{k}}_{\mathrm{d}}$, ${\mathrm{s}}_{\mathrm{d}}$, and ${\mathrm{p}}_{\mathrm{d}}$ denote the kernel size, stride, and padding, respectively.

$${\mathrm{F}}^{\left(1\right)}={\mathcal{C}}^{\left({\mathrm{k}}_{\mathrm{d}},{\mathrm{s}}_{\mathrm{d}},{\mathrm{p}}_{\mathrm{d}}\right)}\left(\mathrm{I}\right)$$

(1)

Dimensionality Reduction: To optimize channel numbers and reduce computational parameters, 1 × 1 convolutions ${\mathcal{C}}^{\left({\mathrm{k}}_1,{\mathrm{s}}_1,{\mathrm{p}}_1\right)}$ were applied to the second and third feature maps ${\mathrm{F}}^{\left(2\right)}$ and ${\mathrm{F}}^{\left(3\right)}$, where ${\mathrm{k}}_1=$1, ${\mathrm{s}}_1=1$, and ${\mathrm{p}}_1=0$.

$${\mathrm{F}}^{\left(2\right)}={\mathcal{C}}^{\left({\mathrm{k}}_1,{\mathrm{s}}_1,{\mathrm{p}}_1\right)}\left({\mathrm{F}}^{\left(1\right)}\right)$$

(2)

$${\mathrm{F}}^{\left(3\right)}={\mathcal{C}}^{\left({\mathrm{k}}_1,{\mathrm{s}}_1,{\mathrm{p}}_1\right)}\left({\mathrm{F}}^{\left(2\right)}\right)$$

(3)

Upsampling and Fusion: The lower-resolution, semantically rich fourth feature map ${\mathrm{F}}^{\left(4\right)}$ was upsampled using ${\mathcal{U}}^{\left({\mathrm{k}}_{\mathrm{u}}\right)}$ to match the resolution of the lower layer feature map ${\mathrm{F}}^{\left(2\right)}$, where ${\mathrm{k}}_{\mathrm{u}}$ is the upsampling kernel size. Feature fusion was performed by element-wise addition.

$${\mathrm{F}}^{\left(5\right)}={\mathrm{F}}^{\left(2\right)}+{\mathcal{U}}^{\left({\mathrm{k}}_{\mathrm{u}}\right)}\left({\mathrm{F}}^{\left(4\right)}\right)$$

(4)

Prediction: The fused feature map ${\mathrm{F}}^{\left(5\right)}$ underwent a convolutional prediction layer ${\mathcal{P}}^{\left({\mathrm{k}}_{\mathrm{p}},{\mathrm{s}}_{\mathrm{p}},{\mathrm{p}}_{\mathrm{p}}\right)}$ for object classification and localization, where ${\mathrm{k}}_{\mathrm{p}}$, ${\mathrm{s}}_{\mathrm{p}}$, and ${\mathrm{p}}_{\mathrm{p}}$ are the kernel size, stride, and padding, respectively.

$$\mathrm{P}=\mathcal{P}\left({\mathrm{k}}_{\mathrm{p}},{\mathrm{s}}_{\mathrm{p}},{\mathrm{p}}_{\mathrm{p}}\right)\left(\dot{{\mathrm{F}}^{\left(5\right)}}\right)=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{v}\left({\mathrm{F}}^{\left(5\right)}, {\mathrm{k}}_{\mathrm{p}},{\mathrm{s}}_{\mathrm{p}},{\mathrm{p}}_{\mathrm{p}}\right)$$

(5)

The SecondFPN fusion strategy enhanced feature representation and increased the network’s sensitivity to multi-scale objects, demonstrating exceptional performance in complex autonomous driving and robotic navigation scenarios.

3.4. Loss Function

The loss function comprised three components: localization loss ${\mathrm{L}}_{\mathrm{l}\mathrm{o}\mathrm{c}}$, orientation classification loss ${\mathrm{L}}_{\mathrm{dir}}$, and classification loss ${\mathrm{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}$. Each 3D bounding box was represented by a seven-dimensional vector $(\mathrm{x},\mathrm{y},\mathrm{z},\mathrm{w},\mathrm{l},\mathrm{h},\mathsf{\theta })$, where $(\mathrm{x},\mathrm{y},\mathrm{z})$ are the center coordinates, $(\mathrm{w},\mathrm{l},\mathrm{h})$ are the dimensions, and $\mathsf{\theta }$ is the orientation angle.

Localization Loss ${\mathrm{L}}_{\mathrm{l}\mathrm{o}\mathrm{c}}$: The localization loss used the Smooth $\mathrm{L}1$ loss function [37], combining $\mathrm{L}1$ and $\mathrm{L}2$ loss characteristics to measure the discrepancy between predicted and ground truth bounding boxes. It balanced smoothness for small errors with robustness for larger deviations.

$${\mathrm{L}}_{\mathrm{l}\mathrm{o}\mathrm{c}}=\sum _{\mathrm{b}\in \left(\mathrm{x},\mathrm{y},\mathrm{z},\mathrm{w},\mathrm{l},\mathrm{h},\mathsf{\theta }\right)}\mathrm{S}\mathrm{m}\mathrm{o}\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{L}1\left(\mathsf{\Delta }\mathrm{b}\right)$$

(6)

where $\mathsf{\Delta }\mathrm{b}$ represents the residual between predicted $\widehat{\mathrm{b}}$ and ground truth $\mathrm{b}$.

The Smooth $\mathrm{L}1$ loss function [37], as depicted in Figure 3, is defined as

$$\mathrm{SmoothL}1\left(\mathsf{\Delta }\mathrm{b}\right)=\left\{\begin{array}{ll}0.5{\left(\mathsf{\Delta }\mathrm{b}\right)}^2& \mathrm{if}\left|\mathsf{\Delta }\mathrm{b}\right|<1\\ \left|\mathsf{\Delta }\mathrm{b}\right|-0.5& \mathrm{otherwise}\end{array}\right.$$

(7)

Orientation Classification Loss ${\mathrm{L}}_{\mathrm{dir}}$: The orientation classification loss employed the Softmax function to optimize the model’s orientation predictions and minimize classification errors as follows:

$${\mathrm{L}}_{\mathrm{dir}}=-\sum _{\mathrm{i}=1}^{\mathrm{N}}\sum _{\mathrm{c}=1}^{\mathrm{C}}{\mathrm{y}}_{\mathrm{i},\mathrm{c}}\mathrm{l}\mathrm{o}\mathrm{g}\left({\mathrm{P}}_{\mathrm{i},\mathrm{c}}\right)$$

(8)

where $\mathrm{N}$ is the number of instances in a batch, $\mathrm{C}$ is the number of orientation classes, ${\mathrm{y}}_{\mathrm{i},\mathrm{c}}$ is an indicator variable (1 if the true orientation of object $\mathrm{i}$ is $\mathrm{c}$, otherwise 0), and ${\mathrm{P}}_{\mathrm{i},\mathrm{c}}$ is the predicted probability for orientation class $\mathrm{c}$.

Classification Loss ${\mathrm{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}$: The classification loss, using Focal Loss [37], addressed class imbalance by focusing on hard-to-classify examples while down-weighting easy ones as follows:

$${\mathrm{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}=-{\sum }_{\mathrm{a}}{\mathsf{\alpha }}_{\mathrm{a}}(1-{\mathrm{p}}^{\mathrm{a}}{)}^{\mathsf{\gamma }}\log {\mathrm{p}}^{\mathrm{a}}$$

(9)

where ${\mathrm{p}}^{\mathrm{a}}$ is the predicted probability for class $\mathrm{a}$, ${\mathsf{\alpha }}_{\mathrm{a}}$ is a balancing factor, and $\mathsf{\gamma }$ is a modulation term used to reduce the emphasis on well-classified examples.

The total loss $\mathrm{L}$ was a weighted sum of these components:

$$\mathrm{L}={\displaystyle \frac{1}{{\mathrm{N}}_{\mathrm{p}\mathrm{o}\mathrm{s}}}}\left({\mathsf{\beta }}_{\mathrm{l}\mathrm{o}\mathrm{c}}{\mathrm{L}}_{\mathrm{l}\mathrm{o}\mathrm{c}}+{\mathsf{\beta }}_{\mathrm{c}\mathrm{l}\mathrm{s}}{\mathrm{L}}_{\mathrm{c}\mathrm{l}\mathrm{s}}+{\mathsf{\beta }}_{\mathrm{d}\mathrm{i}\mathrm{r}}{\mathrm{L}}_{\mathrm{d}\mathrm{i}\mathrm{r}}\right)$$

(10)

where ${\mathrm{N}}_{\mathrm{p}\mathrm{o}\mathrm{s}}$ is the number of positive anchors. The optimal balancing factors were determined through cross-validation and a comparative analysis of performance metrics across various parameter settings. For example, the alpha_a parameter was optimally set at 0.25 following a stringent experimental protocol. Initially, the dataset underwent categorical distribution scrutiny to detect under-represented classes. Subsequently, a five-fold cross-validation scheme was implemented, adjusting alpha_a incrementally from 0.1 to 0.5 in steps of 0.05. Analysis of F1 scores and precision-recall curves across different alpha_a settings confirmed that the model reached its optimal classification equilibrium at alpha_a = 0.25, characterized by a significant boost in recall for minority classes without compromising precision. Moreover, the incorporation of an early stopping criterion averted overfitting, ensuring the model’s robust generalization capabilities. In practice, the balancing factors were set to ${\mathsf{\alpha }}_{\mathrm{a}}$ = 0.25, $\mathsf{\gamma }$ = 2, ${\mathsf{\beta }}_{\mathrm{l}\mathrm{o}\mathrm{c}}$ = 2, ${\mathsf{\beta }}_{\mathrm{c}\mathrm{l}\mathrm{s}}$ = 1, and ${\mathsf{\beta }}_{\mathrm{d}\mathrm{i}\mathrm{r}}$ = 0.1 to optimize model performance.

4. Experiments and Analysis

An in-depth exploration of experimental design and performance evaluation is conducted. Section 4.1 delineates the characteristics of the selected dataset. Section 4.2 introduces the performance metrics for the experiment. Section 4.3 provides the configuration of the experimental environment. Section 4.4 meticulously analyzes the experimental results to assess the algorithm’s performance.

4.1. Datasets

The KITTI dataset [9] is a key benchmark in autonomous driving, featuring diverse environments such as urban, rural, and highway settings, as shown in Figure 4. It includes varied objects such as vehicles and pedestrians with challenges like occlusion, rotation, multi-object tracking, and scale variations. The dataset is divided into three difficulty levels: easy (clear backgrounds, minor occlusion), medium (dense object overlap), and hard (complex scenes with severe occlusion). MS3D is evaluated on 7481 KITTI samples, with 3712 for training and 3769 for validation, to assess the method’s generalization and robustness. The KITTI-trained 3D object detection algorithm [38] achieves effective real-world application, underscoring the KITTI dataset’s crucial role in enhancing the algorithm’s ability to adapt to complex 3D environments.

4.2. Evaluation Metrics

AP (average precision): This metric evaluates model accuracy by averaging precision at various recall levels. A higher AP indicates superior detection accuracy.

F1 Score: The harmonic mean of precision and recall, reflecting the balance between avoiding false positives and false negatives. Higher F1 scores denote improved overall performance.

IoU (Intersection over Union): This metric measures the overlap between predicted and ground truth bounding boxes. IoU values range from 0 to 1, with higher values signifying better localization accuracy. This study employs an IoU threshold of 0.5.

Frame Rate: This metric quantifies the number of frames processed per second in video, image, or point cloud analysis. It is measured in Hertz (Hz), where 1 Hz equals one frame per second.

4.3. Experimental Setup

The point cloud data are restricted to the spatial dimensions of x-axis [0, 70.4] meters, y-axis [−40, 40] meters, and z-axis [−3, 1] meters, focusing on the vehicle’s front. Voxelization is performed using 0.16 m × 0.16 m × 4 m dimensions, creating vertical columns with a height of 4 m and a grid resolution of 0.16 m × 0.16 m in the xy-plane. Each voxel is limited to 32 points, totaling 40,000 voxels. Experiments are conducted on an Ubuntu 18.04 LTS system with PyTorch 1.7.1, utilizing CUDA 11.0 and cuDNN 8.0.4 for GPU acceleration.

To enhance model generalization, extensive data augmentation is employed, including global rotation, scaling, and flipping of the point clouds, as well as local adjustments near ground truth targets. Additionally, a positive sample filtering strategy [13] reduces the weights of difficult samples with significant overlap, improving training quality and focusing the model on more discriminative features to boost detection accuracy and minimize false positives.

4.4. Experiment Analysis

4.4.1. Accuracy Analysis

Table 2. A performance comparison of MS3D with other methods in the easy category.

Methods	Frame Rate (HZ)	AP			F1Score
		BBOX	BEV	3D	BBOX	BEV	3D
VoxelNet	4.4	0.69	0.82	0.81	0.47	0.49	0.50
SECOND	20	0.78	0.89	0.87	0.51	0.53	0.55
PointPillars	62	0.80	0.90	0.90	0.53	0.55	0.56
MS3D	60	0.84	0.94	0.93	0.54	0.56	0.58

,

Table 3. A performance comparison of MS3D with other methods in the moderate category.

Methods	Frame Rate (HZ)	AP			F1Score
		BBOX	BEV	3D	BBOX	BEV	3D
VoxelNet	4.4	0.67	0.76	0.75	0.46	0.48	0.51
SECOND	20	0.76	0.85	0.84	0.50	0.53	0.54
PointPillars	62	0.78	0.864	0.863	0.53	0.55	0.55
MS3D	60	0.82	0.92	0.91	0.54	0.56	0.56

and

Table 4. A performance comparison of MS3D with other methods in the difficult category.

Methods	Frame Rate (HZ)	AP			F1Score
		BBOX	BEV	3D	BBOX	BEV	3D
VoxelNet	4.4	0.63	0.74	0.73	0.45	0.47	0.49
SECOND	20	0.74	0.78	0.80	0.49	0.52	0.53
PointPillars	62	0.75	0.84	0.83	0.52	0.54	0.53
MS3D	60	0.77	0.89	0.88	0.53	0.55	0.55

presents a comparative analysis of MS3D against the classical algorithms VoxelNet, SECOND, and PointPillars across easy, moderate, and hard difficulty levels on the KITTI dataset.

In Table 2, MS3D demonstrates superior performance in the easy category, achieving AP (average precision) scores of 0.84 for BBOX, 0.94 for BEV, and 0.93 for 3D, outperforming competing methods. Notably, MS3D’s 3D AP surpasses VoxelNet by 12%, highlighting its strength in 3D object detection. In addition, MS3D’s F1 Scores of 0.54, 0.56, and 0.58 in the BBOX, BEV, and 3D categories, respectively, also outperformed other methods, especially in the 3D category, with its F1 Score being 8% higher than Voxelnet, indicating a superior balance between precision and recall.

In the moderate category of Table 3, the AP of MS3D reaches 0.82, 0.92, and 0.91 for the BBOX, BEV, and 3D object detection categories, respectively. This represents improvements of (0.15, 0.16, 0.16), (0.06, 0.07, 0.07), and (0.04, 0.06, 0.05) over VoxelNet, SECOND, and PointPillars, respectively. Additionally, the F1 Scores of MS3D are 0.54, 0.56, and 0.56 for the BBOX, BEV, and 3D categories, which are also better than other methods. Indeed, this indicates that MS3D has a significant advantage in terms of detection accuracy and generalization ability in the medium mode.

In the difficult category, MS3D demonstrates superior accuracy over other methods. As indicated in Table 4, MS3D achieves higher AP scores across the BBOX, BEV, and 3D metrics. Specifically, for the 3D metric, MS3D attains an AP of 0.88, outperforming VoxelNet, SECOND, and PointPillars by 15%, 8%, and 5%, respectively. Furthermore, MS3D consistently delivers higher F1 Scores across the BBOX, BEV, and 3D metrics, underscoring its exceptional performance in complex detection tasks.

Across three diverse categories, both PointPillars and MS3D excel in real-time performance, achieving 62 Hz and 60 Hz, respectively, demonstrating high-efficiency point cloud processing. MS3D, while maintaining rapid processing speeds, significantly enhances detection efficacy with its multi-scale feature fusion approach, offering superior accuracy and robustness, particularly in complex environments, despite a slightly lower frame rate than PointPillars.

4.4.2. Representative Cases

Due to PointPillars’ exceptional performance, ranking second among the evaluated algorithms, this section primarily presents a visual comparison between MS3D (our model) and PointPillars. Figure 5a, Figure 6a and Figure 7a represent the point cloud data labels.

• Easy Category: Minor Occlusion of Small Objects

In scenarios with simple categories, detecting small objects is challenging due to slight occlusions and sparse point cloud data, as shown in Figure 5. Sparse datasets lead to insufficient point density for small or distant targets, complicating detection. The pronounced data voids within sparse datasets lead to a marked reduction in informative content, thereby complicating the extraction of meaningful patterns during the learning process. This difficulty is particularly acute for small or remotely positioned targets, which are intrinsically under-represented in the dataset. The sparsity of the data further compounds the scarcity of feature information critical for detection, thereby intensifying the challenges in achieving precise identification and classification of these objects. Figure 5b demonstrates that traditional methods like PointPillars struggle with missed detections, such as failing to identify a distant small vehicle obscured by closer ones, highlighting limitations in handling occlusions and multi-scale detection.

MS3D mitigates these issues by employing advanced feature fusion techniques, combining upsampling and downsampling to integrate multi-scale features effectively. This strategy enhances the model’s ability to focus on critical features and reduces the impacts of less relevant ones, improving detection performance for small objects. Additionally, 2D convolutions enhance the model’s ability to interpret 2D projections of point cloud data, enabling more accurate detection of small targets under slight occlusions. As illustrated in Figure 5c, MS3D’s feature fusion approach yields higher detection confidence, accurately identifying occluded distant small vehicles and significantly improving small object detection.

2.

Moderate Category: Overlapping and Occluded Multi-Object

In moderate-category scenarios, multi-target overlap and occlusion pose significant challenges due to the unstructured nature of point cloud data, which complicates feature extraction, as shown in Figure 6. For instance, at an intersection, multiple vehicles overlap, with one black car nearly obscured on its left side. As depicted in Figure 6b, traditional methods like PointPillars fail to detect this occluded black car, resulting in detection failures.

In contrast, MS3D, leveraging the SecondFPN, effectively detects the occluded black car. The SecondFPN’s multi-scale feature extraction and fusion mechanisms enable the precise identification of targets across various scales, enhancing detection performance in complex occlusion scenarios. Additionally, the Adam optimizer’s adaptive learning rate facilitates handling the complexities of such scenes. As shown in Figure 6c, MS3D’s feature fusion capabilities significantly improve detection accuracy, effectively mitigating interference from overlapping targets.

3.

Difficult Category: Complex Backgrounds and Intricate Trajectories

In challenging scenarios with complex backgrounds and diverse, dynamically moving targets such as pedestrians, bicycles, and vehicles, detecting objects is highly demanding, as shown in Figure 7. Point cloud data can be contaminated by noise, leading to inaccurate detections. Traditional methods like PointPillars often struggle with distinguishing between background and target features, resulting in false positives and missed detections. As illustrated in Figure 7b, traditional approaches may misclassify traffic signs, curved lane dividers, tall flower beds, and multi-track pedestrians as vehicles. Moreover, a fast-moving red car and a partially occluded blue car (due to backlighting and partial visibility) are not detected, highlighting the limitations of these methods in handling background complexity and incomplete imagery.

In contrast, MS3D effectively detects the fast-moving red car and the severely occluded blue car while minimizing false positives, as shown in Figure 7c. Leveraging SecondFPN’s multi-scale feature extraction and fusion capabilities, MS3D significantly enhances point cloud feature representation, enabling precise focus on critical features and accurate detection of partially visible targets. The incorporation of 2D convolutions improves the model’s interpretation of 2D projections from point cloud data, thereby enhancing detection performance for severely occluded and dynamically moving objects. Additionally, the Adam optimizer’s efficient parameter update strategy facilitates rapid convergence and high accuracy in complex scenes. Collectively, these advancements markedly improve MS3D’s performance in challenging and intricate environments.

Despite advancements, the detection of pedestrians on multiple trajectories, especially in occluded scenarios such as individuals propelling bicycles, continues to pose challenges in terms of missed detections. While MS3D has enhanced detection performance through multi-scale feature fusion, the single fusion strategy struggles to adequately extract crucial information to address occlusion challenges, particularly in cases like cyclists. Additionally, the limitations of 2D Convolutional Neural Networks in boundary and detail processing, compounded by the small size of pedestrian targets, sparsity of point clouds, and motion artifacts caused by complex trajectories, all contribute to the complexity of feature extraction and recognition, resulting in persistent issues of missed detections. Subsequent research endeavors will be directed towards the refinement of motion distortion mitigation strategies and the identification and compensation of motion artifacts to bolster the detection precision for pedestrians in multi-track walking scenarios.

5. Conclusions

MS3D pushes the envelope in 3D object detection for autonomous driving by harmonizing a unique blend of spatial segmentation, sophisticated feature extraction, and integration with 2D CNNs, which together elevate detection efficiency and precision. The innovative SecondFPN module enhances feature representation, markedly reinforcing detection stability. Additionally, the strategic use of data augmentation coupled with positive sample filtering is crafted to amplify discriminative feature learning. The training regimen is fine-tuned with the Adam optimizer and an adaptive learning rate approach, assuring rapid convergence and high accuracy. Benchmark tests on the KITTI dataset reveal that MS3D surpasses state-of-the-art methods, including VoxelNet, SECOND, and PointPillars, particularly in detecting small, distant objects, dealing with scenes of multiple overlapping objects, and maneuvering through complex backgrounds, thus advancing the field of 3D object detection technology.

Despite its current capabilities, MS3D encounters difficulties in detecting pedestrians with complex trajectories in intricate urban environments. Future endeavors will concentrate on developing more sophisticated techniques for motion distortion mitigation strategies and the identification and compensation of motion artifacts to improve the identification and segmentation of pedestrians under adverse conditions. Furthermore, multimodal perception strategies, including teacher–student distillation networks [39,40,41,42], will integrate cross-modal data with point cloud characteristics to enhance environmental comprehension, thus bolstering the generalization and robustness of pedestrian detection in complex scenarios.

Building on this foundation, we will leverage large-scale autonomous driving models [43,44,45] to enhance our 3D object detection algorithms, enabling them to interpret and navigate complex urban environments more effectively. This will contribute to an overall improvement in autonomous driving performance.

To validate the accuracy and reliability of the enhanced MS3D algorithms, we will conduct comprehensive evaluations on complex datasets. Our approach will integrate feature layer fusion, multilevel feature extraction, and sophisticated data augmentation strategies, all within an advanced evaluation framework. Utilizing TensorFlow, JAX’s XLA, and Caffe2, we will refine the algorithms for real-time operation and mobile integration, thereby enhancing the performance metrics in autonomous driving vision technology.