Depth Prior-Guided 3D Voxel Feature Fusion for 3D Semantic Estimation from Monocular Videos

Abstract

Existing 3D semantic scene reconstruction methods utilize the same set of features extracted from deep learning networks for both 3D semantic estimation and geometry reconstruction, ignoring the differing requirements of semantic segmentation and geometry construction tasks. Additionally, current methods allocate 2D image features to all voxels along camera rays during the back-projection process, without accounting for empty or occluded voxels. To address these issues, we propose separating the features for 3D semantic estimation from those for 3D mesh reconstruction. We use a pretrained vision transformer network for image feature extraction and depth priors estimated by a pretrained multi-view stereo-network to guide the allocation of image features within 3D voxels during the back-projection process. The back-projected image features are aggregated within each 3D voxel via averaging, creating coherent voxel features. The resulting 3D feature volume, composed of unified voxel feature vectors, is fed into a 3D CNN with a semantic classification head to produce a 3D semantic volume. This volume can be combined with existing 3D mesh reconstruction networks to produce a 3D semantic mesh. Experimental results on real-world datasets demonstrate that the proposed method significantly increases 3D semantic estimation accuracy.

1. Introduction

Three-dimensional (3D) semantic scene reconstruction is an important task in the fields of computer vision [1,2], involving the reconstruction of 3D scenes from image or video data while simultaneously assigning semantic labels to various parts of the scene [3]. This dual-task capability enables machines to understand and interact with their environment more effectively, facilitating applications in robot navigation [4], autonomous driving [5], augmented reality [6,7], and more. By providing detailed and comprehensive environmental understanding, 3D semantic scene reconstruction enhances machine perception and decision-making processes.

Traditionally, 3D semantic scene reconstruction has relied heavily on 3D data from sensors such as LiDAR and depth cameras [8,9]. While effective, these sensors are often expensive, bulky, and less portable, limiting their widespread use. Consequently, there is a growing interest in leveraging 2D video data, which are more readily available, cost-effective, and easier to handle.

Current methods for 3D semantic scene reconstruction from 2D video data often use the same set of features for both semantic estimation and geometry reconstruction. This method overlooks the distinct requirements of these tasks, semantic estimation benefits from global contextual information, while geometry reconstruction focuses on local feature matching, leading to suboptimal performance. One reasonable method is to apply 3D semantic segmentation algorithms to reconstruct meshes from video data. However, this method is not yet used by existing algorithms and has significant limitations. Although 3D semantic segmentation algorithms [10,11,12] typically perform well on high-quality point clouds sampled from 3D sensor data, point clouds generated from 2D video data via mesh reconstruction are often less accurate and noisier. The lack of geometric detail makes the direct application of 3D semantic segmentation infeasible, as models trained on high-quality 3D data cannot generalize to these less accurate reconstructions, resulting in significant performance degradation. Additionally, existing 3D semantic scene reconstruction methods [13,14] typically back-project 2D image features into 3D voxel volumes without considering occlusions or empty spaces, resulting in inaccurate feature allocations.

To address these issues, we propose a novel method that separates the features used for 3D semantic estimation from those used for 3D mesh reconstruction. Our method leverages a pretrained vision transformer (ViT) network [15], specifically DINOv2 [16], whic h was trained using self-supervised techniques to learn robust visual features without supervision. ViTs, such as DINOv2, utilize self-attention mechanisms to capture long-range dependencies and integrate global contextual information, making them particularly effective for semantic tasks.

We use DINOv2 to extract rich, contextual image features from 2D video frames. These features are then guided by depth priors estimated by a pretrained multi-view stereo (MVS) network during the back-projection process to ensure accurate feature allocation within the 3D voxel volume. This depth-guided method helps prevent irrelevant features from being assigned to voxels, addressing one of the main shortcomings of existing methods.

The back-projected features are averaged within each voxel to create coherent voxel features. This 3D feature volume, composed of unified voxel feature vectors, is then fed into a 3D U-Net proposed by Finerecon [17] with a semantic classification head to produce a 3D semantic volume. This semantic volume can be seamlessly integrated with existing mesh reconstruction networks to generate a 3D semantic mesh, offering a modular method that improves semantic estimation accuracy without being affected by the precision of the reconstructed mesh.

The main contributions of our work are as follows:

• We propose a feature separation method for 3D semantic estimation and 3D mesh reconstruction, addressing the interference between these tasks in existing methods;
• We introduce the use of depth priors from a pretrained MVS network to guide the allocation of image features in 3D voxels, improving the accuracy of feature placement;
• We utilize a pretrained ViT, which is adept at capturing global contextual information through self-supervised learning techniques, enhancing the robustness and accuracy of semantic features;
• Our method produces a 3D semantic volume that can be combined with reconstructed 3D scene that obtained from any existing 3D reconstruction method, enhancing flexibility and applicability;
• We present experimental results if real-world datasets that demonstrate significant improvements in 3D semantic estimation accuracy, validating the effectiveness of the proposed method.

The remainder of this paper is organized as follows: In Section 2, we review related work in the field of 3D semantic scene reconstruction. Section 3 details the proposed method. Section 4 presents the experimental results and analysis, demonstrating the effectiveness of the proposed method. Section 5 discusses the proposed method based on experimental results. Finally, in Section 6, we conclude the paper and discuss potential future directions.

2. Related Works

2.1. Three-Dimensional Semantic Segmentation Using Three-Dimensional Data Obtained from Three-Dimensional Sensor

The field of 3D semantic segmentation has been extensively researched, particularly with the advent of advanced neural network architectures such as PointNet [18] and the sparse convolutional neural network (CNN) [19]. These techniques focus on processing 3D point clouds derived from range sensors, enriched with additional attributes such as RGB values and normal vectors for each point. These methods assign semantic labels to each point in the point cloud, requiring precise 3D geometric inputs. The methods [8,20,21,22] that accept RGB-D depth images as input are also researched a lot. These methods operate in a similar fashion where they first adopt an off-the-shelf 2D semantic segmentation network to obtain 2D semantic labels per pixel, and then fuse the depth and labels into a voxel volume. Various methods for fusing semantic labels into the voxel volume have been proposed. For instance, Pham et al. [8] developed a method that constructs 3D meshes with voxel hashing from depth images and subsequently enhances the initial semantic labeling through super-voxel clustering and a high-order conditional random field to improve labeling coherence. Additionally, Cavallari and Di Stefano [21] and Rosinol et al. [22] have explored techniques that use depth images from 3D sensors to back-project 2D semantic labels into 3D voxels using a voting mechanisms. The voxel label is decided by the maximum voted semantic label. Reliable geometry representations (e.g., the 3D point cloud, or depth images from 3D sensors) are required to guarantee the performance of the algorithms. When applied to estimated geometric representations, there is a noticeable decline in segmentation performance due to the imprecise estimated geometry.

2.2. Three-Dimensional Semantic Scene Reconstruction from Videos

When leveraging RGB video inputs for 3D semantic reconstruction, methods bifurcate them into object-level and scene-level 3D semantic scene reconstructions. Object-level reconstruction focuses on identifying and reconstructing individual objects within a scene with semantic context, whereas scene-level reconstruction aims to semantically interpret and reconstruct the entire scene comprehensively. This distinction underscores the complexity and potential of 3D semantic reconstruction from RGB video inputs.

2.2.1. Object-Level 3D Semantic Reconstruction Methods

Object-level semantic reconstruction methods typically integrate detection and subsequent reconstruction processes [4,6,23,24,25,26,27]. Notable methods include Vid2CAD [6] and Mask2CAD [23], which detect objects within a scene and align CAD models from pre-existing databases to these detected objects in 3D space. However, a significant limitation of these methods is the mismatch between retrieved CAD models and the actual objects present in the scene, which may not always align perfectly in terms of shape and features.

To mitigate reliance on CAD databases, adaptive deformation strategies [24] have been developed, allowing a template shape to better match the object in the scene. The choice of the initial template is crucial, as significant differences between the geometric structure of the template and the target object can lead to inaccuracies. For example, using a spherical template [25] to model objects with complex features can result in significant inaccuracies, highlighting the challenge of template selection for diverse object geometries.

Additionally, implicit representation-based methods for mesh reconstruction [4,26,27] have emerged. These methods learn to reconstruct the mesh conditioned on object categories, significantly impacted by the learned shape priors. A common limitation of these methods is their tendency to successfully reconstruct only objects within the categories they were trained on [4,28]. Objects belonging to previously unseen categories often remain unreconstructed, and structural elements of the scene, such as walls and floors, are typically not reconstructed. This highlights a significant challenge in object-level reconstruction: the need for methods that generalize beyond their training data to encompass a broader array of object types and scene components.

2.2.2. Scene-Level 3D Semantic Reconstruction Methods

The domain of 3D semantic reconstruction from posed videos, particularly within the deep learning community, is relatively unexplored. Early pioneering efforts, such as Atlas [13], integrated the estimation of 3D geometry and semantic information from voxel features derived from a 3D CNN network. The initial approach to feature aggregation involved averaging back-projected multi-view image features without determining their relative importance, resulting in a noisy representation due to the indiscriminate fusion of multi-view information. To address this, CDRNet [29] incorporated a gated recurrent unit (GRU) for a more nuanced integration of multi-view features, selectively emphasizing more relevant features to mitigate noise issues.

Despite these advancements, existing methods often back-project 2D image features into 3D voxel volumes without considering occlusions or empty spaces, leading to inaccurate feature allocations. Moreover, directly applying 3D semantic segmentation algorithms to meshes reconstructed from video data can be problematic due to differences in reconstruction accuracy. These issues necessitate a more refined method for feature allocation and semantic estimation.

3. Three-Dimensional Semantic Volume Estimation via Depth Prior-Guided Three-Dimensional Voxel Feature Fusion

3.1. Overview

Figure 1 provides an overview of the proposed method. This method outputs a 3D semantic volume, where each voxel is classified with a semantic label. The semantic volume can then be combined with existing mesh reconstruction networks to generate a reconstructed 3D semantic scene from a posed RGB video.

Semantic volume estimation is divided into three steps: keyframe processing, depth prior-guided 3D voxel feature fusion, and 3D semantic volume estimation. Given a sequence of keyframes ${\left\{I_i\right\}}_{i=1}^N$ from a video, a pretrained MVS network predicts depth priors ${\left\{D_i\right\}}_{i=1}^N$ for all keyframes. An image encoder is used to extract the image features ${\left\{F_i\right\}}_{i=1}^N$ for all keyframes. These image features are then unprojected to specific voxels in a global 3D voxel volume, guided by the estimated depth priors, to form a 3D voxel feature volume. A 3D semantic volume decoder, consisting of a 3D CNN with the same structure as the one used in Finerecon and a classification head, is then applied to the 3D voxel feature volume to obtain voxel-wise semantic estimation results.

3.2. Keyframe Processing

The image encoder network is composed of pretrained ViT-Small (ViT-S), Dinov2, and a CNN composed of three Conv-BN-ReLU blocks, as shown in Figure 2. The Dinov2-extracted feature consists of 384 channels. Direct utilization of this feature in a 3D task is computationally expensive and can lead to excessive memory usage. The CNN module serves two main purposes: first, it reduces the channel size, thus alleviating memory and computational burden; second, it refines the feature to better suit the 3D semantic task.

During training, Dinov2 is frozen, and only the CNN module is trained. This approach reduces the computational load, leading to faster convergence. The depth priors are obtained from an off-the-shelf MVS network [30].

3.3. Depth Prior-Guided 3D Voxel Feature Fusion

This section introduces the multi-view semantic feature fusion process depicted in Figure 3. In this process, the estimated depth images ${\left\{D_i\right\}}_{i=1}^N$ from the MVS module guide the mapping of image features ${\left\{F_i\right\}}_{i=1}^N$ to a 3D voxel volume.

Assuming $\left(x, y, z\right)$ is the coordinate of voxel $v_j$ in the world coordinate system, the mapped pixel coordinates $\left(\widehat{x}, \widehat{y}\right)$ and the projected depth, $\widehat{z}$, on keyframe $i$ can be computed as follows:

$$\left[\begin{array}{c}\widehat{x}\\ \widehat{y}\\ \widehat{z}\end{array}\right]=\mathsf{\Pi }K\left[\begin{array}{cc}{R}_i& t_i\\ 0& 1\end{array}\right]\left[\begin{array}{c}\begin{array}{c}x\\ y\\ z\end{array}\\ 1\end{array}\right],$$

(1)

where $\mathsf{\Pi }$ represents perspective mapping; $R_i$ and $t_i$ are the extrinsic parameters of the camera for keyframe $i$. $K$ represents the intrinsic parameters of camera.

The feature assignment of keyframe $i$ for voxel $v_j$ can be computed as follows:

$$V_i(:, x, y, z)=\left\{\begin{array}{c}{F}_i\left(:, \widehat{x}, \widehat{y}\right) if \left|\widehat{z}-D_i(\widehat{x}, \widehat{y})\right|\le \delta \\ 0 otherwise \end{array}\right.,$$

(2)

where $:$ denotes the slicing operation, $D_i(\widehat{x}, \widehat{y})$ is the sampled depth value from the estimated depth image of keyframe $i$, and $F_i\left(:, \widehat{x}, \widehat{y}\right)$ is the sampled image feature on the image feature plane. The threshold, $\delta$, is defined as the absolute distance between the estimated depth value and the projected depth value of the voxel on the 3D frame. It determines whether $F_i\left(:, \widehat{x}, \widehat{y}\right)$ will be assigned to voxel $v_j$.

Voxels $v_{j-1}$, $v_j$, and $v_{j+1}$ are neighboring voxels along a camera ray. they project the same position onto the image plane of keyframe $i$, and the projected values of them between the estimated depth value in that position are smaller than $\delta$, an thus an identical sampled image feature is added to the them, as shown in Figure 3. This depth prior-guided approach helps to avoid the allocation of image features to voxels that should be empty or occluded by ensuring that only voxels with depth values close to the estimated depth are considered for image feature assignment. It can also mitigate the impact of depth estimation inaccuracies by preventing voxels that represent actual 3D geometry from being ignored.

Image features from other keyframes are also back-projected in the same manner, resulting a multi-view feature volume ${\left\{V_i\right\}}_{i=1}^N$. By averaging these back-projected features at the voxel level, a fused semantic feature volume is constructed. This volume is then refined by a 3D CNN [17] with a semantic classification head for 3D semantic estimation.

3.4. Implementation Details

Keyframe selection is based on a relative camera pose, following the method described in [31]. The voxel size of the voxel volume is set to 4 cm, consistent with the settings used in most existing mesh reconstruction networks [13,14,32]. The threshold, $\delta$, is set to 24 cm. The network is implemented in PyTorch 2.0 and both trained and tested on a Linux server equipped with six RTX A6000 graphic cards. During training, voxel volume chunks of scenes with fixed dimensions (3.84 m × 3.84 m × 2.24 m) are randomly cropped and fed into the network. The training process lasts approximately 22 h. For each cropped chunk, 20 keyframes are selected to generate the 3D voxel feature volume.

The loss used for training the network is defined as follows:

$${\mathcal{L}}_s=\frac{1}{\left|V\right|}\sum _{v\in V}CE\left(S^{*}\left(v\right),S\left(v\right)\right)$$

(3)

where ${\mathcal{L}}_s$ is the 3D semantic loss, $S^{*}$ is the ground truth 3D semantic volume, $S$ is the estimated 3D semantic volume, and $CE$ denotes the cross entropy loss.

4. Experimental Results and Analysis

4.1. Datasets and Metrics

The experiments were conducted on two popular existing datasets for 3D semantic scene segmentation, namely the ScanNet v2 dataset [33] and the SceneNN dataset [34]. The ScanNet v2 dataset was divided into training, validation, and test sets according to the official splits, with the ground truth for the test set hidden. This dataset is a common benchmark in 3D semantic scene reconstruction studies, including works like Atlas, Finerecon and Panopticfusion [20]. It includes 1513 scans of 707 scenes, covering various types of real-world indoor spaces, such as offices, apartments and bathrooms. Following existing works [13,29], the network is trained solely on the training set, and evaluated offline on the validation set and online on the test set. Semantic labels from the reconstructed semantic mesh are transferred to the ground truth mesh using K-nearest neighbor (KNN) to obtain the semantic results. This approach ensures that the comparison is based on consistent geometric alignment between the reconstructed and ground truth meshes. The benchmark provides semantic estimation results for 20 indoor object categories using the mean intersection over union (mIoU) metric. To further verify the generalization capability, we also report evaluation results on the SceneNN dataset, which includes 100 diverse scenes, captured using a different sensor to ScanNet v2. Both datasets provide comprehensive coverage of real-world indoor scenarios.

4.2. Experimental Results of ScanNet v2 Dataset

4.2.1. Quantitative Comparisons

Table 1. Evaluation of 3D semantic estimation on the ScanNet v2 test set.

Methods	mIoU
Atlas [13]	0.340
NeuralRecon [14] + Semantic-heads *	0.279
VoRTX [32] + Semantic-heads *	0.132
CDRNet [29] *	0.391
Ours	0.648

* The results are reported by CDRNet.

presents the mIoU results for 3D semantic estimation evaluated for the ScanNet v2 test set reported by the ScanNet v2 benchmark. Our method achieves a significantly higher mIoU score of 0.468, outperforming all of the compared methods. Among the methods compared, CDRNet achieves the second-highest mIoU score of 0.391. NeuralRecon [14] with added semantic heads and VoRTX [32] with added semantic heads, which are reimplemented by CDRNet, achieve mIoU scores of 0.279 and 0.132, respectively. Atlas achieves an mIoU score of 0.340. Although Atlas offers better performance compared with NeuralRecon and VoRTX, it still falls significantly short compared with our method.

Figure 4 illustrates the per-category evaluation of 3D semantic estimation on the ScanNet v2 validation set, comparing our method with Atlas. Our method consistently outperforms Atlas across all categories, demonstrating its superior generalization capability and robustness. We chose to compare only with Atlas due to the availability of detailed dataset processing methods and pretrained models in their code repository, which enabled a fair and reproducible comparison.

4.2.2. Qualitative Comparisons

Figure 5 presents qualitative comparisons of 3D semantic estimation results on the ScanNet v2 validation set, showcasing the ground truth (GT), Atlas, and our proposed method. For a more intuitive comparison, we directly compare the 3D semantic estimation results on the 3D ground truth mesh, which helps to mitigate the influence of mesh quality variations. Semantic labels from the reconstructed semantic mesh are transferred to the ground truth mesh using KNN to obtain the semantic results. The red boxes highlight areas where our method correctly labels objects that Atlas misclassifies. Our method demonstrates superior performance, accurately identifying and labeling various objects such as cabinets, beds, and shelves, resulting in a closer resemblance to the ground truth.

4.3. Generalization Experimental Results on SceneNN Dataset

4.3.1. Quantitative Comparisons

Table 2. Evaluation of 3D semantic estimation using the SceneNN dataset.

Methods	mIoU
Atlas [13] *	0.314
NeuralRecon [14] + semantic-heads *	0.159
CDRNet [29] *	0.367
Ours	0.518

* Results reported by CDRNet.

presents the evaluation of 3D semantic estimation using the SceneNN dataset, demonstrating the generalization capability of various methods trained exclusively on the ScanNet v2 training set without further fine-tuning on the SceneNN dataset. Despite the lack of specific adaptation to SceneNN, our method achieves the highest mIoU score of 0.518, significantly outperforming the compared methods. Atlas attains an mIoU of 0.314, that of NeuralRecon with added semantic heads reaches 0.159, and CDRNet achieves an mIoU of 0.367. These results, reported by CDRNet, underscore our method’s superior ability to be generalized across different datasets. The substantial margin by which the performance of our approach surpasses that of Atlas and NeuralRecon highlights its robustness in producing accurate semantic estimations in diverse and unseen environments.

4.3.2. Qualitative Comparisons

Figure 6 provides visual comparisons between the performance of Atlas and that of our proposed method on the SceneNN dataset for generalization experiments. The red boxes highlight areas where our method correctly labels objects that Atlas misclassifies.

In the first row, our method accurately labels the chairs and tables, particularly in the edge areas between these objects, while Atlas fails to do so. The second row demonstrates that Atlas misclassifies a chair as a sofa and fails to recognize the table, whereas our method correctly identifies both objects. In the third row, Atlas does not output correct labels for the cabinet and counter, which are accurately labeled by our method. The last row shows that Atlas fails to correctly estimate the refrigerator and sink, and also over-segments the table, partially classifying it as a table and partially classifying it as a cabinet. In contrast, our method provides accurate semantic labels for all these objects, demonstrating superior generalization capability on the SceneNN dataset.

5. Discussion

This study presents a novel method for 3D semantic estimation from monocular videos, leveraging depth priors to guide voxel feature fusion. Our approach effectively addresses the allocation of image features within 3D voxel volumes and separates the features used for semantic estimation from those used for geometry reconstruction. The experimental results for the ScanNet v2 and SceneNN datasets highlight our method’s superior performance and generalization capability compared with existing methods like Atlas and CDRNet. The use of depth priors and a pretrained ViT for image feature extraction contributes significantly to the robustness of our approach, ensuring accurate semantic estimation even in challenging scenarios. Despite these promising results, the reliance on depth priors means that our method’s performance may be influenced by the quality of depth estimation. In future work, incorporating depth reliability information, which indicates the likelihood of depth existence in different voxels, could enhance the robustness of the network. Additionally, the inference speed of the proposed method is slow due to the computationally intensive processes involved in depth prior estimation and ViT image feature extraction. Future work could also explore the development of more lightweight depth estimation networks and image feature extraction methods to improve the speed of inference and move towards real-time performance. Additionally, investigating dynamic feature recognition, such as identifying and tracking moving objects (e.g., humans), could further enhance the applicability of our method in real-world scenarios.

6. Conclusions

In this paper, we introduced a depth prior-guided 3D voxel feature fusion method for 3D semantic estimation from monocular videos. Our method achieves significant improvements in semantic estimation accuracy and demonstrates strong generalization capabilities across different datasets. The integration of depth priors and a pretrained ViT enhances the robustness and accuracy of our approach, making it a valuable contribution to the field of 3D semantic scene reconstruction. Future work may focus on optimizing the feature fusion process to handle the variations in different depth priors.