Interactive Efficient Multi-Task Network for RGB-D Semantic Segmentation

Abstract

Semantic segmentation is significant for robotic indoor activities. However, relying solely on RGB modality often leads to poor results due to limited information. Introducing other modalities can improve performance but also increases complexity and cost, making it unsuitable for real-time robotic applications. To address the balance issue of performance and speed in robotic indoor scenarios, we propose an interactive efficient multitask RGB-D semantic segmentation network (IEMNet) that utilizes both RGB and depth modalities. On the premise of ensuring rapid inference speed, we introduce a cross-modal feature rectification module, which calibrates the noise of RGB and depth modalities and achieves comprehensive cross-modal feature interaction. Furthermore, we propose a coordinate attention fusion module to achieve more effective feature fusion. Finally, an instance segmentation task is added to the decoder to assist in enhancing the performance of semantic segmentation. Experiments on two indoor scene datasets, NYUv2 and SUNRGB-D, demonstrate the superior performance of the proposed method, especially on the NYUv2, achieving 54.5% mIoU and striking an excellent balance between performance and inference speed at 42 frames per second.

1. Introduction

In recent years, improving robot perception capabilities [1,2] has become a hotspot in various applications. Recent research employs sensors such as lasers or LIDAR [3,4], but the absence of visual information limits optimal performance. Consequently, semantic segmentation [5], which effectively utilizes visual data, has gained prominence. It aims to classify images at the pixel level, assigning each pixel to specific categories.

With the emergence of deep learning, an increasing number of high-performance models [6,7,8,9] have shown remarkable results, gradually replacing traditional semantic segmentation methods. However, in complex indoor scenes, relying solely on the RGB modality is insufficient for refined segmentation due to the lack of object boundary and geometric information. However, depth data, which contains rich position, contour, and geometric information, is easily collected using RGB-D sensors such as Microsoft Kinect, making it an excellent modality. Thus, various approaches for RGB-D semantic segmentation [10,11,12] have been studied. Nonetheless, these methods possess limited interaction within cross-modality features and often ignore the impact of noise of both modalities.

Moreover, most RGB-D semantic segmentation works focus on feature fusion [13,14,15] between RGB and depth modality. Although these methods provide instructive fusion modules to integrate the two kinds of information, fully utilizing both modalities for detailed semantic segmentation remains a challenge.

On the other hand, the increased complexity introduced by depth information substantially raises inference time, making it challenging for real-time applications. In semantic segmentation, numerous studies [16,17,18] employ different techniques to minimize computational costs. However, for RGB-D semantic segmentation, most methods do not focus on the balance issue of performance and speed, causing slow inference speed.

To solve the above problems, we propose an interactive efficient multi-task RGB-D semantic segmentation network, IEMNet. By introducing a cross-modality feature rectification module [15], feature interaction and rectification are increased by leveraging channel-wise and spatial-wise correlations. In addition, we also propose a coordinate attention fusion module for capturing long-range interactions and achieving comprehensive feature fusion. Meanwhile, lightweight and efficient architecture [19] is utilized to ensure efficient inference speed. Without affecting inference speed, we also use a multi-task decoder [19], training the model with semantic segmentation and instance segmentation tasks simultaneously, further enhancing semantic segmentation performance. Evaluations of two indoor scene datasets NYUv2 [20] and SUNRGB-D [21] demonstrate that IEMNet achieves an optimal balance between performance and speed compared to other advanced methods.

2. Related Work

RGB-D semantic segmentation. Incorporating depth modality in RGB modality can increase segmentation performance. CNN-based RGB-D semantic segmentation has achieved great advances in recent years. Various methods have been proposed for higher accuracy in different aspects. CANet [10] employed a three-branch encoder comprising RGB, deep, and mixed branches, effectively complementing information via a co-attention module. PGDENet [11] utilized depth data with a dual-branch encoder and depth enhancement module. FRNet [12] proposed cross-level enriching and cross-modality awareness modules to boost representative information and obtain rich contextual information. FSFNet [13] introduced a symmetic cross-modality residual fusion module. CEN [14] proposed a novel feature fusion mechanism that replaces the current channel feature with a feature on the same channel from the other modality according to the scale factor of the batch normalization layer. CMX [15] performed a feature fusion module with a cross attention mechanism. Shapeconv [22] introduced a shape-aware convolutional layer to utilize the shape information from depth modality. SGNet [23] proposed spatial information-guided convolution to help integrate the RGB feature and 3D spatial information. The SA-Gate [24] employed a separation-and-aggregation gating operation for jointly filtering and recalibrating both modalities. Different from the previous works, we improve feature interaction and rectification with spatial and channel correlation and propose a coordinate attention fusion module to integrate features from RGB and depth modality.

Efficient semantic segmentation. The objective of efficient semantic segmentation is to achieve high-quality segmentation accuracy with low computational cost. Several efficient segmentation networks have been reported to satisfy this requirement. MiniNet [16] introduced a multi-dilated depthwise convolution with fewer parameters and better performance. WFDCNet [17] proposed a depthwise factorized convolution to separate dimension and improve computational efficiency. FANet [18] presented a fast attention module and manually added a downsampling layer in ResNet to reduce the computational cost. FRNet [25] used an asymmetric encoder–decoder architecture with factorized and regular blocks, making a trade-off between accuracy and speed. DSANet [26] employed a channel split and shuffle to reduce the computation and maintain higher segmentation accuracy. In this work, we utilize a shallow backbone and a simple architecture to ensure efficient inference speed.

3. Proposed Method

3.1. Overview

We propose an Interactive Efficient Multi-Task RGB-D Semantic Segmentation Network (IEMNet) following the EMSANet [19], as shown in Figure 1. The network extracts RGB and depth information through two encoder branches with ResNet as backbone, and replaces each 3 × 3 convolution of the original ResNet with a Non-Bottleneck-1D-Block (NBt1D) to improve performance. The NBt1D consists of a 3 × 1 convolution layer, a ReLU activation function, and a 1 × 3 convolution layer.

Moreover, the Cross-Modal Feature Rectification Module (CM-FRM) [15] is introduced and placed between two encoder branches to integrate and rectify information from both the modalities. In this way, the interacted and rectified features will be fed into the next stage to further promote feature extraction. Meanwhile, we also propose the coordinate attention fusion module (CAFM) to replace the RGB-D fusion module in EMSANet, enabling more comprehensive feature fusion through the coordinate attention mechanism, merging two feature maps into one, and sending it to the decoder via the skip connection.

Our decoder consists of two branches [19]: one for semantic segmentation and the other for instance segmentation. The semantic branch includes three decoder modules, a 3 × 3 convolution layer, and two learned upsampling blocks. It incorporates shallow structure with skip connections and applies 4× upsampling to reduce computational cost. Each decoder module contains a 3 × 3 convolution layer, three NBt1D blocks, and a learned upsampling block. Additionally, we calculate loss on a multi-scale to improve performance and training effectiveness. The instance branch is essentially the same as the semantic branch, except for the task heads. Instance segmentation results are represented by 2D gaussian heatmaps encoded from their centers and offset vectors pointing from each pixel to the corresponding instance center in the x and y directions.

3.2. Cross-Modal Feature Rectification Module

To address noise rectification and facilitate feature interaction, we introduces the Cross-Modality Feature Rectification Module (CM-FRM) [15], which calibrates features from both modalities at different stages in the encoder. The detailed structure of the CM-FRM is depicted in Figure 2, where it performs feature rectification in both channel and spatial dimensions by leveraging information from the other modality.

Channel-wise feature rectification. Given the features of two modalities ${\mathrm{RGB}}_{\mathrm{in}}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$ and ${\mathrm{Depth}}_{\mathrm{in}}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$, maximum and average pooling are first applied simultaneously across their channel dimensions, obtaining four output vectors. These vectors are concatenated and processed by a multi-layer perceptron (MLP) with a sigmoid activation function to obtain the channel weights, which are then split into ${\mathrm{W}}_{\mathrm{RGB}}^{\mathrm{C}}$ and ${\mathrm{W}}_{\mathrm{Depth}}^{\mathrm{C}}$. Then, they are multiplied with the input separately on channel dimension, obtaining the channel-wise rectificated features as follows:

$$\begin{array}{cc}\hfill & \hfill {\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{C}}={\mathrm{W}}_{\mathrm{Depth}}^{\mathrm{C}}\otimes {\mathrm{Depth}}_{\mathrm{in}},\hfill \\ \hfill & \hfill {\mathrm{Depth}}_{\mathrm{rec}}^{\mathrm{C}}={\mathrm{W}}_{\mathrm{RGB}}^{\mathrm{C}}\otimes {\mathrm{RGB}}_{\mathrm{in}},\hfill \end{array}$$

(1)

where ⊗ denotes channel-wise multiplication.

Spatial-wise feature rectification. Given the same features of two modalities ${\mathrm{RGB}}_{\mathrm{in}}$ and ${\mathrm{Depth}}_{\mathrm{in}}$, they are concatenated on channel dimension and processed by two convolutional layers and a ReLU function. A sigmoid activation function is then applied to obtain the feature map $\mathrm{F}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times 2}$. Afterwards, $\mathrm{F}$ is split into two spatial weight maps ${\mathrm{W}}_{\mathrm{RGB}}^{\mathrm{S}}$ and ${\mathrm{W}}_{\mathrm{Depth}}^{\mathrm{S}}$. Then, they are multiplied with the input separately on spatial dimension, obtaining the spatial-wise rectificated features as follows:

$$\begin{array}{cc}\hfill & \hfill {\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{S}}={\mathrm{W}}_{\mathrm{Depth}}^{\mathrm{S}}\ast {\mathrm{Depth}}_{\mathrm{in}},\hfill \\ \hfill & \hfill {\mathrm{Depth}}_{\mathrm{rec}}^{\mathrm{S}}={\mathrm{W}}_{\mathrm{RGB}}^{\mathrm{S}}\ast {\mathrm{RGB}}_{\mathrm{in}},\hfill \end{array}$$

(2)

where ∗ denotes spatial-wise multiplication.

The final rectificated features of CM-FRM are obtained as follows:

$$\begin{array}{cc}\hfill & \hfill {\mathrm{RGB}}_{\mathrm{out}}={\mathrm{RGB}}_{\mathrm{in}}+{\lambda }_{\mathrm{C}}{\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{C}}+{\lambda }_{\mathrm{S}}{\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{S}},\hfill \\ \hfill & \hfill {\mathrm{Depth}}_{\mathrm{out}}={\mathrm{Depth}}_{\mathrm{in}}+{\lambda }_{\mathrm{C}}{\mathrm{Depth}}_{\mathrm{rec}}^{\mathrm{C}}+{\lambda }_{\mathrm{S}}{\mathrm{Depth}}_{\mathrm{rec}}^{\mathrm{S}},\hfill \end{array}$$

(3)

where ${\lambda }_{\mathrm{C}}$ and ${\lambda }_{\mathrm{S}}$ are two hyperparameters, and both are set to 0.5 by default. Note that ${\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{C}}$ and ${\mathrm{RGB}}_{\mathrm{rec}}^{\mathrm{S}}$ are obtained by ${\mathrm{Depth}}_{\mathrm{in}}$ multiplying the corresponding weights. Utilizing the information of the other modality, the CM-FRM can achieve comprehensive feature rectification and interaction.

3.3. Coordinate Attention Fusion Module

Inspired by [27], we propose a coordinate attention fusion module (CAFM) that receives the rectificated features and applies coordinate attention for feature fusion. As shown in Figure 3, CAFM consists of two stages, the coordinate attention stage and the fusion stage. Through the coordinate attention mechanism, our network is able to capture long-range interactions and know which features to pay more attention to and which features to suppress.

Coordinate attention stage. Given the rectificated features ${\mathrm{RGB}}_{\mathrm{out}}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$ and ${\mathrm{Depth}}_{\mathrm{out}}\in {\mathbb{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$, spatially factorized adaptive pooling is first applied to both features to aggregate features along the two spatial directions, respectively, yielding four direction-aware feature maps. Then, we concatenate the feature maps on spatial dimension after the reshape operation and process the result with a 1 × 1 convolution and a non-linear activation function, yielding

$$\mathrm{Z}=\delta \left({\mathrm{Conv}}_{1\times 1}\left([{\mathrm{RGB}}^{\mathrm{h}},{\mathrm{RGB}}^{\mathrm{w}},{\mathrm{Depth}}^{\mathrm{h}},{\mathrm{Depth}}^{\mathrm{w}}]\right)\right),$$

(4)

where $[.,.]$ denotes the concatenation operation along the spatial dimension, $\delta$ is a non-linear activation function, and $\mathrm{Z}$ is the intermediate feature map.

Then, $\mathrm{Z}$ is split into two separate tensors ${\mathrm{G}}^{\mathrm{h}}$ and ${\mathrm{G}}^{\mathrm{w}}$. Another two 1 × 1 convolutions are utilized to separately transform ${\mathrm{G}}^{\mathrm{h}}$ and ${\mathrm{G}}^{\mathrm{w}}$ to tensors with the same channel number to the input. Then, we split ${\mathrm{G}}^{\mathrm{h}}$ and ${\mathrm{G}}^{\mathrm{w}}$ as attention weights for two modalities. They are multiplied with the input separately. The process is expressed as follows:

$$\begin{array}{cc}\hfill & \hfill {\mathrm{G}}_{\mathrm{RGB}}^{\mathrm{h}},{\mathrm{G}}_{\mathrm{Depth}}^{\mathrm{h}}={\mathcal{F}}_{\mathrm{split}}\left(\sigma \left({\mathrm{Conv}}_{1\times 1}\left({\mathrm{G}}^{\mathrm{h}}\right)\right)\right),\hfill \\ \hfill & \hfill {\mathrm{G}}_{\mathrm{RGB}}^{\mathrm{w}},{\mathrm{G}}_{\mathrm{Depth}}^{\mathrm{w}}={\mathcal{F}}_{\mathrm{split}}\left(\sigma \left({\mathrm{Conv}}_{1\times 1}\left({\mathrm{G}}^{\mathrm{w}}\right)\right)\right),\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \hfill {\mathrm{RGB}}_{\mathrm{out}}^{\mathrm{CA}}={\mathrm{RGB}}_{\mathrm{out}}\otimes {\mathrm{G}}_{\mathrm{RGB}}^{\mathrm{w}}\otimes {\mathrm{G}}_{\mathrm{RGB}}^{\mathrm{h}},\hfill \\ \hfill & \hfill {\mathrm{Depth}}_{\mathrm{out}}^{\mathrm{CA}}={\mathrm{Depth}}_{\mathrm{out}}\otimes {\mathrm{G}}_{\mathrm{Depth}}^{\mathrm{w}}\otimes {\mathrm{G}}_{\mathrm{Depth}}^{\mathrm{h}},\hfill \end{array}$$

(6)

where ⊗ denotes matrix multiplication.

Finally, ${\mathrm{RGB}}_{\mathrm{out}}^{\mathrm{CA}}$ and ${\mathrm{Depth}}_{\mathrm{out}}^{\mathrm{CA}}$ are concatenated on the channel dimension as the output of coordinate attention stage.

Fusion stage. In this stage, we use a simple channel embedding [15] to merge the features. Two 1 × 1 convolutions, a 3 × 3 depthwise convolution, and an intermediate ReLU activation function constitute the main path of the channel embedding. A 1 × 1 convolution forms the skip connection path. The concatenated features pass through these two paths separately and are then added to obtain the final output, which serves as the input for each stage of the decoder.

3.4. Loss Function

For semantic segmentation, we utilize the cross-entropy loss on a multi-scale, which is formulated as follows:

$${\mathcal{L}}_{\mathrm{sem}}=\frac{1}{4}\sum _{\mathrm{i}=1}^4{\mathcal{L}}_{\mathrm{CE}}({\mathrm{S}}_{\mathrm{out}}^{\mathrm{i}},{\mathrm{S}}_{\mathrm{gt}}^{\mathrm{i}}),$$

(7)

where ${\mathrm{S}}_{\mathrm{out}}^{\mathrm{i}}$ and ${\mathrm{S}}_{\mathrm{gt}}^{\mathrm{i}}$ denote semantic predictions and labels in a different scale.

For instance segmentation, Mean Squared Error (MSE) loss and L1 loss are used for center and offset supervision, respectively. The instance loss is computed as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{ins}}=\frac{1}{4}{\lambda }_{\mathrm{MSE}}\sum _{i=1}^4{\mathcal{L}}_{\mathrm{MSE}}({\mathrm{C}}_{\mathrm{out}}^{\mathrm{i}},{\mathrm{C}}_{\mathrm{gt}}^{\mathrm{i}})+\\ \hfill \frac{1}{4}{\lambda }_{\mathrm{L}1}\sum _{i=1}^4{\mathcal{L}}_{\mathrm{L}1}({\mathrm{O}}_{\mathrm{out}}^{\mathrm{i}},{\mathrm{O}}_{\mathrm{gt}}^{\mathrm{i}}),\end{array}$$

(8)

where ${\mathrm{C}}_{\mathrm{out}}^{\mathrm{i}}$ and ${\mathrm{C}}_{\mathrm{gt}}^{\mathrm{i}}$ denote center predictions and labels in different scales, ${\mathrm{O}}_{\mathrm{out}}^{\mathrm{i}}$ and ${\mathrm{O}}_{\mathrm{gt}}^{\mathrm{i}}$ denote offset predictions and labels in different scales, ${\lambda }_{\mathrm{MSE}}$ and ${\lambda }_{\mathrm{L}1}$ are the loss weight of the MSE loss and L1 loss, and we set them as 2 and 1 as default.

The total loss is organized as:

$$\mathcal{L}={\lambda }_{\mathrm{sem}}{\mathcal{L}}_{\mathrm{sem}}+{\lambda }_{\mathrm{ins}}{\mathcal{L}}_{\mathrm{ins}},$$

(9)

where ${\lambda }_{\mathrm{sem}}$ and ${\lambda }_{\mathrm{ins}}$ are hyperparameters, denoting task weights of semantic and instance segmentation.

4. Experiment

4.1. Datasets and Evaluation Measures

We evaluate the proposed method on two indoor datasets, NYUv2 and SUNRGB-D. NYUv2 [20] contains 1449 indoor RGB-D images with detailed annotations for semantic and instance segmentation. The standard dataset split is adopted, 795 images for training and the remaining 654 images for testing. SUNRGB-D [21] integrates multiple indoor RGB-D datasets, including NYUv2, with a total of 10,335 indoor images, annotated in 37 classes. Of these, 5285 images are used for training, and the remaining 5050 images are used for testing. The instance annotations of SUNRGB-D are extracted using 3D bounding boxes and semantic annotations [19]. Moreover, we use the Hypersim [28] dataset for pretraining. Hypersim is a realistic synthetic dataset, including 77,400 samples with masks for semantic and instance segmentation. The high-quality and large number of samples make it highly suitable for pretraining. We evaluated the proposed IEMNet and existing SOTA methods in terms of the mean intersection over union (mIoU), pixel accuracy (PA) [29], and frames per second (FPS).

4.2. Implementation Details

Our model was implemented based on Pytorch, and all training and testing processes were completed on the RTX A5000 GPU. During the experiments, the pre-trained weights from [19] on ImageNet were used to initialize the encoders of the two branches, and the model was trained for 500 epochs with a batch size of 8. We used the AdamW [30] optimizer with a weight decay of 0.01 for training, while the initial learning rate was set to $6\times {10}^{-5}$ and $4\times {10}^{-5}$ for NYUv2 and SUNRGB-D, and the poly learning rate schedule was applied to further adjust the learning rate. For data augmentation, random scaling, random cropping, and random flipping were employed, and for RGB images, slight color jitter augmentation was added in the HSV color space.

4.3. Quantitative Results on NYUv2 and SUNRGB-D

Table 1. Quantitative results on NYUv2 and SUNRGB-D.

Method	Backbone	NYUv2		SUNRGB-D		FPS
		PA (%)	mIoU (%)	PA (%)	mIoU (%)
ShapeConv * [22]	ResNext101 32x8d	76.4	51.3	-	-	25.97
ShapeConv * [22]	Res101	-	-	82.2	48.6	25.97
SGNet * [23]	Res101	76.8	51.1	82.0	48.6	N/A
SA-Gate [24]	Res50	77.9	52.4	82.5	49.4	24.71
CMX * [15]	SegFormer-B5	80.1	56.9	83.8	52.4	11.13
ESANet (pre. SceneNet) [31]	Res34NBt1D	-	51.6	-	48.5	43.66
EMSANet (pre. Hypersim) [19]	Res34NBt1D	78.1	53.3	81.9	48.5	46.23
Ours (Semantic only)	Res34NBt1D	76.7	50.3	82.0	48.1	42.97
Ours	Res34NBt1D	76.8	51.3	81.9	48.3	42.97
Ours (pre. Hypersim)	Res34NBt1D	78.8	54.5	82.3	49.1	42.97

*: additional test-time augmentation, N/A: no implementation available, pre. SceneNet: additional pretraining on SceneNet, pre. Hypersim: additional pretraining on Hypersim.

, Figure 4 and Figure 5 show the quantitative results of the proposed method and other methods on NYUv2 and SUNRGB-D, including the results after pretraining on the Hypersim and fine-tuning on the corresponding datasets. Task weights ${\lambda }_{\mathrm{sem}}$ and ${\lambda }_{\mathrm{ins}}$ were set as 1:3 for NYUv2 and 1:2 for SUNRGB-D, determined through extensive experiments. The pretraining process uses only 20% of Hypersim data in each epoch to accelerate training, with an initial learning rate of $6\times {10}^{-5}$ and task weights of 1:2 for semantic and instance segmentation. Fine-tuning was performed on NYUv2 and SUNRGB-D, with the same task weights and learning rate. FPS is calculated on NYUv2.

It can be observed that introducing the instance segmentation task significantly improves performance on both datasets compared to using only semantic segmentation. Combining multiple tasks does not increase the computational cost during testing, as only semantic results are predicted. Meanwhile, our method gets further improvement after pretraining on Hypersim without increasing the inference cost. Using data several times larger than the original training set for pretraining achieves a huge improvement for our method, as the impact of the data on the model performance is significant. However, our method still achieves enhancement on this basis, which can be seen from the comparison with EMSANet and subsequent ablation experiments. In terms of mIoU comparison, our method ranks just behind CMX, but our inference speed is much faster, reaching 42.97 FPS. In terms of FPS comparison, our method is only slower than ESANet and EMSANet, but the mIoU is higher, achieving 54.5% on NYUv2 and 49.1% on SUNRGB-D, surpassing most methods. Thus, compared to other methods, our IEMNet achieves the optimal balance between performance and inference speed.

4.4. Qualitative Results on NYUv2

We visualize five semantic segmentation results of ESANet, EMSANet, and our method on the NYUv2 dataset, as shown in Figure 6. Visually, the method proposed in this paper performs better in segmentation accuracy than the ESANet and EMSANet. For example, our method IEMNet correctly identifies the TV cabinet and the washbasin while the ESANet and EMSANet wrongly classify them with other classes. It demonstrates that our proposed method enhances the network’s ability to utilize depth information through more comprehensive feature interaction and fusion, resulting in better performance.

4.5. Ablation Study on NYUv2

To investigate the impact of different components on segmentation performance and inference speed, we conducted ablation studies using only semantic decoder on NYUv2 and SUNRGB-D. The same training strategy and various model settings, including the CM-FRM, CAFM, and different backbones, are employed as shown in

Table 2. Ablation for CM-FRM, CAFM and Backbone.

Backbone	CM-FRM	Fusion	NYUv2	SUNRGB-D	FPS
			mIoU (%)	mIoU (%)
Res34NBt1D	×	RGB-D Fusion	49.20	46.81	46.18
Res34NBt1D	✓	RGB-D Fusion	50.27	47.88	43.26
Res34NBt1D	×	CAFM	49.41	47.11	45.28
Res34NBt1D	✓	CAFM	50.38	48.10	42.97
Res18	✓	CAFM	46.04	45.17	63.37
Res34	✓	CAFM	47.06	46.36	54.57
Res50	✓	CAFM	49.86	47.05	28.03
Res18NBt1D	✓	CAFM	46.74	46.09	55.21

, where the RGB-D Fusion module is the original fusion module in the EMSANet.

Firstly, it can be seen that our CAFM outperforms RGB-D Fusion in fusing cross-modal features and achieving better results on both datasets. The significant improvement after introducing the CM-FRM demonstrates the effectiveness of enhanced feature interaction and calibration. Although using the CM-FRM and CAFM increases the inference time, the FPS remains at 42.97, achieving efficient speed. Note that the FPS is very slow when using Res50. This is because the block and number of layers are different with Res18 and Res34, causing a significant increase in parameter and computational complexity. Moreover, using NBt1D, PA, and mIoU is generally better, while the inference speed decreases. This discrepancy with [19,31,32] is due to their FPS calculations being performed on a robotic platform, while ours are on the Pytorch platform. Thus, NBt1D is more suitable for robotic platforms compared to normal convolution. Considering the performance and inference speed of various backbones, using Res34NBt1D as the backbone achieves better balance between performance and speed.

5. Conclusions

In order to achieve good balance between performance and speed for RGB-D semantic segmentation, we propose an interactive efficient multitask RGB-D semantic segmentation network. Based on the EMSANet, the cross-modal feature rectification module and coordinate attention fusion module are introduced into the encoder branches to accomplish feature rectification and fusion at different scales, constructing an interactive encoder structure that enables comprehensive feature interaction. Furthermore, without compromising the inference speed of the semantic segmentation task, our method incorporates an instance segmentation task, thereby enhancing the semantic segmentation performance. Results of the NYUv2 and SUNRGB-D datasets indicate that the proposed method exhibits optimal performance in balancing performance and speed under indoor scenes. In future work, we will port our method to robotic platforms to verify its performance. In addition, integrating more tasks into the current framework is expected to further improve performance, such as depth estimation.