SegForest: A Segmentation Model for Remote Sensing Images

Abstract

The accurate estimation of forest area is of paramount importance for carbon sequestration projects, ecotourism and ecological safety. Forest segmentation using remote sensing images is a crucial technique for estimating forest area. However, due to the complex features, such as the size, shape and color of forest plots, traditional segmentation algorithms struggle to achieve accurate segmentation. Therefore, this study proposes a remote sensing image forest segmentation model named SegForest. To enhance the model, we introduce three new modules: multi-feature fusion (MFF), multi-scale multi-decoder (MSMD) and weight-based cross entropy loss function (WBCE) in the decoder. In addition, we propose two new forest remote sensing image segmentation binary datasets: DeepGlobe-Forest and Loveda-Forest. SegForest is compared with multiple advanced segmentation algorithms on these two datasets. On the DeepGlobe-Forest dataset, SegForest achieves a mean intersection over union (mIoU) of 83.39% and a mean accuracy (mAcc) of 91.00%. On the Loveda-Forest dataset, SegForest achieves a mIoU of 73.71% and a mAcc of 85.06%. These metrics outperform other algorithms in the comparative experiments. The experimental results of this paper demonstrate that by incorporating the three proposed modules, the SegForest model has strong performance and generalization ability in forest remote sensing image segmentation tasks.

1. Introduction

Forest ecosystems are a vital component of terrestrial ecosystems, representing the largest, most widespread, complex and resource-rich ecosystems on land. The interactions between forest ecosystems and the atmosphere, involving exchanges of energy, water, carbon dioxide and other compounds, significantly influence and regulate the climate. Forests play a pivotal role in global carbon cycling, water cycling, the mitigation of global climate change, climate regulation, soil conservation and environmental enhancement [1,2]. Moreover, apart from providing a diverse array of ecological services, the forest area exhibits a strong correlation with sustainable economic development [3] and assumes a critical role in carbon sequestration engineering, ecotourism and ecological security [4,5,6]. Therefore, effective forest resource management and monitoring is critical to ensure sustainable development. Remote sensing techniques, such as satellite and unmanned aerial vehicle (UAV) image segmentation are widely employed for forest area estimation and land survey [7,8,9]. Remote sensing image segmentation refers to the process of analyzing and processing remote sensing image data to partition different regions or objects within the image into distinct sub-regions or sub-objects based on their independent characteristics. However, traditional methods often exhibit low precision, along with high computational complexity and time consumption, making them unsuitable for complex forest remote sensing image segmentation applications. Therefore, improving the efficiency and precision of segmentation tasks in complex forest scenes remains a significant challenge.

Classic segmentation algorithms, including thresholding [10], edge-based [11], region-based [12], graph-based [13] and energy-based [14] techniques, are often limited by their computational complexity and struggle to achieve a satisfactory balance between computational efficiency and accuracy. Consequently, these algorithms may not be suitable for effectively processing forest remote sensing images. To address this problem, Wang et al. proposed a scalable graph-based clustering method called SGCNR, which implements non-negative relaxation to reduce computation complexity [15]. Although SGCNR showed some improvement over classic segmentation algorithms, its accuracy was limited and insufficient for precise forest parcel segmentation. Although other conventional machine learning techniques, such as random forest [16], support vector machine [17,18] and conditional random field [19], have enhanced accuracy and robustness, they may not fully meet the precision requirements for forest remote sensing image segmentation.

In recent years, deep convolutional neural networks (DCNNs) have demonstrated powerful feature extraction and object representation capabilities compared to traditional machine learning methods [20]. The most advanced methods based on full convolutional networks (FCNs) [21] have demonstrated great progress. For example, Wu et al. developed an FCN architecture optimized for polarimetric synthetic aperture radar (SAR) imagery to classify wetland complexes [22]. Li et al. improved the encoder architecture of U-Net++ and added a local attention mechanism for the accurate segmentation of forest remote sensing maps [23]. Nevertheless, fixed geometric configurations constrain local receptive fields and contextual information within a short range. The segmentation of forest remote sensing images remains a demanding task.

To overcome this limitation and increase the receptive field and contextual information over long-term features, self-attention mechanisms have gained widespread usage in computer vision [24,25,26]. Self-attention mechanisms have consequently been integrated with DCNNs by some researchers. Fu et al. proposed a recursive and parsimonious attention module and applied it to the ResNet model to achieve the high-precision recognition of remote sensing scenes [27]. Moreover, some researchers have implemented transformers in remote sensing image segmentation tasks [28]. Advanced self-attention mechanisms enable models to effectively capture various spatial-scale feature information in remote sensing image segmentation tasks, thereby improving segmentation accuracy, particularly when handling large area remote sensing images with complex spatial structures and features.

The encoders of different models can effectively extract the spatial feature information of various scales in remote sensing images by utilizing self-attention mechanisms. However, forest remote sensing images contain complex information of various sizes, colors and shapes, and it is crucial to make better use of multi-scale feature information obtained by encoders for improving the IoU performance of the segmentation models. To solve this issue, we propose SegForest, a high-performance segmentation network that fully utilizes the feature information of various scales. This paper introduces a multi-feature fusion (MFF) module to better fuse features of different scales. Furthermore, an innovative multilevel decoding multi-scale and multi-decoder (MSMD) module is proposed to utilize features at fine scales more extensively. To improve model training, we introduce the weight-based cross-entropy (WBCE) loss function designed specifically for the MSMD. Additionally, two datasets, DeepGlobe-Forest and Loveda-Forest, are specifically created for the forest remote sensing image segmentation tasks in this paper.

2. Datasets

The satellite images used in DeepGlobe-Forest are extracted from the DigitalGlobe + Vivid Images dataset [29]. The images cover areas taken over Thailand, Indonesia and India. The ground resolution of the image pixels is 50 cm/pixel. These images consist of three channels (red, green and blue) each. Each of the original Geotiff images is 19,584 × 19,584 pixels. We crop 194 RGB images of resolution 2448 × 2448 from the original images. These images are labeled into two semantic categories, namely, forest and background. The class distribution can be seen in

Table 1. Class distributions in the DeepGlobe-Forest dataset.

Class	Label	Pixel Count	Proportion
Background	0	537.13 M	46.20%
Forest	1	625.46 M	53.80%

. Figure 1 shows some examples of DeepGlobe-Forest.

The Loveda-Forest dataset was created based on the Land-cOVE dataset for Domain Adaptation (Loveda) [30]. The Loveda dataset is constructed using 0.30 m images acquired in July 2016 from Nanjing, Changzhou and Wuhan, covering a total area of 536.15 km² with red, green and blue bands at a spatial resolution of 0.30 m. Following geometric registration and preprocessing, each region is covered by non-overlapping images with a resolution of 1024 × 1024. We retain only the forest classification from the original dataset, while unifying other classifications as background classifications. To balance the number of forest and background pixels, we carefully select some images. We change the forest label from the original dataset to 1, while changing all other labels to 0 to serve as the background.

Table 2. Class distributions in the Loveda-Forest dataset.

Class	Label	Pixel Count	Proportion
Background	0	332.37 M	48.32%
Forest	1	355.49 M	51.68%

displays the class distribution, while partial examples of Loveda-Forest are shown in Figure 2.

3. Methods

This section presents SegForest, an efficient and robust segmentation framework for remote sensing image forest segmentation. The SegForest model consists of two main components: a transformer encoder for extracting multi-scale features; and an efficient decoder for multi-scale feature fusion, resulting in the final semantic segmentation mask. The encoder design is based on SegFormer, but we redesigned the entire decoder structure due to its inefficiency in fusing multi-scale features. Figure 3 illustrates the structure of SegForest. The following section introduces the decoder that we designed.

3.1. Multi-Scale Feature Fusion

Traditionally, coarse-to-fine segmentation neural networks have typically relied solely on features from coarse-scale subnetworks within fine-scale subnetworks, restricting information flow. However, the diverse sizes of forest blocks in remote sensing images present a significant challenge, necessitating multi-scale features. If the decoder cannot effectively use the complex multi-scale features, accurate segmentation results are unachievable.

To optimize the utilization of multi-scale features, we propose the integration of a multi-scale feature fusion (MFF) module, as depicted in Figure 4. This module enables our model to produce information streams from various scales. The MFF block leverages the Transformer block’s results from each scale by employing convolutional layers to merge the multiscale features. In this model, the MFF formulation is as follows:

$$MF{F}_1^{\mathrm{out}}=MF{F}_1\left(T{B}_1^{\mathrm{out}},{\left(T{B}_2^{\mathrm{out}}\right)}^{\uparrow },{\left(T{B}_3^{\mathrm{out}}\right)}^{\uparrow },{\left(T{B}_4^{\mathrm{out}}\right)}^{\uparrow }\right),$$

(1)

$$MF{F}_2^{\mathrm{out}}=MF{F}_1\left({\left(T{B}_1^{\mathrm{out}}\right)}^{\downarrow },T{B}_2^{\mathrm{out}},{\left(T{B}_3^{\mathrm{out}}\right)}^{\uparrow },{\left(T{B}_4^{\mathrm{out}}\right)}^{\uparrow }\right),$$

(2)

$$MF{F}_3^{\mathrm{out}}=MF{F}_1\left({\left(T{B}_1^{\mathrm{out}}\right)}^{\downarrow },{\left(T{B}_2^{\mathrm{out}}\right)}^{\downarrow },T{B}_3^{\mathrm{out}},{\left(T{B}_4^{\mathrm{out}}\right)}^{\uparrow }\right),$$

(3)

where $T{B}_k^{\mathrm{out}}$ is the feature map of the Kth scale output of the Transformer blocks, $MF{F}_k^{\mathrm{out}}$ denotes the MFF output of the K-th scale, ↑ represents upsampling, and ↓ represents downsampling. As a result, SegForest significantly enhances its multi-scale feature fusion ability and thus improves forest segmentation performance.

3.2. Multi-Scale Multi-Decoder

The decoder in SegFormer first performs an upsampling of the fine-scale features and then concatenates the features from all scales. Finally, the decoder outputs the resulting feature map through a convolution operation. However, this structure may result in the underutilization of the detailed features and the lack of effective fusion among different scales. To mitigate this issue, we introduce a novel multiscale multi-decoder (MSMD) structure in SegForest. The decoder architecture of the second and third levels can be observed in Figure 5a, and the first level’s decoder structure is depicted in Figure 5b.

In Figure 5, $\mathrm{Transpose}{}_{\mathrm{k}+1}{}^{\mathrm{out}}$ refers to the convolutional transpose output at a finer level of scale and ${\mathrm{MFF}}_{\mathrm{k}}^{\mathrm{out}}$ refers to the MFF output at the same level of scale. We chose convolutional transpose instead of upsampling to reduce the information loss caused by upsampling. The green dashed line in Figure 5a represents that this part only runs when training. We let the decoders of each scale output the result to calculate the loss during training so that each decoder can be trained to obtain better performance.

This multi-scale, multi-decoder structure allows the features at each scale to be fully utilized, allowing for more accurate results when dealing with forest segmentation of varying shapes and sizes.

3.3. Loss Function

The SegForest model employs a multi-scale multi-decoder structure. To optimize the model training, we suggest using a weight-based cross-entropy loss function (WBCE) in the following formula:

$$Loss=0.8L_{output}+0.13L_{outpu{t}_2}+0.07L_{outpu{t}_3},$$

(4)

where $L_{output}$ is the cross-entropy loss of the final output of the network, and $L_{outpu{t}_2}$ and $L_{outpu{t}_3}$ are the cross-entropy losses of the output of level 2 and level 3, respectively; see Figure 3. The formula for calculating the cross-entropy loss L for each component is shown below:

$$L=\frac{1}{b}\sum _{j=1}^b\sum _{i=1}^n-ylog\left(\widehat{y}\right)-(1-y)log(1-\widehat{y}),$$

(5)

where b is the batch size, n is the total number of samples within the training set, y is the true distribution, and $\widehat{y}$ is the probability distribution of the model output.

Experimental results show that the weight-based cross-entropy loss function improves model performance. Specifically, it enables the model to better handle the complex area, size and shape of the forests in remote sensing images by fusing the outputs of different scales.

4. Experiments

4.1. Experimental Settings

SegForest is implemented in PyTorch [31]. Training on the Globe-Forest and Loveda-Forest datasets is performed for 160,000 iterations with an initial learning rate of 0.00006, which is linearly decreased from the 1500th iteration onwards. A crop size of 512 × 512 is used on both datasets. We do not employ commonly used techniques, such as OHEM, auxiliary loss or class balance loss for simplicity. The experimental setup consists of a single RTX3060 graphics card, an i9-12900hs CPU and 64 GB of memory.

4.2. Evaluation Metrics

This study utilizes two evaluation metrics, intersection over union (IoU) [32] and accuracy (Acc), to assess the performance of the models. The formulas for calculating mIoU and IoU are presented below:

$$IoU=\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}},$$

(6)

$$mIoU=\frac{1}{k+1}\sum _{i=0}^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}},$$

(7)

where $p_{i}j$ denotes the prediction of category i into category j, and $k+1$ is the total number of categories. Similarly, the accuracy and mAcc formulas are presented below:

$$Accuracy=\frac{TP+TN}{TP+FP+FN+TN},$$

(8)

$$mAcc=\frac{1}{k+1}\sum _{i=0}^k\frac{T{P}_i+T{N}_i}{T{P}_i+F{P}_i+F{N}_i+T{N}_i},$$

(9)

where $TP$ represents the true positive, $TN$ the true negative, $FP$ the false positive, $FN$ the false negative, and $k+1$ signifies the total number of categories.

4.3. Comparison to State of the Art Methods

We benchmark SegForest against multiple state-of-the-art models, including advanced fully convolutional segmentation models and transformer models, on two datasets: DeepGlobe-Forest and Loveda-Forest. The comparison comprises deeplabv3+ [33], pidnet [34], pspnet [35], knet [36], segformer [24], mask2former [25] and segnext [26,37].

Table 3. Performance of models on DeepGlobe-Forest dataset.

	IoU		mIoU	Accuracy		mAcc
	Forest	Background		Forest	Background
Deeplabv3+	77.69	79.67	78.68	87.42	88.70	88.06
Pidnet-s	78.78	80.96	79.87	87.35	90.21	88.78
Pspnet	79.86	80.79	80.33	91.17	87.22	89.20
Knet-s3-r50	80.23	81.22	80.73	91.24	87.63	89.44
Segformer	79.98	81.71	80.85	89.02	89.80	89.41
Mask2former	80.52	81.61	81.06	91.09	88.17	89.63
Segnext	80.60	81.84	81.22	90.69	88.71	89.70
SegForest	82.80	83.99	83.39	91.79	90.20	91.00

Bold font means the best performance.

presents a summary of the results on DeepGlobe-Forest, which includes the mIoU and accuracy results. The highest evaluation index score is highlighted in bold. Our method outperforms the Segformer approach that uses the same encoder. Specifically, our approach improves mIoU by 2.54% and mAcc by 1.59%. The results indicate that Segformer’s simple and lightweight all-MLP (multi-layer perceptron) decoder cannot achieve the full potential of the encoder due to the complex size, edge and shape features of forest plots in remote sensing images. By incorporating our proposed multi-scale feature fusion module and multi-scale multi-decoder module to SegForest, our network demonstrates superior performance. In comparison to other benchmarks, SegForest exhibits the highest IoU in both the forest and background, which is crucial for the remote sensing image forest area and forest biomass estimation tasks that require forest area information. Pidnet records the highest accuracy in the forest category, which is only 0.01% higher than SegForest. Nevertheless, SegForest attains the highest score in the background category, as well as the mean accuracy. Figure 6 showcases some prediction result examples of each method on the DeepGlobe-Forest dataset. As illustrated in Figure 6, our method’s segmentation results are significantly closer to the ground truth on the forest edges, which is particularly important for forest expansion monitoring. The experiment demonstrates SegForest’s superior performance in forest segmentation from remote sensing images.

Table 4. Performance of models on Loveda-Forest dataset.

	IoU		mIoU	Accuracy		mAcc
	Forest	Background		Forest	Background
Deeplabv3+	64.22	75.88	70.05	80.37	84.85	82.61
Pidnet-s	64.36	74.08	69.22	84.82	80.85	82.84
Pspnet	62.68	77.08	69.88	73.93	89.18	81.56
Knet-s3-r50	65.99	76.16	71.08	84.11	83.45	83.78
Segformer	64.63	76.31	70.47	80.36	85.35	82.86
Mask2former	65.67	76.83	71.25	81.69	85.30	83.50
Segnext	64.42	76.96	70.69	78.31	87.01	82.66
SegForest	68.38	79.04	73.71	82.98	87.14	85.06

Bold font means the best performance.

presents the results obtained on Loveda-Forest, reporting values for mIoU and accuracy. The highest evaluation index score is highlighted in bold. SegForest outperforms Segformer in all scores, with improved mIoU and mAcc by 3.75% and 2.20%, respectively. Moreover, when compared with state-of-the-art methods, SegForest achieves the highest score for each IoU score, demonstrating its effectiveness. Knet exhibits the highest accuracy in forest classification, with a 1.13% advantage over SegForest, while Pspnet displays the highest accuracy in background classification, with a 2.04% benefit over SegForest. Nonetheless, SegForest achieves the highest mean accuracy, representing its balanced and comprehensive performance. Figure 7 presents the predictions obtained by the different methods on Loveda-Forest.

4.4. Ablation Studies

We conducted a series of ablation studies on the DeepGlobe-Forest dataset to assess the efficacy of the individual modules. The results of these studies are presented in

Table 5. Results of ablation studies on DeepGlobe-Forest dataset.

MFF	MSMD	WBCE	IoU		mIoU	Accuracy		mAcc
			Forest	Background		Forest	Background
			79.98	81.71	80.85	89.02	89.80	89.41
✓			81.54	83.75	82.65	89.83	90.13	89.98
	✓		80.53	81.64	81.09	88.71	90.18	89.45
	✓	✓	81.31	82.90	82.11	89.95	90.12	90.04
✓	✓	✓	82.80	83.99	83.39	91.79	90.20	91.00

✓ indicates the addition of this module.

, where MFF denotes the multi-feature fusion module, MSMD represents the multi-scale multi-decoder module, and WBCE denotes the weight-based cross-entropy loss function.

The addition of only the MFF module increases the model’s mIoU by 1.80% and mean accuracy by 0.57%. The MFF module improves performance by allowing the decoder to effectively leverage features across scales. The addition of only the MSMD module results in more modest improvements, increasing mIoU and mean accuracy by 0.24% and 0.04%, respectively. By adding the WBCE loss function, the performance is further improved, increasing the mIoU and mean accuracy by 1.02% and 0.59%, respectively. The MSMD and WBCE modules enable hierarchical decoding at different scales, thereby resulting in a significant performance boost. The best performance is achieved by adding all three modules simultaneously, improving the mIoU and mean accuracy by 2.54% and 1.59%, respectively, and enabling the model to better handle complex forest conditions in remote sensing images.

5. Discussion

Experimental comparisons show that SegForest outperforms other models for forest segmentation tasks, especially achieving the best mIoU and mAcc on the DeepGlobe-Forest and Loveda-Forest datasets. However, it is observed that SegForest does not always achieve the optimal accuracy. This issue may be due to the transformer encoder, which is not completely applicable to segmentation tasks. Therefore, there are plans to design a transformer encoder that is more suitable for segmentation tasks. Additionally, ablation studies were conducted on the DeepGlobe-Forest dataset to validate three proposed modules (MFF, MSMD and WBCE) and their performance differences. Specifically, the MFF module enables the model to obtain information flow from different scales and better fuse features from each scale. The MSMD module allows step-by-step decoding at different scales, enabling the model to fully utilize fine-scale features. Moreover, the WBCE module significantly improves post-training performance without increasing the model parameters. The integration of these three modules fortifies SegForest’s handling of intricate size, shape and color features of forest areas in remote sensing imagery. Further improvement measures will be explored to enhance SegForest’s performance, such as introducing more feature fusion strategies and more effective complexity control methods.

One of the challenges of this study is the dataset. Currently, we use the DeepGlobe-Forest and Loveda-Forest datasets, which have low spatial resolution and lack precise annotations. Therefore, a key focus of our future work is to develop a new, more detailed remote sensing image dataset specific to forest segmentation. We plan to use drones equipped with multispectral lenses to capture higher spatial resolution images and multispectral data, which will then be annotated more precisely. Additionally, we will design a network that incorporates multispectral features to achieve even higher accuracy in remote sensing image segmentation [38].

6. Conclusions

This study proposes SegForest, a powerful model for forest segmentation tasks in remote sensing images. We proposed three modules to enhance the utilization of different scales of feature information in the model, namely the multi-feature fusion (MFF) module, multi-scale multi-decoder (MSMD) module and weight-based cross-entropy (WBCE) loss function. Additionally, we introduced two forest remote sensing image segmentation datasets named DeepGlobe-Forest and Loveda-Forest. Both datasets are binary classification datasets, including forest and background pixels in equal quantity. We evaluated SegForest with numerous state-of-the-art methods on these two datasets, and it achieved the highest mIoU of 83.39% and 73.71% on DeepGlobe-Forest and Loveda-Forest, respectively, demonstrating its excellent performance. We also conducted a series of ablation studies to verify the performance of the three proposed modules.