CD_HIEFNet: Cloud Detection Network Using Haze Optimized Transformation Index and Edge Feature for Optical Remote Sensing Imagery

Abstract

Clouds in optical remote sensing images are an unavoidable existence that greatly affect the utilization of these images. Therefore, accurate and effective cloud detection is an indispensable step in image preprocessing. To date, most researchers have tried to use deep-learning methods for cloud detection. However, these studies generally use computer vision technology to improve the performances of the models, without considering the unique spectral feature information in remote sensing images. Moreover, due to the complex and changeable shapes of clouds, accurate cloud-edge detection is also a difficult problem. In order to solve these problems, we propose a deep-learning cloud detection network that uses the haze-optimized transformation (HOT) index and the edge feature extraction module for optical remote sensing images (CD_HIEFNet). In our model, the HOT index feature image is used to add the unique spectral feature information from clouds into the network for accurate detection, and the edge feature extraction (EFE) module is employed to refine cloud edges. In addition, we use ConvNeXt as the backbone network, and we improved the decoder to enhance the details of the detection results. We validated CD_HIEFNet using the Landsat-8 (L8) Biome dataset and compared it with the Fmask, FCN8s, U-Net, SegNet, DeepLabv3+ and CloudNet methods. The experimental results showed that our model has excellent performance, even in complex cloud scenarios. Moreover, according to the extended experimental results for the other L8 dataset and the Gaofen-1 data, CD_HIEFNet has strong performance in terms of robustness and generalization, thus helping to provide new ideas for cloud detection-related work.

1. Introduction

With the rapid development of remote sensing technology, the field of satellite remote sensing has entered the era of Big Data [1,2,3]. Remote sensing is now widely used in environmental protection, land resource surveys, disaster monitoring and other fields [4,5,6]. According to the long-term observations of the International Satellite Cloud Climatology Project (ISCCP), more than 60% of the Earth’s surface is regularly covered by clouds [7]. Therefore, optical remote sensing images are easily disturbed by the atmosphere and clouds, and many images are occluded by clouds. Due to cloud occlusion, the information for ground objects can be attenuated or even lost and the texture of and spectral information from the images may be changed at the same time, which adversely affects the subsequent production of images and seriously reduces the utilization rate of remote sensing images [8,9]. Furthermore, studying the distribution of clouds is helpful for investigations of climate change trends, and it is also significant for the assessment of the global surface radiation budget [10,11,12]. Therefore, cloud detection is key to subsequent image identification, classification and interpretation, and it is the premise that guarantees the production of seamless spatiotemporal remote sensing products [13]. Moreover, effective cloud detection is also an important step in cloud removal [14,15]. Cloud detection can also be used to eliminate images with excessive cloud coverage in order to reduce the burden on data storage and improve product production efficiency. Efficient and accurate cloud detection in remote sensing images has also become a hot issue in the field of remote sensing.

Cloud detection is a semantic segmentation process for pixel-level image classification based on actual semantic information, and it is also a binary classification problem based around distinguishing cloud and non-cloud areas. In the past few decades, much research has been undertaken on cloud detection using different methods. These methods include the spectral threshold method, the texture analysis method and the machine-learning method [16,17,18]. The spectral threshold method mainly relies on the spectral characteristics of clouds and involves designing different combinations of various bands for detection. This method is widely used with MODIS data and Landsat data. Jedlovec [19] used 19 channels of MODIS data and terrain data for cloud detection in the popular MODIS Cloud Mask cloud detection method. Irish et al. [20] proposed an automatic cloud cover assessment (ACCA) algorithm for Landsat 7 images that uses spectral characteristics to perform the first scan and then the radiance and temperature characteristics to perform the second scan and obtain cloud detection results. Subsequently, Zhu et al. [21] proposed the function of mask (Fmask) algorithm to extract clouds and cloud shadows with high accuracy in Landsat data. In recent years, in view of the variability in remote sensing images, a dynamic threshold has been developed as part of the spectral threshold method to obtain more accurate detection results [22,23]. However, most of these methods require the use of many bands, and threshold determination is complicated. Moreover, most of them are designed for specific sensors and cannot be generalized to other sensors, so it is difficult to for these methods to be universal.

With the continuous improvement in the spatial resolution of remote sensing images, the texture features of clouds and ground objects are becoming more obvious in images. Therefore, cloud detection methods based on texture analysis have also attracted the attention of many scholars. The most widely used texture features of clouds are the gray level co-occurrence matrix (GLCM), the fractal dimension and the boundary feature [17]. Kittler and Pairman [24] used the GLCM to construct the feature space of clouds, and this method was able to effectively distinguish cloud and non-cloud areas. However, the method still misjudged thin cloud areas. Cao et al. [25] constructed a two-dimensional feature space with the fractal dimension and GLCM and used a plane classifier to distinguish clouds and ground objects in order to achieve fast and accurate cloud detection. Wang et al. [26] adopted four edge features for cloud recognition, which reduced the misjudgment rate for some mountainous and snowy areas, but it was difficult to distinguish clouds in completely white images of snow. Although this method has improved accuracy compared to the threshold method, the extraction of texture features entails significant computation costs and it still does not address the problem of threshold determination.

Cloud detection methods based on machine learning can be divided into traditional machine-learning methods and deep-learning methods. Although traditional machine-learning methods, such as clustering [27], artificial neural networks [28], support vector machines [29] and random forest algorithms [30], can solve the threshold problem and improve cloud detection accuracy, they require manual selection of features and cannot complete end-to-end learning. In recent years, deep learning has been developing with unprecedented speed. Compared to traditional algorithms, deep learning has strong learning portability and can extract complex nonlinear features. As a typical deep-learning method, convolutional neural networks (CNNs) are widely used in natural language processing, image classification and other fields. Since fully convolutional networks (FCNs) [31] were proposed in 2014, semantic segmentation networks have become the dominant method for image segmentation, including cloud detection. Mohajerani et al. [32] applied an FCN for cloud detection in Landsat data and obtained an accuracy of 88%, but the types of underlying surfaces studied in this method were limited, resulting in poor generalization. Lu et al. [33] used SegNet with multi-scale filters to reduce the probability of identifying other objects as clouds, but the ability to distinguish clouds and snow still required improvement. Peng et al. [34] found that DeepLabv3+ has high accuracy and stability for cloud recognition because of the atrous spatial pyramid pooling (ASPP) module, which makes it possible to obtain advanced features of different scales, but it has the shortcomings including spatial information loss and overly smooth boundaries.

Researchers have also explored networks specifically designed for cloud detection. Zhan et al. [35] added low-level spatial information to a network to improve cloud discrimination. Mohajerani et al. [36] redesigned a fully convolution network to combine global and local features for cloud detection, naming the new method CloudNet. In addition, some studies have also found that fusing CNNs with handcrafted features can improve the performance of cloud detection networks. Zhang et al. [37] added Gabor-filtering feature extraction and channel attention modules to U-Net so that the network could learn rich details and the accuracy of cloud detection could be improved. Guo et al. [38] fused mean, Gabor and Laplacian features to make the network more advanced and accurate. The deep-learning method can achieve end-to-end learning and has high accuracy and universality, but the existing semantic segmentation networks still need improvements for cloud detection. A general segmentation network is mainly divided into two parts: the encoder and decoder. In the encoder part, in order to obtain the global information for image and reduce the amount of calculation, the network needs multiple downsampling operations, but the downsampling process leads to spatial information loss, resulting in inaccurate boundary definitions and affecting the accuracy of cloud detection. Moreover, most cloud detection networks are based on the application of computer vision to the original network. Although the related texture features are integrated, they do not incorporate the unique band feature information from remote sensing images, which means that such networks have certain limitations for cloud detection.

Researchers have also come to understand the difference between remote sensing images and natural images and have studied semantic segmentation networks based on the particularity of remote sensing images. Su et al. [39] used the normalized difference water index (NDWI) together with original images as the input for the DeepLabv3+ network in order to investigate water extraction and found that using the NDWI images as input resulted in the network paying much more attention to water extraction. Therefore, inspired by this work, and in view of the lack of attention paid to the multi-band rate characteristics of remote sensing images in existing cloud detection networks and the difficulty of detecting cloud edge details, this paper proposes a new cloud detection network using the haze-optimized transformation (HOT) index and edge feature extraction based on the most common red, blue, green and near-infrared bands (CD_HIEFNet). The main contributions of this paper are as follows:

• We used the HOT index image and the multispectral (MS) image together as the input of the network, which added the spectral characteristics of the clouds and enabled the network to distinguish difficult regions that are easily confused with clouds, so that the network could distinguish regions that are easily confused with clouds;
• We deployed an edge feature detection (EFE) module to enhance the extraction of cloud boundary details in the network. This made the network fit the cloud boundary well and allowed it to detect various cloud types;
• In our structure, we adopted ConvNeXt [40] as the backbone network. In the decoder stage, we used a structure to fuse shallow and deep features from bottom to top to compensate for the loss of edge and local information, which made it possible to recover boundary information effectively and obtain accurate results;
• CD_HIEFNet has great cloud detection performance for Landsat-8 (L8) Biome datasets. Moreover, the extended experiments showed that CD_HIEFNet had good generalization performance, which is important in practical applications.

The remainder of this paper is arranged as follows. In Section 2, the theoretical methodology for and structural design of CD_HIEFNet are described. In Section 3, the data, the experiments and the corresponding experimental results used to verify the superior performance of CD_HIEFNet are described. Section 4 presents the conclusion.

2. Methodology

Current semantic segmentation networks generally employ an encoder–decoder structure. The encoder usually uses image classification networks, such as VGGNet [41] and ResNet [42], to extract the image features through multi-layer convolution. Many improved modules have also been proposed for semantic segmentation networks, such as the ASPP module [43], to fuse multi-scale spatial information. The decoder stage projects the discriminative features learned by the encoder semantically into the pixel space to obtain a dense classification. Existing decoding mechanisms include the skipping connection [44] and inverse pooling upsampling [45]. A good semantic segmentation network needs not only discriminative abilities at the pixel level but also the ability to incorporate the discriminative features learned by the encoder at different stages.

Our proposed CD_HIEFNet network is also based on the encoder–decoder structure, and the overall framework of the network is shown in Figure 1. As shown in the figure, the initial input of our network consists of two parts: (1) red, green, blue and near-infrared band remote sensing images; (2) HOT index spectral feature images. The encoder uses ConvNeXt with good accuracy and scalability as the backbone network. In addition, the EFE module was added to this part to enhance the attention paid to edge information, as well as the ASPP module to fuse multi-scale and global context information. The backbone network, the HOT index and the EFE module are described in detail in Section 2.1, Section 2.2 and Section 2.3.

At the decoder stage, the fusion of feature information from different stages and different scales can improve the final segmentation results. The feature pyramid network (FPN) [46] uses a pyramid architecture with lateral connections based on the inherent multi-scale pyramid hierarchy of the encoder, enabling the network to obtain precise semantic features and feature locations. We borrowed the pyramid structure of the FPN. In accordance with the downsampled pyramid structure of the classification network, we started by upsampling the feature map with the lowest spatial resolution but the strongest semantic spatial resolution by a factor of 2 (using bilinear interpolation upsampling) and then used element-wise addition to merge the upsampled map with the corresponding level map (a 1 × 1 convolutional layer was applied to the corresponding level map to change the channel dimensions and ensure an equal number of channels). This process was iterated until the result was generated at the top level of the classification network pyramid structure. After this operation, the output results of each layer had complementary relationships, and convolution fusion could be used to obtain accurate segmentation results. In addition, similar to the holistically nested edge network (HDE) [47], we performed 3 × 3 convolution on the abovementioned results from each layer and generated three side output saliency probability maps: Side 1, Side 2 and Side 3. Then, we upsampled them to the original image size and concatenated them, performing 1 × 1 convolution to generate the final saliency probability map.

2.1. Backbone Network

The backbone network plays a crucial role in improving both the efficiency and accuracy of segmentation. Focusing on accuracy, efficiency and scalability, CNNs have launched many representative networks [40], such as VGGNet, ResNet, MobileNet [48] and EfficientNet [49]. However, vision transformers (ViTs) [50] have been widely used in image classification since their development, and they have advantages such as scalable behavior and multi-head attention. On the other hand, in computer vision downstream tasks such as object detection and semantic segmentation, vanilla ViTs face difficulties due to the increased complexity of the attention mechanism. Swin Transformers [51] solve the problem by using a hierarchical structure similar to that used in CNNs and local attention, making these transformers viable as a general visual backbone network; however, it also shows that CNNs are still needed in image segmentation tasks. Therefore, ConvNeXt networks have been built with standard convolutional modules, but they borrow the structure and training mode from transformers to obtain high accuracy and scalability comparable to transformers in the networks [40].

Cloud detection is usually applied in the context of large-scale remote sensing image data, so efficiency is also an issue to be considered. Therefore, we selected the improved version of ConvNeXt-T with the smallest amount of parameters as the backbone network. ConvNeXt-T is composed of a stack of four stages, as shown in

Table 1. The architecture of our backbone network.

Stage	Input	Output	Operator	Dilation Rate
1	512 × 512 Cin	128 × 128 × 96	Conv 4 × 4, stride 4 LN ConvNeXt block × 3	1
2	128 × 128 × 96	64 × 64 × 192	Downsample ConvNeXt block × 3	1
3	64 × 64 × 192	32 × 32 × 384	Downsample ConvNeXt block × 9	1
4	32 × 32 × 384	32 × 32 × 768	LN Conv 3 × 3, stride 1 ConvNeXt block × 3	2

Note: C_in represents the channel number of the input backbone network.

. Stage 1 uses a 4 × 4 convolution with a stride of 4 and layer normalization (LN) to directly downsample the image to one quarter of the input image size and then goes through three ConvNeXt blocks (Figure 2a). Stage 2 consists of a downsampling block (Figure 2b) and three ConvNeXt blocks. Stage 3 is also composed of a downsampling block, as well as nine ConvNeXt blocks. Stage 4 is different from the original ConvNeXt network structure. Chen et al. [52] found that too much downsampling can make the spatial resolution overly low, making it difficult to recover spatial information. However if no downsampling is performed, too much memory space will be used and the complexity will be increased. The authors proved that the best downsampling step to maintain efficiency and accuracy is 1/16. Therefore, in the last stage, we used ordinary convolution with a kernel size of 3 × 3 instead of downsampling, and then three ConvNeXt blocks. At the same time, the dilation rate of the convolution was changed to 2 to ensure that the network continued to acquire deep features without any reduction in the receptive field.

2.2. HOT Index Extraction

Numerous spectral characteristic indexes are available for cloud detection using the spectral threshold method, but many of them use the mid-infrared and thermal infrared bands, which are not universally applicable. Most remote sensing images only contain red, green, blue and near-infrared bands. The HOT index only needs red and blue bands, which have strong universality and high computational efficiency. Therefore, we used the HOT index as the cloud spectral feature index in this study, and the HOT index feature image and the original MS image were used as the input of the network.

The HOT index was originally designed to detect clouds and haze in Landsat 7 data [53], and it has been modified by Zhu et al. [21]. It is currently widely used in thin-cloud detection, cloud removal and fog removal. The design principle of the HOT index is that the reflectivities of cloud and non-cloud areas in the blue and red bands have certain differences. Generally speaking, the HOT index of a cloud is relatively large, so the cloud area can be effectively extracted. The formula is as follows [21]:

$$\mathrm{HOT}=\mathrm{B} - 0.5 \times \mathrm{R} - 0.08$$

(1)

where B is the top of atmosphere (TOA) reflectance of the blue band and R is the TOA reflectance of the red band.

2.3. Edge Feature Extraction Module

In the process of cloud detection, the cloud body is easy to detect due to it being quite different from other ground objects. However, the cloud boundary can easily be mixed with the surrounding pixel information. In addition, cloud boundaries are complex and diverse, making them difficult to accurately detect. In fact, accurate detection of boundaries can greatly improve the accuracy of cloud detection. Therefore, inspired by the Gabor feature extraction module of Zhang et al. [37], we employed the EFE module to enhance the edge detection ability of our network. We adopted the commonly used edge detection operator known as the Sobel operator for edge feature extraction.

The Sobel operator uses two 3 $\times$ 3 kernels convolved with the image to calculate approximations of the derivatives—one for horizontal changes and the other for vertical changes. The calculations are shown in Equation (2):

$$\begin{array}{c}\begin{array}{cc}\hfill G_x=& \left[f\left(x+1,y-1\right)+2f\left(x+1,y\right)+f\left(x+1,y+1\right)\right]\hfill \\ \hfill & -\left[f\left(x-1,y-1\right)+2f\left(x-1,y\right)+f\left(x+1,y+1\right)\right]\hfill \end{array}\\ \begin{array}{cc}\hfill G_y=& \left[f\left(x-1,y-1\right)+2f\left(x,y-1\right)+f\left(x+1,y-1\right)\right]\hfill \\ \hfill & -\left[f\left(x-1,y+1\right)+2f\left(x,y+1\right)+f\left(x+1,y+1\right)\right]\hfill \end{array}\end{array}$$

(2)

where $f\left(x,y\right)$ is the gray value at $\left(x,y\right)$ in the image, and $G_x, G_y$ are approximations of the gray partial derivatives in the horizontal and vertical directions, respectively, which result from the plane convolution of the original image in both directions.

Then, for each point in the image, the final gradient can be obtained from the square root of the square sum of the gradients $G_x$ and $G_y$. In practical applications, in order to improve efficiency, an approximation calculation without the square root is used. The calculation formula is shown in Equation (3):

$$G=\left|G_x\right|+\left|G_y\right|$$

(3)

where $G$ is the final gradient.

There is a large amount of edge detail information in low-level features. Considering the computational efficiency of the network and in order to obtain as much edge information as possible, we only added the EFE module to the initial input image. As shown in Figure 3, the EFE module first performs a 3 $\times$ 3 convolution to the input image. It can add affluent channels and extract low-level features, which is beneficial to the Sobel operator to obtain fine edge details. After using the Sobel operator, it is possible to automatically learn the difference between the image after edge extraction and the image without edge extraction through the subtraction and convolution operations. Then, the learned difference information is added to the image without edge extraction to obtain an edge enhancement effect, which helps the network accurately locate and restore edge details. Moreover, in this module, so as to speed up the training convergence of the network and reduce overfitting, batch normalization (BN), which is often employed for normalization in networks, is used after each convolution.

2.4. Focal Loss

Sample imbalance is a common phenomenon in the application of deep learning to vision tasks. In the process of cloud detection, this imbalance may even be obvious: in some images, large clouds appear continuously, while other images have very small proportions of clouds. This situation makes it difficult for networks to learn small-scale categories, resulting in poor model accuracy. The key to alleviating this problem is to increase the weight of small-scale categories and thus ensure the balance between different categories. However, the method for manually setting the weight requires many attempts, and the degree of automation and universality is poor. Therefore, we used focal loss [54] as the loss function in this study, as it can automatically adjust the weight according to the difficulty learning the sample to ensure the network works well for cloud detection.

Focal loss is a loss function proposed for target detection when the foreground and background are unbalanced. It makes the network focus on learning difficult samples by reshaping the standard cross-entropy. Equation (4) is the standard cross-entropy formula [55], and Equation (5) is the formula for focal loss:

$$CE\left(p_t\right)=-\log \left(p_t\right)$$

(4)

$$\mathrm{FL}\left(p_t\right)=-{\left(1-p_t\right)}^{\gamma }\log \left(p_t\right)$$

(5)

where $p_t$ is the model prediction probability, and $\gamma$ is the set parameter; we set it to 2 in accordance with the best experimental result obtained in the original paper [54]. When $p_t$ is small, this means that the sample has been misclassified and the corresponding ${\left(1-p_t\right)}^{\gamma }$ increases, so that the network will not ignore learning the misclassified sample. Conversely, when $p_t$ approaches 1, this indicates that the sample has been classified well. The ${\left(1-p_t\right)}^{\gamma }$ then changes to 0 and the sample is down-weighted.

3. Results and Discussion

3.1. Dataset Processing

Model training for deep-learning networks requires large and high-quality datasets. The L8 global cloud cover assessment validation dataset “L8 Biome Cloud Validation Masks” (L8 Biome) [56] is an internationally recognized cloud detection dataset. The L8 Biome dataset is used in most cloud detection studies due to its rich selection of surface types, the variety of which is beneficial for the performance of networks. We also chose the L8 Biome dataset as our source for model training and validation. We further selected the L8 Spatial Procedures for Automated Removal of Cloud and Shadow (SPARCS) dataset [57] and Gaofen-1 (GF-1) data to verify the generalization ability of the network model.

3.1.1. Datasets

• The L8 Biome dataset: This dataset includes 96 L8 images sampled from all over the world with sizes of 8000 × 8000 (30 m resolution) and manually generated cloud masks. The dataset has eight types of scene—urban, forest, shrubland, grass, snow, barren, wetlands and water—and each scene type contains 12 images. In order to ensure data heterogeneity and diversity, images are selected with different paths/rows and cloud patterns;
• The SPARCS dataset: This consists of 80 sub-images of L8 images, and the size is 1000 × 1000 (30 m resolution). The purpose of the dataset was to select 12 additional scenarios to use in evaluating the classifier and to reduce the risk of overfitting. Therefore, it was used as the extended experimental data in this study;
• GF-1 data: The GF-1 satellite is equipped with a panchromatic/multispectral (PMS) camera. The PMS camera can acquire panchromatic images with a resolution of 2 m and MS images with a resolution of 8 m (four bands—blue, green, red and near-infrared), with sizes of approximately 5000 × 5000. The spectral range and the spatial resolution of GF-1 are different from those of L8. Therefore, we also used the MS images as extended experimental data to further verify the scalability of the network.

3.1.2. Pre-Processing

The original cloud masks of the L8 Biome Dataset are divided into four categories: cloud shadow, clear, thin cloud and cloud. In this study, the main purpose was to detect clouds, so we combined the cloud masks into two categories: cloud and non-cloud. Since the calculation of the HOT index requires the TOA reflectance, we first performed radiometric calibration to convert the DN value to the TOA reflectance. Next, we selected the red, green, blue and near-infrared bands from the original image to synthesize the four-band images, and then we calculated the HOT index feature image. We also performed the same radiometric calibration and feature calculation process for the SPARCS dataset and the GF-1 data.

As the L8 image size was too large and our hardware processing capabilities for the experiment were limited, we cut the image into non-overlapping sub-images with sizes of 512 $\times$ 512 and obtained 18,466 sub-images. Then, we randomly divided the sub-images into the training set, the validation set and the test set accodring to the ratio of 6:1:3. The training set was used to train the network model, the validation set was used to adjust the model parameters during the training process and the test set was only used to evaluate the model performance and did not participate in the training process. The SPARCS dataset and GF-1 data were also cropped to the size of 512 $\times$ 512.

In addition, in order to ensure the network had strong generalization and robustness, we used data augmentation to enhance the number and complexity of training samples. We used a data augmentation strategy with random flips and random rotations. Figure 4 shows the original image and the data augmentation results. Finally, our dataset contained 33,237 training images, 1847 validation images and 5540 testing images.

3.2. Experiment Settings

3.2.1. Implementation Details

We implemented our network in the open source Pytorch [58] framework and executed it on a 64-bit Ubuntu 18.04 computer with two GeForce RTX 3090 GPU 24 GB cards. We used the stochastic gradient descent (SGD) optimizer with an initial learning rate of 0.001, momentum of 0.9 and weight decay of 0.0005. The cosine annealing learning rate strategy [59], which uses a slow-acceleration, slow-decline mode, reduces the learning rate through the cosine function. It has a good effect on network performance. Therefore, we used the cosine annealing learning rate strategy as the learning rate decay method, with a minimum learning rate of 1 × 10⁻⁸. When training the model, we set the batch size to 16 and carried out 40,000 iterations. To prevent overfitting, the drop path was added to the ConvNeXt block. The specific steps of our cloud detection model training and verification are shown in Algorithm 1.

Algorithm 1. Cloud detection model training and verification

Input:$Dat{a}_{train}, Dat{a}_{val},Dat{a}_{test}$ are the data for model training, validation and testing, respectively; $SPARC{S}_{img}$ and $G{F}_{img}$ are the images for the extended experiment; $iter$ is the number of iterations; $maxiter$ is the maximum number of iterations; $net$ is the initial network.

Output: model prediction results: $Dat{a}_{pred}, SPARC{S}_{pred}, G{F}_{pred}$; model: $ne{t}_{iter}, mode{l}_{iter}, mode{l}_{best}$; evaluation index: $O{A}_{val}, O{A}_{test}, P{R}_{test}, R{R}_{test}, F{1}_{test}, mIO{U}_{test}$.

1: while $iter$ < $maxiter$ do

2: $net$ $\leftarrow Dat{a}_{train}$

3: $ne{t}_{iter}\leftarrow$ update net parameters

4: if $iter$ % 200 == 0 then

5: $O{A}_{val}\leftarrow ne{t}_{iter}$ evaluate $Dat{a}_{val}$

6: $mode{l}_{iter}\leftarrow$ save the $ne{t}_{iter}$

7: end if

8: end while

9: $mode{l}_{best}\leftarrow$ choose the best model in $mode{l}_{iter}$ by $O{A}_{val}$

10: $Dat{a}_{pred}\leftarrow mode{l}_{best}$ predict $Dat{a}_{test}$

11: $O{A}_{test}, P{R}_{test}, R{R}_{test}, F{1}_{test}, mIO{U}_{test}\leftarrow$ compare $Dat{a}_{pred}$ with $Dat{a}_{test}$

12: $SPARC{S}_{pred}, G{F}_{pred}\leftarrow mode{l}_{best}$ perform cloud detection for $SPARC{S}_{img}$ and $G{F}_{img}$

3.2.2. Evaluation Metrics

In order to quantitatively evaluate the performance of our network, we used the overall accuracy (OA), precision ratio (PR), recall ratio (RR), F1 score and the mean intersection of union (mIOU) as evaluation metrics. Their calculation formulae are expressed as follows:

$$\mathrm{OA}=\frac{TP+TN}{TP+FP+TN+FN}$$

(6)

$$\mathrm{PR}=\frac{TP}{TP+FP}$$

(7)

$$\mathrm{RR}=\frac{TP}{TP+FN}$$

(8)

$$\mathrm{F}1 \mathrm{score}=\frac{2\mathrm{PR}*\mathrm{RR}}{\mathrm{PR}+\mathrm{PR}}$$

(9)

$$\mathrm{mIOU}=\frac{1}{N+1}{\displaystyle \sum }_{i=0}^N\frac{TP}{TP+FN+FP}$$

(10)

where $TP$ denotes the number of cloud samples that are correctly predicted as cloud samples, $TN$ denotes the number of non-cloud samples that are correctly predicted as non-cloud samples, $FP$ denotes the number of non-cloud samples that are wrongly predicted as cloud samples and $FN$ denotes the number of cloud samples that are wrongly predicted as non-cloud samples.

3.3. Ablation Experiments

To verify the effectiveness of our proposed ideas, we analyzed the network prediction accuracy of the different modules in our model under the ConvNeXt backbone network. For a fair comparison, all cases were trained, validated and tested on the same dataset with the same hyperparameters. The results are shown in

Table 2. Comparison of performance in the ablation experiments.

Method	OA	PR	RR	F1 Score	mIOU
ConvNeXt	95.52%	93.87%	94.76%	94.31%	91.06%
ConvNeXt + HOT index	95.78%	94.47%	94.84%	94.66%	91.55%
ConvNeXt + EFE	96.28%	95.54%	95.08%	95.31%	92.53%
ConvNeXt + HOT index + EFE	96.47%	95.59%	95.51%	95.55%	92.90%

Note: Bold represents the best results.

. Representative cloud detection comparison results are also shown in Figure 5.

• Effectiveness of the HOT index: Spectral feature information is helpful in extracting clouds. In Table 2, it can be seen that, compared with the network without the HOT index, most of the evaluation indicators for the network with the HOT index were improved. The greatly improved PR shows that the addition of the HOT index spectral feature information was indeed beneficial for cloud detection. In Figure 5a–c, we can also see that the addition of the HOT index added the spectral feature information of the cloud to the network, which made it possible to effectively eliminate some confusing non-cloud pixels around thin clouds to improve accuracy;
• Effectiveness of the EFE module: Edge information plays a critical role in cloud detection. In Table 2, it is can be seen that the performance improvement resulting from the EFE module was much greater than that of the HOT index. The EFE module increased the OA value from 95.52% to 96.28%, the PR from 93.87% to 95.54%, the RR from 94.76% to 95.08%, the F1 score from 94.31% to 95.55% and the mIOU from 91.06% to 92.53%. In Figure 5c, it can be seen that the addition of the EFE module made the detection results more accurate, and the boundary fitting of the cloud was also strengthened. However, there were some pixels that were mistaken for non-cloud areas due to weak boundaries between thick and thin clouds;
• Effectiveness of the fusion of the HOT index and the EFE module: As shown in Table 2, the best performance for the network resulted from the fusion of the HOT index and the EFE modules, with which the OA increased by 1.0%, the PR increased by 1.7%, the RR increased by 0.8%, the F1 score increased by 1.2% and the mIOU increased by 1.8%. It can be seen in Figure 5f that the models that fused the two modules exhibited more accurate cloud detection results and finer edges. Only adding the EFE module caused a weak boundary error (see Figure 5e), but the spectral feature information added by the HOT index alleviated this situation. Moreover, the edge information extracted by the EFE module enabled some pixels with similar spectral features to be distinguished and the performance improved.

3.4. Comparative Experiments

To demonstrate the effectiveness and accuracy of the network, we compared the proposed CD_HIEFNet network with classical methods, including Fmask [21], FCN8s (the prediction result is upsampled by a factor of 8) [31], U-Net [44], SegNet [45], DeepLabv3+ (the backbone network is Xception) [43] and CloudNet [36]. They were evaluated using the same dataset with the same training parameter settings. The quantitative results are shown in

Table 3. The quantitative results for different networks.

Method	OA	PR	RR	F1 Score	mIOU
Fmask	88.75%	88.03%	84.27%	86.11%	79.16%
FCN8s	95.56%	94.58%	94.22%	94.40%	91.14%
U-Net	93.03%	89.14%	92.98%	91.02%	86.38%
SegNet	93.13%	90.65%	91.90%	91.27%	86.61%
DeepLabv3+	95.55%	95.39%	93.51%	94.44%	91.12%
CloudNet	94.72%	92.98%	93.64%	93.31%	89.55%
CD_HIEFNet	96.47%	95.59%	95.51%	95.55%	92.90%

Note: Bold represents the best results.

. Furthermore, in order to show the performance of the network with various surface types, we also selected representative cloud detection results for nine surface types for visualization and qualitative analysis: barren, forest, grass, shrubland, snow, urban, building and wetlands, as shown in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14, respectively.

As can be seen from Table 3, deep-learning methods had obvious advantages over Fmask. Fmask combines multi-band spectral information for cloud detection, which can have a certain detection effect, but simple decision-making based on thresholds still resulted in poor detection results. Due to the ability to fit complex features using deep learning, other networks all had high accuracy for cloud detection. For FCN8s, VGG16 net was used for feature extraction, and the deep features and shallow features were well-fused at the decoder stage, so it showed good detection results. In the evaluation results for U-Net, SegNet and CloudNet, the RRs were higher than the PRs, but DeepLabv3+ had a higher PR than RR. The reason for this is that U-Net and SegNet use skip connections and inverse pooling upsampling mechanisms, respectively, at the decoder stage to closely combine deep and shallow features, allowing them to better restore pixel details. CloudNet uses shortcut connections in each block to capture the global and local features and present details. DeepLabv3+, on the other hand, simply fuses with one of the shallow features to restore the detailed information, resulting in lower recall. However, at the encoder stage, DeepLabv3+ introduces the ASPP module to obtain multi-scale information, and this module can acquire more global and accurate semantic features, so the precision of the cloud detection results was greatly improved compared to other networks. Our proposed network CD_HIEFNet performed the best in the detection results among all the networks. It not only had high accuracy but also maintained a low missed-detection rate and a high recall rate. The results benefited from our designed encoder and decoder. We selected ConvNeXt as the backbone network, added the HOT index image and introduced the ASPP module to obtain more accurate semantic features. The improved decoder enabled our network to combine the deep and shallow features closely and locate the boundary accurately and the EFE module enhanced the edge information, resulting in the network achieving optimal detection of edge and detail information.

In Figure 6, it can be seen that the seven methods exhibited good detection of clouds for the bare land surface type. Although FCN8s combined the features of shallow and deep layers, the direct upsampling by a factor of 8 still did not result in rich details in the detection results. Fmask could perform boundary fitting and broken cloud detection well by judging multi-band information pixel by pixel. U-Net, SegNet and CloudNet also enhanced the details because of the fusion of features at different stages. However, as seen in the red circles in Figure 6c,e,f,h, these methods resulted in certain false detections. Both DeepLabv3+ and CD_HIEFNet used the ASPP module to reduce false detections and thus achieved accurate results, but, because CD_HIEFNet also had the EFE module added to improve the decoder, the boundaries and details of our network were better fitted than that with DeepLabv3+.

In Figure 7 (the forest surface) and Figure 8 (the grass surface), we can see that the other classic networks were able to detect most of the thick cloud areas, but there were some missed detections in thin cloud areas. CD_HIEFNet obtained accurate and comprehensive spectral feature information for the distinction between thin clouds and thick clouds due to the addition of the HOT index image input, so it could basically detect thin clouds. This also illustrates the importance of spectral feature information in remote sensing images for deep-learning cloud detection.

As seen in the red circle in Figure 9 (the shrubland surface), the skip connection used in the U-Net network helped in the upsampling recovery but, although it had a certain effect, it was still difficult to recover small details. The inverse pooling upsampling adopted by SegNet records the pooling indices to restore pixels, and it performed relatively well for the presentation of boundary and detail information. However, the undifferentiated record pooling index resulted in many misjudgments in this network. CloudNet captured global and local features at each stage and was very effective for fine cloud detection, but it mixed some invalid global information, resulting in misjudgments. DeepLabv3+ had no missed detections because of the ASPP module, but possibly irrelevant global information led to many misjudgments in cloud detection. Fmask and CD_HIEFNet avoided these two situations, but the results for Fmask had holes inside clouds, while the results for CD_HIEFNet were the closest to the manually generated cloud masks.

As shown in Figure 10, for snow discrimination, all the networks except for CD_HIEFNet misjudged some snow areas as clouds. This shows that CD_HIEFNet still performed well in an indistinguishable situation such as with clouds and snow, which proves the effectiveness of our added modules. As seen in Figure 11 and Figure 12 (the urban and building surfaces), Fmask, U-Net, SegNet, DeepLbv3+ and CloudNet all produced different degrees of misjudgment for bright surfaces and building areas (the red circles in Figure 12a). FCN8s showed no misjudgments, but the performance for detailed information was poor. In contrast, CD_HIEFNet not only showed no misjudgments but also demonstrated an excellent presentation of broken clouds and boundaries. Similarly, as seen in Figure 13 and Figure 14, for the results for water and wetlands surfaces, CD_HIEFNet demonstrated the best broken cloud detection and boundary fitting, but the rest of the methods also performed well.

In summary, the seven methods demonstrated good detection for cloud areas. Fmask is a pixel-by-pixel cloud detection method that performs well with details but only through threshold judgment, so the detection results included holes in the clouds and some misjudgments. FCN8s can perform rough detection but, due to the direct upsampling by a factor of 8, there were certain difficulties in the boundary fitting and detection of thin clouds and broken clouds. U-Net, based on skip connections, showed certain improvements in thin cloud detection and boundary positioning, but false detections often occurred. SegNet has great advantages for small object detection and detail recovery due to the inverse pooling upsampling, but since the pooling coefficient is the equal weight recovery, there is a lot of unnecessary information, leading to significant noise and discontinuity in the detection results. CloudNet is a fully convolutional network specially designed for cloud detection that can obtain the global and local features in each convolution block. It could present details well but showed errors due to some invalid global information. The introduction of the ASPP module in DeepLabv3+ produced cloud detection results that were generally better than the previous networks, but there were still false detections and missed detections for snow and thin clouds, respectively. CD_HIEFNet added the HOT index and EFE modules to provide the network with spectral and edge feature information at the same time, and thus the network not only had good cloud detection across various surface types but also certain improvements in the challenging detection of thin clouds, broken clouds and snow.

3.5. Extended Experiments

In order to verify the extendibility of the proposed CD_HIEFNet network, we used the trained model to perform cloud detection with the SPARCE dataset and GF-1 images.

3.5.1. Experiments in the SPARCS Dataset

We chose the SPARCE dataset to verify the generalization of the CD_HIEFNet network to other L8 images. Some representative results are shown in Figure 15.

Figure 15d–f,j,k show the cloud detection results for CD_HIEFNet with the barren, vegetation, urban, water, and snow surface types, respectively. It was found that CD_HIEFNet had excellent detection results for different cloud types with different surface types, irrelevant of whether broken clouds or thin clouds were included. As Figure 15c shows, CD_HIEFNet did not misjudge the highlighted building area on the basis of its detection of broken clouds, which indicates the strong robustness of the network. Similarly, as shown in Figure 15h, CD_HIEFNet also discriminated between snow and clouds well and detected clouds accurately. However, there were still some missed detections for very small clouds, as shown in Figure 15b, probably because such a small cloud type did not exist in the training dataset. As seen in Figure 15l, CD_HIEFNet achieved a good fit for the boundaries of thin clouds and broken clouds, which indicates the effectiveness of our network.

To sum up, CD_HIEFNet had good generalization and robustness with other L8 images. It not only accurately detected clouds with different surface types but also achieved successful detection for indistinguishable situations, such as snow, bright objects, and thin clouds. However, for cloud types that did not exist in the original dataset, CD_HIEFNet still demonstrated some missed detections, so it is necessary to establish a more comprehensive cloud training dataset to improve our network’s performance.

3.5.2. Experiments with GF-1 Images

Several GF-1 images with scene types corresponding to the L8 Biome dataset were selected for generalization experiments. Representative results are shown in Figure 16.

Similarly to the experiments described in Section 3.5.1, we again selected the barren, vegetation, urban, water, and snow surface types for cloud detection, as shown in Figure 16a–c,g,h. As can be seen in Figure 16d,e, CD_HIEFNe demonstrated powerful detection for bare land and vegetation areas and effectively detected both thin and broken clouds. The results in Figure 16c show that CD_HIEFNet could effectively distinguish clouds from highly reflective areas, such as buildings in urban areas. However, as seen in Figure 16c,d, some broken clouds were missed, probably because the resolution of GF-1 images is much higher than that of L8 images. Figure 16k shows that our network achieved effective detection in snow areas. Similarly, Figure 16l shows that CD_HIEFNet still had good results for boundary fitting of thin clouds and broken clouds.

In general, CD_HIEFNet achieved good cloud detection results with the extended GF-1 images. This shows that the network has strong scalability for data from different sensors, proving that the model trained with existing datasets was able to be quickly applied to other, new sensors. Of course, if the model can be fine-tuned with the data from new sensors, a better-fitting network will be produced.

4. Conclusions

In this paper, we proposed an encoder–decoder network named CD_HIEFNet that uses the HOT index and edge feature extraction for optical remote image cloud detection. First, we added the HOT index feature image and the MS image input together to make the network effectively eliminate confusing non-cloud pixels around thin clouds. Secondly, we used an EFE module to increase the edge information, so the network could effectively detect isolated clouds and fragmented clouds and fit the cloud edges. Finally, we chose ConvNeXt as the backbone network in the encoder and fused the shallow and deep features from bottom to top to compensate for the edge and local information loss at the decoder stage, which further improved the accuracy and generalization of the network.

The ablation experimental results showed that the proposed HOT index and EFE module indeed contributed to the improvement in the performance of the network. Moreover, the comparative experiments and generalization experiments both proved the superiority of CD_HIEFNet. CD_HIEFNet could perform accurate and efficient cloud detection with most surface types, even for thin or broken cloud detection and snow discrimination. Therefore, CD_HIEFNet has great practical application prospects and can provide new insights for cloud detection.

Nevertheless, due to the existing cloud training datasets, our network still suffers from certain limitations. In the future, we will build richer and more comprehensive cloud training datasets to increase the network generality. Moreover, in practical applications, efficiency is an issue that must be considered. We will also explore the lightweight performance of the network and optimize the model to balance efficiency and accuracy.