Clean Collector Algorithm for Satellite Image Pre-Processing of SAR-to-EO Translation

Abstract

In applications such as environmental monitoring, algorithms and deep learning-based methods using synthetic aperture radar (SAR) and electro-optical (EO) data have been proposed with promising results. These results have been achieved using already cleaned datasets for training data. However, in real-world data collection, data are often collected regardless of environmental noises (clouds, night, missing data, etc.). Without cleaning the data with these noises, the trained model has a critical problem of poor performance. To address these issues, we propose the Clean Collector Algorithm (CCA). First, we use a pixel-based approach to clean the QA60 mask and outliers. Secondly, we remove missing data and night-time data that can act as noise in the training process. Finally, we use a feature-based refinement method to clean the cloud images using FID. We demonstrate its effectiveness by winning first place in the SAR-to-EO translation track of the MultiEarth 2023 challenge. We also highlight the performance and robustness of the CCA on other cloud datasets, SEN12MS-CR-TS and Scotland&India.

1. Introduction

The field of remote sensing has undergone a remarkable transformation in recent years, driven by the synergistic advancement of satellite technology and deep learning. Satellite systems have evolved rapidly, bringing about significant improvements in data acquisition. These advancements include enhanced spatial and temporal resolutions, expanded spectral ranges, and more sophisticated sensor capabilities [1,2,3]. Parallel to this, the exponential growth in computational power, coupled with the development of increasingly sophisticated deep learning algorithms, has unlocked new possibilities for processing and analyzing the vast quantities of satellite data now available [4].

In particular, deep learning models have shown exceptional promise in tackling various remote sensing challenges. Convolutional Neural Networks (CNNs) have proven highly effective for tasks such as image classification and object detection [5,6], while generative adversarial networks (GANs) have demonstrated their utility in image-to-image translation tasks [7,8,9,10,11,12]. These approaches have significantly enhanced the accuracy and efficiency of applications ranging from environmental monitoring and urban planning to disaster response [13].

Satellite remote sensing data possess remarkable capabilities in capturing a diverse array of scenarios, rendering it extensively employed in diverse tasks, including environmental monitoring [14,15], change detection [16,17,18], and disaster assessment [19,20,21].

SAR has the advantage of capturing images regardless of weather conditions or time of day. For instance, SAR can accurately observe terrain, buildings, and objects even in cloudy weather or during night-time. In contrast, electro-optical (EO) imagery relies on visible light, making it more intuitive and easier for humans to interpret. Converting SAR imagery to the optical domain, therefore, provides easily interpretable images regardless of weather and lighting conditions. As a result, various research efforts and competitions have focused on developing SAR-to-EO conversion models. It is important to note that EO data, being based on visible light, are highly susceptible to weather variations, which can impact data quality and consistency in SAR-EO conversion. Consequently, effective pre-processing has emerged as a critical factor in maximizing the potential of deep learning in satellite remote sensing applications [22]. However, its susceptibility to noise, arising from factors such as weather, clouds, smoke, and lighting conditions, imposes substantial constraints on the utilization of electro-optical (EO) data. Specifically, the presence of noise poses a significant challenge in continuous EO imagery tasks for environmental monitoring. To surmount these limitations, a methodology incorporating synthetic aperture radar (SAR) is being proposed [23,24], offering advantages such as the ability to characterize surface properties and collect data during both day and night, even under adverse weather conditions. Nonetheless, SAR data present challenges in human interpretation, limiting their practicality. In addressing these hurdles, a solution has been suggested to convert SAR data into EO data, facilitating easier comprehension and interpretation by humans.

Cabrera Armando et al. [21] proposed a methodology to learn better mapping between SAR and EO images by leveraging additional complementary modalities, such as OpenStreetMap, infrared (IR), latitude, and longitude. Wang Haixia et al. [25] proposed a parallel generative adversarial neural network (Parallel-GAN) consisting of the backbone image translation subnetwork to convert SAR images to EO images and an EO image reconstruction subnetwork to extract hierarchical optical features. Wang Lei et al. [26] combined the advantages of CycleGAN [27] and pix2pix [28] to propose a supervised cycle-consistent adversarial neural network (S-CycleGAN) to preserve both land cover and structure information. While these methods perform well, they assume that the dataset is noise-free. Specifically, the public datasets [29,30,31] used in most studies have been cleaned by experts to remove various real-world noises, and existing methodologies have not focused on data cleaning. If a model is trained on a noisy dataset, the performance degradation can be critical.

Among the real-world noises in EO data, cloud occlusion causes the most problems by hiding geographic information. To address these issues, a cloud removal task is proposed to convert cloud images into cloud-free images. Methods that utilize generative models [32,33,34] to translate cloud images into cloud-free images are effective, but may differ from real-world geographic information when texture information is lacking (half or all clouds). The use of time series data [35,36] has also been proposed to mitigate these problems, but collecting time series data from the same region can be quite time consuming and expensive. In addition, the use of other modalities [37,38] to preserve texture and geographic information has been considered, but like the methods mentioned above, they require refined data free of real-world noise to train high-performance models.

To address these issues, we propose the Clean Collector Algorithm (CCA), which effectively refines real-world noisy data (clouds, night, missing data). The CCA consists of three main stages. Firstly, using the QA60 band and experimentally obtained pixel values, we eliminate images with significant cloud cover. Secondly, we remove night and pixel missing data by calculating the ratio of brightness values in the HSV space to pixels with low values. Lastly, to handle the remaining cloud data that could not be removed by the QA60 band, a hand-crafted constructed subset of cloud EO data is created, and the Fréchet inception distance (FID) values of each EO data are used. We have confirmed that the FID values allow us to approximately assess the extent of cloud cover in the data. Figure 1 displays example images from each stage.

We participated in the MultiEarth 2023 challenge [39] for environmental monitoring in the Amazon rainforest and achieved first place in the SAR-to-EO translation track by applying the CCA proposed in this paper. The dataset of the competition was collected in the Amazon rainforest with high rainfall, and it contains a large amount of environmental noise (weather, clouds, night and day, missing pixels, etc.).

To validate the effectiveness of the CCA, we evaluated it on an evaluation dataset of MultiEarth 2023 that we built ourselves. To prove that our method is not biased toward one dataset, we also validated its performance on two existing datasets, SEN12MS-CR-TS [40] and Scotland&India [41], which contain real-world noise. The code is available at https://github.com/cjf8899/MultiEarth2023_SAR2EO_1st_Place_Solution (accessed on 11 November 2024).

2. Methodology

EO data collected in the real world are unrefined and contain real-world noise such as clouds, weather, illumination differences, etc. An image translation model trained on data containing such noise may produce noisy results during translation and experience performance degradation, e.g., a translation model trained on data containing a lot of clouds may produce clouds in the translation results. To address these issues, we propose the Clean Collector Algorithm (CCA) for effective data cleaning in three stages.

2.1. QA60, Outlier

The datasets collected by Sentinel-2 are multi-spectral datasets that support monitoring studies by observing soil, coastal, and vegetated areas. There are various bands according to wavelength (B1 B12, QA60, etc.) in these data. QA60 has a spatial resolution of 60 m and is a bitmask band that contains cloud mask information. Using this cloud mask information, we can easily identify images containing clouds. Note that it is not necessary to perform this task if the ingested datasets do not provide a cloud mask.

Furthermore, we found that in the datasets collected by Sentinel-2, images containing clouds tend to be above a certain pixel value. Figure 2 shows a graph of the maximum pixel value for each EO image (B4, B3, and B2 bands corresponding to RGB channels) versus the FID value for that image (the method for building the FID is in Section 2.3). In this figure, we can see that a clear image with no clouds has low pixel values and high FID values, while an image with clouds has high pixel values and low FID values. Note that it is not necessary for datasets built from images that are already normalized.

Therefore, as a first stage, we excluded images with $1\in M_{QA}$ from the training dataset, where $M_{QA}$ is the QA60 band cloud mask. Based on the analysis in Figure 2, we also considered pixels above a threshold $\alpha$ as indicative of cloud data and excluded them from the training dataset. We empirically used an $\alpha$ of 4096.

2.2. Missing, Night-Time

Due to satellite sensor malfunctions and errors in data collection, remote sensing images are missing a lot of information, as shown in Figure 1b, which reduces their utility and hinders subsequent interpretation. In addition, these missing images can change the distribution of training data, which can adversely affect learning.

Furthermore, the collected data could be taken at any time of the day and include night-time data. Such night-time data are difficult to interpret due to the lack of illumination and the limited self-luminescence of objects, resulting in very sparse shapes, structures, and characters of objects.

The main purpose of translating SAR to EO is that it should be easily interpretable by humans, but these issues can negatively impact that goal. To remove these problematic data, we convert the image to HSV space. HSV space provides a value representing the brightness of that pixel. Missing data have a value of 0 because there are missing pixels, and night data have low values because it is dark. Therefore, we minimize the impact of dark data and missing data on the training data by removing images with an average value lower than 30. In addition, we remove images where the number of pixels in the RGB space with a pixel value of less than 10 exceeds 30% of the total pixels because the percentage of missing data is small and may not be removed by the brightness value.

2.3. Fréchet Inception Distance

While the QA60 band mask has the potential to remove images where the data contain a lot of clouds, it has the limitation of not being able to remove shallow and few clouds. To address these limitations, we utilize the Fréchet inception distance (FID) value, a feature-based similarity measure, to remove residual cloud data. The FID is used in image generation tasks to measure the quality of the generated image and as a metric to assess the similarity of the distribution to the real image. Inspired by this concept, we can calculate and quantify the distribution similarity between a subset of clouds and an EO image.

The last stage is as follows. To start with, we hand-select and build a cloud subset of 1000 images consisting of images that contain a lot of clouds. With the set of cloud images we built, we chose images that are thick and have at least 4/5 of the total area of the image covered by clouds so that we can identify cloud images using the FID score. Then, we calculated the FID values between the cloud subset and the EO data to create a value set $S=\left\{S_i\right|i=1,2,\dots ,N\}$, where N represents the number of EO data pre-processed in the previous two stages. We found that the FID value can roughly determine the degree of cloud cover contained in the data. We set a threshold of 0.3 through several experiments and that threshold is used in Equation (1) to remove EO data with values lower than the threshold:

$$F_{th}=\min \left(S\right)+(\max \left(S\right)-\min \left(S\right))\times \beta$$

(1)

where $\beta$ is a hyperparameter that determines the quality of the training dataset; we used 0.4 for all experiments.

To summarize, we propose the CCA with three stages to remove images containing real-world noise that affects performance during training. The corresponding python-like CODE is shown in Algorithm 1.

Algorithm 1 Python Implementation of CCA

# M_QA : QA60 band mask # D_cloud : Cloud subset of 1000 images # FID : Functions to calculate the FID score per-image stage1_2_list = [] # Passed through stages 1,2 set stage3_list = [] # Passed through stage 3 set for img, M_QA in D:     # Stage 1: QA60 & Outlier remove     if (1 in M_QA) or (img > alpha):         continue     # Stage 2: Missing & Night-time remove     img = normalize(img) # [0,255]     img_hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)     brightness = np.mean(img_hsv[:,:,2])     img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)     threshold = img.shape[0] ∗ img.shape[1] ∗ 0.3     missing_num = len(img_gray[img_gray<=10])     if (30 > brightness) or (missing_num > threshold):         continue     stage1_2_list.append(img) # Stage 3: Frechet inception distance remove S = FID(np.array(stage1_2_list), D_cloud) S_min, S_max = np.min(S), np.max(S) F_th = S_min + (S_max − S_min) ∗ beta # Equation~(1) for img, score in zip(stage1_2_list, S):     if score > F_th:         stage3_list.append(img)

3. Experiments

3.1. Dataset

• MultiEarth 2023 The MultiEarth 2023 dataset is a collection of remote sensing data with sentinel-1 and 2 bands, containing 780,706 and 5,241,119 images respectively, all in 256 × 256 size. It focuses on Earth ecosystem analysis through challenges like that of the Amazon rainforest, which contains lots of clouds.
• SEN12MS-CR-TS The SEN12MS-CR-TS dataset is a multi-modal and multi-temporal dataset designed for global and all-season cloud removal techniques. It consists of 53 diverse regions of interest, each covering about 40 × 40 km². The dataset includes over 700 patches per region, with a patch size of 256 × 256.
• Scotland&India The Scotland&India dataset consist of 444 samples and all in 256 × 256 size. The data include clear sky images from 2019 and cloudy images from 2020.

3.2. Baseline Selection

We selected three GAN-based methods, pix2pix, pix2pixHD, and SPADE, to compare the performance of the CCA.

• pix2pix pix2pix proposes a conditional generative adversarial network (cGAN) for image-to-image translation. The model can perform various image translation tasks in a single unified framework and has become the basis for various GAN-based models.
• pix2pixHD pix2pixHD is an extended version of pix2pix and proposes a model that enables high-resolution image-to-image translation. It is designed to generate high-resolution images (2048 × 1024 resolution) that the existing pix2pix model has difficulty processing, making it very useful for realistic image generation and complex scene processing.
• SPADE Spatially-adaptive Denormalization (SPADE) is a GAN-based model designed to generate high-resolution images for tasks like image synthesis and transformation. It excels at producing realistic images from semantic layouts (segmentation maps) and effectively overcomes the limitations of previous GAN-based models.

3.3. Implementation Details and Metrics

We pre-processed the SAR data following the official papers of SEN12MS-CR-TS and Scotland&India. Also, we generated three-channel SAR data as described by Cabrera Armando et al. [21]. The SAR and EO data from all datasets we used for training and inference were normalized to 0∼1 when input into the model.

To evaluate the effect of our CCA, we selected metrics such as the SSIM, PSNR, MAE, and MSE to assess the performance of the generated EO data. These evaluation metrics were used to evaluate the visual quality difference and loss of quality information between the EO data generated by the translation model and the real EO data.

3.4. Quantitative Results

To evaluate the performance of our proposed CCA methods, we compared them to several baseline methods on the MultiEarth 2023, SEN12MS-CR-TS, and Scotland&India datasets.

Table 1. Performance comparison of different SAR-to-EO translation models on the MultiEarth 2023, SEN12MS-CR-TS, and Scotland&India datasets with and without the CCA. ↑ indicates higher values are better (for SSIM and PSNR), ↓ indicates lower values are better (for MAE and MSE), The best results are in bold.

Method	MultiEarth 2023				SEN12MS-CR-TS				Scotland&India
	SSIM↑	PSNR↑	MAE↓	MSE↓	SSIM↑	PSNR↑	MAE↓	MSE↓	SSIM↑	PSNR↑	MAE↓	MSE↓
pix2pix	0.449	14.814	0.165	0.040	0.482	13.232	0.189	0.054	0.555	15.803	0.142	0.034
pix2pixHD	0.444	16.066	0.132	0.030	0.566	16.483	0.130	0.027	0.527	15.931	0.147	0.040
SPADE	0.407	15.668	0.148	0.034	0.462	14.661	0.169	0.049	0.522	14.592	0.168	0.037
pix2pix + CCA	0.593	19.696	0.086	0.013	0.539	14.179	0.165	0.044	0.700	18.950	0.105	0.022
pix2pixHD + CCA	0.503	19.634	0.086	0.013	0.601	18.085	0.105	0.020	0.709	20.301	0.111	0.030
SPADE + CCA	0.493	17.590	0.116	0.024	0.495	15.390	0.152	0.039	0.535	15.041	0.156	0.034

shows the quantitative results of the experiments measured by popular evaluation metrics in image translation tasks such as the SSIM, PSNR, MAE, and MSE. All three baseline methods were outperformed when the CCA was applied to all datasets. On the MultiEarth 2023 dataset, the earliest proposed pix2pix model among the evaluation models performed the best on all evaluation metrics with +0.144, +4.882, −0.079, and −0.027 compared to the case without applying the CCA. In addition, if we compare the pix2pix model with the CCA to the other models without the CCA, the pix2pix model achieves better performance on all datasets. These results indicate that the noise in the training data has a significant impact on performance.

3.5. Ablation Study

We conducted an ablation study to evaluate the contribution of each phase of the CCA. Our goal is to convert SAR images to high-quality EO images by removing noisy EO images during training.

Table 2. Classification results of CCA.

	Performance Metrics				Confusion Matrix
	Accuracy	Precision	Recall	F1-Score	TN	FP	FN	TP
Score	0.9746	0.9513	0.7913	0.8639	19,910	92	474	1797

shows the results of the ablation study to evaluate the contribution of each stage of the CCA. These results were obtained using the pix2pixHD model. The baseline model, without any CCA stages, struggled with the inherent challenges posed by noisy EO images, particularly those affected by cloud coverage. This initial performance highlighted the limitations of handling raw data with significant atmospheric disturbances and inconsistencies.

The QA60&Outlier stage effectively identified and removed cloud-covered areas and outliers, especially at cloud edges, resulting in cleaner and more reliable outputs. This stage underscored the importance of preemptively addressing cloud-related distortions and outliers, which had been a major source of error in the baseline model.

The Missing&Night-time stage, which addressed gaps in data coverage and the inclusion of night-time images, which often lack sufficient detail for accurate analysis. Through excluding these lower-quality inputs, the model became more consistent in generating images with higher visual fidelity. The cumulative effect of these first two stages led to noticeable improvements in image clarity, reducing artifacts and inconsistencies that had hindered the baseline model.

The final stage moved beyond addressing large-scale issues like clouds and missing data, focusing instead on more nuanced distinctions in image quality. By leveraging the FID score, the model was able to detect and eliminate subtle defects, such as residual atmospheric distortions, that previous stages might have overlooked. This resulted in a significant leap in image quality, with reconstructions that were both visually clearer and quantitatively superior. The substantial improvement in PSNR (3.568 dB increase), along with the significant reductions in MAE (−0.046) and MSE (−0.017) from the baseline to the final stage, underscores the effectiveness of the complete CCA pipeline. These enhancements translated into visually clearer and more accurate EO image reconstructions from SAR inputs, with fewer artifacts and distortions caused by atmospheric conditions or data quality issues.

Furthermore, to directly evaluate the ability of the CCA algorithm to remove cloud data, we directly label a portion of the Multi-Earth 2023 data to perform cloud dataset classification. We randomly took 22,273 samples and labeled 2271 clean images and 20,002 noisy images. We evaluated the CCA algorithm on 1000 cloud data that are not included in the sample.

The 1000 cloud images utilized in the FID score step can be seen in Figure 3a. As can be seen, the selected data were of dense and thick clouds. In SAR-EO image conversion, images with shallow, broadly shaped clouds and images with small but thick clouds also cause performance degradation when training. With the cloud dataset in (a), we were able to successfully separate an even mixture of cloud data, unlike the few and shallow cases in (b).

Table 3. Ablation study results comparing the performance of the pix2pixHD model on the MultiEarth 2023 dataset with three stages of the CCA. The best results are in bold. Also, blue text indicates a performance improvement.

CCA	MultiEarth 2023
	SSIM↑	PSNR↑	MAE↓	MSE↓
-	0.444	16.066	0.132	0.030
QA60&Outlier	0.472	17.954	0.111	0.022
+ Missing&Night-time	0.489	18.212	0.098	0.017
+ FID score	0.503 (+0.059)	19.634 (+3.568)	0.086 (−0.046)	0.013 (−0.017)

identifies the clouds with a high performance of 0.97 in this experiment.

These results illustrate the cumulative benefits of the CCA pipeline. Each stage contributed meaningfully to improving the model’s performance, addressing both large-scale issues like cloud coverage and more subtle image quality factors. The final output demonstrated substantial improvements in both clarity and accuracy, highlighting the value of a multi-stage approach to handling noisy EO data.

3.6. Qualitative Results

Figure 4 qualitatively compares the image translation results of pix2pix, pix2pixHD, and SPADE for each dataset. As shown in the figure, without applying CCA, both Pix2Pix and SPADE struggle to learn the texture of EO data due to cloud interference, leading to blurry images. Pix2PixHD is particularly impacted, often generating clouds even when they are absent in the EO data. In addition, pix2pixHD is highly affected by night-time images, generating darker images compared to the other models. On the other hand, the model with CCA is less affected by real-world noise such as clouds, generating a cleaner image. SPADE generates better quality images compared to the case without applying CCA, and pix2pixHD, which generated the most clouds before applying CCA, generates images without clouds.

4. Conclusions

In this paper, we introduced the CCA (Clean Collector Algorithm) to address the challenges posed by noisy data in SAR-to-EO translation. The CCA is a three-stage process designed to remove specific sources of noise, such as clouds, night-time images, and missing data, ensuring cleaner and more reliable EO outputs. Even in utilizing a sensor with masking information about clouds, such as the QA60, it is not possible to remove all clouds that are actually present. Through incorporating FID scores, our method was able to effectively address these residual cloud artifacts, improving both visual quality and quantitative performance. This demonstrates that the precise handling of atmospheric distortions, cloud cover, and low-quality inputs is essential for generating accurate EO images from SAR data. While high-performance models are important, the careful pre-processing and removal of noisy data unlock the full potential of SAR-to-EO translation. Our results are validated across three benchmark datasets and further evidenced by a first place finish in the MultiEarth 2023 challenge.

The cloud removal method using the CCA’s FID score effectively filters out even faint clouds to some extent, but further research is needed for the detection and removal of very shallow clouds, such as foggy clouds. We hope that this work encourages further exploration into noise handling techniques for SAR-to-EO translation and broader remote sensing applications, where data quality plays a fundamental role in model success.