Serial GANs: A Feature-Preserving Heterogeneous Remote Sensing Image Transformation Model

Abstract

In recent years, the interpretation of SAR images has been significantly improved with the development of deep learning technology, and using conditional generative adversarial nets (CGANs) for SAR-to-optical transformation, also known as image translation, has become popular. Most of the existing image translation methods based on conditional generative adversarial nets are modified based on CycleGAN and pix2pix, focusing on style transformation in practice. In addition, SAR images and optical images are characterized by heterogeneous features and large spectral differences, leading to problems such as incomplete image details and spectral distortion in the heterogeneous transformation of SAR images in urban or semiurban areas and with complex terrain. Aiming to solve the problems of SAR-to-optical transformation, Serial GANs, a feature-preserving heterogeneous remote sensing image transformation model, is proposed in this paper for the first time. This model uses the Serial Despeckling GAN and Colorization GAN to complete the SAR-to-optical transformation. Despeckling GAN transforms the SAR images into optical gray images, retaining the texture details and semantic information. Colorization GAN transforms the optical gray images obtained in the first step into optical color images and keeps the structural features unchanged. The model proposed in this paper provides a new idea for heterogeneous image transformation. Through decoupling network design, structural detail information and spectral information are relatively independent in the process of heterogeneous transformation, thereby enhancing the detail information of the generated optical images and reducing its spectral distortion. Using SEN-2 satellite images as the reference, this paper compares the degree of similarity between the images generated by different models and the reference, and the results revealed that the proposed model has obvious advantages in feature reconstruction and the economical volume of the parameters. It also showed that Serial GANs have great potential in decoupling image transformation.

1. Introduction

In recent years, there have been more and more applications of remote sensing images in environmental monitoring, disaster prevention, intensive farming, and homeland security. In practice, optical images are widely used due to their high spectral resolution and easy interpretation. The disadvantage is that they are sensitive to meteorological conditions, especially clouds and haze, which severely limits their use for observation and monitoring of ground targets [1]. In contrast, synthetic aperture radar (SAR) sensors can overcome adverse meteorological conditions by creating images using a longer wavelength of radio waves to obtain all-day and all-weather continuous observations. Although SAR images have significant advantages over optical images, their application is still limited by the difficulty of SAR image interpretation. First, because synthetic aperture radar is a side range-measuring instrument, the imaging effect is affected by the distance between the target and the antenna, which can lead to geometric distortion in SAR images [2]. Therefore, compared with optical images, it is more difficult for human eyes to understand the details of SAR images. Secondly, synthetic aperture imaging is a coherent imaging method in which the radio waves in the radar beam are aligned in space and time. While this consistency provides many advantages (required by the synthetic aperture process to work), it also leads to a phenomenon called speckle, which reduces the quality of SAR images and makes image interpretation more challenging [3]. Therefore, it is difficult to distinguish structural information directly from SAR images, which may not necessarily become easier with the increase in spatial resolution [4]. Considering the above two points, how to effectively use and interpret the target and scene information in SAR images has become an important issue that users of SAR data need to pay attention to. Under the condition of reasonable use of SAR image amplitude information, if the SAR image can be converted into a near-optical representation that is easy to recognize by human eyes, this will create new opportunities for SAR image interpretation.

Deep learning is a powerful tool for the interpretation of SAR images. Some scholars have reconstructed clear images by learning hidden nonlinear relations [5,6,7,8,9,10]. This type of method uses a residual learning strategy to overcome speckle noise by learning the mapping between the speckle image and the corresponding speckle-free reconstruction so that it can be further analyzed and explained. Although this mapping learning may be an ill-posed problem, it also provides a useful reference for SAR image interpretation.

In addition to convolutional neural networks, image translation methods in the field of natural images and human images provide other ideas for SAR-to-optical image transformation, such as through conditional generative adversarial networks (CGANs) [11,12,13,14]. This type of method separates the style and semantic information in image transformation, so it can transform from the SAR image domain to the optical image domain, and also ensures the transformed images have the prior structural information of the SAR images and the spectral information of optical images. In previous studies, CGANs were first applied to the translation tasks of text to text [15], text to image [16], and image to image [17,18], and are suitable for generating unknown sequences (text/image/video frames) from known conditional sequences (text/image/video frames). In recent literature, the applications of CGANs in image processing were mostly in image modification. This includes single image super-resolution [17], interactive image generation [18], image editing [19], image-to-image translation [11], etc. CGANs have been used in SAR-to-optical transformation in recent years. In the literature [20,21,22], different improved SAR-to-optical transformation models based on CycleGAN and pix2pix have been proposed. The general idea of these models is to improve the model structure and loss function, but they are not designed specifically for the differences of imaging principle between SAR images and optical images, so they do not have universal applicability.

In order to solve the problem of heterogeneous image transformation in principle, as shown in Figure 1a, we decomposed the SAR-to-optical transformation task into two steps: the first step was to implement the transformation from the SAR image domain to optical grayscale image domain through the Despeckling GAN. In this step, we aimed to suppress the speckle effect of SAR images and reconstruct the semantic structural information and texture details of SAR images. In the second step, we transformed the optical grayscale images obtained in the first step into optical color images through the Colorization GAN. The two subtasks are relatively independent and have low coupling, which can reduce the semantic distortion and spectral distortion in the process of direct SAR-to-optical transformation.

The main contributions of this paper are as follows.

• Unlike the existing methods of direct image translation, this paper proposes a feature-preserving SAR-to-optical transformation model, which decouples the SAR-to-optical transformation task into SAR-to-gray transformation and gray-to-color transformation. This design effectively reduces the difficulty of the original task, enhancing the feature details of the generated optical color images and reducing spectral distortion.
• In this paper, Despeckling GAN is proposed to transform SAR images into optical grayscale images, and its generator is improved on the basis of the U-net [11]. In the processing, Despeckling GAN guides SAR images to generate optical grayscale images based on the texture details of SAR images by gradient maps, thus enhancing the semantic and feature information of transformed images [23].
• In this paper, Colorization GAN is proposed for despeckled grayscale image colorization. Its generator adopts a convolutional self-coding structure. We establish short-skip connections in different levels and long-skip connections between the same level of encoding and decoding. This structure design enables different levels of image information to flow in the network structure, to generate more realistic images with hue information.

The rest of this paper is structured as follows. Section 2 introduces the materials involved in this paper. Section 3 introduces the method in detail, including the network structure and the loss function. In Section 4, the experimental results are given, which are discussed and evaluated based on indexes. Section 5 shows the discussion of this paper. The last part of the paper (Section 6) gives the conclusions and prospects for future work.

2. Materials

Due to the lack of a large number of paired SAR and optical image datasets, deep learning-based SAR-to-optical translation research has mainly followed the idea of the CycleGAN [12] model; that is, unpaired image transformation. With the decrease in the cost of remote sensing images, a new idea has been presented to solve the cross-modal transformation, by using an image transformation method based on the Generative Adversarial Network. In the literature [24], Schmitt et al. published the SEN1-2 dataset to promote SAR and optical image fusion in deep learning research. The SEN1-2 dataset is a traditional remote sensing image dataset obtained by the SAR and optical sensors of the Sentinel-1 and Sentinel-2 satellites. As part of the Copernicus Project of the European Space Agency (ESA), Sentinel satellites are used for remote sensing tasks in the fields of climate, ocean, and land detection. The mission is being carried out jointly by six satellites with different observation applications. Sentinel-1 and Sentinel-2 provide the two most conventional SAR and optical images respectively, so they have been widely studied in the field of remote sensing image processing. Sentinel-1 is equipped with a C-band SAR sensor, which enables it to obtain high-positioning-accuracy SAR images regardless of weather conditions [25]. In its unique SAR imaging mode, the nominal resolution of Sentinel-1 is not less than 5 m, while providing dual-polarization capability and a very short equatorial access time (about 1 week) [26]. In the SEN1-2 dataset, Sentinel-1 images were collected in the interference wide (IW) swath mode, and the result obtained is the ground-range-detected (GRD) products. These images contain the backscatter coefficient in dB scale for every pixel spacing of 5 m in azimuth and 20 m in range. In order to simplify the operation, the dataset pays more attention to the VV polarization data and ignores the data of VH polarization. Sentinel-2 consists of two polar-orbiting satellites in the same orbit, with a phase difference of 180 degrees [27]. For the Sentinel-2 part of the dataset SEN1-2, the researchers used red, green, and blue channels (i.e., Bands 4, 3, and 2) to generate realistic RGB grid images. Because cloud occlusion will affect the final effect, the cloud coverage of the Sentinel-2 image in the dataset is less than or equal to 1%. SEN1-2 is composed of 282,384 pairs of related image patches, which come from all over the world and all weathers and seasons. It is the first large, open dataset of this kind and has significant advantages for learning a cross-modal mapping from SAR images to optical images. With the aid of the SEN1-2 dataset, we were able to build a new model that is different from the previous methods, the Serial GANs model proposed in this paper. Figure 2 shows some examples of image pairs in SEN1-2.

3. Method

The heterogeneous transformation from SAR images to optical images is an ill-posed problem. The transformation results are often not ideal due to speckle noise, SAR image resolution, and other factors. Inspired by the ideas of pix2pix, CycleGAN and pix2pixHD, as shown in Figure 3a–d, this paper attempted to introduce optical grayscale images as the intermediate transformation domain $Y$. The transformation task from the SAR image domain $X$ to the optical color image domain $Z$ was completed in two steps by two generators ($P$ and $Q$) and two discriminators ($D_Y$ and $D_Z$) as shown in Figure 3e. First, the generator $P$ completes the mapping: $X\to Y$, in which the SAR image is transformed into the optical grayscale image, and the corresponding discriminator $D_Y$ is used to promote the transformation of the SAR image in the source domain $X$ to the optical grayscale image in the domain $Y$, which is difficult to distinguish from the real optical grayscale image. Then, the generator $Q$ completes the mapping: $Y\to Z$, in which the optical grayscale image is transformed to the optical color image, and the corresponding discriminator $D_Z$ is used to promote the transformation of the optical grayscale image in the intermediate domain $Y$ to the optical color image in the domain $Z$, which is difficult to distinguish from the optical color image. In this way, the original transformation process from the SAR image to the optical color image is divided into two steps, reducing the semantic distortion and feature loss in the process of direct transformation from the SAR image to the optical color image.

As shown in Figure 4, the transformation from SAR images to optical images can be defined as the mapping transformation $T=P○Q(P:X\to Y,Q:Y\to Z)$, from the source domain $X$ to the target domain $Y$. Suppose that $x^{(i)}$ is a random sample taken from the SAR image domain $X$, and its distribution function is ${\mathbb{P}}^{(i)}(x)$, and the random sample $x^{(i)}$ mapped to the optical grayscale image domain is $y^{(i)}$. The final task of the network proposed in this paper is $T:X\to Z$, in which the final distribution function is $\mathbb{P}\{T○x^{(i)}=z^{(i)}|x^{(i)}\}$ generated from our network.

3.1. Despeckling GAN

Generator $P$: As shown in Figure 5a, this paper used an improved U-net as the generator of Despeckling GAN. The input SAR image was encoded and decoded to output the optical grayscale image. A structure similar to the convolutional self-encoding network enables the generation network to better predict the optical grayscale image corresponding to the SAR image. The encoding and decoding process of the generator works on multiple levels to ensure that the overall contour and local details of the original SAR image are extracted on multi-scales. In the decoding process, the network upsamples the feature map of the previous level to the next level through deconvolution and adds the feature map of the same level in the encoding process through a long-skip connection to get an average merge (Merge). In U-net, this process is completed by concatenation. At the same time, skip connections are also used in each residual block, which has the advantage of overcoming the gradient disappearance problem of the network during training.

Discriminator $D_Y$: As shown in Figure 5b, PatchGAN, which is commonly used in GAN, was used as the discriminator. The process of heterogeneous image transformation includes the transformation of the content part and feature detail part. The content part refers to the similarity in content between the generated image and the original image, and the feature detail part refers to the similarity in features between the generated image and the target image. With PatchGAN, feature details can be maintained [11].

The loss function of the Despeckling GAN generator includes CGAN loss, $L_1$ loss, and feature-preserving loss. Based on the premise of the existing paired training data, this paper used the CGAN loss function to improve the performance of the generator. Through supervised training, the generator $P$ learns the mapping from $X$ to $Y$, and this makes the discriminator $D_Y$ judge true. The network structure of the discriminator has the function of distinguishing fake images from real images. Therefore, the CGAN loss from $X$ to $Y$ is:

$${ℒ}_{\mathrm{GAN}}\left(P,D_Y\right)={\mathbb{E}}_{x^{(i)},y^{(i)}}\left[\log D_Y\left(x^{(i)},y^{(i)}\right)\right]+{\mathbb{E}}_{x^{(i)}}\left[\log \left(1-D_Y\left(P\left(x^{(i)}\right),x^{(i)}\right)\right)\right].$$

(1)

In the reconstruction loss design, the $L_1$ loss is used to minimize the difference between the optical gray image and the generated image.

$${ℒ}_{\mathrm{Recon}}\left(P,x^{(i)}\right)=\mathbb{E}\left[{\Vert P\left(x^{(i)}\right)-x^{(i)}\Vert }_1\right].$$

(2)

In the best state $T^{*}$, the output of the network $T^{*}\left(x^{(i)}\right)$ should be similar to the optical gray image $y^{(i)}$. In order to preserve the feature details of SAR images, this paper proposed a gradient-guided feature-preserving loss [28]. If $M(\cdot )$ denotes the operation to calculate the image gradient map, the loss of feature-preserving is:

$${ℒ}_{\mathrm{FP}}\left(P,x^{(i)}\right)=\mathbb{E}\left[M\left(P\left(x^i\right)\right),M\left(y^{(i)}\right)\right].$$

(3)

For images $I$, $M(\cdot )$ is as follows:

$$\begin{array}{l}{I}_x\left(x,y\right)=I\left(x+1,y\right)-I\left(x-1,y\right),\\ I_y\left(x,y\right)=I\left(x,y+1\right)-I\left(x,y-1\right),\\ \nabla I\left(x,y\right)=\left(I_x\left(x,y\right),I_y\left(x,y\right)\right),\\ M\left(I\right)={\Vert \nabla I\Vert }_2.\end{array}$$

(4)

Specifically, the operation $M(\cdot )$ can be easily implemented by convolution with a fixed convolution kernel.

Therefore, the total training loss of the Despeckling GAN is:

$${ℒ}_{{\mathrm{GAN}}^1}=\arg \underset{G}{\min }\underset{D}{\max }{ℒ}_{GAN}\left(P,D_Y\right)+{\beta }_1{ℒ}_{\mathrm{Recon}}\left(P,x^{(i)}\right)+{\gamma }_1{ℒ}_{\mathrm{FP}}\left(P,y^{(i)}\right).$$

(5)

where ${\beta }_1$ and ${\gamma }_1$ are weighted values.

3.2. Colorization GAN

Colorization GAN completed the transformation from optical gray images to optical color images. Its principle comes from [29], which proved that, compared with Figure 6a, the colorization result of Figure 6b was better, so the latter was adopted in this paper. When a single channel gray image ${\widehat{y}}^{(i)}\in {ℝ}^{H\times W\times 1}$ is input, the model learns the mapping ${\widehat{z}}_{ab}^{{}^{(i)}}=Q\left({\widehat{y}}^{(i)}\right)$ from the input gray channel to the corresponding $Lab$ space color channels ${\widehat{z}}_{ab}^{{}^{(i)}}\in {ℝ}^{H\times W\times 2}$, where, $H$ and $W$ represent the height and width respectively. Then, the RGB image ${\widehat{z}}^{(i)}$ is obtained by synthesizing ${\widehat{z}}_{ab}^{{}^{(i)}}$ and ${\widehat{y}}^{(i)}$. The advantage of this method is that it can reduce the ill-posed problem, such that the colorization result is closer to the real image.

As shown in Figure 7, the generator of the Colorization GAN uses a convolutional self-coding structure, which establishes short-skip connections within different levels and long connections between the same levels of encoding and decoding. This kind of structure design enables different levels of image information to flow in the network so that the hue information of the generated image is more real and full. The discriminator of the Colorization GAN is PatchGAN [11]. Recent studies have shown that adversarial loss helps to make colorization more vivid [29,30,31], and this paper also followed this idea. During training, we input the reference optical color image and the generated image one by one into the discriminator; the discriminator output was 0 (fake) or 1 (real). According to the previous methods, the loss of the discriminator is the sigmoid cross-entropy.

Among them, the adversarial loss is expressed as follows:

$${ℒ}_{\mathrm{GAN}}\left(Q,D_Z\right)={\mathbb{E}}_{y^{(i)},z^{(i)}}\left[\log D_Z\left(y^{(i)},z^{(i)}\right)\right]+{\mathbb{E}}_{y^{(i)}}\left[\log \left(1-D_Z\left(y^{(i)},Q\left(y^{(i)}\right)\right)\right)\right].$$

(6)

In order to make the generated color distribution closer to the color distribution of the reference image, we defined the ${ℒ}_1$ loss in the $Lab$ space, which is expressed as follows:

$${ℒ}_1\left(Q\right)={\mathbb{E}}_{y^{(i)},z^{(i)}}\left[{\Vert Q\left(y^{(i)}\right)-z^{(i)}\Vert }_1\right].$$

(7)

Therefore, the total loss function of the Colorization GAN model is as follows:

$${ℒ}_{{\mathrm{GAN}}^2}=\arg \underset{G}{\min }\underset{D}{\max }{ℒ}_{GAN}\left(Q\left(y^{(i)}\right),z^{(i)}\right)+{\beta }_2{ℒ}_1\left(Q,y^{(i)}\right).$$

(8)

where ${\beta }_2$ is a weighted value.

4. Experiments and Results

As the SEN1-2 dataset covers the whole world and contains 282,384 pairs of SAR and optical color images across four seasons, some of which are overlapped, in order to facilitate the training, the original dataset was randomly sampled according to the stratified sampling method. The dataset was divided into the training dataset, validation dataset, and test dataset, and their respective proportions were about 6:2:2. The experiment of the proposed method was carried out on the computing platform of two 11G GPU GeForce RTX 2080Ti and i9900k CPUs using PyTorch. The input size of the images was 256 × 256, and the batch size was set to 10. In the experimental simulation, 200 epochs were set in the GAN training and optimized by the Adam optimizer. The sum of parameters was set to 0.5 and 0.999, respectively. The initial learning rate of the experiment was set to 0.0002. The first 100 epochs remained unchanged and then decreased to 0 according to the linear decreasing strategy.

Considering that season and landscape will affect the training results of the model, we selected image pairs of different seasons and landscapes and followed the principle of equilibrium [32]. As shown in

Table 1. Number of different types of images selected in our dataset.

Season	Landscape	Training	Validation	Test	Total
Spring	River valley	279	90	74	443	2218
	Mountains and hills	280	91	73	444
	Urban residential area	275	97	70	442
	Coastal city	278	87	77	442
	Desert	277	96	74	447
Summer	River valley	278	92	77	447	2219
	Mountains and hills	276	96	74	446
	Urban residential area	279	93	70	442
	Coastal city	276	93	72	441
	Desert	275	95	73	443
Fall	River valley	278	96	70	443	2215
	Mountains and hills	275	89	74	441
	Urban residential area	278	95	72	445
	Coastal city	279	91	75	445
	Desert	277	89	77	442
Winter	River valley	277	91	78	446	2218
	Mountains and hills	275	94	73	442
	Urban residential area	278	95	71	444
	Coastal city	278	92	75	445
	Desert	277	93	71	441
Total	-	5545	1855	1470	8870

, the number of SAR and optical image pairs in four seasons is approximately the same, and the number of image pairs of different landscapes in each season is also approximately the same.

4.1. Experiment 1

In order to verify the effectiveness of the proposed method, four groups of experiments were designed using the same dataset and different conditions. The four groups of experiments were carried out according to the single variable principle. In Group 1, the unimproved generators $P$ and $\mathcal{Q}$ were used, and the loss function included $GAN$ loss and reconstruction loss. In Group 2, the improved generators $P$ and $Q$ were used, and the loss function included $GAN$ loss and reconstruction loss. In Group 3, the unimproved generators $P$ and $Q$ were used, and the loss function included $GAN$ loss, reconstruction loss, and feature-preserving loss. In Group 4, the improved generators $P$ and $Q$ were used, and the loss function included reconstruction loss, reconstruction loss, and feature-preserving loss. The relationship between the four groups of experiments is shown in

Table 2. Grouping experiments under different conditions.

	Original Loss	Improved Loss	Original Networks	Improved Networks
Group 1	√		√
Group 2	√			√
Group 3		√	√
Group 4		√		√

.

As shown in Figure 8, the first column shows the SAR images collected by the SEN-1 satellite. The second column shows the SAR images collected by the SEN-2 satellite. The third, fourth, fifth, and sixth columns show the experimental results of Group 1, Group 2, Group 3, and Group 4, respectively. Through visual comparative analysis, it can be seen that improving the network structure and loss function can improve the quality of SAR-to-optical transformation, especially by enhancing the feature detail information of the generated image. It can map the SAR image to the optical color image to the maximum extent and help the interpretation of the SAR image.

In order to compare the detailed information of the generated images, Figure 9 shows the detailed comparison between the SEN-2 images and the four groups of experimental results. According to the subjective evaluation criteria, the results of improving the model and loss function at the same time are closer to the SEN-2 images. Only improving the loss function can improve the details of the generated images, but its effect is inferior to that of improving the model. The detailed comparison of the four groups of experimental results once again proves that the improvement measures proposed in this paper are effective. By comparing the two situations of improving model and improving loss function, it can be found that improving model contributes more to the results.

In order to quantify the effectiveness of the method, the final transformation effect (IQA) was measured by calculating the structural similarity (SSIM) [33,34], and the feature similarity (FSIM) [35]. Both indexes were calculated between the generated image $\widehat{z}$ and the corresponding SEN-2 image $z$. Assuming that the generated image is $\widehat{z}$, and the corresponding SEN-2 image is $z$, the SSIM calculation formula is as follows:

$$SSIM\left(\widehat{z},z\right)={\left[l\left(\widehat{z},z\right)\right]}^{\alpha }{\left[c\left(\widehat{z},z\right)\right]}^{\beta }{\left[s\left(\widehat{z},z\right)\right]}^{\gamma },\alpha ,\beta ,\gamma >0$$

(9)

Among them:

$$l\left(\widehat{z},z\right)=\frac{2{\mu }_{\widehat{z}}{\mu }_z+c_1}{{\mu }_{\widehat{z}}^2+{\mu }_z^2+c_1}$$

(10)

$$c\left(\widehat{z},z\right)=\frac{2{\sigma }_{\widehat{z}z}+c_2}{{\sigma }_{\widehat{z}}^2+{\sigma }_z^2+c_2}$$

(11)

$$s\left(\widehat{z},z\right)=\frac{{\sigma }_{\widehat{z}z}+c_3}{{\sigma }_{\widehat{z}}{\sigma }_z+c_3}$$

(12)

$l\left(\widehat{z},z\right)$, $c\left(\widehat{z},z\right)$, and $s\left(\widehat{z},z\right)$ in the equation represent the brightness comparison, contrast comparison, and structural comparison, respectively. ${\mu }_{\widehat{z}}$ and ${\mu }_z$ represent the mean of $\widehat{z}$ and $z$, ${\sigma }_{\widehat{z}}$ and ${\sigma }_z$ represent the standard deviation of $\widehat{z}$ and $z$, ${\sigma }_{\widehat{z}z}$ represents the covariance of $\widehat{z}$ and $z$, and $c_1$, $c_2$ and $c_3$ are constant constants (so that the parent of the equation is not zero). In actually, $\alpha$ = $\beta$ = $\gamma$ = 1, $c_3=c_2/2$, SSIM is represented as:

$$SSIM\left(\widehat{z},z\right)=\frac{\left(2{\mu }_{\widehat{z}}{\mu }_z+c_1\right)\left({\sigma }_{\widehat{z}z}+c_2\right)}{\left({\mu }_{\widehat{z}}^2+{\mu }_z^2+c_1\right)\left({\sigma }_{\widehat{z}}^2+{\sigma }_z^2+c_2\right)}$$

(13)

Another index, the FSIM, is a feature similarity evaluation index, which uses phase consistency (phase consistency (PC)) and gradient features (gradient magnitude (GM)), as follows:

$$FSIM=\frac{{\displaystyle {\sum }_{x\in \Omega }{S}_L\left(x\right)\cdot P{C}_m\left(x\right)}}{{\displaystyle {\sum }_{x\in \Omega }P{C}_m\left(x\right)}}.$$

(14)

Which:

$$S_{PC}\left(x\right)=\frac{2P{C}_1\left(x\right)\cdot P{C}_2\left(x\right)+T_1}{P{C}_1^2\left(x\right)+P{C}_2^2\left(x\right)+T_1}$$

(15)

$$S_G\left(x\right)=\frac{2G_1\left(x\right)\cdot G_2\left(x\right)+T_2}{G_1^2\left(x\right)+G_2^2\left(x\right)+T_2}$$

(16)

$$S_L\left(x\right)={\left[S_{PC}\left(x\right)\right]}^{\alpha }\cdot {\left[S_G\left(x\right)\right]}^{\beta }$$

(17)

$S_{PC}\left(x\right)$, $S_G\left(x\right)$, and $S_L\left(x\right)$ represent the phase consistent (PC) similarity, gradient feature (GM) similarity, and PC-GM fusion similarity, respectively.

The similarity indicators of the four experimental schemes were calculated as

Table 3. The model generated result indicators under different improvement measures. The number in bold indicates the optimal value under the corresponding index.

Scheme	SSIM	FSIM
No Improvement	0.2428	0.9000
Improved Network	0.2432	0.9023
Improved Loss	0.2435	0.9015
Both Improvements	0.2442	0.9042

. By comparing the results of the second, third, and first rows of the table, it can be seen that after improving the generator structure and loss function, both SSIM- and FSIM-generating images had been significantly improved, and the combined use of improved generators and loss functions obtained better results than improving the generator structure or loss functions alone.

4.2. Experiment 2

In order to verify the performance of the proposed method in preserving the SAR image features, the proposed algorithm was compared with pix2pix, CycleGAN, and pix2pixHD, respectively. During training, the Serial GANs train the generator $P$ and the discriminator $D_Y$ first, and then the training generator $Q$ and the discriminator $D_Z$, respectively, with 200 epochs. In Figure 10, the first column shows the SAR images collected by the SEN-1 satellite, the second column shows the optical color images collected by the SEN-2 satellite, and the third, fourth, and fifth columns show the experimental results of pix2pix, CycleGAN, and pix2pixHD, respectively. According to the results, the proposed method can significantly preserve the details of SAR images in the process of heterogeneous transformation, with results as good as pix2pixHD. What is more, the volume of parameters of the model proposed in this paper was significantly lower than in the pix2pixHD model.

In order to compare the details of the images generated by different models, Figure 11 shows the details of the results generated by the proposed method compared with the four methods of pix2pix, CycleGAN and pix2pixHD. According to the subjective evaluation criteria, the results of the proposed method and pix2pixHD are closer to the Sentinel satellite image. The generation results of pix2pix and CycleGAN are inferior to the first two methods. Although the results of the proposed method and pix2pixHD are not significantly different, the subsequent comparison will show that the proposed method is superior to pix2pixHD.

In order to quantitatively measure the advantages of the method, four image quality evaluation indexes (IQA), including PSNR, SSIM, FSIM, and MSE, were selected to quantitatively evaluate the method. As shown in

Table 4. Comparison of the indexes between the images generated by four methods and the SEN-2 images. The number in bold indicates the optimal value under the corresponding index.

	PSNR	SSIM	MSE	FSIM
pix2pix [11]	13.8041	0.2431	0.0673	0.8987
Cycle GAN [21]	13.5052	0.2314	0.0749	0.9039
pix2pix HD [13]	13.4112	0.2347	0.0780	0.9046
Ours	13.9267	0.2442	0.0669	0.9042

, it can be seen from the data that the proposed model achieved the best in PSNR, SSIM, and MSE, and the second-best in FSIM.

The above experimental results show the effectiveness of the proposed method and the superiority to pix2pix and CycleGAN from both qualitative and quantitative aspects. In order to further illustrate that our method is better than pix2pixHD, we draw a performance comparison diagram reflecting the model size and FSIM value. As shown in Figure 12, although the FSIM value of our method is 0.0004 lower than that of pix2pixHD, the model size of our method is about half of that of pix2pixHD, so the advantage of our method is more obvious.

5. Discussion

The existing SAR-to-optical method is a one-step transformation method; that is, it directly transforms SAR images into optical RGB images. However, spectral and texture distortions inevitably occur, reducing the accuracy and reliability of the final transformation result. Moreover, the direct use of CycleGAN and pix2pix in SAR-to-optical transformation only reconstructs the original image at the pixel level, without restoring the spectrum and texture. Such results may not be suitable for further image interpretation. Inspired by image restoration and enhancement technology, a Serial GAN image transformation method is proposed here and used for SAR-to-optical tasks.

Based on SEN 1-2 SAR and optical image datasets, the effectiveness of the proposed method was verified through ablation experiments. Through qualitative and quantitative analysis with several SOTA image transformation methods, the superiority of the proposed method was verified. The image transformation method we proposed uses SAR images as prior information to restore and reconstruct SAR images based on the gradient contour and spectrum. The advantage of this is avoiding the mixing distortion caused by directly transforming the SAR image into an optical image, and the final transformation result has better texture detail and an improved spectral appearance. At the same time, our method does not simply involve learning the SAR-optical mapping but restores and reconstructs the SAR image from both the texture information and the spectral information so that it has an interpretation advantage similar to that of the optical image. Note that our proposed method was better than CycleGAN and pix2pix in the index of the transformation results, and some indexes were better than pix2pixHD. From an indicator point of view, this difference was small. However, from intuitive observation, the method proposed in this paper was significantly better than CycleGAN and pix2pix. The reason for this is that our method is not a simple transformation but the reconstruction of SAR images, which restores SAR images from the perspective of image theory. In comparison with the SOTA model pix2pixHD, the proposed method has no obvious advantage in the test value, but the parameter size of the model is about half that of pix2pixHD, which means that our method has more advantages in application. However, the proposed method also has some potential limitations. First, although we considered different seasons and different land types (urban, rural, semi-urban, and coastal areas) in the training data, supervised learning inevitably depends on the data. For different SAR image resolutions and speckle conditions, the results of the transformations will be different. In addition, because supervised learning requires a large number of training samples, the training effect of the model may not be ideal for a dataset with a small sample size. Therefore, problems arising from transfer learning, weakly supervised learning, and cross-modal technology will need to be solved in the future.

6. Conclusions and Prospects

To address the problem of feature loss and distortion in SAR-to-optical tasks, this paper proposed a feature-preserving heterogeneous image transformation model using Serial GANs to maintain the consistency of heterogeneous features, and reduce the distortion caused by heterogeneous transformation. An improved U-net structure was adopted in the model, which was used for SAR image Despeckling GAN, and then the image was colored by Colorization GAN to complete the transformation from a SAR image to an optical color image, which effectively alleviated the uncertainty of transformation results caused by information asymmetry between heterogeneous images. In addition, the end-to-end model architecture also enabled the trained model to be directly used for SAR-to-optical image transformation. At the same time, this paper introduced the feature-preserving loss, which enhanced the feature details of the generated image by constraining the gradient map. Through intuitive and objective comparison, the improved model effectively enhanced the detail of the generated image. In our view, Serial GANs have great potential in other heterogeneous image transformations.

Furthermore, they can provide a common framework for SAR image and photoelectric image transformation. In the future, we will consider incorporating multisource heterogeneous images into a Multiple GANs hybrid model to provide support for the cross-modal interpretation of multisource heterogeneous remote sensing images.