Two-Way Generation of High-Resolution EO and SAR Images via Dual Distortion-Adaptive GANs

Abstract

Synthetic aperture radar (SAR) provides an all-weather and all-time imaging platform, which is more reliable than electro-optical (EO) remote sensing imagery under extreme weather/lighting conditions. While many large-scale EO-based remote sensing datasets have been released for computer vision tasks, there are few publicly available SAR image datasets due to the high costs associated with acquisition and labeling. Recent works have applied deep learning methods for image translation between SAR and EO. However, the effectiveness of those techniques on high-resolution images has been hindered by a common limitation. Non-linear geometric distortions, induced by different imaging principles of optical and radar sensors, have caused insufficient pixel-wise correspondence between an EO-SAR patch pair. Such a phenomenon is not prominent in low-resolution EO-SAR datasets, e.g., SEN1-2, one of the most frequently used datasets, and thus has been seldom discussed. To address this issue, a new dataset SN6-SAROPT with sub-meter resolution is introduced, and a novel image translation algorithm designed to tackle geometric distortions adaptively is proposed in this paper. Extensive experiments have been conducted to evaluate the proposed algorithm, and the results have validated its superiority over other methods for both SAR to EO (S2E) and EO to SAR (E2S) tasks, especially for urban areas in high-resolution images.

1. Introduction

Electro-optical (EO) satellite imagery has been widely utilized for land surface property analysis [1,2,3,4,5]. However, poor visibility during the night and the presence of occlusions, such as clouds and haze, are hindering the practical deployment of EO data [6,7,8,9]. While weather and illumination determine the quality and reliability of EO imagery, synthetic aperture radar (SAR) provides an all-weather all-time imaging platform. Due to its unique sensor characteristics, clouds and other weather conditions have minimal effects on SAR imagery, which significantly enables the feasibility of SAR as a valuable source of information for earth observation in real-world settings. Since the analysis of SAR imagery conventionally requires expert knowledge and can be time-consuming, one potential and promising solution for robust, fast, and low-cost earth observation is to develop AI models specifically targeted for SAR imagery.

AI-based algorithms have made huge progress in processing natural images [10], and other imaging modalities have also benefited [11,12,13,14]. Yet, high acquisition costs and lack of large-scale, high-quality SAR imagery datasets pose challenges for intelligent remote sensing applications that automatically identify crucial features of buildings and installations, given that data quality and quantity play the most important roles in all AI-based deep learning methods. Unlike SAR imagery, large-scale EO datasets with annotations are publicly available and accessible. By leveraging annotated EO datasets, synthesized labor-free SAR datasets can improve the training stability and performance of large neural networks in the SAR imagery domain. Image-to-image translation (I2I) with deep generative adversarial networks (GANs) [15] has shown great potential in mapping images in two different domains while preserving the main content, such as style transfer and super-resolution. Although GAN-based I2I is widely used in the natural image domain, its application in remote sensing imagery requires further exploration and improvements due to significant divergences in imaging properties. SAR-to-EO (S2E) image translation has been studied in the past with the main objective of synthesizing EO from SAR imagery to aid the interpretation of EO images and compensate for deficiencies due to atmospheric conditions [9,16]. The other direction, EO to SAR (E2S), is seldom explicitly considered from the perspective of dataset augmentation via SAR image synthesis to tackle the data scarcity problem.

The vast majority of S2E methods are applicable to the E2S task. However, one common obstacle hampers the effectiveness of GAN-based I2I algorithms for EO-SAR imagery mapping in both directions, which is not adequately addressed by existing methods. Optical and radar imagery mainly differ from each other in terms of radiometric and physical image formation principles. These differences have introduced nonlinear distortions to EO-SAR image pairs within the same viewing regions, resulting in insufficient pixel-wise correspondence. Current supervised I2I algorithms have a relatively low tolerance for misalignment between training pairs [17], and unsupervised I2I algorithms struggle with local information loss [18,19]. Moreover, we noticed that existing S2E methods are evaluated on datasets with relatively low resolution, i.e., 5 m for Sentinel-1/2. Unlike rural and natural environments, the geometrical resolution of remote sensing data primarily determines its competence for various tasks in urban areas. High-resolution SAR images provide more detailed spatial and textural features of the Earth’s surface [20], opening up possibilities for high-level vision tasks in the remote sensing domain, such as building footprint extraction for urban planning and cargo ship detection for harbor monitoring. To address the limitations in remote sensing image translation, we propose a novel I2I algorithm designed to tackle the nonlinear distortions between EO and SAR. In our proposed algorithm, we design a two-way distortion-adaptive module to mitigate the ambiguity caused by non-uniform distortions, enhancing the pixel-level supervision during the training phase and facilitating the performance of both S2E and E2S tasks. Additionally, to promote further development of remote sensing image translation with high-resolution data, we introduce a new sub-meter resolution EO-SAR dataset, SN6-SAROPT. This new benchmark EO-SAR dataset features the challenges of modality transfer tasks in fine-scale remote sensing imagery, using metadata from SpaceNet 6 [21]. Heterogeneous land cover categories, including urban regions with varying building densities, as well as natural regions like farmland and forests, are all presented in SN6-SAROPT. The key contributions of this work are summarized as follows:

• We construct a new benchmark high-resolution (0.25 m spatial resolution) EO-SAR dataset SN6-SAROPT, which is comprised of over 700 non-overlapping image pairs (Capella Space’s X-band quad-pol SAR of size $1024\times 1024$ and Maxar WorldView 2 EO of size $512\times 512$) covering the port of Rotterdam, the Netherlands;
• We present a GAN-based I2I algorithm for EO-SAR images with a distortion-adaptive module to handle nonlinear distortions caused by different imaging characteristics of optical and radar sensors. To the best of our knowledge, this is the first work that models the nonlinear distortions between two imaging domains via a trainable network for remote sensing applications;
• Extensive experiments on both low-resolution and high-resolution datasets are conducted and have demonstrated the superiority of the proposed method for both S2E and E2S tasks, especially for high-resolution remote sensing data in urban areas.

This paper is organized as follows: Related works on image translation and EO-SAR datasets are reviewed in Section 2. In Section 3, our new benchmark dataset construction is introduced. The proposed novel GAN-based I2I algorithm for EO-SAR with a two-way distortion-adaptive module is described in detail in Section 4. In Section 5, experiments of both E2S and S2E image translation on multiple datasets are conducted and analyzed. Lastly, conclusions are given in Section 6.

2. Related Works

2.1. I2I Translation in Remote Sensing

Since the deep generative GAN model was proposed in 2014 [15], its application to vision tasks has been studied and evolved to tailor to a wide and diverse variety of deployment configurations. Image translation is one of the most common tasks where GANs have ushered in a revolution. As a generic training framework for generative model approximation, GAN-based models typically consist of a generator G and a discriminator $D_Y$, trained in an adversarial manner, as illustrated in Figure 1. Given training images from source domain X and target domain Y, I2I aims to learn a mapping function G, such that given any unseen image in domain X, it can synthesize a fake image indistinguishable from the real image from domain Y, yet keep the semantic content preserved. The generator and discriminator are optimized alternatively to compete with each other. The objective of the generator is to generate fake images capable of fooling the discriminator, while the discriminator is trained to differentiate fake images from real ones. The overall training objective of GANs is known as the adversarial loss; it can be expressed as follows:

$$arg\underset{G}{min}\underset{D_Y}{max}{\mathcal{L}}_{GAN}(G,D_Y)={\mathbb{E}}_{y\sim Y}[log{D}_Y\left(y\right)]+{\mathbb{E}}_{x\sim X}[log(1-D_Y\left(G\left(x\right)\right))]$$

(1)

Since the only supervising signal is from the discriminator $D_Y$, which makes predictions based on high-level features, the detailed reconstruction performance of the generator G is sub-optimal. Thus, on top of the original GANs optimized with Equation (1), recent GAN-based I2I algorithms have exploited additional loss terms for further enhancement. According to the availability of training data, GAN-based I2I algorithms can be summarized into two types: unsupervised and supervised methods. For the unsupervised methods, images from the source and target domain are required. For the supervised ones, training images from both domains need to be paired. Pix2Pix [17] is one representative supervised I2I algorithm, in which an extra loss term is added to guide the generator G to not only fool the discriminator $D_Y$ but also produce fake images close to the ground-truth in the image space. The training objective is given in Equation (2). In addition to the adversarial loss term $\mathcal{L}GAN$, a pixel-level reconstruction loss with a weighting coefficient $\lambda L_1$ in the form of $L_1$ norm is used to enforce that the fake images are similar to real ones in the target domain.

$$\begin{array}{cc}\hfill & arg{min}_G{max}_{D_Y}{\mathcal{L}}_{GAN}(G,D_Y)+{\lambda }_{L_1}{\mathcal{L}}_{L_1}\left(G\right)\\ \hfill & {\mathcal{L}}_{L_1}\left(G\right)={\mathbb{E}}_{x\sim X,y\sim Y}\left[\right||G\left(x\right)-{y\left|\right|}_1]\end{array}$$

(2)

CyleGAN [18] is one representative unsupervised I2I algorithm and it introduces the idea of transitivity into the I2I task. Two sets of generators (G and F) and discriminators ($D_Y$ and $D_X$) are used to learn two mapping functions, namely $G,D_Y$ for $X\to Y$ and $F,D_X$ for $Y\to X$. Instead of minimizing the distance between real and fake images, a cycle-consistency loss is imposed to regularize the mapping functions. The assumption adopted in CycleGAN is that the two mapping functions should be inverse to each other so that an image after forward-backward translations should be consistent with itself: $F\left(G\right(x\left)\right)\approx x$ and $G\left(F\right(y\left)\right)\approx y$ for $x\in X$ and $y\in Y$. As given in Equation (3), a new loss term $\mathcal{L}cyc$ with a weighting coefficient $\lambda L_{cyc}$, together with two adversarial losses, forms the final training objective of CycleGAN.

$$\begin{array}{cc}\hfill & arg{min}_{G,F}{max}_{D_X,D_Y}{\mathcal{L}}_{GAN}(G,D_Y)+{\mathcal{L}}_{GAN}(F,D_X)+{\lambda }_{L_{cyc}}{\mathcal{L}}_{cyc}(G,F)\\ \hfill & {\mathcal{L}}_{cyc}(G,F)={\mathbb{E}}_{x\sim X}\left[\right||x-{F\left(G\left(x\right)\right)\left|\right|}_1]+{\mathbb{E}}_{y\sim Y}\left[\right||y-{G\left(F\left(y\right)\right)\left|\right|}_1]\end{array}$$

(3)

By virtue of the cycle-consistency constraint, paired images from the source and target domains are no longer required. In CycleGAN, mapping functions $\{G,F\}$ in both directions are learned and the new supervising signal comes in the form of self-reconstruction. The illustration of comparisons between Pix2Pix and CycleGAN is shown in Figure 2.

Both Pix2Pix and CycleGAN were applied to remote sensing image translation in previous works [19,22,23]; the empirical results show that these two GAN-based I2I algorithms are superior to others, but there is still significant room for improvement due to relatively inferior IQA values compared to the natural image domain. Additionally, several GAN-based algorithms have been proposed recently, specifically tailored for EO-SAR translation tasks. Yang et al. [19] proposed FG-GAN to tackle the detailed deficiencies in unsupervised methods. Wang et al. [24] incorporated a vision transformer (ViT) into the GAN-based I2I framework to capture long-distance feature correlations, resulting in a hybrid cGAN. In [25], Tan et al. designed Serial GANs, which decouple the translation process into two stages (despeckling and colorization) to enhance image quality. Our method differs from these works in the following two aspects: 1. Data: Geometric distortions are barely discussed and studied in the previous works due to the false sense of success on relatively low-resolution SAR datasets, e.g., SEN1-2. The focus of our work is to explore and handle novel challenges in I2I translation tasks for sub-meter high-resolution EO-SAR images. 2. Problem Setting: FG-GAN is specifically designed for unsupervised EO-SAR translation. Hybrid cGAN requires additional category information, such as the land type associated with each pair. Serial GANs are limited to single-polarization SAR. Our model, however, is designed for general supervised I2I translation tasks, with no specific requirements on polarization or additional information. Therefore, we dig into the current limitations of Pix2Pix and CycleGAN on high-resolution EO-SAR images and aim to further boost the performance by incorporating prior physical knowledge into the algorithm.

As an active data collection, SAR images are the result of received backscatter after sending electromagnetic waves to the Earth’s surface. In contrast, passive optical sensors, similar to the human visual system, create more intuitive EO images for human perception. Due to the disparate imaging mechanisms, paired EO and SAR images differ not only in style but also in geometric changes. Specifically, the geometric distortions in SAR images can alter the shape of buildings, terrains, and almost all installations with significant height changes. These geometric distortions are non-linear and non-uniform, as volume background natural objects like trees and grass will be almost free from elevation changes. Therefore, geometric distortions, including foreshortening and layover induced by the side-looking nature of SAR images, not only create interpretation barriers for non-experts but also hinder the utilization of deep neural networks due to poor pixel-wise correspondence between EO and SAR images. For example, the use of $L_1$ loss in image space, as proposed by Pix2Pix, can introduce unnecessary noise. Similar challenges are also encountered in other tasks, such as the image-matching problem. Traditional methods [26,27] that rely on hand-crafted descriptors have been designed for image matching between SAR and EO. However, these methods fail on complex geometric distortions as they only consider low-level features like edges and corners. More recent methods [28] that utilize deep neural networks have been proposed for more robust image matching via high-level feature extraction in the remote sensing domain, but their proposed solutions to address geometric distortion are in an implicit form.

Although the unsupervised I2I algorithm (CycleGAN) can remove the rigorous pixel-level correspondence requirements set out in the supervised I2I algorithm (Pix2Pix), it has been shown that the absence of paired training examples poses a highly under-constrained condition, which may result in undesirable solutions [29,30]. More importantly, for our image translation task in the remote sensing domain, a narrow solution space is desired to minimize the uncertainty in the outputs. Thus, the two-way distortion-adaptive module proposed in this paper aims to enhance the pixel-wise correspondence between the EO and SAR domains by rectifying the geometric distortions via two neural networks. Similar approaches of utilizing extra neural networks for noise elimination during I2I tasks have been designed for medical image analysis [31]. However, unlike those works whose focus is on estimating image misalignment in the target domain only, our method models geometric distortions in both domains with two individual neural networks. The main motivation of the two-way distortion-adaptive module is that geometric distortions are generally much more complicated than misalignment, so domain-specific biases may be captured in the training phase if only images from a single domain are used. Our proposed distortion-adaptive module can better filter out the biases and obtain a reliable domain-agnostic geometric distortion field. More importantly, for our image translation task in the remote sensing domain, a narrow solution space is desired to minimize the uncertainty in the outputs. Thus, the two-way distortion-adaptive module proposed in this paper aims to enhance the pixel-wise correspondence between the EO and SAR domains by rectifying the geometric distortions via two neural networks. Similar approaches of utilizing extra neural networks for noise elimination during I2I tasks have been designed for medical image analysis [31]. However, unlike those works whose focus is on estimating image misalignment in the target domain only, our method models geometric distortions in both domains with two individual neural networks. The main motivation of the two-way distortion-adaptive module is that geometric distortions are generally much more complicated than misalignment, so domain-specific biases may be captured in the training phase if only images from a single domain are used. Our proposed distortion-adaptive module can better filter out the biases and obtain a reliable domain-agnostic geometric distortion field.

2.2. EO-SAR Datasets

Among the existing remote sensing datasets, the vast majority are solely focused on optical images, and few SAR-specific datasets exist, let alone well-organized EO-SAR datasets. Paired EO-SAR datasets for data fusion and image translation tasks have only been introduced in the last few years after the launch of several SAR satellites. In particular, Sentinel-1A [32], which has been operated by the European Space Administration (ESA) since 2014, has provided publicly available land monitoring SAR data at no cost. The SEN1-2 dataset, published in 2018 [33], which utilizes SAR images from Sentinel-1 and EO images from Sentinel-2, has fostered the exploration of deep learning approaches for SAR-EO data analysis. Despite this growing trend of research works on I2I, the development of object-level high-resolution remote sensing applications is still lagging behind the emerging advances in the natural image domain, and the major limitation is the lack of high-quality datasets. For example, the resolution of SAR data from Sentinel-1A is down to 5 m, which is a coarse spatial resolution where only region-level tasks can be performed. Given the high acquisition cost, open-source high-resolution SAR datasets are scarce. One of the most established data sources for higher spatial resolution is the satellite TerraSAR-X [34], launched by a partnership between the German Aerospace Center (DLR) and EADS Astrium, which provides high-resolution SAR data with a GSD of 1 m. We have selected two commonly-used and representative paired EO-SAR datasets, one with low resolution and the other with high resolution, for further illustration and comparison.

2.2.1. SEN1-2

SEN1-2 [33] is a dataset consisting of 282,384 paired EO-SAR image patches of size $256\times 256$. It covers the entire globe and all four seasons. The raw SAR and EO data are collected from Sentinel-1 and Sentinel-2 satellites, respectively. Sentinel-1 is a C-band SAR satellite with a resolution of 5 m, and the SAR images in SEN1-2 are acquired under the interferometric wide swath (IW) mode with a single VV polarization.

2.2.2. SAR2Opt

SAR2Opt [23], published in 2022, is a high-resolution dataset comprising 2076 paired EO-SAR image patches of size $600\times 600$. The coverage of SAR2Opt is around 70 km${}^2$ over multiple cities around the world. The raw SAR and EO data are collected from the TerraSAR-X satellite and Google Earth Engine respectively. TerraSAR-X is an X-band SAR satellite with a resolution of 1 m, and the SAR images in SAR2Opt are acquired under a high-resolution spotlight mode with single polarization.

Sample images from SEN1-2 and SAR2Opt are given in Figure 3 and Figure 4, respectively. It can be observed that low-resolution SAR images in SEN1-2 cover much wider areas than the high-resolution ones in SAR2Opt. Macro-scale tasks, including vegetation monitoring [35,36,37], ocean observation [38,39], and even natural disaster management, such as flood water delineation [40,41,42] and typhoon estimation [43,44], have benefited from the SAR imagery of Sentinel-1. However, more complex tasks [4,5,45], which require discrimination of fine structures such as single buildings in cities and vessels near harbors, are far beyond the scope of datasets with coarse resolution. In the SAR2Opt dataset, more details and features of urban areas are preserved in the SAR images. Although some scattering events still occur in the resolution cell, the phenomenon of over-averaging with the surrounding background is largely alleviated.

3. Construction of the Novel Dataset SN6-SAROPT

The expanded version of the SpaceNet 6 dataset [21] (E-SN6) is exploited to construct a new well-organized EO-SAR benchmark dataset. The raw EO and SAR images in E-SN6 are provided in the following forms:

• A full Maxar WorldView 2 optical image of size 22,800 × 16,202 and spatial resolution of 0.5 m, without cropping;
• A total of 202 overlapping SAR image strips are included in the dataset, each with a size of ∼2800 × 40,000 and a spatial resolution of 0.25 m. Four channels of SAR intensity information, i.e., HH, HV, VH, and VV, and two channels derived from Pauli polarimetric decomposition, i.e., Alpha² and Beta², are contained.

Sample images of the raw EO and SAR data are illustrated in Figure 5. Upon acquisition of the raw data, we conducted the following two steps to generate a well-organized dataset: SAR image processing and EO-SAR Matching.

3.1. SAR Image Processing

As a common phenomenon in SAR images, speckle noise formed by coherent interference of reflected electromagnetic waves degrades visual quality and causes interpretation difficulty. Thus, for the raw SAR image strips in E-SN6, speckle reduction is performed using a Wiener 2D adaptive filter. Magnitude adjustments, including gamma correction and white balancing, are also conducted to enhance the final image quality.

3.2. EO-SAR Matching

The full-size optical image is cropped into patches of size $512\times 512$ without any overlapping regions. Meanwhile, the geo-coordinates of each patch are manually annotated. For the SAR strips, they are cropped into patches of size $1024\times 1024$, which is two times larger than the optical patches due to the higher resolution of SAR. Similar to the optical patches, the geo-coordinates of SAR patches are annotated first and then used for EO-SAR patch co-registration. We end up with 724 EO-SAR pairs for this new dataset. We have further categorized the image pairs into four land types: building, forest, river, and road. The image pairs for the four categories are 402, 113, 141, and 68, respectively. A comparison with existing paired EO-SAR datasets is presented in

Table 1. Comparisons of paired EO-SAR datasets.

Name	Source of SAR	Resolution of SAR	Coverage	Channel
SN6-SAROPT	Capella Space’s X-band quad-pol sensor	0.25 m	Rotterdam (724 pairs, 512 × 512)	4
SAR2Opt [23]	TerraSAR-X	1 m	Asia cities (2076 pairs, 600 × 600)	1
SARptical [46]	TerraSAR-X	1 m	Berlin (Over 10,000 pairs, 112 × 112)	1
QXS-SAROPT [47]	Gaofen-3	1 m	Port cities (20,000 pairs, 256 × 256)	1
SEN1-2 [33]	Sentinel-1/2	down to 5 m	Multiple locations (282,384 pairs, 256 × 256)	1
SEN1-2MS [48]	Sentinel-1/2	down to 5 m	Multiple locations (282,384 pairs, 256 × 256)	2
So2Sat-LCZ42 [49]	Sentinel-1/2	down to 5 m	Multiple locations (400,673 pairs, 32 × 32)	2

, and sample image patches are given in Figure 6. Even though SN6-SAROPT has no significant advantage over others in terms of image patch numbers, it is worth noting that this is mainly due to the fact that overlap is avoided in the construction. For the current high-resolution datasets, we have noticed that overlap has inflated the scale of datasets, and the overlapping region in a single patch is 60% for SAR2Opt, more than 50% for SARptical, and 20% for QXS-SAROPT. More importantly, the resolution and polarization channels of SN6-SAROPT, which are intrinsic properties that determine the quality and quantity of carried information, have an undoubted ascendancy over the existing ones. Especially for I2I tasks in the remote sensing domain, additional spatial-wise and channel-wise information contained in SN6-SAROPT can explore the full potential of SAR imagery.

4. Methodology

Given two sets of paired EO and SAR images, denoted as domain X and domain Y, respectively, the objective is to learn a mapping function G for the E2S task, such that for any previously unseen EO image, the generated fake SAR image is as similar as possible to real SAR images, and to learn a mapping function F for the S2E task, such that for any previously unseen SAR image, the generated fake EO image is as similar as possible to real EO images.

4.1. Two-Way Distortion-Adaptive Module

To better eliminate noise arising from geometric distortions, a two-way distortion-adaptive module is proposed. As illustrated in Figure 7, the overall training framework takes advantage of both Pix2Pix and CycleGAN, incorporating an additional network called the distortion-adaptive (DA) module in both directions. Each DA module aims to learn the geometric changes between the two domains. Specifically, DA-SAR models the geometric changes from EO to SAR and vice versa for DA-EO. Since both image style and geometric structure contribute to the final visual appearance, learning geometric changes can be difficult if image styles are changing simultaneously. To address this, image translation is performed first and only images from the same domain are fed into each DA module. DA-SAR takes the fake SAR image $G\left(x\right)$ and the real SAR image y as inputs and produces a distortion field ${\varphi }^{SAR}$. The distortion field ${\varphi }^{SAR}$ is then used to perform a resampling process on fake SAR image $G\left(x\right)$ to obtain $\mathcal{R}(G\left(x\right),{\varphi }^{SAR})$. DA-EO takes the fake EO image $F\left(y\right)$ and real EO image x as inputs and produces a de-distortion field ${\varphi }^{EO}$. The de-distortion field ${\varphi }^{EO}$ is then used to perform a resampling process on the fake EO image $F\left(y\right)$ to obtain $\mathcal{R}(F\left(y\right),{\varphi }^{EO})$. The objective of the two-way distortion-adaptive module is to enhance pixel-level correspondence between the image pairs by explicitly modeling geometric distortions between SAR and EO domains. Therefore, we expect that the resampling operations with the guidance from the two fields will produce outputs similar to the real images, i.e., $\mathcal{R}(G\left(x\right),{\varphi }^{SAR})\approx y$ and $\mathcal{R}(F\left(y\right),{\varphi }^{EO})\approx x$. Both the distortion field ${\varphi }^{SAR}$ and de-distortion field ${\varphi }^{EO}$ have the size of $H\times W\times 2$ for image patches $x,y$ of size $H\times W$, which specifies the shifts of each pixel in horizontal and vertical directions. The image sampling strategy in the Spatial Transformer [50] is adopted in our method to achieve a differentiable resampling mechanism. The general 2D image resampling operation can be defined as follows:

$$V_i^c=\sum _{n=1}^H\sum _{m=1}^W{U}_{n,m}^{c}k(p-n;{\Phi }_h)k(q-m;{\Phi }_v)\forall i\in [1\dots HW]\forall c\in [1\dots C]$$

(4)

where $V_i^c$ denotes the value of pixel i at coordinates $(p,q)$ of the output image in channel c, $U_{n,m}^c$ denotes the pixel value at coordinates $(n,m)$ of the input image in channel c, and $\{{\Phi }_h,{\Phi }_v\}$ are the parameters of interpolation function $k(·)$ in horizontal and vertical axes, respectively. With ${\varphi }^{SAR}$ and ${\varphi }^{EO}$, the resampling results can be expressed as follows:

$$\begin{array}{cc}\hfill & {\mathcal{R}}_i^c(G\left(x\right),{\varphi }^{SAR})={\sum }_{n=1}^H{\sum }_{m=1}^W{G\left(x\right)}_{n,m}^c\delta ({\varphi }_i^{SAR}\left[0\right]+p-n)\delta ({\varphi }_i^{SAR}\left[1\right]+q-m)\\ \hfill & \forall i\in [1\dots HW]\forall c\in [1\dots C]\end{array}$$

(5)

$$\begin{array}{cc}\hfill & {\mathcal{R}}_i^c(F\left(y\right),{\varphi }^{EO})={\sum }_{n=1}^H{\sum }_{m=1}^W{F\left(y\right)}_{n,m}^c\delta ({\varphi }_i^{EO}\left[0\right]+p-n)\delta ({\varphi }_i^{EO}\left[1\right]+q-m)\\ \hfill & \forall i\in [1\dots HW]\forall c\in [1\dots C]\end{array}$$

(6)

where $\delta (·)$ is the Kronecker delta function, ${\varphi }_i^{SAR}\left[0\right]$ and ${\varphi }_i^{SAR}\left[1\right]$ denote the values of pixel i at coordinates $(p,q)$ of the distortion field ${\varphi }^{SAR}$ and similar notations apply to ${\varphi }^{EO}$. Equations (5) and (6) reduce to ${\mathcal{R}}_i^c(G\left(x\right),{\varphi }^{SAR})={G\left(x\right)}_{n,m}^c$ with $\{n={\varphi }_i^{SAR}\left[0\right]+p,$$m={\varphi }_i^{SAR}\left[1\right]+q\}$ and ${\mathcal{R}}_i^c(F\left(y\right),{\varphi }^{EO})={F\left(y\right)}_{n,m}^c$ with $\{n={\varphi }_i^{EO}\left[0\right]+p,m={\varphi }_i^{EO}\left[1\right]+q\}$, respectively. As the coordinates of images should be integers, bi-linear interpolation is used because of the float data type of the two fields. Therefore, the final results are:

$$\begin{array}{cc}\hfill & {\mathcal{R}}_i^c(G\left(x\right),{\varphi }^{SAR})={\sum }_{r\in \mathcal{N}}(1-|r\left[0\right]-n\left|\right)(1-|r\left[1\right]-m\left|\right){G\left(x\right)}_r^c\forall i\in [1\dots HW]\forall c\in [1\dots C]\\ \hfill & n={\varphi }_i^{SAR}\left[0\right]+p,m={\varphi }_i^{SAR}\left[1\right]+q\end{array}$$

(7)

$$\begin{array}{cc}\hfill & {\mathcal{R}}_i^c(F\left(y\right),{\varphi }^{EO})={\sum }_{r\in \mathcal{N}}(1-|r\left[0\right]-n\left|\right)(1-|r\left[1\right]-m\left|\right){F\left(y\right)}_r^c\forall i\in [1\dots HW]\forall c\in [1\dots C]\\ \hfill & n={\varphi }_i^{EO}\left[0\right]+p,m={\varphi }_i^{EO}\left[1\right]+q\end{array}$$

(8)

where $\mathcal{N}$ represents the set of locations of four neighbors of coordinates $(n,m)$, i.e., $\mathcal{N}=\left\{\right(\lceil n\rceil ,\lceil m\rceil ),(\lfloor n\rfloor ,\lfloor m\rfloor ),(\lceil n\rceil ,\lfloor m\rfloor ),(\lfloor n\rfloor ,\lceil m\rceil \left)\right\}$.

4.2. Overall Training Scheme

With the two-way distortion-adaptive module, it is possible to achieve the pixel-level reconstruction of real images. Since the two generators G and F have already performed the task of style/texture transfer, and the resampling operations have rectified the geometric distortions, we can safely minimize the $L_1$ distance between the synthetic and real images:

$${\mathcal{L}}_{L_1}(G,F,{\varphi }^{SAR},{\varphi }^{EO})={\mathbb{E}}_{x\sim X,y\sim Y}\left[\right||\mathcal{R}(G\left(x\right),{\varphi }^{SAR})-{y\left|\right|}_1+\left|\right|\mathcal{R}(F\left(y\right),{\varphi }^{EO})-{x\left|\right|}_1]$$

(9)

The two fields ${\varphi }^{SAR}$ and ${\varphi }^{EO}$ can be highly uneven in order to achieve nearly-zero distance in Equation (9), which may give trivial solutions of over-fitted DA modules and under-fitted generators. Hence, a gradient loss is imposed as a regularization constraint:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{smooth}(G,F,{\varphi }^{SAR},{\varphi }^{EO})={\mathbb{E}}_{x\sim X,y\sim Y}\left[\right||\nabla {\varphi }^{SAR}{\left|\right|}^2+\left|\right|\nabla {\varphi }^{EO}{\left|\right|}^2]\\ \hfill & \nabla {\varphi }^{SAR}=({\varphi }^{SAR}[:,1:,:]-{\varphi }^{SAR}[:,:-1,:],{\varphi }^{SAR}[1:,:,:]-{\varphi }^{SAR}[:-1,:,:])\\ \hfill & \nabla {\varphi }^{EO}=({\varphi }^{EO}[:,1:,:]-{\varphi }^{EO}[:,:-1,:],{\varphi }^{EO}[1:,:,:]-{\varphi }^{EO}[:-1,:,:])\end{array}$$

(10)

In addition, cycle-consistency loss is also incorporated into our methods for a more controllable solution space. To this end, we have the following full training objective:

$$\begin{array}{cc}\hfill & arg{min}_{G,F,{\varphi }^{SAR},{\varphi }^{EO}}{max}_{D_X,D_Y}{\mathcal{L}}_{GAN}(G,D_Y)+{\mathcal{L}}_{GAN}(F,D_X)+{\lambda }_{L_{cyc}}{\mathcal{L}}_{cyc}(G,F)\\ \hfill & +{\lambda }_{L_1}{\mathcal{L}}_{L_1}(G,F)+{\lambda }_{L_{smooth}}{\mathcal{L}}_{smooth}(G,F,{\varphi }^{SAR},{\varphi }^{EO})\end{array}$$

(11)

where ${\mathcal{L}}_{cyc}(G,F)$ is defined in Equation (3) and ${\lambda }_{L_{cyc}},{\lambda }_{L_1},{\lambda }_{L_{smooth}}$ are weights of the corresponding loss terms, respectively. Similar to the two discriminators, $D_Y$ and $D_X$, the two DA modules will not be used during the inference phase, because the ground-truth images in the target domain are not available.

5. Experiments

To provide more comprehensive experimental results, the performances on both E2S and S2E translation tasks are assessed. According to the empirical results presented in previous work [23], we have selected the three top-performing GAN-based I2I algorithms for the S2E task on both high-resolution and low-resolution EO-SAR datasets. These algorithms are CycleGAN [18], Pix2Pix [17], and NICEGAN [51], and they will be compared to our proposed method. Since NICEGAN is a holistic approach that has incorporated multiple components, e.g., multi-scale discriminator, residual attention mechanism, and cycle consistency, two variants of NICEGAN are tested: NICEGAN with and without cycle consistency loss, denoted as NICEGAN(C) and NICEGAN(NC), respectively. Three benchmark datasets, i.e., SEN1-2, SAR2Opt, and SN6-SAROPT, are evaluated. For SEN1-2, we randomly selected 500 images from each sub-dataset (spring, summer, fall, and winter), and then keep $10\%$ of the images for testing and the remaining $90\%$ for training. For SAR2Opt, the original train/test split is adopted. Similar to SEN1-2, we split the new dataset (SN6-SAROPT) by randomly selecting $10\%$ for testing and the remaining $90\%$ for training.

5.1. Experiment Setup

All experiments were conducted on a server running Ubuntu 18.04 with 4 NVIDIA GeForce RTX 2080 Ti GPUs, 128 GiB RAM, and an Intel Xeon CPU E5-1660. The programming environment used was Python 3.6.13 and PyTorch 1.8.1. The Adam optimizer with a learning rate of $1\times {10}^{-4}$ was employed, and the total number of training epochs was set to 80. The network architecture used for generators G and F mainly consisted of two downsampling convolutional layers, nine residual blocks, and two upsampling convolutional layers. For the discriminator $D_X$ and $D_Y$, a five-layer convolutional neural network is used. Similar to the previous works [31,52], U-Net architecture is used for the two DA modules. The hyperparameters in Equation (11) are set as follows: ${\lambda }_{L_{cyc}}=10$, ${\lambda }_{L_1}=20$ and ${\lambda }_{L_{smooth}}=10$ for SAR2Opt and SN6-SAROPT; ${\lambda }_{L_{cyc}}=10$, ${\lambda }_{L_1}=0.5$ and ${\lambda }_{L_{smooth}}=0.5$ for SEN1-2. The model collapse is observed when ${\lambda }_{L_1}$ and ${\lambda }_{L_{smooth}}$ take large values for SEN1-2, so they are adjusted to lower levels.

5.2. Evaluation Metrics

Four evaluations metrics are used to assess the quality of generated images: PSNR, SSIM [53], FID [54], and LPIPS [55]. Note that all metrics used in this work belong to full-reference image quality assessment (FR-IQA), where reference images are required. No-reference Image Quality Assessment (NR-IQA) is not considered, due to the fact that most NR-IQA scores are solely dependent on the images to be assessed, and pre-established models are based on natural images, which have significant differences from remote sensing imagery.

• Peak signal-to-noise ratio (PSNR): Based on the average $L_2$ distance between two images, x and y of size $C\times H\times W$, also known as the mean-squared error (MSE). PSNR is defined as:

  $$PSN{R}_{(x,y)}=10{log}_{10}\left(\frac{1}{MSE}\right),MSE=\frac{1}{HWC}\sum _{i=1}^{HWC}{(x_i-y_i)}^2$$

  (12)
• Structural SIMilarity (SSIM): Image similarity is measured by comparing the contrast, luminance, and structural information:

  $$SSI{M}_{(x,y)}=\frac{(2{\mu }_x{\mu }_y+C_1)(2{\sigma }_{xy}+C_2)}{({\mu }_x^2+{\mu }_y^2+C_1)({\sigma }_x^2+{\sigma }_y^2+C_2)}$$

  (13)

  where ${\mu }_{x/y}$, ${\sigma }_{x/y}$, and ${\sigma }_{xy}$ denote the mean, standard deviation, and covariance of the corresponding images respectively, and $C_1$ and $C_2$ are two constants.
• Fréchet Inception Distance (FID): FID is defined as the Wasserstein-2/Fréchet distance between the feature embeddings of images from the last pooling layer of the Inception-V3 model pre-trained on ImageNet.

  $$FI{D}_{(x,y)}=\left|\right|m_x-m_y{\left|\right|}_2^2+\mathrm{Tr}(C_x+C_y-2{\left(C_x{C}_y\right)}^{0.5})$$

  (14)

  where $m_{x/y}$ and $C_{x/y}$ denote the mean and covariance of the corresponding feature embeddings, respectively.
• Learned perceptual image patch similarity (LPIPS):

  $$LPIP{S}_{(x,y)}=\sum _l\frac{1}{H_l{W}_l}\sum _{h,w}\left|\right|w_l(s_x^l-s_y^l){\left|\right|}_2^2$$

  (15)

  where $s_{x/y}^l$ and $w_l$ denote the corresponding normalized feature embeddings and scaling coefficient of layer l.

5.3. Quantitative Evaluations

The quantitative results are presented in

Table 2. Comparisons of IQA scores on SAR2Opt for S2E (SAR to EO) and E2S (EO to SAR) tasks. The best and second best results are highlighted as red bold and blue italic, respectively.

Methods	S2E				E2S
	PSNR↑	SSIM↑	FID↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	LPIPS↓
Ours	15.72	0.240	178.76	0.491	15.22	0.203	116.29	0.380
CycleGAN [18]	13.23	0.072	164.29	0.581	12.13	0.058	188.01	0.567
Pix2Pix [17]	14.60	0.110	237.09	0.531	13.78	0.111	168.95	0.380
NICEGAN(NC) [51]	11.66	0.026	308.59	0.674	11.70	0.069	366.85	0.582
NICEGAN(C) [51]	13.70	0.189	186.63	0.556	12.12	0.101	151.89	0.437

,

Table 3. Comparisons of IQA scores on SN6-SAROPT for S2E (SAR to EO) and E2S (EO to SAR) tasks. The best and second best results are highlighted as red bold and blue italic, respectively.

Methods	S2E				E2S
	PSNR↑	SSIM↑	FID↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	LPIPS↓
Ours	15.14	0.188	255.22	0.461	18.10	0.262	130.81	0.377
CycleGAN [18]	13.49	0.146	200.47	0.485	16.78	0.105	137.58	0.416
Pix2Pix [17]	14.71	0.143	355.48	0.525	17.73	0.114	277.15	0.397
NICEGAN(NC) [51]	11.63	0.157	411.64	0.715	11.11	0.079	448.96	0.723
NICEGAN(C) [51]	13.75	0.143	323.57	0.522	17.14	0.217	159.77	0.418

and

Table 4. Comparisons of IQA scores on SEN1-2 for S2E (SAR to EO) and E2S (EO to SAR) tasks. The best and second best results are highlighted as red bold and blue italic, respectively.

Methods	S2E				E2S
	PSNR↑	SSIM↑	FID↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	LPIPS↓
Ours	10.92	0.027	219.41	0.615	12.87	0.016	163.29	0.459
CycleGAN [18]	11.41	0.005	177.85	0.610	10.14	0.035	168.18	0.450
Pix2Pix [17]	11.53	0.061	241.02	0.621	11.82	0.014	141.73	0.388
NICEGAN(NC) [51]	10.23	0.059	336.32	0.749	11.37	0.001	169.08	0.449
NICEGAN(C) [51]	10.46	0.051	199.26	0.617	10.32	0.005	116.92	0.430

for the three datasets. Our method has shown significant improvements on both S2E and E2S tasks for high-resolution datasets SAR2Opt and SN6-SAROPT. In Table 2 and Table 3, seven out of the eight best scores of evaluation metrics are achieved by our method. Although the FID scores of our method on the S2E task are slightly worse than those of CycleGAN, we consider this as a result of the evaluation bias of FID, which only addresses high-level feature similarity and ignores low-level reconstruction quality. Moreover, we observed that the scores on the E2S task are generally better than those on the S2E task, indicating an imbalance in the difficulties of the two transfer tasks. We believe this is due to the fact that EO images contain more information compared to SAR images, which makes the E2S task easier than the S2E task under the same conditions. More visual samples from SAR2Opt and SN6-SAROPT will be presented in the following section. For SEN1-2, our method performs comparably to others, as shown in Table 4. However, the advantages of our proposed DA module are not fully realized on low-resolution EO-SAR datasets. Nonlinear distortions due to different imaging characteristics of optical and radar sensors are less severe in low-resolution SAR images, limiting potential improvements. Additionally, evaluation results for each land type in SN6-SAROPT are provided in

Table 5. Comparisons of IQA scores per category of SN6-SAROPT for S2E (SAR to EO) and E2S (EO to SAR) tasks. The best and second best results are highlighted as red bold and blue italic, respectively.

Land Type	Methods	S2E				E2S
		PSNR↑	SSIM↑	FID↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	LPIPS↓
Building	Ours	14.31	0.108	283.07	0.469	17.90	0.238	139.60	0.401
	CycleGAN [18]	12.36	0.053	205.02	0.492	16.44	0.096	146.15	0.447
	Pix2Pix [17]	13.63	0.063	374.92	0.515	17.43	0.105	330.29	0.431
	NICEGAN(NC) [51]	11.11	0.079	448.96	0.723	17.52	0.129	482.44	0.724
	NICEGAN(C) [51]	12.94	0.057	377.93	0.529	16.75	0.188	177.67	0.458
Forest	Ours	16.75	0.163	337.81	0.500	17.90	0.260	222.16	0.357
	CycleGAN [18]	15.06	0.150	328.70	0.557	16.57	0.087	210.20	0.370
	Pix2Pix [17]	16.85	0.159	388.83	0.564	17.50	0.107	226.37	0.353
	NICEGAN(NC) [51]	11.42	0.151	494.87	0.759	16.77	0.133	431.06	0.751
	NICEGAN(C) [51]	13.87	0.106	371.68	0.607	17.16	0.227	224.88	0.360
River	Ours	16.28	0.446	307.21	0.393	18.75	0.320	214.14	0.324
	CycleGAN [18]	15.49	0.418	296.19	0.392	17.71	0.135	239.29	0.369
	Pix2Pix [17]	16.15	0.364	353.59	0.506	18.63	0.137	308.49	0.337
	NICEGAN(NC) [51]	13.26	0.390	389.40	0.639	18.13	0.145	432.20	0.708
	NICEGAN(C) [51]	15.86	0.423	365.65	0.421	18.38	0.287	246.56	0.354
Road	Ours	14.79	0.149	439.59	0.500	18.27	0.282	217.43	0.381
	CycleGAN [18]	13.15	0.098	353.40	0.521	17.18	0.127	257.67	0.418
	Pix2Pix [17]	14.30	0.104	488.10	0.553	17.94	0.127	396.68	0.395
	NICEGAN(NC) [51]	11.48	0.119	525.64	0.759	17.99	0.146	481.53	0.736
	NICEGAN(C) [51]	13.81	0.110	464.75	0.552	16.73	0.218	288.81	0.428

. We have observed correlations between the effectiveness of our method and the urbanization level of the region. For building and road areas, our method outperforms others in seven out of eight scores. While for forest and river areas, the performances are still generally superior, but slightly degraded compared to more urbanized areas. Since human-made objects in high-resolution SAR images are more vulnerable to noises arising from geometric distortions than natural objects, the empirical results have successfully validated our assumption that the proposed DA module is capable of addressing such an issue.

The comparison of model complexity is presented in

Table 6. Comparisons of model complexity.

Methods	Number of Trainable Parameters (M)				Test Time
	Generator	Discriminator	DA Modules	Total	(ms/per image)
Ours	22.76	5.51	4.12	32.40	41.76
CycleGAN [18]	22.76	5.51	-	28.29	35.55
Pix2Pix [17]	54.41	2.77	-	57.18	8.82
NICEGAN(NC) [51]	9.45	11.72	-	21.17	62.23
NICEGAN(C) [51]	18.57	93.75	-	112.32	74.14

, where the number of trainable parameters and test time are compared among the algorithms. The proposed algorithm has a comparable number of parameters to CycleGAN and is less complex than Pix2Pix and NICEGAN(C). The additional computational cost brought by the proposed DA modules is not significant. For the test time, Pix2Pix achieved significantly faster inference than others.

5.4. Qualitative Evaluations

Visual samples generated by different models are shown in Figure 8, Figure 9, Figure 10 and Figure 11. Because the model collapse is observed for NICEGAN(NC), the visual results of NICEGAN(NC) are omitted. SN6-SAROPT: In Figure 8, both global and local structures of the EO images have been correctly reconstructed by our method. Significant structure deformations are observed in images generated by NICEGAN(C). Pix2Pix fails to preserve the global structure information and, thus, ends up with meaningless outputs. CycleGAN performed better than Pix2Pix and NICEGAN(C), as most of the spatial structures are correctly captured. However, many wrong interpretations are made by CycleGAN, e.g., the white storage tanks in the real EO images are incorrectly reconstructed for the first two samples. In Figure 9, similar observations can be made, i.e., NICEGAN(C) fails to generate realistic images; Pix2Pix totally ignores the overall structure information; and CycleGAN is prone to generating wrong details. For example, the upper right building blocks in the real SAR images of the first two samples were only correctly produced by our method. SAR2Opt: For the first sample in Figure 10, NICEGAN(C) generates checkerboard artifacts; CycleGAN fails to preserve the detailed structures; and the roads produced by Pix2Pix are not as clear as those reconstructed by our method. For the other two samples, Pix2Pix has only learned partial texture information. NICEGAN(C) and CycleGAN have generated incorrect items, such as blue buildings in the second sample and vegetation in the third sample. In Figure 11, the translation results of E2S are particularly distinct among the four methods. Specifically, Pix2Pix introduces severe noise and produces blurry boundaries for the installations. CycleGAN and NICEGAN(C), on the other hand, only adjust the color tone of the EO images without making significant changes to other content. In contrast, our method has accurately captured the correlations between the two modalities regarding both texture and contour information. To summarize, we can make the following observations: (1) Despite cycle consistency being effective in preventing model collapse, NICEGAN(C) is still prone to significant artifacts. (2) Pix2Pix suffers from a loss of structural information and only learns texture correlations. (3) The outputs from CycleGAN may seem reasonable at first glance without comparing them to ground-truth images, but most of the generated local details deviate from the real ones upon closer inspection. (4) The proposed DA module provides extra supervision signals via the rectification of geometric distortions, such that not only the image style in terms of color and texture is learned, but also local reconstruction quality is substantially enhanced.

5.5. Ablation Study

To validate the effectiveness of each module in the proposed method, an ablation study was conducted to further analyze the contribution of each. The comparisons of IQA scores from our method and its three variants are given in

Table 7. Ablation study on SAR2Opt: IQA scores of our proposed method and its three variants. The best and second best results are highlighted as red bold and blue italic, respectively.

Variants	S2E				E2S
	PSNR↑	SSIM↑	FID↓	LPIPS↓	PSNR↑	SSIM↑	FID↓	LPIPS↓
Ours w/o DA modules	13.07	0.078	185.69	0.572	12.72	0.018	274.66	0.940
Ours w/o DA-SAR	15.26	0.220	161.95	0.483	11.39	0.084	117.24	0.516
Ours w/o DA-EO	13.60	0.097	189.56	0.579	15.33	0.199	143.14	0.416
Ours	15.72	0.240	178.76	0.491	15.22	0.203	116.29	0.380

. Referring back to the overall training framework in Figure 7, DA-SAR is designed to map the distortion field ${\varphi }^{SAR}$ and is jointly trained with generator G, which translates images from the EO domain to the SAR domain, i.e., the gradients of loss ${\mathcal{L}}_{L_1}$ are back-propagated through both of them. Therefore, DA-SAR is expected to have a great impact on the performance of the proposed method on the E2S task. Similar reasoning applies to DA-EO, and we expect that the performances on the S2E task are largely dependent on the learning of the de-distortion field ${\varphi }^{EO}$. The results in Table 7 have validated the above analyses. Considerable performance degradation happens when DA-SAR is removed on the E2S task or DA-EO is removed on the S2E task. Moreover, we observed that the performances of our method on the E2S task are not only determined by DA-SAR but also enhanced by DA-EO. By just removing DA-EO, three out of the four evaluation metrics have worsened on the E2S task. While for the S2E task, both PSNR and SSIM scores degraded after removing DA-SAR. Visual samples are provided in Figure 12 and Figure 13 for the S2E and E2S tasks, respectively. By comparing the generated images with the ground-truth, the importance of each DA module is demonstrated. In Figure 12, the generated EO images show the wrong land type after removing DA-EO, i.e., buildings in the first sample and bare grounds in the latter two samples all change to grasslands. After removing DA-SAR, several buildings in the first sample disappear, and the upper-left region of the second sample changes to grassland. In Figure 13, the generated SAR images treat shadows/backgrounds as targets after removing DA-SAR. Moreover, the boundaries of buildings in the generated SAR images become blurry after removing DA-EO. Therefore, the synergistic effects of the proposed two-way distortion-adaptive module are confirmed, i.e., each single DA network is indispensable.

6. Conclusions

In this paper, we address two limitations in the current GAN-based I2I studies for EO-SAR mapping tasks: (1) The lack of high-resolution datasets gives a false sense of success that existing methods have solved the problem, and (2) the non-linear distortions between EO and SAR images arising from different sensors are ignored. We introduce a new dataset, SN6-SAROPT, which provides EO-SAR image patches with a resolution of 0.25 m, the highest among all publicly available datasets. We also design a novel algorithm that incorporates a two-way distortion-adaptive module into translation networks. We conduct extensive experiments on three datasets with resolutions ranging from 5 m to 0.25 m for both E2S and S2E tasks. The empirical results demonstrate the superiority of our method over others, especially for high-resolution datasets where more regions have severe geometric distortion issues.

For future work, we plan to incorporate high-level tasks into the learning of EO-SAR mapping. Firstly, high-level tasks, including (but not limited to) object detection, classification, and anomaly detection, can provide more comprehensive evaluations of the transfer model. As one of the main motivations of EO-SAR mapping is data augmentation via synthetic image generation, the performance of the detector/classifier trained with synthetic data is a good indicator of the efficacy of the deployed I2I translation network. Secondly, current I2I translation algorithms for remote sensing applications are mainly focused on general solutions, whereas task-specific image mapping will benefit downstream applications by selectively emphasizing task-discriminative information.

The defense against the misuse of generated images for remote sensing applications is also worth further investigation. In the natural image domain, widely accessible online image-generation platforms have already raised ethical and legal concerns. While remote sensing imagery is not as common as natural images, the trustworthiness of data is becoming more important than ever due to its extensive use in security-critical scenarios.