Unsupervised Remote Sensing Image Super-Resolution Guided by Visible Images

Abstract

Remote sensing images are widely used in many applications. However, due to being limited by the sensors, it is difficult to obtain high-resolution (HR) images from remote sensing images. In this paper, we propose a novel unsupervised cross-domain super-resolution method devoted to reconstructing a low-resolution (LR) remote sensing image guided by an unpaired HR visible natural image. Therefore, an unsupervised visible image-guided remote sensing image super-resolution network (UVRSR) is built. The network is divided into two learnable branches: a visible image-guided branch (VIG) and a remote sensing image-guided branch (RIG). As HR visible images can provide rich textures and sufficient high-frequency information, the purpose of VIG is to treat them as targets and make full use of their advantages in reconstruction. Specially, we first use a CycleGAN to drag the LR visible natural images to the remote sensing domain; then, we apply an SR network to upscale these simulated remote sensing domain LR images. However, the domain gap between SR remote sensing images and HR visible targets is massive. To enforce domain consistency, we propose a novel domain-ruled discriminator in the reconstruction. Furthermore, inspired by the zero-shot super-resolution network (ZSSR) to explore the internal information of remote sensing images, we add a remote sensing domain inner study to train the SR network in RIG. Sufficient experimental works show UVRSR can achieve superior results with state-of-the-art unpaired and remote sensing SR methods on several challenging remote sensing image datasets.

1. Introduction

Remote sensing images are currently being used in many applications, such as weather alerts [1], land cover classification [2,3], and target detection [4,5,6]. In fact, high-resolution (HR) remote sensing images are required in these applications for precise prediction. While visible images have rich textures and detailed content in their HR images, the visual resolution of most remote sensing images is poor and limited by various sensors in satellites due to their high altitude. Moreover, monsoons, clouds, and detector blur and noise can also degrade the quality of remote sensing images. Consequently, it is difficult to obtain high-quality satellite images, and this undoubtedly will constrain their further applications. One cost-efficient technique to increase the spatial resolution of remotely sensed data is image super-resolution (SR).

Image SR is an important issue in image processing tasks. It can be defined as a key preprocessing stage for image identification, image classification, and image segmentation, etc. The purpose of SR is to scale up a single degraded low-resolution (LR) image to a corresponding HR image in software without having to update the hardware, thereby keeping hardware costs in hand, especially considering the scaling factor up to $8\times$ (8 times). Nonetheless, SR incurs two challenges: the high number of potential super-resolution results from the corresponding LR, and the difficulty in defining a model to produce an optimal SR result. With the evolution of deep learning methods and models, SR research is rapidly improving. Deep learning SR models have demonstrated the capability to identify likely HR reconstructions by capturing detailed prior information about images. These methods have achieved remarkable performance by leveraging convolutional neural networks (CNNs) [7,8,9,10,11,12,13] and generative adversarial networks (GANs) [14,15,16,17,18].

All existing SR models can be classified into two kinds of techniques, supervised or unsupervised SR, depending on the training data. Most models are trained in a supervised manner, in that indispensable HR/LR image pairs should be provided in the model training. Essential HR image datasets including visible clean datasets, such as DIV2K [19], SET5 [20], URBAN [21], and B100 [22], are adopted to generate paired HR/LR training data by downsampling the HR images. Regarding the unsupervised category, these SR approaches estimate the HR information in the result with the LR input image itself. This is because in most practical applications, including remote sensing images, infrared images, real-world visible images, and HR datasets are always unavailable. Recently, some work [23,24,25,26] has adopted unpaired training in unsupervised SR. Specifically, they built the models to enhance the resolution of LR data by other HR image datasets and trained the model with these unpaired HR/LR counterparts.

Real-world visible image SR [25,26,27,28,29,30,31] deals with unpaired training. Real-world images are captured by various visible sensors such as cameras, smartphones, and surveillance cameras, but it can be arduous to obtain HR/LR pairs due to the shaking of sensors or changing backgrounds. Recently, CycleGAN [32] was proposed to translate an image from a source domain to a target domain in the absence of training paired examples. Some recent work applied CycleGAN to real-world SR. Bulat et al. [26] first applied CycleGAN to real-world facial SR. Specifically, guided by an unpaired LR real-world face, they exploited CycleGAN to transfer a LR clean facial image to the real-world domain. Then, a stacked SR network after CycleGAN scaled up the resolution of this LR simulated real-world face to the expected scale. In their pipeline, they trained CycleGAN and the SR network together with several loss functions. Yuan et al. [31] improved Bulat’s model and used a cycle-in-cycle network in natural real-world image SR. Their work used two CycleGANs, one to learn real-world degradation similar to Butal et al. [26], and the next to perform the SR task. To maintain the content consistency, an additional CycleGAN was used to ensure that the final result can go back to the initial input. However, the recent work on the zero-shot super-resolution network (ZSSR) [33] offered another way of performing unsupervised SR. They first trained paired HR/LR synthetic image pairs in a single SR network and downsampled the LR real image to an expected scale. Then, they applied these downsampled LR real image/LR real image pairs to continue to train the SR network in a test time, which can also be regarded as online training. This training strategy can be used to explore the inner relationship of real-world images. In this paper, we take full advantage of both CycleGAN and ZSSR in our unsupervised remote sensing SR model.

In remote sensing scenarios, to the best of our knowledge, there is very limited work on the training setting of unpaired HR/LR images. Most remote sensing SR methods [34,35,36,37,38] simply treat the initial remote sensing image from the dataset and its downsampled counterpart as HR/LR. However, there are two key problems in this setting. Because the resolution of the initial remote sensing image is low and lacks details, it is improper to treat it as the target. In addition, the degradation between HR/LR remote sensing is unknown and more complicated than when it is simply downsampled. To deal with an unpaired setting, Haut et al. [39] proposed an unsupervised learning model for generating remote sensing images from unpaired random noise and improved the resolution of remote sensing imagery. Later, Wang et al. [40] improved this noise generation strategy. In their implementation, they encoded the reference image into the latent space as the migration image prior and updated noise maps to obtain a more stable and detailed reconstruction. However, noise itself has meaningful high-frequency information; this constrains the further improvement of remote sensing images.

Rather than random noise, visible HR clean images have more meaningful textures and can provide sufficient high-frequent information. In this paper, we use HR clean images as a target to guide remote sensing SR. Nonetheless, the domain distance between the visible clean images and remote sensing images is much greater than that between the visible clean images and the real-world images, and the direct use of the models in real-world SR is more likely to cause domain shift in the reconstruction. How to explore the ability of an HR clean target in the reconstruction without domain shift is a significant challenge in remote sensing SR.

To resolve this problem, we proposed a novel unsupervised GAN-based and cross-domain super-resolution network for remote sensing images, which we call UVRSR. The network is divided into two learnable branches: the visible image guide branch (VIG) and the remote sensing image guide branch (RIG). The design of the VIG is to explore the high frequency information and textures in HR clean images for the network. To avoid domain shift in the reconstruction, we design a novel domain-ruled(DR) discriminator to ensure domain consistency. The DR discriminator learns to determine the remote sensing domain by remote sensing the target, and it learns to determine the content by the HR visible target. Moreover, it enhances the target domain to avoid domain shift in the reconstruction. In addition, to learn the remote sensing inner relationship, we add a RIG to the SR network.

In summary, our contributions are:

• To the best of our knowledge, this is the first work to perform cross-domain SR in the absence of HR/LR image pairs, as well as the first to apply visible images to assist in remote sensing domain image SR;
• This paper proposes a novel two-branch network, UVRSR, to produce an SR remote sensing domain image with an HR visible image and an unpaired LR remote sensing image. A CycleGAN-based learnable branch VIG is proposed to dig the rich textures from HR visible images, and another learnable branch, RIG, is built to explore the internal information of remote sensing images;
• We design a novel domain-ruled (DR) discriminator to determine the SR output to the target remote sensing domain without domain shift in the reconstruction;
• Experiments show that UVRSR can achieve superior results when compared with state-of-the-art unpaired and remote sensing SR methods on the remote sensing UC Merced and the NWPU-RESISC45 datasets.

The paper is organized as follows. Section 2 investigates work related to our proposed method. Section 3 describes the theory, structure and training of our proposed UVRSR model. Section 4 illustrates our experimental work and analysis in multiple remote sensing datasets. Section 5 further discusses the effects of our proposed model. Section 6 contains our conclusions and future work indications.

2. Related Work

2.1. Remote Sensing Super-Resolution

The first work on remote sensing SR was proposed by Tsai et al. [41]. These authors applied frequency-domain to upscale LR images by multiple downsampled images. Then, with the proposal of various image processing techniques, including domain wavelet transformation (DWT), maximum a posterior (MAP), total variation (TV), iterative back-projection (IBP) and projection onto convex sets (POCS), in the following years, remote sensing SR rapidly developed. Tao et al. [42] first applied the DWT method in remote sensing SR. The SR result is obtained by the inverse discrete wavelet transform. Wang et al. [43] proposed a MAP method to iteratively produce SR remote sensing images and keep spectral characters of multi-spectral images. Li et al. [44] embedded the improved elastic registration method in a modified IBP method to reconstruct better detailed remote sensing images. Yuan et al. [45] proposed a regional spatially adaptive total variation model, which reduced some pseudo-edges. Recently, deep learning networks have boosted the performance of remote sensing SR. Feng et al. [46] proposed a denoising and residual dense-based SR network in a unified framework to acquire better reconstruction quality. Liu et al. [47] proposed a saliency-guided image super-resolution model to strengthen the salient object in the remote sensing images. Jiang et al. [37] proposed a GAN-based edge-enhancement network (EEGAN) for robust satellite image SR reconstruction, along with an adversarial learning strategy that is insensitive to noise. Haut et al. [38] integrated a visual attention mechanism within a residual-based network to enhance feature presentation. Xu et al. [18] proposed a GAN-based model and combined the attention mechanism and pyramidal convolution in the generator.

2.2. Super-Resolution Frameworks in Deep Learning

Much of the initial effort in single-image super-resolution (SISR) has focused on the design of neural network architectures. Dong et al. [7] proposed one of the earliest SR methods based on a CNN. DRRN [8] and DRCN [9] used recursive neural networks to address the difficulty of training very deep networks. VDSR [48] used a deeper convolution network and residual learning with an extremely high learning rate to quickly train the network. Some recent researchers made the layers in their network dense to learn better and stronger features. Inspired by the success of ResNet [49] in image classification, multiple papers have used residual learning in recent SR work. SRResNet [14] demonstrated high-quality reconstructions via the ResNet architecture. EDSR [10] removed the batch normalization layers in ResNet and used residual scaling to achieve faster, more stable training. Inspired by both DenseNet [50] and ResNet, RDN [11] used residual dense blocks to learn deeper features before scaling up. Attention mechanisms have also been used in the SR task. RCAN [12] applied a residual channel attention block to learning the interdependence of different channels. Wang et al. [51] proposed a feature distillation pyramid residual group to extract features efficiently from the LR inputs. Subsequent work, including [13,52,53], exploited non-local attention in SR. Mei et al. [13] proposed a cross-scale non-local (CS-NL) attention module to explore cross-scale feature correlation in image representation.

GANs are networks that can learn to produce realistic samples, and, in particular, images, from Gaussian noise. However, they also find use in imaging problems as an additional loss term that encourages realistic reconstructions. For example, SRGAN [14] exploited GAN loss for SR together with perceptual loss (it uses features of a pretrained VGG) to reconstruct more realistic textures and details. ESRGAN [15] built a residual-in-residual dense block, which provided a further improvement over the perceptual loss. Ma et al. [17] exploited a GAN-based model to concentrate more on geometric structures with the help of gradient maps and gradient loss.

2.3. Unpaired Training in Super-Resolution

Unpaired/Unsupervised training is widely used in image restoration, such as image dehazing [54], deraining [55], deblurring [56] and denoising [57]. In super-resolution, real-world visible SR is a hot application. Real-world visible images are a complicated degradation that always lack reference high quality images. Accordingly, there are no paired HR/LR real-world images in the training. Recently, CycleGAN [32] was proposed for image style transfer. Subsequently, many other researchers used this network in unsupervised SR. Bulat et al. [26] first applied CycleGAN to an unsupervised SR task. Yuan et al. [31] used a cycle-in-cycle network in natural real-world image SR. One CycleGAN is exploited to the domain transfer, and the other is used to maintain content consistency with the initial input. Wei et al. [24] adopted domain-gap-aware training and domain-distance-weighted supervision strategies in their model to address the domain gap between real-world images and clean images. Liu et al. [33] proposed a conditional variation autoencoder in a joint image denoising and SR model. A cycle training was also adopted for stable training. ZSSR [33] offered a new method of this unpaired training in real-world SR. The authors trained paired HR/LR synthetic image pairs first, then applied down-sampled LR real images and LR real image pairs to continue to train the SR network in a test time. This training strategy can explore the inner relationship of real images. Moreover, Haut et al. [39] built a unsupervised generative network that injected random noise to generate HR remote sensing reconstructions. Wang et al. [40] improved this noise-generative model in [39], adding a reference image as the migration image prior to the generation step to enhance the textures and structured information of SR remote sensing images.

3. Methodology

In this section, we explain our proposed UVRSR, including its two learnable branches and total training.

3.1. Overall Architecture

As shown in Figure 1, UVRSR is composed of two learnable branches: VIG and RIG. VIG is trained by unpaired HR visible images and LR remote sensing images. Similar to real-world image SR, we first utilize a CycleGAN [32] and transfer an LR visible image to a remote sensing domain with an unpaired remote sensing image. Next, a residual-based SR network is stacked after the CycleGAN and produces a remote sensing domain SR image. The perceptual loss [58] is applied to match perceptual content with the HR clean visible target and the SR image. We define this process as content consistency. To avoid domain shift in the reconstruction, we design a novel domain-ruled (DR) discriminator to keep domain consistency. Unlike many other traditional discriminators, the proposed DR discriminator has three inputs: the SR output, the HR clean visible target and a remote sensing target. RIG is trained by paired LR remote sensing images and its lower counterparts. A downsampled LR remote sensing image is fed to the same SR network in VIG. In the training of RIG, L1 loss and adversarial loss are applied to match the content between the SR remote sensing output and the LR remote sensing target.

3.2. Formulation

As shown in Figure 1, the training of UVRSR consists of two learnable branches. To make full use of the guided HR visible target, UVRSR exploits a visible guided branch (VIG), defined as $G_{VIS}$ (see Figure 1’s upper branch). Then, to further study the inner relationship of the remote sensing images, a remote sensing guided branch(RIG) is further designed to train the network (see Figure 1’s lower branch), defined as $G_{RS}$.

In the VIG, given a LR input visible image $x_{lr}^{vis}$, the final output is the SR image in the remote sensing domain, defined as $y_{sr}^{rs}$. The $x_{lr}^{vis}$ is simply downsampled by the HR visible target $x_{hr}^{vis}$ with a bicubic operator. To stabilize the adversarial training, in [59], the authors added a noisy image as well as a LR input to their variational autoEncoder GAN-based model. In our UVRSR, inspired by [59], we add a zero-mean Gaussian noise map $noise$ to $x_{lr}^{vis}$ with the same size before CycleGAN, as follows:

$$\begin{array}{c}\hfill {\widehat{x}}_{lr}^{vis}=noise+x_{lr}^{vis}\end{array}$$

(1)

To transfer ${\widehat{x}}_{lr}^{vis}$ to the expected remote sensing domain, we also use a CycleGAN as in [24,26,31] in real world SR. We treat CycleGAN as a process operator that can be formulated as $CycG(\ast )$. The output of CycleGAN is described as the simulated remote sensing image ${\widehat{y}}_{lr}^{rs}$, which should have the same content as the LR input $x_{lr}^{vis}$, but a different domain. This transfer process is as follows:

$$\begin{array}{c}\hfill {\widehat{y}}_{lr}^{rs}=CycG\left({\widehat{x}}_{lr}^{vis}\right)\end{array}$$

(2)

Next, an SR network, defined as $SR(\ast )$, is used to reconstruct ${\widehat{y}}_{lr}^{rs}$ to an SR image $y_{sr}^{rs}$ in a fixed scale factor r. The SR process is:

$$\begin{array}{c}\hfill y_{sr}^{rs}=SR\left({\widehat{y}}_{lr}^{rs}\right)\end{array}$$

(3)

In the end, the full visible guided branch can be formulated as follows:

$$\begin{array}{c}\hfill y_{sr}^{rs}=G_{VIS}\left({\widehat{x}}_{lr}^{vis}\right)=SR\left(CycG\left({\widehat{x}}_{lr}^{vis}\right)\right)\end{array}$$

(4)

In the RIG, we train the SR network in VIG in another way. In ZSSR [33], the authors applied the down-sampled LR real images and initial LR real image pairs (low LR/LR) to continue to train a SR network which had been trained by synthetic data. The inner relationship between low LR/LR real images can be further explored in the training. Inspired by ZSSR, we downsampled the input LR remote sensing image $x_{lr}^{rs}$ to a lower size $x_{lrlow}^{rs}$ in the scale r, then fed $x_{lrlow}^{rs}$ to the same SR network ($SR(\ast )$) in the VIG. The SR network learns to generate $y_{lr}^{rs}$ by minimizing the distance between $y_{lr}^{rs}$ and $x_{lr}^{rs}$. The remote sensing guided branch can be formulated as:

$$\begin{array}{c}\hfill y_{lr}^{rs}=G_{RS}\left(x_{lrlow}^{rs}\right)=SR\left(x_{lrlow}^{rs}\right)\end{array}$$

(5)

3.3. Visible Image Guided Branch Training

In the training of this branch, the visible image plays an important role and guides the network to generate an SR image with rich textures and details. As shown in Figure 1, the key modules in VIG include a CycleGAN, an SR network and a DR discriminator. We illustrate the details of each, respectively.

3.3.1. CycleGAN

The architecture of CycleGAN is shown in Figure 2. First, the noisy input ${\widehat{x}}_{lr}^{vis}$ is set into CycleGAN and then transferred from the visible domain to the remote sensing domain. In CycleGAN, the first generator $G_{rs}$ is a CNN-based network that has several convolution layers. The function for this generator is to produce a remote sensing version of input $x_{lr}^{vis}$ with the same size.

We specify the filter sizes by using the letters k, which denotes the (square) kernel size; n, which denotes the number of output filters; and s, which denotes the stride size, followed by the corresponding number. The first Conv layer of CycleGAN to process the noisy input ${\widehat{x}}_{lr}^{vis}$ into the feature space has filters with $k=7$, $n=64$ and $s=1$. SRGAN [14] benefited from their proposed residual-based basic layers with efficient feature representation. However, in their basic layers, the BN normalization layers tend to introduce unpleasant artifacts and increase computing cost. In our work, we use its basic layers without all BN normalization layers as our basic layers in CycleGAN. Here we have five stacked basic layers in $G_{rs}$. In addition, adversarial training is used to ensure that the output ${\widehat{y}}_{lr}^{rs}$ transfers to the proper remote sensing domain. The discriminator $D_{rs}$ for the remote sensing domain has the first Conv layer with the filters $k=3$, $n=64$ and $s=2$. Then, the following three Conv layers with BN layers are stacked after the first layer, which have the filters $k=2$, $n=128$, $s=2$; $k=2$, $n=256$, $s=2$; and $k=2$, $n=512$, $s=1$, respectively. The final Conv layer of $D_{rs}$ to process the features to $1/0$ (true/false) has the filters $k=3$, $n=1$ and $s=1$. The training of discriminator $D_{rs}$ can be trained as:

$$L_{\mathrm{Drs}}={\mathbb{E}}_{z_{lr}^{rs}}|D_{rs}(z_{lr}^{rs}){-1|}_2^2+{\mathbb{E}}_{x_{lr}^{vis}}{|D_{rs}(G_{rs}({\widehat{x}}_{lr}^{vis}))|}_2^2,$$

(6)

where $z_{lr}^{rs}$ denotes an unpaired LR remote sensing image for domain guidance, and ${\mathbb{E}}_{z_{lr}^{rs}}$ and ${\mathbb{E}}_{x_{lr}^{vis}}$ denote the expectation with respect to the random variable $z_{lr}^{rs}$ and ${\widehat{x}}_{lr}^{vis}$.

A cycle training method was used to avoid bias in the content between the output ${\widehat{y}}_{lr}^{rs}$ and input $x_{lr}^{vis}$ during the domain transfer. Specifically, we put ${\widehat{y}}_{lr}^{rs}$ to another generator $G_{vis}$ so that it could come back to the initial visible domain without content bias. We define the output of $G_{vis}$ as the cycle visible image $y_{cycle}^{vis}$, and in the cycle training, we suppose $y_{cycle}^{vis}$ could return to LR visible input $x_{lr}^{vis}$ as close as possible. This is achieved by further adversarial training with a discriminator $D_{cycle}$:

$$L_{\mathrm{Dcycle}}={\mathbb{E}}_{x_{lr}^{vis}}|D_{cycle}\left(x_{lr}^{vis}\right){-1|}_2^2+{\mathbb{E}}_{{\widehat{y}}_{lr}^{rs}}{\left|D_{cycle}\left(G_{vis}\left({\widehat{y}}_{lr}^{rs}\right)\right)\right|}_2^2,$$

(7)

where ${\mathbb{E}}_{x_{lr}^{vis}}$ and ${\mathbb{E}}_{{\widehat{y}}_{lr}^{rs}}$ denote the expectation with respect to the random variable $x_{lr}^{vis}$ and ${\widehat{y}}_{lr}^{rs}$. The architectures of $G_{vis}$ and $G_{rs}$, as well as $D_{rs}$ and $D_{cycle}$, are the same, as shown in Figure 3.

3.3.2. SR Netowrk

The output of CycleGAN ${\widehat{y}}_{lr}^{rs}$ is then put into an SR network to produce the HR result $y_{sr}^{rs}$. As illustrated in the introduction section, the HR visible clean image can provide sufficient textures and details; in this sense, we apply the HR visible clean image $x_{hr}^{vis}$ as a target for reconstruction. We train the SR network by minimizing the content distance between $x_{hr}^{vis}$ and $y_{sr}^{rs}$. Recently, ESRGAN [15] produced a residual-in-residual dense block (RRDB) to provide a deeper and more efficient feature representation block. Each RRDB combines multi-level residual network and dense connections without BN normalization layers.

In our UVRSR, we used their 5 stacked RRDBs after the first two Conv layers, which have filters of $k=3$, $n=64$, and $s=1$. Then we used a pixel shuffle layer to scale up the resolution of features, followed by two stacked Conv layers having the filters of $k=3$, $n=64$, and $s=1$. Finally, the Conv layer with the filter $k=3$, $n=3$, and $s=1$ is used to produce the SR image from 64-dimensional features.

In the real-world SR, there are also no real-world domain HR targets. Recent works [16,24,25,26,27,28] have regarded HR visible clean images as the target and trained the network with paired simulated real-world images and HR visible clean images in an end-to-end manner. This setting is based on one hypothesis that the domains of HR real-world visible image and HR clean visible images are the same. Actually, as shown in Figure 4a, their domain distance is close. Nonetheless, in Figure 4b, the domain gap between remote sensing images and visible clean image is much larger. If we directly treat the HR clean visible image $x_{hr}^{vis}$ as the target, as in real-world SR, it will probably cause a domain shift in $y_{sr}^{rs}$; in other words, $y_{sr}^{rs}$ will change to a visible domain in the reconstruction.

To avoid that, we train the SR network in two ways: content consistency and domain consistency. Through these, we expect the SR network to produce $y_{sr}^{rs}$ as the visible target $x_{hr}^{vis}$ but to remain the target remote sensing domain. There is a likely trade-off between domain and content for $y_{sr}^{rs}$ in the super-resolution process.

The architecture of the SR network is shown in Figure 5.

In content consistency, perceptual loss [58] between $x_{hr}^{vis}$ and $y_{sr}^{rs}$ based on a pretrained 19-layer VGG network is used for better content reconstruction. Perceptual loss can be defined as:

$$\begin{array}{c}\hfill L_{\mathrm{content}}={\mathbb{E}}_{{\widehat{y}}_{lr}^{rs}}{|{\varphi }_{\mathrm{VGG}}\left(y_{sr}^{rs}\right)-{\varphi }_{\mathrm{VGG}}\left(x_{hr}^{vis}\right)|}_2^2,\end{array}$$

(8)

where ${\mathbb{E}}_{{\widehat{y}}_{lr}^{rs}}$ denotes the expectation with respect to the random variable, and ${\widehat{y}}_{lr}^{rs}$. ${\varphi }_{\mathrm{VGG}}$ denotes the high-level features produced by a pretrained 19-layer VGG network.

3.3.3. DR Discriminator

For domain consistency, we proposed a novel DR discriminator in VIG. We fed three inputs into the DR discriminator, including SR output $y_{sr}^{rs}$, the HR visible clean target $x_{hr}^{vis}$ and an RS domain target ${\widehat{z}}_{hr}^{rs}$ of the same size. To avoid domain shift in the reconstruction, we add an unpaired remote sensing image (different content with SR image) as the domain target for the DR discriminator. The reason for designing three inputs here is simple: the HR visible target $x_{hr}^{vis}$ can help the DR discriminator learn the content validity, while the remote sensing image ${\widehat{z}}_{hr}^{rs}$ helps it to ensure the domain validity. Thus, the DR discriminator learns the trade-off between the content validity and domain validity of the SR image. In this paper, because there is no HR remote sensing image, we upsample the $z_{lr}^{rs}$ in the CycleGAN to the scale of r.

We feed the ${\widehat{z}}_{hr}^{rs}$ to the DR discriminator as real with the weight of $\alpha$, and we also feed the visible target $x_{hr}^{vis}$ to the discriminator as real with the weight of $\beta$. The SR network learns to produce a detailed SR image with rich texture in the remote sensing domain to fool the DR discriminator. The reason we call it a domain-ruled discriminator is that in SR reconstruction, the domain validity is more important and problematic than content validity; therefore, the domain should rule the validity in the discriminator. Consequently, we set the weights of domain validity and content validity with a large distance $\alpha >>\beta$ ($\alpha =0.9$, $\beta =0.1$ in this paper). The design of the DR discriminator is shown in Figure 6. The architecture follows the discriminators in Figure 3, and in the training, we set two different outputs of DR discriminator with different weights. The training of the DR discriminator $D_{DR}$ can be formulated as:

$$\begin{array}{cc}{L}_{\mathrm{D}3}=& \alpha \ast {\mathbb{E}}_{x_{hr}^{rs}}|D_{DR}\left(x_{hr}^{rs}\right){-1|}_2^2+\alpha \ast {\mathbb{E}}_{x_{hr}^{rs}}{\left|D_{DR}\left(y_{sr}^{rs}\right)\right|}_2^2+\\ & \beta \ast {\mathbb{E}}_{x_{hr}^{vis}}|D_{DR}\left(x_{hr}^{vis}\right){-1|}_2^2+\beta \ast {\mathbb{E}}_{x_{hr}^{vis}}{\left|D_{DR}\left(y_{sr}^{rs}\right)\right|}_2^2,\end{array}$$

(9)

where ${\mathbb{E}}_{x_{hr}^{rs}}$ and ${\mathbb{E}}_{x_{hr}^{vis}}$ denote the expectation with respect to the random variable $x_{hr}^{rs}$ and $x_{hr}^{vis}$, respectively. $\alpha$ and $\beta$ denote the weights of the domain validity and content validity in the DR discriminator.

3.4. Remote Sensing Image-Guided Branch Training

In RIG, we train the SR network in VIG in the guidance of remote sensing images. Because the HR remote sensing images are unavailable, to learn the inner relationship of remote sensing images, inspired by ZSSR [33], we use an LR remote sensing image $x_{lr}^{rs}$ and its smaller $x_{lrlow}^{rs}$ to train the same SR network in the VIG. We first exploit the bicubic operator to downsample $x_{lr}^{rs}$ with the scale factor r. We define the output of the SR network as $y_{lr}^{rs}$. Then, the SR network is trained with this paired and the target in supervised training. We define L1 loss in the training as:

$$\begin{array}{c}\hfill L_{rs}={\mathbb{E}}_{x_{lrlow}^{rs}}{|y_{lr}^{rs}-x_{lr}^{rs}|}_1,\end{array}$$

(10)

where ${\mathbb{E}}_{x_{lrlow}^{rs}}$ denotes the expectation with respect to the random variable $x_{lrlow}^{rs}$.

3.5. Total Training

In this section, we illustrate the training details of UVRSR. As described in the previous sections, in the VIG, we first train the CycleGAN for domain transfer. We have three losses for CycleGAN: cycle loss, the adversarial loss of the remote sensing domain and the adversarial loss for cycle content consistency. The cycle loss computes the L1 loss between the cycle visible image $y_{cycle}^{vis}$ and the LR visible image $x_{lr}^{vis}$:

$$\begin{array}{c}\hfill L_{\mathrm{cycle}}={\mathbb{E}}_{x_{lr}^{vis}}{|x_{lr}^{vis}-y_{vis}^{cycle}|}_1\end{array}$$

(11)

The adversarial loss of the remote sensing domain related to Equation (6) is:

$$\begin{array}{c}\hfill L_{\mathrm{GAN}1}={\mathbb{E}}_{{\widehat{x}}_{lr}^{vis}}{|D_{rs}\left(G_{rs}\left({\widehat{x}}_{lr}^{vis}\right)\right)-1|}_2^2\end{array}$$

(12)

while the adversarial loss for the visible domain related to Equation (7) is:

$$\begin{array}{cc}\hfill L_{\mathrm{GAN}2}=& {\mathbb{E}}_{{\widehat{y}}_{lr}^{rs}}{\left|D_{rs}\left(G_{vis}\left({\widehat{y}}_{lr}^{rs}\right)-1\right)\right|}_2^2\hfill \end{array}$$

(13)

Therefore, the total loss to train the CycleGAN is:

$$\begin{array}{c}\hfill L_{\mathrm{CycleGAN}}={\alpha }_1\ast L_{\mathrm{cycle}}+{\beta }_1\ast L_{\mathrm{GAN}1}+{\gamma }_1\ast L_{\mathrm{GAN}2},\end{array}$$

(14)

where ${\alpha }_1$,${\beta }_1$ and ${\gamma }_1$ are the weights of $L_{\mathrm{cycle}}$,$L_{\mathrm{GAN}1}$ and $L_{\mathrm{GAN}2}$, respectively.

The SR network is trained after feeding the output ${\widehat{y}}_{lr}^{rs}$ of CycleGAN. Equation (8) calculates the training loss of content consistency. For domain consistency, related to Equation (9), the GAN loss of the DR discriminator can be formulated as:

$$\begin{array}{c}\hfill L_{\mathrm{domain}}={\mathbb{E}}_{y_{sr}^{rs}}{\left|D_{DR}\left(y_{sr}^{rs}-1\right)\right|}_2^2,\end{array}$$

(15)

where ${\mathbb{E}}_{y_{sr}^{rs}}$ denotes the expectation with respect to the random variable $y_{sr}^{rs}$.

The total loss for the SR network is calculated as:

$$\begin{array}{c}\hfill L_{\mathrm{SR}}={\alpha }_2\ast L_{\mathrm{content}}+{\beta }_2\ast L_{\mathrm{domain}},\end{array}$$

(16)

where ${\alpha }_2$ and ${\beta }_2$ are the weights of $L_{\mathrm{content}}$ and $L_{\mathrm{domain}}$, respectively.

The total loss of the VIG is:

$$\begin{array}{c}\hfill L_{\mathrm{VIG}}=\zeta 1\ast L_{\mathrm{SR}}+\zeta 2\ast L_{\mathrm{CycleGAN}},\end{array}$$

(17)

where ${\zeta }_1$ and ${\zeta }_2$ are the weights of $L_{\mathrm{SR}}$ and $L_{\mathrm{CycleGAN}}$, respectively.

We further train the SR network with the paired $x_{lr}^{rs}$ and its smaller $x_{lrlow}^{rs}$. The loss of RIG is equal to $L_{rs}$ in Equation (8). In the end, we train and optimize both two learnable branches together. The total training loss for our UVRSR is:

$$\begin{array}{c}\hfill L_{\mathrm{Total}}=\eta 1\ast L_{\mathrm{VIG}}+\eta 2\ast L_{\mathrm{RIG}},\end{array}$$

(18)

where ${\eta }_1$ and ${\eta }_2$ are the weights of $L_{\mathrm{VIG}}$ and $L_{\mathrm{RIG}}$, respectively.

4. Experiments and Analysis

In this section, we first describe the datasets and quality assessment metrics used in this paper, then illustrate the detailed implementation of UVRSR and the comparison with state-of-the-art methods.

4.1. Datasets

As the setting of our proposed UVRSR, we used two types of datasets to train our model: an HR visible clean image dataset and an LR remote sensing image dataset. For the HR visible clean image dataset, we used the DIV2K dataset, while for the remote sensing image dataset, we chose two classic remote sensing image datasets: UC Merced [60] and NWPU-RESISC45 [61]. The details of each dataset are as follows.

4.1.1. DIV2K

There are adequate HR visible clean image datasets for image processing tasks, such as DIV2K [19], SET5 [20], URBAN [21], and B100 [22]. Among all these datasets, DIV2K is the most popular for super-resolution tasks. DIV2K was first used in the NTIRE 2017 Challenge on Single Image Super-Resolution. It contains 800 high-definition HR training images, 100 HR test images and 100 HR images for validation. The resolution of the these 1000 HR images are about $2k\times 1k$ pixels. The content in the DIV2K contains abundant natural scenes, including animals, human, buildings, and plants; see the first row in Figure 7. It can provide diverse content and sufficient details for the training of our UVRSR.

4.1.2. UC Merced

The UCMerced dataset is one of the most popular datasets in remotely sensed data processing. The UCMerced dataset contains a total of 2100 images of the surface of the earth; see the second row in Figure 7. It is composed of 21 types of low-light land use images, including agricultural areas (AGI), airplanes (APL), baseball diamonds (BD), beaches (BE), buildings (BU), chaparral (CP), dense residential areas (DR), forests (FO), freeways (FW), golf courses (GC), harbors (HA), intersections (IS), medium residential areas (MR), mobile-home parks (MHP), overpasses (OP), parking lots (PL), rivers (RI), runways (RW), sparse residential areas (SR), storage tanks (ST), and tennis courts (TC). Each category has 100 images with a resolution of $256\times 256$ pixels, with a spatial resolution of 0.3 m per pixel in the RGB color space. In particular, these images were originally extracted from aerial orthoimagery downloaded from the United States Geological Survey (USGS) National Map.

4.1.3. NWPU-RESISC45

The NWPU-RESISC45 dataset is a large-scale remote sensing dataset with a number of scene classes, images per class and total number of images. The NSPU-RESISC45 dataset contains a total of 31,500 images of the surface of the Earth; see the last row in Figure 7. This database was created by Northwestern Polytechnical University (NWPU). It is composed of 45 types of scene classes, including airplanes (APL), airports (AP), baseball diamonds (BD), basketballs (BB), courts (CO), beaches (BE), bridges (BR), chaparral (CP), churches (CH), circular farmland (CF), clouds (CL), commercial areas (CA), dense residential areas (DR), desert (DE), forests (FO), freeways (FW), golf courses (GO), courses (CR), ground track fields (GTF), harbors (HB), industrial areas (IA), intersections (IS), islands (IL), lakes (LA), meadows (ME), medium residential areas (MR), mobile home parks (MHP), mountains (MT), overpasses (OP), palaces (PA), parking lots (PL), railways (RW), railway stations (RS), rectangular farmland (RF), rivers (RI), roundabouts (RD), runways (RU), sea ice (SI), ships (SH), snowbergs (SB), sparse residential areas (SR), stadiums (SD), storage tanks (ST), tennis courts (TC), terraces (TR), thermal power stations (TPS), and wetlands (WL). Each class has 700 images with a size of $256\times 256$ pixels in the RGB color space. The spatial resolution varies from about 30 m to 0.2 m per pixel for most of the scene classes.

4.2. Quality Assessment Metrics

Due to the unpaired training implementation in this paper, in the testing, there are no HR target remote sensing images to measure the quality of the SR images. Therefore, to use the reference-based image quality assessment (IQA) metrics of PSNR, as followed by [37], we set the test images in the UC Merced and NWPU-RESISC45 datasets as targets and used the degradation method (simply a bicubic downsampling operator in this paper) to generate corresponding LR input images. PSNR, short for “peak signal-to-noise ratio”, is an objective standard for image evaluation. The PSNR value approaches infinity as the MSE approaches zero; a higher PSNR value provides a higher image quality. It can be calculated as:

$$\mathrm{PSNR}=10lo{g}_{10}\frac{{(2^n-1)}^2}{\mathrm{MSE}},$$

(19)

where n is the image bit width, and MSE is the mean square error defined as:

$$\mathrm{MSE}=\frac{{\sum }_{j=1}^N\left({\sum }_{i=1}^M{\left({\widehat{Y}}_{i,j}-X_{i,j}\right)}^2\right)}{MN},$$

(20)

where $X_{i,j}$ is the ground truth image, and ${\widehat{Y}}_{i,j}$ is the reconstructed image. $MN$ is the size of ${\widehat{Y}}_{i,j}$ and $X_{i,j}$.

In addition, for a more comprehensive comparison, as in real-world visible image SR, we also use two non-reference-based IQA metrics to measure the quality of the reconstruction images: NIQE [62] and perceptual index (PI). NIQE and PI can directly assess the quality of remote sensing images without ground truth, which is more practical in application. Lower measures of both metrics relate to higher image quality.

NIQE is based on constructing “quality-aware” features and fitting them to a multivariate Gaussian (MVG) model. The quality-aware features are derived from a regular natural scene statistic (NSS) model. The quality of the reconstruction image is expressed as the distance between the quality-aware NSS feature model and the MVG fit to the features extracted from the reconstruction image. It is defined as:

$$D\left({\nu }_1,{\nu }_2,{\rho }_1,{\rho }_2\right)=\sqrt{\left({\left({\nu }_1-{\nu }_2\right)}^T{\left(\frac{{\rho }_1+{\rho }_2}{2}\right)}^{-1}\left({\nu }_1-{\nu }_2\right)\right)},$$

(21)

where ${\nu }_1$, ${\nu }_2$ and ${\rho }_1$, ${\rho }_2$ are the mean vectors and covariance matrices of the natural MVG model and the reconstruction image’s MVG model.

PI is now one of popular perceptual metrics applied in image quality assessment. The definition of PI is calculated by Ma’s score [63] and NIQE as follows:

$$PI=\frac{1}{2}\left((10-M{a}^{\prime }sscore)+\mathrm{NIQE}\right),$$

(22)

4.3. Implementation Details

In the training of our model, we adopt the Adam optimizer with ${\beta }_1=0.5$ and ${\beta }_2=0.999$. The initial learning rate is ${10}^{-4}$, and it decays every 60 epochs. Specifically, in the training, we first train the CycleGAN with the weights ${\alpha }_1=10$, ${\beta }_1=3$ and ${\gamma }_1=1$ after 10 epochs; then, we train the CycleGAN and SR network together with the weights ${\alpha }_1=5$, ${\beta }_1=1$ and ${\gamma }_1=1$. The weight for CycleGAN is ${\alpha }_2=5$, and it is ${\beta }_2=1$ for the SR network in VIG. Finally, we set the weights for total loss in VIG to be ${\zeta }_1=1$ and ${\zeta }_2=1$. We train the SR network with the loss in Equation (18) together, and the weights are ${\eta }_1=1$, ${\eta }_2=3$.

For the $4\times$ (4 times) training setting, we cropped the HR visible clean images in DIV2K to obtain HR target images of $256\times 256$ pixels and to form minibatches of 8 images. We downsampled the HR target images to $64\times 64$ pixels to become the LR visible inputs in the VIG. We cropped the LR remote sensing images in the UC Merced and NWPU-RESISC45 training datasets of $64\times 64$ pixels to CycleGAN and the DR discriminator in the VIG. In the RIG, we also cropped the LR remote sensing images of $64\times 64$ pixels and downsampled them to $16\times 16$ pixels as lower-version counterparts. In this paper, we also trained and tested in the scale of $2\times$ (2 times/twice) and the extreme scale of $8\times$ (8 times). Thus, in the training, the size of $256\times 256$ pixels in $4\times$ setting remained the same, but the $64\times 64$ pixels were all changed to $128\times 128$ pixels and $32\times 32$ pixels, corresponding to scale factors of $2\times$ and $8\times$. Additionally, $16\times 16$ pixels were changed to $32\times 32$ pixels and $8\times 8$ pixels, corresponding to scale factors of $2\times$ and $8\times$. We also used data augmentation through rotations and horizontal flipping. In the training, we divided two remote sensing datasets images into training images and test images. To be specific, in the UC Merced dataset, the images of numbers 0–39 in 10 categories were used for training, and numbers 40–69 were used for testing; for the NWPU-RESISC45 dataset, the images of numbers 1–40 in 10 categories were used for training, and numbers of 41–70 were used for testing. The categories we chose are demonstrated in the next subsection.

4.4. Comparison with State-of-the-Art Methods

To validate the effectiveness of our proposed method, we compared our UVRSR trained with a fixed scale ($2\times$, $4\times$, $8\times$) to three state-of-the-art(SoTA) unpaired SR methods: CinCGAN [31], ZSSR [33], and Bulat et al.’s method [26]. We also compared UVRSR with two remote sensing image methods: Haut et al.’s method [38] and EEGAN [37] on the UC Merced dataset and the NWPU-RESISC45 test set. As shown in

Table 1. Quantitative comparisons with state-of-the-art methods on the UC Merced dataset and NWPU-RESISC45 test set. The best results (PSNR/NIQE/PI) are highlighted.

Class/Method	Bicubic	Haut [38]	EEGAN [37]	ZSSR [33]	Bulat [26]	CinCGAN [31]	UVRSR (Ours)
$2\times$ scale factor
APL	29.12/5.77/5.64	31.78/4.22/4.16	30.94/2.29/2.14	31.36/4.89/4.82	29.40/4.71/4.53	30.98/2.29/2.24	31.16/2.05/2.00
BE	28.92/5.71/5.64	31.72/4.14/4.08	30.84/2.25/2.11	31.26/4.82/4.99	29.16/4.79/4.85	30.86/2.25/2.19	31.20/2.12/2.07
BU	28.80/5.85/5.70	31.47/4.34/4.27	30.57/2.40/2.27	31.12/4.86/4.70	29.30/4.76/4.62	30.62/2.47/2.43	30.88/2.22/2.16
FW	28.39/6.05/5.89	31.25/4.48/4.36	30.45/2.51/2.56	30.87/5.08/4.82	28.62/4.95/5.15	30.39/2.54/2.63	30.53/2.38/2.42
HA	28.75/5.84/5.79	31.52/4.26/4.19	30.62/2.38/2.49	31.19/5.12/4.74	28.67/4.82/4.99	30.44/2.41/2.47	30.79/2.22/2.29
MR	29.59/5.42/5.37	32.29/3.73/3.57	31.43/1.86/1.73	31.83/4.43/4.49	29.94/4.25/4.06	31.26/2.13/2.14	31.52/1.65/1.58
OP	29.32/5.60/5.49	32.02/4.20/4.01	31.33/1.99/1.91	31.73/4.75/4.68	29.56/4.60/4.49	31.31/2.17/2.06	31.62/1.97/2.05
RI	29.47/5.51/5.43	32.07/3.87/3.85	31.40/1.94/1.89	31.76/4.35/4.27	29.81/4.43/4.29	31.20/2.15/2.01	31.37/1.84/1.76
ST	29.42/5.57/5.42	32.02/4.02/4.06	31.33/2.07/2.01	31.64/4.62/4.77	29.74/4.52/4.36	31.27/2.10/2.15	31.28/1.87/1.94
TC	29.18/5.46/5.52	31.82/4.15/4.23	31.04/2.15/1.99	31.39/4.78/4.65	29.32/4.82/4.61	30.96/2.32/2.11	31.24/2.02/1.89
AVG	29.10/5.68/5.59	31.90 /4.13/4.08	30.99/2.18/2.11	31.42/4.77/4.69	29.35/4.67/4.60	30.93/2.26/2.23	31.16/2.03/2.02
$4\times$ scale factor
APL	23.82/6.43/6.57	28.38/4.53/4.49	25.24/2.39/2.42	25.76/5.13/4.97	23.77/6.77/6.62	24.96/2.62/2.58	25.20/2.31/2.33
BE	24.12/6.38/6.52	28.53/4.62/4.42	25.14/2.45/2.36	25.81/4.93/5.09	23.85/6.65/6.54	25.17/2.56/2.49	25.34/2.41/2.34
BU	23.26/7.09/7.26	27.86/4.85/4.93	24.32/2.66/2.79	24.79/5.42/5.29	22.43/7.12/6.95	24.21/2.75/2.74	24.37/2.54/2.58
FW	23.32/6.72/6.68	28.09/4.77/4.81	24.54/2.67/2.72	24.96/5.32/5.14	22.25/7.23/7.08	24.36/2.86/2.75	24.76/2.52/2.48
HA	23.79/6.50/6.52	28.22/4.56/4.35	24.86/2.59/2.68	25.43/5.34/5.26	23.54/6.89/6.73	24.82/2.55/2.65	24.90/2.39/2.37
MR	24.86/5.93/6.07	28.92/4.26/4.17	25.52/2.15/2.02	26.35/4.52/4.55	23.89/6.32/6.37	25.31/2.26/2.24	25.78/1.96/2.08
OP	24.25/6.45/6.38	28.68/4.39/4.33	25.16/2.36/2.42	25.88/4.92/4.84	23.54/6.80/6.68	25.06/2.42/2.39	25.29/2.29/2.26
RI	24.81/6.03/6.11	28.85/4.35/4.30	25.49/2.23/2.26	26.47/4.63/4.68	24.07/6.35/6.22	25.42/2.31/2.40	25.62/2.06/2.14
ST	24.43/6.29/6.35	28.62/4.49/4.42	25.22/2.31/2.22	26.06/4.85/4.89	23.77/6.59/6.80	25.29/2.43/2.31	25.39/2.29/2.23
TC	24.19/6.46/6.39	28.42/4.44/4.37	25.27/2.38/2.49	25.56/4.93/4.84	23.67/6.75/6.68	25.26/2.35/2.29	25.32/2.24/2.28
AVG	24.06/6.43/6.49	28.46 /4.53/4.46	25.08/2.42/2.44	25.71/5.00/4.96	23.48/6.75/6.67	24.99/2.51/2.49	25.20/2.30/2.31
$8\times$ scale factor
APL	17.46/7.92/7.95	20.66/5.88/5.91	19.59/5.24/5.15	16.95/8.25/8.30	16.85/8.56/8.52	19.32/5.36/5.18	20.85/3.82/3.73
BE	17.62/7.80/7.72	20.77/5.84/5.96	19.71/5.06/4.87	17.18/8.08/8.15	16.92/8.42/8.36	19.62/5.39/5.45	21.26 /3.65/3.28
BU	16.70/8.39/8.21	19.68/6.54/6.26	18.52/5.75/5.61	16.33/8.48/8.56	16.24/8.92/8.71	18.75/5.68/5.59	20.15/4.52/4.35
FW	16.84/8.23/8.06	19.92/6.34/6.22	18.83/5.53/5.42	16.53/8.12/8.09	16.04/8.79/8.85	18.92/5.62/5.51	19.89/4.65/4.55
HA	17.22/8.10/7.92	20.47/5.93/6.05	19.27/5.40/5.21	16.82/8.34/8.46	16.41/8.63/8.69	19.25/5.42/5.44	20.26/4.05/3.86
MR	18.56/7.24/7.16	21.51/5.32/5.40	20.73/4.85/4.62	18.71/7.42/7.34	19.25/6.87/6.92	20.82/4.58/4.53	21.49/3.49/3.36
OP	17.92/7.62/7.56	21.04/5.75/5.68	20.22/5.03/4.88	17.79/7.81/7.77	17.64/7.86/7.75	20.17/5.11/4.87	21.14/3.62/3.65
RI	18.53/7.36/7.25	21.34/5.44/5.56	20.39/4.97/4.85	18.35/7.52/7.43	18.82/7.16/6.92	20.47/4.89/4.79	21.50/2.81/2.94
ST	18.24/7.44/7.42	21.06/5.68/5.49	20.25/4.92/4.76	18.02/7.66/7.54	18.31/7.58/7.46	20.19/5.12/4.82	21.25/3.57/3.40
TC	17.64/7.85/7.79	20.88/5.84/5.90	19.73/5.06/4.88	17.02/8.18/8.38	16.95/8.42/8.36	20.08/5.04/4.93	21.06/3.67/3.54
AVG	17.67/7.80/7.70	20.73/5.86/5.84	19.72/5.18/5.03	17.37/7.99/8.02	17.39/8.12/8.05	19.79/5.22/5.11	20.88/3.79/3.67

, we used the same 10 classes of the images in the UC Merced dataset test set and the NWPU-RESISC45 dataset test set to test, including airplanes (APL), buildings (BU), beaches (BE), freeways (FR), harbors (HA), medium residential areas (MR), overpasses (OP), rivers (RI), storage tanks (ST) and tennis courts (TC). Then we combined the images from the same class in both datasets together. We calculated both the reference-based index PSNR and the non-reference based indexes NIQE and PI for each method with a scale of $2\times$, $4\times$ and $8\times$. We also calculated the average number of each index. Regarding the methods, Haut [38] and ZSSR [33] are distortion-based (CNN-based) methods, while CinCGAN [31], Bulat’s method [26], EEGAN [37] and UVRSR (our method) are perceptual-based (GAN-based) methods. According to the demonstration in ESRGAN [15], distortion-based methods always perform better regarding PSNR (distortion loss) than perceptual-based methods, and perceptual-based methods perform better in perception loss (PI, NIQE). In Table 1, UVRSR achieved the top performance in terms of the PI and NIQE metrics, which shows that the reconstructed images tend to be more realistic than other methods, especially in large factors of $4\times$ and $8\times$. Regarding PSNR, UVRSR outperforms all perceptual-based methods and is on par with the top performing distortion-based methods. We also observed that in the large scale factor of $8\times$, our proposed UVRSR demonstrated the best PSNR results, even though it is a perceptual-based method. Moreover, we offered a visual comparison in Figure 8, Figure 9 and Figure 10 with the scales of $2\times$, $4\times$ and $8\times$. The visual examples clearly demonstrate that our UVRSR has more textures and is always sharper than other approaches.

5. Discussion

In this section, we further discuss the effects of the proposed USRVR.

5.1. Effect of Two Learnable Branches

As illustrated in the previous sections, UVRSR consists of two learnable branches: VIG and RIG. In VIG, the SR network is guided by HR visible images. They can provide sufficient high frequency information for reconstruction. In RIG, the inner relationship of remote sensing images is explored for the SR network. In this subsection, we assess the impact of each branch for the final reconstruction. In

Table 2. Analysis of the learnable branches. The best performance is highlighted.

Branches	$2\times$	$4\times$	$8\times$
VIG + RIG	30.79/2.22/2.29	24.90/2.39/2.37	20.26/4.05/3.86
VIG	30.66/2.28/2.35	24.76/2.44/2.41	20.12/4.11/3.93
RIG	30.72/2.34/2.43	24.80/2.52/2.48	20.14/4.19/3.98

, we present the results on the learnable branches when UVRSR is trained and tested with scale factors of $2\times$, $4\times$ and $8\times$. From the table, the final choice for UVRSR achieves the best performance, which proves that the best choice for the proposed UVRSR trains RIG and VIG together. In addition, we also find an interesting conclusion from Table 2, which is that RIG performs better in the PSNR metric than VIG, while VIG has better PI results than RIG. This indicates that VIG can provide more realistic details than RIG, and RIG can provide more structural information than VIG.

5.2. Role of the DR Discriminator

The proposed DR discriminator is proposed in VIG for remote sensing domain consistency in the reconstruction. As demonstrated in Section 3, the inputs for the DR discriminator are SR output (SR), the HR visible clean target (VT) and RS domain target (RT). Here, we analysis the importance of using DR discriminators. In

Table 3. Analysis of DR discriminators. The best performance is highlighted.

Inputs of Discriminator	$2\times$	$4\times$	$8\times$
SR + VT + RT	31.16/2.05/2.00	25.20/2.31/2.33	20.85/3.82/3.73
SR + VT	30.75/2.10/2.07	24.52/2.39/2.38	20.28/4.03/3.92
SR + RT	31.08/2.19/2.13	24.92/2.46/2.49	20.66/4.18/4.11

, the combination SR + VT + RT is the proposed DR discriminator, the combination SR + VT is the traditional setting of the discriminator and SR + RT is another design of discriminator. We train and test UVRSR with/without DR discriminators for $2\times$, $4\times$ and $8\times$ scale factors. As shown in Table 3, UVRSR trained with DR discriminators (with all SR + VT + RT inputs) yields the best accuracy. We also provide a visual comparison in Figure 11, which clearly shows that the result with DR discriminator (SR + VT + RT) has more detailed and realistic results than the others. In addition, even though the result of SR + VT has more textures than that of SR + RT, it is also closer to the visible domain. On the other hand, this observation indicates that domain shift will occur with traditional settings of the discriminator, (SR + VT. The visual result of SR + RT and the DR discriminator (SR + VT + RT) maintain the remote sensing domain. With the help of the HR visible target, the result of the DR discriminator is sharper than SR + RT.

6. Conclusions

In this paper, we proposed a novel cross-domain super-resolution method called UVRSR, which allowed training to be conducted with unpaired HR visible images and LR remote sensing images. It enhanced the capability for HR visible images to assist in the reconstruction of remote sensing images. It is the first work to apply visible images to assist remote sensing domain SR, and is also the first to perform cross-domain SR without HR/LR training pairs. It combines the advantages of HR visible images and remote sensing images, or, in other words, the advantages of visible realistic details and remote sensing structural information. In UVRSR, to learn more detailed images without domain shift in the reconstruction, we propose a novel two-branch training strategy and a domain-ruled discriminator. The two-branch training, which included the visible image-guided branch (VIG) and the remote image-guided branch (RIG), has different functions for the SR network. VIG is designed to explore sufficient high frequency information from the HR visible target, while RIG is meant to learn the inner relationship in the remote sensing domain. The proposed DR discriminator further helps the SR network to produce a more detailed remote sensing reconstruction and keep the domain consistency in the meantime. We demonstrated that UVRSR achieved state-of-the-art performance including unpaired and remote sensing SR methods. Extensive experiments on the datasets of UC Merced and NWPU-RESISC45 show that our UVRSR achieved SoTA results in the scale of $2\times$, $4\times$ and $8\times$. In the future, we will incorporate this unpaired training strategy in other domain SR scenarios.