Super-Resolution of Remote Sensing Images for ×4 Resolution without Reference Images

Abstract

Sentinel-2 satellites can provide free optical remote-sensing images with a spatial resolution of up to 10 M, but the spatial details provided are not enough for many applications, so it is worth considering improving the spatial resolution of Sentinel-2 satellites images through super-resolution (SR). Currently, the most effective SR models are mainly based on deep learning, especially the generative adversarial network (GAN). Models based on GAN need to be trained on LR–HR image pairs. In this paper, a two-step super-resolution generative adversarial network (TS-SRGAN) model is proposed. The first step is having the GAN train the degraded models. Without supervised HR images, only the 10 m resolution images provided by Sentinel-2 satellites are used to generate the degraded images, which are in the same domain as the real LR images, and then to construct the near-natural LR–HR image pairs. The second step is to design a super-resolution generative adversarial network with strengthened perceptual features, to enhance the perceptual effects of the generated images. Through experiments, the proposed method obtained an average NIQE as low as 2.54, and outperformed state-of-the-art models according to other two NR-IQA metrics, such as BRISQUE and PIQE. At the same time, the comparison of the intuitive visual effects of the generated images also proved the effectiveness of TS-SRGAN.

1. Introduction

The applications of satellite remote-sensing images are broad, involving agriculture, environmental protection, land use, urban planning, natural disasters, hydrology, climate, etc. [1]. With the continuous updating of optical instruments and other equipment, the spatial resolution of satellite images is constantly improving. For example, Worldview-3/4 satellites can collect eight bands of multi-spectral data with a ground resolution of 1.2 M [2]. However, Worldview-3/4 satellites data cost to acquire, and when covering a large area or performing a multi-temporal analysis, the data cost can be restrictive. Therefore, open-access data with acceptable spatial quality can be considered, such as Landsat [3] or Sentinel [4] data. Sentinel-2 updates remote-sensing images of every location in the world for free approximately every five days, and these remote-sensing images are becoming a more and more important resource for applications. Sentinel-2 uses two satellites to achieve remote-sensing coverage at the equator on a global scale, and provides a multi-resolution layer composed of 13 spectral bands, among which, 10 M resolution images are provided in four RGBN bands, 20 M resolution images are provided in six bands, and 60 M resolution images are provided in the other three bands [4]. The bands with 10 and 20 M resolution are usually used for land cover or water mapping, agriculture or forestry, whereas the band with 60 M resolution is mainly used for water-vapor monitoring [5]. Due to the open data distribution strategy, the 10 M resolution remote-sensing images provided by Sentinel-2 are becoming important resources for some applications. However, such spatial resolution is still slightly insufficient for many applications. In order to make the most of freely available Sentinel-2 images, and to achieve spatial resolution of about 2 m, it is worth considering some post-processing methods to obtain the spatial enhancement of LR images and to recover the high-frequency details to generate HR images. In order to improve the spatial resolution of Sentinel-2 images, some researchers [6,7,8,9,10,11,12] fused the data of the several bands of Sentinel-2 with 60, 20 and 10 M spatial resolutions to obtain higher spatial-resolution images; however, this paper focuses on SR directly using 10 M resolution images.

Yang et al. [13] and Gou et al. [14] studied a supervised SR model based on dictionary learning, and provided an effective solution by using sparse coding technology. Pan et al. [15] applied structural self-similarity and compressed sensing to SR tasks. Zhang et al. [16] and Li et al. [17] adopted several different image representation spaces in SR to achieve higher performance.

Deep learning has been attracting more and more attention in the field of SR [18]. Deep learning does not need to directly map the relationship between HR and LR domains. As long as there are enough training data, a deep learning network in principle can learn very complex non-linear relationships [19]. Among these techniques, the convolutional neural network (CNN) can make better use of the high-order features of the images to create HR images, and can significantly improve the performance of SR imaging [18]. Dong et al. [19] proposed the SRCNN, which has a strong learning ability. It is based on the CNN, and adopts pixel loss to optimize the network. The result was too smooth without consideration of the perceptual quality. Additionally, on this basis, Kim et al. [20] and Zhang et al. [21] introduced residual learning models, Tai et al. [22] introduced recursive learning models, and Hu et al. [23] introduced an attention mechanism to optimize the deep-learning architecture to improve performance, but these models also had the problem of over-smoothing, because they all relied solely on pixel loss to optimize the network.

Goodfellow et al. [24] proposed the GAN training of two models at the same time, one of which was called the generator (G), and the other was called the discriminator (D). SRGAN [25], proposed by Leding et al., was a pioneering way to implement SR based on GAN theory, and because of its ability to generate images with rich texture and high quality, GAN has been widely used in SR. Wang et al. [26] further improved the SRGAN model, proposed ESRGAN, used a more complex and denser residual-layer combination in the generator, and deleted the batch normalization layer. As an SR model based on GAN was gradually applied to the field of satellite remote sensing, Ma et al. [27] proposed transfer GAN (TGAN) to solve the shortcomings of poor quantity and quality of remote-sensing data. Haut et al. [28,29] and Lei et al. [30] designed the network to form LR–HR image pairs by downsampling the public remote-sensing images, and tested different network architectures. Targeting the remote-sensing images provided by Sentinel-2, Gong et al. [31] proposed the Enlighten-GAN SR model, which adopts an internal inconsistency loss and cropping strategy, and achieved good results in gradient similarity measurement (GSM) for the medium resolution remote-sensing images of Sentinel-2. Sentinel-2 can provide images with a spatial resolution of up to 10 m. In the task of upgrading the resolution from 10 to 2 m, the SR model based on GAN has encountered a great challenge that mainly comes from the lack of real HR images at 2 M resolution. Some papers [32,33] used the 10 M resolution images of Sentinel-2 and 2 M HR images of worldview satellites to form LR–HR image pairs to construct training datasets. Galar et al. [32] proposed an SR model based on the enhanced depth residual network (EDSR), and Salgueiro et al. [33] proposed an RS-ESRGAN model based on the ESRGAN. All the proposed models could enhance the 10 M channel of Sentinel-2 to 2 m. However, by using the unnatural low–high image pairs consisting of Sentinel-2 and Worldview images, and other models using BiCubic downsampling to construct LR–HR image pairs [21,25,26,34,35,36,37], the track details related to frequency will be lost [38]. In order to solve this problem, inspired by the blind SR model KernelGAN [39] and the blind image denoising model [40], we explicitly estimated the degradation kernel of LR–HR image pairs of natural images through GAN, estimated the distribution of the degraded noises at the same time, and degraded the 10 M resolution images of Sentinel-2, to construct near-natural LR–HR image datasets. On the basis of these datasets, with reference to the structures of SRGAN, PatchGAN, and VGG-128, TS-SRGAN was designed to implement an SR of Sentinel-2 images from 10 M to 2.5 M.

2. Dataset

For the convenience of the following analysis, we initially present the datasets used in training and testing. The model proposed in this paper is aimed at Sentinel-2 images, so we used the SEN12MS [41] dataset to train and test the models. SEN12MS contains complete multi-spectral information in geocoded images; it also includes SAR and multi-spectral images provided by Sentinel-1 and Sentinel-2, and land cover information obtained by MODIS system. This paper mainly focuses on 10 M resolution images of red (B4), green (B3) and blue (B2) bands in multi-spectral images, namely, RGB color images with 10 m resolution. SEN12MS gives Sentinel-2 cloudless images of the region of interest (ROI) at specified time intervals. SEN12MS divides the images into patches with 256 × 256 pixels, which span 128 pixels so that the overlap rate between the adjacent patches is 50%. SEN12MS takes 50% overlap as an ideal compromise between the independence of patches and the maximum number of samples. The SEN12MS dataset obtained a randomly sampled ROI based on four seeds (1158, 1868, 1970 and 2017), and the distribution of ROI is shown in Figure 1.

In this paper, TS-SRGAN used a part of SEN12MS named ROIs1158, which is composed of 56 regions of interest across the globe generated from 1158 seeds from 1 June 2017, to 31 August 2017. ROIs1158 is divided into 56 subsets by region, totaling 40,883 pieces of 256 × 256 pixel images. We randomly selected the subset “ROIs1158_spring_106” as the test dataset (ROI_Te), which contains 784 test images; the remaining 55 subsets, including 40,099 images, were used as the source images dataset (ROI_Src), and ROI_Src was degraded, to generate the LR image dataset (ROI_LR). The source images $I_{\mathrm{src}}$ in ROI_Src were directly used as HR images $I_{\mathrm{HR}}$ in the training, which formed the LR-HR image-pairs dataset (ROI_Tr) with the images $I_{\mathrm{LR}}$ in ROI_LR dataset one by one. This paper compares the performances of the newly proposed models, including EDSR8-RGB [32], RCAN [21], and RS-ESRGAN [33], with the traditional model of BiCubic [42]. BiCubic directly uses ROI_Te for interpolation testing without training; RCAN takes the images in ROI_Src as LR images, and generates HR images by BiCubic-interpolating every image to form an LR–HR image pair dataset; the models of EDSR8-RGB and RS-ESRGAN refer to the models proposed in [32] and [33], respectively, to construct a dataset based on ROI_Src.

3. Methods

3.1. Structure of TS-SRGAN

We used TS-SRGAN to generate 2.5 M resolution images $I_{\mathrm{SR}}$ from 10 M resolution source images $I_{\mathrm{src}}$ of Sentinel-2, in two stages. In the first stage, KernelGAN was used to implement the estimation of the explicit degradation kernel of $I_{\mathrm{src}}$ images, and then, along with injecting the degraded noise, the source images $I_{\mathrm{src}}$ were degraded to LR images $I_{\mathrm{LR}}$, which we combined with HR images $I_{\mathrm{HR}}$ (equivalent to $I_{\mathrm{src}}$) to construct LR-HR image pairs ($I_{\mathrm{LR}}$,$I_{\mathrm{HR}}$). In the second stage, the dataset $\left\{(I_{\mathrm{LR}},I_{\mathrm{HR}})\right\}$ was used to train the super-resolution generative adversarial network (SR-GAN), which consisted of a super-resolution generator (SR-G), a super-resolution discriminator (SR-D), and a super-resolution perceptual-feature extractor (SR-F). TS-SRGAN represents the Sentinel-2 image SR model proposed in this paper, and the structure of TS-SRGAN is shown in Figure 2.

3.2. Degraded Model

Here, we introduce an image degradation model based on the GAN. The natural pairing mapping between HR and LR images can be approximately understood as the degradation relationship between them, and the degradation process can be expressed as follows:

$$I_{LR}=\left(I_{SRC}*k^s\right){\downarrow }_s+n$$

(1)

where $k^s$ and $n$ represent the degradation kernel and degraded noise, respectively, and $s$ represents the scaling factor. The quality of the degradation kernel and degraded noise determine the relevance between LR–HR image pairs and natural image pairs, and the accuracy of the extracted mapping features from LR and HR resolution images determines the quality of images generated by SR.

3.2.1. Degradation Kernel Estimation

Here, we first consider the noise-free degradation process, assuming that the noise-free LR image $I_{LR\_cl}$ is the result of the downsampled HR image $I_{SRC},$ by using the degradation kernel through the scaling factor $s$:

$$I_{LR\_cl}=\left(I_{SRC}*k^s\right){\downarrow }_S$$

(2)

In this paper, KernelGAN is used to estimate the image degradation kernel $k^s$, which is a blind SR degradation kernel estimation model based on Internal-GAN [43] and a completely unsupervised GAN, requiring no extra training data except the image $I_{SRC}$ itself [39]. KernelGAN uses only the images $I_{SRC}$ for training, to learn the distribution of internal pixel patches, with the goal of finding the image-specific degradation kernel and the best degradation kernel to retain the distribution of pixel patches for each scale of the image $I_{SRC}$. More specifically, the goals are to “generate” downsampled images, and to make the pixel patch distribution of the downsampled images as close to that of the images $I_{SRC}$ as possible. The essence of the model is the extraction of the cross-scale recursive characteristics between LR and HR images through deep learning, and GAN in KernelGAN can be understood as the matching tool for pixel patch distribution. The implementation process of KernelGAN is shown in Figure 3, which illustrates training, using a single input image to learn the distribution of internal pixel patches of the cropped patch. There is a kernel generator (kernel-G) and a kernel discriminator (kernel-D). Both the kernel-G and the kernel-D are fully convolutional, which means that the network is applied to the pixel patch rather than to the whole image. With the given input of images $I_{SRC}$, the kernel generator will learn to downsample to $I_{LR\_cl}$, whose goal is to make the discriminator indistinguishable from the input images $I_{SRC}$ at the pixel-patch level.

The objective function of KernelGAN is defined as follows:

$$G^{*}({\mathbf{I}}_{SRC})=\underset{G}{\mathrm{argmin}}\underset{D}{max}\{{\mathbb{E}}_{x~\mathrm{patches}({\mathbf{I}}_{SRC})}[|D({\mathbf{I}}_{SRC})-1|+|D(G({\mathbf{I}}_{SRC}))|]+\mathcal{R}\}$$

(3)

where $G$ represents the generator and $D$ represents the discriminator. Additionally, $\mathcal{R}$ is the regularization term optimized by the degradation kernel $k^s$:

$$\mathcal{R}={\alpha }_{\mathrm{s}\text{\_}1}{\mathcal{L}}_{\mathrm{s}\text{\_}1}+{\alpha }_{\mathrm{b}}{\mathcal{L}}_{\mathrm{b}}+{\alpha }_{\mathrm{sp}}{\mathcal{L}}_{\mathrm{sp}}+{\alpha }_{\mathrm{c}}{\mathcal{L}}_{\mathrm{c}}$$

(4)

where ${\mathcal{L}}_{\mathrm{s}\text{\_}1},{\mathcal{L}}_{\mathrm{b}},{\mathcal{L}}_{\mathrm{sp}},{\mathcal{L}}_{\mathrm{c}}$ represent losses, and ${\alpha }_{\mathrm{s}\text{\_}1},{\alpha }_{\mathrm{b}},{\alpha }_{\mathrm{sp}},{\alpha }_{\mathrm{c}}$ represent constant coefficients. In this study, the constant coefficients were set according to experience as ${\alpha }_{\mathrm{s}\text{\_}1}=0.5,{\alpha }_{\mathrm{b}}=0.5,{\alpha }_{\mathrm{sp}}=5,{\alpha }_{\mathrm{c}}=1$. The losses are defined as the following equations, respectively:

$${\mathcal{L}}_{s\_1}=\left|1-{\displaystyle \sum _{i,j}{k}_{i,j}}\right|$$

(5)

where $k_{i,j}$ represents the parameter value of each point of the degradation kernel, and the goal of ${\mathcal{L}}_{s\_1}$ is that the sum of $\left\{k_{i,j}\right\}$ is 1.

$${\mathcal{L}}_{\mathrm{b}}={\displaystyle \sum _{i,j}\left|k_{i,j}\cdot m_{i,j}\right|}$$

(6)

The goal of ${\mathcal{L}}_{\mathrm{b}}$ is to punish the non-zero value near the boundary, and $m_{i,j}$ is the constant mask of weight, which increases exponentially with the distance from the center of $\left\{k_{i,j}\right\}$.

$${\mathcal{L}}_{\mathrm{sp}}={\displaystyle \sum _{i,j}{\left|k_{i,j}\right|}^{1/2}}$$

(7)

The goal of ${\mathcal{L}}_{\mathrm{sp}}$ is the sparsity of $k_{i,j}$ to avoid excess smoothness of the interior kernel.

$${\mathcal{L}}_{\mathrm{c}}={‖\left(x_0,y_0\right)-\frac{{\displaystyle {\sum }_{i,j}}{k}_{i,j}\cdot (i,j)}{{\displaystyle {\sum }_{i,j}}{k}_{i,j}}‖}_2$$

(8)

The goal of ${\mathcal{L}}_{\mathrm{c}}$ is to have the center of $\left\{k_{i,j}\right\}$ in the center of the interior kernel, and $\left(x_0,y_0\right)$ represents the indices of the center.

Kernel-G can be regarded as an image downsampling model, which implements linear downsampling mainly through the convolution layer, and the network contains no nonlinear activation unit. A nonlinear generator is not used here because it is possible for the nonlinear generator to generate physically unnecessary solutions for the optimization targets, for example, to generate an image that is not downsampled but contains effective pixel patches. In addition, because the single-layer convolution layer cannot converge accurately, we use the multi-layer structure of linear convolution layers, as in Figure 4.

The goal of kernel-D is to learn the distribution of pixel patches in input images $I_{SRC}$ and to distinguish between the real patches and fake patches in the distribution. The real patches are cropped from input images $I_{SRC}$, and the fake patches are cropped from $I_{LR\_cl}$ generated by kernel-G. We use the fully convolutional pixel-patch discriminator introduced in [44] to learn the pixel patch distribution of every single image, as shown in Figure 5.

The convolution layer used in kernel-D does not perform pooling operations, instead it implicitly acts on each pixel block and finally generates a hot map (D-map), on which each position corresponds to one cropped patch input. The hot map output by kernel-D represents the possibility of each pixel extracting the surrounding pixel patches from the original pixel patch distribution, and it is used to distinguish the real patches from the fake patches. The loss is defined as the pixel-wise mean-square error between the hot map and the label map. The label map refers to all 1 labels of the real patches and all 0 labels of the fake patches.

After the training of KernelGAN, we do not focus on the generator network, but convolute the convolution layers of kernel-G with the stride of 1, successively, to extract the explicit degradation kernel. Meanwhile, the training of KernelGAN is based on one single input image, $I_{SRC}$, which means that each input image trains one degradation kernel, and many degradation kernels generated by the training image set will be randomly selected and used in the subsequent steps. The graphical examples of some degradation kernels are shown in Figure 6.

3.2.2. Generation and Injection of Noise

As opposed to the direct downscaling methods, such as BiCubic, we explicitly inject additional noise into $I_{LR\_cl}$, so as to keep the noise distributions of $I_{LR}$ and $I_{SRC}$ images as consistent as possible. Due to the large variance of the patches with rocky content [38], and inspired by [40,45], when extracting noise-mapping patches we control the variance within a specific range under the condition:

$$D\left(n_i\right)<{\mathsf{\sigma }}_{max}$$

(9)

where $D\left(·\right)$ represents the variance function, and ${\mathsf{\sigma }}_{max}$ represents the maximum value of the variance. The noise-mapping patches are extracted from images selected from the images of ROI_Src randomly, and a certain number of noise patches are extracted to construct the dataset (ROI_Noi). The noise-mapping patches used for the noise-injection process are randomly selected from ROI_Noi.

To sum up, the process of generating LR images in ROI_LR from the source images in ROI_Src can be expressed as Equation (10), where I and j are randomly selected:

$$I_{LR}=\left(I_{SRC}*k_i^s\right){\downarrow }_s+n_j$$

(10)

3.3. SR-GAN

SR-GAN consists of a super-resolution generator (SR-G), super-resolution discriminator (SR-D) and perceptual-feature extractor (SR-F). SR-G generates a ×4 high-resolution image through learning the characteristics of the training set data. SR-D and SR-F compare the generated image with the ground truth image, respectively. SR-D feeds back pixel-wise loss and adaptive loss to SR-G, and SR-F feeds back perceptual loss to SR-G, realizing SR-D and SR-F’s supervision of SR-G. SR-G was designed on the basis of the ESRGAN [26]. As an ESRGAN discriminator may introduce more artifacts [38], SR-D was designed on the basis of PatchGAN [44]. The perceptual-feature extractor was designed on the basis of VGG-19 [46], so as to introduce the perceptual loss [47], which can strengthen the extraction of low-frequency features and improve the effect of visual perception.

The loss ${\mathcal{L}}_{\mathrm{SR}}$ of SR-GAN consists of three parts, including pixel-wise loss ${\mathcal{L}}_{\mathrm{x}}$ [26], perceptual loss ${\mathcal{L}}_{\mathrm{p}}$ and adversarial loss ${\mathcal{L}}_a.$

$${\mathcal{L}}_{\mathrm{SR}}={\alpha }_x{\mathcal{L}}_{\mathrm{x}}+{\alpha }_{\mathrm{p}}{\mathcal{L}}_{\mathrm{p}}+{\alpha }_{\mathrm{a}}{\mathcal{L}}_a$$

(11)

where ${\alpha }_x,{\alpha }_{\mathrm{p}}\mathrm{and}{\alpha }_{\mathrm{a}}$ are constant coefficients, and the constant coefficients were set according to experience as ${\alpha }_x=0.01,{\alpha }_{\mathrm{p}}=1,{\alpha }_{\mathrm{a}}=0.005$. The losses ${\mathcal{L}}_{\mathrm{x}},{\mathcal{L}}_{\mathrm{p}}\mathrm{and}{\mathcal{L}}_a$ are defined as Equations (12), (13) and (16).

$${\mathcal{L}}_{\mathrm{x}}={\mathbb{E}}_{I_{LR}}{‖G(I_{LR})-I_{HR}‖}_1$$

(12)

Pixel-wise loss ${\mathcal{L}}_{\mathrm{x}}$ uses L1 distance to evaluate the pixel-wise content loss between $G\left(I_{LR}\right)$ and $I_{HR}$.

$${\mathcal{L}}_{\mathrm{p}}={\lambda }_f{\mathcal{L}}_f+{\lambda }_t{\mathcal{L}}_t$$

(13)

Perceptual loss ${\mathcal{L}}_{\mathrm{p}}$ evaluates the perceived differences in content and style among different images, and consists of feature reconstructing loss ${\mathcal{L}}_f$ related to content and style reconstructing loss ${\mathcal{L}}_t$, where ${\lambda }_f\mathrm{and}{\lambda }_t$ denote constant coefficients, and ${\mathcal{L}}_f$ and ${\mathcal{L}}_t$ can be expressed as follows:

$${\mathcal{L}}_f=\frac{1}{C_j{H}_j{W}_j}{‖{\varphi }_j\left(G\left(I_{LR}\right)\right)-{\varphi }_j\left(I_{HR}\right)‖}_2^2$$

(14)

$${\mathcal{L}}_t={‖\frac{1}{C_j{H}_j{W}_j}{\displaystyle \sum }_{h=1}^{H_j}{\displaystyle \sum }_{w=1}^{W_j}\left[\begin{array}{c}{\varphi }_j{(G\left(I_{LR}\right))}_{h,w,c}{\varphi }_j{(G\left(I_{LR}\right))}_{h,w,c^{\prime }}-{\varphi }_j{(I_{HR})}_{h,w,c}{\varphi }_j{(I_{HR})}_{h,w,c^{\prime }}\end{array}\right]‖}_F^2$$

(15)

where ${\varphi }_j\left(I\right)$ represents the characteristic diagram obtained at level $j$ of the convolution layer after the image $I$ inputs SR-F, and the shape of the obtained characteristic diagram is $C_j\times H_j\times W_j$ ($\mathrm{Channel}\times \mathrm{Height}\times \mathrm{Width}$). ${‖·‖}_F^2$ represents the square Frobenius norm.

$${\mathcal{L}}_a={{\displaystyle \sum }}_{n=1}^N-D\left(G\left(I_{LR}\right)\right)$$

(16)

Adversarial loss, ${\mathcal{L}}_a$, can enhance the texture details of the image, making the visual effect of the generated image more realistic.

The structure of SR-G is shown in Figure 7. Based on the ESRGAN model, and with the RRDB [39] structure, it was trained in the constructed LR-HR image pairs $\left(I_{LR},I_{HR}\right)$, and the resolution of the generated images was magnified ×4.

Due to the discriminator in ESRGAN possibly introducing more artifacts, we used the patch discriminator instead of the VGG-128 discriminator in the ESRGAN model, and SR-D was designed based on PatchGAN [44]. In addition, the patch discriminator was used instead of the VGG-128 discriminator, out of consideration for the following aspects: VGG-128 is applied to images with a size of 128 pixels, which makes training on large scales less powerful; VGG-128, when using a fixed fully connected layer, is better at handling global features rather than local features. [34]. For this reason, we use the patch discriminator, which has a fully convolutional structure, and with the receptive field fixed. Each output value of SR-D depends only on the local fixed patches, so that we can optimize the local details. The average value of all local errors is used as the final error, to guarantee global consistency. The structure of SR-D is shown in Figure 8.

Based on the VGG-19 [46] model, this paper introduces the perceptual-feature extractor to extract the perceptual loss ${\mathcal{L}}_{\mathrm{p}}$, that is, to extract the inactive features in VGG-19. The perceptual loss can enhance the low-frequency features of the images and make the images generated by the generator look more realistic. The structure of the perceptual-feature extractor is shown in Figure 9.

4. Experiments and Results

Training Details

The proposed model, TS-SRGAN, and other models, such as EDSR8-RGB, RCAN, RS-ESRGAN and RealSR, were run in a Pytorch environment, using the modules provided by the “sefibk/KernelGAN” project [39], “xinntao/BasicSR” project [48] and “Tencent/Real-SR” project [38] in the Github library. BiCubic can be obtained directly, using Matlab functions to perform interpolation operations.

TS-SRGAN first generates an LR–HR image pair dataset (ROI_Tr) based on a training dataset (ROI_Src) for training and testing. We randomly selected 2134 images from 40,099 images of ROI_Src to generate a degraded kernel dataset (ROI_Ker) through KernelGAN training, one by one, namely, $k_i^s\in \left\{\mathrm{ROI}\_\mathrm{Ker}\right\},i\in \left\{1,2\cdots 2134\right\}$; and then randomly selected 4972 images from 40,099 images of ROI_Src to extract noise patches, one by one, to form a noise patch dataset (ROI_Noi), namely, $n_j\in \left\{\mathrm{ROI}\_\mathrm{Noi}\right\},j\in \left\{1,2\cdots 4972\right\}$; finally, we used the degradation kernel and injected noise to perform degrading operations on the images in ROI_Src, one by one. In the processing of each image, the degradation kernel and injected noise were randomly selected from ROI_Ker and ROI_Noi.

The network structural parameters of kernel-G and the kernel-D and the constant coefficients of losses of KernelGAN have been mentioned above, so we will not repeat them here. In the training phase, both the generator and the discriminator used an ADAM optimizer with the parameters ${\beta }_1=0.5, {\beta }_2=0.999$; the learning rates of kernel-G and kernel-D were both set to 0.0002, decrementing by ×0.1 every 750 iterations, and the network was iteratively trained for 3000 epochs.

SR-G used the “RRDBNet” model in the “BasicSR” project, and SR-D used the “NlayerDiscriminator” model in the “Real-SR” project. The networks’ structural parameters and the constant coefficients of losses have been mentioned above; therefore, we will not repeat them here. The image was magnified by 4 times, and during the training phase both the generator and the discriminator used the ADAM optimizer with the parameters ${\beta }_1=0.9, {\beta }_2=0.999$; the learning rates of SR-G and SR-D were both set to 0.0001, and the network was iteratively trained for 60,000 epochs.

Some convolutional layers were used in TS-SRGAN, and these convolutional layers play a vital role. After many tests, it became known that the parameters of the convolutional layer in the network needed to be set as in

Table 1. Settings of specific parameters for convolutional layers of TS-SRGAN.

	In_Channels	Out_Channels	Kernel_Size	Stride	Padding	Oher Parameters
Kernel-G	3	64	7	1	0	default
	64	64	5	1	0	default
	64	64	3	1	0	default
	64	64	1	1	0	default
	64	64	1	1	0	default
	64	64	1	1	0	default
	64	1	1	2	0	default
Kernel-D	3	64	7	1	0	default
	64	64	1	1	0	default
	64	1	1	1	0	default
SR-G	64	32	3	1	0	default
	96	32	3	1	0	default
	128	32	3	1	0	default
	160	32	3	1	0	default
	192	64	3	1	0	default
	64	64	3	1	0	default
	64	3	3	1	0	default
SR-D	3	64	4	2	0	default
	64	128	4	2	0	default
	128	256	4	2	0	default
	256	512	4	1	0	default
	512	1	4	1	0	default
SR-F	3	64	3	1	0	default
	64	64	3	1	0	default
	64	128	4	2	0	default
	128	128	4	2	0	default
	128	256	4	1	0	default
	256	256	4	1	0	default
	256	512	4	1	0	default
	512	512	4	1	0	default
	512	1	4	1	0	default

, to achieve the ×4 resolution images TS-SRGAN and obtain the image quality we wanted.

The EDSR8-RGB, RCAN, RS-ESRGAN and RealSR models implemented training and testing under the frameworks of BasicSR [48] and Real-SR [38], with the parameter-setting schemes which have been proven to achieve the best results in references [21,32,33]. The parameters used in the implementations are detailed in

Table 2. Settings of specific parameters for the models implemented in the framework of BasicSR.

Model	EDSR8-RGB	RCAN	RS-ESRGAN	RealSR
Network	network_g: type: EDSR num_in_ch: 3 num_out_ch: 3 num_feat: 256 num_block: 32 upscale: 4 res_scale: 0.1 img_range: 255. rgb_mean: [0.4488, 0.4371, 0.4040]	network_g: type: RCAN num_in_ch: 3 num_out_ch: 3 num_feat: 64 num_group: 10 num_block: 20 squeeze_factor: 16 upscale: 4 res_scale: 1 img_range: 255. rgb_mean: [0.4488, 0.4371, 0.4040]	network_g: type: RRDBNet num_in_ch: 3 num_out_ch: 3 num_feat: 64 num_block: 23 network_d: type: VGGStyleDiscriminator128 num_in_ch: 3 num_feat: 64	network_G: type: RRDBNet num_in_ch: 3 num_out_ch: 3 num_feat: 64 num_block: 23 network_D: type: NLayerDiscriminator num_in_ch: 3 num_feat: 64 num_layer: 3
Traing	optim_g: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.99] scheduler: type: MultiStepLR milestones: [2 × 105] gamma: 0.5 total_iter: 3 × 105	optim_g: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.99] scheduler: type: MultiStepLR milestones: [2 × 105] gamma: 0.5 total_iter: 3 × 105	optim_g: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.99] optim_d: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.99] scheduler: type: MultiStepLR milestones: [5 × 104, 1 × 105, 2 × 105, 3 × 105] gamma: 0.5 total_iter: 4 × 105	optim_g: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.999] optim_d: type: Adam learning rate: 1 × 10−4 weight_decay: 0 betas: [0.9, 0.999] scheduler: type: MultiStepLR milestones: [5 × 103, 1 × 104, 2 × 104, 3 × 104] gamma: 0.5 total_iter: 6 × 105

.

As the source images used are already the highest resolution (10 m) images of Sentinel-2, there are no real ground truth images (2.5 M resolution) that can be compared with the generated images in reality, and some image-quality assessment metrics commonly used, such as, PSNR and SSIM, are not applicable in this scenario. Therefore, we adopted non-reference image quality assessment (NR-IQA) metrics, including NIQE [49], BRISQUE [50] and PIQE [51]. The evaluation values of NIQE, BRISQUE and PIQE can be calculated by the corresponding functions NIQE, BRISQE and PIQE in Matlab; the output results of the three functions are all within the range of [0, 100], where the lower the score, the higher the perceived effect.

We randomly selected one sub-dataset, “ROIs1158_spring_106,” in ROIs1158, as the testing dataset (ROI_Te) containing 784 images. The remote-sensing images in ROI_Te were collected from the ground areas, as shown in Figure 10. In the figure we marked eight regions with strong geographic features, and the ×4 generated images of these regions are shown subsequently, to visually compare the differences among those models.

We used the BiCubic, EDSR8-RGB, RCAN, RS-ESRGAN and TS-SRGAN models to process 784 images in ROI_Te to generate ×4 HR images, and used Matlab to calculate the evaluation values of NIQE, BRISQUE and PIQE, one by one, for the images. The histograms were drawn according to the distributions of evaluation metric values, as shown in Figure 11, Figure 12 and Figure 13, and the mean and extreme values based on the evaluation values are provided in

Table 3. Statistics of NIQE, BRISQUE and PIQE evaluation values.

	BiCubic	EDSR8-RGB	RCAN	RS-ESRGAN	RealSR	TS-SRGAN
NIQE mean	6.349	5.296	4.329	3.337	3.144	2.544
NIQE max	7.607	6.381	5.260	4.608	4.389	4.120
NIQE min	5.639	4.086	3.180	2.678	2.483	1.816
BRISQUE mean	55.662	49.041	46.564	22.786	21.961	16.408
BRISQUE max	61.535	60.014	58.167	33.340	43.458	43.405
BRISQUE min	44.464	42.699	35.306	7.886	3.641	3.424
PIQE mean	94.635	79.374	60.333	14.186	13.849	13.231
PIQE max	100.000	33.340	77.966	25.524	25.562	25.293
PIQE min	50.000	65.191	25.925	7.707	7.123	6.900

. The histograms and table show that the TS-SRGAN model is superior to the other models in a variety of NR-IQA metrics.

Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21 show the generated images of eight regions with strong geographic features selected in “ROIs1158_spring_106” to visually compare the differences among the models. Through the comparison of the images of various terrains in Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20 and Figure 21, it can be clearly seen that the images processed by the traditional BiCubic method are the most bleary and smooth, due to the inherent deficiencies of the interpolation algorithm. The EDSR8-RGB, RCAN and RS-ESRGAN models cannot correctly distinguish the noise with sharp edges, resulting in blurred results, and even indistinguishable houses and roads. As shown in the TS-SRGAN results, the dividing lines among the objects and backgrounds, such as roads, bridges and houses, are much clearer, which indicates that the noise we estimated was closer to the real noise. By using different combinations of degradation (e.g., blur and noise), TS-SRGAN has obtained LR images sharing the same domain with real images, which can avoid the generated LR images being too smooth and fuzzy. Using the super-resolution network trained by the domain-consistent data, TS-SRGAN generates HR images with clearer boundaries and better perception. Compared with the EDSR8-RGB, RCAN and RS-ESRGAN models, the TS-SRGAN’s results are clearer, and have no ambiguity.

5. Conclusions

In this paper, based on the latest and most widely recognized GAN technologies, including KernelGAN, ESRGAN, PatchGAN, etc., we introduced the degradation kernel estimation and noise injection to perform SR for Sentinel-2 satellites’ remote-sensing images, and improved the highest-resolution images from 10 M to 2.5 M resolution. Through the combination of the degradation kernel and injected noise, we obtained LR images in the same domain with real images, and obtained the near-natural LR–HR image pairs. On the basis of near-natural LR–HR image pairs, we used a GAN, combined with an ESRGAN-type generator, a PatchGAN-type discriminator and a VGG-19-type feature extractor, used the perceptual loss, and focused on the visual characteristics of the images, so that our results have clearer details and better perceptual effects. Compared with the SR models of Sentinel-2, such as EDSR8-RGB, RCAN, RS-ESRGAN and RealSR, the main difference in our model lies in the construction of LR–HR image pairs for the training datasets. In the scene of training with natural LR–HR image pairs, there was no significant difference in the effect for SR images obtained with those models; however, in the scene with only LR images and no HR prior information, compared with RCAN, which constructs the image pairs through BiCubic and EDSR8-RGB and RS-ESRGAN, which use WorldView satellite HR images to construct the image pairs, TS-SRGAN has obvious advantages in the quantitative comparison of the non-reference image quality assessment and the intuitive visual effects.