Learning the Frequency Domain Aliasing for Real-World Super-Resolution

Abstract

Most real-world super-resolution methods require synthetic image pairs for training. However, the frequency domain gap between synthetic images and real-world images leads to artifacts and blurred reconstructions. This work points out that the main reason for the frequency domain gap is that aliasing exists in real-world images, but the degradation model used to generate synthetic images ignores the impact of aliasing on images. Therefore, a method is proposed in this work to assess aliasing in images undergoing unknown degradation by measuring the distance to their alias-free counterparts. Leveraging this assessment, a domain-translation framework is introduced to learn degradation from high-resolution to low-resolution images. The proposed framework employs a frequency-domain branch and loss function to generate synthetic images with aliasing features. Experiments validate that the proposed domain-translation framework enhances the visual quality and quantitative results compared to existing super-resolution models across diverse real-world image benchmarks. In summary, this work offers a practical solution to the real-world super-resolution problem by minimizing the frequency domain gap between synthetic and real-world images.

1. Introduction

Single-image super-resolution (SR) aims to enhance the quality of low-resolution images by reconstructing them into high-resolution counterparts. Learning-based methods [1,2,3,4,5,6,7,8,9,10], such as SRCNN [1], VDSR [2], LapSRN [3], RCAN [4], SRGAN [5], and ESRGAN [6], have made significant advancements in achieving impressive results. Typically, these methods require pairs of high-resolution (HR) and low-resolution (LR) images for training. However, acquiring such HR–LR image pairs for real-world scenarios proves challenging. Even if obtained through methods like optical zoom [11,12,13,14,15,16], they fail to address issues stemming from differences in depth of field, illumination, and perspective [17]. These disparities impact the performance of end-to-end learning models and result in blurry SR outcomes [17]. Moreover, these methods are constrained by limited degradation diversity, making it challenging to apply them effectively to various scenarios with differing degradation distributions. As a result, their adoption remains limited, primarily due to issues related to misalignment and a lack of adaptability.

Considering that there are currently many widely used HR datasets [18,19,20,21,22], a practical approach for real-world SR involves creating LR images from HR images by using a degradation model. This process leads to the formation of HR–LR image pairs, which can serve as the training dataset. Some methods [1,2,3,4,5,6] use HR images and LR images generated by blurring and downsampling to train SR models. When real-world images are taken as input, SR results are disturbed by severe artifacts and noise. This is due to the mismatch between the degradation model and the real-world degradation process. Some recent methods extend the classical degradation model through random strategies, or recombine steps such as blur, downsampling, and noise [23,24,25,26]. These methods greatly expand the range of degradation distributions covered and are able to deal with more complex degradation scenarios. However, at the cost of improved generalization capabilities, these methods lack detail-recovery capabilities and tend to produce over-smoothed results. Furthermore, these predefined, constrained degradation steps often prove inadequate for covering the complexity of degradation in real-world images.

Since the degradation processes in the real world are complex and diverse, domain-translation-based methods are proposed to adaptively learn degradation models for images with different degradation distributions [27]. The reference HR image is considered to be from the source domain, while the real-world LR image is considered to be from the target domain. The degradation model is considered to be the domain-translation process from the source domain to the target domain. Since the image contents in the reference HR and real-world LR are not directly consistent, the translation process from the source domain to the target domain is learned under an adversarial training framework [28] to generate synthetic LR images that conform to the degradation distribution of the target domain. Then, image pairs consisting of HR and synthetic LR images can be used to train the SR model in a supervised learning manner. KernelGAN [29] first generates LR images from HR using a deep linear generator. Bulat et al. [30] used a generator with an encoder–decoder structure for the domain-translation task. The CycleGAN [31] architecture for image-to-image translation is also used to learn domain translation from HR to LR images, as well as LR to HR images [32,33,34]. In this framework, the degradation process and the SR model are jointly trained, and cycle consistency loss ensures that the content of the image does not change during the degradation and SR processes. To facilitate integration with more advanced SR methods, more methods train the degradation and SR processes separately [17,30,35,36,37,38,39,40]. Zhou et al. [37] proposed a color-guided domain-mapping network to alleviate color shift in domain-translation processes. Luo et al. [38] proposed a probabilistic degradation model, which studies the degradation process by modeling it as a random variable and covers more diverse degradation distributions. Son et al. [40] proposed an adaptive data loss to allow the downsampler to adaptively learn the degradation process of real-world images. These learning-based methods can learn the degradation process of real-world images to generate LR images with smaller gaps from the real-world domain, thereby training SR models that can restore better image reconstruction quality.

However, these methods are all based on classical degradation models, focusing on establishing accurate spatial blur kernels and noise, without paying attention to the frequency-domain gap between synthetic LR images and real-world LR images. The frequency distributions between the synthetic LR image and the real-world LR image show obvious differences [27]. Ji et al. [39] focused on the frequency-domain features of synthetic images and used a frequency-consistent adaptation to construct images that were consistent with real-world images. However, when the image contents are inconsistent, it is inappropriate for the frequenc- consistent adaptation to directly compare their frequency spectral densities. Furthermore, this frequency domain consistency adaptation method does not point out the reason for the frequency-domain gap between synthetic images and real-world images, and adopts an indiscriminate comparison of densities at all frequencies.

This work points out that the main reason for the frequency-domain gap between synthetic LR images and real-world LR images is the aliasing phenomenon, which is widely present in real-world images [41,42,43]. However, existing degradation models [17,25,26,29,30,35,36,37] ignore the impact of aliasing on images [39,44]. Aliasing is an inherent property of sampling equipment [45,46,47]. Aliasing occurs when the spatial frequency of the scene surpasses the Nyquist limit defined by the imaging device’s resolution [48,49]. Therefore, aliasing widely exists in images captured by acquisition devices such as cameras and mobile phones in real scenes [41,42,43]. Aliasing reduction [50,51,52] has always been a classical problem in the fields of signal processing and image processing. However, in the field of real-world image super resolution, few works [39] focus on the impact of aliasing on images.

Therefore, this paper proposes a method to evaluate aliasing in LR images that have undergone unknown degradation. The method involves generating an alias-free copy of the LR image to be evaluated and then computing the $L1$ distance between this copy and the LR image. The greater the distance calculated, the more serious the aliasing. Compared with the previously discussed frequency-consistent adaptation method, this approach provides a more intuitive and accurate means of quantifying the degree of aliasing. The consistency of image contents between the copy and the LR image enhances the precision of the results.

On the basis of this measurement method, this work proposes a domain-translation-based method that generates frequency features similar to real-world images. On the one hand, a frequency-domain branch extracts the aliasing features in the degradation process. On the other hand, a loss function is proposed to guide the generation of synthetic images with aliasing features. Both the domain-translation generator and the frequency-domain loss function reduce the frequency-domain gap between synthetic LR images and real-world LR images, so the proposed method can help the SR model reconstruct more realistic image details.

The main contributions of this work are as follows:

• This work points out the frequency-domain gap between the synthetic and real-world image. A method is proposed to measure the degree of frequency-domain aliasing in images that undergo unknown degradation;
• A domain-translation framework is proposed to generate frequency-domain features that are similar to real-world images, including a branch to extract aliasing features and a loss function related to the degree of aliasing;
• The proposed domain-translation framework is proven to help the SR model achieve better reconstruction quality on real-world images.

2. Related Work

2.1. Image Pair-Based Methods

The challenge in real-world SR primarily stems from the shortage of real-world HR–LR image pairs in most scenarios [27]. To address this issue, various approaches have been proposed [11,12,13,14,15,16]. These methods involve capturing images of the same scene at different resolutions, effectively creating datasets containing HR and LR images derived directly from real-world sources. Some methods [11,12,13,14] involve using zoom lenses to acquire HR and LR images with long and short focal lengths, respectively. SR models trained with these datasets have the capacity to learn the degradation distribution present in real-world scenarios, resulting in improved performance on LR images characterized by the same degradation distribution. Nonetheless, challenges persist due to disparities in depth of field, lighting conditions, and perspective, leading to inevitable misalignment between HR and LR images [17]. These alignment discrepancies significantly impact the efficacy of SR models. Moreover, existing datasets often lack diversity in terms of scenes and degradation distributions, making it challenging to generalize to real-world images that exhibit varying degradation distributions.

To address complex and diverse real-world SR challenges, degradation models that are consistent with real-world degradation processes and have generalization capabilities are necessary. Alternatively, domain-translation methods for unpaired HR and LR images offer a viable solution.

2.2. Degradation Modeling-Based Methods

In many cases, only reference HR images and real-world LR images are available due to the absence of HR–LR image pairs. There is no direct content correspondence between reference HR and real-world LR images. To reconstruct SR images using reconstruction-based SR methods, it is necessary to determine the parameter set for the degradation model between the HR and LR images. Efrat et al. [53] proposed that an accurate degradation model is more important than using advanced image priors. Classical degradation models typically involve steps like blurring, downsampling, and noise. Most current research efforts [54,55,56,57,58,59,60,61] concentrate on the precise recovery of blur kernels. Several approaches [57,58,59,60,61] optimize degradation parameters and intermediate super-resolved images through iterative prediction and correction techniques. However, when compared to the vast and diverse degradation space in the real world, these methods still exhibit limitations in terms of diversity and face the risk of failure in realistic scenarios.

For learning-based SR methods, it is crucial to determine the degradation model that transforms reference HR images into their corresponding LR counterparts, in order to train the SR model. One intuitive approach [62] involves synthesizing various LR images with multiple degradation distributions to train the SR model. This strategy aims to enhance the model’s generalization performance. Several methods seek to expand upon the classical degradation model by introducing new degradation frameworks [23,24,25,26]. For instance, Zhou et al. [23] employed Generative Adversarial Network [28] (GAN) to construct a blur kernel pool and used these kernels in conjunction with HR images to generate LR images. Ji et al. [24] proposed RealSR, a degradation framework that employed blind kernel estimation to construct blur kernels and noise pools. The degradation process involves random sampling of blur kernels and noise from these pools to degrade HR images. Zhang et al. [25] used a random shuffled strategy to extend the classic degradation model and proposed a practical degradation model BSRGAN. Wang et al. [26] proposed Real-ESRGAN, a high-order degradation model that simulates ringing artifacts through a sinc filter.

Nevertheless, in their pursuit of improved generalization, these methods often produce over-smoothed results and require extensive datasets and training resources.

2.3. Domain-Translation-Based Methods

Consider the reference HR image and the real-world LR image as residing in different domains: HR images in the source domain and LR images in the target domain. In this context, both the degradation process and the SR process can be treated as domain-translation challenges. One approach is to leverage the well-established CycleGAN architecture [31], typically used in style transfer. This strategy [32,33,34] involves the joint training of the degradation and SR stages. Cycle-consistency loss enforces content correspondence within the cycle, while style consistency between different domains is often assessed through adversarial loss. These methods possess the ability to adaptively learn the degradation and SR processes for images with varying distributions. These methods can achieve more accurate recovery of image details compared to degradation models based on generalization ability. However, these methods exhibit limited flexibility, making it challenging to integrate with advanced SR methods. Some methods [17,35] leverage the cycle structure solely within the degradation process, opting to train the degradation and SR processes separately. But, unlike the style transfer problem, which usually assumes a one-to-one correspondence between elements in domains, SR poses an ill-posed challenge. In the field of SR, a single LR image corresponds to countless HR images. The deterministic degradation relationship provided by the cycle structure struggles to account for the random variables in real-world degradation.

Another approach [29,30,36,37,38] is to directly employ GAN [28] for unsupervised learning of domain translation within the degradation process, resulting in the generation of HR–LR image pairs. The subsequent SR process is executed by deep convolution neural network (CNN) based SR methods through supervised learning. In contrast to the explicit modeling of degradation found in degradation modeling-based methods, this strategy constitutes implicit modeling of degradation. Fritsche et al. [36] proposed DSGAN to learn the domain-translation process from HR images to unpaired LR images in an unsupervised manner. DSGAN trains two models for different datasets, including SDSR [36] and TDSR [36]. Zhou et al. [37] proposed a color-guided domain-mapping network to alleviate color shift during domain translation, named CARB [37]. Luo et al. [38] proposed a probabilistic degradation model (PDM), which studies the degradation process as a random variable and models the mapping from a prior random variable to the degradation process. Son et al. [40] simulated the distribution of LR images in the target domain by generalizing low-frequency loss and designed an adaptive data loss (ADL). These methods can cover a wider range of unknown degradation distributions and is more robust when dealing with degradations that lack definition or are affected by random factors. Compared with methods based on the CycleGAN [31] framework, this strategy of separately modeling degradation can be easily integrated with advanced SR methods. With no cycle-consistency constraint, these methods rely on content consistency measured through the distance between low-frequency components of HR images and their corresponding LR images. Style consistency is still represented by adversarial loss.

However, the domain gap exists in the frequency domain between synthetic LR images and real-world LR images, which is overlooked by the existing methods [39]. Real-world images exhibit frequency-domain aliasing phenomena, while synthetic images tend to reduce aliasing features due to the low-pass filtering [40]. When applied to real-world images, SR models trained on such synthetic images often mistake aliasing phenomena as image content and amplify these features [44].

3. Method

3.1. Classical Degradation Model

Generally, LR image ${\mathit{I}}_{LR}$ can be obtained from HR image ${\mathit{I}}_{HR}$ by the classical degradation model [63,64]:

$${\mathit{I}}_{LR}=({\mathit{I}}_{HR}\otimes \mathit{k}){\downarrow }_s+\mathit{n}$$

(1)

where $\mathit{k}$ denotes a degradation kernel, ⊗ is a spatial convolution, ${\downarrow }_s$ indicates downsampling and decimation by a scale factor s, and $\mathit{n}$ is a noise term. The order of these steps is not fixed; the classical degradation model can also be expressed as

$${\mathit{I}}_{LR}=\left({\mathit{I}}_{HR}{\downarrow }_s\right)\otimes \mathit{k}+\mathit{n}$$

(2)

The classical degradation model is widely used to synthesize HR–LR image pairs by degradation modeling-based methods. Furthermore, the classical degradation model is also combined with the probabilistic model in domain-translation methods [38].

3.2. Frequency-Domain Aliasing

There is a frequency-domain gap between real-world LR images and LR images synthesized by Equation (1) or Equation (2). Image downsampling process ${\downarrow }_s$ can be understood as sampling and decimation, which causes aliasing. Figure 1 shows the impact of several degradation processes on signals in the frequency domain. Figure 1a represents direct downsampling. Figure 1b and Figure 1c represent the degradation processes of Equation (1) and Equation (2), respectively. The influence of noise is not considered. Among them, ${\omega }_M$ is the highest inherent frequency of the image ${\mathit{I}}_{HR}$, ${\omega }_S$ is the sampling frequency, and ${\omega }_K$ is the cutoff frequency of the low-pass filter $\mathit{k}$. Compared with direct downsampling, the degradation processes represented by Equations (1) and (2) limit the frequency of ${\mathit{I}}_{LR}$ within low-pass windows, reducing aliasing. Assume that ${\mathit{I}}_{HR}$ generates ${\mathit{I}}_{LR}$ through an unknown degradation process ${\mathcal{D}}^{\times s}(·)$, and the scale factor between ${\mathit{I}}_{HR}$ and ${\mathit{I}}_{LR}$ is s, which can be expressed as

$${\mathit{I}}_{LR}={\mathcal{D}}^{\times s}\left({\mathit{I}}_{HR}\right)$$

(3)

Figure 2 selects three sets of image samples and shows the frequency-domain images of ${\mathit{I}}_{HR}$ and ${\mathit{I}}_{LR}$ after the above-mentioned degradations. The direct downsampling is denoted as ${\mathcal{D}}_B^{\times s}(·)$. And the degradation process in Equations (1) and (2) are expressed as ${\mathcal{D}}_1^{\times s}(·)$ and ${\mathcal{D}}_2^{\times s}(·)$, respectively ($\mathit{k}$ using Gaussian low-pass filter). The samples come from the real-world dataset City100 [11], so an LR image that has experienced the real-world degradation process ${\mathcal{D}}_R^{\times s}(·)$ could be used as a reference. The first row of each group is the spatial-domain image. The second row of each group is the corresponding frequency-domain image. Consistent results with the previous analysis can be obtained from the samples. Compared with ${\mathcal{D}}_B^{\times s}(·)$, ${\mathcal{D}}_1^{\times s}(·)$ and ${\mathcal{D}}_2^{\times s}(·)$ limit the frequency-domain information to low-pass windows. Real-world LR images ${\mathcal{D}}_R^{\times s}\left({\mathit{I}}_{HR}\right)$ exhibit frequency-domain characteristics that are more like ${\mathcal{D}}_B^{\times s}\left({\mathit{I}}_{HR}\right)$, rather than ${\mathcal{D}}_1^{\times s}\left({\mathit{I}}_{HR}\right)$ or ${\mathcal{D}}_2^{\times s}\left({\mathit{I}}_{HR}\right)$. In the spatial domain, ${\mathcal{D}}_R^{\times s}\left({\mathit{I}}_{HR}\right)$ is not as smooth as ${\mathcal{D}}_1^{\times s}\left({\mathit{I}}_{HR}\right)$ and ${\mathcal{D}}_1^{\times s}\left({\mathit{I}}_{HR}\right)$, but have jagged edges and background stripes, similar to ${\mathcal{D}}_B^{\times s}\left({\mathit{I}}_{HR}\right)$. These phenomena indicate that the classical degradation model cannot simulate the aliasing phenomenon existing in real-world images. The step-by-step degradation models do not accurately represent complex degradation processes in the real world.

In order to more accurately describe the aliasing features of these LR images generated through different degradation processes, this work proposes a method to evaluate aliasing in LR images. Interpolate ${\mathit{I}}_{LR}$ in Equation (3) to the size of ${\mathit{I}}_{HR}$, expressed as

$${\mathit{I}}_{LR,up}=\left({\mathcal{D}}^{\times s}\left({\mathit{I}}_{HR}\right)\right){\uparrow }_s$$

(4)

where ${\uparrow }_s$ indicates upsampling by a scale factor s. In order to measure the degree of aliasing caused by the degradation process ${\mathcal{D}}^{\times s}(·)$ on ${\mathit{I}}_{HR}$, assume that there is an alias-free copy of ${\mathit{I}}_{LR}$, which is the solution to the following optimization problem:

$${\widehat{\mathit{I}}}_F=\underset{{\mathit{I}}_F}{\mathrm{argmin}}\parallel {\mathit{I}}_F-{\mathit{I}}_{LR,up}\parallel +\lambda \mathcal{H}\left({\mathit{I}}_F\right)$$

(5)

where the first item represents that the contents of ${\mathit{I}}_F$ and ${\mathit{I}}_{LR,up}$ are consistent. $\mathcal{H}(·)$ is a constraint that ensures that there is no aliasing in ${\mathit{I}}_F$. $\lambda$ is the regularization constant. As shown in the last row of Figure 1, ${\mathit{I}}_F$ can be obtained from ${\mathit{I}}_{HR}$ through a low-pass filter $\mathit{k}$ in the absence of aliasing. Therefore, $\mathcal{H}(·)$ can be defined as

$$\mathcal{H}\left({\mathit{I}}_F\right)=\mathcal{C}({\mathit{I}}_F,\mathit{k})=\parallel {\mathit{I}}_{HR}\otimes \mathit{k}-{\mathit{I}}_F\parallel +\mu \mathcal{G}\left(\mathit{k}\right)$$

(6)

where $\mathcal{C}(·)$ is the non-aliasing constraint on ${\mathit{I}}_F$ that introduces variable $\mathit{k}$, and $\mathcal{G}(·)$ is the prior constraint of the filter $\mathit{k}$, usually using regularization term related to gradient. $\mu$ is the regularization constant. Equation (5) is rewritten as

$$({\widehat{\mathit{I}}}_F,\widehat{\mathit{k}})=\underset{{\mathit{I}}_F,\mathit{k}}{\mathrm{argmin}}\parallel {\mathit{I}}_F-{\mathit{I}}_{LR,up}\parallel +\lambda \parallel {\mathit{I}}_{HR}\otimes \mathit{k}-{\mathit{I}}_F\parallel +\mu \mathcal{G}\left(\mathit{k}\right)$$

(7)

When a signal is aliased, its high-frequency components additively spill into the low-frequency components [65]. Treat aliasing as an additive noise [66,67,68] and denote it as $\mathit{\eta }$; then,

$${\mathit{I}}_{LR,up}={\mathit{I}}_F+\mathit{\eta }$$

(8)

Then, $\mathit{k}$ can be obtained by solving

$$\widehat{\mathit{k}}=\underset{\mathit{k}}{\mathrm{argmin}}\parallel {\mathit{I}}_{HR}\otimes \mathit{k}-{\mathit{I}}_{LR,up}+\mathit{\eta }\parallel +\mu \mathcal{G}\left(\mathit{k}\right)$$

(9)

During the downsampling process, aliasing mostly occurs near areas containing high-frequency information, that is, areas where gradient changes are dramatic. $\eta$ has less impact on areas where gradient changes are relatively gentle. Divide the image ${\mathit{I}}_{LR,up}$ and ${\mathit{I}}_{HR}$ into N sample blocks. The i-th sample is denoted as ${\mathit{I}}_{LR,up}^i$ and ${\mathit{I}}_{HR}^i$, and ${\mathit{x}}^i$ is the center pixel of this window. The metric to measure the usefulness of gradients in each block [69] is defined as

$$r\left({\mathit{x}}^i\right)=\frac{\parallel {\sum }_{\mathit{y}\in {\mathit{I}}_{HR}^i}\nabla {\mathit{I}}_{HR}\left(\mathit{y}\right)\parallel }{{\sum }_{\mathit{y}\in {\mathit{I}}_{HR}^i}\parallel \nabla {\mathit{I}}_{HR}\left(\mathit{y}\right)\parallel +\beta }$$

(10)

where ∇ is the operator of first-order spatial derivatives and $\beta$ is a constant coefficient. A small r implies that a flat region is involved, which causes neutralization of many gradient components. A large r implies existing strong image structures in the local region [69]. Next, define a weight w that is negatively related to r:

$$w\left({\mathit{x}}^i\right)=\exp (-\parallel r\left({\mathit{x}}^i\right){\parallel }^{\alpha })$$

(11)

where $\alpha$ is a constant coefficient. Use w as the weight to find the average blur kernel of N blocks. Since $\mathit{\eta }$ of the flat area is very small and has different directions, it will be ignored by the averaging process. The weight w of such a block is set large. The $\mathit{\eta }$ of blocks containing high-frequency areas will interfere with the solution of the blur kernel as noise. Thus, the weight w of such blocks is set small. The average blur kernel can be approximated as

$$\overline{\mathit{k}}=\underset{\mathit{k}}{\mathrm{argmin}}\sum _{i=1}^{N}w\left({\mathit{x}}^i\right)\parallel {\mathit{I}}_{HR}^i\otimes \mathit{k}-{\mathit{I}}_{LR,up}^i\parallel +\mu \mathcal{G}\left(\mathit{k}\right)$$

(12)

In the case of $\lambda >1$, ${\widehat{\mathit{I}}}_F$ can be obtained from Equation (7) as

$${\widehat{\mathit{I}}}_F={\mathit{I}}_{HR}\otimes \overline{\mathit{k}}$$

(13)

$\mathit{\eta }$ can be obtained from Equation (8) as

$$\mathit{\eta }={\mathit{I}}_{LR,up}-{\widehat{\mathit{I}}}_F$$

(14)

This work proposes to use the L1 distance ${\parallel \mathit{\eta }\parallel }_1$ to quantify the degree of aliasing of ${\mathit{I}}_{LR}$ obtained by ${\mathit{I}}_{HR}$ through unknown degradation ${\mathcal{D}}^{\times s}(·)$. Figure 2 shows the visualization of the result ${\mathit{I}}_{LR}$ and $\mathit{\eta }$ produced by different degradation processes. There are three sets of samples. The first row of each set is ${\mathit{I}}_{HR}$ and ${\mathit{I}}_{LR}$ generated by the corresponding degradation processes. The second row is the corresponding images in the frequency domain, and the third row is the visualized images of $\mathit{\eta }$ calculated by the proposed method. The corresponding L1 distance ${\parallel \mathit{\eta }\parallel }_1$ is marked below the visualization of $\mathit{\eta }$. The visualization and the L1 distance of $\mathit{\eta }$ both show that the degree of aliasing caused by the bicubic downsampling degradation ${\mathcal{D}}_B^{\times s}(·)$ is similar to the real-world degradation ${\mathcal{D}}_R^{\times s}(·)$. In contrast, the currently commonly used degradation models ${\mathcal{D}}_1^{\times s}(·)$ and ${\mathcal{D}}_2^{\times s}(·)$ have obvious gaps with the real-world degradation model ${\mathcal{D}}_R^{\times s}(·)$.

3.3. Downsampling with Domain Translation

To reduce the frequency-domain gap between synthetic and real-world images, this work proposes a domain-translation-based downsampling framework to generate aliasing features through the frequency-domain branch. The framework is based on the probabilistic degradation model [38]. Since the step-by-step degradation process cannot simulate real-world aliasing, this paper proposes to separate the aliasing information from the reference image through the frequency-domain branch, and then compensate the image before downsampling to generate LR images with aliasing. As shown in Figure 3, the degradation process of the framework involves three steps:

$$\left\{\begin{array}{c}{\mathit{I}}_F={\mathit{I}}_{HR}\otimes \mathit{k}\hfill \\ {\mathit{I}}_C=\mathrm{netD}({\mathrm{netE}}_1\left({\mathit{I}}_F\right)+{\mathrm{netE}}_2\left(\mathit{\eta }\right)){\downarrow }_s\hfill \\ {\mathit{I}}_{LR}={\mathit{I}}_C+\mathit{n}\hfill \end{array}\right.$$

(15)

where $\mathit{k}$ is the blur kernel generated by the Kernel Generator, $\mathit{n}$ is the spatial-domain noise generated by the Noise Generator, netE₁ and netE₂ are two encoders, and netD is a decoder. $\mathit{\eta }$ is the aliasing feature produced by the frequency-domain branch. Compensate ${\mathit{I}}_F$ with $\mathit{\eta }$ before downsampling.

The main function of the frequency-domain branch is to generate the aliasing feature $\mathit{\eta }$. Compared with the method of obtaining $\mathit{\eta }$ mentioned in Section 3.2, a more intuitive method is to directly use a frequency-domain filter ${\mathit{K}}_H$ to separate this part of aliasing information from the aliased image:

$$\mathit{\eta }=\mathrm{IFFT}\{{\mathit{K}}_H\ast \mathrm{FFT}\left\{\left({\mathcal{D}}_B^{\times s}\left({\mathit{I}}_{HR}\right)\right){\uparrow }_s\right\}\}$$

(16)

where the image undergoes the degradation process ${\mathcal{D}}_B^{\times s}(·)$ and is used as a reference image to provide aliasing information.

To model the distribution of the blur kernel ${\mathit{K}}_H$, a prior random variable ${\mathit{z}}_m$ is defined, and a generator netM is used to learn the mapping from ${\mathit{z}}_m$ to the frequency domain filter:

$${\mathit{K}}_H=\mathrm{netM}\left({\mathit{z}}_m\right){\mathit{z}}_m\sim \mathcal{N}(\mathbf{0},\mathbf{1})$$

(17)

If the height, width, and number of channels of ${\mathit{I}}_{HR}$ are h, w, and c, then

$${\mathit{z}}_m\in {\mathbb{R}}^{f_m\times h\times w}{\mathit{K}}_H\in {\mathbb{R}}^{h\times w\times c}$$

(18)

where $f_m$ is the dimension of the normal distribution ${\mathit{z}}_m$.

Finally, the aliasing information $\mathit{\eta }$ is integrated into ${\mathit{I}}_F$ for compensation before downsampling. In this way, compared to classical degradation processes, ${\mathit{I}}_{LR}$ generated by this proposed framework contains aliasing information and has a smaller domain gap compared to real-world images.

3.4. Frequency-Domain Loss

In this work, a frequency-domain loss is proposed to guide the domain-translation network to generate frequency-domain features that are consistent with real-world images. In order to guide the frequency-domain branch to generate $\mathit{\eta }$ that conforms to the aliasing distribution, ${\mathit{\eta }}_B$ that corresponds to ${\mathcal{D}}_B^{\times s}(·)$ degradation is used as prior knowledge. The optimization of netM is formulated as

$$\underset{\mathrm{netM}}{\mathrm{argmin}}{\parallel \mathrm{IFFT}\{\mathrm{netM}\left({\mathit{z}}_m\right)\ast \mathrm{FFT}\left\{\left({\mathcal{D}}_B^{\times s}\left({\mathit{I}}_{HR}\right)\right){\uparrow }_s\right\}\}-\gamma {\mathit{\eta }}_B\parallel }_1$$

(19)

where $\gamma$ is a constant coefficient. Equation (19) can be simply written as

$$\underset{\mathit{\eta }}{\mathrm{argmin}}{\parallel \mathit{\eta }-\gamma {\mathit{\eta }}_B\parallel }_1$$

(20)

In order to ensure that the generated image ${\mathit{I}}_{LR}$ contains aliasing features, the aliasing degree $\parallel {\mathit{\eta }}_{LR}{\parallel }_1$ related to ${\mathit{I}}_{LR}$ is measured through the method proposed in Section 3.2. ${\mathit{\eta }}_{B,down}$ is a downsampled version of ${\mathit{\eta }}_B$, which serves as a prior. In this way, the overall frequency-domain loss function is defined as

$${\mathcal{L}}_{fre}=\parallel \mathit{\eta }-\gamma {\mathit{\eta }}_B{\parallel }_1+\upsilon \parallel \parallel {\mathit{\eta }}_{B,down}{\parallel }_1-\tau \parallel {\mathit{\eta }}_{LR}{\parallel }_1\parallel$$

(21)

where $\upsilon$ and $\tau$ are weights for the regularizer term. The entire frequency-domain loss function consists of two parts. The first part uses ${\mathit{\eta }}_B$ to guide the generator to generate $\mathit{\eta }$ that conforms to the aliasing distribution. The second part uses the downsampled version ${\mathit{\eta }}_{B,down}$ to constrain the final generated ${\mathit{I}}_{LR}$ to satisfy the aliasing degree.

Since the degradation process represented by ${\mathcal{D}}_B^{\times s}(·)$ is deterministic, ${\overline{\mathit{k}}}_B$ of the processes ${\mathcal{D}}_B^{\times s}(·)$ is independent of image and is only related to the downsampling factor s. As such, replace ${\overline{\mathit{k}}}_B$ with fixed blur kernel calculated through a large number of samples in advance. This approach avoids recalculating ${\overline{\mathit{k}}}_B$ every time the loss is calculated.

3.5. Overall Loss

Synthetic image ${\mathit{I}}_{LR}$ needs to meet the following two requirements: the image content is consistent with ${\mathit{I}}_{HR}$ and the image distribution is consistent with the target domain images. In order to preserve the image content across different scales, the data loss ${\mathcal{L}}_{data}$ is defined to measure the distance between the low-frequency components of ${\mathit{I}}_{HR}$ and the synthetic image ${\mathit{I}}_{LR}$:

$${\mathcal{L}}_{data}={\parallel {\mathcal{P}}^{\times s}\left({\mathit{I}}_{HR}\right)-\mathcal{P}\left({\mathit{I}}_{LR}\right)\parallel }_1$$

(22)

where $\mathcal{P}(·)$ is a low-pass filtering and ${\mathcal{P}}^{\times s}(·)$ is a combination of low-pass filtering and downsampling by s. The adversarial loss ${\mathcal{L}}_{adv}$ is used to enforce ${\mathit{I}}_{LR}$ to follow the distribution of the target domain images:

$$\left\{\begin{array}{c}{\mathcal{L}}_{adv}^g=-\mathbb{E}\left[\log \left(\mathcal{F}\left({\mathit{I}}_{LR}\right)\right)\right]\hfill \\ {\mathcal{L}}_{adv}^d=-\mathbb{E}\left[\log \left(\mathcal{F}\left({\mathit{I}}_{LR}^{real}\right)\right)\right]+\mathbb{E}\left[\log (1-\mathcal{F}\left({\mathit{I}}_{LR}\right))\right]\hfill \end{array}\right.$$

(23)

where ${\mathcal{L}}_{adv}^g$ and ${\mathcal{L}}_{adv}^d$ are the adversarial losses of the generator and discriminator, respectively. $\mathcal{F}(·)$ is the discriminator and ${\mathit{I}}_{LR}^{real}$ is the real-world image of the target domain.

The total loss function of the degradation model is

$$\left\{\begin{array}{c}{\mathcal{L}}^g={\mathcal{L}}_{data}+{\mathcal{L}}_{adv}^g+{\mathcal{L}}_{fre}\hfill \\ {\mathcal{L}}^d={\mathcal{L}}_{adv}^d\hfill \end{array}\right.$$

(24)

4. Experiments

4.1. Datasets and Training Details

Datasets. In order to test the effect of the proposed method on real-world images, 400 images were selected from the HR dataset DIV2K [18] as the source domain HR images. Real-world LR datasets or datasets with unknown and complex degradation were selected as target domain images, including DRealSR [12] and NTIRE2020 [70] Track1 and Track2. The source domain HR image and the target domain LR image were unpaired, which means that their contents were inconsistent. A portion of the target domain LR images were selected for training, and another portion of the target domain images were chosen for testing and validation. The proposed domain-translation-based downsampling method was used to learn the translation from the source domain to the target domain, and the loss function proposed in Equation (24) was used.

Evaluation metrics. For validation sets with ground-truth references, such as NTIRE2020 Track1, we used PSNR, SSIM [71], and LPIPS [72] as evaluation metrics. For the validation set without ground-truth references, we used NIQE [73], NRQM [74], and PI [75] as evaluation metrics.

Implementation and training details. During training, ${\mathit{z}}_m$ was set as $f_m=128$. The HR images were cropped into $128\times 128$, and the LR images were cropped into $64\times 64$ for scale factor 2 and $32\times 32$ for scale factor 4. In comparative experiments and ablation experiments, the network was trained for 40 epochs with a learning rate of $1\times {10}^{-4}$. The batch size was set as 24.

4.2. Comparison with Other Domain-Translation Based Methods

The proposed method is compared with several state-of-the-art real-world SR methods based on domain translation, including SDSR [36], TDSR [36], CARB [37], PDM [38], and ADL [40]. For fair comparison, the SR model used by the proposed method is consistent with the SR model used by these methods, including EDSR [76] and ESRGAN [6]. EDSR is PSNR-oriented and supervised by L1 and L2 losses. ESRGAN is perceptual-oriented and supervised by perceptual loss. Therefore, when EDSR is the SR model, PSNR and SSIM are used as the main evaluation metrics. When ESRGAN is the SR model, LPIPS (based on perceptual similarity) is used as the main evaluation metric.

The results on NTIRE2020 Track 1 [70] are presented in

Table 1. Quantitative comparison with domain-translation-based methods on NTIRE2020 Track 1 with ground-truth references. ↑ denotes the larger the better. ↓ denotes the smaller the better. The best results are denoted in red, and the second best are denoted in blue.

Methods	PSNR ↑	SSIM↑	LPIPS↓
PDM + EDSR	21.099	0.6044	0.3794
ADL + EDSR	28.942	0.8004	0.3248
SDSR + ESRGAN	23.096	0.4479	0.5619
TDSR + ESRGAN	21.949	0.3901	0.6024
CARB + ESRGAN	28.483	0.7968	0.3285
ADL + ESRGAN	24.688	0.6437	0.3063
Ours + EDSR	29.366	0.8033	0.3107
Ours + ESRGAN	25.661	0.7636	0.2930

. When combined with the PSNR-oriented network EDSR, the proposed method in this work attains the best performance in terms of PSNR and SSIM. Similarly, when combined with the perceptual-oriented network ESRGAN, it achieves the best LPIPS results, indicating superior perceptual quality. This underscores that, regardless of being PSNR-oriented or focused on perceptual quality, incorporating the method in this work leads to the best results. The LR images in NTIRE2020 Track 1 are obtained by an undisclosed degradation operator on the HR dataset Flickr2k [19]. This degradation operator [70] generates structured artifacts produced by the kind of image-processing pipelines found on very low-end devices. Figure 4 provides a visual comparison of three sets of samples. The SR model in the first row of each group of samples is EDSR; both LR images are amplified by bicubic interpolation, and ground-truth HR images are used as references. The SR model in the second row is ESRGAN. In the results with EDSR as the SR model, the method proposed in this work achieved the sharpest edge while maintaining the denoising ability. As a perception-oriented method, ESRGAN obviously has stronger detail-recovery capabilities when used as the SR model, but it also bears the risk of amplifying complex noise. In each set of samples, “SDSR + ESRGAN” and “TDSR + ESRGAN” are plagued by complex noise. “ADL + ESRGAN” over-enlarges some details that do not exist in the ground-truth HR image. “CARB + ESRGAN” is too blurry to take advantage of the perception-oriented methods. “Ours + ESRGAN” is cleaner than other methods, indicating that the proposed method can better model complex noise in the spatial domain and frequency domain.

Table 2. Quantitative comparison with domain-translation-based methods on NTIRE2020 Track 2 without ground-truth references. ↑ denotes the larger the better. ↓ denotes the smaller the better. The best results are denoted in red, and the second best are denoted in blue.

Methods	NIQE↓	NRQM↑	PI↓
SDSR + ESRGAN	6.744	4.630	6.057
TDSR + ESRGAN	4.365	4.985	4.690
CARB + ESRGAN	8.459	2.256	8.101
PDM + ESRGAN	6.714	4.231	6.241
ADL + ESRGAN	5.229	3.352	5.938
Ours + ESRGAN	4.423	5.158	4.632

shows the results of these methods on NTIRE2020 Track 2 [70]. The method proposed in this work still achieves the best results in NRQM and PI. The LR images in NTIRE2020 Track 2 are real-world images taken by an iPhone3 and come from DPED [22]. NTIRE2020 Track 2 contains more unknown and complex real-world degradation processes. Since the methods used for comparison all provide the perceptual-oriented SR model ESRGAN, the method proposed in this work also uses ESRGAN as the SR model. Figure 5 shows the visual comparison of three sets of samples. LR images amplified by bicubic interpolation are used as references. The results of “SDSR”, “TDSR”, and “ADL” are all affected by severe noise, while the results of “CARB” are still blurry. “PDM” has a better ability to remove noise in the spatial domain, but there is still some complex noise related to the image structure, as shown in the first sample. The method proposed in this work obtains the cleanest results, and there is no structural noise like in the results of “PDM”.

4.3. Comparison with Other Degradation Modeling-Based Methods

In order to have an intuitive comparison of the effects of different methods, comparisons with several methods based on degradation modeling are also provided, including RealSR [24], BSRGAN [25], and Real-ESRGAN [26]. These methods are all based on predefined degradation steps and have strong generalization capabilities for real-world LR images. Since these methods are perceptual-oriented and use ESRGAN or improved ESRGAN [26] as SR models, the method proposed in this work is also combined with improved ESRGAN [26] for comparison. The real-world dataset DRealSR [12] is selected as the validation set. The LR images of DRealSR are derived from the real world, and have ground-truth images as references. PSNR, SSIM, and LPIPS are evaluation metrics. Due to the perceptual quality orientation, the results of LPIPS are more important.

Table 3. Quantitative comparison with degradation modeling-based methods on DRealSR with ground-truth references. ↑ denotes the larger the better. ↓ denotes the smaller the better. The best results are denoted in red, and the second best are denoted in blue.

Methods	PSNR↑	SSIM ↑	LPIPS↓
RealSR	23.088	0.7122	0.2438
BSRGAN	28.147	0.8128	0.1824
Real-ESRGAN	26.656	0.8013	0.1875
Ours + ESRGAN	26.799	0.8188	0.1779

shows the results of theses methods on DRealSR. The method proposed in this work achieved the best LPIPS and SSIM results. Figure 6 shows the visual comparison of three samples. LR images amplified by bicubic interpolation and ground-truth HR images are used as references. The results show that, at the expense of generalization ability, “BSRGAN” and “Real-ESRGAN” tend to produce over-smoothed results that ignore texture details, such as the shape of the petals in the first sample. Since the predefined degradation model may not match the real degradation model, there is a gap between the results of “RealSR” and the ground truth. The method proposed in this work can learn the specific degradation process, so it achieves the visual results that are the closest to ground truth.

4.4. Ablation Studies

In order to verify the effect of the proposed frequency-domain branch and frequency-domain loss function, comparative experiments were designed. The probabilistic degradation model without frequency-domain branch and frequency-domain loss function is used as a baseline. On this basis, the effects of the frequency-domain branch and the frequency-domain loss function are independently verified. The results are shown in

Table 4. Ablation studies on frequency-domain branch and frequency-domain loss. The test set is NTIRE2020 Track 1 with ground-truth references. ↑ denotes the larger the better. ↓ denotes the smaller the better. The best results are denoted in red.

Frequency Branch	Frequency Loss	PSNR↑	SSIM ↑	LPIPS↓
		26.397	0.7335	0.3919
✓		26.719	0.7416	0.4005
	✓	28.842	0.7909	0.3242
✓	✓	29.366	0.8033	0.3107

. The effect of having frequency-domain branch or frequency-domain loss function is better than the baseline. When both are available, SR has the best effect.

In order to verify the role of the random variable, ${\mathit{z}}_m$ is fixed to zero in the comparative experiment.

Table 5. Ablation studies on the random variable ${\mathit{z}}_m$. The test set is NTIRE2020 Track 1 with ground-truth references. ↑ denotes the larger the better. ↓ denotes the smaller the better. The best results are denoted in red.

${\mathit{z}}_{\mathit{m}}$	PSNR ↑	SSIM ↑	LPIPS↓
	28.909	0.7996	0.3456
✓	29.366	0.8033	0.3107

shows that, when ${\mathit{z}}_m$ is randomly sampled, a better SR effect is achieved than when ${\mathit{z}}_m$ is fixed to zero. The reason is speculated that ${\mathit{z}}_m$ can simulate the influence of random factors and avoid overfitting.

5. Conclusions

This work proposes a method to evaluate frequency-domain aliasing in images suffering from unknown degradation. This is the first time that the effect of aliasing on images has been considered in a degradation model. Furthermore, a domain-translation framework is proposed that aims to leverage unpaired HR and LR images to generate synthetic datasets for training. This framework combines frequency-domain branch and frequency-domain loss function to generate synthetic images with real-world aliasing characteristics, thereby reducing the domain gap between synthetic images and real-world images. Experimental results demonstrate that the proposed method could improve the visual effects and quantitative results of super-resolution models on real-world images. This illustrates the practicality of the method proposed in this work and marks the possibility of improving super-resolution technology for real-world scenarios. In the future, the proposed method will be extended to estimate degradation models with real-world noise. Another direction is to combine domain-translation methods with reconstruction-based methods, with the goal of accurately estimating the degradation model and reconstructing high-resolution images, even when there is only a single low-resolution image or there are real-world images from different sources.