Multiframe Super-Resolution of Color Images Based on Cross Channel Prior

Abstract

Color images have a wider range of applications than gray images. There are two ways to extend the traditional super-resolution reconstruction method to color images: Super resolution reconstructs each channel of the color image individually; Change the RGB color bands into YCrCb color bands, then super-resolution reconstructs the luminance component and interpolates the chrominance components.These algorithms cannot effectively utilize the property that the edges and textures are similar in the RGB channels, and the results of those methods may lead to color artifacts. Aiming to solve these problems, we propose a new super-resolution method based on cross channel prior. First, a cross channel prior is proposed to describe the similarity of gradient in RGB channels. Then, a new super-resolution method is proposed for color images via combination of the cross channel prior and the traditional super-resolution methods. Finally, the proposed method reconstructs the color channels alternately. The experimental results show that the proposed method could effectively suppress the generation of color artifacts and improve the quality of the reconstructed images.

1. Introduction

Image super-resolution (SR) is an effective image enhancement technology. It uses mathematical methods to increase image resolution without changing the imaging system hardware. It has great advantages in terms of technology and cost, and is widely used in scientific research and engineering [1,2]. Traditional SR methods are mainly based on gray images. Compared with gray images, color images could provide more information and are widely used in digital television, remote sensing, medical imaging, and cultural relic protection and display [3,4,5]. Color image SR is gradually becoming a new direction of SR research.

In 1984, Tsai and Huang [6] proposed the concept of multiframe super-resolution (MFSR) reconstruction, proved the theoretical feasibility of the SR algorithm and successfully applied it to the processing of Landsat satellite images. Kim, Bose and Valenzaule [7,8] improved Tsai’s algorithm by considering image noise, image blurring, and image registration which extended the application of SR algorithm. Rhee and Kang [9] used discrete cosine transform (DCT) instead of discrete Fourier transform (DFT) in SR algorithm, which improved the computational efficiency of the algorithm. After more than 30 years of development, the current SR methods can be divided into two categories: reconstruction-based and learning-based methods.

The learning-based SR technology of a single image has received widespread attention in recent years [10]. By establishing a training set of high-resolution (HR) images corresponding to low-resolution (LR) images, a suitable data dictionary [11], neural network [12], or adversarial network [13] can be trained to achieve the effect of SR reconstruction. This type of methods has good SR effects and high efficiency, but the training process often requires a lot of computing resources, and the reconstruction results are limited to the training set. In contrast, MFSR methods use the sub-pixel displacement between the LR images to recovery the lost high-frequency information, which would improve the true (optical) resolution of the image [14].

The reconstruction-based methods can be divided into two categories: frequency domain methods and spatial domain methods. The frequency domain method uses the aliasing that exists in the LR images to reconstruct the HR image, the principle is straightforward, the computational complexity of the algorithm is low, and it is easy to implement in hardware. Therefore, the frequency methods can only handle the global translational motion and cannot utilize the image prior information. The spatial domain methods mainly include iterative back projection (IBP) [15,16,17], the convex set projection method (POCS) [18,19,20], the maximum a posteriori (MAP) probability estimation method [21,22,23] and other methods. These methods use the sub-pixel information existing between LR images to provide additional information to reconstruct images. The algorithms reconstruct well and are mainly used in scientific research, satellite remote sensing and other fields.

The above-mentioned SR algorithm research mainly concentrates on grayscale images. For color images with a wider range of applications, there is not much targeted research, mainly involving independently applying SR reconstruction technology to the three RGB channels of color images for reconstruction [24,25,26]. Since the independent reconstruction process ignores the correlation between the color channel components of the color image, the SR reconstruction of the color image is prone to artifacts. Some scholars have proposed a new method to extend the single-channel SR reconstruction method to color images based on the characteristics of human eyes being more sensitive to brightness information [27]. Yang et al. [11] converted the color image from the RGB color space to the YCrCb color space, performed SR reconstruction of luminance information and performed simple interpolation processing on chrominance components. Xu et al. [28] improved Yang’s method with a new conversion method and an improved TV regularization method. The new conversion method allows the luminance channel to contain more texture information, and the improved TV regularization method can effectively suppress image artifacts. This type of method combines the intensity change information of each component of the image as a whole for processing, which effectively reduces the amount of calculation while ensuring the basic quality of the image, and at the same time overcomes the problem of ignoring channel correlation in independent calculations. However, separating image brightness and chrominance information will inevitably break the correlation between color channels. Therefore, the reconstruction results of this type of method still face the problem of color artifacts in the reconstruction results.

Starting from the analysis of the correlation between the channels of the color image, combining the imaging model and prior information in the traditional grayscale SR reconstruction algorithm, this paper proposes an effective color image SR reconstruction algorithm. The main contributions of this article are listed below,

• Introduce a new image prior knowledge in multiframe SR algorithm for color images, which can describe the correlation between each channel of the image well.
• Propose a new SR algorithm based on the introduced cross-channel prior, which effectively utilizes the correlation between color channels and improves the reconstructed image quality.
• Design experiments to verify the proposed algorithm and compare it with other algorithms to show the effect.

Through simulation experiments and real data testing, our algorithm can effectively utilize color image channel correlation and the reconstruction results are better than the presented algorithms.

2. Materials and Methods

2.1. Observation Model

A set of LR images can be obtained from an HR image through a set of operations such as rotation, displacement, bluring, down-sampling, and mixed noisy. The mathematical expression of the observation model is

$$y_{ic}=SK{M}_i{x}_c+N_i,i=1,2,\dots ,s,c=r,g,b,,$$

(1)

where $y_{ic}$ and $x_c$ denote the cth band(R,G,or B) of the ith LR image and the cth band of the HR image. S represents the down-sampling matrix, K represents the blurring matrix, $M_i$ represents the warping matrix, and $N_i$ represents the additive noisy.

2.2. Multiframe Super-Resolution Methods for Grayscale Image

The purpose of the MFSR method is to reconstruct the HR image from a set of LR images. It is an ill-conditioned problem based on the observation model and cannot be solved directly. Therefore, a regularization method is used in this problem.

$$\tilde{x}=arg\underset{x}{min}\sum _{i=1}^s{∥y_i-SK{M}_{i}x∥}_2^2+\lambda \Gamma \left(x\right),$$

(2)

where ${∥y_i-SK{M}_{i}x∥}_2^2$ is the data fidelity terms, which measure the correlation between the solution and the data, and $\Gamma \left(x\right)$ is the regularization cost function, which imposes a penalty on the estimated value x to obtain a stable solution. The parameter $\lambda$ is a scalar value that controls the tradeoff between the data fidelity term and the regularization terms.

The selection of the regularization term is often related to the a priori information of the image, and different regularization terms will lead to different reconstruction results. Here, we use the total variation (TV) regularization [29] to test our algorithm. The TV regularization, which is based on the assumption that natural images have small TV norm, is widely employed in image SR methods. The expression of the TV regularization is as follows,

$$\Gamma \left(x\right)={∥\nabla x∥}_{tv}={∥{\Delta }_{h}x∥}_1+{∥{\Delta }_{v}x∥}_1,$$

(3)

where ${\Delta }_h$ and ${\Delta }_v$ respectively represents the horizontal and vertical gradient of the image.

Given the current estimation of the blur kernel K and the motion matrix $M_i$, we can estimate the HR image by Conjugate Gradient (CG) method.

2.3. Cross-Channel Prior

The current MFSR algorithm can reconstruct HR grayscale images; therefore, the main idea of the algorithm in this paper is to enhance the SR results of color images by using cross channel prior.

The color image consists of RGB color channels. The pixel values of the three color channels are different, but the shape information is similar. i.e., the edges of the object are presented in all color channels in the same location the same shape. This corss-channel prior can be mathematically approximated as,

$$\frac{\nabla x_r}{x_r}\approx \frac{\nabla x_g}{x_g}\approx \frac{\nabla x_b}{x_b},$$

(4)

where $x_r$, $x_g$, $x_b$ represent the RGB channels of the image x.

2.4. Proposed Method

By adding the cross-channel prior, our algorithm can be formulated as the following optimization problem,

$${\tilde{x}}_c=\underset{x_c}{min}\sum _{i=1}^s{∥y_{ci}-SK{M}_i{x}_c∥}_2^2+{\lambda }_c{∥\nabla x_c∥}_{tv}+\sum _{l\ne c}^{}{\beta }_{cl}{∥\nabla x_c·x_l-\nabla x_l·x_c∥}_1,$$

(5)

where the first and second terms are the same as the terms of the SR methods applied to the grayscale images, and the third term represents the cross-channel prior. The parameters ${\lambda }_c,{\beta }_{cl}$ with $c,l\in r,g,b$ are weights for the image prior and the cross-channel prior, respectively.

2.5. Deconvolution Algorithm

The conjugate gradient method is used to solve the RGB channels of the image alternately, where each channel is solved while other two channels kept constant. The deconvolution algorithm for the RGB channels are similar, so we take the R-channel of the image as an example to illustrate the deconvolution algorithm. First, since $x_g$ and $x_b$ are fixed parameters, the cross-channel prior terms can be written as,

$$\begin{array}{c}{\displaystyle \sum _{l\ne r}}\left[{∥{\Delta }_h{x}_r·x_l-{\Delta }_h{x}_l·x_r∥}_1+{∥{\Delta }_v{x}_r·x_l-{\Delta }_v{x}_l·x_r∥}_1\right]\\ ={\displaystyle \sum _{l\ne r}}\left[{∥D_l{\Delta }_h{x}_r-D_{hl}{x}_r∥}_1+{∥D_l{\Delta }_v{x}_r-D_{vl}{x}_r∥}_1\right]\end{array},$$

(6)

where $D_l$ denotes the diagonal matrix with the diagonal taken from the parameter $x_l$, and $D_{hl}$ and $D_{vl}$ are the diagonal matrices with ${\Delta }_h{x}_l$ and ${\Delta }_v{x}_l$ as the diagonal elements, respectively.

Since the $l_1$ norm cannot be solved directly using the gradient method, the majorization–minimization (MM) method [30] is used in this paper to approximate the $l_1$ norm. By introducing an auxiliary vector w, the $l_1$ norm can be written as:

$${∥Ax∥}_1=\sqrt{{\left(Ax\right)}^2}\le \frac{w+{\left(Ax\right)}^2}{2\sqrt{w}},$$

(7)

where $w>0$, and the equation is established when $w=Ax$.

Therefore, the solution Equation (5) can be optimized in the following form,

$$\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^s}\left[{\left(SK{M}_i\right)}^{T}SK{M}_i\right]+{\lambda }_c[{\Delta }_h^T{W}_h{\Delta }_h+{\Delta }_v^T{W}_v{\Delta }_v]\\ +{\beta }_{cl}{\displaystyle \sum _{l=g,b}}\left[\begin{array}{c}{\left(D_l{\Delta }_h-D_{hl}\right)}^T{W}_{lh}\left(D_l{\Delta }_h-D_{hl}\right)\hfill \\ +{\left(D_l{\Delta }_v-D_{vl}\right)}^T{W}_{lv}\left(D_l{\Delta }_v-D_{vl}\right)\hfill \end{array}\right]\end{array}\right\}{x}_r^{\left(t+1\right)}=\sum _{i=1}^s{\left(SK{M}_i\right)}^T{y}_{ir},$$

(8)

where $x_r^{t+1}$ is the $(t+1)$th estimate to be computed, and the auxiliary matrix

$$W_h=diag\left[{\left({\Delta }_h{x}_r^{\left(t\right)}\right)}^2\right],$$

(9)

$$W_v=diag\left[{\left({\Delta }_v{x}_r^{\left(t\right)}\right)}^2\right],$$

(10)

$$W_{lh}=diag\left\{{\left[\left(D_l{\Delta }_h-D_{hl}\right)x_r^{\left(t\right)}\right]}^2\right\},$$

(11)

and

$$W_{lv}=diag\left\{{\left[\left(D_l{\Delta }_v-D_{vl}\right)x_r^{\left(t\right)}\right]}^2\right\},$$

(12)

are calculated by the tth estimate x. The termination condition of our iteration is set as:

$$\frac{{∥x^{\left(t+1\right)}-x^{\left(t\right)}∥}_2^2}{{∥x^{\left(t\right)}∥}_2^2}\le \epsilon .$$

(13)

3. Results

In this paper, we test our proposed algorithm by comparing it with various state-of-the-art SR algorithms. The algorithm results are evaluated by the objective evaluation methods peak signal to noise ratio (PSNR) and structural similarity (SSIM) [31].

3.1. Simulation Images

Four HR images with the size $256\times 256$ are used as original images for simulation experiments. The four original images are show in Figure 1. Each original image is used to create eight synthetic images through image degradation described in Formula (1). The degradation process is as follows. The HR images are first shifted with random sub-pixel displacement and rotated with random angle to creat the image sequence of eight images. Then the image sequence is blurred by a Gaussian kernel of $5\times 5$ size and unit variance, and downsampled by a factor of 2 in both the vertical and horizontal directions. Finally, the LR image sequence is obtained by adding 20 dB Gaussian noise. In order to evaluate our proposed algorithm, four representative methods are used in our experiment:

• Grayscale image SR performed on RGB color channels independently.
• color image SR performed on YCrCb color space [27].
• color image SR performed on chrominance regularization [32].
• color image SR performed on cross-channel prior.

The method in [32] is a learning-based method with color constraint; here, we replace it with an MFSR method using the same color constraint. All methods use the same grayscale SR algorithm [26] to eliminate the error caused by different convolution methods.

The PSNR is the most common and widely used objective evaluation index for images, which evaluates the image quality through the pixel error between images. Given a test image $\widehat{x}$ with size $M\times N$ and its original image x, the PSNR is defined as:

$$\left\{\begin{array}{c}PSNR=10{log}_{10}\frac{L^2}{MSE}\hfill \\ MSE=\frac{1}{3MN}{\displaystyle \sum _{n=1}^N}{\displaystyle \sum _{m=1}^M}{\displaystyle \sum _{c=r,g,b}}{\left[x(n,m,c)-\widehat{x}(m,n,c)\right]}^2,\hfill \end{array}\right.,$$

(14)

where L is the peak value in the image data and if it a 8-bit image, the L is 255. The evaluated results of PSNR is shown in

Table 1. Comparison of the PSNR of different SR methods.

Image	Bicubic	SR Method 1	SR Method 2	SR Method 3	SR Method 4
airplane	23.3448	25.2329	23.6477	25.8175	26.7506
parrot	27.9443	29.2376	27.7819	29.9670	30.8983
boat	24.4134	27.4984	25.8517	29.4892	30.8083
kids	25.1000	25.3873	24.0937	27.8454	28.8605
butterfly	22.3581	24.5738	24.3927	25.7262	26.8961
face	27.5945	29.3142	25.9966	29.3134	30.5205
child	26.4463	28.3403	24.4222	28.3416	30.7550
bird	28.1778	31.1675	29.0075	32.1796	33.6357

. It is obvious that our proposed method outperforms other SR methods.

Smoothing the image will improve the image PSNR, but it will lead to the loss of image texture details. Therefore, it is not accurate to evaluate the algorithm results from the PSNR alone. SSIM [31] is another widely used objective evaluation standard. SSIM can measure image similarity in terms of brightness, contrast, and structure, which are more sensitive to edge information. Hence, the combined use of the SSIM and the PSNR can accurately evaluate the quality of the reconstructed image.

The SSIM is defined as follows:

$$SSIM=\frac{(2{\mu }_x{\mu }_{\widehat{x}}+C_1)(2{\sigma }_{x\widehat{x}}+C_2)}{({\mu }_x^2+{\mu }_{\widehat{x}}^2+C_1)({\sigma }_x^2+{\sigma }_{\widehat{x}}^2+C_2)},$$

(15)

where ${\mu }_x$ and ${\mu }_{\widehat{x}}$ are the means and ${\sigma }_x$ and ${\sigma }_{\widehat{x}}$ are the standard deviations of the original image x and the test image $\widehat{x}$, respectively. ${\sigma }_{x\widehat{x}}$ is the covariance between x and $\widehat{x}$, and $C_1$ and $C_2$ are constants. The SSIM is equal to 1 only if $x=\widehat{x}$. The comparison of the SSIM of different SR methods is show in

Table 2. Comparison of the SSIM of different SR methods.

Image	Bicubic	SR Method 1	SR Method 2	SR Method 3	SR Method 4
airplane	0.7757	0.7965	0.7638	0.8065	0.8956
parrot	0.9319	0.9331	0.9351	0.9396	0.9631
boat	0.8307	0.8620	0.8920	0.9052	0.9388
kids	0.8375	0.8306	0.8425	0.8575	0.9067
butterfly	0.9039	0.9351	0.9437	0.9483	0.9631
face	0.8154	0.8441	0.8263	0.8440	0.8757
child	0.8619	0.8847	0.9027	0.8848	0.9328
bird	0.9380	0.9472	0.9361	0.9519	0.9646

. It appears that our algorithm provides better results than other SR methods.

Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show our experiment’s results of different SR methods. It is clear from the images that our algorithm works better than the other comparison algorithms.

3.2. Real Data

In the real data experiment, a dataset is used to verify our proposed algorithm. The dataset is a sequence of 40 images of size $115\times 138$ obtained from the MDSP dataset [33]. The real data is SR with a magnification factor of 3, and the blur kernel is a Gaussian kernel with variance of 1 and size of $5\times 5$. The SR results of the real data are show in Figure 10.

The traditional bicubic interpolation method has little effect on image SR. The SR method with RGB channels reconstruct independently can restore the lost details by SR and correct color image chromatic aberration due to the robustness of the SR algorithm. The results of SR method 2 are seriously affected by chromatic aberration because that only the luminance channel is reconstructed when SR is performed in the YCbCr color space. When the chromatic aberration is not severe, the SR method 3 can use the chrominance regularization prior to enhance the SR effect, but when the chromatic aberration is severe, the chrominance regularization prior used in SR method 3 will cause the image resolution to be unclear. It is obvious from the figure that our proposed method with cross-channel prior has the best SR results.

4. Conclusions

In this paper, we propose a multiframe SR algorithm for color images based on a new cross-channel prior. The results of the simulation show that our proposed algorithm can effectively suppress the noise and preserve edge details with a good reconstruction effect. The real data experiments show that the proposed algorithm can also suppress chromatic aberration and achieve the state-of-the-art SR results.