Super-Resolution Reconstruction Algorithm for Infrared Image with Double Regular Items Based on Sub-Pixel Convolution

Abstract

In this paper, an adaptive dual-regularization super-resolution reconstruction algorithm based on sub-pixel convolution (MPSR) is proposed. There are two novel features of the algorithm: First, the traditional regularization algorithm and sub-pixel convolution algorithm are combined to enrich the details; then, a regularization function with two adaptive parameters and two regularization terms is proposed to enhance the edge. MPSR firstly enhances the multi-scale detail of low-resolution images; then, regular processing and feature extraction are carried out; finally, sub-pixel convolution is used to fuse the extracted features to generate high-resolution images. The experimental results show that the subjective and objective evaluation indexes (PSNR/SSIM) of the algorithm have achieved satisfactory results.

1. Introduction

Infrared imaging equipment has anti-jamming and strong target recognition ability, and can work at night and in harsh environments. It has been widely applied in industry, military, medicine, life and other fields. However, due to the limited number of pixels in the detector array of the infrared imaging system and the limitation of pixel size, the spatial sampling frequency cannot satisfy the sampling theorem, and infrared diffraction also causes the loss of high-frequency details of the image signal. In addition, the temperature range of real objects in nature is relatively small, and there is heat exchange and absorption with the surrounding environment. Therefore, compared with visible light images, an infrared image has lower resolution, unclear gray level, more concentrated gray distribution and less obvious texture, which is far from meeting the needs of human eyes for resolution. To solve these problems, the researchers applied the super-resolution reconstruction algorithm to the infrared image field.

The deep convolution neural network has been successfully applied to single image super-resolution reconstruction (SISR) [1]. The goal of single image super-resolution reconstruction is to reconstruct high-resolution (HR) images from low-resolution (LR) images. It has a wide range of applications in security, monitoring, satellite, medical imaging and other fields [2], and can be used as a a built-in module for other image restoration or recognition tasks [3,4]. With the rapid update of intelligent algorithms, the image super-resolution algorithm has made great progress compared with the original. Because of the addition of depth learning theory, the image super-resolution reconstruction effect is amazing, and the derivative speed of the new algorithm is very fast. From Figure 1, it can be seen that the super-resolution reconstruction algorithm based on deep learning can be divided into four parts:

• Algorithm based on standard CNN, such as SRCNN [5], FSRCNN [6], ESPCN [7];
• Algorithm based on ResNet, such as VDSR [8], RED [9], CARN [10];
• Algorithm based on GAN, such as SRGAN [11] and SRFeat [12];
• Other model algorithms, such as RDN [13], IDN [14], DBPN [15].

In 2016, Shi proposed a super-resolution reconstruction method based on sub-pixel convolution [7]. This method directly takes low-resolution images as input, which reduces the complexity and has a good reconstruction effect. However, due to the low-resolution, indistinct gray level, centralized gray distribution and unclear texture of infrared images, this algorithm is improved on the basis of a sub-pixel convolution super-resolution reconstruction algorithm. The improved algorithm is more suitable for infrared images and achieves better results.

In this paper, an adaptive dual-regularization super-resolution reconstruction algorithm (MPSR) based on sub-pixel convolution is proposed. This algorithm has obvious advantages in infrared image processing. MPSR firstly enhances the multi-scale detail of low-resolution images, Then the initial reconstruction is carried out with the regular terms of the double regular terms and the double adaptive parameters. Finally, sub-pixel convolution is used to fuse the extracted features to generate high-resolution images. The experimental results show that the subjective and objective evaluation indexes (PSNR/SSIM) of the algorithm have achieved satisfactory results.

The rest of this paper is organized as follows. Section 2 is the detailed introduction of the algorithm proposed in this paper. Section 3 describes the content of the experiment and the analysis of the results. Section 4 is the conclusion.

2. Methods

Due to the low resolution, indistinct gray level, centralized gray distribution and unclear texture of infrared image, the super-resolution reconstruction algorithm based on sub-pixel convolution directly takes a low-resolution image as input, which results in an unsatisfactory reconstruction effect.

Therefore, the adaptive dual-regularization super-resolution reconstruction algorithm based on sub-pixel convolution (MPSR) first enhances the infrared image with multi-scale details. In the second step, another regular term is added to the regular part of the algorithm to enhance the edge of the image, and an adaptive regularization parameter of the constrained fidelity term and two regular items is added. The third step is to extract and activate the features, finally, the sub-pixel convolution is used to generate high-resolution images. The algorithm block diagram of MPSR is shown in Figure 2 and will be described in detail in the following sections.

2.1. Multi-Scale Detail Enhancement

MPSR uses the multi-scale method to improve image details without artifacts. Firstly, the image is convoluted into three Gaussian blur kernels $H_1,H_2,H_3$ of different scales, and the blurred images $B_1,B_2,B_3$ are obtained as follows:

$$B_1=H_1\ast X,B_2=H_2\ast X,B_3=H_3\ast X$$

(1)

Then, the convoluted images are subtracted from the original image to obtain different levels of detail information:

$$D_1=X-B_1,D_2=X-B_2,D_3=X-B_3$$

(2)

Finally, the overall detail image is generated by merging the three layers:

$$D=\left(1-a_1\times \mathrm{sgn}\left(D_1\right)\right)\times D_1+a_2\times D_2+a_3\times D_3$$

(3)

the fine detail $D_1$ expands the gray level differences near the edge, but may saturate the gray levels duo to its excessive overshooting. To overcome this problem, the sgn function is introduced.

The data of $a_1$,$a_2$,$a_3$ and the PSNR of the reconstructed image are obtained by simulation experiment. Figure 3 is drawn from the experimental data. It can be seen that when $a_1$,$a_2$,$a_3$ are set to 0.1, 0.2 and 0.6 respectively, the reconstruction effect is the best.

2.2. Adaptive Regularization Algorithm Based on Maximum a Posteriori Estimation

2.2.1. Related Algorithm

The algorithm in this paper is proposed on the basis of the maximum a posteriori estimation, which is described in detail below.

The implication of the super-resolution reconstruction algorithm based on maximum a posteriori estimation is that if the sequence of low-resolution images is known; the posteriori probability of high-resolution images is maximized as follows:

$${\widehat{x}}_{MAP}=\arg \underset{x}{\max }\left[\mathrm{Pr}\left\{x|y\right\}\right]$$

(4)

According to the Bayesian formula, the following formula is obtained:

$${\widehat{x}}_{MAP}=\arg \underset{x}{\max }\left[\frac{\mathrm{Pr}\left\{x|y\right\}\mathrm{Pr}\left\{x\right\}}{\mathrm{Pr}\left\{y\right\}}\right]$$

(5)

Take the logarithm to the right side of Formula (5):

$${\widehat{x}}_{MAP}=\arg \max \left[\log \mathrm{Pr}\left\{y|x\right\}+\mathrm{logPr}\left\{x\right\}\right]$$

(6)

where $\log \mathrm{Pr}\left\{y|x\right\}$ is the logarithm of the maximum likelihood function; $\log \mathrm{Pr}\left\{x\right\}$ is the logarithm of the prior probability of $x$. If the image noise is assumed to be Gauss noise with mean values of 0 and variance of ${\sigma }_k^2$, the overall probability function for estimating $y_k$ by $x$ is:

$$\mathrm{Pr}(y_k|x)={\displaystyle \prod _{\forall x,y}\frac{1}{{\sigma }_k\sqrt{2\pi }}\exp \left(-\frac{{\left({\widehat{y}}_k-y_k\right)}^2}{2{\sigma }_k^2}\right)}$$

(7)

In Formula (7), ${\widehat{y}}_k$ simulates low-resolution images and is given by ${\widehat{y}}_k=A_k\widehat{x}$, where $A_k$ is a fuzzy matrix. Assuming that the images are independent of each other, the probability density function of low-resolution image sequences can be expressed as Formula (8).

$$\mathrm{Pr}(y_k|x)={\displaystyle \coprod _{\forall k}\left[{\displaystyle \prod _{\forall x,y}\frac{1}{{\sigma }_k\sqrt{2\pi }}\exp \left(-\frac{{\left({\widehat{y}}_k-y_k\right)}^2}{2{\sigma }_k^2}\right)}\right]}$$

(8)

A common prior probability estimation method is to smooth the image, that is, to punish the adjacent pixel units with large differences in the image, and its probability density function can be assumed to be in the following form:

$$\mathrm{Pr}\left(x\right)=\exp \left[-\frac{1}{\lambda }{\Vert Qx\Vert }^2\right]$$

(9)

where $\lambda$ is a control parameter to control the peak of probability density distribution. Substitute Formula (9) into Formula (6):

$${\widehat{x}}_{MAP}=\arg \max \left[\log \frac{N}{{\sigma }_K\sqrt{2\pi }}-{\displaystyle \sum \frac{{\left({\widehat{y}}_k-y_k\right)}^2}{2{\sigma }_k^2}-\frac{1}{\lambda }{\Vert Qx\Vert }^2}\right]$$

(10)

In Formula (10), the first term in the middle bracket on the right side of the equal sign is a constant term, which can be eliminated directly; then the negative sign of the other two terms is changed to a positive sign, and assuming that the noise of each image has the same variance, the above maximization problem can be transformed into the following minimization problem:

$${\widehat{x}}_{MAP}=\arg \min \left[{\displaystyle \sum {\left({\widehat{y}}_k-y_k\right)}^2+\frac{2{\sigma }^2}{\lambda }{\Vert Qx\Vert }^2}\right]$$

(11)

Set ${\alpha }_1=\frac{2{\sigma }^2}{\lambda }$; the above formula can be written as:

$${\widehat{x}}_{MAP}=\arg \min \left[{\Vert y-Ax\Vert }^2+{\alpha }_1{\Vert Qx\Vert }^2\right]$$

(12)

where ${\alpha }_1$ is the regularization parameter. The necessary condition for minimizing the above formula is that the partial differential value of ${\Vert y-Ax\Vert }^2+{\alpha }_1{\Vert Qx\Vert }^2$ with respect to $x$ is zero, as Formula (13).

$$(A^{T}A+{\alpha }_1{Q}^{T}Q)x=A^{T}y$$

(13)

According to Formula (12), regularization parameter ${\alpha }_1$ controls the relative contribution of ${\Vert y-Ax\Vert }^2$ and ${\Vert Qx\Vert }^2$ in the process of solving, and constrains the degree of distortion and smoothness of reconstructed images. If ${\alpha }_1$ is too small, the problem of noise is not solved well, and the reconstructed image will still be distorted. If ${\alpha }_1$ is too large, the reconstructed image will be too smooth, resulting in the loss of image details. The adaptive solution method can solve the above problems. This method makes full use of the information of reconstruction results in the iteration process, updates ${\alpha }_1$ continuously and calculates the reconstructed image. The new reconstructed image will be applied to solve ${\alpha }_1$ in the next iteration, so that the optimal solution of the reconstructed image can be obtained by cyclic iteration.

From the principle of regularization, it is pointless that the regularization parameter is less than zero, so the first condition it should satisfy is to be greater than zero. Larger ${\Vert y-Ax\Vert }^2$ represents that there is a large noise in the model. In order to eliminate the influence of noise on the results, larger ${\alpha }_1$ is needed to regularize it, so ${\alpha }_1$ should be proportional to ${\Vert y-Ax\Vert }^2$. Because $Q$ is a high-pass filter operator and larger ${\Vert Qx\Vert }^2$ represents image with rich edge and texture details, it is necessary to select smaller regularization parameters to maintain image details, so ${\alpha }_1$ should be inversely proportional to ${\Vert Qx\Vert }^2$. According to the three basic attributes mentioned above, the following regularization parameter solving formulas are determined:

$${\alpha }_{1\left(k+1\right)}=\ln \left(\lambda \frac{{\Vert y-A{x}_k\Vert }^2}{{\Vert Q{x}_k\Vert }^2+r}+1\right)$$

(14)

In Formula (14), ${\alpha }_{1\left(k+1\right)}$ is the regularization parameter of the $k+1$ iteration; $x_k$ is the reconstruction image of the k iteration; $r$ is a very small number to ensure that the denominator is not zero; $\lambda$ is the convergence correction factor, and a fixed constant can be selected. In the process of iteration, after regularization parameters are determined, Formula (15) is applied to update the reconstructed image:

$$x_{k+1}=x_k+\left[A^{T}y-\left(A^{T}A+{\alpha }_{1\left(k+1\right)}{Q}^{T}Q\right)x_k\right]$$

(15)

2.2.2. Improvement of Regularization

On the basis of the two-dimensional Laplacian operator, a Prewitt operator is added to enhance the edge of the image. This operator is represented by W.

$$\mathrm{Pr}(x)=\exp \left[-\frac{1}{{\lambda }_1}{\Vert Qx\Vert }^2--\frac{1}{{\lambda }_2}{\Vert Wx\Vert }^2\right]$$

(16)

Then the maximization problem is transformed into the minimization problem. The formula is as Formula (17):

$${\widehat{x}}_{MAP}=\arg \min \left[{\Vert y-Ax\Vert }^2+{\alpha }_1{\Vert Qx\Vert }^2+{\alpha }_2{\Vert Wx\Vert }^2\right]$$

(17)

where ${\alpha }_1$ and ${\alpha }_2$ are regularization parameters.

The necessary condition for minimizing the above formula is that the partial differential value of ${\Vert y-Ax\Vert }^2+{\alpha }_1{\Vert Qx\Vert }^2+{\alpha }_2{\Vert Wx\Vert }^2$ with respect to $x$ is zero, as Formula (18):

$$2A^T\left(Ax-y\right)+2{\alpha }_1{Q}^{T}Qx+2{\alpha }_2{W}^{T}Wx=0$$

(18)

In the process of iteration, after regularization parameters are determined, the following formulas are used to update the reconstructed image:

$$x_{k+1}=x_k+\left[A^{T}y-\left(A^{T}A+{\alpha }_{1\left(k+1\right)}{Q}^{T}Q+{\alpha }_{2\left(k+1\right)}{W}^{T}W\right)x_k\right]$$

(19)

$${\alpha }_{2\left(k+1\right)}=\ln \left(\lambda \frac{{\Vert y-A{x}_k\Vert }^2}{{\Vert W{x}_k\Vert }^2+r}+1\right)$$

(20)

where ${\alpha }_{1\left(k+1\right)}$ and ${\alpha }_{2\left(k+1\right)}$ are the regularization parameter of the k+1 iteration; $x_k$ is the reconstruction image of the k iteration; $r$ is a very small number to ensure that the denominator is not zero; $\lambda$ is the convergence correction factor, and a fixed constant can be selected.

2.3. Combining Regularization with Sub-Pixel Convolution

2.3.1. Basic Theory of Sub-Pixel Convolution

Many super-resolution reconstruction algorithms usually preprocess the low-resolution image in the first step, that is, they are enlarged to the required size by the bicubic interpolation algorithm [16], and then reconstructed. This greatly increases the complexity of reconstruction. The super-resolution reconstruction algorithm of sub-pixel convolution is to extract the features of low-resolution images first, and finally to fuse the extracted features to generate high-resolution images by using sub-pixel convolution. The process is shown in the Figure 4.

As shown in Figure 4, the input of the network does not need interpolation pretreatment, but directly inputs the original LR image. After the convolution layer, the characteristic images with channel number $r^2$ are obtained; $r$ is the magnification factor. Then, through the sub-pixel convolution layer, $r^2$ channels of each pixel correspond to a sub-pixel block with size $r\times r$ in the HR image. Finally, the characteristic image of $H\times W\times r^2$ is rearranged to an HR image of $rH\times rW\times 1$. Therefore, the number of feature outputs of the last convolution layer should be set to the square of the magnification factor $r$, so that the total number of pixels is the same as the HR image.

Compared with using the interpolation function to enlarge the image in the first layer of the network, when using sub-pixel convolution to enlarge the image, the interpolation function can learn automatically in the previous layer of convolution, thus learning better and more complex mapping from LR to HR. Since convolution operations are performed on LR images, smaller convolution cores can be used to extract the same information, which further reduces the computational complexity, so the efficiency will be significantly improved.

For a network composed of L layers, the first L-1 layer can be described as Formula (21) and Formula (25).

$$f^1\left(I^{LR};W_1,b_1\right)=\phi \left(W_1\ast I^{LR}+b_1\right)$$

(21)

$$f^l\left(I^{LR};W_{1:l},b_{1:l}\right)=\phi \left(W_1\ast f^{l-1}\left(I^{LR}\right)+b_l\right)$$

(22)

where $W_l$, $b_l$, $l\in (1,L-1)$ are the weight and deviation of the network, $W_l$ is the 2D convolution layer of size $n_{l-1}\times n_l\times k_l\times k_l$, $n_l$ is the characteristic number of layer $l$, $k_l$ is the filter size of layer $l$, deviation $b_l$ is the vector of length $n_l$, $\phi$ is a non-linear function (or activation function) and the last layer $f^L$ synthesizes.

2.3.2. Combining Regularization with Sub-Pixel Convolution

Due to the insufficient feature extraction of the regularization algorithm in the super-resolution reconstruction of a single image, and the complexity of using the interpolated image as the reconstruction input, in order to solve the above problems, this paper proposes an adaptive dual-regularization super-resolution reconstruction algorithm based on sub-pixel convolution (MPSR), which combines the regularization and sub-pixel convolution to make the original low-resolution image. The resolution image is directly used as the input. Firstly, it is regularized to enhance the details and edges; secondly, it is extracted and activated; finally, it is rearranged with the sub-pixel convolution layer to get a high-resolution image, which not only improves the image quality compared to the original basis, but also improves the algorithm training speed. See Algorithm 1 for details

Algorithm1 MPSR algorithm

1: Input: Low resolution image X, up-sampling factor s.

2: Step 1: Multi-scale detail enhancement for X.

3: Step 2: Initial image reconstruction using regularized objective function.

4: (1) Parameter initialization:

5: ${\alpha }_1=0.02$, ${\alpha }_2=0.008$, $MAX=50$, $\mathrm{ITER}=1$

6: (2) Reconstruction from objective function:

7: $x_{k+1}=x_k+\left[A^{T}y-(A^{T}A+{\alpha }_{1(k+1)}{Q}^{T}Q+{\alpha }_{2(k+1)}{W}^{T}W)x_k\right]$

8: (3) Adjust adaptive parameters:

9: ${\alpha }_{1\left(k+1\right)}=\ln \left(\lambda \frac{{\Vert y-A{x}_k\Vert }^2}{{\Vert Q{x}_k\Vert }^2+r}+1\right)$

10: ${\alpha }_{2\left(k+1\right)}=\ln \left(\lambda \frac{{\Vert y-A{x}_k\Vert }^2}{{\Vert W{x}_k\Vert }^2+r}+1\right)$

11: (4) $\mathrm{ITER}=\mathrm{ITER}+1$

12: (5) If $\mathrm{ITER}<\mathrm{MAX}$:

13:    Yes, Go back to 2);

14:    No, output, $X=x_{k+1}$

15: Step 3: Image magnification by sub-pixel convolution

16: (1) Using convolution layers to extract image features

17: (2) Generation of high-resolution image $I$ by sub-pixel convolution

18: Output: High-resolution image $I$

3. Experiments and Results

3.1. Complexity Analysis

When calculating the complexity of the model, the influence of the activation function on the complexity of the model is neglected because the activation function has little influence on the complexity of the model. The formula for calculating the complexity is as follows:

$${\rm O}\left[\left({\displaystyle \sum _{l=1}^D{K}_l\times C_{l-1}{C}_l}\right)S_I\right]$$

(23)

where $D$ is the network depth, $K_l$ is the convolution core size of layer $l$, $C$ is the number of channels, $S_I$ is the size of the input image. Formula (23) shows that the complexity of the network model is proportional to the size of the HR image, and the impact of the middle layer on the complexity of the network model is relatively greater than that of the first and last layer. In SRCNN, the size of the convolution core in the first layer is set to $9\times 9$, but since our algorithm uses the original low-resolution image as input, the smaller convolution core can be applied to extract the same features. The size of the convolution core in the first layer is $5\times 5$, and the size of the convolution core in the latter layer is $3\times 3$. The following table compares the complexity of the 9-5-5 SRCNN algorithm with that of MPSR algorithm. According to

Table 1. Comparisons of model complexity

Complexity Related Items	SRCNN	MPSR
Conv1	(64, 9, 1)	(64, 5, 1)
Conv2	(32, 5, 64)	(64, 3, 64)
Conv3	(1, 5, 32)	(64, 3, 32)
Conv4	None	(4, 3, 32)
Input image	$S_{HR}$	$S_{LR}$
Number of parameters	57,184	58,048
Complexity	3.9	1

, the complexity of the algorithm MPSR is 3.9 times lower than that of SRCNN.

3.2. Experimental Environment and Parameters Setting

Infrared image dataset is a kind of self-made dataset. These images are taken by infrared imaging equipment. It contains 400 training images and 100 test images. In order to make the experimental results as accurate as possible, the training set should contain as many scenes as possible. In order to synthesize low-resolution images, Gauss filter is used to blur high-resolution images. Due to the small number of samples in the dataset, the original high-resolution image and the blurred low-resolution image are cut to generate more sample pairs. The original image pixel is $320\times 240$, the cut image pixel is $32\times 24$ and finally $4\times {10}^4$ pairs of training samples are generated. During the training, the learning rate is set to ${10}^{-4}$ and the activation function is RELU.

Among them, MAX is the largest number of regular partial iterations, and the maximum number of iterations has an impact on the reconstruction effect and time. As shown in Figure 5.

As can be seen from the Figure 5a, the greater the number of iterations, the greater the PSNR values, the better the reconstruction effect. Figure 5b shows that the larger the number of iterations, the longer the time needed to reconstruct the image. Considering comprehensively, the number of iterations in this experiment is set to 50. In practical work, it can be selected according to the needs.

The experimental environment is as

Table 2. Experimental environment.

Project	Environment/Version
Operating system	ubuntu
CPU	i7-8700k
Memory	32GB
GPU	GTX1080ti
Framework	pytorch
Python IDE	pycharm

.

3.3. Image Quality Metric Parameters

In the evaluation of simulation image test results, subjective evaluation and objective evaluation are used in this paper. The subjective evaluation is the result of human observation [17]. The objective evaluation indexes include peak signal-to-noise ratio (PSNR) and structural similarity degree (SSIM). The formula for obtaining PSNR value is defined as Formula (24):

$$PSNR=10{\log }_{10}\frac{{255}^2}{MSE}$$

(24)

where MSE represents the mean square error. In the objective evaluation index of image processing, PSNR is widely used, which is recognized by researchers. The larger the PSNR value is, the closer the reconstructed image is to the original high-resolution image, and the better the reconstruction effect is.

SSIM (structural similarity index) is another commonly used evaluation index, which judges image quality according to the similarity of image structure before and after reconstruction. The formula is defined as Formula (25).

$$SSIM\left(x,y\right)=\frac{\left(2{\mu }_x{\mu }_x+c_1\right)\left(2{\sigma }_{xy}+c_2\right)}{\left({\mu }_x^2+{\mu }_y^2+c_1\right)\left({\sigma }_x^2+{\sigma }_y^2+c_2\right)}$$

(25)

where ${\mu }_x$ is the average of $x$, ${\mu }_y$ is the average of $y$, ${\sigma }_x^2$ and ${\sigma }_y^2$ are the variance of $x$ and $y$ respectively, and ${\sigma }_{xy}^2$ is the covariance of $x$ and $y$. $c_1={(k_{1}L)}^2$, $c_2={(k_{1}L)}^2$. $L$ reflects the range of pixel values. $k_1=0.01,k_2=0.03$, SSIM value range is $(-1,1)$. The closer the SSIM value is to 1, the better the image reconstruction effect is. When the two images are identical, the value of SSIM is 1.

3.4. Experimental Results

3.4.1. Experiment on the Effectiveness of Algorithm Improvements

MPSR has two improvements. Firstly, because the texture of the infrared image is not obvious, the image details are enhanced before the super-resolution reconstruction, and then the image is reconstructed with the improved biregular objective function. Secondly, the regularization objective function is combined with sub-pixel convolution. Instead of taking the enlarged high-resolution image as the input, the low-resolution image is taken as the input. Finally, the sub-pixel convolution is used to enlarge the image to the required multiple, which not only reduces the complexity, but also enhances the reconstruction effect. The effectiveness of these two parts is verified by experiments, as shown in Figure 6.

Figure 6a is a comparison of the objective functions between single regular item and double regular items. It can be seen from the figure that the effect of image reconstruction with the objective function of double regular items is better than that with single regular item, and the PSNR value is increased by about 1.5 dB

Figure 6b is the comparison of PSNR values before and after algorithm improvement. The former algorithm refers to the traditional regularization algorithm, the improved algorithm is MPSR. It can be seen from the figure that the improved PSNR value has increased by 2.5 dB. As shown above, both improvements in this experiment are effective.

3.4.2. Comparison of Different Algorithms

In order to compare with MPSR, spline interpolation algorithm (SPLINE), adaptive regularization algorithm (MAP), SRCNN and sub-pixel convolution-based reconstruction algorithm (ESPCN) are also applied in the experiment. Eight images in the test set are selected to display the data. The specific images are shown in Figure 7.

The PSNR and SSIM values obtained from reconstructed images using different algorithms are listed in

Table 3. PSNR values of different algorithms.

Metric Parameters	Number	Image Pixel	SPLINE	MAP	SRCNN	ESPCN	MPSR
PSNR (dB)	1	320 * 240	34.0800	35.1562	36.9908	37.0787	38.0546
	2	320 * 240	23.6019	24.5066	25.9310	25.9588	27.5505
	3	320 * 240	31.3120	32.2794	33.2204	33.8099	35.0086
	4	320 * 240	31.7600	32.7605	33.0185	33.6165	36.7969
	5	320 * 240	31.8201	32.2924	33.3726	33.6970	35.8422
	6	320 * 240	30.8223	31.2468	32.1141	32.6161	33.7191
	7	320 * 240	31.0702	31.9873	33.3658	33.7443	34.9114
	8	320 * 240	29.8001	30.8034	32.1989	32.5128	34.1559

and

Table 4. SSIM values of different algorithms.

Metric Parameters	Number	Image Pixel	SPLINE	MAP	SRCNN	ESPCN	MPSR
SSIM	1	320 * 240	0.9113	0.9215	0.9398	0.9429	0.9516
	2	320 * 240	0.7518	0.7875	0.8315	0.8457	0.8804
	3	320 * 240	0.8779	0.8925	0.9023	0.9194	0.9365
	4	320 * 240	0.9594	0.9617	0.9645	0.9690	0.9736
	5	320 * 240	0.9593	0.9625	0.9659	0.9678	0.9702
	6	320 * 240	0.9175	0.9223	0.9290	0.9360	0.9436
	7	320 * 240	0.8099	0.8340	0.8661	0.8782	0.9059
	8	320 * 240	0.8334	0.8542	0.8784	0.8889	0.9144

.

In order to evaluate the effect more intuitively, the data in Table 3 and Table 4 are averaged and plotted in Figure 8.

The abscissa of Figure 8a is the algorithm name, and the ordinate is the PSNR values obtained by reconstructing the image with different algorithms. From Figure 8a, it can be seen that the SPLINE algorithm has the worst effect, the mean value of PSNR after reconstruction is only 30.5 dB, and the traditional regularization algorithm MAP is not ideal. At present, the popular algorithm based on deep learning has a better reconstruction effect than the traditional algorithm, but MPSR proposed in this paper has the best effect. It can be seen from Figure 8a that the PSNR values calculated by this algorithm is about 4 dB higher than that of SPLINE interpolation algorithm, about 2 dB higher than that of SRCNN algorithm, and about 1 dB higher than that of ESPCN algorithm.

The abscissa of Figure 8b is the algorithm name, and the ordinate is the SSIM value obtained by reconstructing the image with different algorithms. Compared with the trend of PSNR, spline algorithm has the worst effect. The mean value of SSIM after reconstruction is less than 0.88, while the mean value of SSIM of MPSR is nearly 0.94, which is nearly 0.03 more than that of SRCNN algorithm and 0.02 more than that of ESPCN algorithm.

The effect of the reconstructed images is as in Figure 9.

Figure 9 are the images reconstructed with different algorithms. Among these five methods, the super-resolution image reconstructed by SPLINE interpolation is the most fuzzy and the edge preserving effect is the worst. SRCNN and ESPCN algorithm have good reconstruction effect and rich detail information, but the edge reconstruction effect is still not ideal. The MPSR algorithm proposed in this paper has obvious advantages in detail information enhancement and edge preservation, and can effectively reduce noise and improve the overall brightness of the image, and the subjective visual effect is better than other algorithms.

4. Conclusions

Common super-resolution reconstruction algorithms usually pre-process low-resolution images; that is, interpolation is enlarged to the required size, and then super-resolution reconstruction is carried out. This greatly increases the complexity of reconstruction. The image super-resolution reconstruction algorithm based on sub-pixel convolution directly takes low-resolution images as input, which reduces the complexity and improves the speed. However, due to the low resolution of the infrared image, the unclear gray level and the centralized gray distribution, the texture is not obvious, and the reconstruction effect is not ideal. In this paper, an adaptive dual-regularization super-resolution reconstruction algorithm based on sub-pixel convolution is proposed. The algorithm regularizes the image before convolution. Compared with the traditional regularization algorithm, the algorithm firstly enhances the multi-scale details of the image, and adds another regularization term to the regularization part of the algorithm to enhance the edge of the image, and adds a constraint fidelity. Compared with other algorithms, the PSNR and SSIM values of the proposed algorithm are significantly improved, and the enhancement of image details and edge preservation are significantly better. Moreover, the proposed algorithm can effectively reduce noise and enhance the overall brightness of the image, and the subjective visual effect is better than other algorithms.