Multiple Degradation Skilled Network for Infrared and Visible Image Fusion Based on Multi-Resolution SVD Updation

Abstract

Existing infrared (IR)-visible (VIS) image fusion algorithms demand source images with the same resolution levels. However, IR images are always available with poor resolution due to hardware limitations and environmental conditions. In this correspondence, we develop a novel image fusion model that brings resolution consistency between IR-VIS source images and generates an accurate high-resolution fused image. We train a single deep convolutional neural network model by considering true degradations in real time and reconstruct IR images. The trained multiple degradation skilled network (MDSNet) increases the prominence of objects in fused images from the IR source image. In addition, we adopt multi-resolution singular value decomposition (MRSVD) to capture maximum information from source images and update IR image coefficients with that of VIS images at the finest level. This ensures uniform contrast along with clear textural information in our results. Experiments demonstrate the efficiency of the proposed method over nine state-of-the-art methods using five image quality assessment metrics.

1. Introduction

Multi-modal images have huge demand in many areas, such as medical imaging, target recognition, and remote sensing [1]. This is due to the advancement in sensor technology. In particular, the infrared (IR) and visible (VIS) sensor combination has distinct advantages. IR sensors capture thermal radiation and represent as IR images. On the contrary, VIS images hold textural details captured from the reflected light on VIS sensors. By considering the mapped IR and VIS sensor data, the fused IR-VIS images have inherent features with good visual understating of a scene [2].

Image fusion aims to extract the vital features from multiple source images and merge them into a single image. In this regard, various algorithms have been developed with certain fusion rules. These are based on saliency detection [3,4,5], sparse representation theory [6,7,8], multi-scale transforms [9,10], convolutional neural networks [11,12], subspace methods [13,14], machine and deep learning [15,16,17], hybrid methods [18,19], and other fusion strategies [20,21].

In spite of the appreciable performance achieved by these methods, the IR-VIS image fusion at different resolutions still remains an unsolved research problem. IR images usually suffer from poor resolution when compared with the corresponding VIS data. However, the existing methods [2,3,4,6,7,9,10,11,12,13,14,15,16,18,19,20,21,22] do not consider resolution differences between IR and VIS source images, leading to inaccurate fusion results. Under this condition, there are two possible cases while fusing the IR and VIS images with different resolution levels. For intuitiveness, these cases are illustrated in Figure 1. Let the dimensions of IR and VIS images be $M\times N$ and $sM\times sN$, respectively. In case 1, the VIS image is downsampled with a factor s before applying the fusion process [23]. The fused output in this procedure still suffers from low resolution. For consistency and also to compare with other methods, the fused image is forcibly converted to an HR image. However, this cannot recover the loss of VIS textures caused by the downsampling operation prior to image fusion.

To address this, we confine to case 2 in our work and eliminate inaccuracy and blur in the fused image. Here, the IR image is first upscaled to $sM\times sN$ before using it as the source image [23]. Hence, both IR and VIS images have the same HR dimensions before applying the fusion process. With the proper upscaling of IR images, both IR and VIS source images contribute to the fused image with improved resolution.

In this paper, we address the problem of image fusion with different resolution VIS-IR source images. We learn a deep CNN by assuming multiple true degradations in real time to improve practicability. The learned multiple degradations skilled network (MDSNet) provides good scalability to recover the IR images with low resolution (LR). Moreover, it creates resolution compatibility between IR-VIS images for the fusion process. Correspondingly, the multi-resolution singular value decomposition captures information from IR-VIS source images in horizontal, vertical and diagonal directions at different levels. The contrast correction on IR image coefficients with that of VIS image coefficients at the finest level ensures better fused images. The rest of this paper is organized as follows. In Section 2, we discuss the degradation model, network structure and loss function of MDSNet. In Section 3, we propose our IR-VIS fusion method by detailing MDSNet upscaling and MRSVD updating. In Section 4, the proposed method is validated over the existing methods based on quantitative metrics and qualitative analysis. Finally, conclusions are presented in Section 5.

2. MDSNet

In recent times, CNNs have shown unprecedented success in extensive applications. However, we limit our discussion on CNNs to super-resolution (SR) imaging in this paper. Existing SR CNN methods apply bicubic downsampling on an HR image, which do not consider true degradation, thus leading to poor performance. To address this issue, we employ the MDSNet for SR that can handle multiple degradations and improve the practicability [24,25]. In general, the degradation model of an LR image p from the corresponding HR image q can be viewed as

$${p=(q\otimes b)\downarrow }_s+n,$$

(1)

where ⊗ represents convolution operation, b is a blur kernel, ${\downarrow }_s$ indicates a downsampling operation with a factor s, and n is additive Gaussian noise. To obtain better insights on CNNs in the context of SR, we analyze the approximate solution to Equation (1) using the maximum a posteriori (MAP) framework as

$$\widehat{q}=arg\underset{q}{min}\frac{1}{2{\sigma }^2}{∥{(q\otimes b)\downarrow }_s-p∥}^2+\lambda \varphi \left(x\right).$$

(2)

Here, the first term $\frac{1}{2{\sigma }^2}{∥{(q\otimes b)\downarrow }_s-p∥}^2$ represents data fidelity and the second term $\lambda \varphi \left(x\right)$ indicates regularization with a trade-off $\lambda$. The above MAP problem in Equation (2) conveys that the SR reconstruction process should obey the degradation model while estimating HR images. In addition, it is clear that the estimated SR image $\widehat{q}$ is a function of LR input p, blur operator b, and standard deviation $\sigma$ representing noise level and $\lambda$.

$$\widehat{q}=\mathrm{f}(p,b,\sigma ;\theta ),$$

(3)

where $\theta$ represents the MAP inference parameter. However, the existing SR networks [5,8,17] limit the single image SR problem to interpolation, neglecting the blurring operation and noise level as $\widehat{q}=\mathrm{f}(p;\theta )$. However, this does not cover true degradations, resulting in poor performance. So, we consider the three inputs $p,b,$ and $\sigma$ together as in Equation (3) while modeling MDSNet. The sizes of $p,b,$ and $\sigma$ are different, which causes dimensionality mismatch while feeding the CNN. Hence, we apply dimensionality stretching and convert all the input parameters to have a similar size. Let us assume an LR image of size $M\times N$ with C number of channels as $M\times N\times C$, a blur operator of size $r\times r$, and a noise level $\sigma$. First, the $r\times r$ blur matrix is vectorized to $r^2\times 1$ and projected onto k-dimensional space using the principal component analysis (PCA) approach. The obtained vector is then concatenated with the noise level and denoted by a vector t, which is finally stretched into $k+1$ degradation maps of size $M\times N$ represented as $M\times N\times (k+1)$. Now, these maps have similar size compared to LR images, thus enabling MDSNet capable of handling three inputs simultaneously.

The MDSNet is designed for multiple degradations for improving the real-time SR performance. We describe the architecture of MDSNet in the following.

• The network takes $M\times N\times (C+k+1)$ images as input by considering $M\times N\times C$ LR images concatenated with $M\times N\times (k+1)$ degradation maps. The MDSNet achieves non-linear mapping using multiple convolutional layers in cascade form [17]. Each layer includes convolution operation (CONV), batch normalization (BN) [26] and rectifier linear unit (ReLU) [27].
• The first “CONV+BN+ReLU” layer operates on $M\times N\times (C+k+1)$ input images.
• All the middle layers are similar to each other, performing “CONV+BN+ReLU” operations.
• The last layer consists of “CONV” operation alone, which produces output HR image patches. The last layer is followed by a sub-pixel convolution layer [28] to convert HR image patches to a single HR image $sM\times sN\times C$.
• We use Adam optimizer [29] to minimize the loss function,

  $$L\left(\theta \right)=\frac{1}{2K}\sum _{i=1}^N{∥f\left(p_i,D_i;\theta \right)-q_i∥}^2,$$

  (4)

  where K is the total number of images including the degradation maps $D_i$ and $p_i,q_i$ are the i-th LR and HR images, respectively. For all scale factors s, we set a total of 12 convolution layers with 128 feature maps in each layer. The training is performed separately for different scaling factors. We use the trained model to enhance the resolution of the input LR-IR image in our proposed method.

3. Proposed Method

In this paper, we aim to solve the different resolution IF problem of IR and VIS images. Let us denote the ratio of IR image resolution and VIS image resolution by a factor s. That is, the IR image has dimensions $M\times N$ and that of the VIS image is $sM\times sN$. Given an LR-IR image ${IR}_p$ and an HR-VIS ${VIS}_q$ image, the fusion process of our proposed MDSNet-MRSVD algorithm is outlined in Figure 2.

3.1. LR-IR Image Upscaling Using MDSNet

Since the IR image ${IR}_p$ has a lower resolution, it is first subjected to MDSNet to generate the corresponding HR-IR image ${IR}_q$. The MDSNet is trained with numerous HR-LR example patches where the LR images are synthesized using Equation (1). However, the sizes of LR images, blur kernel, and noise are different, which causes dimensionality mismatch while feeding the CNN. Hence, we apply dimensionality stretching and convert all the input parameters to have the same size.

To synthesize HR images using MDSNet, it is required to specify the blur operator and noise level for defining the degradation maps. For blur operation, isotropic Gaussian kernel with a fixed width and anisotropic Gaussian kernel with probability density function having zero mean and variable covariance are used. The blur matrix is vectorized and projected onto k-dimensional space using the PCA approach. The obtained blur vector is concatenated with the additive white Gaussian noise and stretched into $M\times N\times (k+1)$ degradation maps. By considering these maps, MDSNet can handle multiple degradations in real time and improve the practicability.

MDSNet is optimized by minimizing the loss function in Equation (4) using Adam optimizer [29]. The learned knowledge is used to produce the HR-IR image with factor s. In other words, the size of HR-IR is $sM\times sN,$ which is similar to that of the HR-VIS image.

3.2. MRSVD Coefficients Generation and Updation

The implementation of MRSVD resembles the discrete wavelet transform (DWT) except the finite impulse response filters are replaced with SVDs [30,31]. However, MRSVD performs similar to DWT but with reduced computations. Moreover, its basis is data dependent unlike a fixed basis in DWT, discrete cosine transform and discrete Fourier transform. Hence, it can capture maximum information from the source IR and VIS images, thereby proving the finest details in the fused image.

The MRSVD of the HR-VIS image is given by

$$\left\{{\Phi }_L^{\mathrm{VIS}},{\left\{{\Psi }_i^{VI{S}_v},{\Psi }_i^{VI{S}_h},{\Psi }_i^{VI{S}_d}\right\}}_{i=1}^L,{\left\{{\mathrm{U}}_i^{\mathrm{VIS}}\right\}}_{i=1}^L\right\}=MRSVD\left(VISq\right)$$

(5)

where ${\Phi }_L^{VIS}$ denotes the L-th level approximation coefficient of the HR-VIS image ${VIS}_q$, ${\left\{{\Psi }_i^{VI{S}_v},{\Psi }_i^{VI{S}_h},{\Psi }_i^{VI{S}_d}\right\}}_{i=1}^L$ are the detailed coefficients of ${VIS}_q$ at different levels i in the vertical, horizontal and diagonal directions. ${\left\{{\mathrm{U}}_i^{\mathrm{VIS}}\right\}}_{i=1}^L$ represents the eigenvectors of ${VIS}_q$ corresponding to $i=1,2,\dots ,L$.

Similarly, the MRSVD of HR-IR can be written as

$$\left\{{\Phi }_L^{\mathrm{IR}},{\left\{{\Psi }_i^{{\mathrm{IR}}_v},{\Psi }_i^{{\mathrm{IR}}_h},{\Psi }_i^{{\mathrm{IR}}_d}\right\}}_{i=1}^L,{\left\{{\mathrm{U}}_l^{\mathrm{IR}}\right\}}_{i=1}^L\right\}=MRSVD(IRq)$$

(6)

where ${\Phi }_L^{\mathrm{IR}}$ denotes the L-th level approximation coefficient of the HR-IR image ${IR}_q$, ${\left\{{\Psi }_i^{{\mathrm{IR}}_v},{\Psi }_i^{{\mathrm{IR}}_h},{\Psi }_i^{{\mathrm{IR}}_d}\right\}}_{i=1}^L$ are the detailed coefficients of ${IR}_q$ at different levels i in the vertical, horizontal and diagonal directions. ${\left\{U_i^{IR}\right\}}_{i=1}^L$ represents the eigenvectors of ${IR}_q$ corresponding to $i=1,2,\dots ,L$.

The HR-IR image ${IR}_q$ is obtained from its LR-IR ${IR}_p$ using MDSNet. In the process of approximation, a non-uniform illumination may be induced in the resulting image due to the external training database considered. To address this issue, we correct the HR-IR image MRSVD approximation coefficient in Equation (5) using the corresponding VIS image coefficient in Equation (6).

$$\zeta =\frac{max\left({\mathrm{U}}_L^{\mathrm{VIS}}\right)}{max\left({\mathrm{U}}_L^{\mathrm{IR}}\right)}$$

(7)

where $\zeta$ is the correction factor for updating the ${\Phi }_L^{\mathrm{IR}}$ coefficient using ${\mathrm{U}}_L^{\mathrm{VIS}}$ and ${\mathrm{U}}_L^{\mathrm{IR}}$ at the L-th level. Now, the updated IR approximation coefficient can be obtained as ${\Phi }_L^{IR}=\zeta {\Phi }_L^{\mathrm{IR}}$ and thus representing the new set of IR coefficients as $\left\{{\Phi }_L^{\mathrm{IR}},{\left\{{\Psi }_i^{{\mathrm{IR}}_v},{\Psi }_i^{{\mathrm{IR}}_h},{\Psi }_i^{{\mathrm{IR}}_d}\right\}}_{i=1}^L,{\left\{{\mathrm{U}}_i^{\mathrm{IR}}\right\}}_{i=1}^L\right\}$.

3.3. Fusion Process

In this section, the fusion process of HR-IR and HR-VIS images using approximation coefficients, detailed coefficients and eigenvector matrices is presented. From Equation (5), the ${VIS}_q$ is decomposed into $\left\{{\Phi }_L^{VIS},{\left\{{\Psi }_i^{{VIS}_v},{\Psi }_i^{{VIS}_h},{\Psi }_i^{{VIS}_d}\right\}}_{i=1^{\prime }}^L,{\left\{U_i^{VIS}\right\}}_{i=1}^L\right\}$, and from Equation (6), ${IR}_q$ is decomposed and updated as $\left\{{\Phi }_L^{\mathrm{IR}},{\left\{{\Psi }_i^{{\mathrm{IR}}_v},{\Psi }_i^{{\mathrm{IR}}_h},{\Psi }_i^{{\mathrm{IR}}_d}\right\}}_{i=1}^L,{\left\{{\mathrm{U}}_i^{\mathrm{IR}}\right\}}_{i=1}^L\right\}$. The approximation coefficients and the eigen matrices are fused by following the average strategy, whereas the detailed coefficients of the fused image are obtained by taking the maximum value of IR and VIS image detailed coefficients,

$$\begin{array}{c}\hfill {\Phi }_L^F=\frac{1}{2}\left({\Phi }_L^{IR}+{\Phi }_L^{VIS}\right)\end{array}$$

(8)

$$\begin{array}{c}\hfill {\Omega }_i^{F_h}=max\left({\Omega }_i^{I{R}_h}+{\Omega }_i^{VI{S}_h}\right)\end{array}$$

(9)

$$\begin{array}{c}\hfill {\Omega }_i^{F_v}=max\left({\Omega }_i^{I{R}_v}+{\Omega }_i^{VI{S}_v}\right)\end{array}$$

(10)

$$\begin{array}{c}\hfill {\Omega }_i^F=max\left({\Omega }_i^{I{R}_h}+{\Omega }_i^{VI{S}_h}\right)\end{array}$$

(11)

$$\begin{array}{c}\hfill U_i^F=\frac{1}{2}\left(U_i^{IR}+U_i^{VIS}\right).\end{array}$$

(12)

Here, ${\Phi }_L^F$ denotes the L-th level approximation coefficient of the fused image. The set $\left\{{\Psi }_i^{{\mathrm{IR}}_v},{\Psi }_i^{{\mathrm{IR}}_h},{\Psi }_i^{{\mathrm{IR}}_d}\right\}$ represents the detailed coefficients of the fused image in the vertical, horizontal and diagonal directions. In addition, $U_i^F$ is the fused eigen matrix corresponding to the IR and VIS images. Finally, we apply the inverse MRSVD process and reconstruct the HR fused image $F_q$ using these fused coefficients $\left\{{\Phi }_L^F,{\left\{{\Psi }_i^{{\mathrm{F}}_v},{\Psi }_i^{{\mathrm{F}}_h},{\Psi }_i^{{\mathrm{F}}_d}\right\}}_{i=1}^L,{\left\{{\mathrm{U}}_i^{\mathrm{F}}\right\}}_{i=1}^L\right\}$.

4. Results and Discussion

In this section, we present the experimental settings of the proposed fusion algorithm, methods and metrics used for comparison. Finally, we present a qualitative and quantitative analysis to validate our MDSNet-MRSVD approach over the existing state-of-the-art methods.

4.1. Datasets and Settings

The resolution ratio between IR and VIS input images is set to $s=4$. However, the dimensions $M\times N$ for IR image and $sM\times sN$ for VIS image vary with the input test images. The problem of super-resolving IR images can be solved using $s=4,s=3,$ or $s=2$. In general, small scaling factors result in a better enhancement of IR images, and thus, considering large upscaling factors is a challenging task in real time. For this reason, we employ MDSNet with $s=4$ to perform super-resolution on IR input images.

The training data should be as close as possible to the data on which predictions are to be made. In our use case, MDSNet training involves blurry and LR data from a security camera. However, we first select a clean large-scale HR image dataset with DIV2K [5] 800 images and BSDS [32] 500 images. Out of these 1300 images, 1200 images are randomly selected for training, and the remaining 100 images are used as validation data. Next, the LR images are synthesized using Equation (1) to generate LR-HR image patch pairs. For this, the width of the blur kernel is set to $[0.2,4]$ for isotropic Gaussian, the angle range of eigenvectors is set to $[0,\pi ]$ to determine the space of anisotropic Gaussian kernel, and the noise range is set to [0, 75]. The PCA approach is implemented by setting $k=15$ dimensional space. The loss function in Equation (4) is minimized using the Adams approach [29], with an initial learning rate and mini-batch size set to ${10}^{-3}$ and 8, respectively. The learning rate reduces by a factor of 10 when the loss is becoming saturated.

The MRSVD of HR-VIS and HR-VIS images in Equations (5) and (6) are implemented with $L=2$ levels, and the corresponding $\zeta$ is computed for the 2nd level as shown in Equation (7). The ${VIS}_q$ coefficients and the updated ${IR}_q$ coefficients are fused using Equations (8)–(12), which are followed by the inverse MRSVD with $L=2$ levels. For testing the MDSNet-MRSVD approach, we select 20 IR-VIS registered HR image pairs from the TNO Human Factors [22] dataset. The IR and VIS images corresponding to each image pair in the dataset have the same resolution. In our experiments, we downscale the IR images to LR by the factor $s=4$.

4.2. Methods and Evaluation Metrics

We compare the proposed method with nine state-of-the-art IF methods, including a deep networks based super-resolution model in the stationary wavelet domain [17], the adversarial game model [22], total variation minimization (TVM) [33], gradient transfer–TVM problem [20], multi-sensor IF model with one level SVD [34], wavelets approach [35], image guided filtering [36], curvelet transform [37], and Laplacian pyramid model [38]. All the methods are implemented using Matlab software on a system with Intel(R) Core(TM) i7-8750H CPU @ 2.20 GHz and 16 GB RAM.

For the comprehensive evaluation of different IF methods, we adopt the peak signal-to-noise ratio (PSNR) [39] and structural similarity index measure (SSIM) [40]. In addition, the total IF performance $Q^{AB/F}$, IF loss $L^{AB/F}$, and IF artifacts $N^{AB/F}$ are also considered for in-depth IF performance analysis [41]. It is noted that the sum of $Q^{AB/F}$, $L^{AB/F}$, and $N^{AB/F}$ should be unity [42]. The metrics SD, EN, and SF depend on the fused image only, whereas PSNR, SSIM, $Q^{AB/F}$, $L^{AB/F}$, and $N^{AB/F}$ depend on both fused and IR and VIS source images. Theoretically, PSNR, SSIM, SD, SF, EN, and $Q^{AB/F}$ should be high, and $L^{AB/F}$, and $N^{AB/F}$ should be low for better IF performance.

4.3. Qualitative Results Analysis

For qualitative comparison of different state-of-the-art IF methods, the visual IF results on seven typical IR-VIS image pairs are presented in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. In Figure 3 and Figure 4, the fused images of the proposed method, existing methods [17,20,22,33,34,35,36,37,38] on test image pair1-duine sequence and test image pair4-tree sequence are given, respectively. Figure 3a,b represent the IR-VIS source images of the duine sequence and Figure 4a,b represent the IR-VIS source images of the tree sequence. Figure 3c and Figure 4c show the results of the proposed MDSNet-MRSVD method, whereas Figure 3d–l and Figure 4d–l are the fused images from the methods in [17,20,22,33,34,35,36,37,38]. Here, we have selected the captains present in the duine sequence and tree sequence as our region of interest. It can be noticed that the thermal information is well preserved in our result. In Figure 3, the captains contrast is higher in Figure 3c than in Figure 3f–j. The fused images in Figure 3d,e,k,l also enhance the contrast, but it comes with an additional blurring.

In Figure 5, Figure 6 and Figure 7, the results of the proposed method and the existing methods [17,20,22,33,34,35,36,37,38] on test image pair7-airplane in trees, test image pair10-two men in front of house, and test image pair13-lake are evaluated. Figure 5a,b represent the IR-VIS source images of airplane in trees, Figure 6a,b represent the IR-VIS source images of two men in front of house, and Figure 7a,b represent the IR-VIS source images of lake. Figure 5c, Figure 6c and Figure 7c show the results of the proposed MDSNet-MRSVD method, whereas the fused images of the methods in [17,20,22,33,34,35,36,37,38] are shown in Figure 5d–l, Figure 6d–l and Figure 7d–l. For airplane in trees, our region of interest is airplane, and for two men in front of house and lake, we highlight the tree branches. Here, the fine textures are greatly recovered in our results. For example, the tree branches corresponding to the test image pair10 and test image pair13 are clearly visible in Figure 6c and Figure 7c, whereas the results in Figure 6d–l and Figure 7d–l suffer from non-uniform illumination along the textures.

Finally, Figure 8 and Figure 9 compare the proposed method with the existing methods [17,20,22,33,34,35,36,37,38] using the results on test image pair16-soldier in trench and test image pair19-marne. Figure 8a,b represent the IR-VIS source images of soldier in trench and Figure 9a,b are the IR-VIS source images of marne. Figure 8c and Figure 9c show the results of the proposed MDSNet-MRSVD method. The fused images of the methods in [17,20,22,33,34,35,36,37,38] are shown in Figure 8d–l and Figure 9d–l. Here, we have selected the stairs present in soldier in trench and marne as our region of interest. We can notice that the stairs are clearer for our method in Figure 8c and Figure 9c compared to the existing methods. Moreover, our results provide good thermal information with minimized blurring effect.

4.4. Quantitative Results Analysis

We first performed qualitative analysis in Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9. It is based on human visual perception and usually focuses on the contrast, edge details, and sharpness of an image. However, the visual results alone are insufficient to prove the significance of the proposed method. Therefore, we also present quantitative analysis which is consistent with visual results and is based on PSNR, SSIM, $Q^{AB/F}$, $L^{AB/F}$ and $N^{AB/F}$. In

Table 1. Average PSNR, SSIM, $Q^{AB/F}$, $L^{AB/F}$ and $N^{AB/F}$ indices of different methods.

Method	PSNR	SSIM	${\mathit{Q}}^{\mathit{A}\mathit{B}/\mathit{F}}$	${\mathit{L}}^{\mathit{A}\mathit{B}/\mathit{F}}$	${\mathit{N}}^{\mathit{A}\mathit{B}/\mathit{F}}$	${\mathit{Q}}^{\mathit{A}\mathit{B}/\mathit{F}}+{\mathit{L}}^{\mathit{A}\mathit{B}/\mathit{F}}+{\mathit{N}}^{\mathit{A}\mathit{B}/\mathit{F}}$
Proposed	34.03	0.663	0.720	0.278	0.002	1
[17]	33.87	0.642	0.680	0.310	0.010	1
[22]	30.71	0.513	0.613	0.250	0.138	1
[33]	33.16	0.549	0.646	0.338	0.016	1
[20]	33.28	0.560	0.651	0.336	0.013	1
[34]	32.31	0.653	0.505	0.493	0.002	1
[35]	32.32	0.656	0.495	0.504	0.001	1
[36]	34.34	0.574	0.712	0.284	0.003	1
[37]	32.22	0.623	0.738	0.237	0.025	1
[38]	32.17	0.617	0.759	0.220	0.021	1

, we show different metrics of the proposed method averaged over 20 IR-VIS test image pairs. The individual metric results of different fusion methods are summarized in Figure 10.

As seen, the proposed method produces outstanding results in terms of PSNR, SSIM, $Q^{AB/F}$, and $N^{AB/F}$. The method in [36] and proposed method are capable of generating PSNR values above 34. In addition, the proposed method has the highest SSIM index of 0.663 compared to other methods. The methods in [37,38] and the proposed method have $Q^{AB/F}$ values of 0.759, 0.738, and 0.720, respectively. In addition to the proposed method, the methods in [35,36] are advantageous with smaller $N^{AB/F}$ indices. In terms of $L^{AB/F}$, the proposed method is comparable to the methods in [22,37,38].

The average runtime values corresponding to the proposed method and the existing methods [17,20,22,33,34,35,36,37,38] are given in

Table 2. Average runtime (in seconds) comparison of different methods on 20 IR-VIS test image pairs.

Proposed	[17]	[22]	[33]	[20]
9.26	14.52	32.56	12.79	3.29
[34]	[35]	[36]	[37]	[38]
7.41	0.24	0.70	9.16	0.07

. It can be noticed that the methods in [35,36,38] are the fastest methods that can execute in 1 s. The proposed method and the methods in [20,34,37] consume less than 10 s. In contrast, the methods in [17,22,33] are the slowest methods with execution times of 14.52, 32.56, and 12.79 s, respectively. Even though the methods in [35,36,38] are the fastest methods, the method in [35] yields very poor fusion performance $Q^{AB/F}$, resulting in high fusion loss $L^{AB/F}$. The methods in [36,38] suffer from poor SSIM indices with more noise compared to the proposed method. In particular, when better visual quality is desired with high PSNR, SSIM indices and low fusion loss, a slight increase in computational time is acceptable in view of improved computational facilities.

Thus, comparing all metrics together, the proposed method can be seen as the best method in terms of SSIM, second best performed in terms of PSNR and $N^{AB/F}$, and third best performed with $Q^{AB/F}$ metrics. Additionally, it gives comparable results in view of the $L^{AB/F}$ metrics as well.

5. Conclusions

In this correspondence, we propose a IR-VIS image fusion algorithm based on MDSNet and MRSVD updation. The proposed method does not demand source images with the same spatial resolution but still produces a high spatial resolution. The multi-degradation handling capabilities of MDSNet and IR image coefficient correction using MRSVD enrich the fused image with clear visible textures and thermal radiation information. Extensive visual comparisons on various IR-VIS test images demonstrate the qualitative superiority of our method. The proposed method yields good results on PSNR, SSIM, and $Q^{AB/F}$ with less fusion loss and artifcats as well.