Infrared Image Adaptive Enhancement Guided by Energy of Gradient Transformation and Multiscale Image Fusion

Abstract

The detail enhancement and dynamic range compression of infrared (IR) images is an important issue and a necessary practical application in the domain of IR image processing. This paper provides a novel approach to displaying high dynamic range infrared images on common display equipment with appropriate contrast and clear detail information. The steps are chiefly as follows. First, in order to protect the weak global details in different regions of the image, we adjust the original normalized image into multiple brightness levels by adaptive Gamma transformation. Second, each brightness image is decomposed into a base layer and several detail layers by the multiscale guided filter. Details in each image are enhanced separately. Third, to obtain the image with global details of the input image, enhanced images in each brightness are fused together. Last, we filter out the outliers and adjust the dynamic range before outputting the image. Compared with other conventional or cutting-edge methods, the experimental results demonstrate that the proposed approach is effective and robust in dynamic range compression and detail information enhancement of IR image.

1. Introduction

The infrared sensor can capture the thermal radiation emitted by the objects, which is less impacted by the dark condition. It is widely applied in detection, scene surveillance, reconnaissance, and navigation, etc. due to its ability to operate 24 h a day. However, IR images do have many obvious shortcomings, including low contrast, weak details, and blurred resolution, when compared with visible images, which may trigger much inconvenience when people observe the images. Consequently, infrared sensors in high dynamic range (>8 bit) are always applied in practical application to capture more details in these years. If displaying the HDR images on normal facilities (8 bit) directly, some information in the original image could not be represented. The procedure to achieve high-quality visualization of HDR infrared image must take the following problems into consideration. First and foremost, the dynamic range of the output should be mapped to be acceptable for the display device. Meanwhile, in order to take advantage of the HDR sensor and guarantee convenience for the following work, weak details should be enhanced. Last but not least, the output should be as visually pleasing as possible.

The core idea of many conventional approaches is adjusting the distribution of the gray level. Those methods usually include linear stretching and curve stretching (logarithmic, gamma, and sigmoid) methods based on histogram equalization [1] (HE) and gradient domain methods.

Linear stretching can compress the dynamic range to be acceptable for the display equipment, while it leads to detail losing. Curve stretching like Gamma transformations can increase the image contrast, while the fitting parameters change when the images are different. For each image, manual selection of parameters with experience is required.

HE enlarges the contrast of the image by redistributing the image pixel values so that the number of pixels in each gray level is approximately equal. Researchers have made plenty of efforts based on HE [2,3,4,5,6,7,8], including global HE-based approaches and local HE-based approaches. The global HE-based approaches like Plateau HE [3,4] and its improved ones [2,7] can make the gray levels distribute more reasonably, but the ability to preserve the details is insufficient. The local HE-based approaches like the partially overlapped sub-block histogram equalization (POSHE) [8] and contrast-limited adaptive histogram equalization (CLAHE) [5] can generate more details, but they are intended to induce artifacts, over-enhancement, and blocking effects. In short, most of the existing HE-based methods cannot avoid the contradiction between the maintaining of local details and the global consistency of the entire image, because they only take the histogram information into consideration.

The main idea of gradient domain [9,10] methods is attenuating the large gradient while expanding the small gradient to produce the modified gradient field, and then reconstruct the result image by solving a Poisson equation. Generally, gradient domain operators are capable of achieving appropriate dynamic range compression and avoiding artifacts like halo and gradient reversal. However, they may be limited in enhancing the local details effectively, and cautious selection of parameters is necessary, which limits the practical application of these methods.

Recently, multiscale decomposition methods are critical and prevalent in the domain of HDR image display. These methods introduce filters to decompose the image into several components followed by processing each component separately, then compose the processed components to yield the result image. F.Durand et al. [11] introduced a method to display HDR images. In their method, the image is decomposed into the base layer and the detail layer by the bilateral filter. Then, they attenuate the contrast of base layer, while the contrast of the detail layer is kept, thereby the details are preserved. Nowadays, an increasing number of researchers have adopted the use of a Guided Filter [12,13] to process the image for its simplicity and efficiency without the gradient reversal artifacts. B. Gu et al. [14] presented an edge-preserving filter with locally adaptive property, which is particularly effective in preserving or enhancing local details. Besides, researchers also applied hat-top transform [15,16] or wavelet transform [17] to decompose the image, then process the layers individually followed by composing them. Although multiscale decomposition methods have strength on details enhancing, they sometimes trigger halo artifacts in strong edges.

Currently, based on the human visual system (HVS), Retinex theory [18,19], researchers have presented various methods [20,21,22,23,24,25,26]. The critical challenge of those algorithms is the contradiction between calculation speed and the processing effect. On the one hand, better results can be obtained by constructing more complex models; on the other hand, the intricate structure of the model increases the calculation complexity, which limits the widespread application of the algorithm. Meanwhile, HVS-based approaches are more appropriate for the visual images with sufficient details rather than the infrared images lack of details in general.

In conclusion, the inadequacies of the existing methods mainly include (1) the contradiction between the maintaining of local details and the global consistency of the entire image, (2) too many parameters need to be selected manually with experience, and (3) poor robustness for dim images lack of details.

Nowadays, infrared and visible light images fusion [27,28] and the fusion of multi-exposure images [29,30] are both research hotspots in image processing. The visible light sensor mainly captures the reflected light so that the visible light image has abundant background information. In contrast, the infrared sensor can capture the thermal radiation emitted by the object, and it is less impacted by the dark condition or the dim weather. Therefore, the fusion of infrared and visible light images can guarantee more complex and detailed scene information. Similarly, each of multi-exposure images has its own unique details. If these details are well fused into one image, a high-quality image with multiple details can be produced.

This paper presents a novel approach based on adaptive transform and image fusion to overcome the problems above and display HDR infrared images on LDR display equipment with appropriate contrast and clear and abundant detail information. Inspired by the idea of image fusion, we transform the original image into multiple brightness by gamma transformation followed by multiscaled guided filter enhancement to keep and enhance the details in the entire image. In order to simplify the selection of parameters, we adapt the energy of the gradient (EOG) to guide the transformation, and the entropy is utilized to guide the multiscaled guided filter enhancement. The experimental results can prove that our method can achieve acceptable results with the fixed parameters. For typical HDR infrared images of various scenes, the effect of our method is robust.

The rest of this paper can be chiefly described as follows. Section 2 describes the fundamental theory and specific steps of our proposed method. In Section 3, our experiment comparison of the methods are described in detail. In Section 4, the conclusion of the paper is presented. Finally, the acknowledgment is made in Section 5.

2. Proposed Theory

The proposed framework is shown in Figure 1. First, in order to keep the weak global details in different areas, we adopt an adaptive Gamma transformation to adjust the original normalized image into multiple brightness levels. Second, the multiscale guided filter is utilized to decompose the images in different brightness individually into a base layer and detail layers. Details in each image are enhanced separately. Third, to obtain the image with global details of the original image, we fuse the enhanced images in each brightness together. Last, we filter out the bad pixels and adjust the dynamic range before outputting the image.

2.1. EOG Guided Gray Distribution Adjustment

Generally, the dynamic range of the HDR IR image (14 bit, 16 bit or more) far exceeds the dynamic range of the typical display range. Linear mapping is widely used due to its simplicity, but it is not suitable for most of the IR images whose gray levels are unevenly distributed. Different gamma correction parameters have different stretching effects on the image. A smaller gamma value can brighten the entire image and increase the contrast in darker areas; a larger gamma value can darken the entire image and increase the contrast in brighter areas. In order to keep the weak global details in different areas, we adjust the original normalized image into multiple brightness levels. However, manual selection of parameters with experience is required for each image in general, which generates inconvenience in the application.

2.1.1. Energy of Gradient

EOG is a well-established method for evaluating the clarity of the infrared image, due to its simplicity and accuracy. Energy of Gradient (EOG) is chosen as the evaluation criteria for the richness of image details. Let $f(x,y)$ be the value of the pixel $(x,y)$. The EOG can be calculated as follows,

$$EOG\left(im\right)=\sum _x\sum _y(f_x^2+f_y^2)$$

(1)

where $im$ is an image, and

$$f_x=f(x+1,y)-f(x,y)$$

(2)

$$f_y=f(x,y+1)-f(x,y)$$

(3)

2.1.2. EOG Guided Gray Distribution Adjustment

Let $S_i=\{S_{bright},S_{morderate},S_{dark}\}$ be set candidates of $\gamma$, and use the EOG function to evaluate and select the optimal value of $\gamma$ for image in each interval. Thus, the original image is adjusted to several images with rich details of multiple brightness levels.

Denote the original normalized image as $I_{input}$. Let $S_1$ (bright), $S_2$ (moderate), and $S_3$ (dim) be the three intervals:

$${\gamma }_i=\underset{\gamma \in S_i}{argmax}EOG\left(I_{input}^{\gamma }\right)i=1,2,3,$$

(4)

where $EOG\left(image\right)$ is the energy of the gradient of the image.

Through calculating, the value of $\gamma$ which can produce the image with the maximum EOG in each brightness intervals is picked out adaptively. Moreover, the details in different areas of the image can be kept separately.

2.2. Multiscale Guided Filter Enhancement

2.2.1. Multiscale Guided Filter Decomposition

He et al. [12] presented the guided image filter (GF), which not only benefits in edge-preserving but also be computationally efficient. Consequently, it is widely applied in the domain of image processing currently. We adopt the guided filter to decompose the image, in which the guide is identical to the filtering input I. The local linear model between the guide image I and the filter output Q is the critical assumption of the guided filter. Q is a linear transformation of I in a window ${\omega }_k$ centered at the pixel k.

$$Q_i=a_k{I}_i+b_k\forall i\in {\omega }_k$$

(5)

where

$$a_k=\frac{{\sigma }_k^2}{{\sigma }_k^2+\epsilon }$$

(6)

$$b_k=\overline{I_k}-a_k{\mu }_k$$

(7)

Here, $\left|\omega \right|$ is the number of pixels of ${\omega }_k$, ${\mu }_k$ is the mean of I in ${\omega }_k$, ${\sigma }_k^2$ is the variance of I in the ${\omega }_k$, and $\overline{I_k}$ is the mean value of I in ${\omega }_k$.

When the area has rich details, the ${\sigma }_k^2$ is relatively large, $a_k$ approaches 1, and $b_k$ tends to 0; the guided filter can keep the details in the local area. When the area has rich details, the ${\sigma }_k^2$ is relatively small, $a_k$ approaches 0, and $b_k$ tends to $\overline{p_k}$; the guided filter behaves as a weighted mean filter. $\epsilon$ is a parameter depended on the image information, which determines whether it is an edge should be preserved.

Therefore, the guided filter behaves as an edge-preserving smoothing operator. For simplicity, we refer to it as $Q=GF\left(I\right)$. Q can be regarded as the base layer of the input image I, which contains the low-frequency information of the input image I, reflects the intensity change of the image on a large scale, while $(I-Q)$ can be regarded as the detail layer, which contains the high-frequency information of input image I, reflecting the details of the image on a small scale.

As introduced above, we can obtain a smoothed base layer and a detail layer by the guided filter. In order to obtain more complete details, we could utilize the guided filter iteratively to obtain the multiscale smoothed images. Meanwhile, the multiscale detail images can be generated. The specific procedure can be described as follows.

$$\left\{\begin{array}{ccc}{B}_1=GF\left(I\right)\hfill & & \\ D_1=I-B_1\hfill & & \\ B_k=GF\left(B_{k-1}\right)\hfill & & k=2,3,\dots \hfill \\ D_k=B_{k-1}-B_k\hfill & & k=2,3,\dots \hfill \end{array}\right.$$

(8)

The $B_i$ is the ith base layer while the $D_i$ is the ith detail layer. Then, we can decompose the original image as follows.

$$I=B_n+D_1+D_2+\dots +D_n$$

(9)

Specifically, in our study and experiment, as is shown in the Figure 1, we decompose the image into three layers: one base layer and two detail layers. Therefore, the multiscale decomposition progress can be described as

$$I=B_2+D_1+D_2$$

(10)

2.2.2. Adaptive Multiscale Guided Filter Composition

Each image can be decomposed into several layers; the details in the infrared image are typically weak. As is shown in Figure 2, the input image is one of the EOG guided transformed images, there are rich details captured by the HDR infrared sensors in the detail layers, but they are too weak to be observed.

The composition of the base layer and detail layers can be described as follows.

$$I_{enhanced}=B_n+{\alpha }_1{D}_1+{\alpha }_2{D}_2+\dots +{\alpha }_n{D}_n$$

(11)

The layers are linearly accumulated together, the value of each coefficient ${\alpha }_i$ gives expression to the importance of the ith detail layer, the more the information in the layer, larger the ${\alpha }_i$. In order to adaptively choose the value of ${\alpha }_i$, we adopt entropy to evaluate the richness of information in each layer:

$$Entropy\left(im\right)=-\sum _{i=0}^1{p}_{i}log{p}_i$$

(12)

where $p_i$ is the probability of gray level i in the image.

$${\alpha }_i=\frac{C·Entropy\left(D_i\right)}{{\sum }_{k=1}^{n}Entropy\left(D_k\right)+z}i=1,2,\dots ,n$$

(13)

C is a fixed coefficient and z is a very small number added to prevent the denominator from being 0.

In Figure 2 the effect of the step is shown; the weak details are enhanced. The weak details in detail layers (b,c) become much clearer in (e,f). What needs to be explained is that the figures of detail layers are stretched 10 times for better visibility; the details in fact are much weaker.

The effect of the proposed multiscale guided filter enhancement in this section can also be obviously reflected in Figure 3. There are two groups of images: panels (a,c) are results of the proposed method without the multiscale guided filter enhancement, in which the information is ambiguous, while panels (b,d) are the results of proposed method, which is much more visually comfortable.

2.3. Image Fusion

Inspired by the research hotspots including infrared and visible light images fusion and the fusion of multi-exposure images, which aim at fusing details in different images of a same scene to obtain an image with rich information, we fuse the images generated by previous steps. We adopt a method [29] with clear mathematical principles and high computational efficiency.

Through the steps above, a set of enhanced images in different brightness from the original image can be generated. We regard those images as multi-exposed images. In order to maintain local details well, we block the image in the fusion. Let $\left\{i_k^n\right\}=\left\{i_k^n\right|1<k<K\}$ be a set of column vectors of $N^2$ dimensions expanded from the blocks at the identical location of the source images that contains K images in multi-brightness. k means the block is from the kth image of the set, n means the nth location. The elements of the vector are value of each pixel in the image block. N is the side length of the image block. n corresponds to the position of the patch in the entire image. In order to express, analyze, and process the feature of a block, the vector $i_k^n$ can be decomposed into three components, including signal strength $p_k^n$. Signal structure $s_k^n$ and mean intensity ${\mu }_{i_k^n}$. The definitions of the components are chiefly as follows.

$$i_k^n=\parallel i_k^n-{\mu }_{i_k^n}\parallel ·\frac{i_k^n-{\mu }_{i_k^n}}{\parallel i_k^n-{\mu }_{i_k^n}\parallel }+{\mu }_{i_k^n}=p_k^n·s_k^n+{\mu }_{i_k^n}$$

(14)

${\mu }_{i_k^n}$ is a vector, in which all the elements equal to the mean value of $i_k^n$.

$$p_k^n=\parallel i_k^n-{\mu }_{i_k^n}\parallel$$

(15)

$$s_k^n=\frac{i_k^n-{\mu }_{i_k^n}}{\parallel i_k^n-{\mu }_{i_k^n}\parallel }$$

(16)

Obviously, the contrast of an image block can be directly reflected by the signal strength component $p_k^n=\parallel i_k^n-{\mu }_{i_k^n}\parallel$. Generally speaking, the higher the contrast, the clearer the block or image. While the excessive contrast may trigger an unrealistic scene. Considering the input images (blocks) are undistorted, we could assume that the block has the largest contrast corresponds to the optimal visibility. Therefore, we choose the highest signal strength of all source image blocks as the signal strength of the fused image block:

$$\widehat{p_n}=\underset{1<k<K}{max}{p}_k^n$$

(17)

Determine the structure of the set of image blocks as a series of unit length vectors $s_k^n=\frac{i_k^n-{\mu }_{i_k^n}}{\parallel i_k^n-{\mu }_{i_k^n}\parallel }(1\le k\le K)$, and each one points to a direction in the vector space. The structure of the fused image block should represent the structures of the series of image blocks. Specifically, the relationship between the structure of fused block and the input blocks is defined in a simple but effective way:

$$\overline{s_n}=\frac{{\sum }_{k=1}^K{\parallel p_k^n\parallel }^{\rho }·s_k^n}{{\sum }_{k=1}^K{\parallel p_k^n\parallel }^{\rho }}$$

(18)

$$\widehat{s_n}=\frac{\overline{s_n}}{\parallel \overline{s_n}\parallel }$$

(19)

The definition of mean intensity of each block is

$$\widehat{{\mu }_n}=\frac{{\sum }_{k=1}^{K}L({\mu }_k,{\mu }_{i_k^n})·{\mu }_{i_k^n}}{{\sum }_{k=1}^{K}L({\mu }_k,{\mu }_{i_k^n})}$$

(20)

where $L({\mu }_k,{\mu }_{i_k^n})$ is a weighting function which is controlled by the mean value of the kth whole image ${\mu }_k$ and the mean value of the current block ${\mu }_{i_k^n}$ in the kth image. $L(·)$ should be relatively large when the block $i_k^n$ is in a well-exposed region, and vice versa. To specify it, we adopted a two-dimensional Gaussian function:

$$L({\mu }_k,{\mu }_{i_k^n})=exp-[\frac{{({\mu }_k-0.5)}^2}{2{\sigma }_g}+\frac{{({\mu }_{i_k^n}-0.5)}^2}{2{\sigma }_l}]$$

(21)

When signal strength $\widehat{p_n}$, signal structure $\widehat{s_n}$, and mean intensity $\widehat{{\mu }_n}$ are computed, the new vector $\widehat{i_n}$, which means the vector of the fused image block, can be defined and the block can be reconstructed:

$$\widehat{i_n}=\widehat{p_n}·\widehat{s_n}+\widehat{{\mu }_n}$$

(22)

The blocks from the source sequence are obtained by a moving window with a fixed stride D. The pixels in the overlapping blocks are averaged to produce the final output of this step.

2.4. Outliers Filtering

Generally, there are still some outliers in the image, which are usually the brightest or darkest. Specifically, the maximum or minimum value in the image may be the outliers of the image, which affect the result of dynamic range adjustment. To cope with the problem, we have adopted a simple and effective method.

For instance, to avoid manual selection of the parameters with experience, we assume that there are two outliers in each row or in each patch with fixed size a. Take an image in size of $a\times b$ as the example. First, take every pixel value in the whole image in descending order. Then, pick out the $ath$ value as the minimum value $f_{min}$ and the $ath$ last value as the maximum value $f_{max}$. Finally, adjust the image according to the effective values.

$$I_{output}=255\ast \frac{\widehat{I}-f_{min}}{f_{max}-f_{min}}$$

(23)

3. Experiment Results

3.1. Experimental Settings

In order to measure the effect and the efficiency of the proposed method.Multiple 16 bits infrared images selected from typical scenes in databases FLIR Thermal Starter Dataset Version 1.3 [31] and LTIR Dataset Version 1.0 [32] were utilized for testing. The information of the images including image size and dynamic range is listed in

Table 1. Basic information of the test images.

Image	Size	Min (Gray Value)	Max (Gray Value)	% of the Total Gray Levels
IM1	480*640	0	16961	25.880
IM2	480*640	9893	16339	9.835
IM3	512*640	1411	3575	3.302
IM4	512*640	5683	8188	3.822
IM5	512*640	6433	8653	3.387
IM6	512*640	5984	11322	8.145
IM7	512*640	6010	10000	6.088

. Meanwhile, four well-established methods (HE [1], CLAHE [5], MSR [23], and Reinhard [24]) and two novel approaches (AHPBC [6] and LEP [14]) were introduced for comparison. In those methods, we select the parameters as the authors advised or with experience.

In Section 2.1.2 (Equation (4)), the three brightness intervals (bright $S_1$, moderate $S_2$, and dark $S_3$) are set as follows, $S_1=[0.1,0.7]$, $S_2=(0.7,1.5]$, and $S_3=(1.5,8]$.

In Section 2.2.1 (Equation (6)), we set the value of $\epsilon$ be related to the variance of the entire image, because the effect of $\epsilon$ is determining whether it is an edge should be preserved. In our experiment, $\epsilon ={\sigma }_i^3$, where ${\sigma }_i$ is the variance of the entire image.

In Section 2.2.2 (Equation (11)), the values of ${\alpha }_k$ determine the enhancement of the details. Throughout our experiment, we obtain two detail layers, and in Equation (13), $C=7$ and $z=0.0001$.

In Section 2.3 (Equation (18)), $\rho$ determines the contribution of each block to the fused block’s structure. Obviously, the contribution increases along with the strength of the block. Theoretically, $\rho >0$ is feasible. We set $\rho =4$ in our experiment. In Equation (21), ${\sigma }_g$ and ${\sigma }_l$ control the spread of the profile along ${\mu }_k$ and $l_k$. We set ${\sigma }_g=0.2$ and ${\sigma }_l=0.5$, a smaller value of ${\sigma }_g$ relative to ${\sigma }_l$ is important to generate results with good visual impression. Additionally, we set the size of blocks and the moving window stride: $N=11$ and $D=2$, as the author advised.

Throughout the paper, parameters mentioned above are adopted for typical infrared images with different characteristics. Results have demonstrated that the proposed method is capable of effectively enhancing IR image.

3.2. Visual Comparisons

To compare the effects of the methods intuitively, the enhanced results of the algorithms are given in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. We discuss the results in detail in Section 4.

3.3. Quantitative Comparison

Generally, good display performance means high clarity and even gray level distribution. In order to do the quantitative comparison, the Tenengrad [33], Entropy, Naturalness Image Quality Evaluator (NIQE) [34], and Perception-based Image Quality Evaluator (PIQE) [35] are introduced. They are widely used in evaluating the quality of an image.

The Tenengrad is written as

$$G_x=\left[\begin{array}{ccc}1& 0& -1\\ \\ 2& 0& -2\\ \\ 1& 0& -1\end{array}\right]$$

(24)

$$G_y=\left[\begin{array}{ccc}1& 2& 1\\ \\ 0& 0& 0\\ \\ -1& -2& -1\end{array}\right]$$

(25)

$$S(x,y)=\sqrt{G_x\ast I(x,y)+G_x\ast I(x,y)}$$

(26)

$$Tenengrad=\frac{1}{n}\ast \sum _x\sum _{y}S(x,y)$$

(27)

where $I(i,j)$ denotes the gray value of the pixel $(x,y)$, * denotes convolution, and n is the number of pixels in image.

The Tenengrad is utilized to reflect the clarity of the whole image. Theoretically, the larger the Tenengrad value is, the higher the contrast, and the better the visibility of the details of the image. The calculation Tenengrad results are listed in

Table 2. The Tenengrad’s value of the images.

Image	Linear Mapping	HE	CLAHE	AHPBC	MSR	Reinhard	LEP	Proposed
IM1	4.2562	18.8087	15.7123	10.5995	4.2137	11.1091	21.5385	22.3971
IM2	4.9403	12.3800	7.4490	4.7770	4.8471	4.7601	11.1299	18.1054
IM3	4.4459	8.9382	4.2856	4.7390	4.1821	4.2448	11.0855	18.5358
IM4	3.1144	4.4115	2.0874	3.6231	3.1346	3.1205	5.9977	7.8913
IM5	4.6191	5.1433	3.9676	4.6468	4.4070	4.6040	9.5314	14.9446
IM6	1.4227	7.8863	2.9639	2.7526	1.4158	1.6008	8.4747	6.7918
IM7	2.0778	8.7989	2.9121	2.9168	2.0738	2.2411	8.9327	6.8261

.

The even distribution of the pixel values of the image is another goal of image enhancement. And the entropy of an image is a common approach to reflect the pixel value’s distribution. Specifically and theoretically, the larger the entropy value is, the more evenly the gray levels distributed. The entropy of an 8 bit image is written as

$$Entropy\left(im\right)=-\sum _{i=0}^{255}{p}_{i}log{p}_i$$

(28)

where $p_i$ is the probability of gray level i in the image. The calculation Entropy results are listed in

Table 3. The entropy of the images.

Image	Linear Mapping	HE	CLAHE	AHPBC	MSR	Reinhard	LEP	Proposed
IM1	5.5735	3.4553	7.0238	5.3511	5.4562	7.3206	7.6118	7.5752
IM2	5.7655	2.5542	6.5089	5.6805	6.0352	5.6525	6.7636	7.1654
IM3	6.3733	2.2374	5.1676	6.1961	6.6940	6.2290	7.0587	7.4246
IM4	5.9289	1.7727	5.1401	5.7168	6.4312	5.9052	5.8555	6.6476
IM5	7.5017	2.8809	5.8623	7.4927	7.4540	7.5118	7.4614	7.8229
IM6	4.5174	1.5771	5.2951	4.3780	4.7277	4.6765	6.1891	6.3499
IM7	4.5504	1.1825	4.5859	4.5556	4.7223	4.6450	5.8029	5.8444

.

NIQE measures the distance between the NSS-based features calculated from image to the features obtained from an image database used to train the model. The features are modeled as multidimensional Gaussian distributions. We calculate it by the Matlab function $niqe\left(\right)$, which returns a non-negative scalar. Theoretically, the lower value of NIQE is, the better the perceptual quality of the image. The results are listed in

Table 4. The Naturalness Image Quality Evaluator (NIQE) values of the images.

Image	Linear Mapping	HE	CLAHE	AHPBC	MSR	Reinhard	LEP	Proposed
IM1	2.4030	7.6034	4.7557	2.3148	2.3713	2.1673	2.5625	2.8865
IM2	3.1617	8.7744	5.8055	3.3352	3.2184	3.2078	3.4816	4.0588
IM3	3.3359	11.5356	7.4972	3.8602	3.3382	3.2956	3.9761	4.3825
IM4	7.5140	11.9366	7.5140	4.2840	4.8551	4.7213	4.2182	4.8106
IM5	3.5719	15.0618	7.2863	3.4961	3.6931	3.6239	3.6369	3.4917
IM6	4.9581	13.3283	7.4248	4.9065	4.8822	4.5561	3.9335	4.0222
IM7	3.9773	11.7403	7.9685	3.7186	3.9794	3.8734	3.5809	3.5237

.

PIQE calculates the no-reference quality score for an image through block-wise distortion estimation. We calculate it by the Matlab function $piqe\left(\right)$, which returns a non-negative scalar in the range [0, 100]. The PIQE score is inversely correlated to the perceptual quality of an image. A low PIQE value indicates high perceptual quality and high PIQE value indicates low perceptual quality. The results are listed in

Table 5. The Perception-based Image Quality Evaluator (PIQE) values of the images.

Image	Linear Mapping	HE	CLAHE	AHPBC	MSR	Reinhard	LEP	Proposed
IM1	33.7017	65.3970	59.9335	33.43665	33.3354	21.2885	22.8901	24.9244
IM2	30.8086	76.7303	74.3567	32.4350	32.8172	34.5472	20.3241	22.4031
IM3	16.0431	82.7883	79.3746	16.5842	19.2051	18.4810	13.8669	30.9339
IM4	74.8258	81.6471	74.8258	35.7224	37.2663	39.0461	36.4915	16.0016
IM5	52.9848	82.5220	80.2671	52.7876	54.4075	49.0024	49.4377	43.2992
IM6	65.9284	81.1321	77.7412	62.9356	64.9387	58.8325	20.6870	16.2914
IM7	58.0052	79.8022	74.0387	56.8523	52.6654	54.8697	16.8590	16.2839

.

3.4. Running Time Comparison

In order to do an efficiency comparison, the above-listed algorithms are tested, using MATLAB R2018b on a personal computer (Intel core i5-8250U; CPU:1.60 GHz; Memory: 8 GB). The size of the tested images are listed in Table 1. The calculation time results are listed in

Table 6. Running time of the test images. Unit: second.

Image	HE	CLAHE	AHPBC	MSR	Reinhard	LEP	Proposed
IM1	0.1140	0.1999	27.7791	0.9225	0.0257	0.8482	2.5096
IM2	0.1195	0.1439	26.8293	0.1770	0.0284	1.0943	2.4500
IM3	0.1111	0.1455	30.8719	0.1070	0.2198	1.1543	2.6872
IM4	0.1158	0.1387	30.6425	0.1040	0.0202	1.1325	2.7946
IM5	0.1253	0.1422	40.4624	0.0990	0.0207	1.0314	2.6288
IM6	0.1112	0.1434	28.5309	0.0981	0.0208	1.2840	2.5608
IM7	0.1463	0.1869	27.4465	0.9937	0.0203	2.1250	2.7588

.

4. Discussion

Image group Figure 4 is an example of infrared images of rich scene information including human, bicycles, benches, ground, and so on. HE and CLAHE enhanced the contrast, while a large amount of local details lost. The AHPBC and MSR can enhance the details to some extent, while the dynamic range of the result image is so small that the visibility is poor. The result of Reinhard is visually comfortable but the some texture information is still ambiguous. LEP can enhance the image well in general, but generates a halo. Compared with other six approaches, our method meets the best performance.

Image group Figure 5 and Figure 6 are examples of low contrast image, which contains many details about the texture. The dynamic range of the original IR image is so narrow that HE and CLAHE fail in the enhancement of the details, while some regions in their results are over-enhanced, and some noises are generated. AHPBC, MSR, and Reinhard can preserve the global contrast but has a relatively weak compatibility in the enhancement of the local details. LEP can successfully enhance the edge of the humans in the image, but some tiny details like the texture of the road are still dim. Our method yields the best enhancement results, producing global detail enhancement without noise generation.

Image group Figure 7 and Figure 8 are examples of foggy images. Details like outlines of trees and human are unobservable in the results of HE and CLAHE, and the backgrounds are distorted. Compared with the original linear mapped image, the results of AHPBC and Reinhard are still blurred, even though there might be a great change in brightness. MSR can increase the contrast to some extent, but the effect on local detail enhancement is relatively weak. The noise in the result of LEP is obvious. It can be indicated by the comparison of the results in Figure 7 and Figure 8 that the proposed method creates the most visually comfortable results, which reveal the details most fully.

Image group Figure 9 and Figure 10 are examples of image with blurred details. Due to low contrast and weak details in the original image, HE and CLAHE not only fail to reproduce the details, but also generate noises. Objects such as trees, buildings, and pedestrians in AHPBC’s results are blurred. The results of MSR and Reinhard are too dark to observe the information. Relatively, the results of LEP and the proposed method are visually pleasing; comparing with the results of LEP, noise in the proposed results is weaker.

The results of the Tenengrad for the test images are shown in the Table 2. In theory, the higher value of Tenengrad, the clearer the entire image. In accordance with the result of visual comparisons, the proposed method and LEP achieve higher Tenengrads.

As being reported in Table 3, comparing about the entropy, the proposed method and LEP have the robust result, and our method obtains the slightly better value than LEP does. Practically, there are more details in the results of our proposed method.

As being reported in Table 4, comparing about the NIQE, lower value of NIQE reflect better perceptual quality of image. the overall difference of AHPBC, MSR, Reinhard, LEP, and our proposed method is not obvious.

As being reported in Table 5, lower value of NIQE reflect better perceptual quality of image. In general, the proposed method and LEP achieve better results, but the average result of our proposed method is the best.

As being reported in Table 6, as our approach introduce multiscale analysis and image fusion, the calculation time of the proposed algorithm is much more than the conventional and famous methods HE, CLAHE, MSR, and Reinhard. Our method runs relatively slower than LEP, but more quickly than AHPBC. How to accelerate our algorithm is one of the key points of our future work. Hardware acceleration is one of our choices. After optimization, our method is very likely to process image on real-time application.

All in all, performance of the proposed algorithm is verified by experiments with images with various characteristics. The above analysis of the results shows that the proposed method has strength in detail enhancement of the HDR infrared image. The dynamic range compression and detail enhancement results are visually comfortable without excessively obvious noise.

5. Conclusions

In this paper, a novel high dynamic range infrared image enhancement method is introduced. This method is capable of compressing the dynamic range, adjusting the gray levels, and enhancing the details effectively. The proposed approach is mainly based on adaptive Gamma correction, multiscale guided filter, and image fusion. First, in order to keep the weak global details in different area, we adopt an EOG-guided Gamma transformation, which is adaptive to adjust the original normalized image into multiple brightness levels. Second, the multiscale guided filter is utilized iteratively to decompose each brightness image into a base layer and several detail layers. Details in each image are enhanced separately and composed adaptively. Third, to obtain the image with global details of the input image, enhanced image in each brightness is fused together. Last, we filter out the bad pixels and adjust the dynamic range before outputting the image. Tested on HDR IR images of different scenes with sundry details and background, the experiment result indicates that the proposed method can compress the dynamic range while higher the contrast, enhance the details effectively, and generate a visually pleasing result. It should be pointed out that in the step of guided transformation, the EOG function is just chosen to guarantee the simplicity and correctness of the algorithm. That is to say, the function could be changed according to the case with flexibility in the future work. Meanwhile, the method of the enhancement of the decomposed layers could also be extended, which also provides new point for the research.