Total Variation Based Neural Network Regression for Nonuniformity Correction of Infrared Images

Abstract

Many existing scene-adaptive nonuniformity correction (NUC) methods suffer from slow convergence rate together with ghosting effects. In this paper, an improved NUC algorithm based on total variation penalized neural network regression is presented. Our work mainly focuses on solving the overfitting problem in least mean square (LMS) regression of traditional neural network NUC methods, which is realized by employing a total variation penalty in the cost function and redesigning the processing architecture. Moreover, an adaptive gated learning rate is presented to further reduce the ghosting artifacts and guarantee fast convergence. The performance of the proposed algorithm is comprehensively investigated with artificially corrupted test sequences and real infrared image sequences, respectively. Experimental results show that the proposed algorithm can effectively accelerate the convergence speed, suppress ghosting artifacts, and promote correction precision.

1. Introduction

Infrared imaging systems generally suffer from the detector’s nonuniform response, so-called nonuniformity (NU) or fixed pattern noise (FPN), which strongly degrades the resolving capability of the infrared image. In order to eliminate the NU or FPN, the fundamental solution is to develop more sophisticated techniques and new materials in the manufacture of infrared sensors. However, the abovementioned development is a slow and difficult process. Therefore, nonuniformity correction (NUC), being an alternative solution based on signal processing methods, is now applied to most infrared imaging applications.

NUC methods are mainly classified into two categories: reference-based NUC (RBNUC) and scene-based NUC (SBNUC). RBNUC employs the response of blackbody radiation at different temperatures to calculate the gain and offset parameter by using linear or high-order fitting. RBNUC is not applicable to many applications because its parameter calibration process inevitably interrupts the normal imaging operation. Even more serious, spatial nonuniformity tends to drift slowly over time, which leads to the failure of RBNUC methods with fixed parameters. To address this problem, many SBNUC approaches have been developed to minimize the correction error resulting from the drifting response of infrared focal plane array (IRFPA) sensors [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16].

SBNUCs are motion-dependent and easily affected by extreme scenes, which makes most of them face the problem of ghosting artifacts. These artifacts are generally generated when global scene movement happens or partial content of the scene slows down or abruptly halts [8]. To suppress the ghosting effect, Harris et al. designed a deghosting module in the constant-statistics method [2,3]. Qian et al. introduced spatial filtering to promote the temporal high-pass filtering method [4], which was then ameliorated by Bai et al. [5] and Zuo et al. [6]. Moreover, the adaptive learning rate [7,8,9], edge-directed parameter estimation [10], edge-preserving spatial filter [11,12,13,14], and temporal statistics thresholding [15] strategies have been proposed in the neural network NUC (NN-NUC) algorithm to prevent ghosting artifacts and detail loss. The abovementioned efforts have been confirmed to be effective but still deficient. Serious ghosting artifacts accompanied with slow convergence speed is still a challenging problem to be solved.

In seeking minor residual correction error and fast convergence rate, we present a novel scene adaptive NUC algorithm in this paper. Specifically, the proposed adaptive NUC method relies on the employment of a total variation (TV) penalty to solve the overfitting problem in the linear regression estimation of correction parameters, which helps to eliminate FPN as possible and preserve the details in the iterative correction process. Moreover, an adaptive gated learning rate is presented to further suppress the ghosting artifacts and retain a higher convergence rate.

The rest of this paper is organized as follows. Section 2 starts with a detailed description of the typical NN-NUC method, and then the proposed TV-regularized NN-NUC algorithm along with its highlight improvements is presented and discussed in Section 3. Following that, the performance of the proposed algorithm is respectively verified with artificially-corrupted sequences and real infrared sequences in Section 4. The paper is ended with a summary in Section 5.

2. Typical NN-NUC Method

Based on the vision bionics research, Scribner et al. presented a retina-like neural-network structure and its corresponding NUC method [1]. The NN-NUC method links every nerve cell in a hidden layer to a pixel in an input layer and implements the calculation of the error function in the hidden layer. Following that, it minimizes the square error function to get the correction parameters iteratively in the output layer based on a least mean square (LMS) algorithm. Finally, the NUC for the IRFPA is implemented according to those parameters.

The response of each detector in IRFPA is generally considered to be linear in most of its dynamic range, the relationship between the scene irradiance $x_{ij}$ and the detector response $y_{ij}$ can be expressed as

$$y_{ij}=a_{ij}{x}_{ij}+b_{ij},$$

(1)

where $a_{ij}$ and $b_{ij}$ are gain and offset parameters associated to the $\left(i,j\right)$ detector, respectively.

According to the linear response model defined in Equation (1), the calibrated output of the $\left(i,j\right)$ detector in IRFPA can be represented as

$${\widehat{X}}_{ij}={\widehat{W}}_{ij}^T\cdot Y_{ij},$$

(2)

where ${\widehat{X}}_{ij}$ and $Y_{ij}={\left(y_{ij},1\right)}^T$ denote the corrected value and the observation value, respectively. ${\widehat{W}}_{ij}={\left({\widehat{g}}_{ij},{\widehat{o}}_{ij}\right)}^T$ stands for the correction parameter matrix, in which ${\widehat{g}}_{ij}=1/a_{ij}$ and ${\widehat{o}}_{ij}=-b_{ij}/a_{ij}$ are the correction parameters for gain and offset, respectively.

Since LMS techniques are utilized to iteratively estimate the correction parameter, the needed error function can be defined as

$$E_{ij}={\left({\widehat{X}}_{ij}-D_{ij}\right)}^2,$$

(3)

where the desired target value $D_{ij}$, being the output of the $\left(i,j\right)$ neuron located at the hidden layer, is calculated by

$$D_{ij}=\frac{1}{{\left(2r+1\right)}^2}{\displaystyle \sum _{m=-r}^r{\displaystyle \sum _{n=-r}^r{Y}_{i+m,j+n}}},$$

(4)

where $r$ denotes the radius of the kernel.

Thereafter, a stochastic gradient descent algorithm is applied to seek ways to minimize the error function $E_{ij}$, and yields the recursive update of correction parameter matrix ${\widehat{W}}_{ij}$. Following that, the adaptive correction results can be obtained from the output layer according to Equation (2).

3. TV-Regularized NN-NUC Algorithm

3.1. Scheme Description

In order to solve the overfitting problem in the LMS regression of the traditional NN-NUC method, a total variation penalty is introduced into the cost function. In the newly presented algorithm, the TV-regularized cost function can be expressed as

$$J_{ij}\left(n\right)=E_{ij}\left(n\right)+\delta \cdot {\mathsf{\Upsilon }}_{TV}\left({\widehat{X}}_{ij}\left(n\right)\right),$$

(5)

where $n$ stands for the frame index, the error function. $E_{ij}\left(n\right)$ acts as the fidelity term to preserve the characters of the original image, and the parameter $\delta$ is a scalar for properly weighting the similarity cost against the regularization term ${\mathsf{\Upsilon }}_{TV}\left({\widehat{X}}_{ij}\left(n\right)\right)$.

In Equation (5), the TV criterion penalizes the entire change in the corrected image as quantified by the $L_1$ norm of the gradient magnitude

$${\mathsf{\Upsilon }}_{TV}\left({\widehat{X}}_{ij}\left(n\right)\right)={\Vert \nabla {\widehat{X}}_{ij}\left(n\right)\Vert }_1,$$

(6)

where $\nabla$ is gradient operator. The advantage of the TV criterion is that it tends to preserve edges in the denoising without severely penalizing steep local gradients [17,18].

According to the steepest descent method, we minimize the cost function and deduce the iterative formula of the correction parameter as

$${\widehat{W}}_{ij}\left(n+1\right)={\widehat{W}}_{ij}\left(n\right)+{\mu }_{ij}\left(n\right)\cdot G_p\left(J_{ij}\left(n\right)\right),$$

(7)

where $G_p(\cdot )$ represents a partial derivative operator, ${\mu }_{ij}\left(n\right)$ is a learning rate that governs the convergence of the algorithm, and the vector ${\widehat{W}}_{ij}\left(n\right)$ that minimizes the Equation (5) will be the solution to

$$G_p\left(J_{ij}\left(n\right)\right)=\{\begin{array}{l}\frac{\partial }{\partial \widehat{g}}\left[J_{ij}\left(n\right)\right]=0\\ \frac{\partial }{\partial \widehat{o}}\left[J_{ij}\left(n\right)\right]=0\end{array}$$

(8)

Thereafter, incorporating the solution of Equation (8) into Equation (7), we can respectively obtain the iterative formula of estimated gain and offset correction parameter

$${\widehat{g}}_{ij}\left(n+1\right)={\widehat{g}}_{ij}\left(n\right)-{\mu }_{ij}\left(n\right)\cdot \left\{F_{ij}\left(n\right)+\delta \cdot R_{ij}\left(n\right)\right\}\cdot y_{ij}\left(n\right),$$

(9)

$${\widehat{o}}_{ij}\left(n+1\right)={\widehat{o}}_{ij}\left(n\right)-{\mu }_{ij}\left(n\right)\cdot \left\{F_{ij}\left(n\right)+\delta \cdot R_{ij}\left(n\right)\right\},$$

(10)

where the components representing the derivative of fidelity and penalty terms can be respectively expressed as

$$F_{ij}\left(n\right)={\widehat{X}}_{ij}\left(n\right)-D_{ij}\left(n\right)$$

(11)

and

$$R_{ij}\left(n\right)=div\cdot \left(\frac{\nabla {\widehat{X}}_{ij}\left(n\right)}{\sqrt{{\left|\nabla {\widehat{X}}_{ij}\left(n\right)\right|}^2+\epsilon }}\right),$$

(12)

where $div(\cdot )$ denotes the divergence operator, and $\epsilon =1\times {10}^{-6}$ is a small constant added to avoid non-differentiability of the TV penalty.

Finally, the calibrated output of the $\left(n+1\right)$th iteration can be given by

$${\widehat{X}}_{ij}(n+1)={W_{ij}}^T(n+1)\cdot Y_{ij}(n+1).$$

(13)

According to the previously described theory, the complete scheme representing the whole process of the proposed TV-regularized neural-network NUC (TVRNN-NUC) method is presented in Figure 1.

3.2. Gated Adaptive Learning Rate

In order to eliminate burn-in ghosting from lack of scene motion, we present a gated adaptive learning rate ${\mu }_{ij}\left(n\right)$, which can be defined as

$${\mu }_{ij}\left(n\right)=\{\begin{array}{ll}\frac{{\eta }_{ij}\left(n\right)}{1+{\sigma }_{Y_{ij}\left(n\right)}},& \left|D_{ij}\left(n\right)-B_{ij}\left(n-1\right)\right|>K\\ 0,& else\end{array}$$

(14)

and

$$B_{ij}\left(n\right)=\{\begin{array}{ll}{D}_{ij}\left(n\right),& \left|D_{ij}\left(n\right)-B_{ij}\left(n-1\right)\right|>K\\ B_{ij}\left(n-1\right),& else\end{array},$$

(15)

where $K$ is the threshold value that determines the updating of learning rate, ${\sigma }_{Y_{ij}\left(n\right)}$ stands for the local standard deviation of the current observation $Y_{ij}\left(n\right)$, which makes ${\mu }_{ij}\left(n\right)$ increase in flat regions and decrease in texture regions. $B_{ij}\left(0\right)=\infty$ is defined to ensure that $\left|D_{ij}\left(1\right)-B_{ij}\left(0\right)\right|>K$ for all $i,j$, and the variable step size coefficient that further solves the problem of decreasing convergence rate caused by gated strategy is given by

$${\eta }_{ij}\left(n\right)=\{\begin{array}{llll}{\eta }_{\max },& & if& {\eta }_{ij}^{\prime }\left(n\right)\ge {\eta }_{\max }\\ {\eta }_{\min },& & if& {\eta }_{ij}^{\prime }\left(n\right)\le {\eta }_{\min }\\ {\eta }_{ij}^{\prime }\left(n\right),& & else& \end{array}$$

(16)

and

$${\eta }_{ij}^{\prime }\left(n\right)=\alpha \cdot {\eta }_{ij}\left(n-1\right)+\beta \cdot E_{ij}\left(n\right),$$

(17)

where $0<{\eta }_{\min }<{\eta }_{\max }$. The constant ${\eta }_{\max }$ denotes the upper limit of step size, which is introduced to guarantee the mean square error (MSE) of the algorithm to be bounded, and ${\eta }_{\min }$ is chosen to provide a lower limit of tracking capability. The initial step size ${\eta }_{ij}\left(0\right)$ is generally set to be ${\eta }_{\max }$. The step size ${\eta }_{ij}\left(n\right)$ is determined by the result of the error function $E_{ij}\left(n\right)$ as well as the parameters $\alpha$ and $\beta$. $\alpha$ should be taken in a range of (0, 1) to provide exponential forgetting. $\beta$ is a small constant that is selected in conjunction with $\alpha$ to satisfy the misadjustment requirements [19]. Obviously, a large prediction error yields faster tracking; on the contrary, a small prediction error restrains the misadjustment.

4. Experiments and Results Analysis

In this section, we will employ artificially-corrupted sequences and real infrared sequences to verify the effectiveness of the newly-developed TVRNN-NUC method, and then compare the performance of the proposed TVRNN-NUC method to that of a typical NN-NUC method [1] and a TV-NUC method [20] with objective and subjective evaluation assessment. For each NUC method, we initialized the estimated gain matrix $\widehat{g}$ and offset matrix $\widehat{o}$ with ones and zeros in all elements, set parameters $\delta =10$, $\alpha =0.97$, $\beta =2\times {10}^{-9}$, and then implemented a proper fine-tuning of the learning rate to pursuit the best performance with a trade-off between convergence speed and stability.

4.1. Simulation with Artificially Corrupted Data

In this simulation, 4000 frames of 471 × 358-sized images collected by A615 camera (FLIR Systems, Inc., Wilsonville, OR, USA) and 500 frames of 532 × 478-sized images acquired by IRT102 imager (JOHO Technology, Wuhan, China) were first extracted from two different infrared sequences and named sequence 1 and sequence 2, respectively. Following that, both of them were artificially corrupted by FPN according to the model in Equation (1). In order to test the correction effect in different noise modes, the stripe gain parameters and random gain parameters with mean 1 and standard deviation 0.15 were respectively applied to sequence 1 and sequence 2, while offset parameters with mean 0 and standard deviation 11.55 were generated as realizations of iid Gaussian random variables and applied to both sequences. Thereafter, the NN-NUC, TV-NUC, and TVRNN-NUC with different gated threshold ($K=1$ and $K=10$) and gated TV (GTV, simplified TVRNN-NUC with gated threshold $K=10$ and fixed step size ${\eta }_{ij}\left(n\right)=1.5\times {10}^{-4}$) were respectively applied to correct the artificially-corrupted data frame-by-frame.

4.1.1. Comparison of Ghost Suppression Performance

In this section, the ghost suppression performance of variant NUC methods is compared by using the subjective performance index and objective visual effect. Firstly, the mean square error (MSE) of the gain estimates [11] is introduced to quantitatively evaluate the ghost suppression performance, which is defined as

$$MS{E}_{gain}\left(n\right)=\left(1/N_r{N}_c\right){\displaystyle \sum _{i=1}^{N_r}{\displaystyle \sum _{j=1}^{N_c}{\left[g_{ij}\left(n\right)-{\widehat{g}}_{ij}\left(n\right)\right]}^2}},$$

(18)

where $N_r$ and $N_c$ are the number of rows and columns in the simulated gain $g$ and estimated gain $\widehat{g}$, respectively.

The curves of MSE gain estimation obtained by applying different NUC methods to sequence 1 are plotted in a logarithmic scale in Figure 2. The NN-NUC showed an obvious non-convergence trend after the 800th frame due to the lack of deghosting ability. On the contrary, the TV-NUC posed a relatively slow descent and started to obtain a lower MSE gain estimation after the 2500th frame when compared to the NN-NUC. For the proposed TVRNN-NUC with $K=1$, the MSE dropped fastest and sustained a lower level in most of the iterative correction process. When global scene motion was negligible, once the scene motion slowed down (from about the 1000th to the 1200th frame), the TVRNN-NUC with larger gated threshold $K=10$ yielded smaller MSE, indicating a better deghosting capability. Moreover, TVRNN-NUC with $K=10$ took a distinct advantage from the use of adaptive step size when compared to the fixed step GTV-NUC with $K=10$. This experiment confirmed that the novel TV-regularized neural network regression model and gated adaptive learning strategy of the proposed TVRNN-NUC resulted in a stable and swift deghosting effect when compared to the existing NN-NUC and TV-NUC.

Figure 3 shows the corrected results of various NUC methods for the corrupted 1210th frame (immediately after the pause of object movement in the scene) in sequence 1. Figure 3a shows the simulated FPN disturbing image. The outputs using the NN-NUC, TV-NUC, GTV-NUC ($K=10$), TVRNN-NUC ($K=1$), and TVRNN-NUC ($K=10$) are shown in Figure 3b–f, respectively. From Figure 3b, we can observe strong ghosting artifacts in the NN-NUC outputs, and most of the ghosting artifacts trended to be suppressed by the TV-NUC except for a small amount of residual. By contrast, the TVRNN-NUC further reduced ghosting and presented the better correction result. Compared to TVRNN-NUC with $K=1$, we can find that TVRNN-NUC and GTV-NUC with $K=10$ eliminated the ghosting artifacts more completely. This phenomenon validates that a higher gated threshold is beneficial for ghosting suppression.

4.1.2. Comparison of Correction Precision and Convergence Rate

The correction results of different NUC methods was quantitatively assessed with both the peak signal-to-noise ratio (PSNR) criterion

$$PSNR\left(n\right)=10{\log }_{10}\frac{{255}^2}{\frac{1}{MN}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{\left[{\widehat{X}}_{ij}\left(n\right)-x_{ij}\left(n\right)\right]}^2}}},$$

(19)

and the roughness index [21], which was employed to measure the roughness of the image and can be defined as

$$\rho \left(n\right)=\frac{{\Vert h_1\otimes \widehat{X}\left(n\right)\Vert }_1+{\Vert h_2\otimes \widehat{X}\left(n\right)\Vert }_1}{{\Vert \widehat{X}\left(n\right)\Vert }_1},$$

(20)

where $h_1$ is a horizontal mask $\left[1,-1\right]$, $h_2=h_1^T$ is a vertical mask, $\otimes$ represents a convolution operator, and ${\Vert \cdot \Vert }_1$ refers to an $L_1$ norm. A lower roughness and higher PSNR means a better correction result.

Figure 4 shows the performance of different NUC methods upon sequence 1. It can be seen clearly from Figure 4a that the PSNR for TVRNN-NUC with $K=1$ and $K=10$ grew faster and finally reached a gap of nearly 2 dB over NN-NUC and TV-NUC. When we focus on the gated threshold, we can find the convergence rate of TVRNN-NUC with $K=10$ was obviously slower than that with $K=1$ but was faster than GTV-NUC with $K=10$. Essentially, for the TVRNN-NUC method, the increased gated threshold led to a better deghosting effect while paying the price of decreasing convergence rate; for the GTV-NUC method, the lacking adaptive variable step size involved in the TVRNN-NUC method pays back a slower convergence. For the TV-NUC method, the PSNR curve keeps a slow increase during the first 900 frames. Thereafter, it decreases suddenly and fluctuates fiercely in the rest of sequence 1.

As can be seen clearly from the roughness curves plotted in a logarithmic scale in Figure 4b, the proposed TVRNN-NUC method obtained a lower roughness value with faster and more stable convergence. Performance assessment for the averaged values of the whole iterative correction processes belonging to each of NUC methods are listed in

Table 1. Mean PSNR (dB) and roughness index (ρ) for artificially-corrupted sequence 1. FPN: fixed pattern noise.

Performance Metrics	FPN-Corrupted Image	Corrected Images
		NN-NUC	TV-NUC	GTV-NUC	TVRNN-NUC (k = 1)	TVRNN-NUC (k = 10)
PSNR (dB)	20.29	28.10	28.31	27.73	29.63	28.27
ρ	0.2760	0.1093	0.1083	0.1191	0.0999	0.1122

, where TVRNN-NUC methods presented an outstanding increment of PSNR over both of NN-NUC and TV-NUC. In addition, TVRNN-NUC methods obtained the smallest reported mean roughness index.

It is worth noting that the performance promotion of the TV-NUC was not distinct over NN-NUC from the entire correction process. The reason lies in that the convergence process of TV-NUC is easily interrupted and restarted by frequent scene switching. In order to further confirm this view, we repeated the above experiment using sequence 2. Different from sequence 1, the scene in sequence 2 is continuous without sudden scene changes. In this case, the TV-NUC method was not disturbed by scene switching, and achieved stable convergence. From the assessment shown in Figure 5 and

Table 2. Mean PSNR (dB) and roughness index (ρ) for artificially-corrupted sequence 2.

Performance Metrics	FPN-Corrupted Image	Corrected Images
		NN-NUC	TV-NUC	GTV-NUC	TVRNN-NUC (k = 1)	TVRNN-NUC (k = 5)
PSNR (dB)	22.17	32.55	33.34	29.84	36.65	33.34
ρ	0.3708	0.0927	0.1041	0.1354	0.0719	0.1029

, we can find that TV-NUC showed an obvious advantage in PSNR and held a slender lead in roughness after about 110 frames of iterative correction when compared to NN-NUC.

Moreover, it is clear that neither TVRNN-NUC nor GTV-NUC with $K=5$ were superior to the TVRNN-NUC with $K=1$. Since the strong edge object in sequence 2 keeps moving without perceptible pause, even if the NUC methods without gated strategy were not bothered by the ghosting artifacts, in this situation, the additional gated strategy led to slow convergence and residual noise rather than suppressing ghosting.

The aforementioned experimental results indicate that our proposed TVRNN-NUC method presented outstanding correction precision and had faster convergence speed. More importantly, the stability of our proposed TVRNN-NUC was outstanding whether the object moved or not and scene switched or not.

4.2. Applications to Real Infrared Image Sequences

To test the practical processing effect, 1970 frames of real infrared data (named sequence 3 with gentle scene motion) collected by a 384 × 288 ULIS Pico384 camera (ULIS, Veurey-Voroize, France) and 200 frames of 320 × 256-sized real infrared data (named sequence 4 with more dramatic scene motion) acquired by RTD3172C imager (IRay, Yantai, China) were adopted to contrast the performance of the fine-tuning TVRNN-NUC with $K=5$ against NN-NUC and TV-NUC. For each method, we carried out a quantitative performance test with the metric of roughness index (ρ).

Figure 6 shows the performance of different NUC methods upon sequence 3. It can be seen clearly that the TVRNN-NUC method converged more rapidly and reached a stable convergence with smallest roughness index. Note that at the motion pause stage (from frame 940 to frame 1110), the NN-NUC method still updated the correction parameters and continued to implement correction, which made the output become over-smoothed (roughness curve reached the concave point of the pause stage). After the resumption of scene motion, the NN-NUC method began to quickly recover, but noticeable ghosting existed for the following 50 frames at least (this phenomenon can be observed in Figure 7). The TV-NUC method slowed down the correction with the TV-norm changed faintly in the pause stage, which resulted in a comparatively better deghosting effect. In contrast, the TVRNN-NUC method adopted a gated strategy and TV regularization to simultaneously further suppress the ghosting and avoid over-smoothing effects. The reason lies in that the edge-preserving total variation regularizer contributes to smooth out the FPN without the need to increase the average window size of target value $D$ in the fidelity term. This is why the proposed TVRNN-NUC can avoid the damage of the over-smoothing effect from the averaging target.

In order to observe the corrected results and deghosting effects of the three methods after the motion pause intuitively, take the correction results of the 1155th frame shown in Figure 7 as an example. Figure 7a shows the input image. The corrected results of the NN-NUC, TV-NUC, and TVRNN-NUC are shown in Figure 7b–d, respectively. By comparing the visual effect with the naked eye, we can easily find that the proposed TVRNN-NUC method suppressed the FPN better without perceptible ghosting artifacts and preserved more high-frequency details, which made the corrected scene seem sharper and more distinguishable.

Figure 8 shows the curves of roughness obtained by different NUC methods for sequence 4. The proposed TVRNN-NUC acquired lowest roughness and required only 20 iterations to reach convergence. From the visual effect comparisons shown in Figure 9, we can see clearly that the correction results of TVRNN-NUC provided distinct details while suppressing the annoying fixed pattern noise and ghosting artifacts simultaneously. Meanwhile, NN-NUC produced obvious ghosting artifacts on the region near the roof and lost much of the detailed spatial information, and TV-NUC had partial grid-shaped fixed pattern noise remaining interspersed in the background.

It is worth nothing that the TVRNN-NUC method needed about 200 iterative corrections to reach convergence in sequence 3 and only 20 iterations in sequence 4. The reason lies in that the global scene motion of sequence 4 was more violent than sequence 3.

The above testing results validate the fact that our proposed TVRNN-NUC method can effectively eliminate nonuniformity and suppress ghosting artifacts more completely for real corrupted sequences. At the same time, abundant scene motion is beneficial to promote the convergence rate of the TVRNN-NUC. In other words, the increase of scene motion could be regarded as a reasonable idea to overcome the limitation of slow convergence introduced by the gated strategy.

4.3. Comparison of Real-Time Performance

In order to verify the real-time performance, we corrected the abovementioned four testing sequences with different length and frame size and then evaluated the processing efficiency on a 2.4 GHz Dell desktop PC.

Table 3. Processing frame frequency (frames/second) for different testing sequences.

Index	NN-NUC	TV-NUC	GTV-NUC	TVRNN-NUC
Sequence 1 (471 × 358)	272	210	46	43
Sequence 2 (532 × 478)	199	115	31	28
Sequence 3 (384 × 288)	580	442	86	81
Sequence 4 (320 × 256)	628	549	112	99

gives the actually achievable processing frame frequency for different frame sizes. The results reported in Table 3 indicate that the proposed TVRNN-NUC demanded more computational load than others, which is the cost of its outstanding correction performance. Even so, the TVRNN-NUC can afford to process considerably large frames in excess of the real-time requirement in serial computing mode. Once implemented on specially-designed parallel hardware (e.g., Graphics Processing Unit (GPU) or Field Programmable Gate Array (FPGA)), the TVRNN-NUC will be further accelerated by parallel convolution computation.

5. Conclusions

While many scene-adaptive NUC approaches have been presented to suppress FPN in infrared images, they may not be able to solve ghosting artifacts accompanied with slow convergence speed. To this end, we introduce a total variation penalty into the objective function to solve the overfitting problem existing in the LMS regression of the NN-NUC method. Moreover, an adaptive gated learning rate is presented to further suppress the ghosting effect while retaining a higher convergence rate. Experiments with both of simulated data and real scenes have successfully demonstrated that the proposed TVRNN-NUC method achieved a higher correction precision with a faster convergence rate. Moreover, the correction results of our proposed method had a sharper visual effect without perceptible ghosting artifacts.