On-Board Flickering Pixel Dynamic Suppression Method Based on Multi-Feature Fusion

Abstract

The blind pixel suppression is the key preprocess to guarantee the real-time space-based infrared point target (IRPT) detection and tracking. Meanwhile, flickering pixels, as one of the blind pixels, is hard to suppress because of randomness. At present, common methods adopting a single feature generally need to accumulate dozens or hundreds of frames to ensure detection accuracy, which cannot update flickering pixels frequently. However, with low detection frequency, the flickering pixels are easily miss detected. In this paper, we propose an on-board flickering pixel dynamic suppression method based on multi-feature fusion. The visual and motion features of flickering pixels are extracted from the result of IRPT detection and tracking. Then, the confidence of flickering pixel evaluation strategy and selection mechanism of flickering pixel are introduced to fuse the above features, which achieves accurate flickering pixel suppression using a dozen frames. The experimental results evaluated on the real image of four scenarios show that the blind pixel false detection rate of the proposed method is no more than 1.02%. Meanwhile, evaluated on the simulated image, the flickering pixel miss suppression rate is no more than 2.38%, and the flickering pixel false suppression rate is 0. The proposed method could be an addition to most other IRPT detection methods, which guarantees the near-real-time and reliability of on-board IRPT detection applications.

1. Introduction

Space-based infrared point target (IRPT) detection is one of the most important applications of the on-board intelligent information processing technology, which has been wildly used in both the civil and military fields. Blind pixel suppression is the essential preprocess for all infrared camera applications. Owing to the limitation of the material, craftwork, and working environment, some infrared pixels will experience permanent or temporary functional degradation, namely blind pixels. Especially, blind pixels with a visual feature similar to that of IRPT can easily lead to false detection of IRPT and obstruct the real-time performance of the IRPT detection algorithm. Indeed, threshold filter [1], coding mechanism [2], and support vector machine [3], etc., are proposed to eliminate false alarm sources and have obtained good results in the IRPT detection algorithm. However, these sophisticated methods consume most computing resources to eliminate more indistinguishable false alarm sources from background. These blind pixels have a catastrophic effect on on-board intelligent information processing if they are not properly suppressed. Therefore, an effective blind pixel suppression method is vitally needed for on-board IRPT detection application.

Generally, blind pixels are divided into constant blind pixels and flickering pixels [4,5]. The constant blind pixels which appear as loog-live over-bright or over-dark can be corrected by the existing method effectively. However, considering that the flickering pixels can oscillate randomly between several levels, flickering pixels suppression is a hard task. Nowadays, blind pixel suppression is divided into two steps: blind pixel detection and blind pixel compensation. The blind pixel detection method can be classified into two categories. The first category is the calibration-based method [6,7,8], which uses the blackbody to obtain uniform reference images and detect blind pixels by comparing each pixels’ response rate, deviation factor, noise statistics, and other properties. This method is easy to implement and has a good detection effect for constant blind pixels. However, flickering pixels cannot be detected accurately by this method due to their randomicity. More importantly, this method requires additional hardware equipment, such as blackbody and electromechanical parts, which will consume valuable hardware resources of satellites.

The other category is the scene-based blind pixel detection method, which utilizes image processing and computer vision to statistically analyze the scene image. Although this method has a lower accuracy compared to the calibration-based method, it can operate during the system’s normal mission and does not require other equipment. Namely, this method has better real-time performance and adaptability. Presently, one of the most important tools for scene-based blind pixel detection methods is image registration. Korchev et al. [9] proposed a method that created bad pixel masks by frame registration using simple pixel differences, and produced good results for detecting moving, low-contrast targets at the same time. Liu et al. [10] adopted a registration-based method that could correct the fixed pattern noise with less error using the estimation of scene motion. In addition, Liu et al. [11] used the correlated phase information to make the registration more precise and adopted linear mapping to estimate true pixel value. However, the efficiency of image registration methods may influence the real-time performance of these methods. For instance, the registration based on the Fourier transform would make some methods difficult to apply to real-time hardware [11]. In addition, a good image registration effect asks for rich scene details, which limits the application for some scenarios. Recently, analyzing the intensity of adjacent pixels is also efficient. Tchendjou and SIMEU [12] proposed real-time and online defective pixel detection and correction algorithms. By using basic arithmetic operations on adjacent pixels, these algorithms have been implemented on an FPGA platform. Additionally, Cao et al. [13] proposed a scene-based bad pixel dynamic correction method. According to the self-adaptive median filter and the keyframe technique, this method can search for blind pixels with high detection accuracy. Song et al. [14] also gave an adaptive median filtering method to detect real-time blind and flickering pixels, in which a human-vision-based algorithm is applied to avoid misjudging blind pixels. However, these methods also have limitations. For instance, they detect blind pixels using high spatial frequency, which can easily cause the IRPT to be falsely identified as blind pixels. Therefore, these methods need dozens or hundreds of frames to ensure the accuracy of blind pixel detection, which may cause miss detection of the flickering pixel with low flickering frequency.

The blind pixel compensation method is applied to overcome the bad effect of the blind pixel which is detected by the previous step. At present, the existing blind pixel compensation methods [15,16,17,18] have achieved great compensation effect. These methods use adversarial networks, dual-band information, or regression model to compensate the blind pixel, and the rich scene details are preserved. Despite their success in estimated accuracy, aforementioned methods also have a limitation, since they generally treat each blind pixel as the constant blind pixel. However, the flickering pixels still retain a part of the information acquisition ability [19]. Therefore, how to choose the flickering pixel that should be compensated is worth studying.

In this paper, in order to address the problems mentioned above, we proposed an on-board flickering pixel dynamic suppression method based on multi-feature fusion. The visual and motion features of flickering pixels were extracted from the result of IRPT detection and tracking, which economizes computing resources. Meanwhile, the above features are fused by the confidence of flickering pixel (CFP) evaluation strategy and selection mechanism to guarantee the accuracy of flickering pixel detection with a dozen frames. Finally, CFP and visual features of flickering pixels are combined to compensate flickering pixels dynamically and accurately. This method overcomes the drawbacks associated with the conventional scene-based method that depends on a single feature consuming large amounts of frames to ensure detection accuracy.

2. Methodology

As shown in Figure 1, the proposed method is embedded in a IRPT detection process and, the block diagram briefly summarizes the processing steps of the proposed method including visual feature extraction, motion feature extraction, multi-feature fusion, and blind pixel compensation.

2.1. Visual Feature Extraction Based on Facet Model

A point target is a target whose actual image area on the surface of the detector is smaller than one pixel of the detector [20]. However, the imaging process of the practical optical system is influenced by diffraction and aberration. Therefore, IRPT imaging is not an ideal point but a speckle. Conversely, blind pixels are mainly caused by the limitations of the sensor, and their image is entirely unrelated to the optical system. Hence, blind pixels appear as bright or dark points in the infrared focal plane array (IRFPA).

From the above results and analysis, we propose a visual feature extraction method based on the facet model using the result of IRPT detection as input to distinguish blind pixels from suspected IRPT. The facet model was proposed by Robert M. Haralick [21]. This model considers the discrete pixel response as surface, and a polynomial function is used to estimate the surface in a small neighborhood [22]. Therefore, the spatial domain of the image can be partitioned into connected smooth regions called facets. Obviously, the blind pixels cannot be estimated by the facet model accurately because of its pulse shape. On the contrary, the intensity of IRPT decreased gradually from the center to the edge, which could be estimated more accurately by using the facet model. Therefore, we regard the detected IRPTs as potential blind pixels, and estimate them based on the facet model. Then, the estimation error of the facet model is segmented by the threshold to extract blind pixels. A detailed description of the steps is provided bellow.

Firstly, we use a bivariate cubic function f in canonical form as Equation (1) to fit the 3 × 3 neighborhood.

$$f\left(r,c\right)={{\displaystyle \sum }}_{i=1}^8{k}_i{p}_i,$$

(1)

where, $\left(r,c\right)\in R\times C$, $R,C=\left\{-1,0,1\right\}$ are symmetric index sets of row and column respectively. $p_i\in \left\{1,r,c,r^2-2/3,rc,c^2-2/3,r\left(c^2-2/3\right),c\left(r^2-2/3\right)\right\}$ are 2-D discrete orthogonal Chebyshev polynomials, and $k_i$ is the fitting coefficient which could be deduced by the least-squares algorithm as shown in Equation (2).

$$k_i=\frac{{{\displaystyle \sum }}_{\left(r,c\right)\in R\times C}{P}_i\left(r,c\right)I\left(r,c\right)}{{{\displaystyle \sum }}_{\left(r,c\right)\in R\times C}{P}_i{}^2\left(r,c\right)},$$

(2)

where $I\left(r,c\right)$ is the response intensity of pixel $\left(r,c\right)$. Obviously, $I\left(r,c\right)$ is independent of the remainder in Equation (2). Therefore, we can define this remainder as fixed filters ${\omega }_i$ in Equation (3) so that $k_i$ could be calculated by convolution.

$${\omega }_i=\frac{P_i\left(r,c\right)}{{{\displaystyle \sum }}_{\left(r,c\right)\in R\times C}{P}_i{}^2\left(r,c\right)},$$

(3)

Then, the estimation of the facet model can be deduced through Equations (1) and (2) as follows:

$$f_{facet}\left(i,j\right)=k_1-\frac{2}{3}{k}_4-\frac{2}{3}{k}_6,$$

(4)

where $\left(i,j\right)$ is the coordinate of potential blind pixels, and $f_{facet}\left(i,j\right)$ is the estimation of the facet model. The non-zero fitting coefficients could be calculated by the Equation (3) as follows:

$$\{\begin{array}{c}{\omega }_1=\frac{1}{9}\left[\begin{array}{ccc}1& 1& 1\\ 1& 1& 1\\ 1& 1& 1\end{array}\right]\\ {\omega }_4=\frac{1}{6}\left[\begin{array}{ccc}1& 1& 1\\ -2& -2& -2\\ 1& 1& 1\end{array}\right]\\ {\omega }_6={\omega }_4{}^T\end{array},$$

(5)

Up to now, we could estimate the response intensity of the 3 × 3 neighborhood of potential blind pixels quickly. Figure 2 shows simulated blind pixel and IRPT and their estimation by the above-mentioned facet model. It can be clearly seen that the estimation of IRPT is more accurate.

Subsequently, the estimation error of the facet model could be determined based on the following criterions as

$$Dev\left(i,j\right)=\frac{{{\displaystyle \sum }}_{m=-1}^1{{\displaystyle \sum }}_{n=-1}^1\sqrt{{\left[I\left(m+i,n+j\right)-f_{facet}\left(m+i,n+j\right)\right]}^2}}{{{\displaystyle \sum }}_{m=-1}^1{{\displaystyle \sum }}_{n=-1}^1\sqrt{{\left[I\left(m+i,n+j\right)-mean\left(pic\right)\right]}^2}},$$

(6)

where $Dev\left(i,j\right)$ is the estimation error of the facet model while $mean\left(pic\right)$ represents the mean of the current image. Note that the denominator of Equation (6) is used in the normalization of the estimation error for easy threshold comparison.

Finally, we set up a visual feature model of blind pixels (VFBP model) and define ${\mathrm{Th}}_{\mathrm{IM}}$ as the threshold of estimation error. The results of matching VFBP model are expressed by $X_{IM}\left(i,j\right)$, which is written as

$$X_{IM}\left(i,j\right)=\{\begin{array}{c}1, Dev\left(i,j\right)<{\mathrm{Th}}_{\mathrm{IM}}\\ 0, Dev\left(i,j\right)\ge {\mathrm{Th}}_{\mathrm{IM}}\end{array},$$

(7)

If $Dev\left(i,j\right)$ is smaller than ${\mathrm{Th}}_{\mathrm{IM}}$, $X_{IM}\left(i,j\right)$ will be assigned to 0, which means this potential blind pixel might be an IRPT. On the contrary, $X_{IM}\left(i,j\right)$ will be set to 1, which means this pixel might be a blind pixel. In this paper, ${\mathrm{Th}}_{\mathrm{IM}}$ is set to 0.30. According to the change of practical optical system, this threshold needs to be adjusted.

2.2. Motion Feature Extraction

In this paper, the IRPTs are divided into two categories: space target and aircraft. The movement of the space target is restricted to a specific orbit. Meanwhile, the movement of aircraft is also constrained by specific flight envelop [23]. Therefore, the moving path of IRPT can be treated as smooth curvilinear motion within a short time. In terms of blind pixel, although blind pixels cannot move, the inter-frame correlation may cause an error path formed by adjacent blind pixels. Therefore, the blind pixels appear as static points or winding paths.

According to the above analysis, we set up two kinds of motion feature models of blind pixels (MFBP model) to extract the motion feature of blind pixels. The static MFBP model with IRPT detection results as input is used to detect isolated blind pixels. Meanwhile, the dynamic MFBP model with IRPT tracking results as input detects adjacent blind pixels by evaluating the curvature degree of the path.

The static MFBP model firstly calculates the frequency that a pixel is detected as the blind pixel. Then, compare this frequency with the threshold to detect isolated blind pixels, as shown in Equation (8).

$$X_{MOst}\left(i,j\right)=\{\begin{array}{c}0, {{\displaystyle \sum }}_{f=F-F_{length}+1}^F{X}_d\left(f,i,j\right)/F_{length}<{\mathrm{Th}}_{MOs\mathrm{st}}\\ 1, {{\displaystyle \sum }}_{f=F-F_{length}+1}^F{X}_d\left(f,i,j\right)/F_{length}\ge {\mathrm{Th}}_{MOs\mathrm{st}}\end{array},$$

(8)

where $X_d\left(f,i,j\right)$ is the result of IRPT detection. If $X_d\left(f,i,j\right)$ is 1, it means that pixel $\left(i,j\right)$ is detected as potential blind pixel in $f$th frame. $F$ presents current frame number while $F_{length}$ presents the frame range of testing. Finally, $X_{MOst}\left(i,j\right)$ is the result of static MFBP model matching. If $X_{MOst}\left(i,j\right)$ equals to 1, it means that pixel $\left(i,j\right)$ may be blind pixel. Conversely, if $X_{MOst}\left(i,j\right)$ equals to 0, this pixel may be normal. In terms of parameter selection, the threshold of static MFBP model (${\mathrm{Th}}_{MOs\mathrm{st}}$) is set to 0.3. However, this threshold needs to be adjusted according to frame frequency and moving state of IRPT. It is necessary to point out that, considering computational complexity and reliability, $F_{length}$ is set to 10 to 20 in this paper.

Dynamic FPMF model also evaluates within $F_{length}$ frames and takes the path of IRPT tracking as input. Suppose that $(i_{fn},j_{fn})$ denotes the path points forms a certain path, and the subscript fn means the frame number. Using the result of IRPT tracking as input, pick up direction vectors from the path, we now describe the procedure as follows. Firstly, traverse all path points in frame order as the starting point of each direction vector. Then, traverse the path points whose distance to the starting point is greater than $\sqrt{2}$ and the frame number is greater than the starting point, and choose the path point with the smallest frame number among them as the end point of the corresponding direction vector. Let $\overrightarrow{D_n}$ be direction vectors and the subscript n means order number of direction vectors. Finally, the threshold of the dynamic MFBP model (${\mathrm{Th}}_{MO\mathrm{dy}}$) was introduced to evaluate the winding degree of a path, as shown in Equation (9).

$$X_{MOdy}(i_{fn},j_{fn})=\{\begin{array}{c}0, \frac{1}{n-1}{{\displaystyle \sum }}_{j=1}^{n-1}\frac{\overrightarrow{D_{j+1}}·\overrightarrow{D_j}}{\left|\overrightarrow{D_{j+1}}\right|·\left|\overrightarrow{D_j}\right|}<{\mathrm{Th}}_{MO\mathrm{dy}}\\ 1, \frac{1}{n-1}{{\displaystyle \sum }}_{j=1}^{n-1}\frac{\overrightarrow{D_{j+1}}·\overrightarrow{D_j}}{\left|\overrightarrow{D_{j+1}}\right|·\left|\overrightarrow{D_j}\right|}\ge {\mathrm{Th}}_{MO\mathrm{dy}} or n\le 3\end{array},$$

(9)

where $X_{MOdy}(i_{fn},j_{fn})$ presents the matching result of the dynamic MFBP model. The left side of the inequality denotes the mean value of the angle transformation of the path direction vector. If this value is larger than or equal to ${\mathrm{Th}}_{MO\mathrm{dy}}$, $X_{MOdy}(i_{fn},j_{fn})$ will be set to 1, which means this path is tortuous and these path points may be blind pixels. Furthermore, if the number of available direction vectors (n) is less than or equal to 3, $X_{MOdy}(i_{fn},j_{fn})$ will be set to 1 also, which means the pixels forming this path are too close and they might be blind pixel blocks. Note that ${\mathrm{Th}}_{MOs\mathrm{dy}}$ is set to 30. This threshold also needs to be adjusted according to frame frequency and moving state of IRPT.

2.3. The Strategy of Flicking Pixel Suppression

The confidence evaluation mechanism based on multi-features fusion was introduced to evaluate the blinking frequency of a pixel. We use $C\left(i,j\right)$ to represent the confidence of flickering pixel of pixel $\left(i,j\right)$. The value range is 0 to 1. The greater the confidence of flickering pixel of a pixel is, the higher the blinking frequency would be. In addition, the step size of updating confidence is defined as $c$. Considering the iteration speed and reliability of the confidence, we set $c$ ranging from 0.02 to 0.05. The higher the $c$, the faster the confidence update, but the lower the reliability.

All the pixels’ confidences are assigned to 0 at the beginning. If a certain pixel is judged as a flickering pixel by the VFBP model and one of the MFBP models at the same time, the confidence of this pixel will be increased a unit $c$, as shown in Equation (10).

$$C(i,j)=\{\begin{array}{c}1 C(i,j)+c>1\\ C(i,j)+c C(i,j)+c\le 1\end{array} if(X_{IM}\left(i,j\right)=1)\&\&\left(X_{MOst}\left(i,j\right)=1\left|\right|X_{MOdy}\left(i,j\right)=1\right),$$

(10)

If a certain pixel is judged as normal pixel by VFBP model or MFBP model, the confidence of this pixel will be decreased a unit $c$, as shown in Equation (11).

$$C(i,j)=\{\begin{array}{c}0 C(i,j)-c\le 0\\ C(i,j)-c C(i,j)-c>0\end{array} if(X_{IM}\left(i,j\right)=0)\left|\right|\left(X_{MOst}\left(i,j\right)=0\&\&X_{MOdy}\left(i,j\right)=0\right),$$

(11)

Especially, a threshold was introduced to maintain the scale of flickering pixels’ table. ${\mathrm{Th}}_{\mathrm{re}}$ is the reserve threshold and is set to 0.5 $c$. When CFP of a pixel is greater than ${\mathrm{Th}}_{\mathrm{re}}$, the pixel will be added to the flickering pixels’ table. Otherwise, it will be removed from the table. In addition, due to changes in working conditions, some flickering pixels will return to normal. Therefore, we introduce forgetting factor (F) with values from 0.96 to 0.98. If a certain pixel within the flickering pixels’ table is not updated CFP in the current frame, the CFP of this pixel will be decreased by multiplying the forgetting factor. The block diagram of proposed confidence evaluation mechanism of flickering pixel is shown in Figure 3.

Subsequently, the dynamic compensation strategy based on confidence was proposed to compensate the flickering pixel accurately to frame. The direct compensation threshold ${\mathrm{Th}}_{\mathrm{D}}$ and the compensation after validation threshold ${\mathrm{Th}}_{\mathrm{V}}$ are introduced. If the CFP of a certain pixel is equal to or greater than ${\mathrm{Th}}_{\mathrm{D}}$, this pixel will be compensated directly. Correspondingly, if the confidence of a certain pixel is equal to or greater than ${\mathrm{Th}}_{\mathrm{V}}$ but less than ${\mathrm{Th}}_{\mathrm{D}}$, this pixel will be validated by the VFBP model again. If it is judged as flickering pixel by the VFBP model in current frame, this pixel will be compensated. Otherwise, this pixel will not be compensated. In this paper, for saving computing resources, flickering pixels would be compensated by the mean filter or median filter. Alternatively, the flickering pixels are simply suppressed by skipping the IRPT detection process for these pixels.

Finally, the above parameters and thresholds selection are discussed. Indeed, these values are important because they determine the effect of flickering pixel suppression. The CFP fluctuates nonlinearly with the forgetting factor and steps size of updating CFP. Therefore, it is hard to give a clear interpretation of fluctuations of CFP. Moreover, the relation between blinking frequency of flickering pixel and the corresponding evolution of CFP should be determined to select the thresholds ${\mathrm{Th}}_{\mathrm{V}}$ and ${\mathrm{Th}}_{\mathrm{D}}$. Firstly, we introduce the 90th-percentile CFP (${\mathrm{Q}}_{90\%\mathrm{CFP}}$) to determine thresholds. The probability that the CFP of a pixel with a certain blinking frequency is greater than ${\mathrm{Q}}_{90\%\mathrm{CFP}}$ is 90%. Then, we define that the flickering pixel with blinking frequency equal to or greater than 80% will be suppressed directly, which corresponds to ${\mathrm{Th}}_{\mathrm{D}}$. The blinking frequency corresponding to ${\mathrm{Th}}_{\mathrm{V}}$ is 20%. Subsequently, we use uniformly distributed random numbers to simulate the blinking frequency of flickering pixel. Finally, through ten million iterations, the distribution of ${\mathrm{Q}}_{90\%\mathrm{CFP}}$ with different blinking frequency (BF), forgetting factor (F) and step size of updating CFP (c) is observed, as shown in Figure 4.

In this paper, the selection principle of parameters and thresholds is to make the blue line (${\mathrm{Th}}_{\mathrm{D}}$) and the red line (${\mathrm{Th}}_{\mathrm{V}}$) showed in Figure 4 uniformly distributed within the range of CFP. For instance, when the forgetting factor is equal to 0.98 and the step size of updating CFP is equal to 0.05, there will be a huge space under the red line (${\mathrm{Th}}_{\mathrm{V}}$), as shown in Figure 4c. It shows that flickering pixel needs more frames to accumulate CFP. Correspondingly, when forgetting factor is equal to 0.96 and step size of updating CFP is equal to 0.01, there will be a relatively huge space above the blue line (${\mathrm{Th}}_{\mathrm{D}}$), as is shown in Figure 4a. It denotes the flickering pixel needs more frames to weaken CFP when returning normal briefly. Following these principles, some recommended ranges of parameters and thresholds are selected and highlighted in the dotted box in Figure 4. Meanwhile, some typical selection of parameters and thresholds are listed in

Table 1. Typical selection of parameters and thresholds.

Step Size of Updating CFP (c)	0.01	0.03	0.05
Forgetting factor	0.98	0.97	0.96
${\mathrm{Th}}_{\mathrm{V}}$	0.17	0.30	0.37
${\mathrm{Th}}_{\mathrm{D}}$	0.95	0.97	0.96
${\mathrm{Th}}_{\mathrm{re}}$	$0.5 c$

.

3. Experimental Results and Discussion

3.1. Experimental Data

In order to measure the performance of the proposed method in this paper, we used MWIR 640 × 512 pixels Integrated Detector Dewar Cooler Assembly to obtain experimental data.

Table 2. Specifications of the MWIR detector used to obtain experimental data.

Format	640 × 512
Pixel size	15 μm
Spectral Range	3–5 μm
F-number	4
Noise Equivalent Temperature Difference (NETD)	30 mk
Framerate	50 Hz
Bits per pixel	14 bits

provides an overview of its specifications. In this letter, IRPT we mainly study is space target or aircraft. Therefore, we obtained sky background and deep space background with their IRPT as experimental data, as shown in Figure 5. In Figure 5, the scenarios from the top row to bottom row are sky background, sky background with IRPT, deep space background, and deep space background with IRPT, respectively. Each data set has five sequences, and each sequence includes 150–200 frames. Figure 6 shows the details about the acquisition of experimental data. In these data sets, the sky background and its IRPT are the real shots of the sky and the aircraft as shown in Figure 6a; However, the deep space background and its IRPT are simulated by blackbody and target board as shown in Figure 6b. In Figure 6c, the blackbody penetrates the target hole to form an IRPT. Then, we can control the field of view movement in a certain direction by moving the camera. In the image, the IRPT will move in the opposite direction of the camera movement.

Due to the limited number of flickering pixels of real experimental data, the simulated flickering pixels with random blinking frequency were added to the real experimental data to make a quantitative analysis on the effect of flickering pixel suppression.

3.2. Evaluation Criterion

In this paper, the evaluation criteria consist of two parts, blind pixel detection evaluation criterion and flickering pixel suppression evaluation criterion.

We adopt the number of detected blind pixels, the false detection rate of detected blind pixels, and the average running time per frame to evaluate the effect of blind pixel detection. The number of detected blind pixels ($N_{bp}$) denotes the number of constant blind pixels and flickering pixels detected by a certain method. The false detection rate of detected blind pixels is given by Equation (12).

$$R_{fbp}=\frac{N_{fbp}}{N_{bp}}·100\%,$$

(12)

where $N_{fbp}$ is the number of normal pixels that are falsely detected as blind pixels by a method. In this paper, we obtain calibration data, and $N_{fbp}$ can be measured by the method given in national standard [24]. Finally, the average running time per frame ($T_{pf}$) can present the operating efficiency of a method. The proposed method and other methods are implemented under MATLAB R2018a with an Intel Core 2.80 GHz processor and 8 GB of physical memory.

The number of suppressed flickering pixels, the miss suppression rate of flickering pixels, and the false suppression rate of flickering pixels form the evaluation criterion for flickering pixel suppression. The number of suppressed flickering pixels ($N_{sf}$) means the total number of frames suppressed for all flickering pixels. The miss suppression rate of flickering pixel ($R_{msf}$) is defined in Equation (13).

$$R_{msf}=\frac{{{\displaystyle \sum }}_{n=1}^N{N}_{msf}\left(n\right)}{{{\displaystyle \sum }}_{n=1}^N{N}_{rf}\left(n\right)}·100\%,$$

(13)

where $N$ is the flickering pixel amount, and $N_{msf}\left(n\right)$ refers to the frame amount that should be suppressed but not be suppressed of the $nth$ flickering pixels. $N_{rf}\left(n\right)$ is the frame amount of blinking of $nth$ flickering pixel. Similarly, false suppression rate of flickering pixel ($R_{fsf}$) is given by Equation (14).

$$R_{fsf}=\frac{{{\displaystyle \sum }}_{n=1}^N{N}_{fsf}\left(n\right)}{{{\displaystyle \sum }}_{n=1}^N{N}_{rf}\left(n\right)}·100\%,$$

(14)

where $N_{fsf}\left(n\right)$ refers to the frame amount that false suppressed of the $nth$ flickering pixel. As is mentioned above, the evaluation of flickering pixel suppression uses simulation data. Therefore, the $N_{rf}\left(n\right)$, $N_{fsf}\left(n\right)$, and $N_{msf}\left(n\right)$ in Equations (13) and (14) can be recorded during simulation.

3.3. Results and Discussion

According to the experimental data and evaluation criterion above, the proposed method is compared with the scene-based bad pixel dynamic correction (SBBPDC) method [13] and the calibration-based adaptive threshold bad pixel detection (CBATBPD) method [9]. In this experiment, we used the median filter and threshold segmentation to detect IRPT and use the interframe matching to track IRPT, in which our proposed method was embedded. The above IRPT process methods are one of the most common and the simplest methods in IRPT application. It is necessary to point out that our proposed method can also be embedded in any IRPT detection and tracking method.

Figure 7 shows the blind pixel maps of the first sequence of four scenarios, which are detected by SBBPDC, CBATBPD, and our proposed method. The top row to bottom row of Figure 7 is sky background, sky background with IRPT, deep space background, and deep space background with IRPT, respectively. The left column to right column of Figure 7 is the proposed method, SBBPDC, and CBATBPD, respectively. The blue points denote the blind pixels detected by our method. The red triangular rims and green square rims represent the blind pixels that are not detected by SBBPDC and CBATBPD, respectively. It is clear in the left column of Figure 7 that our method has detected many blind pixels that are not detected by other methods. Similarly, the red triangles and green squares denote the blind pixels detected by SBBPDC and CBATBPD, respectively, and the blue round rims are the blind pixels that are miss detected by our method. Obviously, there are only a few blind pixels that cannot be detected by the proposed method compared with other methods. As the middle column of Figure 7 shown, the SBBPDC cannot detect some adjacent blind pixels compared with our proposed method. This is because the adaptive threshold segmentation of this method erroneously suppresses a part of adjacent blind pixels. Furthermore, due to the movement of the background, some edges of the image usually move out quickly and do not have the same part for inter-frame registration. Therefore, the CBATBPD cannot detect the blind pixels at some edges of the field of view as shown in Figure 7. In conclusion, our method adopts multi-feature to accurately identify the real targets from potential targets. It can detect more real blind pixels compared with other methods. The blind pixel maps of other sequences have similar results as the first sequence. However, space does not permit their presentation.

In Figure 8, we respectively plotted the number of blind pixels detected by the above-mentioned methods in each frame. Figure 8a–d illustrates the data of the first sequence of sky background, sky background with IRPT, deep space background, and deep space background with IRPT, respectively. It is obvious that the proposed method can detect more blind pixels with fewer frames compared with other methods. SBBPDC uses the keyframe to improve the detection accuracy and efficiency, which depends on the change of background. Therefore, the detection rate of SBBPDC tends to slow down in deep space because the change of background is not obvious. Additionally, CBATBPD needs hundreds of frames to ensure detection accuracy.

Subsequently, we calculate the $N_{bp}$, $R_{fbp}$, and $T_{pf}$ mentioned in Section 3.2 of the above three methods in

Table 3. The situation of blind pixels detected in four application scenarios.

Blind Pixel Detection Method	Application Scenarios
	Cloud			Cloud with Target			Deep Space			Deep Space with Target
	${\mathit{N}}_{\mathit{b}\mathit{p}}$	${\mathit{R}}_{\mathit{f}\mathit{b}\mathit{p}}(\%)$	${\mathit{T}}_{\mathit{p}\mathit{f}}\left(\mathbf{s}\right)$	${\mathit{N}}_{\mathit{b}\mathit{p}}$	${\mathit{R}}_{\mathit{f}\mathit{b}\mathit{p}}(\%)$	${\mathit{T}}_{\mathit{p}\mathit{f}}\left(\mathbf{s}\right)$	${\mathit{N}}_{\mathit{b}\mathit{p}}$	${\mathit{R}}_{\mathit{f}\mathit{b}\mathit{p}}(\%)$	${\mathit{T}}_{\mathit{p}\mathit{f}}\left(\mathbf{s}\right)$	${\mathit{N}}_{\mathit{b}\mathit{p}}$	${\mathit{R}}_{\mathit{f}\mathit{b}\mathit{p}}(\%)$	${\mathit{T}}_{\mathit{p}\mathit{f}}\left(\mathbf{s}\right)$
SBBPDC	62	1.61	0.71	69	1.45	0.99	37	0	0.71	32	0	0.89
CBATBPD	37	10.8	1.17	34	8.82	1.02	—	—	—	—	—	—
Proposed	98	1.02	0.36	100	1.00	0.46	40	0	0.34	38	0	0.34

. The false detection rate of the proposed method is less than 1.02% in different scenarios, while the average running time per frame of the proposed method is less than 0.46 s. The proposed method could detect more blind pixels with less false detection rate and less time. It can be clearly seen that the proposed method is superior to others in four application scenarios.

In order to discuss the time consumption of our proposed method in practical applications, we recorded the average frame time consumption of each step in different scenarios in

Table 4. The average frame time consumption of each step of our method used in the above experiment in different scenarios.

	Scenarios	Cloud	Cloud with Target	Deep Space	Deep Space with Target
Time Consumption
IRPT detection (Median filter and threshold segmentation)		0.111 s	0.121 s	0.109 s	0.112 s
IRPT tracking (Interframe matching)		0.156 s	0.247 s	0.135 s	0.141 s
Blind pixels detection and compensation		0.091 s	0.089 s	0.091 s	0.090 s
Total		0.358 s	0.457 s	0.335 s	0.343 s

. The average frame time consumption of blind pixels detection and compensation is about 0.09 s. After code optimization and parallel acceleration, the time-consuming situation of our method can be further improved. Furthermore, in practical applications, our method can work once every two to three frames to ensure the near-real-time suppression of blind pixels. Moreover, it is important to note, owing to the limitation of the transmission bandwidth and computing resource, the current frame rate of IRPT detection and tracking application usually only reaches 5~20 frames per second.

In the terms of flickering pixel suppression, our proposed method is to achieve frame-accurate dynamical flickering pixel suppression. For instance, the pixel (347,487) was detected as a flickering pixel in the second sequence of cloud scenario and Figure 9 shows the suppression situation of this pixel. The CFP of this pixel was less than ${\mathrm{Th}}_{\mathrm{V}}$ before the 18th frame, so this pixel was not suppressed. After the 18th frame, our method accurately suppresses this pixel when the response is too low.

Meanwhile, we use the simulated data to make a quantitative analysis on the flickering pixel suppression. Figure 10 shows the randomly added flickering pixels and miss-suppressed flickering pixels of our method. Additionally, Figure 10a–d is sky background, sky background with IRPT, deep space background, and deep space background with IRPT, respectively. Blue points show the flickering pixels added randomly, red circles indicate the flickering pixels which miss-suppressed by our proposed method. We calculate the $N_{sf}$, $R_{fbp}$, and $R_{fsf}$ mentioned in Section 3.2 of different simulated scenarios in

Table 5. The situation of flickering pixels suppressed by proposed method.

Application Scenarios	${\mathit{N}}_{\mathit{s}\mathit{f}}$	${\mathit{R}}_{\mathit{m}\mathit{s}\mathit{f}}(\%)$	${\mathit{R}}_{\mathit{f}\mathit{s}\mathit{f}}(\%)$
Cloud	10,080	1.77	0
Cloud with target	9500	2.07	0
Deep space	7331	2.38	0
Deep space with target	7519	2.07	0

. The results showed that there is no false suppression of flickering pixels in all scenarios. In addition, the miss suppression rate of flickering pixels is less than or equal to 2.38% in all scenarios.

4. Conclusions

In this paper, an on-board flickering pixel dynamic suppression method based on multi-feature fusion is proposed. This method is embedded in the IRPT detection process, and the results of IRPT detection and tracking are used to extract the visual and motion features of the flickering pixel. A small amount of extra computing resource was consumed. Therefore, our proposed method can achieve real-time uninterrupted blind pixel detection without interrupting IRPT detection. Furthermore, the confidence of the flickering pixel evaluation strategy and the selection mechanism of flickering pixel are introduced to fuse the above features, which can achieve frame-accurate dynamical flickering pixel suppression. Experimental results have shown that our proposed method is superior to others in blind pixel detection and flickering pixel suppression. The proposed method can be an addition to most other IRPT detection processes, and it has significant meaning that the ability to suppress blind pixels guarantees near-real-time IRPT detection.