A Method for Visualization of Images by Photon-Counting Imaging Only Object Locations under Photon-Starved Conditions

Abstract

Recently, many researchers have been studying the visualization of images and the recognition of objects by estimating photons under photon-starved conditions. Conventional photon-counting imaging techniques estimate photons by way of a statistical method using Poisson distribution in all image areas. However, Poisson distribution is temporally and spatially independent, and the reconstructed image has a random noise in the background. Random noise in the background may degrade the quality of the image and make it difficult to accurately recognize objects. Therefore, in this paper, we apply photon-counting imaging technology only to the area where the object is located to eliminate the noise in the background. As a result, it can be seen that the image quality using the proposed method is better than that of the conventional method and the object recognition rate is also higher. Optical experiments were conducted to prove the denoising performance of the proposed method. In addition, we used the structure similarity index measure (SSIM) as a performance metric. To check the recognition rate of the object, we applied the YOLOv5 model. Finally, the proposed method is expected to accelerate the development of astrophotography and medical imaging technologies.

1. Introduction

Recently, research on estimating and visualizing photons under photon-starved conditions has been conducted in various fields such as aerospace optics, medical optics, etc. Furthermore, there is a new field of research using photon-counting detectors (PCDs), which detect photon energy to visualize images [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]. The Hubble Space Telescope (HST), launched from Earth in 1990, is a space telescope developed by the National Aeronautics and Space Administration (NASA) and the European Space Agency (ESA) that is used for observational astronomy from space [2]. The HST is equipped with a variety of cameras, spectrographs, and optical instruments, including a spectrograph optimized for ultraviolet observation, which uses PCDs with photon-counting technology that perform better than a CCD (Charge-Coupled Device) at estimating and visualizing ultraviolet photons even under photon-starved conditions such as in space [2,3,4,5]. As a result, the HST has enabled the observation of the expanding universe as well as the Hubble constant measurement project, which has provided us with a deeper understanding of the universe and its structure [6,7].

Hounsfield was awarded the 1979 Nobel Prize in Physiology or Medicine for developing and first introducing the medical imaging technology computed tomography (CT) into medicine [8,9]. X-ray CT is a medical imaging technique that non-destructively reconstructs three-dimensional (3D) images of human internal structures. More recently, advances in X-ray CT imaging technology have led to the development of photon-counting computed tomography (PCCT), which uses PCDs to estimate individual incoming X-ray photons and measure their energy levels [10,11,12]. PCDs can significantly reduce image noise and increase spatial resolution, and k-edge imaging can be used to measure the concentration of specific elements for material discrimination. Moreover, these technological advances will reduce radiation dose by at least 30–40% compared to traditional X-ray CT imaging techniques [10]. Techniques, such as using PCDs to estimate individual incoming X-ray photons and measure their energy levels, have been developed by many researchers in recent years [2,3,4,5,6,7,8,9,10,11,12,13,14]. However, PCDs may have some imperfections. First, charge sharing occurs when an X-ray photon arrives near the boundary between two pixels. This causes the X-ray photon to be detected both times at the wrong energy, reducing spatial resolution. Second, two independent photons may arrive at the same pixel at very high speeds, causing the signal to “pile up” and be interpreted by the electronic processing unit as a single photon. Finally, PCDs can be susceptible to electron noise, similar to CCDs, and suffer from the problem that measurements become uncertain when low-energy photons are detected [15]. Many researchers have studied these problems [20], and research is ongoing to accurately estimate photons to visualize images.

In general, photon-counting imaging techniques use a specialized sensor such as an EM-CCD camera, rather than a general camera, to acquire images. However, these specialized sensors require high prices and are difficult to commercialize. To overcome this problem, in this paper, we use a computational algorithm with a general camera to implement a photon-counting imaging technique [18,21,22,23,24,25,26]. Photon-counting imaging techniques, which estimate photons in a statistical way, may be an alternative to the conventional PCD problem. The computational photon-counting imaging method is an image processing technique that solves the hardware problems of photon-counting imaging by using software methods similar to the physical photon-counting imaging method. Photon-counting imaging techniques can visualize images by estimating photons that rarely occur in a unit of time and space based on Poisson distribution [16,17,18,19,21,22,23,24,25,26]. However, it also has a problem of random noise, and photons can be detected from objects as well as from the background, which degrade the visual quality of the image. Several studies have applied filters to remove the random noise in this background, most notably the median filter or the Kalman filter [21,22]. Using these filters can reduce the quality of the image because they remove not only background noise, but also information about the object. To solve these problems, in this paper, we propose a novel photon-counting imaging method to classify object location and the number of photons by section (COLaNoPS). Since the conventional photon-counting imaging technique applies a Poisson random process to all of the image, we can recognize the random noise from both the object and the background. Therefore, we need to remove noise from the background by applying Poisson distribution only to the area where the object exists. To estimate the location of an object, we assume that the photon energy at the location of the object is higher than that of the background. A threshold for the presence of an object is defined by measuring the photon energy as a section of a certain size moves throughout it. This algorithm reduces noise in the background by applying Poisson distribution only to the area in which the object is present, based on the threshold. As the section moves throughout the image, it applies spatial overlap to the area where the photons overlap each other to improve the photon energy of the object. As a result, we can estimate the photon energy of the object, improving the visual quality of the image.

This paper is organized as follows. In Section 2, we describe conventional photon-counting imaging techniques and the proposed method that estimates the photon energy of the object to reduce background noise. Then, we perform the optical experiment and show the results in Section 3. Finally, in Section 4, we present our conclusions and future works.

2. Reducing Background Noise by Estimating Photons Only in the Object Area

2.1. Photon-Counting Imaging

The human eye can see objects by converting the properties of light reflected from them into electrical signals through the rod and cone cells of the retinal cells. The image sensor (CCD or complementary metal-oxide semiconductor (CMOS) sensor) in the camera is a device similar to the human eye, which uses the photoelectric effect to detect the nature of reflected light and can visualize objects. Therefore, the image sensor in the camera suffers from the inability to visualize objects under photon-starved conditions. To solve this problem, photon-counting imaging techniques [16,17,18,19,21,22,23] can be one of the alternatives to visualizing images under photon-starved conditions. Photon-counting imaging technology estimates photons for an object by applying a statistical process of Poisson distribution to each image pixel under photon-starved conditions. Furthermore, the accuracy of the estimated photons can be improved by applying the maximum likelihood estimation (MLE) and a Bayesian approach such as the maximum a posteriori (MAP). MLE and MAP are the main methods for solving the classification problems using probability. MLE is the method of selecting the class with the maximum likelihood, and MAP is the method of selecting the class with the maximum posterior probability [16,19]. We apply a Poisson distribution to probabilistically estimate the photons of an object under photon-starved conditions. We apply MLE and MAP to the estimated photons for solving the classification problem of determining whether a photon occurs or not. Since photons rarely occur in unit time and space under photon-starved conditions, we can assume that they follow a Poisson distribution, and the Poisson distribution is defined by the following equations:

$${\lambda }_E(x,y)=\frac{I_E(x,y)}{{\sum }_{x=1}^{N_x}{\sum }_{y=1}^{N_y}{I}_E(x,y)}$$

(1)

$$C_E\left(x,y\right)\left|N_p{\lambda }_E(x,y)\right.\sim Poisson\left[N_p{\lambda }_E\left(x,y\right)\right],$$

(2)

where ${\lambda }_E(x,y)$ is the normalized intensity of the image at position $(x,y)$ of each elemental image in the array, $I_E$ is the intensity of the elemental image, $N_x$ and $N_y$ are the total number of image pixels, and x and y are the pixel positions of each elemental image, respectively. In addition, $C_E$ is the estimated photons in the elemental image, and $N_p$ is the expected number of photons for each elemental image. The likelihood function and log-likelihood function of the ${\lambda }_E$ normalized elemental images are defined as follows [16,17,18,19,21,22,23]:

$$P\left(C_{kl}|N_p{\lambda }_{kl}\right)=\prod _{k=0}^{k-1}\prod _{l=0}^{L-1}\frac{e^{{-N}_p{\lambda }_{kl}}{\left(N_p{\lambda }_{kl}\right)}^{C_{kl}}}{C_{kl}!},$$

(3)

$$L\left(N_p{\lambda }_{kl}|C_{kl}\right)\propto \sum _{k=0}^{K-1}\sum _{l=0}^{L-1}\left\{C_{kl}log\left[N_p{\lambda }_{kl}\right]\right\}-\sum _{k=0}^{K-1}\sum _{l=0}^{L-1}{N}_p{\lambda }_{kl},$$

(4)

where ${\lambda }_{kl}$ is assumed to be a matrix of k rows and l columns all extracted independently from the elemental images. $P\left(C_{kl}\right|N_p{\lambda }_{kl})$ is the likelihood function, $L\left(N_p{\lambda }_{kl}\right|C_{kl})$ is the log-likelihood function. The MLE for estimating by maximizing the likelihood function is defined as follows:

$$\frac{\partial L\left(N_p{\lambda }_{kl}|C_{kl}\right)}{\partial {\lambda }_{kl}}=\frac{C_{kl}}{{\lambda }_{kl}}-N_p=0,$$

(5)

$$\therefore {\lambda }_{kl}^{MLE}=\frac{C_{kl}}{N_p}.$$

(6)

In Equations (5) and (6), we can define the MLE for the normalized elemental image ${\lambda }_{kl}$ as ${\lambda }_{kl}^{MLE}$ by taking a partial derivative of $L\left(N_p{\lambda }_{kl}\right|C_{kl})$ to ${\lambda }_{kl}$. As a result, we can derive the MLE of each elemental image as shown in Figure 1 [16,17,18].

Figure 1 shows the process of the photon-counting imaging by MLE. However, in MLE, the estimation accuracy may be low because it uses uniform distribution as the prior information, which means that the occurrence of photons for each pixel has the same probability. Therefore, more specific prior information is required to obtain better estimation accuracy. A Bayesian approach such as MAP uses the specific statistical distribution as the prior information. In this paper, we assume that the normalized elemental image with the expected number of photons $N_p{\lambda }_{kl}$ follows a Gamma $(\Gamma )$ distribution because a general image has a [0 255] pixel range. The MAP method uses the statistical parameters $\alpha$ and $\beta$ of the image’s prior probability distribution, which is defined as a $\Gamma$ distribution. To estimate ${\lambda }_{kl}$, a posterior distribution is calculated and maximized by multiplying the likelihood function for each elemental image using prior information [19,21,22,23].

$$\pi \left(N_p{\lambda }_{kl}\right)=\frac{{\beta }^{\alpha }}{\Gamma \left(\alpha \right)}{\left(N_p{\lambda }_{kl}\right)}^{\alpha -1}{e}^{-\beta N_p{\lambda }_{kl}},N_p{\lambda }_{kl}>0$$

(7)

$$\mu =\frac{\alpha }{\beta },{\sigma }^2=\frac{\alpha }{{\beta }^2}\to {\alpha }^2=\frac{{\mu }^2}{{\sigma }^2},\beta =\frac{\mu }{{\sigma }^2},$$

(8)

$$\pi \left(N_p{\lambda }_{kl}|C_{kl}\right)\sim Gamma\left(C_{kl}+\alpha ,\left(1+\beta \right)\right),$$

(9)

where $\pi \left(N_p{\lambda }_{kl}\right)$ is a $\Gamma$ distribution of the normalized elemental image with the expected number of photons, which is a prior probability distribution and a conjugate prior of Poisson distribution; $\alpha$ and $\beta$ are the statistical parameters of the $\Gamma$ distribution and they are both positive; $\mu$, ${\sigma }^2$ are the mean and variance of $N_p{\lambda }_{kl}$; $\pi \left(N_p{\lambda }_{kl}\right|C_{kl})$ is modeled as a conjugate family of distributions for ease of calculating the $Gamma$ distribution, respectively [22,23].

$$\therefore {\lambda }_{kl}^{MAP}=\frac{C_{kl}+\alpha }{N_p\left(1+\beta \right)},C_{kl}>0.$$

(10)

By using Equations (7)–(9), we can calculate the posterior distribution and the estimator of elemental images as written in Equation (10). The original image is reconstructed using the mean of the posterior distribution from each elemental image. That is, ${\lambda }_{kl}^{MAP}$ can be defined as the mean of a posterior distribution by MAP [23].

Figure 2 shows the reconstructed images by MLE and MAP under photon-starved conditions. The MAP method can reconstruct the image by estimating the photons more accurately than the MLE method under these conditions with the same number of photons.

Figure 3 shows a noisy image from the background when the image is reconstructed by estimating photons with MAP. To visualize the noise, we increased the brightness by 40% and decreased the contrast by 40%. As shown in Figure 3, the MAP method can estimate photons more accurately than the MLE method to visualize the image under photon-starved conditions. However, since the MAP method applies a Poisson distribution to all areas, random noise occurs in the background. In this case, the noise in the background degrades the image quality of the object and makes it difficult to recognize the object accurately. In the next subsection, we propose a new photon-counting imaging technique to solve these problems.

2.2. Proposed Method

To remove background noise from conventional photon-counting imaging techniques, we propose a novel photon-counting method to classify object location and the number of photons by section (COLaNoPS). The proposed method, COLaNoPS, estimates photons in the section where an object is located by calculating the overall intensity of the section rather than applying Poisson random distribution in all areas, and visualizes it by applying a Poisson distribution only where the object is located to prevent generating noise in the background. Furthermore, spatial overlap is used in areas where photons overlap as the section moves, improving the image quality of objects and the recognition rate of objects. The proposed method assumes that the overall intensity value of the section in the presence of the object is higher than in the absence of the object. The method for estimating the location of an object calculates the overall intensity of the section and the overall intensity of the image as a section of a certain size is moved. By comparing the background pixel intensity to the object pixel intensity, we define the threshold for the presence of the object using the median value. The thresholds for determining the presence of an object are as follows.

$$I_{inten}\left(x,y\right)=\sum _{k=0}^{k-1}\sum _{l=0}^{L-1}{I}_{kl}{andK}_{inten}\left(x,y\right)=\sum _{k=0}^{k-1}\sum _{l=0}^{L-1}{K}_{kl}$$

(11)

$${\gamma }_m(x,y)=Med\left\{\frac{K_{inten}\left(x,y\right)}{I_{inten}\left(x,y\right)}\times 100\right\}$$

(12)

$${\mu }_p=\{\begin{array}{cc}{N}_p,\hfill & K_{inten}\left(x,y\right)>{\gamma }_m\left(x,y\right)\hfill \\ 0,\hfill & K_{inten}\left(x,y\right)\le {\gamma }_m\left(x,y\right),\hfill \end{array}$$

(13)

where $I_{inten}(x,y)$ is the overall pixel intensity of the original image and $K_{inten}(x,y)$ is the overall pixel intensity of the section. ${\gamma }_m$ is the threshold to determine the presence of the object, ${\mu }_p$ is the estimated number of photons in the object based on the threshold, and $N_p$ is the expected number of photons in each elemental image. The presence of an object is determined based on a defined threshold to estimate photons in the object.

$${\widehat{\lambda }}_{kl}^{MAP}=\frac{{\widehat{C}}_{kl}+\alpha }{{\mu }_p\left(1+\beta \right)},{\mu }_p\ge 0$$

(14)

$${\tilde{C}}_{kl}\left|{\mu }_p{\widehat{\lambda }}_{kl}^{MAP}\right.\sim Poisson\left({\mu }_p{\widehat{\lambda }}_{kl}^{MAP}\right).$$

(15)

Using Equations (14) and (15), we can calculate the posterior distribution and estimator of the area in which the object exists. ${\widehat{\lambda }}_{kl}^{MAP}$ is the posterior mean of the photons estimated in the object by the Bayesian approach MAP method, and ${\tilde{C}}_{kl}$ is the photon estimated from the object of each elemental image. The area in which the object exists is modeled using an estimated posterior mean. As a result, we can define a posterior mean from the MAP method of the photons estimated in the area where the object exists.

$$R\left(x,y\right)=\frac{1}{O_{spatial}\left(x,y\right)}\sum _{i=0}^{N_x-1}\sum _{j=0}^{N_y-1}{\widehat{\lambda }}_{ij}^{MAP}\left\{x-S_i,y-S_j\right\},N_{ij}>S_{ij},$$

(16)

where R is the elemental image reconstructed by the COLaNoPS method; $O_{spatial}$ is the matrix of spatial overlaps of photons in the object as the section moves; and $S_i$ and $S_j$ is the section’s x and y-axis shifting pixel, respectively. Finally, the photon-counting imaging technique is applied only to the areas that are estimated to be objects; where photons overlap, the image can be reconstructed by applying the spatial overlapping.

Figure 4 illustrates a flowchart of the COLaNoPS method. We use the median value to define a threshold for the presence of an object, and we calculate the spatial overlap in the space where photons overlap due to section movement. Finally, we visualize the image by estimating the photons for the object under photon-starved conditions.

Figure 5 shows the results of the conventional photon-counting imaging technique and the proposed COLaNoPS method under photon-starved conditions, and shows that the COLaNoPS method improves the image quality of objects more than the conventional photon-counting imaging technique.

3. Experimental Setup and Results

3.1. Experimental Setup

In this subsection, we describe the experimental setup to compare the conventional photon-counting imaging technique with the COLaNoPS method. Figure 6a shows the experimental setup in this paper. Figure 6b is the image obtained by the experimental setup. Photon-starved conditions are set by controlling the amount of light in this experiment. In this experiment, we use a Nikon D5300 to take experimental scenes because our proposed method is the computational algorithm with a general camera to implement photon-counting imaging technique as we mentioned earlier in the first section. All three objects are metal objects, and they are located at different distances from the camera. The distances from the camera to the objects are 400 mm, 430 mm, and 460 mm, with a distance difference of 30 mm for each object.

Table 1. Camera specifications and setup.

Setup	Nikon D5300
Resolution	2992 × 2000
Sensor size	23.5 mm × 15.7 mm
Section size	400 × 400
Section shifting pixel	50
Focal length	5 mm
ISO	160
Shutter speed	Normal	5 s
	Extremely low-light	180 s

shows the specifications and setup of the camera used in this experiment. The number of photons is gradually increased from 100,000 to 1,100,000 based on images obtained under photon-starved conditions.

3.2. Results

In this subsection, we show the results of the conventional method and the proposed method. Figure 7a is the image obtained under normal light conditions, and Figure 7b is the image obtained under photon-starved conditions. As shown in Figure 7b, the object cannot be visualized because it is too dark. Figure 7c is the reconstruction result by applying conventional photon-counting imaging techniques to the image in Figure 7b, which is not visible to the human eye. Figure 7d shows the reconstruction result by applying the COLaNoPS technique to the image in Figure 7b. For fair comparison, the number of photons applied to the images in Figure 7c and Figure 7d is 100,000. As a result, the reconstructed result by the conventional photon-counting imaging technique as shown in Figure 7c can visualize the object better than the image in Figure 7b, but it cannot accurately recognize the object due to the insufficient number of estimated photons. In contrast, the reconstruction result obtained using the COLaNoPS method as shown in Figure 7d can visualize the object better than the reconstruction result in Figure 7c, and the object can be visualized more clearly than the conventional photon-counting imaging technology when the same number of photons is estimated.

Figure 8 shows the background noise generated by the conventional photon-counting imaging method and the proposed method, and shows the improvement in the quality of the objects. In this experiment, we increase the brightness by 40% and decrease the contrast by 40% to visualize the background noise and object noise. Figure 8a is the result of the conventional photon-counting imaging technique. It shows the random noise in the background as the Poisson distribution is applied in all areas. Furthermore, objects cannot be accurately recognized when the number of estimated photons is insufficient. Figure 8b is the result of the COLaNoPS method. It shows that the background noise is also removed by applying the Poisson distribution only to the area where the object exists. Furthermore, by applying spatial overlap to the areas where photons overlap as the section moves, the object can be visualized more clearly and accurately than the result obtained using the conventional photon-counting imaging technique as shown in Figure 8a.

Figure 9 shows the images reconstructed for each object by the conventional photon-counting imaging technique and the COLaNoPS method, where the number of photons is 100,000. Figure 9a,d,g show reference images obtained under normal light conditions. The objects are listed in the order of car, truck, and bus. All of them are identifiable. Figure 9b,e,h show the results of applying the conventional photon-counting imaging technology. The insufficient photon count estimation makes it difficult to identify the object. Figure 9c,f,i is the result of applying the COLaNoPS method. As shown in Figure 9, in the conventional method, the numbers or letters cannot be recognized accurately due to random noise. On the other hand, we can recognize numbers or letters accurately using the proposed method.

Figure 10 shows the result of increasing the number of photons by 1,100,000. Figure 10a,d,g show the reference image obtained under normal light conditions. The objects are listed in the order of car, truck, and bus, and all of them are identifiable. Figure 10b,e,h show the result of applying the conventional photon-counting imaging technology. The objects are listed in the order of car, truck, and bus, and it is difficult to identify them due to the lack of estimated photons, similar to Figure 9. Figure 10c,f,i show the result of applying the COLaNoPS method. The objects are listed in the same order and the objects can be recognized. As shown in the results of Figure 9 and Figure 10, we can see that, as the number of assumed photons increases, the photons for the object are more accurately estimated. As a result, our proposed method can reduce the background noise and visualize the letters or numbers on the surface of the object more accurately than the conventional photon-counting imaging technique.

Although it is noticed that the proposed method can reconstruct the image under photon-starved conditions more accurately than the conventional method, the numerical comparison analysis may be required to verify our proposed method. In this paper, we use the structural similarity index measure (SSIM) metric for numerical comparison [27]. In SSIM analysis, we measured the change in SSIM as we incremented the photons from 100,000 to 1,100,000. Figure 11 shows the SSIM results for the conventional photon-counting imaging technique and the COLaNoPS method via various numbers of photons. Figure 11a shows the performance metrics of SSIM on car images. The average SSIM of the proposed method is 0.6917, while that of the conventional method is 0.4987, and there is a difference of about 1.387 times. Figure 11b shows the performance metrics of SSIM on truck images, where the average SSIM of the proposed method is 0.6363 and that of the existing method is 0.4336, and a difference of about 1.467 times. Figure 11c shows the performance metrics of SSIM for bus images, and the average SSIM of the proposed method is 0.6697, while that of the existing method is 0.3801, a difference of about 1.761 times. It can be seen that the SSIM value increases with the position of the object, because the closer the object is, the more photons can be estimated, so the image can be reconstructed clearly. However, photon-counting imaging cannot accurately estimate photons for dark objects, and dark objects are subject to frequent random noise. As a result, dark objects such as cars and trucks have a lower SSIM. As shown in Figure 11, the proposed method in this paper shows better SSIM results than the conventional method.

Figure 12 shows the reconstructed images for each object using the conventional photon-counting imaging technique and the COLaNoPS method with 100,000 photons. Figure 12a,d,g show the reference images obtained under normal lighting conditions and show a magnified image of the numbers or letters on objects. Figure 12 b,e,h show the results of applying the conventional photon-counting imaging technique. It is difficult to identify the numbers or letters on objects due to insufficient photons. Figure 12c,f,i present the results of applying the COLaNoPS technique. As shown in Figure 12, the numbers or letters cannot be recognized accurately with the conventional method due to random noise, while they can be recognized accurately with the proposed method.

Figure 13 uses the peak signal-to-noise ratio (PSNR) [28] metric to compare numbers and letters on objects. In the PSNR analysis, we measured the change in PSNR when the number of photons increased from 100,000 to 1,100,000. Figure 13 shows the PSNR results of the conventional photon-counting imaging technique and the COLaNoPS method for different numbers of photons. Figure 13a shows the performance metric of PSNR for the image of letters on the car. The average PSNR of the proposed method is 15.94 and that of the conventional method is 14.85. Their difference is about 1.073 times. Figure 13b shows the performance metrics of PSNR for numbers or letters on the truck, where the average PSNR of the proposed method is 11.78 and that of the conventional method is 11.31. Their difference is about 1.041 times. Figure 13c shows the performance metrics of PSNR for characters on the bus, where the average PSNR of the proposed method is 11.44 and that of the traditional method is 7.647. Their difference is about 1.497 times. As shown in Figure 13, the proposed method in this paper shows better PSNR results than the traditional method.

Figure 14 shows the object recognition rate by applying the YOLOv5 model [29] to the results of the conventional photon-counting imaging technology and the COLaNoPS method. Figure 14a shows the recognition rate of objects for the image obtained under normal light conditions, where all objects are correctly recognized with a high recognition rate. Figure 14b shows the object recognition rates for images obtained under photon-starved conditions. Only the bus object is correctly recognized, and the truck is incorrectly recognized as a car. Figure 14c shows the recognition rate of objects for the reconstructed image using the conventional photon-counting imaging technology. Only the truck and bus objects are recognized, and it shows that the recognition rate of the two objects is low. Figure 14d shows the recognition rate of objects for the image reconstructed using COLaNoPS technology. It can be seen that all objects, such as car, truck, and bus, are recognized more accurately, and the recognition rate is higher than that of the conventional photon-counting imaging technique.

Finally, the proposed method removes background noise by applying a Poisson distribution only to the areas where objects exist under photon-starved conditions, and improves image quality by applying spatial overlap. As a result, the object can be visualized by estimating only the photons for the object, and the recognition rate of the object can be improved.

4. Conclusions

In this paper, we have proposed the COLaNoPS method to remove random noise from the background by estimating only photons for an object under photon-starved conditions. The proposed method has solved the background noise problem by calculating the total intensity of the background and the total intensity of the object as the section moves to estimate where the object exists, and applying spatial overlap to the areas where photons overlap. In addition, to verify the effect of the proposed method, we have compared the numerical analysis of the proposed method using SSIM performance metrics and the YOLOv5 model. As a result, the proposed method has been able to accurately estimate photons only for an object better than the conventional method, and improves the image quality of the object and can improve the object recognition rate of deep learning techniques.

In this experiment, we have applied the median, mean value, and the average of the maximum and minimum values to define the threshold for determining the presence of an object, and found that the median value can accurately classify the object. Figure 15 shows the difference in SSIM among median, mean, and min-max average. The median and mean values can classify the objects correctly, and we can see that the SSIM for the median value is slightly higher than that for the mean value. The min-max average does not classify correctly, so only the bus is visualized. Furthermore, we can see that the min-max average has the lowest SSIM.

The proposed method is expected to contribute to the overall development of technologies utilizing photon energy, including astrophotography, medical imaging, photon encryption, autonomous driving, and AR/VR technologies.

The method proposed in this paper is a technology that visualizes 2D images in a situation where the number of photons is insufficient, but it can be visualized as a 3D image, and it is judged that the distance information of the object and the image quality of the object can be improved when reconstructed as a 3D image [30,31,32].