Modified Multiresolution Convolutional Neural Network for Quasi-Periodic Noise Reduction in Phase Shifting Profilometry for 3D Reconstruction

Abstract

Fringe profilometry is a method that obtains the 3D information of objects by projecting a pattern of fringes. The three-step technique uses only three images to acquire the 3D information from an object, and many studies have been conducted to improve this technique. However, there is a problem that is inherent to this technique, and that is the quasi-periodic noise that appears due to this technique and considerably affects the final 3D object reconstructed. Many studies have been carried out to tackle this problem to obtain a 3D object close to the original one. The application of deep learning in many areas of research presents a great opportunity to to reduce or eliminate the quasi-periodic noise that affects images. Therefore, a model of convolutional neural network along with four different patterns of frequencies projected in the three-step technique is researched in this work. The inferences produced by models trained with different frequencies are compared with the original ones both qualitatively and quantitatively.

1. Introduction

The fringe projection is one method without contact that permits to measure heights from objects to generate 3D objects, and it is considered one of the most reliable for this aim [1,2,3].

The acquisition of 3D information is very essential in many areas, e.g., computer vision [4,5,6], industrial applications [7,8,9], optics [10,11], and biomedical applications [12,13,14], among others [15]. However, this method presents an inconvenience in the final 3D reconstruction due to the quasi-periodic noise [16,17,18,19,20] that is produced during the acquisition of images at the stage of phase unwrapping [21,22]. This stage of phase unwrapping recovers the 3D information from the image capture depending on the number of images. In this work, we apply the three-step technique [1,17], and therefore three images are required. This quasi-periodic or Moire noise, as is also known, has the particularity of affecting the shape of the 3D object [8,23,24,25], as it is shown in Figure 1, and it depends on the frequency of the pattern employed in the projection. This number of frequencies affects the way the noise appears in the images, as it is shown below in Figure 1 and Figure 2.

The reduction or elimination in periodic or quasi-periodic noise, known as Moire noise, began as soon as the first digital images could be obtained; however, it was not until it was analyzed in terms of how much noise was produced that research began on ways to attenuate or eliminate it from the images. Once the noise on the images was detected and analyzed, it was found that it is formed in a repetitive pattern and in different ways. Many studies have been conducted to reduce or eliminate such quasi-periodic noise, some processing the image in its spatial domain [18], others in its frequency domain [17]. In recent studies, thanks to the advances in artificial intelligence, specifically in the field of deep learning, images can be processed for different tasks, including image reduction or restoration. Convolutional neural networks are networks composed of neurons and are part of deep learning. They are composed of many layers of stacked neurons. These networks are designed to process an image by convolution, which is a technique that infers a pixel by calculating an average from information of neighboring pixels [26].

When all pixels of an image are completely processed, a complete image is produced by a model that is trained for this specific task. Many tasks are carried out with convolutional neural networks, such as classification [27,28], segmentation [29,30], restoration [31,32], object detection [33,34], among other tasks [35,36,37,38,39].

In this work, we propose a convolutional neural network to restore images affected by quasi-periodic noise in the process of 3D reconstruction by using the technique of fringe projection in three steps. The trained model will act as an image pre-processor by reducing the repetitive pattern present in the affected images, whose pattern appears like horizontal fringes that affect the surface of an object, improving the speed of this stage and obtaining an accurate 3D object. The convolutional neural network is based on the same architecture proposed by Sun [40], namely the multiresolution convolutional neural network, for the reduction in Moire patterns in digital images whose parameters will be described below. Section 3 will describe the results and Section 4 will present the conclusions.

2. Materials and Methods

The software Blender (https://www.blender.org/) emulates a 3-step fringe profilometry system and the 3D object models used for generating the database were acquired from platform Turbosquid and were free to use. Here, 75 different 3D models were used, and in Figure 3 some examples of these models are shown.

In the simulated system, a lamp is used to project the fringes over objects; then, pictures are acquired with four different patterns. For capturing the images, a camera with a focal length setting of 28 mm [41] was selected and the size of the captured images was 512 × 512 pixels. With the simulation system, a database of 1350 images with different objects at different positions was generated, but as four frequencies were applied to the pattern projected over the object, the total images were 5400. Each scene was composed of 16 different pictures plus 12 more that correspond to the references −3 for each different pattern projected shared by every scene. Figure 4 shows a complete set of all pictures of a single scene that conform to the generated database [17,41].

Figure 2 shows a single picture affected by four different patterns of quasi-periodic noise, and the process to obtain such images is shown in Figure 5.

All images were obtained using a laptop with NVIDIA GeForce RTX 3060 graphic card with 6 Gb of memory RAM, 16 Gb of memory RAM, and an I7-10750H processor @2.60 GHz. The images for training were 90% of 1050 images, for validation 10% of 1050 images, and 300 additional images were used for the test set. All this was only used to train a model with one single frequency, which was either 4, 8, 16, or 32. For the training of a model with multiple frequencies, were combined images with all four frequencies, adding up to a total of 4200 images, 90% for the training set, 10% for the validation set, and 1200 images affected with the four different patterns for the test set. Figure 6 shows some images with quasi-periodic noise at different frequencies and their respective targets generated by Blender.

The database of images generated with Blender includes three images of different 3D models with a pattern of 4 different frequencies and shifting 120° degrees. Applying a phase unwrapping algorithm (in this case, the PEARLS algorithm [21]) to obtain an absolute phase image, results in the generation of an image with noise known as quasi-periodic noise or Moire noise, as shown in Figure 2. Such noise is inherent in the technique of 3-step fringe profilometry to obtain the heights of an object from images using a single camera and affects the final 3D object reconstructed by altering its shape and losing 3D information.

When the stage to obtain the absolute phase of images pre-processed with the PEARLS algorithm is finished, the images are used to generate a database. The algorithm called PEARLS (Phase Estimation using Adaptive Regularization based on Local Smoothing) is described in the following pseudocode:

1.

Each pixel $(x,y)$ in $h\in H$

(a)

The zero-order phase $\widetilde{{\phi }_h}\left(x,y\right)$ estimate is calculated;

(b)

Adaptive window size is applied to estimates $\widetilde{{\phi }_h}\left(x,y\right)$ to properly select a window size $h^{+}(x,y)$;

(c)

Compute first-order phase estimates with adaptive window size;

(d)

end.

2.

Unwrap the phase $\widetilde{{\phi }_{h^{+}}}$ using one of the procedures developed for noise-free data.

For further information, see [21].

The database is then used as the source for training a CNN to learn to reduce or eliminate the quasi-periodic noise present in images that are affected by such noise. Finally, the trained model is used in the stage of pre-processing to generate a 3D object as a filter of noise. The process of the methodology to generate the images to train a model to reduce the noise in images is shown in Figure 5.

Once the database of images is obtained, a convolutional neural network based on Multiresolution-CNN proposed by Sun [40] is applied. The proposed model was modified to have 9 layers after the down-sampling and up-sampling operation with 3 × 3 kernel and 64 channels which were completely convolutional, contrary to the original one that had 5 layers in this stage of the architecture. In addition to this change, a layer as input and output of grayscale images or one channel was added as well. Figure 7 shows the architecture developed and implemented.

The original Multiresolution-CNN model was developed to reduce the Moire noise in color and white and black images. Therefore, the proposed model was modified to be trained with images that contain both quasi-periodic and Moire noise and reduce such noise. The trained model to reduce such noise was finally used as part of pre-processing images to generate a 3D object, improving the speed and the quality of the 3D objects generated. The novelty of this paper relies on the proposal of a modified multiresolution CNN in order to reduce the quasi-periodic noise on phase shifting profilometry at four different frequencies to generate a more reliable 3D reconstruction of an object.

The model was trained using a set of 1050 grayscale images with a size of 549 × 540 but adjusted to a size of 512 × 512 before feeding the model, and five experiments were carried out, one for every frequency present in the images. The last experiment was carried out using the set of images of every frequency gathered in one set of 4200 images. The projected frequency patterns were 4, 8, 16, and 32 fringes. Every experiment was performed using the optimizer Adam() [42,43] and the MSELoss() function [44] to calculate the training and validation loss. Internally, the algorithm took 10% of images randomly to be used as validation set in every training.

The fringe profilometry method allows for obtaining information on object heights through images. Therefore, a large number of images with a wide variety of shapes, surfaces, and contours are required to remove this specific noise. Although there are techniques to augment data and give them variety during the training of models for noise reduction and image restoration, this first approximation was carried out without data augmentation. This is performed in order to observe the results obtained and make the corresponding improvements. However, since this is a specific noise to be reduced, we leave some training techniques for future work. For now, we just add a large variety of objects to have a model trained with enough data to generalize to the greatest number of possible scenarios or objects to reduce or restore noise in images affected by quasi-periodic noise.

The selection of the neural network architecture is based on a previously published article, wherein a comparative study between three different architectures was carried out and the most appropriate neural network for this purpose was selected using performance criteria [23].

2.1. Optimizer and Loss Function

The optimizer Adam() has the advantage of requiring little memory, and it is computationally efficient and has an adaptive estimation to calculate moments of first and second order.

$$m_t={\beta }_1{m}_{t-1}+\left(1-{\beta }_1\right)g_t$$

(1)

$$v_t={\beta }_2{v}_{t-1}+\left(1-{\beta }_2\right)g_t^2,$$

(2)

$g_t$ evaluates the gradient in a timestep t, $m_t$ calculates the average of moving, $v_t$ is the squared gradient, and ${\beta }_1$ and ${\beta }_2$ calculate the decay rates for every moment estimates.

Equation (3) calculates the MSELoss() function

$$l(x,y)=L={\left\{l_1,\dots ,l_N\right\}}^T,l_n={(x_n-y_n)}^2,$$

(3)

the batch-size is represented by N, x, and y which represent the dimensions that form a matrix of a given size with n elements [42,43].

2.2. IMMSE

The inverse mean square error (IMMSE) is a metric used to evaluate the quality of reconstructed images by comparing the original image with the generated image. The IMMSE formula is based on the calculation of the mean square error (MSE), but is applied in an inverse manner [45]. This equation is shown below

$$IMSSE={\displaystyle \frac{1}{mn}}\sum _{i=1}^m\sum _{i=j}^n{(I(i,j)-K(i,j))}^2$$

(4)

where I is the original image, K is the processed image, m number of rows, n number of cols, and $K(i,j)$ is the value of the corresponding pixel in the reconstructed image. The IMMSE provides a measure of how similar the two images are, where higher values indicate better quality.

2.3. PSNR (Peak Signal-to-Noise Ratio)

Peak Signal-to-Noise Ratio (PSNR) is a widely used metric to assess the quality of compressed or reconstructed images. PSNR measures the ratio of the maximum power of a signal (the original image) to the noise that affects the quality of its representation (the reconstructed image) [45,46]. It is defined as follows:

$$PSNR(f,g)=10lo{g}_{10}({255}^2/MSE(f,g))$$

(5)

where

$$MSE(f,g)={\displaystyle \frac{1}{MN}}\sum _{i=1}^M\sum _{j=1}^N{(f_{ij}-g_{i,j})}^2$$

(6)

A higher PSNR indicates that the reconstructed image is more similar to the original, i.e., it has less noise. Typical PSNR values for high-quality images are in the range of 30–50 dB.

3. Results

The results were obtained by experimenting with different parameters and the parameters proposed by the author of the model, which were based on the model proposed in [40]. These parameters are summarized in

Table 1. Parameters used during network training for comparison, trained with images affected by quasi-periodic noise at four different patterns (4, 8, 16, and 32 frequencies), as seen in Figure 2.

Parameter	Pattern 1 (Number of Fringes 4)	Pattern 2 (Number of Fringes 8)	Pattern 3 (Number of Fringes 16)	Pattern 4 (Number of Fringes 32)	Pattern 5 (Multifrequency Pattern)
Batch size	4	4	4	4	4
Initials weights	Gaussian random (average = 0.0, standard deviation = 0.01)	Gaussian random (average = 0.0, standard deviation = 0.01)	Gaussian random (average = 0.0, standard deviation = 0.01)	Gaussian random (average = 0.0, standard deviation = 0.01)	Gaussian random (average = 0.0, standard deviation = 0.01)
Bias	0.0	0.0	0.0	0.0	0.0
Learning rate	0.007	0.007	0.007	0.007	0.007
Optimizer	Adam()	Adam()	Adam()	Adam()	Adam()
Training loss	MSELoss()	MSELoss()	MSELoss()	MSELoss()	MSELoss()
Validation loss	MSELoss()	MSELoss()	MSELoss()	MSELoss()	MSELoss()
Test planing (train, val)	90%, 10%	90%, 10%	90%, 10%	90%, 10%	90%, 10%
Images size (Width, Height)	$512\times 512$ pixels	$512\times 512$ pixels	$512\times 512$ pixels	$512\times 512$ pixels	$512\times 512$ pixels
Set train images	1050	1050	1050	1050	4200
Set validation images	105	105	105	105	420
Set test images	300	300	300	300	300

.

In addition to the parameters shown in Table 1, 50 epochs were set and, if the obtained model had a better validation loss, it was saved as the best, but if a bad validation loss was obtained, the model was penalized. After training the model, the results obtained after every epoch were charted to show the evolution of the training and validation loss and were scaled for a better appreciation of the loss. The graphs of the evolution of every training are shown in Figure 8.

Figure 8 shows the training and validation loss of every model trained. The evolution of training and validation loss shows a constant decline, which is consistent and shows that the model is effectively “learning”, and it is also observed that the learning is performed rapidly and at the end the rate of learning is very low.

The time of learning and the training and validation loss are shown in

Table 2. Time employed to perform each training and training and validation loss reached during network training for comparison using images with four different patterns (4, 8, 16, and 32 frequencies), as seen in Figure 2.

	Pattern 1 (Number of Fringes 4)	Pattern 2 (Number of Fringes 8)	Pattern 3 (Number of Fringes 16)	Pattern 4 (Number of Fringes 32)	Pattern 5 (Multifrequency Pattern)
Training loss	0.10275	0.11939	0.09801	0.08825	0.12041
Validation loss	0.11187	0.10390	0.10042	0.09749	0.10443
Training time (HH:MM:SS)	0:59:37	1:08:49	0:58:12	1:00:16	5:20:22

.

According to the data obtained after performing the training of the models, these trainings took around one hour to complete, while the training with the set that contains all the images with the four frequencies lasted a bove five hours because its set contained more than 4000 images. The training and validation loss reached values equal to or below 0.1, indicating a constant learning by the trained models.

3.1. Inferences Obtained from Images Affected with Quasi-Periodic Noise of 4 Frequencies

The inferences obtained from images affected by quasi-periodic noise composed of four frequencies using all the trained models are shown in Figure 9, and the 3D reconstructions are shown in Figure 10.

The profiles obtained from these inferences, the ground-truth image, and the original image affected by quasi-periodic noise of four frequencies are compared and are charted in Figure 11. The heights are normalized from 0.0 to 1.0 and the x-axis represents pixels.

The error between the inferences made by the models trained and the ground-truth image is identified using the PSNR, SSIM, IMMSE, and the MSE Profile between the inference and the ground-truth image. The measures obtained for the images affected by quasi-periodic noise of four frequencies are summarized in

Table 3. Measures obtained with model trained with images affected by noise of four frequencies.

Inference	IMMSE	SSIM	PSNR	MSE (Profile)
1	0.022	0.871	64.676	0.064
2	0.017	0.879	65.767	0.048
3	0.033	0.828	62.900	0.089
4	0.046	0.793	61.547	0.124
5	0.012	0.873	67.263	0.034

.

3.2. Inferences Obtained from Images Affected with Quasi-Periodic Noise of 8 Frequencies

The inferences obtained from images affected by quasi-periodic noise composed of 8 frequencies using all the trained models are shown in Figure 12, and the 3D reconstructions are shown in Figure 13.

The profiles obtained from these inferences, the ground-truth image, and the original image affected by quasi-periodic noise of eight frequencies are compared and are charted in Figure 14. The heights are normalized from 0.0 to 1.0 and the x-axis represents pixels.

The error between the inferences made by the models trained and the ground-truth image is identified using the PSNR, SSIM, IMMSE, and the MSE Profile between the inference and the ground-truth image. The measures obtained in images affected by quasi-periodic noise of eight frequencies are summarized in

Table 4. Measures obtained with model trained with images affected by noise of 8 frequencies.

Inference	IMMSE	SSIM	PSNR	MSE (Profile)
1	0.017	0.882	65.838	0.048
2	0.012	0.889	67.488	0.031
3	0.025	0.846	64.224	0.063
4	0.036	0.813	62.561	0.095
5	0.007	0.878	69.646	0.018

.

3.3. Inferences Obtained from IMAGES Affected with Quasi-Periodic Noise of 16 Frequencies

The inferences obtained from images affected by quasi-periodic noise composed of 16 frequencies using all the trained models are shown in Figure 15, and the 3D reconstructions are shown in Figure 16.

The profiles obtained from these inferences, the ground-truth image, and the original image affected by quasi-periodic noise of 16 frequencies are compared and are charted in Figure 17. The heights are normalized from 0.0 to 1.0 and the x-axis represents pixels.

The error between the inferences made by the models trained and the ground-truth image is identified using the PSNR, SSIM, IMMSE, and the MSE Profile between the inference and the ground-truth image. The measures obtained in images affected by quasi-periodic noise of 16 frequencies are summarized in

Table 5. Measures obtained with model trained with images affected by noise of 16 frequencies.

Inference	IMMSE	SSIM	PSNR	MSE (Profile)
1	0.014	0.886	66.517	0.043
2	0.009	0.903	68.549	0.025
3	0.017	0.897	65.771	0.050
4	0.028	0.872	63.609	0.082
5	0.005	0.914	71.465	0.011

.

3.4. Inferences Obtained from Images Affected with Quasi-Periodic Noise of 32 Frequencies

The inferences obtained from images affected by quasi-periodic noise composed of 32 frequencies using all the trained models are shown in Figure 18, and the 3D reconstructions are shown in Figure 19.

The profiles obtained from these inferences, the ground-truth image, and the original image affected by quasi-periodic noise of 32 frequencies are compared and are charted in Figure 20. The heights are normalized from 0.0 to 1.0 and the x-axis represents pixels.

The error between the inferences made by the models trained and the ground-truth image is identified using the PSNR, SSIM, IMMSE, and the MSE Profile between the inference and the ground-truth image. The measures obtained in images affected by quasi-periodic noise of 32 frequencies are summarized in

Table 6. Measures obtained with model trained with images affected by noise of 32 frequencies.

Inference	IMMSE	SSIM	PSNR	MSE (Profile)
1	0.010	0.905	68.307	0.027
2	0.005	0.923	71.543	0.011
3	0.010	0.922	68.098	0.028
4	0.019	0.901	65.273	0.054
5	0.002	0.927	75.116	0.002

.

4. Discussion

The inferences obtained from each trained model with a different set of images affected with quasi-periodic noise of different frequencies show, in every case, a better performance when the model is trained with a set that contains images affected with different frequencies instead of using only a set of images with one frequency. Although an image affected with quasi-periodic noise of only four frequencies appears to show a better similarity with the ground-truth image, preserving better details of the object, it is difficult for the trained models to obtain a better inference in quantitative terms. This is observed in the metrics shown in the Table 3, Table 4, Table 5 and Table 6. In quantitative terms, in the four inferences performed with 4, 8, 16, and 32 fringes, the IMMSE value is reduced in each inference compared to the original model. In addition, it is observed that the inferences with model 5, which was trained with multiple frequencies, presented a better performance in the SSIM, PSNR, and MSE (Profile) metrics, compared to models 1 to 4.

At first glance at the images with quasi-periodic noise, it can be seen that those that are affected by a lower frequency of such noise lose fewer details of the 3D object. However, as the number of fringe frequencies in the projected patterns increases, this quasi-periodic noise decreases, but only in size. It therefore merges and blends with the details of the 3D object, making it almost impossible to determine what is noise and what is part of the 3D information. Another effect shown by using a low fringe frequency in the projected patterns is that the final height of the object inferred by the model better preserves the original height of the object. This is clearly appreciated in the images that compare the profiles at each frequency of the projected pattern analyzed, where the inferences represented as model 3 (fringe pattern with 16 frequencies) and model 4 (fringe pattern with 16 frequencies) are always lower in normalized height.

It is expected that training using images affected by noise of the same frequency may adequately restore images with similar noise present; however, it was found that performance improved when using images affected by other frequencies. Therefore, training was carried out wherein all the images affected by different frequencies were put together, achieving better results. Although better results were obtained by generalizing the training data more by combining affected images with different frequencies, another limitation was the number of images. By increasing the number of images in the database, it may be possible to further improve the results obtained.

Generating data using Blender allows us to obtain data in a way that is very similar to real data. On the one hand, extensive methodologies must be followed, such as calibration of cameras and objects that do not have restrictions of any kind, while synthetic data save us time and can be used freely. Since models tend to mimic real-world objects, it is possible to use them to represent even people or people’s faces roughly, but without the inconvenience of having to obtain them from real people. Furthermore, one can include images of objects captured from the real world, and carry out the 3D reconstruction process at the testing stage.

These results show the difficulty of eliminating the quasi-periodic noise that affects this particular fringe profilometry method for 3D reconstruction, even when trying with different frequencies. Trying different frequencies was found out that the speed of acquisition of image by fringe profilometry of 3-step, while less the frequency pattern projected is faster than with a high frequency. The next research will aim to improve the inferences obtained by either increasing the number of images in the training set or trying other models of convolutional neural networks or networks known as GAN.

5. Conclusions

The experiments performed using a set of images affected with quasi-periodic noise of four different frequencies show how these frequencies affect the 3D object reconstructed and the results obtained when an inference is generated after training a CNN model with these images. Quantitative results show better performance when the model is trained with a set of images that contains, in this case, a quasi-periodic noise pattern of four different frequencies showing that images affected with a higher frequency are the ones that obtain a better result and visually show greater similarity with the ground-truth image.

On the other hand, using a model trained to reduce noise in images obtained in PSP increases the speed of image pre-processing to obtain a 3D object. Trying different frequencies to produce images with different kinds of noise helps to create a high variety of such noise in datasets to train models of CNNs, generating good results both quantitatively and qualitatively.