Multi-Depth Computer-Generated Hologram Based on Stochastic Gradient Descent Algorithm with Weighted Complex Loss Function and Masked Diffraction

Abstract

In this paper, we propose a method to generate multi-depth phase-only holograms using stochastic gradient descent (SGD) algorithm with weighted complex loss function and masked multi-layer diffraction. The 3D scene can be represented by a combination of layers in different depths. In the wave propagation procedure of multiple layers in different depths, the complex amplitude of layers in different depths will gradually diffuse and produce occlusion at another layer. To solve this occlusion problem, a mask is used in the process of layers diffracting. Whether it is forward wave propagation or backward wave propagation of layers, the mask can reduce the occlusion problem between different layers. Otherwise, weighted complex loss function is implemented in the gradient descent optimization process, which analyzes the real part, the imaginary part, and the amplitude part of the focus region between the reconstructed images of the hologram and the target images. The weight parameter is used to adjust the ratio of the amplitude loss of the focus region in the whole loss function. The weight amplitude loss part in weighted complex loss function can decrease the interference of the focus region from the defocus region. The simulations and experiments have validated the effectiveness of the proposed method.

1. Introduction

The holographic three-dimensional (3D) display is considered one of the most ideal and promising 3D display technologies since it can reconstruct the whole optical wave field of the 3D scene. With the developments of computing technology, computer-generated holograms (CGHs) can be used to perform reconstruction of a 3D scene [1,2,3,4,5]. Double-phase holograms (DPHs) [6,7], the Gerchberg–Saxton (GS) algorithm [8], and its extended forms [9,10] are classic CGHs methods to generate the hologram. In recent years, optimizing the phase-only hologram by the stochastic gradient descent (SGD) method has been proposed [11,12,13,14] and developed rapidly.

In the process of SGD, the initial random phase is constantly updated as the loss value converges by iterating. The optimization process can approach the global minimum point of the loss function and obtain a high-quality hologram [15]. A traditional SGD method to generate multi-depth holograms is to compare the amplitude loss values between the reconstructed images of the hologram and the primitives at each depth and add the loss values together as a total loss. However, because there are no phase constraints in the optimization process, the phase distributions in the simulation reconstructed results are nonuniform. In addition, due to the non-ideal optical reconstruction system, this nonuniform phase distribution status will become worse; therefore, the optical reconstructed images are not consistent with simulation results, which are contaminated by speckle noise. Chen et al. [14] proposed a complex loss function based on the SGD method. Complex loss function can optimize both the amplitude term and phase term of the hologram. The phase distribution in the reconstructed results is nearly uniform and the speckle noise in the reconstructed results can be suppressed well. To use complex loss function, multiple images in different depths reconstruct each input image as the target planes at the corresponding depth [16,17]. The wave propagation of images in multi-depth to the target plane can be simulated by Fresnel diffraction or angular spectrum [18,19]. In the wave propagation process of multiple layers in different depths, the layers behind will gradually diffuse as the wave propagation and the occlusion will occur at the boundary of the previous layer. This occlusion problem will aggravate as the number of layers increases, especially when the contents of the input images are complex. Zhang et al. [20] adopted a mask to solve this occlusion problem. The mask is implemented in the process of forward propagation layer by layer and the usage of the mask leads to an outstanding display effect. Inspired by this method, we design a mask to solve occlusion problems between different layers. Whether it is forward wave propagation or backward wave propagation, the mask can decrease the interference between different layers. Otherwise, we add the amplitude loss of the focus region to complex loss function and adopt a weight parameter to adjust the ratio, which can reduce the interference of the focus region from the defocus region and promote the attention of observers to the focus region.

In this paper, the SGD method with weighted complex loss function and masked multi-layer diffraction is proposed to generate high-quality holograms of a 3D scene with multi-depth. Firstly, input image layers in different depths will diffract from the corresponding depth to the target plane. In the process of diffraction, a mask is used to solve the occlusion problem. By repeating the first step, we can get different target planes in different depths. Then, by comparing the real part, the imaginary part, and the amplitude part of the focus region of the holographic reconstructed images with that of the target planes at each depth and then adding them together as weighted complex loss function, the initial hologram can be updated constantly as the optimization process is close to the global minimum point of the loss function. Because of the usage of weighted complex loss function, the amplitude and the phase information of the holographic reconstructed images will be optimized simultaneously. Moreover, the weight parameter is used to adjust the ratio of the amplitude loss of the focus region in the whole loss function, and the amplitude loss part of the focus region can decrease the interference of the focus region from the defocus region. The proposed method can improve the quality of the hologram of the 3D scene with multi-depth.

2. Methods

The diagram of the proposed method is shown in Figure 1. Firstly, we adopt the target planes formed by layers diffracting in different depths as the primitives at each depth. Through the back and forward wave propagation of different layers, we can obtain the target planes at corresponding depths. The distance between adjacent layers is $\mathsf{\Delta }z$ and the mask is used to solve the occlusion problem in the diffracting process of layers. Then, the hologram is reconstructed in the different depths ($z_1=z_r$, $z_2=z_r+\mathsf{\Delta }z$). The sum of the real loss, the imaginary loss, and the amplitude loss of the focus region between the target planes and the reconstructed images are implemented as weighted complex loss function. We use weighted complex loss function for the SGD method updating the initial random phase in the process of the SGD. After many iterations of the SGD method, we can get a high-quality multi-depth hologram.

We take the forming process of the $1$st target plane as an example. The $2$nd layer propagates back to the $1$st layer to form the $1$st target plane. The angular spectrum method (ASM) is used to simulate wave propagation. The complex amplitude of the $i$th layer after wave propagation is given by:

$$U_i(x,y)=IFFT\{FFT[A_i(x,y)]\times H_i\},$$

(1)

where $A_i\left(x,y\right)$ is the amplitude of the $i$th layer, $FFT$ represents the fast Fourier transform operator, and $IFFT$ represents inverse $FFT$. $H_i$ represents the transform function in ASM. The expression of $H_i$ is expressed by the following equation:

$$H_i=\left\{\begin{array}{l}{e}^{i\frac{2\pi }{\lambda }{z}_i}\sqrt{1-{(\lambda f_x)}^2-{(\lambda f_y)}^2},if\sqrt{f_x{}^2+f_y{}^2}<\frac{1}{\lambda }\\ 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}otherwise\end{array}\right.,$$

(2)

where $z_i$ is the diffraction distance, $\lambda$ is the wavelength, and $f_x$, $f_y$ are the spatial frequencies in the x and y dimensions. To solve the occlusion problem in the diffracting process, the complex amplitude of the focus layer (the $1$st layer) after wave propagation needs to multiply a mask:

$$M(m,n)=\left\{\begin{array}{l}0,\hspace{1em}\hspace{1em}\hspace{1em}0<m<960,0<n<1080\\ 1,\hspace{1em}\hspace{1em}\hspace{1em}960<m<1920,0<n<1080\end{array}\right.,$$

(3)

where m and n are the numbers of sampling points of the object in the x and y dimensions, respectively. By setting the mask with a binary value, it is effective to pass the wavefront where the region of $960<m<1920, 0<n<1080$ and let the rest block. Then, the complex amplitude of the $2$nd layer (the defocus layer) $U_2\left(x,y\right)$ is added to the product of the mask and the complex amplitude of the $1$st layer (the focus layer) $U_1\left(x,y\right)$, which forms the complex amplitude of the $1$st target plane:

$$C_1(x,y)=U_1(x,y)\times M(m,n)+U_2(x,y),$$

(4)

Repeating the analogous operation of the forming process of the $1$st target plane, the $2$nd target plane can be obtained. However, unlike the formation process of the $1$st target plane, the formation process of the $2$nd target plane is the forward propagation between layers.

We adopt a random phase $p_{initial}$ in the first iteration of the SGD as the hologram. In the process of SGD, the import phase updated by the previous iteration is reconstructed by ASM, which is described as:

$$R_i=IFFT\{FFT[h]\times H_r\},$$

(5)

where $R_i$ is the complex amplitude of the reconstructed result of the hologram in different depths, $h$ represents the wavefront of $p_{initial}$, which is expressed as $h=e^{i{p}_{initial}}$. $H_r$ represents the transform function in the process of reconstruction, the expression of which is:

$$H_r=\left\{\begin{array}{l}{e}^{-i\frac{2\pi }{\lambda }{z}_r}\sqrt{1-{(\lambda f_x)}^2-{(\lambda f_y)}^2},\hspace{1em}\hspace{1em}if\sqrt{f_x{}^2+f_y{}^2}<\frac{1}{\lambda }\\ 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}otherwise\end{array}\right.,$$

(6)

where $z_r$ is the diffraction distance between the hologram and the $j$th target plane, $\lambda$ is the wavelength, and $f_x$, $f_y$ are the spatial frequencies in the x and y dimensions.

The loss function plays an important role in the SGD optimization process. Images in different depths will focus at the corresponding depth while the other image will be blurry, so the focus region needs more attention than the defocus region [21]. In the proposed method, the sum of the real loss, the imaginary loss, and the amplitude loss of the focus region between the reconstructed images of the hologram and the target planes are implemented as weighted complex loss function, which is given by:

$$Los{s}_{sum}={\displaystyle \sum _{j=1}^{n}MSEloss(R_r,R_t)}+MSEloss(I_r,I_t)+m\times MSEloss(A_r\times M_j,A_j\times M_j),$$

(7)

where $Los{s}_{sum}$ is the loss sum, n is the number of the planes, $R_r$, $I_r$ and $A_r$ are the real part, the imaginary part, and the amplitude of the holographic reconstructed image, $R_t$, $I_t$ and $A_j$ are the real part, imaginary part, and the amplitude of the $j$th target plane, the parameter $m$ represents the weight, and $M_j$ represents a binary mask highlighting the focus region of the holographic reconstructed results and the target planes. MSEloss is the mean squared error, which can be expressed as:

$$MSEloss=\frac{1}{mn}{{\displaystyle \sum _{m,n}\left[S_r(m,n)-S_t(m,n)\right]}}^2,$$

(8)

where S_r and S_t represent the results of the holographic reconstructed image and the target planes, respectively.

The amplitude loss part of the focus region in weighted loss function can make the boundary between the focused region and the defocused region clearer. By setting $m$ and $M_j$, it can promote the attention of observers to the focus region and control the display effect of the focus region and the defocus region.

The SGD method can optimize the initial phase in succession based on weighted complex loss function by iterating. Finally, the hologram of the 3D scene and its reconstructed image can be obtained. The mask can solve the occlusion problem between different layers in the process of the images diffracting in different depths. In addition, weighted complex loss function can improve the quality of the hologram and make the edge between the focus region and the defocus region clear.

3. Results

3.1. Simulation Results

To verify the feasibility of the method, we performed relevant experiments. Figure 2a ‘butterfly’ and (b) ‘zebra’ from the DIV2K dataset are used to represent two different depth layers of the 3D scene in the experiments. The resolutions of ‘butterfly’ and ‘zebra’ including the black areas are all 1080 × 1920 pixels. The distance $z_r$ between the hologram and the first target plane is 230 mm. To observe the influence of the occlusion effect when the distance between adjacent layers is different, we set the distance between adjacent layers $\mathsf{\Delta }z$ to 10 mm, 30 mm, and 50 mm, respectively. The spatial light modulator (SLM) has 1080 × 1920 pixels, the pixel pitch is 8 μm, and the wavelength of the light source is 639.0 nm. During simulations, the initial phase $p_{initial}$ is supposed to be a random phase with a range of (−π, π). In a large number of simulation experiments, we find that the display effect between the focus region and the defocus region will achieve an appropriate balance when the weight parameter is between 4.5 and 5. Between 4.5 and 5, we adopt 4.8 as the weight parameter $m$in weighted complex loss function. The number of the iteration in the process of SGD is set to 100 and the learning rate of SGD is set to 0.1. Pytorch 1.10.0 and Python 3.8.0 are implemented to optimize the process of SGD. The Adam optimizer is used to update the learning rate in the optimization process.

Figure 3 shows the results (the $\mathrm{fir}$st target plane and the second target plane) after the wave propagation of layers in different depths without the mask and with the mask when the distance $\mathsf{\Delta }z$ is 10 mm, 30 mm, and 50 mm, respectively. As shown in Figure 3a₁–l₁, the occlusion problem of the wave propagation without the mask is aggravated according to the increase of the propagation distance. The usage of the mask can solve this occlusion problem well.

Figure 3 shows the simple vertical occlusion case. Otherwise, we use the mask in a complex situation to verify the validity of the mask. As shown in Figure 4, we use ‘dragon’ and its depth map to simulate complex diffracting processes.

By setting the depth values in different ranges, we can obtain different layers. After the wave propagation of layers, we can get the target plane. Figure 5 is the comparison of the results after the wave propagation of layers generated by the ASM without the mask and with the mask. We can see that the edge between different layers after the wave propagation with the mask is clearer. The mask is still effective while handling irregular graphics.

After obtaining the $\mathrm{fir}$st target plane and the $\sec o$nd target plane, we use the SGD method with weighted complex loss function to update the initial phase and obtain high-quality holographic reconstructed images. Because the amplitude loss part of the focus region is used in the whole loss function, the focus regions of the reconstructed images acquire more attention than the defocus regions and obtain higher quality. The comparisons of the reconstructed results between the SGD method with complex loss function and the proposed method when $z_r$= 230 mm and $\mathsf{\Delta }z$ is 30 mm are shown in Figure 6. The interference between the focus regions and the defocus regions of the proposed method is slighter than that of the SGD method with complex loss function.

Figure 7 shows the SGD method with amplitude loss function to generate multi-depth holograms. The SGD method with amplitude loss function only calculates the amplitude loss values between the reconstructed images of the hologram and the primitives at each depth.

The comparisons of reconstructed results among the SGD method with amplitude loss function and the proposed method are given in Figure 8. Figure 8a–d gives the reconstructed results of the SGD method with amplitude loss function. Because of only calculating the amplitude loss values between the reconstructed images of the hologram and the primitives at each depth, there are no phase constraints in the optimization process, so the reconstructed image quality both of the focus regions and the defocus regions is unsatisfactory. As shown in Figure 8e–h, the reconstructed results generated by the proposed method are of high quality, and the focus regions of the reconstructed images almost do not contain the interference from the defocus regions. (a₁–h₁) and (a₂–h₂) are the details of the previously reconstructed images in the red box region. The peak signal-to-noise ratio (PSNR) values and structural similarity (SSIM) values in the red box region of the proposed method are all higher than that of the SGD method with amplitude loss function, which shows that the quality and display effect of the reconstructed images in different depths generated by the proposed method are all better.

As shown in

Table 1. Comparison of the optimization time for 100 steps.

Method	${\mathit{z}}_{\mathit{r}}$ = 230 mm, $\mathsf{\Delta }\mathit{z}$= 10 mm	${\mathit{z}}_{\mathit{r} }$= 230 mm, $\mathsf{\Delta }\mathit{z}$= 30 mm	${\mathit{z}}_{\mathit{r}}$= 230 mm, $\mathsf{\Delta }\mathit{z}$= 50 mm
The proposed method	48.9338 s	47.5858 s	48.8018 s
The SGD method with complex loss function	30.6387 s	28.7206 s	28.1467 s

, we compared the time consumed between the SGD method with complex loss function and the proposed method. Because the loss function of the proposed method is calculated at each target plane, the proposed method does consume more time than the SGD method with complex loss function. It is indeed a limitation of the proposed method.

3.2. Optical Results

The optical setup is shown in Figure 9. The 4-f filter system is used in the optical reconstruction process of the hologram, and the focal lengths of the two lenses (lens 1 and lens 2) are both 150.0 mm. A phase-only SLM (8 µm pixel pitch, 1080 × 1920 pixels) is used, which is provided by HOLOEYE company. The frame rate and the phase modulation range of the SLM are 60 Hz and [0, 2π], respectively. The wavelength of the input red light laser is 639.0 nm. The hologram is displayed on the SLM and the reconstructed results were captured by a complementary metal oxide semiconductor (CMOS) camera (Canon 60D).

Figure 10a,b shows the optical reconstructed results of the SGD method with weighted complex loss function when the mask is not used in the process of the diffraction. Because the mask is not used, the results contain serious interference between different layers. Figure 10c,d shows the optical reconstructed results of the SGD method with complex loss function when the mask is used in the process of diffraction. The aliasing between different layers is slighter than the former method. Figure 10e,f gives the optical reconstructed results of the proposed method. Because of the usage of the mask and weighted complex loss function, the interference between the focus regions and the defocus regions of the proposed method is the slightest among these three methods, and the image edge are also the clearest.

The comparisons of the optical reconstructed results between the SGD method with amplitude loss function and the proposed method are given in Figure 11. As shown in Figure 11a–d, the optical reconstructed results of the SGD method with amplitude loss function are not consistent with the simulation results, which are still contaminated by speckle noises because the optical reconstruction system is not an ideal system and it aggravates the nonuniform status of the phase distribution. The optical reconstructed images of the proposed method are shown in Figure 11e–h, and their display effect is more excellent than that of the SGD method with amplitude loss function. Compared with the SGD method with amplitude loss function, the optical reconstructed images generated by the proposed method agree well with the simulation results, and the speckle noise is suppressed well. Because the proposed method optimizes phase and amplitude simultaneously, the phase distribution of the optical reconstructed image is more uniform than that of the SGD method with amplitude loss function. Therefore, the influence from the error of a nearly uniform phase distribution is smaller than a nonuniform phase distribution even in a non-ideal optical reconstruction system.

4. Conclusions

In conclusion, we propose a new SGD method to generate a high-quality hologram of a 3D scene with multi-depth. The mask decreases the interference between different layers in the diffracting process of different layers, which ensures the focus region is not contaminated by the defocus region. Weighted complex loss function can improve the quality and control the display effect of the focus region and the defocus region of the hologram.