Curved Holographic Augmented Reality Near-Eye Display System Based on Freeform Holographic Optical Element with Extended Field of View

Abstract

At present, most near-eye display devices adopt flat substrates, which have problems such as limited field of view (FOV) and bulky shape, while the curved structure is expected to expand the FOV with appropriate volume. In this paper, we propose a curved holographic augmented reality (AR) near-eye display system based on holographic optical element (HOE) with the ability to expand the FOV. The system includes a display source and a HOE with curved substrate. We analyze the system by exploiting the diffraction theory between plane and curved surface, and a layered and weighted FOV optimization method using particle swarm optimization algorithm is proposed to realize the optimization of the phase of freeform HOE. Numerical and experimental results show that the proposed curved holographic near-eye display system can realize cylindrical AR display and expand the FOV of the system. It is expected to be applied to the holographic AR near-eye display in the future.

1. Introduction

With the increasing demand for information acquisition in today’s era, augmented reality (AR) technology has become a hot topic in display. Holographic optical elements (HOEs) have been discovered due to their thinness, transparency, and unique angle/wavelength selectivity. Nowadays HOEs are used to fabricate holographic AR near-eye display devices, which reduce the volume of the optical system to a certain extent and obtain better light transmittance [1,2]. According to the different record light waves, HOEs can achieve different optical functions. For example, Mukawa H et al. [3,4,5,6] used holographic optical waveguides to make AR display lenses, and HOEs were used as in-coupling and out-coupling combiner; Kim S et al. [7,8,9,10,11,12,13] implemented AR displays in the form of Maxwell displays, and HOE functions as holographic mirrors or holographic lenses. In summary, the current implementation of holographic AR near-eye display devices mostly relies on planar substrates. But the system still has problems such as limited field of view (FOV) [14] and insufficient size [15,16]. At this time, the curved structure is expected to bring larger FOV [17,18,19] and a more eligible appearance [20,21]. However, there is still a lack of research on holographic AR near-eye display based on curved substrate.

The concept of curved HOEs was proposed in the 1970s [22]. Subsequently, Fairchild derived the ray tracing formulas for curved HOEs in 1982 [23]. In 1986, Peng and Frankena investigated the aberration of spherical HOEs [24]. In 2019, Kiseung Bang et al. analyzed the influence of the surface curvature of curved HOEs on the imaging characteristics of optical systems using Uchida’s beta value method (BVM) [25]. According to the above related theories, the application of curved HOEs in holographic near-eye display has gradually gained attention. Travis et al. used curved HOEs to fabricate curved wedge waveguides [17,18]. But the design of curved waveguides is extremely complex, requiring consideration of the number of reflections and steps for rays of different incident angles and wavelengths inside the waveguide. Bang et al. prepared a HOE recorded on a planar substrate. Then pasted it on a curved substrate, which was used as an imaging combiner for the retinal projection display system. However, the deformation of the HOE after attaching to the curved substrate resulted in aberrations such as vertical astigmatism. Additionally, the exit pupil size of the retinal projection is small, and even a slight shift can cause the image to disappear. Tian Shu et al. proposed that polynomials could be used to characterize the phase of curved HOEs [26]. It could be used to achieve AR display on curved surfaces, and the calculation was more complicated. However, there are still few theories about the design and optimization of the phase of curved HOEs, and there is also a lack of a system that truly uses curved HOEs for near-eye imaging.

In this paper, we construct a curved holographic near-eye display system based on freeform HOE. The proposed system based on reflective HOE achieves a larger FOV compared to planar structures by adopting a curved structure. An off-axis tilted point source method from plane to curved surface is employed. This method simulates the curved imaging results accurately. Several characteristic images are used to layer the entire system’s FOV when optimizing the phase of HOE. Each characteristic image has a corresponding characteristic phase. These serve as the initial optimized phase distribution of the HOE. And particle swarm optimization algorithm is introduced to optimize the weight of each characteristic phase. The results show that the FOV of near-eye display can be enlarged using curved HOE. Both simulation and experiment confirm the feasibility of this method.

The rest of this paper is organized as follows. Section 2 outlines the proposed architecture of the system and principles for implement. Section 3 details the simulation and experimental results of the proposed system. Section 4 summarizes the research findings.

2. Principles and Methods

In this work, we propose a curved holographic AR near-eye display system based on freeform HOEs with extended FOV which uses reflective curved HOE. Then the curved phase distribution is optimized. The optimization method can be divided into two parts, including initial phase calculation and single-objective optimization.

2.1. System Design

The proposed curved holographic AR near-eye display system is shown in Figure 1, which consists of a display source that generates a target image, a reflective curved HOE that displays a virtual image, and a transparent curved glass substrate.

First of all, the projection image of the display source needs to be completely incident on the HOE at a certain tilt angle, and then the imaging unit diffracts the target image to form a virtual image entering the human eye. The imaging plane is located far in front of the hologram. It is necessary that the display source must not obstruct the viewing field. At the same time, because the HOE and the substrate are transparent, it does not affect the entry of external light. Since this imaging mode is similar to concave mirror imaging, it has the function of magnifying the image. So it has the advantage of expanding the FOV. The specific benefits are discussed in the next section.

2.2. Field of View

The difference between the FOV of curved structure and planar structure in near-eye display devices is analyzed. The width of the HOE is projected horizontally at $W$. The distance between the human eye and the center point of the HOE remains unchanged, and the intersection point of the edge line of the curved HOE with $O{O}_1$ is $O_2$. In addition, the length of $O{O}_2$ is $d_1$, and the length of $O_1{O}_2$ is $d_2$. As shown in Figure 2, assuming that the width of the curved HOE and the planar HOE is the same. The FOV of the two can be expressed by the following equation:

$$FO{V}_P={\theta }_P=2\arctan ({\displaystyle \frac{\frac{W}{2}}{d_1+d_2}})$$

(1)

$$FO{V}_C={\theta }_C=2\arctan ({\displaystyle \frac{\frac{W}{2}}{d_1}})$$

(2)

$FO{V}_P$ is the FOV of the near-eye display with planar structure, and $FO{V}_C$ is the FOV of the near-eye display with curved structure. It can be seen that when the width of the HOE which is projected horizontally and the distance between the human eye and the HOE are constant, the holographic near-eye display system with curved structure has larger FOV than the planar structure.

2.3. Calculation Principle

At present, most of the diffraction theories used for calculation are based on planes. Thus, in order to better analyze the diffraction rules of curved optical elements, it is necessary to develop diffraction theories suitable for curved surfaces. As we know, when the diffraction system satisfies the paraxial condition in the actual diffraction problem, the direction factor $D(\alpha )=1$. Taking the Cartesian coordinate system, the plane coordinate of the diffraction aperture is $(\xi ,\eta )$ and the coordinate of the investigation plane is $(x,y)$. The diffraction aperture plane is parallel to the investigation plane. The Kirchhoff diffraction integral formula for diffractive objects of arbitrary properties and arbitrary illumination conditions in Cartesian coordinate systems can be obtained. The complex amplitude of the observation point P can be expressed as

$$E(P)={\displaystyle \frac{1}{j\lambda }}{\displaystyle \int {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}A(\xi ,\eta )\frac{\exp (jkr)}{r}}}d\xi d\eta$$

(3)

$A(\xi ,\eta )=B(\xi ,\eta )T(\xi ,\eta )$ is the complex amplitude of the light wave through which the diffracted object is transmitted. $B(\xi ,\eta )$ is the distribution of the complex amplitude of the light wave emitted by any monochromatic light source when it reaches the plane $(\xi ,\eta )$ of the diffraction aperture, and the complex amplitude transmission coefficient of the diffractive object of any nature is $T(\xi ,\eta )$.

In the system proposed in this paper, the diffraction surface is an inclined plane and the observation surface is a curved surface. So, the paraxial condition is not satisfied, and the direction factor cannot be ignored. Depending on the Rayleigh-Sommerfeld diffraction formula, here we take the direction factor $D(\alpha )=\cos (n,r)$. n represents the normal of the diffraction surface, and r is the diffraction direction. Apparently, the diffraction field of the observation point P on the investigation plane can be represented by the following equation.

$$E(P)={\displaystyle \frac{1}{j\lambda }}{\displaystyle \int {\displaystyle \underset{-\infty }{\overset{\infty }{\int }}A(\xi ,\eta )\frac{\exp (jkr)}{r}}}\cos (n,r)d\xi d\eta$$

(4)

2.4. Optimization Principle

In order to achieve the optimization of curved HOE, its phase needs to be designed. We use the layered and weighted FOV method and the particle swarm optimization algorithm, as shown in Figure 3. Here’s a detailed explanation of these two steps.

In the first step, N feature images are selected to stratify the FOV of the whole system. Each feature image is calculated based on the diffraction compensation (DC) or approximate compensation (AC) method [27], which is shown in the step 1 of Figure 3. Then the difference between the complex amplitude of the diffracted image and the ideal image and the feedback parameter k are introduced as the initial amplitude distribution of the next iteration afterwards, which can accelerate the convergence speed of the overall calculation. So as to complete a round of iterations, and each feature image completes the same number of iterations in this way. Ultimately, we can obtain N feature phase distributions of curved HOE.

After completing the first step of optimization, the N feature phases are added linearly as the initial optimized phase of the curved HOE, and then the weights are used as variables. The expression is as

$${\mathrm{\Phi }}_{opti}=a_1 \times {\mathrm{\Phi }}_1+a_2 \times {\mathrm{\Phi }}_2\dots \dots +a_n{\mathrm{\Phi }}_n\hspace{1em}{a}_k\in \left[0,1\right] ,\hspace{1em}k=1, 2 \dots n$$

(5)

$a_1, a_2 \dots a_n$ are the weights of feature phases. The N weights are used as variables, followed by the value range set between 0 and 1. An image with universal applicability is introduced, and this representative image is calculated by the inclined point source method mentioned in Section 2.3. Through the iterative calculation, the weight values of feature phases in each round are updated, so the optimized phase of the curved HOE is updated at the same time. The particle swarm optimization algorithm is selected to carry out the above single-objective optimization process. With the increase of the number of iterations and speed, the weight finally converges to a certain value, and the final optimized phase distribution is obtained.

3. Numerical Simulation and Experimental Results

In order to verify the feasibility of the proposed system to achieve the near-eye display effect, we carry out numerical simulation and optical experiments on the proposed system, and the FOV of the curved structure and the planar structure are compared.

3.1. Numerical Simulation Method

For the calculation of diffraction between the inclined plane and the curved surface in the proposed system, the first step is to unify the coordinate system. Taking the calculation between two planes as an example. Assuming that the inclined plane $\mathrm{\Sigma }$ is in the coordinate system $x^{\prime }{y}^{\prime }{z}^{\prime }$ and the plane $\mathrm{\Pi }$ is in the coordinate system $xyz$. The transformation relationship unifying the two planes into the same coordinate system can be represented by the following equation:

$$\left\{\begin{array}{l}{u}^{\prime }=u\times \cos \theta +deltax\\ v^{\prime }=v\\ w^{\prime }=-u\times \sin \theta +deltaz\end{array}\right.$$

(6)

$$\left\{\begin{array}{l}deltax=d\times \sin \theta \\ deltaz=d\times \cos \theta \end{array}\right.$$

(7)

$u^{\prime }$ is the coordinate in x′-direction after conversion. u is the coordinate in x-direction before conversion. $v^{\prime }$ is the coordinate in y′-direction after conversion. $v$ is the coordinate in y-direction before conversion. $\theta$ is the angle between the inclined plane and the vertical direction. $w^{\prime }$ is the distance between calculated point and the inclined plane $\mathrm{\Sigma }$, and d is the distance between the center point of the plane $\mathrm{\Pi }$ and the center point O of the inclined plane $\mathrm{\Sigma }$.

Next, the coordinate conversion relationship between the inclined plane and the curved surface is discussed, which can be calculated as

$$\left\{\begin{array}{l}{u}_2^{\prime }=z_d\times \sin \theta +u_2\times \cos \theta +deltax\\ v_2^{\prime }=v_2\\ w_2^{\prime }=z_d\times \cos \theta -u_2\times \sin \theta +deltaz\end{array}\right.$$

(8)

$u_2^{\prime }$ is the coordinate in x′-direction after conversion. $u_2$ is the coordinate in x-direction before conversion. $v_2^{\prime }$ is the coordinate in y′-direction coordinate after conversion. $v_2$ is the coordinate in y-direction before conversion. $w_2^{\prime }$ is the distance between calculated point and the inclined plane $\mathrm{\Sigma }$. Suppose point A is the point on the curved surface that is closest to the center point O of the inclined plane in the z-direction, and $z_d$ is the distance between the calculated point on the curved surface and point A in the z-direction. The definition of $deltax$, $deltaz$, d and $\theta$ is the same as in the Equation (7).

Through the above formula, the oblique surface point source method can be realized, which can achieve a larger range and more accurate diffraction calculation compared with the original diffraction theory. Besides, a fast calculation method is also proposed. The diffraction calculation from the inclined plane to the plane is completed by using Equations (4), (6) and (7), and then the planar phase is compensated to the curved phase by using the phase approximation. If the optical path from the plane ${\mathrm{\Pi }}_1$ to the curved surface ${\mathrm{\Pi }}_2$ is $\mathrm{\Delta }d(x,y)$, the compensation phase can be expressed as

$${\mathrm{\Phi }}_P=k\times \mathrm{\Delta }d(x,y)\hspace{1em}\hspace{1em}k={\displaystyle \frac{2\pi }{\lambda }}$$

(9)

3.2. Numerical Simulation Results

In the numerical simulation, the resolution and pixel spacing of the target image are set to 1920 × 1080 and 3.7 μm, respectively. The resolution and pixel spacing of the HOE are set to 1920 × 1080 and 3.74 μm, respectively. The horizontal distance between the center point of the oblique display source and the edge of HOE is 20 mm, and the horizontal distance between the edge of HOE and the human eye is 25 mm. Then the curved HOE is set to a cylindrical surface with a horizontal curvature radius of 100 mm. When the display source is incident at different tilt angles, the diffraction of HOE is also different. In this paper, an inclination angle of 30 degrees is used as an example.

Next, we optimize the phase of the curved HOE. Select three feature images in the preliminary optimization, while set the number of iterations to 10. To complete the iteration of the particle swarm algorithm, the number of particles is set to 150 and the number of iterations is 25 times. At the same time, we define the signal-to-noise ratio (PSNR) of the image as the evaluation function for each iteration, which can be expressed as

$$PSNR=10\times {\log }_{10}({\displaystyle \frac{{255}^2}{MSE}})$$

(10)

$MSE$ is the mean square deviation of the ideal image and the diffraction image, which can be expressed as

$$MSE={\displaystyle \frac{1}{MN}}\underset{m=1}{\overset{M}{\mathrm{\Sigma }}}\underset{n=1}{\overset{N}{\mathrm{\Sigma }}}{\left[I_0(m,n)-I_r(m,n)\right]}^2$$

(11)

M and N are the number of pixels in the horizontal and vertical directions of the two images. $I_0(m,n)$ and $I_r(m,n)$ represent the pixel values of the ideal image and the diffracted image, separately. We compare the simulation results before and after optimization, as shown in Figure 4b,c, the corresponding PSNR values are also appended.

Based on the above system, we analyze the FOV of the planar and curved structure. For a reflective HOE with a fixed focal length, the record light waves are one plane wave and one spherical wave. For two coherent plane waves with an angle of $\theta$ between their wave vectors, the spatial period P of the interference field is a fixed value, and it can be calculated by

$$P={\displaystyle \frac{\lambda }{2\sin \frac{\theta }{2}}}$$

(12)

Therefore, the spatial period of the HOE recorded by a spherical wave and a plane wave is variable. However, due to the limitation of holographic exposure materials, when the calculated spatial period is less than a certain value, the interference field generated by the two light waves cannot be recorded on the material. It illustrates that the HOE has a certain effective area when recorded, which ultimately leads to the finite FOV.

From the above analysis, we calculate the difference between the FOV of the planar HOE and the curved HOE under certain conditions.

In the simulation, the wavelength of light is 532 nm. The refractive index of the material is 1.2, whose minimum spatial period is 0.00023 mm. Besides, the number of points sampled on the HOE surface is 500. As shown in Figure 4d–f, the FOV of a near-eye display system with planar structure and a near-eye display system with curved structure are compared under four different conditions. Figure 4d shows the relationship between the FOV and the focal length of the system under the condition that the width of HOE is 60 mm, the incident angle of the parallel light is 45° and the radius of curvature of the curved substrate is 75 mm. Figure 4e shows the relationship between the FOV and the width of the HOE as the focal length of the system is 40 mm, the incident angle is 45° and the radius of curvature is 75 mm. Figure 4f shows the relationship between the FOV and the radius of curvature when the focal length is 40 mm, the width of HOE is 60 mm and the incident angle is 45°. The solid line represents the FOV of the planar HOE, while the dotted line represents the FOV of the curved HOE. It is obvious that in all three cases, the FOV of the curved HOE is equal to or greater than that of the planar HOE. Therefore, through the simulation results, we verify the advantage of the curved structure in near-eye display system, which can be reflected by the FOV.

3.3. Experimental Results and Discussion

The steps of manufacture of HOE are introduced, in which the laser beam is divided into two paths after passing through the polarization beam splitter. Subsequently, through the collimation system, they both become parallel beams. One beam is used as a reference light to illuminate the HOE, and a spatial light modulator (SLM) is introduced into the path of the other beam to load the phase compensation factor. Then the lens is used to change the beam into a divergent spherical wave with a specific focal length as the signal light. The SLM used in the experiment has a resolution of 3840 × 2160 and a pixel spacing of 3.74 μm, respectively.

Utilizing the above optical path, two planar HOEs with different focal length are fabricated. Second, one of them is removed from the flat substrate and attached to the cylindrical glasses, as shown in Figure 5, where the red area framed on the HOE is the valid area recorded. At the same time, the focal length of the system with planar structure is 45 mm, which should be the same as the system with curved structure, and the projection width of the two should be equal in the horizontal direction too.

Next, we test the FOV of the HOE with planar substrate and the HOE with curved substrate under the above conditions. In this paper, the manufacture and testing of HOE are based on the laser with a dominant wavelength of 532 nm. The optical path for testing is shown in Figure 6. It includes a collimation system, a HOE, and a semicircular. It should be noted that the effective area of the HOE should be all covered by the parallel light passing through the collimation system. Then the diffraction of the parallel light will be concentrated and the converging angle measured by a semicircular is the FOV of the HOE. The results of the measurements are shown in

Table 1. FOV of the planar HOE and the curved HOE.

Shape of Substrate	FOV/°
Planar surface	11.15
Curved surface	11.98

, where the FOV is the average of the measured results.

Due to the size limitation of the SLM, the effective area of the HOE is finite, which leads to the tiny influence of the substrate shape on the FOV. But it can still be obtained from the analysis of the data that the mean FOV of the HOE with curved substrate is greater than the mean FOV of the HOE with planar substrate, and the difference is about 0.83°. In addition, through the previous analysis, it is not difficult to know that when the effective area of the HOE enlarges, the curved substrate will show a more significant advantage in expanding the FOV than the traditional planar substrate. Therefore, compared with the existing planar near-eye display system, the curved near-eye display system we designed has the function of expanding the FOV. And compared with the curved near-eye display system proposed by others, our work can make it easier to design the freeform phase of HOE, which can improve the imaging quality to a certain extent.

4. Conclusions

In this work, we propose a curved holographic AR near-eye display system based on freeform HOE with extended FOV. The system consists of a display source and a curved HOE. Besides, we propose a layered and weighted FOV optimization method using the curved hologram calculation and particle swarm optimization algorithm, so as to realize the optimal design of the phase of the freeform HOE. We verify that the curved structure of the system brings larger FOV than the traditional planar structure by numerical simulation. Furthermore, the FOV of the HOE with planar structure and the HOE with curved structure are measured by optical experiments, which further proves the advantage of the system in expanding the FOV.

Due to the size limitation of the SLM, the size of the HOE made in this work is limited. Although the curved near-eye display system has the ability to expand the FOV of the system compared with the planar near-eye display system, the difference between the FOV of the two is still small, and a larger HOE in size can be made by using stitching method in fabrication in the future. In addition, the way of record and reconstruction in our work introduces a certain number of aberrations [28]. In future research, we will adopt some approaches to representing the aberrations of the system and assigning certain weights to optimize and correct them. The research which mainly revolves around image quality will be conducted furtherly as well. In summary, the proposed curved holographic AR near-eye display system can expand the FOV, perform modulation of curved phase, and has a simple structure.