Perceived Brightness and Resolution of Holographic Augmented Reality Retinal Scan Glasses

Featured Application

This work can be applied to the display calibration and characterization of augmented reality glasses with a retinal scan architecture.

Abstract

Augmented reality display performance depends strongly on features of the human visual system. This is especially true for retinal scan glasses, which use laser beam scanning and transparent holographic optical combiners. Human-centered approaches allow us to go beyond conventional optical metrology and evaluate display performance as it is perceived in actual augmented reality use cases. Here, we first present a theoretical formula for the retinal scan luminance and ambient contrast ratio calculated from optical powers, wavelengths, field of view, and human pupil diameter. As a promising insight, we found that the pupil diameter dependence is beneficial in assimilating the virtual image luminance to the ambient luminance. Second, we designed and performed a psychophysical experiment to assess perceived resolution in augmented reality settings using a fully functional retinal scan glasses prototype. We present the results of the trials and illustrate how this approach can be used in the further development of augmented reality smart glasses.

1. Introduction

Retinal scan (“RS”) glasses are near-eye displays that use laser beam scanning for retinal image projection. A collimated light beam is projected via an optical combiner and through the human pupil onto the retina [1]. Recent advances in holographic manufacturing [2] have enabled the use of holographic optical elements (“HOEs”) as transparent, diffractive optical combiners [3]. This allows for lightweight augmented reality displays to be integrated into all-day wearable smart glasses with a small formfactor.

Research and development in the field is progressing quickly [4]. First efforts have been made to define new parameters like the ambient contrast ratio (“ACR”) [5] that are necessary to characterize the performance of holographic augmented reality displays [6] in real visual environments. We elaborate on these efforts and show that for the special case of RS displays, interactions with the human eye must be considered.

Most conventional display technologies such as LCD, LED, or waveguides emit light from each pixel into a large solid angle. Their emission profiles roughly obey Lambert’s cosine law to keep luminance constant across different viewing angles. We use the term “Lambertian” to describe such displays. From Lambertian displays, only a small portion of the emitted light enters the human eye at any given time. In contrast, all light emitted by RS displays is confined to a narrow scanning beam which passes through the pupil. This makes RS displays not only energy efficient but also leads to important consequences for fundamental display parameters like the brightness and resolution.

One challenge is that some conventional measurement devices such as luminance cameras cannot be used directly with RS glasses due to the interaction of the projection system with the human pupil. Instead, the luminance of an RS display must be computed in a certain way to achieve accurate results. We introduce a theoretically derived formula to calculate the retinal scan luminance and ambient contrast ratio. This formula incorporates various parameters including optical powers, wavelengths, field of view, and human pupil diameter. Building on this formula, we designed and conducted a psychophysical study to measure the perceived resolution of RS glasses in augmented reality settings.

While contrast and resolution perception have been investigated in many previous studies as fundamental parameters of human visual perception [7], augmented reality glasses provide new opportunities to take the measurements of these parameters outside of controlled laboratory environments. This work presents novel theory and methods for such out-of-laboratory measurements.

2. Materials and Methods

2.1. Retinal Scan Glasses

The BML500P RS glasses prototype available for this study (Figure 1) was developed by Bosch Sensortec and presented at the Consumer Electronics Show 2020 [8].

A light engine in the right temple of the glasses projects images via a transparent holographic optical element through the pupil onto the retina. Two oscillating micromirrors steer light from three low-power RGB laser diodes in a raster scanning pattern to form an image at a refresh rate of 60 Hz.

2.1.1. Brightness in Retinal Raster Scanning

The human eye detects retinal illuminance as brightness. Retinal illuminance ${\mathrm{E}}_{\mathrm{R}}$ is defined as luminous flux ${\mathsf{\Phi }}_{\mathrm{V}}$ per retinal area ${\mathrm{A}}_{\mathrm{R}}$.

$${\mathrm{E}}_{\mathrm{R}}={\displaystyle \frac{{\mathsf{\Phi }}_{\mathrm{V}}}{{\mathrm{A}}_{\mathrm{R}}}}$$

(1)

Display brightness is usually denoted in terms of luminance L, which is defined as the luminous flux ${\mathsf{\Phi }}_{\mathrm{V}}$ emitted into a solid angle $\mathsf{\Omega }$ by a light source with area ${\mathrm{A}}_{\mathrm{S}}$.

$$\mathrm{L}={\displaystyle \frac{{\mathsf{\Phi }}_{\mathrm{V}}}{{\mathsf{\Omega } \mathrm{A}}_{\mathrm{S}}}}$$

(2)

Luminance is a measure for the light source, while the perceived brightness depends on the illuminance on the retina. When comparing Lambertian displays with each other, luminance can be used as a proxy for perceived brightness because luminance changes lead to proportional retinal illuminance changes.

This is, however, not valid for a comparison between Lambertian and RS displays. The size of the human pupil influences the luminous flux ${\mathsf{\Phi }}_{\mathrm{V}}$ from a Lambertian display but not from an RS display, as shown in Figure 2.

For this reason, measuring the luminance of an RS image with a luminance camera leads to inconsistent results: the larger the entrance pupil of the luminance camera (to which it is calibrated using a Lambertian light source), the smaller the measured RS luminance.

In the first part of the Results Section, we show that it is nevertheless possible to calculate an equivalent luminance for RS displays. This luminance depends on the RS luminous flux, RS field of view, and human pupil size and is directly comparable with the luminance of Lambertian displays.

2.1.2. Focus and Resolution in Retinal Raster Scanning

Figure 3 shows the impact that ocular imperfections have on Lambertian and RS images. The circle of confusion for individual pixels is smaller for RS images due to the strong collimation and long depth of field of the scanning light beam.

While this effect can be illustrated clearly, it is difficult to assess the differential impact on display performance with methods from conventional display metrology since the human eye needs to be brought “into the loop”.

2.2. Perceived Resolution Study Design

To measure the perceived resolution of the BML500P RS glasses prototype, we conducted a psychophysical discrimination test with 20 participants (17 males, 3 females; aged 18–60 years, mean ± sd: 36.0 ± 7.7 years). We implemented a prescreening routine for visual acuity and contrast sensitivity with Landolt rings and a test for color vision with color plates to ensure a minimum standard level of vision ability among all participants. Participants who could not pass these screening tests were not included in the study. Inclusion criteria were right-eye dominancy, a right monocular visual acuity of at least 0.8 decimal (screening results: logMAR mean ± sd: −0.073 ± 0.060), a right monocular contrast vision threshold below 15% Weber contrast (mean ± sd: 4.2 ± 0.8%), no color vision deficiency, and no known eye diseases. The experiment was approved by the ethics committee of the Medical Faculty of the University of Tübingen (Code 410/2022B02) and conducted according to the provisions of the Declaration of Helsinki. Informed consent was obtained from all participants.

The study setup is shown in Figure 4. Participants were equipped with the RS glasses on a chinrest 2.3 m in front of a 60-inch LED monitor (PN-R603; Sharp Corporation; Sakai, Osaka, Japan). The light-isolated examination room was illuminated at an average of 2500 lux, equal to outdoor illumination on a cloudy day.

Participants were shown a right-eye monocular stimulus in the glasses with a size of 11.15 ± 0.22 deg × 14.99 ± 0.17 deg [mean ± sd from 10 measurements]. The transparent RS glasses superimposed this virtual image onto the real image from the LED monitor. Psychopy [9] was used to control the monitor and RS glasses stimuli simultaneously. To minimize distractions, participants wore a black eye patch over their left eye.

The glasses stimulus was picked randomly from a selection of monochromatic line patterns with different spatial frequencies from 1.8 cyc/deg to 16.0 cyc/deg and a full monochromatic rectangle. The line pattern consisted of lines with equal digital pixel width for both projection and non-illuminated dark background lines. The spatial frequency range of the lines was determined by the 480 × 320 pixels that could be set within the software configuration of the prototype. The pattern with the highest spatial frequency had a width of one digital pixel.

For high digital spatial frequencies, the resulting optical lines started to blur into a full rectangle. The task of the participants was to report whether they saw lines or a rectangle using a wireless keyboard. The spatial frequency of the lines was adjusted based on the participants’ responses using a 3-down 1-up staircase algorithm to determine the highest resolvable spatial frequency. The algorithm terminated after three reversals. There was no time limit for responses. To ensure that a staircase level could not be passed by only identifying rectangles, a line pattern was always presented after two correct responses. A 0.5 s pause between stimuli was added to avoid afterimages and to reduce cues from stimulus transitions.

We performed 15 trials each for horizontal and vertical lines for different stimulus colors (laser wavelengths red [640 nm], green [521 nm], blue [452 nm]), stimulus brightness (maximum radiant flux red [272 nW]), green [736 nW], blue [1263 nW], equiluminant radiant flux green [66 nW], blue [1145 nW]), and LED monitor backgrounds [black, red, green, blue, orange] to measure the influence of different stimulus and background conditions on the perceived resolution.

The LED monitor was set to a luminance of 27 cd/m² for each color and 1 cd/m² for the black background. The right spectacle lens with the holographic optical combiner had an optical transmittance of ${\mathsf{\tau }}_{\mathrm{C}}$ = 0.90.

The optical power of the retinal scan glasses was measured by setting an optical power meter (PM160 power meter; Thorlabs, Inc.; Newton, NJ, USA) in the beam path such that all light contributing to the image was captured by the optical sensor. The wavelengths of the retinal scan glasses as well as the luminance of the LED monitor were measured with a spectroradiometer (JETI spectraval 1511; JETI Technische Instrumente GmbH; Jena, Germany). The wavelength measurement only depends on the relative magnitudes of optical power. Therefore, measuring the wavelengths of retinal scan glasses with a spectroradiometer will provide accurate results, while measuring the luminance directly will not.

Generally, the perceived resolution threshold not only depends on the spatial frequency but also on the contrast [10]. If the contrast between stimuli and background is too small, the perceived resolution threshold decreases across spatial frequencies [11]. Previous research showed that this influence is persistent up to contrast levels of about 0.5 [12]. Since we wanted to avoid the influence of contrast on the perceived resolution in this study, we needed to ensure that the contrast between the retinal scan stimuli and the Lambertian background stayed above 0.5. As shown in Section 2.1.1, however, this contrast can only be calculated correctly if the brightness for both display types is denoted in the same physical unit. We therefore first derive a formula for this calculation in the first part of the Results Section before presenting the results of the study in the second part of the section.

3. Results

3.1. Retinal Scan Luminance and Ambient Contrast Ratio

To express the brightness of an RS display in terms of luminance, we determine the luminance of a Lambertian display that would cause the same retinal illuminance ${\mathrm{E}}_{\mathrm{R}}$ as the RS glasses (see

Table A1. Mathematical symbols and their meaning for the derivation of the retinal scan luminance and ambient contrast ratio.

Symbol	Meaning
${\mathrm{A}}_{\mathrm{P}}$	Area of human pupil
${\mathrm{A}}_{\mathrm{R}}$	Area of image on retina
${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Lam}}}$	Area of image on retina for a Lambertian system
${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Scan}}}$	Area of image on retina for a retinal scan system
${\mathrm{A}}_{\mathrm{S}}$	Area of a light source (e.g., area of a display panel)
${\mathrm{ACR}}_{\mathrm{RS}}$	Ambient contrast ratio of a retinal scan system
${\mathrm{d}}_{{\mathrm{P}}_{\mathrm{mm}}}$	Diameter of human pupil in millimeters
${\mathrm{E}}_{\mathrm{R}}$	Retinal illuminance
${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}$	Retinal illuminance from a Lambertian system
${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}$	Retinal illuminance from a retinal scan system
${\mathrm{f}}_{\mathrm{e}}$	Focal distance of the human eye
${\mathrm{K}}_{\mathrm{m}}$	Luminous efficacy constant
$\mathrm{L}$	Luminance
${\mathrm{L}}_{\mathrm{A}}$	Ambient luminance
${\mathrm{L}}_{\mathrm{Lam}}$	Luminance of a Lambertian system
${\mathrm{L}}_{\mathrm{RS}}$	Luminance of a retinal scan system
$\mathrm{R}$	Distance between human pupil and a Lambertian display
$\mathrm{V}\left(\mathsf{\lambda }\right)$	Luminous efficiency function
$\mathsf{\alpha }$	First projection angle of a retinal scan system
$\mathsf{\beta }$	Second projection angle of a retinal scan system
$\mathsf{\lambda }$	Wavelength
${\overline{\mathsf{\lambda }}}_{\mathrm{blue}}$	Average wavelength of blue retinal scan system primary
${\overline{\mathsf{\lambda }}}_{\mathrm{green}}$	Average wavelength of green retinal scan system primary
${\overline{\mathsf{\lambda }}}_{\mathrm{red}}$	Average wavelength of red retinal scan system primary
${\mathsf{\tau }}_{\mathrm{C}}$	Transparency of retinal scan system optical combiner
${\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}$	Radiant flux from retinal scan system
${\mathsf{\Phi }}_{\mathrm{blue}}$	Radiant flux from blue retinal scan system primary
${\mathsf{\Phi }}_{\mathrm{green}}$	Radiant flux from green retinal scan system primary
${\mathsf{\Phi }}_{\mathrm{red}}$	Radiant flux from red retinal scan system primary
${\mathsf{\Phi }}_{\mathrm{V}}$	Luminous flux
${\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Lam}}}$	Luminous flux from a Lambertian system
${\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Scan}}}$	Luminous flux from a retinal scan system
$\mathsf{\Omega }$	Solid angle

in Appendix A for an overview of mathematical symbols).

$${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}={\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}$$

(3)

To find the RS retinal illuminance ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}$, we first calculate the luminous flux ${\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Scan}}}$ from the radiant flux ${\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}$, the luminous efficacy constant ${\mathrm{K}}_{\mathrm{m}}=683 {\displaystyle \frac{lm}{W}}$, and the luminous efficiency function $\mathrm{V}\left(\mathsf{\lambda }\right)$. The luminous efficacy constant ${\mathrm{K}}_{\mathrm{m}}$ tells us how many lumens result from 1 watt of optical power. It acts as the conversion factor from radiometric to photometric quantities. The luminous efficiency function $\mathrm{V}\left(\mathsf{\lambda }\right)$ denotes how bright light with different wavelengths appears to the human eye. In practice, the resulting integral is approximated by a finite sum of measurements.

$${\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Scan}}}={\mathrm{K}}_{\mathrm{m}}\int \mathrm{V}\left(\mathsf{\lambda }\right) {\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right) \mathrm{d}\mathsf{\lambda }\approx {\mathrm{K}}_{\mathrm{m}}{\displaystyle \sum }_{\mathsf{\lambda }}\mathrm{V}\left(\mathsf{\lambda }\right) {\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right)$$

(4)

Next, we calculate the illuminated retinal area ${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Scan}}}$ geometrically from the RS field of view angles $\mathsf{\alpha }$ and $\mathsf{\beta }$ and the eye’s focal distance ${\mathrm{f}}_{\mathrm{e}}$.

$${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Scan}}}={\mathrm{f}}_{\mathrm{e}} 2\tan \left({\displaystyle \frac{\mathsf{\alpha }}{2}}\right) {\mathrm{f}}_{\mathrm{e}} 2 \tan \left({\displaystyle \frac{\mathsf{\beta }}{2}}\right)=4{ \mathrm{f}}_{\mathrm{e}}{⠀}^2\tan \left({\displaystyle \frac{\mathsf{\alpha }}{2}}\right) \tan \left({\displaystyle \frac{\mathsf{\beta }}{2}}\right)$$

(5)

Plugging Equations (4) and (5) into Equation (1) gives us the retinal illuminance for an RS display.

$${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}={\displaystyle \frac{{\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Scan}}}}{{\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Scan}}}}}={\displaystyle \frac{{\mathrm{K}}_{\mathrm{m}}{{\displaystyle \sum }}_{\mathsf{\lambda }}\mathrm{V}\left(\mathsf{\lambda }\right) {\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right)}{4{ \mathrm{f}}_{\mathrm{e}}{⠀}^2\tan \left(\frac{\mathsf{\alpha }}{2}\right) \tan \left(\frac{\mathsf{\beta }}{2}\right)}}$$

(6)

Next, we express the Lambertian display retinal illuminance ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}$ in terms of the Lambertian display luminance $\mathrm{L}$ [13]. For an observer with pupil area ${\mathrm{A}}_{\mathrm{P}}$ at distance $\mathrm{R}$ to the display, the solid angle $\mathsf{\Omega }$ will be given by the ratio of ${\mathrm{A}}_{\mathrm{P}}$ and ${\mathrm{R}}^2$.

$$\mathsf{\Omega }={\displaystyle \frac{{\mathrm{A}}_{\mathrm{P}}}{{\mathrm{R}}^2}}$$

(7)

Plugging Equation (7) into Equation (2) and rearranging terms yields the luminous flux ${\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Lam}}}$ entering the pupil as a function of the Lambertian display luminance ${\mathrm{L}}_{\mathrm{Lam}}$, pupil area ${\mathrm{A}}_{\mathrm{P}}$, display area ${\mathrm{A}}_{\mathrm{S}}$, and display distance $\mathrm{R}$.

$${\mathrm{L}}_{\mathrm{Lam}}={\displaystyle \frac{{\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Lam}}}}{\frac{{\mathrm{A}}_{\mathrm{P}}}{{\mathrm{R}}^2} {\mathrm{A}}_{\mathrm{S}}}} \iff { \mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Lam}}}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{Lam}} {\mathrm{A}}_{\mathrm{P}} {\mathrm{A}}_{\mathrm{S}}}{{\mathrm{R}}^2}}$$

(8)

To find the retinal image area ${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Lam}}}$, we use the lens formula from geometrical optics. A display with area A_S at a distance R to the eye creates an image with area A_R at focal distance ${\mathrm{f}}_{\mathrm{e}}$ on the retina. The magnification between ${\mathrm{A}}_{\mathrm{S}}$ and ${\mathrm{A}}_{\mathrm{R}}$ is given by the square of the ratio between image distance ${\mathrm{f}}_{\mathrm{e}}$ and object distance $\mathrm{R}$.

$${\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Lam}}}={\mathrm{A}}_{\mathrm{S}}{\left({\displaystyle \frac{{\mathrm{f}}_{\mathrm{e}}}{\mathrm{R}}}\right)}^2$$

(9)

Plugging Equations (8) and (9) into Equation (1) gives us the retinal illuminance for a Lambertian display:

$${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}={\displaystyle \frac{{\mathsf{\Phi }}_{{\mathrm{V}}_{\mathrm{Lam}}}}{{\mathrm{A}}_{{\mathrm{R}}_{\mathrm{Lam}}}}}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{Lam}} {\mathrm{A}}_{\mathrm{P}} {\mathrm{A}}_{\mathrm{S}}}{{\mathrm{R}}^2 {\mathrm{A}}_{\mathrm{S}}{\left(\frac{{\mathrm{f}}_{\mathrm{e}}}{\mathrm{R}}\right)}^2}}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{Lam}} {\mathrm{A}}_{\mathrm{P}}}{{\mathrm{f}}_{\mathrm{e}}{⠀}^2}}$$

(10)

Finally, we can set both retinal illuminances ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}$ and ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}$ as equal.

$${\displaystyle \frac{{\mathrm{L}}_{\mathrm{Lam}} {\mathrm{A}}_{\mathrm{P}}}{{\mathrm{f}}_{\mathrm{e}}{⠀}^2}}={\displaystyle \frac{{\mathrm{K}}_{\mathrm{m}}{{\displaystyle \sum }}_{\mathsf{\lambda }}\mathrm{V}\left(\mathsf{\lambda }\right) {\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right)}{4{ \mathrm{f}}_{\mathrm{e}}{⠀}^2\tan \left(\frac{\mathsf{\alpha }}{2}\right)\tan \left(\frac{\mathsf{\beta }}{2}\right)}}$$

(11)

After rearranging terms and renaming ${\mathrm{L}}_{\mathrm{Lam}}$ to ${\mathrm{L}}_{\mathrm{RS}}$, we arrive at Equation (12) for the luminance ${\mathrm{L}}_{\mathrm{RS}}$ of an RS display.

$${\mathrm{L}}_{\mathrm{RS}}={\displaystyle \frac{{\mathrm{K}}_{\mathrm{m}}{{\displaystyle \sum }}_{\mathsf{\lambda }}\mathrm{V}\left(\mathsf{\lambda }\right) {\mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right)}{4{ \mathrm{A}}_{\mathrm{P}} \tan \left(\frac{\mathsf{\alpha }}{2}\right)\tan \left(\frac{\mathsf{\beta }}{2}\right)}}$$

(12)

To calculate the pupil area ${\mathrm{A}}_{\mathrm{P}}$, we measure the pupil diameter ${\mathrm{d}}_{\mathrm{p}}$ in mm.

$${\mathrm{A}}_{\mathrm{P}}={\displaystyle \frac{1}{4}}\mathsf{\pi } {\left(0.001{ \mathrm{d}}_{{\mathrm{P}}_{\mathrm{mm}}}\right)}^2$$

(13)

For an RS display with narrow RGB laser spectral linewidths, we approximate ${\displaystyle \sum }_{\mathsf{\lambda }}\mathrm{V}\left(\mathsf{\lambda }\right){ \mathsf{\Phi }}_{{\mathrm{E}}_{\mathrm{Scan}}}\left(\mathsf{\lambda }\right)$ in Equation (12) with radiant powers ${\mathsf{\Phi }}_{\mathrm{i}}$ and average wavelengths ${\overline{\mathsf{\lambda }}}_{\mathrm{i}}$.

$${\mathrm{L}}_{\mathrm{RS}}={\displaystyle \frac{{\mathrm{K}}_{\mathrm{m}}\left(\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{red}}\right) {\mathsf{\Phi }}_{\mathrm{red}}+\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{green}}\right) {\mathsf{\Phi }}_{\mathrm{green}}+\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{blue}}\right) {\mathsf{\Phi }}_{\mathrm{blue}}\right)}{\mathsf{\pi } {\left(0.001{ \mathrm{d}}_{{\mathrm{P}}_{\mathrm{mm}}}\right)}^2\tan \left(\frac{\mathsf{\alpha }}{2}\right)\tan \left(\frac{\mathsf{\beta }}{2}\right)}}$$

(14)

${\mathrm{K}}_{\mathrm{m}}$ and $\mathrm{V}\left(\mathsf{\lambda }\right)$ are constant and tabularized [14]. ${\mathsf{\Phi }}_{\mathrm{i}}$, $\mathsf{\alpha }$, and $\mathsf{\beta }$ must be measured for the RS display. The only remaining variable is the pupil diameter ${\mathrm{d}}_{{\mathrm{P}}_{\mathrm{mm}}}.$ Changes in pupil diameter can be caused by different factors [15]. In most circumstances, however, the main contributor is the ambient luminance ${\mathrm{L}}_{\mathrm{A}}$. We use this fact to express the RS luminance as a function of ambient luminance instead of pupil size.

Several mathematical models have been developed for this relationship [16]. We picked the model from Stanley and Davies [17] for binocular background adaptation with an adaptive field size a of (60 deg)². The model allows us to express pupil diameter ${\mathrm{d}}_{\mathrm{P}}$ in mm as a function of ambient luminance ${\mathrm{L}}_{\mathrm{A}}$ in cd/m² for an average observer.

$${\mathrm{d}}_{{\mathrm{P}}_{\mathrm{mm}}}=7.75-5.75\left({\displaystyle \frac{{\left(\frac{{\mathrm{L}}_{\mathrm{A}} 3600}{846}\right)}^{0.41}}{{\left(\frac{{\mathrm{L}}_{\mathrm{A}} 3600}{846}\right)}^{0.41}+2}}\right)$$

(15)

Plugging Equation (15) into Equation (14) allows us to calculate the RS luminance ${\mathrm{L}}_{\mathrm{RS}}$ as a function of the ambient luminance ${\mathrm{L}}_{\mathrm{A}}$.

$${\mathrm{L}}_{\mathrm{RS}}={\displaystyle \frac{{\mathrm{K}}_{\mathrm{m}}\left(\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{red}}\right) {\mathsf{\Phi }}_{\mathrm{red}}+\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{green}}\right) {\mathsf{\Phi }}_{\mathrm{green}}+\mathrm{V}\left({\overline{\mathsf{\lambda }}}_{\mathrm{blue}}\right) {\mathsf{\Phi }}_{\mathrm{blue}}\right)}{\mathsf{\pi }{\left(0.001\left(7.75-5.75\left(\frac{{\left(\frac{{\mathrm{L}}_{\mathrm{A}} 3600}{846}\right)}^{0.41}}{{\left(\frac{{\mathrm{L}}_{\mathrm{A}} 3600}{846}\right)}^{0.41}+2}\right)\right)\right)}^2\tan \left(\frac{\mathsf{\alpha }}{2}\right)\tan \left(\frac{\mathsf{\beta }}{2}\right)}}$$

(16)

Expressing the RS luminance in terms of the ambient luminance not only eliminates the need for direct pupil size measurements but also simplifies another useful quantity, the “retinal scan ambient contrast ratio” ${\mathrm{ACR}}_{\mathrm{RS}}$, which is defined by the RS luminance ${\mathrm{L}}_{\mathrm{RS}}$, the combiner transparency ${\mathsf{\tau }}_{\mathrm{C}}$, and the ambient luminance ${\mathrm{L}}_{\mathrm{A}}$.

$${\mathrm{ACR}}_{\mathrm{RS}}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{RS}}}{{\mathsf{\tau }}_{\mathrm{C}} {\mathrm{L}}_{\mathrm{A}}}}$$

(17)

In non-transparent displays, the contrast ratio is determined from the brightest and darkest pixel’s luminance. RS displays do not project any light into fully dark pixels. The luminance of the darkest RS display pixel is therefore given by the luminance of the darkest background element within the field of view. This could be measured for a fixed head orientation and homogeneous background in a controlled laboratory setting. For most natural visual scenes, however, such a pixel-precise background luminance would be difficult to determine. For those cases, we use the average ambient luminance ${\mathrm{L}}_{\mathrm{A}}$ as an estimate.

Since the ambient light passes through the spectacle lens, its luminance is reduced by the transmittance ${\mathsf{\tau }}_{\mathrm{C}}$ of the HOE combiner. Optical power measurements of the RS display are taken at the eye position, so ${\mathsf{\tau }}_{\mathrm{C}}$ does not influence the numerator in Equation (17). The transmittance could however influence the pupil diameter in Equation (15). This will depend on the size of the combiner in comparison to the adaptive field size. Since the HOE region in our glasses prototype was small, and, more importantly, monocular, we neglected this influence. If this does not hold true for other prototypes, ${\mathsf{\tau }}_{\mathrm{C}}$ can be included as an additional coefficient to ${\mathrm{L}}_{\mathrm{A}}$ in Equation (15).

Figure 5 shows the RS luminance ${\mathrm{L}}_{\mathrm{RS}}$ and the RS ambient contrast ratio ${\mathrm{ACR}}_{\mathrm{RS}}$ as a function of the ambient luminance ${\mathrm{L}}_{\mathrm{A}}$ for a fixed set of RS glasses parameters.

3.2. Perceived Resolution Study Results

This study was conducted in a controlled environment which allowed us to determine the pupil size directly. We measured the pupil diameter of all participants right before the experiment while they faced the monitor on the chinrest and found a mean ± sd pupil diameter of 3.3 ± 0.4 mm within a range from 2.5 mm to 4.0 mm. With the optical measurement values from the glasses, we determined ${\mathrm{L}}_{\mathrm{RS}}$ according to Equation (14). We then used the luminance of the LED monitor backgrounds for ${\mathrm{L}}_{\mathrm{A}}$ and the transmittance of the combiner ${\mathsf{\tau }}_{\mathrm{C}}$ to calculate ${\mathrm{ACR}}_{\mathrm{RS}}$ according to Equation (17).

Table 1. Trial parameters and results from staircase algorithm in perceived resolution study.

Trial	Trial Parameters				Number of Participants That Passed Staircase Resolution Level Horizontal [H] and Vertical [V]
	Glasses	Monitor	LRS [cd/m2]	ACRRS	4H 3.59 c/d	3H 4.78 c/d	2H 7.17 c/d	1H 14.35 c/d	4V 4.00 c/d	3V 5.34 c/d	2V 8.01 c/d	1V 16.01 c/d
1	red	black	83	92	20	17	0	0	20	17	0	0
2	red	green	83	3	20	19	0	0	20	18	0	0
3	red	red	83	3	20	19	0	0	19	19	0	0
4	green	black	929	1032	20	20	4	0	20	20	0	0
5	green	black	83	92	20	20	5	0	20	20	0	0
6	green	red	929	38	20	20	5	0	20	20	0	0
7	green	red	83	3	20	20	6	0	20	20	0	0
8	green	green	929	38	20	20	6	0	20	20	0	0
9	green	green	83	3	20	20	5	0	20	20	0	0
10	blue	black	91	101	20	20	4	0	20	20	0	0
11	blue	black	83	92	20	20	5	0	20	20	2	0
12	blue	orange	91	4	20	20	3	0	20	20	1	0
13	blue	orange	83	3	20	20	4	0	20	20	1	0
14	blue	blue	91	4	20	20	5	0	20	20	1	0
15	blue	blue	83	3	20	20	5	0	20	20	1	0

provides the trial parameters and results from the perceived resolution study. The left side of the table shows the trial parameters including the ambient contrast ratio ACR_RS for the retinal scan stimuli and the LED monitor background. Since ACR_RS stayed well above 0.5 for all trials, we can now assume that the perceived resolution threshold is only a function of the spatial frequency and not the contrast. The right side of the table shows the number of participants that successfully passed each staircase level. “Passing” a staircase level means that the participant provided three correct answers in a row at that level during the staircase algorithm. The level number corresponds to the digital pixel width of the line patterns. All 20 participants were able to complete horizontal level 4H (3.59 cyc/deg) and vertical level 5V (3.20 cyc/deg) in all trials. No participant was able to complete horizontal level 1H (14.35 cyc/deg) or vertical level 1V (16.01 cyc/deg) in any trial.

The endpoints of the staircase algorithm provide an overview over the results but do not represent all individual datapoints. We therefore additionally fitted sigmoidal psychometric functions to the trial data. We determined the perceived resolution threshold from the inflection point of the functions and calculated the 95% confidence intervals. Figure 6 shows two separate fits for horizontal and vertical lines. This gives us a perceived resolution of 7.09 cyc/deg (95% CI: 6.91 to 7.27) for horizontal and 7.43 cyc/deg (95% CI: 7.02 to 7.89) for vertical lines.

We fitted separate psychometric functions for trials with different stimulus and background conditions but found similar thresholds for all of them. The biggest differences can be seen for trials with different stimulus colors. Participants reached a threshold of 6.68 cyc/deg (95% CI: 6.09 to 7.28) in trials with red stimuli, a threshold of 7.39 cyc/deg (95% CI: 7.29 to 7.48) in trials with green stimuli and a threshold of 7.28 cyc/deg (95% CI: 7.10 to 7.47) in trials with blue stimuli (Figure 7). These results agree with observations from Table 1, since participants performed slightly worse in the staircase algorithm for red stimuli as well.

Participants achieved a threshold of 7.11 cyc/deg (95% CI: 6.89 to 7.33) in trials with maximum stimulus luminance and a threshold of 7.39 cyc/deg (95% CI: 7.27 to 7.50) in trials with equal stimulus luminance (Figure 8).

They reached a threshold of 7.25 cyc/deg (95% CI: 7.01 to 7.49) in trials with a black background, a threshold of 7.20 cyc/deg (95% CI: 7.02 to 7.38) in trials with complementary color backgrounds (red/green, green/red, blue/orange) and a threshold of 7.36 cyc/deg (95% CI: 7.18 to 7.53) in trials with the same color backgrounds (Figure 9).

4. Discussion

RS glasses use an optical architecture that is different from conventional display technologies. We explained how the human pupil influences the perceived brightness of an RS display and presented a physical derivation for the RS luminance ${\mathrm{L}}_{\mathrm{RS}}$. Since the pupil adapts to ambient luminance, RS glasses appear darker in dark environments and brighter in bright environments. For an observer with standard pupil dynamics, the RS display luminance multiplies fivefold from mesopic to bright photopic environments.

For conventional displays, Troland is commonly used to describe retinal illuminance as a function of pupil size [18]. Although Troland accounts for the pupil area ${\mathrm{A}}_{\mathrm{P}}$ in Equation (10), one still needs to make an assumption about the eye’s focal distance ${\mathrm{f}}_{\mathrm{e}}$ (i.e., use a value for a standard eye) to calculate an actual Troland value [13]. In our derivation, however, ${\mathrm{f}}_{\mathrm{e}}$ cancels out in Equation (11) after setting ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Lam}}}$ equal to ${\mathrm{E}}_{{\mathrm{R}}_{\mathrm{Scan}}}$. This means that ${\mathrm{L}}_{\mathrm{RS}}$ is independent from any assumptions about the eye apart from the pupil size.

We picked the model from Stanley and Davies [17] to express pupil size as a function of ambient luminance. Using another model with additional parameters such as age [16] could further increase the accuracy but would also increase the complexity. The fundamental relationship between pupil size and RS luminance holds independently from this model choice.

Using the ambient luminance ${\mathrm{L}}_{\mathrm{A}}$ and combiner transmittance ${\mathsf{\tau }}_{\mathrm{C}}$, we defined the ambient contrast ratio for RS glasses as ${\mathrm{ACR}}_{\mathrm{RS}}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{RS}}}{{\mathsf{\tau }}_{\mathrm{C}} {\mathrm{L}}_{\mathrm{A}}}}$. In some previous research [4,5], the ambient contrast ratio has been defined as $\mathrm{ACR}={\displaystyle \frac{{\mathrm{L}}_{\mathrm{on}}+{\mathsf{\tau } \mathrm{L}}_{\mathrm{A}} }{{\mathrm{L}}_{\mathrm{off}}+\mathsf{\tau } {\mathrm{L}}_{\mathrm{A}}}}$, where ${\mathrm{L}}_{\mathrm{on}}$ and ${\mathrm{L}}_{\mathrm{off}}$ are the luminance of on and off pixels, $\mathsf{\tau }$ is the see-through transmittance, and ${\mathrm{L}}_{\mathrm{A}}$ is the ambient luminance. RS displays do not project light into dark pixel areas. ${\mathrm{L}}_{\mathrm{off}}$ is always zero, so the denominator of the two formulas does not differ. More thought must be put into the ambient light term ${\mathsf{\tau } \mathrm{L}}_{\mathrm{A}}$ in the numerator. If the term is included, the ambient contrast ratio would equal 1 for an off display. This seems in conflict with the conventional notion of a contrast ratio. We therefore propose to make a distinction between a standard “ambient contrast ratio” that does not include the background term ${\mathsf{\tau } \mathrm{L}}_{\mathrm{A}}$ in the numerator, and a second quantity, potentially called the “additive ambient contrast ratio”, that does include it. In this paper, we use the standard ambient contrast ratio.

With the derivation for the brightness and contrast of visual stimuli, we were finally able to design and evaluate our psychophysical study for the perceived resolution. Overall, we found consistent results across 15 trials with different stimulus parameters. This study had a gamified design, but we provided no reward for correct answers. Instead, we instructed the participants to give accurate feedback according to their perception. Several participants reported after the study that they did not find any visual cues to manipulate the results even if they had wanted to, hinting toward a robust experimental design.

From the field-of-view size and the number of digital pixels, we can infer that the pixels were not perfectly quadratic. This led to different spatial frequencies for horizontal and vertical staircase levels (Table 1). The fact that the correct response ratios in Figure 6 still follow the trend of a common overall sigmoidal function further speaks to the validity of our measurement approach.

The 0.8 decimal (0.097 logMAR) minimum visual acuity that participants needed to achieve to be included in the study corresponds to a visual resolution of 24.0 cyc/deg [19]. Since we found an average perceived resolution threshold of 7.2 cyc/deg across all trials, we conclude that the perceived resolution was limited by the optical resolution of the RS glasses and not the visual perception of any of the participants.

Our results indicate that the background conditions did not have a notable influence on the perceived resolution within the range of conditions we tested. This could, however, still be the case for other background parameter ranges. As expected, the RS luminance and the ambient contrast ratio, which was above 0.5 in all trials, did not influence the perceived resolution thresholds, despite varying over several orders of magnitude. These findings agree with the previous literature in which it is argued that the neural response in the visual pathway saturates, and further contrast increases do not lead to improved perceived resolution thresholds [7]. We see our study as a first step toward more human-centered characterizations of augmented reality glasses and recognize the potential for follow-up studies in additional parameter ranges. In particular, the RS luminance and ambient contrast ratio could be decreased in a future study to values around and below 0.5 to also investigate the effect of contrast levels on the perceived resolution. Since calculating and setting the ambient contrast ratio of retinal scan glasses stimuli are only possible with the correct equation for the RS luminance, a follow-up experiment that varies stimulus contrast in addition to stimulus resolution would further illustrate the benefits of our theoretical derivation.

We neither measured nor controlled for the polarization of light from the retinal scan glasses. The influence of the linear polarization from the laser source on the visual perception of retinal scan displays could be an interesting aspect for further investigation, especially when compared with conventional display architectures.

We also measured the RS glasses’ optical display resolution according to the IEC 61947-2 standard [20] and found a maximum value of 3.4 cyc/deg for monochromatic green images. In our psychophysical study, we found a perceived resolution threshold of 7.4 cyc/deg for monochromatic green images.

The perceived resolution threshold adds additional interpretability to the IEC 61947-2 optical resolution threshold, which only uses a modulation depth of 30% to determine the spatial frequency cutoff, independent of visual perceptibility. Knowing the number of pixels per degree that humans can distinguish in the glasses helps in taking better-guided design decisions for future prototypes; for example, for finding the necessary field-of-view and holographic combiner area size to show image content with a certain desired size and resolution.

Furthermore, in accordance with the ideas presented in Section 2.1.2, we expect the ratio between the perceived and optical resolution to be higher for retinal scan displays than for conventional Lambertian displays. A promising next step to test this claim would be to conduct a second, comparative study with a Lambertian augmented reality display that only differs in its optical architecture and is otherwise as identical to our retinal scan glasses prototype as possible.

The BML500P prototype used in this study still had some known limitations. In Figure 4, one can see brightness inhomogeneities, particularly in the top left corner of the green and blue stimuli. Such inhomogeneities typically arise from local variations in the diffraction efficiency of the HOE. This is a manufacturing issue that will be resolved in the future.

The second limitation was the small exit pupil of the single eyebox prototype. The RS display is only visible if the narrow light beam fully passes through the pupil. If this alignment is lost, the image is not visible anymore. Therefore, we added a 5 min fitting procedure for each participant at the beginning of our study. The nose bridge of the glasses could be quickly adjusted to fit to the pupil position of each participant (see Figure 1, left). In the future, several approaches such as eyebox replication or eyebox steering can be implemented to remediate this issue.

5. Conclusions

We found a physical derivation for the perceived brightness and ambient contrast ratio of holographic augmented reality retinal scan glasses. Building on those results, we designed and conducted a psychophysical study to determine the perceived resolution of a fully functional retinal scan glasses prototype.

The perceived brightness as quantified by the retinal scan luminance ${\mathrm{L}}_{\mathrm{RS}}$ changes with pupil size: it increases for small pupils and decreases for large pupils. As a result, retinal scan displays appear brighter in bright environments and darker in dark environments in comparison to conventional displays. This is a unique and advantageous property of retinal scan glasses for augmented reality use cases.

The psychophysical study produced consistent perceived resolution thresholds of about seven cycles per degree across all trials despite varying background and stimulus conditions. This result shows that within the range of parameters we tested, the optical resolution of the retinal scan glasses and not the human visual perception was the limiting factor for the perceived resolution. Overall, this study demonstrated how both the display and the background can be included in a psychophysical experiment to better understand and optimize augmented reality display performance for realistic use cases.