Enhancing a Display’s Sunlight Readability with Tone Mapping

Abstract

The sunlight readability of display devices, such as notebook computers, transparent displays, vehicle displays, and augmented reality, is a significant technical challenge due to degraded image quality. To mitigate this problem, by fitting the human eye function, we propose a tone mapping method on a mobile phone display panel to enhance low grayscale image readability under bright ambient light. Additionally, we adapt a mini-LED backlight model to simulate real images under different ambient lighting conditions. Both experimental and simulated results indicate that high luminance displays with an optimized gamma value significantly enhance sunlight readability and image quality. Moreover, global color rendering can alleviate color shift. Such a method is also valid for the optimization of optical see-through devices under diverse environmental conditions.

1. Introduction

Sunlight readability is a longstanding problem for flat panel displays (such as mobile phones, laptops, smart watches, and automotive displays) and optical see-through displays, including transparent displays, head-up displays (HUDs), and head-mounted displays (HMDs) [1,2,3,4]. Due to the wide range of lighting conditions, from sunlight to starlight, and varying environmental temperatures, it is essential for these displays to provide a high peak brightness, acceptable ambient contrast ratio (ACR > 5:1), extreme temperature tolerance, and long lifetime [5]. Especially under strong ambient light, preventing image washout due to inadequate display brightness remains a grand challenge.

Presently, the tandem organic light-emitting diode (OLED) display has been widely used in high-end smartphones, laptops, and smartwatches due to its high peak brightness, excellent black state (at dark ambient), and saturated color performance [6,7,8,9]. However, keeping a high peak brightness would reduce the OLED’s lifetime and efficiency [10]. To overcome these issues, micro-LED (µLED) has advantages because its brightness can be easily controlled by boosting the driving current and duty ratio. However, the nonuniformity of transferred µLED chips and high fabrication cost remain the major barriers for its widespread application [11,12,13]. Commercial infotainment displays, laptops, and tablets require a demanding level of sunlight readability. In comparison with OLED and μLED displays, mini-LED backlit liquid crystal displays (mLCDs) are more favorable because of their high brightness without burn-in issues, excellent reliability, especially at elevated temperatures, and low cost [10,14]. At the 2023 SID Automotive Displays conference, BOE demonstrated an 800-nit 42.2″ curved mLCD integrated infotainment display prototype, featuring 1439 local dimming zones. In addition, innovative approaches, such as advanced backlight driving algorithms, tailored mini-LED light spread function (LSF), and liquid crystal layer optimization, are able to further increase the backlight uniformity [15,16,17], reduce the power consumption [18], and mitigate the undesirable halo effect [19]. Despite these advancements, the LCD panel transmittance is often below 10% due to significant losses from polarizers, opaque thin-film transistors, and color filters, which in turn limit the peak brightness. For optical see-through devices, the situation gets worse because the impact of ambient light is more pronounced. For HUDs such as transparent windshield displays, a ~4000-nit µLED panel is preferred [20]. For HMDs such as augmented reality (AR) eyeglasses, the low efficiency of waveguide (~1%) significantly reduces the peak luminance [21]. Moreover, simply increasing the peak luminance leads to a higher power consumption. These limitations hamper peak luminance and readability under intense ambient light conditions, especially for low grayscale images. To address these issues, Sharp has proposed an adaptive tone mapping model based on reducing the contrast response between the human vision system and original image [22], but it is challenging to apply to high resolution displays with a high frame rate. Alternatively, gamma correction methods have been proposed for image enhancement under ambient lighting conditions [23,24], image contents [25], and human eye function. In 2006, Devlin et al. [26] proposed a luminance remapping based on human eye just-noticeable difference (JND) measurements to compensate for the loss of contrast, with visual calibration performed by the viewers themselves. Kykta et al. [27] applied two gamma curves to fit the human eye function, though they did not consider ambient light. In 2014, Blankenbach et al. [28] corrected the display outputs based on the Weber fraction with color rendering. However, several problems still exist for the abovementioned methods. First, in practical applications, the environment and display luminance may change constantly, requiring a dynamic gamma correction. Second, clipping effect and color shift still remain to be overcome [29]. In addition, the image enhancement of emerging transparent displays and head-up displays via eye function is lacking. Therefore, comprehensive image analysis with tone mapping in advanced display systems is essential.

In this paper, we start with the tone mapping method by fitting the gamma curve to the human eye function under different ambient conditions, including adaptive brightness control to provide users with a better readability of low grayscale images. Next, we simulate and optimize the mini-LED backlit LCD model for analyzing the image quality in real cases. To validate our simulation results, we conduct a human visual experiment using a commercial display panel with variable peak luminance in a uniform lightbox. The measured image quality is analyzed and compared to the simulated results. Afterwards, the potential clipping effect due to limited display peak luminance is discussed. Additionally, the color shift and global color rendering are evaluated. Finally, the tone mapping for optical see-through displays with improved image quality is analyzed.

2. Tone Mapping

2.1. Tone Mapping for DICOM GSDF

After considering the ambient light reflectance, the luminance (L) of an 8-bit display panel at different grayscales can be expressed as follows:

$$L=L_r\times {\left(\frac{G_r}{255}\right)}^{\mathrm{ϓ}}+L_g\times {\left(\frac{G_g}{255}\right)}^{\mathrm{ϓ}}+L_b\times {\left(\frac{G_b}{255}\right)}^{\mathrm{ϓ}}+\frac{I_a}{\pi }\times R$$

(1)

$$L_p=L_r+L_g+L_b,$$

(2)

where L_P, Υ, I_a, and R represent the peak luminance, gamma value, ambient light illuminance, and surface reflectance of the panel; and L_r,g,b and G_r,g,b describe the peak luminance and the gray level (0–255) of red, green, and blue colors, respectively. In Equation (1), the gamma value can vary from 2.2 to 2.6, depending on the L_P and applications [30]. For a typical vehicle display, I_a can be in the range between 1 lux (starlight) and 100 k lux (direct sunlight) depending on the driving conditions [31], and the surface reflectance is assumed to be 1%.

Figure 1a shows the simulated grayscale luminance of a 1000-nit display panel with Υ = 2.2 as the ambient light illumination increases from 1 lux to 100 k lux. Under bright ambient, the low grayscale contents are washed out by the surface reflection. To provide a premium visual experience for users, we compare the results to human eye function. The National Electrical Manufacturers Association (NEMA) application to Digital Imaging and Communications in Medicine (DICOM) grayscale standard display function (GSDF) standardizes the luminance response perceived by human eyes in units of just-noticeable difference (JND) based on the human-contrast-sensitivity model [32]. There are two major problems for existing displays that use a constant gamma value. The first problem is that, as depicted in Figure 1b, the luminance of the display is insufficient and only 215 JNDs are found for DICOM GSDF under direct sunlight. For example, when the image content is below 100 nits, there are 90 gray levels for the gamma 2.2 curve, while only 42 JNDs exist in human eye function according to DICOM. That means 48 gray levels are lost because they cannot be detected by the human eye due to a lack of JNDs, which results in a lower readability of these low-luminance images.

The second problem is described in Figure 2a, which depicts the mismatching between gamma 2.2 (blue curve) and the DICOM function (red curve). The gray level of the gamma curve and JND of DICOM are both normalized to the same scale for a fair comparison. The DICOM luminance level [L(j)] as a function of JND index (j) can be expressed as follows:

$$\begin{array}{c}{\log }_{10}\left[L\left(j\right)\right]\hfill \\ ={\displaystyle \frac{a+c\times \ln \left(j\right)+e\times {\left[\ln \left(j\right)\right]}^2+g\times { \left[\ln \left(j\right)\right]}^3+m\times { \left[\ln \left(j\right)\right]}^4 }{1+b\times \ln \left(j\right)+d\times {\left[\ln \left(j\right)\right]}^2+f\times {\left[\ln \left(j\right)\right]}^3+h\times {\left[\ln \left(j\right)\right]}^4+k\times {\left[\ln \left(j\right)\right]}^5}}\hfill \end{array}$$

(3)

where a, b, c, d, e, f, g, h, k, and m are constants that can be found in [32].

For a certain display peak luminance (L_P) and reflected ambient light (L_R = $\frac{I_a}{\pi }\times R$), the display luminance ranges from (L_R) to (L_P + L_R). The corresponding JND index in the DICOM model will be j_low and j_high. The mismatch (M) between the DICOM and the gamma curve can be described as follows:

$$M={\sum }_{j=j_{low}}^{j_{high}}|[L\left(j\right)-L_R-L_P\times {\left(\frac{\mathrm{j}-{\mathrm{j}}_{low}+1}{{\mathrm{j}}_{high}-{ j}_{low}+1}\right)}^{\mathrm{ϓ}}]/L\left(j\right)|$$

(4)

The optimized gamma value Υ can be achieved by minimizing M, as shown in the green curve in Figure 2a. Such a gamma correction by decreasing the gamma value from 2.2 to 1.46 helps to increase the luminance of low grayscale image contents and prevent them from being washed out. By calculating the fitted gamma values shown in Figure 2b, we plot the gamma value after tone mapping as a function of panel peak luminance and reflected ambient light intensity based on the DICOM GSDF. From Figure 2b, we notice that (1) a dynamic gamma value is required to match with human eye function based on ambient conditions and display brightness, and (2) the display panel with a high peak luminance prefers a large gamma value.

Figure 2c compares the readability of the tone-mapped grayscale image to the gamma 2.2 curve. Tone mapping is performed through exponential fitting to the DICOM function. After tone mapping by increasing the panel luminance of low grayscale contents, the gray level difference is more distinguishable by human eyes. On the other hand, a noticeable clipping effect can be found for large grayscale contents because of the increased gamma value and insufficient JND levels (215 compared to 255 gray levels), which only happens under strong ambient light conditions. This phenomenon will be discussed in Section 2.5.

2.2. Auto-Brightness Control

In the above analysis, the peak luminance of the display panel remains constant, while, in practical applications, the peak luminance is adaptive to the ambient light intensity. Therefore, we test the auto-brightness on a commercial smartphone display panel, a Google Pixel 7 Pro (Foxconn, New Taipei, Taiwan) with a surface reflectance of ~0.9% by adjusting ambient light illuminance from 0 lux to 12,000 lux. A D65 LED light box (DLS LED Color Viewing Light v7 S, Just Normlicht, Inc., Langhorne, PA, USA) in a dark room with a tunable light intensity from 0 lux to 12,000 lux is applied to simulate the daytime ambient lighting condition. The luminance of the display is measured by the luminance difference between the white image (gray level = 255) and the dark image (by turning off the phone) to cancel the surface reflectance in the calculation. The measured results (open circles) are plotted in Figure 3a. Combined with the gamma correction at different screen luminance values in Figure 2b, the adaptive tone mapping gamma values are obtained as Figure 3b depicts.

2.3. Mini-LED Backlit LCD Model

Before analyzing the image quality, we adapt a mini-LED backlit LCD model to simulate and optimize the image contents on the display. The simulation is constructed based on an LCD panel with a resolution of 1920 × 1200. The spacing between mini-LED chips is set at 0.5 mm, corresponding to a length of 16 pixels. The backlight unit (BLU) is divided into 40 × 25 local dimming zones, and each zone accommodates 3 × 3 LEDs. The LSF employed in the simulation is derived from Gaussian superposition and can be expressed as follows:

$$LSF\left(x,y\right)={\sum }_{i=1}^N{a}_i{e}^{\frac{{\left(x-u_{x,i}\right)}^2+{\left(y-u_{y,i}\right)}^2}{{\sigma }_i^2}},$$

(5)

where u_x and u_y are the spatial position of each LED, a_i is the parameter determining the peak luminance of LEDs, and σ is a variable for adjusting the full width at half maximum (FWHM) of the LED.

According to Equation (5), the backlight uniformity, which is determined by the LSF and FWHM of the mini-LED radiation pattern, is subsequently computed. The parameter σ is adjusted to attain various FWHM/pitch ratios, and we apply the uniformity standard as presented below:

$$\mathit{Uniformity}=(1-\frac{L_{\mathit{max}}-L_{\mathit{min}}}{L_{\mathit{avg}}}) \times 100\%,$$

(6)

where L_min, L_max, and L_avg are the minimum, maximum, and average values of the luminance of the backlight, respectively. Figure 4 shows the calculated uniformity of the mini-LED backlight as a function of the FWHM/pitch ratio. As the FWHM/pitch ratio increases, the uniformity rises rapidly and then saturates gradually. When the ratio exceeds 1, the uniformity slightly fluctuates because of the excessive light overlapping in the middle of the neighboring LEDs. The uniformity exceeds 98% when the FWHM/pitch ratio is 1.2 (σ = 0.41), 1.8 (σ = 0.60), and 2.3 (σ = 0.78). Previous research selects the FWHM/pitch ratio = 1.8 (σ = 0.60) [33]; however, a higher FWHM/pitch ratio intensifies the halo effect and reduces the light efficiency. As a result, we set the ratio at 1.2 in our simulated image test.

To generate simulated images and determine the grayscale of each local dimming zone in the BLU, the image data are transformed into a Hue Saturation Value (HSV) color model [34]. To determine the grayscale of each local dimming zone in the BLU, the root-mean-square (RMS) algorithm is applied to determine the gray level of each local dimming zone, and the light distribution of the mini-LED backlight is calculated. However, the RMS method may induce problems such as decreased peak luminance and a strong clipping effect. Therefore, to avoid image quality degradation, in Section 2.5, we will introduce the clipping effect in detail.

2.4. Image Quality

To evaluate the image quality degradation of a display under different lighting conditions, we conducted the measurement experiment in a D65 lightbox using a camera (SONY α6100, Tokyo, Japan; lens f# = 3.5–5.6, focal length = 16–50 mm). The measurement was conducted in a dark room and the only light source was the D65 lightbox. The original image shown in Figure 5a contains sufficiently low grayscale contents. Since the maximum illuminance of the lightbox is still much lower than that of direct sunlight (10⁵ lux), we tested three peak luminance values (Case I: 188 nits, Case II: 340 nits, and Case III: 566 nits) for a Google Pixel 7 Pro under the same ambient light illuminance of 12,000 lux. Confirmed by measurement, the default gamma value is set at 2.2 for all three cases. In other words, tone mapping is not applied to this device. As a comparison, according to the calculated results in Figure 2b, the corresponding tone-mapped gamma values for Case I, II, and III are 1.58, 1.83, and 2.07, respectively.

As illustrated in Figure 5b, for Case I with Υ = 2.2, the low gray level image contents (e.g., snow mountains) are all washed out by the strong ambient light. After applying tone mapping, the low grayscale contents are more distinguishable for Case I in both measurement and simulation. For example, details in the snow mountain and the starry sky are clearer. As the display peak luminance increases from 188 nits to 340 nits, the tone-mapped gamma value increases from 1.58 to 1.83, and the image washout is less obvious compared to Case I. On the other hand, for minor details, such as the ridges of the snow mountain, the tone-mapped images can be distinguished more easily. However, as the display peak luminance further increases to 566 nits, the difference for Case III is barely noticeable because the tone mapping gamma value is 2.07, which is rather close to 2.2. Therefore, display panels with a high peak luminance do not require a major gamma correction under strong ambient light, which is beneficial to avoid image washout. However, in realistic cases, direct sunlight (100 k lux) is much stronger than the experimental ambient light intensity (12 k lux). This considers that the peak luminance of most display panels is in the range of 1000 nits to 3000 nits, which is still much weaker than sunlight. Under such circumstances, tone mapping is still necessary to mitigate the image washout.

Figure 6 depicts the image quality before and after tone mapping, which is quantitatively analyzed by multi-scale structure similarity (ms-ssim) [35,36]. The reference image is chosen from Figure 5a. The ms-ssim of the tone-mapped image is consistently better than that of gamma 2.2 without tone mapping. As the display luminance keeps increasing, both sets of test images have a higher similarity to the reference image. Additionally, for displays with a high luminance, the difference between the tone-mapped images and gamma 2.2 images gets smaller, indicating that these two images are more difficult to distinguish. However, solely increasing the display peak luminance leads to higher power consumption and an undesirable thermal effect.

2.5. Clipping Effect

As discussed in Section 2.1, the clipping effect becomes significant when the panel luminance is not sufficient under strong ambient conditions. The minimum display luminance requirement for the human eye is to include at least 255 JND levels in DICOM GSDF as calculated in Figure 7a. For example, to obtain 255 JND levels in human vision under 100 k lux ambient light, a 1400-nit display panel is necessary to display an 8-bit image. After applying the tone mapping with a gamma value of 1.58 (Figure 7b), the simulated grayscale image in Figure 7c depicts a better visibility for low gray level contents and an acceptable visibility for the counterpart.

2.6. Color Shift

Under intense ambient lighting conditions, the color of the display shifts toward the white point. If tone mapping is applied to the display, the color coordinates will further shift toward the white point because the luminance of low grayscale content increases. Color rendering (CR) is required to compensate for this color shift and can be achieved by adjusting the transmittance of the LC layer:

$$T_{LC,(r,g,b)}={\left(\frac{G_{\left(r,g,b\right)}}{255}\right)}^{\gamma }+C{R}_{(r,g,b)}$$

(7)

The term $[{\left(\frac{G_{\left(r,g,b\right)}}{255}\right)}^{\gamma }+C{R}_{(r,g,b)}]$ is in the range of [0, 1]. The value of CR_(r,g,b) can be calculated by minimizing the color coordinate difference between the original picture without ambient light and the rendered picture with dynamic ambient light. Instead of sub-pixel color rendering, the value of CR is a global modulation for all the pixels to avoid image nonuniformity.

During our optimization of CR, we applied 18 reference colors in the Macbeth ColorChecker. The color performance will be more accurate to the original picture if more colors are considered. The average color shift of these 18 reference colors is shown in Figure 8. The color shift becomes more severe as the ambient light illuminance increases. For displays with fixed Υ = 2.2 (blue curve), the color shift is recognizable when ambient light illuminance is 10⁴ lux. After applying tone mapping, the color shift (red curve) is very high under dim ambient light conditions. This can be attributed to the large gamma value when the reflected ambient light intensity is low, as shown in Figure 2b. However, this is not problematic because the auto-brightness control limits the display peak luminance in a dark room, as described in Figure 3. Under intense ambient light, applying tone mapping slightly increases the color shift compared to the display with Υ = 2.2. Displays with CR (green curve) help increase the JND to 2 × 10⁴ lux. Figure 8b compares the simulated color performance of six selected mixed colors of a 1000-nit display under direct sunlight (10⁵ lux). In this case, the CR value is −0.16, which shifts the color coordinates away from the white point. Overall, this color rendering method avoids pixel-level modulation while it slightly disrupts the fitting to the DICOM function. Therefore, CR is optional and dependent on specific applications.

3. Optical See-Through Displays

Optical see-through displays such as HUDs and HMDs all face tremendous image washout issues due to intense ambient light. Like Equations (1) and (2), the luminance of an 8-bit transparent display panel at different grayscales can be expressed as follows:

$$L=L_r\times {\left(\frac{G_r}{255}\right)}^{\mathrm{ϓ}}+L_g\times {\left(\frac{G_g}{255}\right)}^{\mathrm{ϓ}}+L_b\times {\left(\frac{G_b}{255}\right)}^{\mathrm{ϓ}}+\frac{I_a}{\pi }\times T$$

(8)

$$L_p=L_r+L_g+b$$

(9)

where T = 70% is approximately the transparency of the display panel or the optical combiner. According to previous studies and fitting to the eye function in Figure 9a, we calculate Υ corrected tone mapping and consider three cases with varying ambient light intensity: room with artificial light (~290 lux, black vertical dashed lines), outdoor under big trees (~1700 lux, blue vertical dashed lines), and open playground (~14,350 lux, red vertical dashed lines) [37]. For rooms with artificial light, the transparent TV panel or AR eyeglasses with 1000-nit peak luminance require an optimized gamma value of 2.17, which is already close to the typical indoor display gamma value of 2.2. On the other hand, for outdoor applications, tone mapping and high peak luminance are necessary to improve the readability of low grayscale images. As shown in Figure 9a with a blue dot and Figure 9b, a 1000-nit display requires a gamma value of 1.15 when using AR eyeglasses on an open playground. However, such a low luminance is not sufficient to avoid image washout. Figure 9c depicts that the character image without tone mapping is still washed out even when L_P increases to 4000 nits. After applying tone mapping (Υ = 1.34), the image content is more distinguishable.

It is difficult to directly analyze the image quality of a transparent display because the final image on the panel always contains both display and background information unless an optical dimmer is applied. As shown in Figure 10a, we select the reference image as an open playground without any image content from the display. When an image appears on the display, the ms-ssim of the image with respect to the reference image will decrease. In this case, higher image readability leads to a smaller ms-ssim, or a larger [1 − (ms-ssim)] value. Figure 10b shows [1 − (ms-ssim)] as a function of the peak luminance of the display based on the images in Figure 9c. The readability of the tone-mapped image is consistently better than that of gamma 2.2.

4. Conclusions

True tone mapping through gamma correction under different ambient conditions and the display peak luminance is discussed to fit with the human eye’s function. To enable practical applications, we simulate the images based on a mini-LED backlit LCD model, and the FWHM-to-pitch ratio is optimized at 1.2 to achieve the highest backlight uniformity. Combining these two methods with experimental validation, we find that low grayscale information is more distinguishable by the human vision system, which is beneficial for infotainment automotive displays and mobile phones to improve their sunlight readability. In addition, the minimum peak display luminance required to alleviate the clipping effect is analyzed by including at least 255 JNDs in the human visual system. Color rendering can reduce the color shift; however, this may result in a less fit to the DICOM function. Finally, after applying tone mapping and adjusting the peak luminance of the display to the optical see-through displays, 4000-nit brightness is required to achieve a premium visual experience for daily use.