A Novel Anatomy Education Method Using a Spatial Reality Display Capable of Stereoscopic Imaging with the Naked Eye

Featured Application

A stereoscopic display without a head-mounted display is highly useful and promising for anatomy education.

Abstract

Several efforts have been made to use virtual reality (VR) and augmented reality (AR) for medical and dental education and surgical support. The current methods still require users to wear devices such as a head-mounted display (HMD) and smart glasses, which pose challenges in hygiene management and long-term use. Additionally, it is necessary to measure the user’s inter-pupillary distance and to reflect it in the device settings each time to accurately display 3D images. This setting is difficult for daily use. We developed and implemented a novel anatomy education method using a spatial reality display capable of stereoscopic viewing with the naked eye without an HMD or smart glasses. In this study, we developed two new applications: (1) a head and neck anatomy education application, which can display 3D-CG models of the skeleton and blood vessels of the head and neck region using 3D human body data available free of charge from public research institutes, and (2) a DICOM image autostereoscopic viewer, which can automatically convert 2D CT/MRI/CBCT image data into 3D-CG models. In total, 104 students at the School of Dentistry experienced and evaluated the system, and the results suggest its usefulness. A stereoscopic display without a head-mounted display is highly useful and promising for anatomy education.

1. Introduction

Several efforts have been made to use virtual reality (VR) and augmented reality (AR) for medical and dental education and surgical support. In addition, these technologies have been shown to be helpful during use in several VR- and AR-assisted surgeries [1,2,3,4], and beneficial in several reports of anatomy education using VR and AR [5,6,7,8,9]. The current methods still require wearing devices, such as a head-mounted display (HMD) and smart glasses, which pose challenges in hygiene management and long-term use. Several reports have also reported eye fatigue and sickness when using HMD and AR smart glasses [10,11,12,13,14,15,16,17,18].

Additionally, it is necessary to measure the user’s interpupillary distance (IPD) and to reflect it in the device settings each time to display 3D images accurately. Measuring the user’s IPD takes 2–3 min each time, which makes the device unsuitable for daily use in a busy medical field, or an educational field in which dozens of students use it for a short time. Microsoft HoloLens, a common type of AR smart glasses, has a specification that requires the 3D-CG model to be placed at least 1 m from the eye to be seen in true 3D [19]. In other words, 3D-CG models located as close as 30 cm in proximity do not appear truly three-dimensional. This is a structural problem of AR smart glasses that requires a solution. It is necessary to display 3D-CG models with perfect stereoscopic effects within a close range of 30 to 50 cm for surgery and anatomy education. In the past, 3D displays that can be viewed stereoscopically with the naked eye have been researched and developed. However, these displays are not practical, as it is challenging to achieve both a high-quality image and high-speed screen refresh rates [20,21].

This study developed and implemented a novel anatomy education method using a spatial reality display (Sony, ELF-SR1, Tokyo, Japan, 2020) capable of stereoscopic viewing with the naked eye without an HMD or smart glasses [22]. The spatial reality display (SRD) can display a 3D-CG model with perfect three-dimensionality within a close range of 30 to 50 cm. The SRD can display 3D-CG models at 4K resolution and offers high-speed screen refresh rates, making it practical. Although 3D displays that do not require 3D glasses exist, they have low resolutions and cannot present highly accurate 3D images due to a phenomenon called “crosstalk”, in which the image displayed to one eye is mixed with the image displayed to the other eye [20,21]. An SRD overcomes these problems of conventional 3D displays and is the first truly practical product available for purchase on the general market. Its price is approximately USD 5000, and it requires a PC with a value of about USD 2000 to run the application. Currently, there are no commercially available applications for SRD, so users must develop their own. There are also no applications for medical and dental education. This study developed and evaluated the usefulness of two new applications for a spatial reality display: (1) a head and neck anatomy education application called SR Anatomy, which can display 3D-CG models of the skeleton and blood vessels of the head and neck region using 3D human body data available free of charge from public research institutes, and (2) a DICOM image autostereoscopic 3D viewer called DSR View, which can automatically convert 2D CT/MRI/CBCT image data into 3D-CG models.

In total, 104 students at the School of Dentistry experienced and evaluated SR Anatomy. In addition, 12 dentists working at the University Hospital experienced DSR View and commented on its use. To evaluate the accuracy of DSR View, we compared the interdental distance of the actual dental education model and the distance of the 3D-CG naked-eye stereoscopic display of the DICOM data obtained by CT imaging from the actual model. We analyze the results of these assessments and discuss their usefulness as a new anatomy education method.

2. Materials and Methods

2.1. Hardware: A Spatial Reality Display

This study developed and implemented a novel anatomy education method using a spatial reality display (SRD) capable of stereoscopic viewing with the naked eye [22]. Sony’s SRD can display a 3D-CG model with perfect -dimensionality within a close range of 30 to 80 cm. The SRD can display 3D-CG models at 4K resolution and offers high-speed screen refresh rates, making it practical. This spatial reality display is also called an Eye-sensing Light Field Display.

This display SRD utilizes 3 of Sony’s core technologies: (1) real-time sensing technology with high speed and high accuracy, (2) real-time ray rendering technology, and (3) high-precision 3D display technology.

(1) Real-time sensing technology with high speed and high accuracy. This technology uses Sony’s original high-speed vision sensor and face-recognition technology to achieve high detection accuracy and low-latency eye recognition. To make the viewer feel as if the object is there, it is necessary to constantly display an image with the correct viewpoint for both of the user’s eyes. With this technology, the positions of the left and right eyes can be calculated in real time in both the horizontal and vertical directions and in the depth direction.

(2) Real-time ray rendering technology. This technology first assumes that an object is placed in the display device based on the user’s location. Then, the technology acquires an image of the user looking at it with both eyes. It then converts the image into a kind of trompe l’oeil on the display surface. This is then combined with the micro-optical lens technology [23,24], which consistently delivers correct images to the user’s left and right eyes to produce an actual light source image from the display panel. This image, created in real time, is displayed with high precision and speed, and the display always presents an image with the correct viewpoint to both eyes. As a result, the user feels as if the subject is actually there.

(3) High-precision 3D display technology. The display makes full use of Sony’s high-speed vision sensor and its unique micro-optical lens, which is designed to follow the line of sight to significantly reduce crosstalk and to provide high-definition images compared to conventional 3D displays. Figure 1 shows the SRD and a user pointing to the 3D-CG model.

2.2. Software

In this study, we developed 2 new applications for the SRD. The SRD was connected to a desktop computer (Mouse Computer, Tokyo, Japan, 2019, CPU: Intel Core i7-9700K 3.60 GHz, RAM: 16 GB, GPU: NVIDIA GeForce RTX 2060 SUPER, OS: Windows 10 Pro) via a USB cable and an HDMI cable, and the following applications were run on the PC.

2.2.1. Head and Neck Anatomy Education Application: SR Anatomy

SR Anatomy can display 3D-CG models of the skeleton and blood vessels of the head and neck region using 3D human body data available free of charge from public research institutes [25]. The user can easily rotate, scale, and move the model using a keyboard or a commercially available game controller. Even first-time users can operate SR Anatomy intuitively. The user can also switch between transparent bone-only display and artery-only display in the same way. The arteries in the maxillofacial region are annotated with names, which can be shown or hidden by pressing a button on the keyboard or a controller. In combination with the Leap Motion controller [26], users can also manipulate the 3D model via hand and finger movements. Figure 2 and Figure 3 show examples of SR Anatomy, whereas Figure 4 and Figure 5 show examples of finger manipulation.

2.2.2. DICOM Image Autostereoscopic 3D Viewer DSR View

DSR View can automatically create 3D-CG models and display them in the SRD with the naked eye by selecting a folder containing DICOM data output from the computed tomography scan (CT)/magnetic resonance imaging (MRI)/cone beam computed tomography (CBCT) modalities. DICOM data are the standard format for cross-sectional images taken by CT/MRI/CBCT. DSR View provides naked-eye stereoscopic viewing of the DICOM data outputs from each of the 15 CT/MRI/CBCT modalities and is compatible with nearly all models. The threshold value can be changed arbitrarily. The user can easily rotate, scale, and move the model using a keyboard or a commercially available game controller. First-time users can also operate DSR View intuitively. Figure 6 and Figure 7 show examples of DSR View.

2.3. Statistical Analysis

We performed Fisher’s exact probability test on the cross-tabulations using SPSS. The significance level was set at p = 0.05.

3. Results

3.1. Evaluation of SR Anatomy

In total, 104 students at the School of Dentistry experienced and evaluated SR Anatomy. The participants were all second-year students of the School of Dentistry, Kanagawa Dental University, who participated in the head and neck dissection training curriculum. The head and neck dissection training curriculum consisted of 20 periods—the method proposed in this paper was used for one of these periods (1.5 h). When the students practiced head and neck dissection, they experienced SR Anatomy individually in a separate room in the same building as the dissection training room. The SR Anatomy experience per student lasted about 3 min. Students completed a questionnaire before and after the experience. The response time was 5 min each. We conducted a questionnaire survey of the 104 students before and after the experience and received valid responses from 99 students. We excluded seven inconsistent responses from the 99 respondents, leaving 92 for analysis. The validity of the survey questions and quantities was verified and determined by several anatomy faculty members and a faculty member with extensive experience in statistical analysis. The content of the questionnaire survey (Supplementary Materials) was reviewed and approved by the Ethics Review Committee of the Kanagawa Dental University.

In their impressions of the spatial reality display (SRD) experience, 53% of the students answered that their experience went extremely well and 44% answered that it went very well. Regarding their comprehension of anatomy using the SRD, 18% of the students answered that it went extremely well and 64% answered that it went very well. Before the SRD experience, in the application of SRD to anatomy education, 21% of the students answered that the application of SRD is absolutely necessary and 62% answered that it is necessary. After the SRD experience, in the application of SRD to anatomy education, 38% of the students answered that SRD application is absolutely necessary and 56% answered that it is necessary.

The relationship between students’ comprehension of anatomy using the SRD and their impressions of the SRD experience was statistically significant (p = 0.004). The relationship between their comprehension of anatomy using the SRD and the amount of information displayed by the SRD was also statistically significant (p = 0.006). It was suggested that the level of understanding of anatomy by the SRD was related to satisfaction with SRD use and the amount of sufficient information in the SRD. Figure 8 and Figure 9 show examples of the experiences of dental students, Figure 10 shows their impressions of the SRD experience, and Figure 11 shows their comprehension of anatomy using the SRD. Additionally, Figure 12 shows the necessity of the application of the SRD in anatomy education before the SRD experience, and Figure 13 shows the necessity of the application of the SRD in anatomy education after the SRD experience.

Table 1. Correlation between the comprehension of anatomy using the SRD and impressions of the SRD experience.

		Impressions of SRD Experience				p *
		Extremely Satisfied	Moderately Satisfied	Neither	Moderately Dissatisfied
Comprehension of anatomy using SRD	Extremely well	14	3	0	0	0.004
		82.4%	17.6%	0.0%	0.0%
	Very well	31	29	1	0
		50.8%	47.5%	1.6%	0.0%
	Moderately well	3	9	1	1
		21.4%	64.3%	7.1%	7.1%

* Fisher’s exact test.

shows the correlation between students’ comprehension of anatomy using the SRD and their impressions of the SRD experience, and

Table 2. Correlation between the comprehension of anatomy and the amount of information displayed by the SRD.

		Amount of Information Displayed by SRD				p *
		Extremely Satisfied	Moderately Satisfied	Neither	Moderately Dissatisfied
Comprehension of anatomy using SRD	Extremely well	15	2	0	0	0.006
		88.2%	11.8%	0.0%	0.0%
	Very well	28	30	1	2
		45.9%	49.2%	1.6%	3.3%
	Moderately well	5	6	2	1
		35.7%	42.9%	14.3%	7.1%

* Fisher’s exact test.

shows the correlation between their comprehension of anatomy using the SRD and the amount of information displayed by the SRD.

3.2. Evaluation of DSR View

3.2.1. Subjective Assessment by Dentists

In total, 12 dentists working at the University Hospital experienced DSR View and commented on its use.

• “It looks more three-dimensional than smart glasses.”
• “It is lovely to be able to grasp complex three-dimensional structures easily.”
• “It allows us to grasp the volume of the lesion and realize the need for measurement.”
• “It is convenient to be able to display three-dimensional images from DICOM data immediately.”

3.2.2. Objective Verification of the Accuracy of DSR View

To verify the accuracy of DSR View, we compared the interdental distance of the actual dental education models (SHOFU INC., Japan) [27] and the distance of the 3D-CG naked-eye stereoscopic display of the DICOM data obtained by CT imaging from the actual models. In this evaluation, we used five different actual models for dental education. We captured CT (CANON MEDICAL SYSTEMS, Aquilion Lightning, Ohtawara, Japan) images of five different models and acquired DICOM data. DSR View displayed the DICOM data of each model three-dimensionally to the naked eye. The measurer was able to see the model in a truly three-dimensional manner and accurately measure any interdental distances using a caliper. We manually measured the horizontal interdental distance and the depth interdental distance for each model with a high-precision digital display caliper (Shinwa Sokutei, Digital Ruler No.19975, Sanjyo, Japan) capable of measuring in 0.01 mm increments for the substantive model. The same digital caliper was used by the measurer to manually measure the same part of the model displayed by DSR View. For each model, we measured two positions in the horizontal direction and two locations in the depth direction, for a total of four positions. Position a in the horizontal direction is the smallest interdental distance; Position b is the largest interdental distance. Position a in the depth direction is the maximum interdental distance; Position b is the length of the first premolar in the depth direction. The measurement results are shown in

Table 3. Measurement results in the horizontal direction.

Model No.	Position	Actual Model	DSR View	Difference
1	a	31.51	31.51	0.00
	b	41.65	41.65	0.00
2	a	5.10	5.10	0.00
	b	34.45	34.45	0.00
3	a	18.27	18.27	0.00
	b	41.97	41.97	0.00
4	a	40.29	40.29	0.00
	b	44.92	44.92	0.00
5	a	18.38	18.38	0.00
	b	41.64	41.64	0.00
				(mm)

and

Table 4. Measurement results in the depth direction.

Model No.	Position	Actual Model	DSR View	Difference
1	a	13.02	13.02	0.00
	b	7.25	7.25	0.00
2	a	10.33	10.33	0.00
	b	7.21	7.21	0.00
3	a	18.01	18.01	0.00
	b	7.83	7.83	0.00
4	a	43.56	43.56	0.00
	b	7.24	7.24	0.00
5	a	17.70	17.70	0.00
	b	7.37	7.37	0.00
				(mm)

, where the difference was less than 0.01 mm. Figure 14 and Figure 15 show examples of measurements in the horizontal direction, whereas Figure 16 and Figure 17 show examples of measurements in the depth direction. Table 3 shows the measurement results in the horizontal direction, and Table 4 shows the measurement results in the depth direction.

4. Discussion

This study developed and implemented a novel anatomy education method using a spatial reality display (SRD: Sony, ELF-SR1, Tokyo, Japan, 2020) capable of stereoscopic viewing with the naked eye without an HMD or smart glasses. The SRD has a significantly better performance than conventional 3D displays [20,21] and does not experience problems with crosstalk or display delay. The stereoscopic effect and image quality of the displayed 3D-CG models are practical. As a fully commercial product, it is now readily available for purchase in Japan and the U.S., as of June 2021. This study developed and evaluated the usefulness of two new applications for spatial reality display: (1) a head and neck anatomy education application called SR Anatomy and (2) a DICOM image autostereoscopic 3D viewer called DSR View.

The dental students’ experiences with SR Anatomy suggest the usefulness of this application. In addition, the level of understanding of anatomy using the SRD was related to students’ satisfaction with SRD use and the sufficient amount of information in the SRD.

The University Hospital dentists’ experiences with DSR View also suggest the usefulness of this application.

There was no difference in the measurement results between the five actual models and the 3D-CG model created from the DICOM data of the CT imaging of the models and displayed in DSR View. This result suggests that the DICOM data displayed in DSR View are accurate.

Overall, these results suggest that the two applications developed in this study are useful in anatomy education, and a new method of anatomy education was established. A stereoscopic display without a head-mounted display is very useful and promising for anatomy education.

5. Conclusions

This study developed and implemented a novel anatomy education method using a spatial reality display (SRD) capable of stereoscopic viewing with the naked eye. Sony’s SRD can display a 3D-CG model with perfect three-dimensionality within a close range of 30 to 80 cm. The SRD can display 3D-CG models at 4K resolution and offers high-speed screen refresh rates, making it practical. Although 3D displays that do not require 3D glasses exist, they have low resolutions and cannot present highly accurate 3D images due to a phenomenon called “crosstalk”, in which the image of one eye is mixed with the image of the other eye. The SRD overcomes the problems associated with conventional 3D displays and is the first truly practical product widely available for purchase.

Currently, there are no commercially available applications for the SRD. There are also no applications for medical and dental education. This study developed and evaluated the usefulness of two new applications for the SRD. The evaluation results suggest that the two applications developed in this study are helpful in anatomy education, and a new method of anatomy education was established. Because each student is not currently able to be given their own SRD, the experience time per student is limited. However, this problem can be addressed by increasing the number of SRDs and devising a means to experience them. In addition, the short experience of about 3 min was sufficiently practical in this study.

A stereoscopic display without a head-mounted display is highly useful and promising for anatomy education. In the future, practical 3D displays will be devised by other companies, and prices will decrease. For 3D displays to gain popularity in the education field, software engineers and faculties must work closely together. It is essential to periodically evaluate the educational effects of 3D displays and to continuously improve educational methods.