Augmented Reality Implementations in Stomatology

Abstract

Augmented reality has a wide range of applications in many areas that can extend the study of real objects into the digital world, including stomatology. Real dental objects that were previously examined using their plaster casts are often replaced by their digital models or three-dimensional (3D) prints in the cyber-physical world. This paper reviews a selection of digital methods that have been applied in dentistry, including the use of intra-oral scanning technology for data acquisition and evaluation of fundamental features of dental arches. The methodology includes the use of digital filters and morphological operations for spatial objects analysis, their registration, and evaluation of changes during the treatment of specific disorders. The results include 3D models of selected dental arch objects, which allow a comparison of their shape and position during repeated observations. The proposed methods present digital alternatives to the use of plaster casts for semiautomatic evaluation of dental arch measures. This paper describes some of the advantages of 3D digital technology replacing real world elements and plaster cast dental models in many areas of classical stomatology.

1. Introduction

Augmented reality (AR) covers many research areas [1,2,3,4,5], including biomedicine, neurology and engineering. The implementation of these methods in stomatology [6,7,8,9,10,11] extends the medical study of real dental objects into the digital world. This paper describes how real dental objects can be replaced by their digital models in the present cyber-physical environment, which combines augmented and virtual reality [7]. The efficiency of the use of AR in stomatology can be seen during surgical navigation [12], manipulation in implantology [13,14], and applications in maxillofacial surgery [15] as well.

The initial stage of digitization and the introduction of computational methods in stomatology include the replacement of physical plaster casts models by digital models [16,17] that are recorded by digital cameras, as presented in Figure 1. This approach overcomes several of the limitations of traditional plaster models; for example, they can be easily damaged, require time for model fabrication, and require space to be set aside for their storage [18]. This use of digital technologies allows orthodontists to more accurately analyze dental problems, predict their treatment, more precisely specify the refabrication tasks, and it simplifies the collaboration with further specialists, remote research laboratories and educational institutions.

The use of advanced 3D scanners, as presented in Figure 2a, and wireless communication links allows the construction of more accurate digital models (see Figure 2b) that are based on stereolithography (STL) data. The use of AR for user-friendly intra-oral scanning [19] allows detailed analysis of dental objects, restoration of tooth cavities and tooth alignment in the computational environment, and then enables the return from the digital to real world again. This approach is often used for surgical navigation and for the study of dental arch models during the orthodontic treatment.

In all these cases, digital technologies are extensively combined with computational intelligence to construct and analyze digital models with sufficient accuracy allowing for reducing the examination of real objects in the clinical practice. Specific mathematical methods are used in this stage for digital rotation, shifting and scaling of digital objects, and for their registration, feature learning and classification.

Three-dimensional (3D) modelling is a rapidly developing area that uses digital signal processing methods to analyse 3D bodies and their time evolution. The 3D data processing forms the basis of modern robotic and information systems, including the possible implementation of virtual reality methods. The results of these technologies include printing of spatial bodies, including dental objects.

The whole methodology related to the AR in stomatology is based upon the use of specific sensors, cameras, contactless scanners [18], and (wireless) communication links, followed by 3D object reconstruction. Applications of these methods assume the knowledge of optics [20], engineering [21], biomechanics, information technologies [22,23], and material sciences.

Mathematical and computational methods used for data processing in this area include procedures of signal and image analysis, data denoising methods [24,25], morphological operations, image registration [26], stereo vision methods, and statistical evaluation of the results. Functional transforms—including wavelet analysis and specific optimization methods—are important for feature extraction and classification. Meanwhile, 3D data processing also assumes the application of geometrical modelling [27]. Point cloud data processing and the 3D modelling of spatial objects form further specific research areas, which are based upon the use of digital signals and image processing methods.

This paper surveys selected methods related to the 3D modelling in stomatology, which allow the implementation of AR, and the examination and treatment of stomatological disorders. It presents methods of data acquisition by camera systems [16,28] and digital evaluation of plaster casts by simple and more advanced mathematical methods.

Special attention is devoted to processing the data recorded by an intra-oral contactless scanner [20,29,30,31,32,33,34], as presented in Figure 2a, which allows the construction of 3D models as a basic step in the whole complex digitization process. The resulting data are then de-noised and analyzed to select the dental measures [22] that are important for the treatment of specific disorders. Image registration methods are then used to compare images during the treatment and to analyze its time evolution.

2. Methods

Augmented reality in stomatology uses different methods to process data obtained by panoramic radiography, to analyse images of dental casts, or to process data acquired by intra-oral scanning devices. No ethical approval was required for this study.

2.1. Panoramic Radiography

Dental panoramic radiographs are a form of focal plane tomography and they are often used for clinical examination in the diagnosis of dental diseases. These methods are routinely applied in dental practice, despite the limitations of two-dimensional (2D) analysis. Figure 3 presents an example of the resulting image, which can be further analysed by specific image processing [35] and edge detection methods.

The computational methods that are used in this area include common edge detectors (Roberts, Sobel, Previtt and Canny methods), which are based upon the use of 3 × 3 matrix kernels that are convolved with the image for estimation of the first derivative in the horizontal ($G_x$) and vertical ($G_y$) directions, followed by edge gradient evaluation using relation

$$G=\sqrt{G_x^2+G_y^2}.$$

(1)

Recently, several new computer-assisted approaches [36,37] have been proposed that have better performance efficiency and allow the detection of edges in complex medical images with different artefacts.

2.2. Camera Systems in Orthodontic Measures Evaluation

Dental casts still play an important role in the diagnosis and in the planning of the treatment of dental disorders. They are the gold standard in the evaluation of the dental arch shape, in measuring dental arch distances, and tooth positions. Associated digital models can simplify their use because they allow access to the patients’ records through the computer network, they increase the possibility to share the models with other specialists during the therapy, and they enable accurate measurements for the diagnosis setups using selected image processing methods.

While manual measurements of plaster casts can be used, it is possible to create digital models of them [22] and collect these measurements using general image processing methods applied to images from a single camera, as presented in Figure 4. Manually detected rectangular regions A and B can be then used for the automatic detection of their extreme values applied to the thresholded image. If we know the camera resolution and the distance between image pixels, it is possible to evaluate the distance between selected teeth during the treatment.

The multicamera spatial modelling approach can be used to reduce the errors caused by the use of a single camera. Figure 5 presents the system of two cameras located at a selected distance c to follow a specific point C on a plaster cast in 3D space. Cameras $R1$ and $R2$ together with the object C form a triangle that can be used to determine its spatial coordinates. The angular and spatial resolution of both cameras in the selected coordinate system was evaluated during the calibration stage [22]. Corresponding objects $C1,C2$ detected by both cameras can be then specified by their coordinates $[x_{C1},y_{C1},z_{C1}]$ and $[x_{C2},y_{C2},z_{C2}]$ and their distance can be evaluated using relation

$$D=\sqrt{{(x_{C2}-x_{C1})}^2+{(y_{C2}-y_{C1})}^2+{(z_{C2}-z_{C1})}^2}.$$

(2)

Given that the process of the treatment in dentistry includes also time evolution aspects, it is often necessary to compare the dental arch parameters changes. The dental plane evaluated from the spatial coordinates of the teeth tops from separate observations can be then used [22] to define its parameters

$$a_{1}x+a_{2}y+a_{3}z+a_4=0$$

(3)

using the least square method and the selected Cartesian coordinate system. Coefficients ${\left\{a_i\right\}}_{i=1}^4$ were then used [22] to rotate all of the objects around individual coordinates into the horizontal plane, which allows digital comparison of individual teeth positioning.

Figure 6a,b present locations of the centres of individual teeth in 3D space as evaluated by the proposed method [22] and the plane of their positions in the selected Cartesian coordinate system. The teeth centers specified in dental planes and rotated into the horizontal position are presented in Figure 6c, together with the dental arch curve. This enables locations of the teeth to be evaluated during treatment from the spatial coordinates that have been obtained. This transform allows us to detect changes in the dental arch with a stereo matching algorithm [38,39]. Corresponding pairs of images obtained before and after the dental treatment enable us to use 2D registration to evaluate the results of the operations [40].

The use of camera systems is closely related to the appropriate illumination [23] and separation of overlapping regions. Figure 7 presents the results of segmentation applied to the image of a dental arch acquired with combined light sources to enhance image contours, which is processed by moving average and median filtering to reject the high-frequency and impulsive image noise components. Figure 7a presents the results of the circular Hough transform, which allows us to detect individual objects, and the results of the segmentation process using the region growing and convex hull methods in Figure 7b.

The detail segmentation of overlapping regions that is presented in Figure 8 is based on separation of individual objects using specific methods based upon definition of convex regions in the observed image [23]. The following detail segmentation can use morphological methods, watershed transform and region growing by multiple seed points to substantially improve the result of this segmentation process.

2.3. Scanning Technologies in Spatial Model Construction

The scanning technologies that are used in dentistry [41,42] include triangulation using laser light, parallel confocal imaging, or fringe interferometry based upon projection of light patterns.

Data acquisition systems mostly include sensors located in a handheld camera on a wand that collects surface data inside the mouth cavity, as presented in Figure 2a, and a communication system that transmits these data into a computer for further processing. Either laser or white light is used to illuminate the scanned area and its reflection is acquired by specific sensors. Signal and image processing methods are then used to analyse and visualise the resulting 3D model.

The widely used Trios intra-oral scanning device uses advanced parallel confocal imaging. In principle, this device [43,44,45,46] captures sets of 2D images of the oral cavity and then combines them into a spatial model using optical sectioning technology [41] and geometrical modelling. STL data exported from the Trios intra-oral scanner into the Matlab and Comsol computational environments were used to construct and analyze digital models of the lower and upper dental arches. The resolution of the Trios scanner [47] was 41.2 points/mm${}^2$ and each model was constructed from about 300,000 vertices.

Algorithms have also been developed to process the point cloud datasets acquired by the intra-oral scanner, including the selected signal processing methods, imaging technologies [32,41], and mathematical methods to construct the surface areas. Delaunay triangulation and Voronoi diagrams form the basis of the computational geometry used in this area.

Specific methods of data processing presented in [34] included the transform of recorded datasets into the selected 3D coordinate system, forming a matrix $\mathbf{A}$ and using the chosen grid size of the plane $[x,z]$, as presented in Figure 9. The evaluated data were then de-noised by a median filter as a nonlinear digital filtering technique using the moving window and a mask of 9 by 9 elements, resulting in a new matrix $\mathbf{B}$. The selection of the contour level density was then applied to detect specific areas of the individual teeth. These regions of interest were then used to estimate the dental arch parameters.

The coordinates of evaluated tops of each tooth were used [34] for parabolic approximation of the dental arch by a function

$$f\left(x\right)=c_1{x}^2+c_{2}x+c_3$$

(4)

for x in a range of minimum and maximum values on the horizontal axes. The coefficients of this function formed further measures associated with each dental arch.

Image rigid registration methods that are used for acquired 3D dental arch data can be then applied to study the reproducibility of the proposed method in the analysis of dental arches and to follow the time evolution of observed dental measures during the treatment. The results presented in [34] used datasets acquired by an intra-oral scanner and recorded in the STL format, which were analyzed and processed in the MATLAB (ver. R2019a, The MathWorks, Inc., Natick, MA, USA) and COMSOL Multiphysics (ver. 5.3, COMSOL, Inc., Stockholm, Sweden) computational, simulation, and visualization environments.

3. Results

Figure 2b presents a selected STL model of a lower dental arch obtained by an intra-oral scanner; further details are presented in [34]. The contour plot of the evaluated digital model in 3D space is presented in Figure 9a. Figure 9b presents a 3D plot of its selected body (a molar tooth) with the associated contour plot in Figure 9c, which is used for detail visualisation of its shape and to specify the regions of interest.

Figure 10 presents digital models of the left and the right canine teeth. The details of their contours presented in Figure 10c,d were used to precisely locate the teeth tops coordinates $x_k,z_k$ associated with the value

$$y_k=max\left(y(i,j)\right) \mathrm{for} i,j\in U_k,$$

(5)

where $U_k$ specifies the rectangular region around the k-th tooth. This semiautomatic specification of the teeth tops coordinates allows us to estimate the distances between the left and the right canine teeth (for $k=L3$ and $k=R3$, respectively) by relation (2), which defines a valuable coefficient used to specify the dental arch and to evaluate the treatment. An alternative approach based upon object rotation into the horizontal plane is presented in [22,34].

The evaluation of distances between canine teeth was done for 20 observations of the same individual in [34]. The difference in standard deviation to process the lower and upper dental arches was explained by the less convenient scanning of the upper dental arch, which was controlled on the monitor of the scanner itself.

Figure 11 presents the process of the rigid registration [48] of the input and base images (in Figure 11a,b, respectively) of the same individual. The fixed points are selected from the 3D model and the tops of the individual teeth. The image registration includes scaling, rotation and translation of all data points $\left\{x\right(i,j),z(i,j\left)\right\}$ into their new locations $\left\{X\right(i,j),Z(i,j\left)\right\}$, which can be performed by matrix multiplication

$$[X(i,j),Z(i,j)]=[x(i,j),z(i,j),1]\left[\begin{array}{cc}a\left(3\right)& b\left(3\right)\\ a\left(2\right)& b\left(2\right)\\ a\left(1\right)& b\left(1\right)\end{array}\right]$$

(6)

with optimized values of the transform matrix.

The distances between symmetrical values evaluated for the base and registered images by Equation (4), as described in [34], are lower than 4% in this case. The results of the registration presented in Figure 11c can be further used to evaluate the treatment and time comparison of dental arch changes.

Intraoral scanners provide stereographic files that can be used for both digital analysis of real objects and for their 3D printing [49,50,51,52]. The accuracy depends on the scanner and printer technology used, as presented in Figure 12.

AR in stomatology is closely related to real data acquired as a plaster cast or as a digital model. In the computational environment, these data can be analysed by numerical methods and computational intelligence tools can be used to evaluate the information that is necessary for diagnostics and the treatment of dental disorders. In this way, the real world can be combined with virtual reality, while the computational environment extends direct observations and allows for a more detailed understanding of real situations and processes.

Digitization has become a regular part of dentistry, orthodontics and maxillofacial surgery. The 3D reconstructive imagery based on a cone beam computed tomography (CBCT), intraoral and facial scans and computer-aided design techniques and technologies is more precise and can replace 2D X-rays, photographs, making impressions, models, splints and fabricating prostheses. In addition, the education and training of new practitioners is simpler due to virtual patient programs, dental software, testing devices, audiovisual aids and so on. Digitization permits the use of many new and more effective methods in the development of stomatology.

4. Discussion

This paper summarizes selected aspects of the use of AR in dentistry with a wide range of applications during the analysis of dental bodies, during surgical navigation, restorative dentistry, and orthodontics. It also presents the use of selected multidimensional signal processing methods and different computational intelligence tools in this area. Special attention is paid to a brief description of scanning technologies and an intra-oral 3D scanner used for dental data acquisition and their 3D modelling in the selected coordinate system.

The results present the possibility of using a spatial model for semiautomatic detection of specific parts of dental objects and to evaluate dental arch parameters during the treatment of dental disorders. They also demonstrate how the specification of regions of interest for individual teeth on the 3D surface plot can be improved by the contour lines of the surface, together with their selected density. The results also show that contour plots allow more precise analysis of the shape of the individual teeth.

The separation of individual objects of dental arches points to the possibility of dental object description, which will allow their mathematical analysis and evaluation of dental arch parameters. Selected digital models of dental objects can be used for their 3D printing and physical modelling.

Using AR provides the dental patients more freedom and comfort to navigate around their treatment plans and choices. It also enables a better interaction of patients with dentists to select and implement their optimum treatment.

Future studies related to digitization in stomatology will be probably devoted to the use of digital models for more detailed mathematical evaluation of dental parameters and to classify dental disorders using a set of selected dental features. Specific studies might be further devoted to 3D printing of individual teeth and to the analysis of appropriate printing materials. It seems at the moment that these 3D physical prints can form an alternative to classical dental replacements in the near future.

5. Conclusions

Aesthetic dentistry and implants are very promising AR-based research areas. Using AR, it is possible to point the camera around and see how the implant would look in different positions and locations in the patient’s mouth. In this way, AR can seamlessly be assimilated into existing technology today. AR allows us to overlay information in the real world. In the case of learning dental restoration, a practitioner might point a device at the mouth and overlay it with video or other information. AR would then provide them with additional information to interact with the machine in the real world.

Applications of AR in stomatology cover both digital modelling of spatial objects and analysis of signals acquired by different sensors including data obtained by the diffuse reflectance spectroscopy. Machine learning and selected classification and probabilistic methods can be then applied for detail processing of specific object features.

It seems that the future development of the use of AR in stomatology will be closely related to the development of associated visualization methods and parallel investigation of objects in the real world and computational environment. Sophisticated registration methods will also be applied in the comparative mode to adapt mathematical models to changing external conditions.