Intraoperative Use of Mixed Reality Technology in Median Neck and Branchial Cyst Excision

Abstract

The paper considers the possibilities, prospects, and drawbacks of the mixed reality (MR) technology application using mixed reality smartglasses Microsoft HoloLens 2. The main challenge was to find and develop an approach that would allow surgeons to conduct operations using mixed reality on a large scale, reducing the preparation time required for the procedure and without having to create custom solutions for each patient. Research was conducted in three clinical cases: two median neck and one branchial cyst excisions. In each case, we applied a unique approach of hologram positioning in space based on mixed reality markers. As a result, we listed a series of positive and negative aspects related to MR surgery, along with proposed solutions for using MR in surgery on a daily basis.

1. Introduction

The existing methods of instrumental diagnostics in the preoperative phase (ultrasound, MSCT, and MRI) allow a rather accurate detection of topological peculiarities of the median neck and branchial cysts [1,2]. However, the intraoperative image, with the change in the patient’s positioning on the operating table, can considerably differ from the examination results. This leads to difficulties when compared to the images in the surgeon’s mind. Therefore, there is a need for a solution that is based on augmented reality technology and overlays MRI or MSCT data on top of the patient during surgery procedures. This would provide surgeons a visualization of the interrelation between the cystic mass and the great vessels, as well as the interconnection of all neck organs; this would also allow the surgeons to rule out any abnormal changes in them. However, the classical approach of augmented reality is not enough because the final image is formed on a flat screen, which does not allow the surgeon to accurately determine the position of anatomical structures in 3D space. Thus, it is necessary to use wearable glasses that include a stereo display and the use of mixed reality technology [3].

The appearance of the Microsoft HoloLens augmented reality glasses on the market has revolutionized many areas, including medicine [4,5]. This has opened up opportunities for more effective diagnostics and intraoperative treatment through the visualization of various types of data. Now, Microsoft HoloLens is widely used in research planning [6], training [7,8], and in performing surgical operations [9] using mixed reality technology.

If we consider the application of this technology directly in the operating room, the existing solutions can be divided into two categories:

• Medical Data Visualization Systems;
• Navigation Technologies through the Combination of Medical Data with the Patient during the Operation.

In the first case, the main idea is to visualize CT/MRI data as well as information about the patient’s condition through a series of virtual screens, which eliminates the need to use bulky racks with multiple monitors, therefore optimizing the space in the operating room [10]. In the second case, mixed reality glasses are used to compare a virtual model of the patient’s anatomy, which was built based on MRI and CT data, with the real condition of the patient. This is achieved by superimposing these data onto the patient’s body and creating the effect of an “X-ray vision” or a visualization at a separate point in space, where the patient’s data is compared with the current situation during surgery in the form of a freestanding holographic model [11,12].

In the case of using mixed reality technology as a navigation system, one of the main tasks is the patient’s registration (a comparison of virtual MRI data/CT scan with the patient’s position in the operating room). To do this, it is necessary to use spatial positioning systems, which can be marker-based (reference markers) [13] or marker-free (markerless); these systems are based on optical tracking technologies [14], magnetic induction sensors, or 3D scanning [15]. Systems using optical tracking, where infrared cameras are used as tracking devices, have proven themselves well and are actively used in brain surgeries (Medtronics, Brainlab), while providing a positioning accuracy of 1–2 mm. However, these systems have a significant drawback—the markers must always be in the field of view of the recording cameras, which complicates the operation process when several doctors working with the patient block the field of view of the device. Technologies using magnetic induction sensors do not have this disadvantage and offer even higher positioning accuracy, but they are difficult to implement and have a limited scope of work. The approach of performing a 3D scanning of the patient’s body [16] using MR glasses with a built-in depth sensor during a real operation seems almost impossible to use because the patient’s body is often covered with cloth or additionally blocked by other life-supporting systems (e.g., catheter, breathing tube).

Our solution is based on the use of reference AR markers as the easiest way to implement MR technology, which simultaneously provides sufficient accuracy [17]. At the same time, the main problem that this study is aimed at is the search for and the development of an approach that will allow surgeons to perform operations using mixed reality on a daily basis with a large number of patients, thus reducing the time spent on preparing for surgery. Therefore, in this paper, solutions are proposed to simplify the process of preparation—of the patient and for the patient’s registration.

2. Materials and Methods

Here we present 3 clinical cases: 2 median neck cysts in female patients and 1 branchial cyst in a male patient.

All approaches listed below are based on a marker system that uses special AR markers [18] as reference of a 3D model to a certain point in space. These markers have a specific texture that the software can recognize by means of computer vision algorithms; this allows the software to identify its location and orientation relative to the observer. The overall system flowchart is illustrated in Figure 1.

• The main aspects of application for this approach: The markers themselves are made of PLA plastic, while the image-texture coating is applied by means of UV print, which allows for the sterilization of the markers before their application in the operating room (Patent [19]);
• We transformed the CT into a 3D model using a segmentation approach with respect to the set density threshold or the area of interest in the 3D Slicer. This allowed us to isolate all key anatomical structures of the neck as a series of separate three-dimensional models. After that, resulting 3D models were optimized and cleaned up using the procedural toolkit in Houdini (up to 75–95% polygon reduction); we then transformed the MSCT into a 3D model using a segmentation approach with respect to the set density threshold or the area of interest. This allowed us to isolate all key anatomical structures of the neck separately as a series of three-dimensional models;
• We visualized 3D models using mixed reality smartglasses and the developed software, which has preset parameters for referencing the markers to the three-dimensional model. It has an integrated basic interface [20], which allows the user to display the required anatomical elements as well as customize the parameters of the marker tracking system;
• The 3D models were transferred to the HoloLens glasses using TCP via the Wi-Fi network, which alleviated time-consuming app compilations for each procedure.

The approach to marker tracking is based on a combination of the method of analyzing the camera image on the glasses and the spatial mapping system [21], which supports the further positioning of the 3D model using the environment mapping data in case the marker leaves the field of view.

Subjects gave their written informed consent, and the study protocol was not in conflict with the institute’s committee of ethics. The developed approaches of the current study did not impose any additional risks to patients or surgeons because they were not subject to any specific procedure, nor were they required to follow any new rules of behavior. Diagnostics and surgery were performed without any deviation from the standard procedure. All decisions of the surgeons were based on the patient’s medical conditions as well as on the analysis of the obtained CT and MRI data.

3. Results

3.1. Case 1. Fixation Mask

Patient M. was diagnosed with a median neck cyst. The patient underwent MSCT with intravenous contrast enhancement and cystography (Figure 2). The MSCT was performed in the patient’s selected position (the selected position is the uniform position of the patient during examination and surgery; in this example, it is the position in the CT scanner and on the operating table) on her back, in several stages. We used Q-fix (1 and 2) fixators produced in the USA in the form of a perforated thermoplastic cervicofacial mask-frame, with openings for operative access and intubation and designed for radiation therapy.

The examination revealed for the first time that the lump was 5 cm long and consisted of a cyst with a double-blind fistula descending from the cyst mass down to the lower edge of the isthmus of the thyroid gland (Figure 3).

For the preoperative preparation and the subsequent intraoperative hologram application, we used the 3D Slicer application to perform image segmentation and to build a 3D model of bone structures, flows, internal and external jugular vein, common and external carotid artery, larynx and tracheal cartilages, thyroid gland, and the cyst along with the fistula (Figure 3).

Developed Approach

For this approach, we used a Q-fix fixation mask, which allowed us to fix the head and pectoral arch in a certain position, with a possibility to replicate this position in the subsequent surgery [22]. Owing to the fact that the frame holding the mask remained unchanged (the masks were replaced), this marker could then be used multiple times in other surgeries involving this method.

The surgery was performed in the patient’s selected position—on her back, identical to the position during the MSCT (Figure 4).

As a result, the fixation mask helped to ensure the patient’s position on the operating table, identical to that during the MSCT examination. However, the main disadvantage of this method lies in the need for the production of individual masks for each surgery and their installment during the preoperative process. This requires considerable time and labor costs.

3.2. Case 2. Adjustable Navigation Frame

Patient B. was diagnosed with a median neck cyst. During inpatient preoperative treatment in the Maxillofacial Surgery Department, the patient underwent MSCT of the neck, with intravenous contrast enhancement and cystography (Figure 5).

The MSCT of the neck was performed in the patient’s selected position, similar to the intraoperative one, with a full extension of the cervical spine, using a facial frame made of polyamide and a PLA reference marker.

Operating Principle and Conclusion

This solution entailed the use of a special adjustable frame (Patent [23]), which was attached to the patient’s head and adjusted according to the head parameters (Figure 6). It was applied during the CT and was later used again with the customized parameters in the operating room. As a result, the patient’s head was positioned in the same way as during the CT. The hologram positioning was realized by means of a marker installed on the frame (Figure 8). The frame was 3D-printed from polyamide; both the frame and the marker were sterilized.

Initially, the frame was attached to the auricles via metal screws and nuts that generated severe artifacts in the MSCT imaging. This led to a replacement of these parts with polyamide elements. Thus, radiopaque markers were the only metal components of the frame.

This approach included not a manual but a semi-automatic adjustment (matching the initial frame coordinates in the virtual space with the CT coordinate system). The frame hade built-in radiopaque markers made of radiodense material in the form of 2 mm × 2 mm metal adjusting screws. These markers allowed the system to match two coordinate systems (Figure 7).

The advantage this method has over the previous one is that it consists of smaller dimensions and that there is no need to produce individual fixation frameworks (Figure 8). In addition, this solution allows for a significant reduction of patient positioning time. Nonetheless, it still requires some preparations to attach the frame to the patient’s head before the surgery.

3.3. Case 3. Adjustable Navigation Frame

In May of 2021, Patient K., 25 years old, was seeking medical advice in a maxillofacial surgery clinic and was diagnosed with branchial cyst. During inpatient preoperative treatment, the patient underwent MSCT of the neck, with intravenous contrast enhancement and cystography.

The MSCT of the neck was performed in the patient’s selected position, on the back with the head turned all the way to the left (to the healthy side).

Similar to Case 1, Case 3 included a three-step MSCT protocol, with cystography and intravenous contrast enhancement (Figure 9).

Operating Principle and Conclusion

Due to the strictly specific lateral positioning of the patient, the use of the navigation frame from Case 2 was impossible.

The developed anatomical navigation splint approach [24] allowed for the attachment of a splint with anatomical landmarks to any of the patient’s body parts by means of an impression compound applied next to the area of interest prior to CT. In this case, the ear was chosen as a relatively immobile body part with a complex structure.

The splint itself consisted of two parts: a reference frame and a fixing handle (Figure 10). The fixing handle had a cross-shaped base firmly installed into the impression compound. It also contained thin metal rods (10 mm × 0.5 mm × 0.5 mm), which served as radiopaque markers, allowing for further calibration of the coordinate systems (similar to the previous case).

The key advantage of this method is the possibility to reference the splint to any part of the patient’s body, as well as its quick installation. However, the chosen body part needs to have a complex anatomical shape and rigid bone structure.

4. Conclusions

Overall, these proposed solutions can be used for treatments using MR technology on a daily basis. However, the most effective solutions were the radiopaque marks used (in the second and third cases), which allowed us to automate the calibration process. In both cases, the additional time spent on patient preparation was no more than 10 min (excluding the 3D segmentation process). On the other hand, in the first case where face mask was used, the total time for preparation was about an hour due to difficulties with the patient’s positioning.

Resulting hologram positioning accuracy depends on:

• Marker-based 3D visual localization [17]. Translation error from 1 m distance is about 2–3 mm;
• Manufacturing accuracy of fabricated marker holder. For FDM technology, this is about 0.2 mm;
• CT data slice thickness. In our case, all datasets have 1 mm thickness;
• Marker placement.

In the first case, the overall error was approximately 7–15 mm due to our inability to precisely measure the mask’s complex shape. On the other hand, in the second and third cases, we used our own marker holders, so the total error decreased for both cases down to 3–7 mm. Thus, the most significant amount of error was caused by the marker placement. For future development, we are planning to focus on a more accurate marker positioning system that uses multiple smaller markers.

In addition, based on these cases, we compiled a list of benefits and drawbacks for using MR technologies in the operating room:

• The MR technology can be applied in the operating room as an additional auxiliary technique to assess the adequacy of surgical exposure as well as for intraoperative navigation;
• The bloodstream visualization eliminates the risk of great arterial and venous trunks intersection. Additional ligation is almost completely excluded;
• At the current stage of technological development, there is a competition between the brightness of the operating room lighting and the brightness of the hologram, which makes their simultaneous use impossible. Therefore, the MR technology can be implemented for periodic navigation of the wound with the lights dimmed as well as for postoperative follow-up of cysts and fistula excision;
• The direct use of mixed reality technology during surgical manipulations based on the problem of “focal rivalry” [25] is difficult. Therefore, glasses were used only in the process of planning and marking access, as well as in checking the results with the preoperative state.
• Despite the rigid fixation of the patient in Case 1, Cases 2 and 3, which involved no rigid immobilization of the patient relative to the operating table, demonstrated the best accuracy for hologram referencing. We attribute the obtained results to the variants of the reference marker’s attachment to the patients’ body parts and its relation to the surgery area. The closer they were to each other, the more accurately the hologram could reference to the real landmarks.