Serious Games and Mixed Reality Applications for Healthcare

1. Introduction

Serious games are games in which the main goal is not entertainment, but a serious purpose ranging from the acquisition of knowledge to interactive training, to name just a few. These games are emerging as a promising educational technique across various domains (such as the military, education, politics, management, and engineering) and are attracting growing attention in healthcare thanks to the possibility of: increasing the level of interactivity of the application and the motivation of the user, allowing adaptation to the user’s level of competence, flexibility over time, repeatability, and continuous feedback.

In recent years, there has been a growing interest in applying virtual reality (VR) techniques to the development of serious games for creating immersive experiences. VR and augmented reality (AR) have a long history in the healthcare sector, offering the opportunity to develop a wide range of tools and applications aiming at improving the quality of care and efficiency of services for professionals and patients alike.

The best-known examples of VR/AR applications in the healthcare domain include surgical planning and medical training by means of simulation technologies. The techniques used in surgical simulation have also been applied to cognitive and motor rehabilitation, pain management, and patient and professional education.

Recently, healthcare has also become one of the biggest adopters of mixed reality (MR), which merges real and VR content to generate new environments where physical and digital objects not only co-exist but are also capable of interacting with each other in real time. MR encompasses both VR and AR applications.

These novel applications are attracting growing attention from users and researchers, but many cognitive and perceptual issues still need to be completely understood and resolved to fully take advantage of the disruptive potential of these emerging technologies in healthcare, and to minimize side effects of VR/AR technologies such as “cybersickness”. Moreover, efforts should be made to strengthen the experience of “presence”, the level of acceptance, and the compatibility of MR technology with the general population (including the elderly and sick people).

In light of the above, this Special Issue was introduced to gather and publish original scientific contributions that explore opportunities and address the challenges of serious games, VR/AR, and MR applications in healthcare. There were 11 papers submitted to this Special Issue, and 8 papers were accepted.

2. Serious Games in Healthcare

Various topics have been addressed in this Special Issue, mainly focused on the description of innovative serious games with applications in the field of rehabilitation/training of elderly people or patients with degenerative diseases [1,2,3,4,5], and training for self-regulation of mental states [6].

The collected papers cover some of the key points in the development of serious games, namely: development of interactive and engaging tasks to increase user motivation, acquisition and analysis of quantitative data, and adaptation to the user’s conditions.

In Sipatchin et al. [4], VR is proposed as a new assisting tool for patients suffering from macular degeneration. More specifically, a VR game that involves tracking and detecting changes in a moving object is proposed to re-define traditional tasks used for preferred retinal locus development studies and therapy, which can be repetitive, exhaustive, and tedious, ending in a decrease in the subject’s motivation. Gamification, in this context, was intended to develop more entertaining tasks, resulting in greater participant engagement and rapid adaptation.

One of the possibilities offered by serious games for rehabilitation/training is the collection and analysis of quantitative data for different purposes, such as: to assess the initial disease state and/or exercise performance, and to monitor the patient’s progress during therapy, just to name a few.

Three papers of this collection [1,3,5] focus on collecting and analyzing quantitative data, and two of them [1,3] employ artificial intelligence (AI) techniques. AI is becoming an important major asset for serious games to provide more facilities for the learner, but also more knowledge for the supervisor (educator, medical specialist, etc.) [7].

Shapoval et al. [3] describe the integration of biofeedback into a serious game to monitor frailty in an elderly population. The system is suitable for everyday use at home, but also for more in-depth observation of various biological parameters, such as the heart rate and temperature. It features a real-time tracking system of the body position, relying on the use of a webcam and a built-in motion recognition algorithm based on a neural network, for controlling the game and for analyzing the patient’s activity. Acquired data may be subject to medical secrecy rules and will not be available to anyone other than the treating physician and the patient to monitor the outcome and progress of the rehabilitation process for different degrees and types of musculoskeletal disorders. All the data obtained can be visualized and examined at any time, thanks to a cloud interface where all the information is duplicated.

Karapapas et al. [1] propose the use of a serious game for early detection of mild cognitive impairment (MCI), which is an indicative precursor of Alzheimer’s disease. More specifically, their work intends to test whether performance data collected during sessions of a serious game, specifically conceived for cognitive assessment and training of elderly people, can be used to create machine learning (ML) models to classify the patient’s cognitive status.

Investigating the deterioration of cognitive functions is also the goal of the research work by Chen et al. [5]. Their serious game was designed in collaboration with psychologists and clinicians to be as simple as possible, given that the target patients are elderly and that most of them have no experience in playing video games. The aim of the game is to measure the subjects’ performance, in terms of the reaction time and accuracy rate in reaching a target, to classify their cognitive abilities and distinguish normal subjects from MCI subjects and subjects with dementia.

Traditional sensing devices employed for serious games in rehabilitation include optical and inertial sensors. According to [2], the Microsoft Kinect (a 3D sensor based on time-of-flight technology and the intensity modulation technique [8]) is usually the preferred choice to track full-body movements, although accuracy drops when complex movements are executed. In Shapoval et al. [3], the main sensor required is a simple USB webcam: this makes the platform more affordable from the financial standpoint, and easy to install and use.

One work in this Special Issue by Guimarães et al. [2] employs two inertial sensors (one for each user foot) to simplify the setup and reduce costs. This serious game is designed to support personalized and multicomponent clinical interventions for older adults, and, more specifically, the goal is to train strength, balance, cognition, and the pelvic floor muscle, in one single session. In this work, reducing the system complexity was preferred over the use of additional sensors for full-body movement evaluation.

Recent studies have also integrated biosensors into games designed for physical rehabilitation and for the treatment of cognitive and neurological disorders. Biosensors indeed can be integrated into games to study the biological data of the participants during play as well as making games adaptive based on information extracted from the patient’s bio-signals [9].

In this Special Issue, da Costa et al. [6] hypothesized the possibility of developing a “Neurofeedback assisted self-regulation machine” combining the technical, behavioral, psychological, emotional, and electrophysiological components of brain–computer interfaces (BCIs) based on electroencephalography (EEG), neurofeedback training (NFT), mindfulness meditation, and self-regulation of mental states in a single framework. The work investigates “how” a priming intervention right before NFT affects NFT performance and the emotional state, acquired using qualitative emotional state self-reports and quantitative emotional state biomarkers (i.e., galvanic skin response and heart rate variability). The proposed framework could potentially improve the efficacy of self-regulation serious games for therapeutic, performance, or entertainment purposes.

3. Extended Reality Applications and Hybrid Simulation in Healthcare

Among all the papers accepted in this Special Issue, three focused on extended reality (XR) applications and hybrid simulation for the healthcare sector [4,10,11].

Each one of these papers investigated a different aspect of the application of these technologies (VR, AR, MR, and hybrid simulation) in medicine, in particular: introducing VR and gamification in traditional patient treatment [4], using AR and hybrid simulation in medical training [10], and reviewing the current state of the art of AR/MR applications in healthcare simulation [11].

In [4], Sipatchin et al. presented a VR serious game for patients suffering from macular degeneration. This VR application includes both eye tracking of the user and motion tracking of a moving object, and its goal is to modernize conventional therapies for preferred retinal locus development. This novel VR game, by using gamification techniques, can improve patient engagement, motivation, and adaptation.

A hybrid simulator for the training of pedicle screw fixation in the spine was proposed in [10]. In this paper, Condino et al. describe a system for the preoperative planning and rehearsal of spine surgery, with the goal of improving both surgical workflows and postoperative patient outcomes. This hybrid simulation platform combines a 3D printed patient-specific spine model with AR functionalities and virtual X-ray visualization, obviating the need for any harmful radiation during the medical simulation.

Viglialoro et al. [11], in their review paper, summarized and discussed different medical simulators based on AR, MR, and hybrid approaches, analyzing their evaluation and validation as well. This work, by highlighting the drawbacks and advantages of each application, provides some guidelines for developing novel healthcare simulators and also outlines promising research trends in this field for the coming years.