Toward Precision Rehabilitation for Neurological Diseases: Data-Driven Approach to Exergame Personalization ^†

Abstract

Physical rehabilitation of patients affected by neurological diseases is currently an unmet clinical need due to the high cost of health systems and the impact that rehabilitation has on patients and their families. This paper introduces an approach based on the idea of precision rehabilitation, where personalization of the physical rehabilitation for each patient and accessibility to technological tools that support it are pursued. A general architecture that contemplates functional modules related to multiple work areas and different roles is proposed. One of these modules is related to gamification and delves into the design and development of exergames, defined from a general data model, in order to increase the level of patient motivation when exercising. The paper discusses the creation and evaluation of an exergame designed for the rehabilitation of lower limbs.

1. Introduction

Tertiary digital prevention aims to mitigate, through the use of technology, the impact that a disease or permanent injury has on people. Depending on each particular case, the final objective can address goals such as increasing the patient’s level of autonomy, extending their life expectancy, or simply improving their life quality [1].

A particular area of tertiary digital prevention is linked to neurological diseases, since people who suffer from them need physical rehabilitation to a greater or lesser extent. An example is that of stroke survivors needing rehabilitation, whose number in Europe has risen to several million, and it is expected to grow in the next few years. In economic terms, and according to the European Brain Council, the estimated cost for European health systems is around 800 billion euros per year [2], while from the perspective of patients and families that face this process, the impact that this situation has on them is very high. By way of example, in Spain the cost of patients treated by units specialized in stroke is estimated at 27,711 euro per year [3]. These problems can be extrapolated to the global level [4], thereby clearly generating a clinical need that has not yet been met.

Traditionally, technological solutions for rehabilitation have been designed to increase patients’ level of autonomy, so that they can perform rehabilitation exercises at home [5,6]. In contexts where human-resource limitations mean that moving patients/therapists poses a limitation, technology improves accessibility to this type of tools [7]. However, and in order to maximize chances of success of this type of approach, such tools or services must be capable of motivating the patient to use them [8] and, at the same time, of adapting to each patient’s evolution.

With respect to the clinicians’ role, typically related to therapists, they must play a major part in devising technological solutions that support the process of physical rehabilitation. To this end, it is essential to consider, together with patients, a cocreative approach based on participation when designing telemedicine tools. On the one hand, patients need to feel comfortable when correctly using technology, seeing an incentive in it to keep on progressing in the rehabilitation process. On the other hand, the design of this technology has to involve, from the very beginning, medical personnel who eventually supervise and monitor patients’ evolution, considering the needs of each person at any given time.

In this work, we propose the design of an architecture for precision rehabilitation, whose ultimate goal is to benefit from both current hardware technology and existing computational models to provide personalized rehabilitation, individually linked to each patient. The concept of precision rehabilitation borrows ideas from precision medicine [9], a modern approach that has as its ultimate goal the personalization of medical assistance. This level of customization is mainly supported by the application of artificial-intelligence techniques, typically deep learning [10] and Big Data [11], which enable the extraction of patterns that represent the best therapy for a patient.

This paper introduces such an approach in the area of physical rehabilitation of patients affected by neurological diseases to improve the medical care they receive, and to reduce costs faced by health services. Thus, a general architecture consisting of a series of modules is proposed, so that physical rehabilitation can eventually be adapted to each patient. From a general point of view, the architecture contemplates the integration of components from various work areas: (i) artificial intelligence to select the best rehabilitation treatment for each patient, (ii) motivational techniques related to the psychosocial dimension to maintain the patient’s level of commitment, (iii) advanced human–computer interaction systems to increase the patient’s level of autonomy, and (iv) analytics and telemedicine to facilitate the therapist’s work and improve the patient’s quality of life.

Of all the modules proposed in the architecture, the one relating to gamification was developed in this work within a motivational context to encourage patients when making rehabilitation routines. More particularly, this module delves into in the definition of personalized exergames, defined by means of a data model that is introduced in this work. This data model contemplates the definition of three main components: (i) the game scene, which contains the views of tutorial, participation, and results; (ii) the set of actors, which contemplates the elements of the game that implement behaviors, such as the patient’s virtual avatar; and (iii) gameplay, which defines the set of actions that the user has to perform in order to achieve the objectives of the exergame. On the other hand, exergames are stored in a centralized repository so that patients can download them to their remote rehabilitation devices, based on possible prior recommendations from the therapist or suggestions automatically made by the system.

In order to validate the gamification module, we developed an exergame oriented to the physical rehabilitation of lower limbs, which follows conventions established by such a module according to the designed data model. The motivational part of this exergame lies in inviting patients to perform exercises by means of game mechanics that simulates traditional grape-treading. This activity, still carried out manually in regions with winemaking tradition such as in Mediterranean countries, can serve as an example of a recreational activity through which patients perform rehabilitation exercises in an enjoyable way.

The rest of the article is structured as follows. Section 2 discusses important aspects and assumptions of the problems addressed in this work, synthesizing the characteristics of the physical environment in which precision rehabilitation is addressed. Next, Section 3 introduces the proposed architecture, describing the modules that compose it. Section 4 delves into the gamification module and, in particular, into the definition of personalized exergames by means of a data model explicitly created for that. Subsequently, Section 5 discusses an exergame example developed using the data model to perform physical rehabilitation exercises. Finally, Section 6 highlights the main conclusions and devises lines of future work.

2. Problem Statement

The idea of precision rehabilitation inherently entails a direct relationship with the use of technology for treating each patient’s rehabilitation process in a personalized manner. This customization links to a scenario in which most rehabilitation is remotely done by the patient, typically at home, and provided that the patient’s medical conditions or cognitive abilities allow it. In addition, this approach is suitable for contexts where the limitation of human resources is a problem, or moving patients/therapists poses serious restrictions.

Next, the physical environment in which a patient would carry out the rehabilitation routines within the framework of precision rehabilitation is characterized. This environment is graphically recreated in Figure 1.

• Patient has a physical space at home, typically the living-room, in which they carry out rehabilitation exercises.
• The patient has a television to which they can connect, via an HDMI port, a stand-alone device that runs the remote rehabilitation system without the need for a personal computer.
• The device that runs the system is capable of performing the patient’s skeleton tracking and has enough processing capacity to computationally perform simple tasks and give visual feedback through the TV itself.
• There is at least a distance of 1.5 m between device and patient.
• The device is permanently connected to the Internet for downloading periodic updates.
• The therapist can remotely monitor the patient’s progress using a tablet or computer.

We assume that, as far as possible and depending on the type of rehabilitation, it is important to use nonintrusive devices from the point of view of patient use. By way of example, and if it is necessary to detect how the patient moves an arm in a rehabilitation exercise, said detection could be carried out by means of depth cameras and without the patient having to include it in their clothing wearables or colored elastic bands that are easy to detect by cameras. However, if the type of rehabilitation requires precision, for example, finger or hand movements, or if it is necessary to monitor the patient’s condition, then the use of devices that require more intrusive deployment can be considered.

On the other hand, it is assumed that the patient does not have a computing device, such as a PC or a laptop computer, that can be exclusively dedicated to rehabilitation at home. For this reason, we aimed for solutions on the basis of devices that have limited computing capacity, typically to give visual feedback and to provide some basic computation capabilities, but that have an Internet connection to delegate computations to external servers. In this sense, this approach is linked to the paradigm of fog computing [12].

Taken together, these assumptions define a realistic environment in which rehabilitation is carried out from the patient’s home with the aim of increasing autonomy but, at the same time, with a direct link to the rehabilitation system and with the therapists themselves to appropriately guide and adapt the rehabilitation process.

3. Architecture for Precision Rehabilitation

Figure 2 graphically shows the general architecture proposed for precision rehabilitation. This architecture is composed of a series of modules that contemplate functionality from different perspectives: (i) collecting and processing data from multiple sources, (ii) intelligent data analysis for the definition of rehabilitation routines, (iii) integration of human–computer interaction mechanisms, (iv) motivation of patients to increase their level of commitment with the rehabilitation process, and (v) analytics and information visualization so that medical personnel can adequately monitor the rehabilitation.

Sources of information involve offline data, which make up the patient’s clinical history, and online data, which are obtained in real time by means of wearables and sensors in rehabilitation sessions. With respect to offline data, special consideration is given to those linked to the to patient’s medical reports and to records generated in the rehabilitation sessions. The latter typically contain information that can be automatically processed, such as numerical data on a scale that measures the patient’s progress, but also include textual information that requires more complex treatment. With regard to online data, they regard the use of devices that allow patient-skeleton tracking by using depth cameras and wearables to monitor a patient’s condition in terms of heart rate or body temperature, among others, to measure effort and adaptation to the rehabilitation routine. At a general level, a design based on Lambda architecture [13] to manage the data is proposed. On the other hand, data management has to be carried out in such a way that these are anonymized, encrypted, and stored in a secure database.

The artificial-intelligence module learns from data stored in this repository and generates the necessary information to define and personalize rehabilitation sessions according to each patient’s profile. The use of supervised learning [14] is proposed to classify patients depending on how they perform rehabilitation routines and how they progress, together with the integration of a recommendation system [15] that allows to study patients and to display the exercises they carried out or that had significant impact on their evolution.

The module related to patients’ motivation through gamification is dealt with in detail in Section 4. Basically, a scheme is proposed where it is possible to generate personalized exergames for each patient from a data model containing the elements, actors, and game mechanics that make up each game. In this way, it is possible to visualize a scenario in which both patients and therapists have a catalog of available exergames at their disposal.

On the other hand, the computer–human interaction-mechanism module represents the communication channel between patient and rehabilitation system. Depending on a patient’s capabilities and the type of rehabilitation to be carried out, the architecture contemplates aspects that range from the integration of devices that simply provide visual feedback to the use of more advanced schemes based on mixed reality and smart robotics.

Finally, the analytics module is related to the therapist’s role. In this sense, we worked on increasing quality time that therapists dedicate to patients, so that the architecture enables mechanisms to increase patient autonomy in terms of remote rehabilitation. This module contemplates the integration of visualization and interaction mechanisms that allow the adequate monitoring of a patient’s progress, and the possibility of assigning and adjusting the recommended routines by the rehabilitation system.

4. Gamification Module

The gamification module establishes a functional contract between system and exergame designers, allowing them to define new exergames from data models by means of data-driven development [16]. Thus, the gamification module consumes these data models to define the properties and the behavior of the exergames. These data files, defined by using a JSON format, contain all needed information to generate a new exergame or configure an already existing one in the system according to a patient’s needs.

Exergames defined by these data models are hosted in a centralized repository, accessible from the remote rehabilitation system. The user can access this repository or catalog from the menus of the rehabilitation tool, and select the exergame that they want to execute so that the system downloads it to the device. If the user is a therapist, they could also add a selection of exergames to the rehabilitation routine of some of their patients, notifying them about new changes in the routine that would take place during the next rehabilitation session.

When defining new exergames, it is necessary to take into account the elements that really define their nature within the context of this work. These refer to (i) scene, (ii) actors, and (iii) game mechanics.

• Scene. The main scene of the exergame is obligatorily defined according to the contract exposed by the module, by three views that follow one another in order. The first of these, tutorial view, shows the patient an animation with the guide of the exergame as in a tutorial. The second, participation view, allows the patient to directly participate in the exergame for as many times/repetitions as the therapist indicated considering the patient’s routine. Finally, the third, results view, is responsible for displaying the scores obtained by the patient after finishing all the repetitions in the exergame.
• Actors. It groups all game elements that have any kind of behavior, such as playing animations or performing translations in the 3D space. Each exergame has at least one actor: the avatar that replicates the patient’s actions during the game.
• Gameplay. It defines the set of actions that the user must perform to complete a repetition of the exergame. The correct execution of the game mechanics increases the score obtained by the patient during each repetition.

For practical purposes, these three elements are defined in the data model using multiple exchange files that refer to each other. They are governed by a single specification file with exergame metainformation (see Figure 3). This last file maintains references to all other data files that define the exergame. In the same way, it also contains other properties, such as a unique identifier of the exergame, the title of the exergame, the author, and the download price (if it is not free of charge). This file also declares whether the exergame depends on some already existing in the repository, with the objective that the author can reuse other exergames to extend or modify them. It should be noted that an exergame can only depend on another provided that the latter belongs to the same as the former or it is available for free.

Listing 1 shows an example of the content of this file. In this case, the configured exergame would be built on an existing one identified as ‘john-doe/another-exergame’. Thus, the data models would overwrite or add new properties to the exergame, depending on whether they are already defined or not in the base exergame.

Listing 1: Example of master file defining an exergame.

In the case of scenes, a file is kept in the JSON format that references each of the three views that make up the exergame. Thus, each view is defined by a specification file in the GL Transmission Format (glTF) [17] maintained in a JSON structure that includes, in a self-contained way, all the resources of a scene internally encoded using URIs of base-64 data (i.e., a unique .gltf file) or externally referenced (i.e., a .gltf file and another .bin).

Likewise, actors would be stored in another JSON file using a name that must match that of the object node defined in the scene file. The actor-definition file also maintains other properties relating to each actor of the exergame such as, for example, the list of the patient’s joints to which it reacts to interact and whether it takes any action in that case. The supported actions are exposed by the gamification module through several internal functions that allow to manage the state of the exergame. In addition, each action accepts a number of specific properties that affect the execution of the exergame.

Finally, the game mechanics of the exergame is defined by a sequence of actors with whom the patient must interact in order to complete a repetition of the exercise. This data model also allows to define the the effect of these interactions at the actor level, with the existing possibilities of (i) touching the actor or (ii) staying for a certain period of time on the actor. The number of actors who are part of the game mechanics define the precision with which the execution of the repetition is assessed. The lower the number of involved actors in the sequence for a joint are, the less constrained the movement execution for that joint is, and vice versa.

Figure 4 shows an example of a possible exergame in which the patient is invited to clean a window by replicating the established movement by the actors of the game mechanics. In the screenshot, the exergame would find itself executing the view in which the patient directly participates in the repetition, specifically in (a) the fourth of a total of five assigned. (b) Obtained score is shown in this case during the participation view without the patient having to wait to complete the exergame and progress to the results view. (c) The avatar would replicate movements made by the patient by means of direct association between patient’s joints and 3D virtual avatar. By highlighting the joints involved in the movement, that is, (e) the avatar’s right hand in green, the patient is informed that the movement must be carried out with that hand and (d) that the interaction must be made with actors of the color that establishes the game mechanics in the defined order. (f) Those actors of the game mechanics surrounded by an orange circle would indicate to the patient that they have not yet interacted with them. Finally, (g) the scene view also defines some decorative and continuously animated elements.

Similarly, Figure 5 shows content fragments of the data models of the proposed exergame. Thus, the relations of the sequence-actor-1 and sequence-actor-2 actors can be observed throughout the three files that constitute the exergame. In the case of the file that defines the gameplay, ’touch’ mode can be set for sequence-actor-1, indicating that the associated action would simply be executed when the user touches that actor with some joint. On the other hand, sequence-actor-2 defines its interaction in ’keep’ mode, indicating in this case that the action is executed when the user is in contact with that actor by means of some joint for a duration specified in the ’duration’ field (i.e., 2000 milliseconds).

It should be noted that information in this JSON file is declared in the form of a list with the aim of maintaining the order in which the patient must interact with actors. In the case of the file that defines actor behavior, actor sequence-actor-1 would indicate the list of joints to which one would react and the actions that would take for that reaction. In this case, the joint associated with the right hand executes the action of increasing the score (i.e., increase-score) in the amount indicated by the properties of the object params. Finally, the file that defines the scene (in the case of the figure, participation view) would maintain the hierarchy of the elements that form the scene in glTF format, indicating properties such as the transformation of objects (i.e., translation, rotation, and scale), the associated animations, or references to binary data.

5. Experiment Results and Evaluation

5.1. Exergame for Lower-Limb Physical Rehabilitation

The design proposed in the gamification module allows to integrate different types of exergames, validating the system as a flexible environment for physical rehabilitation through games. These games can help during the rehabilitation of patients who suffer from certain conditions caused by neurological disorders by means of exercises performed in relaxed environments that mimic real-world actions. Specifically, and taking advantage of functionality provided by the gamification module, the development of an exergame that motivates patients for lower-limb physical rehabilitation has been addressed. The context of this exergame is winemaking by means of traditional grape treading. This exergame was stored in the remote exergame repository under the name ‘Tread the Grape!’.

A preliminary version of this exergame is available for download in APK format for Android at http://www.esi.uclm.es/www/dvallejo/rehab/thread-the-grape. The application is intended to be run on the standalone device TVico (https://tvico.io/), an Android device that incorporates functionalities for skeleton tracking thanks to a built-in depth sensor.

The game mechanics of the developed exergame consists of alternately lifting knees up to hip height and then lowering them back down to the ground, starting with the right knee and ending with the left. The condition for finishing the exergame is defined by the number of repetitions for which the patient must perform the movement with both knees. Thus, completing a repetition increases the obtained score and the quantity of wine produced in the meter.

In Figure 6, various views that form the exergame are shown. (a) Tutorial view shows the patient’s avatar (left) and another avatar, automatically controlled (right), that repeats the exercise as a guide for the patient. In this case, both avatars are characterized by a robot. In this view, the patient can repeat the movement as many times as they want until they understands how to correctly perform the repetitions. Once the wine meter has reached the maximum value (i.e., the internal score is higher than a threshold), the exergame progresses to the (c) participation schedule, where the patient performs the repetition of the tutorial up to ten times following the sequence of the game mechanics. Note how, in this case, the main joint that must be moved (i.e., the knee) is highlighted with the same color as the actors who form the sequence of the game mechanics. Finally, the (b) results view shows the final score obtained by the patient, and the avatar reproduces a victory animation.

The development of the exergame follows established conventions by the gamification module, keeping the views of the scene in glTF format, and the data models of the actors and the game mechanics in JSON format. In this case, actors who define the sequence of the game mechanics deal with other stage actors during the tutorial and participation views. These last actors refer to the wine meters. This is achieved through the execution of actions that directly refer to these actors.

Thus, Listing 2 shows the partial data model that defines the behavior of actors when the patient interacts with them through the left and right knees. As an example, the complete behavior of sequence-actor-one is provided, who would sequentially execute two actions after the patient interacts through their right knee with it: (i) increase-score to increase the internal score by 15 points, and (ii) actor-scale-relative to increase the size of the referred actor (i.e., wine meter) by 4 units on the vertical axis regarding its current size. The rest of the actors would replicate such behavior, with the difference that actors sequence-actor-three and sequence-actor-four would react to the joint associated with the patient’s left knee.

Listing 2: Partial behavior of actors defining gameplay in “Tread the Grape!” exergame.

5.2. Preliminary Evaluation

The created exergame has been evaluated by seven people in terms of usefulness and ease of use. They were chosen considering their past experience when attending physical-rehabilitation sessions, not necessarily related to neurological diseases. At this point, it was not our intention to assess how the exergame improved a patient’s recovery, but to evaluate how the tool was perceived by those who might have instead used similar technologies. Attendants consisted of five men and two women, ranging from 22 to 41 years old.

The participants played the discussed exergame in Section 5.1, which took them approximately 1 min each. Before that, they were able to see the tutorial mode where a virtual avatar showed them how the exergame is played. Since this avatar replicated how the exercise should be made, it was actually very simple for the participants to understand the game mechanics. After completing the exercise, each participant was requested to fill a questionnaire with questions based on the TAM framework [18]. This allowed us to measure how they perceived the system in terms of usefulness and ease of use. The answers were scored using a Likert scale ranging from 1 (totally disagree) to 5 (totally agree).

Table 1. Statistical values relating to “perceived usefulness” and “perceived ease-of-use” dimensions to scores provided by participants using the tool (1: totally disagree; 5: totally agree). Numbers in parentheses denote standard deviations.

Dimension	Question	Mean	Mode
Perceived usefulness (PU)	1. I would like to use this tool for rehabilitation.	4.14 (0.90)	5
	2. I find it fun to perform the rehabilitation exercises using this tool.	4.43 (0.79)	5
	3. I have felt motivated while performing the exercises.	4.57 (0.53)	5
	4. I have the intention to use this tool if I should need it.	4.14 (0.69)	4
Perceived ease-of-use (PEOU)	1. I am satisfied with how easy it is to use the tool.	4.29 (1.11)	5
	2. Facing the camera for body recognition was an easy task.	4.71 (0.49)	5
	3. I can effectively complete my rehabilitation exercises using this tool.	4.28 (0.76)	4
	4. I prefer to use this tool instead of attending the rehabilitation center.	4.14 (1.07)	4
	5. I feel comfortable using this tool.	4.71 (0.49)	5
	6. It was easy to learn how to use the tool.	5.00 (0.00)	5
	7. Overall, I am satisfied with the tool.	4.43 (0.53)	4

shows the obtained results. The mean values for the addressed statements were all higher than 4, which denotes good reception of the tool by the participants and shows that the exergame represents a good starting point regarding usefulness and usability for remote rehabilitation.

Finally, the involved participants also had the opportunity to leave comments about the tool so that it could be improved on the basis of their suggestions. Most of the comments were positive and stressed how visual feedback contributed to keeping the participants motivated when performing the exercise. The possibility of playing the exergame with another person was also very well-valued. On the other hand, regarding negative comments, most of the participants complained about the skeletal tracking, since they sometimes noticed that their virtual avatar was not correctly performing the leg movements. Additionally, they missed a game mode that involved more than two players.

6. Conclusions and Future Work

In this work, an architecture designed as a base to support the idea of precision rehabilitation was proposed that pursues the personalization of the physical rehabilitation of patients affected by neurological diseases to offer them more individualized treatment and to improve access to technological systems that support this process. In the medium term, the aim is to automate the recommendation and adaptation of physical-exercise routines so that the clinical personnel maintain their role as supervisors but they can simultaneously devote more quality time to their patients.

The gamification module was designed and developed so that, on the basis of the definition of a data model, the generation of personalized exergames for each patient is easy to handle. However, the process of creating new exergames is not yet fully automated and requires certain specific knowledge, such as the creation of the scenes by means of 3D models, and the conversion of models to the supported format or packaging of all binary resources in the exergame repository (e.g., textures, 3D models, animations, sounds).

Preliminary experiments that involved an exergame related to lower-limb rehabilitation were conducted to assess the usefulness and ease of use of the tool. Most of the participants perceived the system as a good starting point, and suggested interesting ideas to improve the current version of the prototype.

A future line of work would consist of facilitating this process for exergame authors through a tool that allows them to build exergames without having to know imposed restrictions by the gamification module or authorship processes of the digital contents that make up the game.