Sustainable Virtual Reality Patient Rehabilitation Systems with IoT Sensors Using Virtual Smart Cities

Abstract

To develop sustainable rehabilitation systems, these should consider common problems on IoT devices such as low battery, connection issues and hardware damages. These should be able to rapidly detect any kind of problem incorporating the capacity of warning users about failures without interrupting rehabilitation services. A novel methodology is presented to guide the design and development of sustainable rehabilitation systems focusing on communication and networking among IoT devices in rehabilitation systems with virtual smart cities by using time series analysis for identifying malfunctioning IoT devices. This work is illustrated in a realistic rehabilitation simulation scenario in a virtual smart city using machine learning on time series for identifying and anticipating failures for supporting sustainability.

1. Introduction

In health systems, sustainability is relevant due to the potential high impact of their failures or interruptions [1]. Among others, users may be injured when performing rehabilitation and nobody may notice it, for example, if the security system is malfunctioning. The data about the evolution of a patient may not be recorded if a sensor collecting data starts malfunctioning, providing wrong measures. The rehabilitation may be misleading if the indications are based on information collected from malfunctioning devices, leading users either to useless rehabilitation exercises or risky exercises that may hurt patients. For instance, a system that does not properly measure heart rate may encourage the user to continue exercising hard in risky elevated heart rates. Therefore, in rehabilitation systems, sustainability improves safety and performance.

Patient rehabilitation systems have provided a wide range of solutions to support part of the rehabilitation at home. For instance, upper limb rehabilitation has been supported with virtual reality (VR) for people with multiple sclerosis with different approaches in the literature [2]. There are different ways of addressing sensory cueing in virtual motor rehabilitation [3]. This sensory cueing relies on the proper functioning of these devices, and sustainable systems become relevant in this context. In virtual rehabilitation environments, many different kinds of external devices may be used. For example, Kinect devices have been widely used in rehabilitation for the capacity of Kinect to track limbs and hands. Kinect was especially useful for tracking rehabilitation of lower limbs [4] as the amount of mechanisms for tracking lower limbs is lower than for upper limbs.

The ways of integrating VR systems and Internet of things (IoT) are increasing, as revealed in [5]. In general, IoT devices allow users to track different activities in rehabilitation. Among others, rehabilitation systems can use smart bands, bodily sensors, smart watches, curtain sensors, magnetic door sensors and smart cupboards. Elaborated VR rehabilitation systems with IoT devices become ecosystem devices that are more error-prone due to the possible failures of IoT devices, problems in the communications among the devices, and generally coordinating for providing the service.

As one can observe in the literature, authorities pursue providing sustainable rehabilitation services. For example, Bessette et al. [6] indicated how Haiti authorities are looking to address the challenge of developing sustainable rehabilitation services, which is still not solved. In this line of research, ECG healthcare monitoring has been incorporated in residential environments [7], resulting in more sustainable healthcare systems.

In this context, the current work proposes a novel approach for designing and developing sustainable rehabilitation systems that use IoT devices and virtual smart-city scenarios. This approach relies on having redundant devices that can be automatically self-managed, and using machine learning with novelty detectors for identifying malfunctioning IoT devices. To the best of the authors’ knowledge, the proposed approach is the first one that uses novelty detectors for identifying malfunctioning IoT devices in rehabilitation systems for making them sustainable through rapid warnings to users and temporary usage of alternative equivalent data.

The remainder of this article is organized as follows. The next section introduces the most relevant related works mentioning the gaps of the literature addressed in this work. Section 3 presents the technique for making IoT rehabilitation systems sustainable. Section 4 illustrates the proposed technique with a VR sustainable system with IoT devices. Section 5 presents some experimentation simulating malfunctioning devices for evaluating the application of the proposed approach over not applying it. Section 6 discusses the most relevant aspects related to the obtained results. Finally, Section 7 mentions the conclusions of this work depicting some of our future research lines.

2. Related Work

In the line of sustainable healthcare, some works have proposed effective mechanisms that users can conduct on their own, reducing specialized medical assistance for making healthcare systems sustainable. For instance, ECG monitoring systems have been developed for residential environments [7], using the appropriate wireless sensor networks. In addition, wearable healthcare sensors have been improved to be sustainable considering peak consumption by patients’ usage [8], taking a step towards sustainable healthcare systems. Finally, trust management models have been improved to make more secure environments for healthcare systems used directly by patients [9]. However, none of these approaches proposed a mechanism for using redundant information in case of failures by means of rapid failure detection with novelty detectors, as proposed in the current approach.

One of the major applications of VR rehabilitation systems is the motor rehabilitation [10]. Some rehabilitation systems focus on upper limbs, while others address lower limbs. Some systems are aimed at training users in getting the right balance and posture. Others focus on locomotion of patients, such as the ones suffering from Parkinson’s. Nevertheless, these systems usually do not consider sustainability for avoiding interruptions in rehabilitation services.

IoT-based smart rehabilitation systems make rehabilitation more natural as it is based on interaction with physical objects [11]. There are even ontology-based methodologies for designing these interactive systems. There is even a methodology of collaboration of IoT devices for providing health services such as the measurement of memory for identifying diseases, such as Alzheimer’s [12]. In all these systems, one can observe that in long rehabilitation, sustainability can be important as it may rely on the system for detecting accidents, such as patient falls, and interrupted services can become problematic for patients. These works lack the appropriate sustainability methodologies that the current work proposes for avoiding these interruptions due to malfunctioning IoT devices.

IoT-based telemedicine has been widely used for prevention of diseases and health promotion. The review by Albahri et al. [13] presented a taxonomy of 14 different groups of applications of IoT-based telemedicine, and they proposed a lifecycle of these applications from its initial definition to its final deployment so that researchers and practitioners can be guided in the development of these applications. Nonetheless, their guide fails to mention how these applications could be adapted to be sustainable and robust against failures of IoT devices.

Rehabilitation has been monitored by combining IoT and VR. For instance, Postolache et al. [14] presented a system for remotely monitoring the rehabilitation of stroke patients by using a serious VR game and tracking the movements with IoT wearable sensors. In this way, medical doctors could evaluate and follow the progress of stroke patients remotely. In particular, this system used a headband and two gloves with sensors in all the fingers. They concluded that these kinds of systems were useful for achieving better rehabilitation results in short periods. They also increased the patient engagement during rehabilitation. However, their system did not consider appropriate measures against sensor failures to ensure a sustainable VR rehabilitation system with IoT devices.

Other kinds of rehabilitation have also been supported by IoT systems. For instance, cancer rehabilitation has also been supported by IoT technology. More concretely, Han et al. [15] proposed an artificial intelligence (AI) recommendation system for supporting decisions of cancer patients based on the information collected by IoT devices. In particular, this system focused on adapting nutrition programs. By contrast, they did not propose any sustainability approach for identifying and handling malfunctioning devices to avoid erroneous nutrition recommendations due to the misinformation caused by IoT failures.

In summary, several IoT and VR approaches have supported rehabilitation in different contexts. However, these systems usually omit measures for making their systems sustainable considering possible malfunctioning IoT devices. In this context, the current work addresses this gap in the literature by providing a technique for identifying malfunctioning IoT devices and ameliorating their impact on rehabilitation systems making these kinds of systems sustainable.

3. Technique for Making IoT Rehabilitation Systems Sustainable

The proposed technique is based on the set up of an initial configuration of devices, considering simulated malfunctioning IoT devices with redundant devices that can help in the case of failures avoiding interrupting the service. The monitoring of the IoT devices is performed by analyzing the input data from these with a novelty detector mechanism in the context of machine learning. Figure 1 presents the block diagram of the proposed technique for making IoT rehabilitation systems sustainable. This diagram follows the common notation of this kind of diagram, starting with a black dot in the beginning and ending with a black dot with an outer circle for the ending. The flow is indicated with the arrows with some rhombuses for conditional bifurcations of flow. In this technique, first, all the IoT devices are registered. Notice that these will be connected to a server, which has a synchronized copy in another server in case of server failure. The IoT device input will be used the with rehabilitation system depending on the specific application. For each IoT device, the system creates a monitoring agent that analyzes each IoT input in order to detect anomalies for identifying malfunctioning devices. After everything is set up, the rehabilitation system starts. Considering the established frequency of checking, in each iteration the system executes all the monitoring agents. Each agent collects data from the corresponding IoT device in order to confirm a dataset for later running the novelty detector. If there are enough data, it runs the novelty detector to identify whether there is any novelty representing that the device is malfunctioning. This approach is based on the assumption that initially all IoT devices start working properly as this is requested to be checked when setting up the system. In this way, when an IoT device malfunctions, it is usually detected with the novelty detector from its data patterns that it is different from the previous ones. One can observe that the last conditional bifurcation is used to keep a loop for repeating the same iterations over and over until the system is stopped.

The proposed approach uses self-management for IoT devices. In case of an IoT device failure, the system uses an alternative IoT sensor for capturing the missing data. This can be done thanks to the initial redundant configuration. The malfunctioning device is unregistered from the server, and the system warns the user about this malfunctioning device so that it can be either fixed or replaced. The malfunctioning can be due to many different possibilities, such as a problem with the charging system, the absence of data from the corresponding sensor, patterns of data that do not make sense and consequently are very different from the previous one, or the lack of coherence among data considering the input data from other sensors. Because these agents are executed in the system, they can access, store and process data from other sensors to check consistency.

This approach uses a novelty detector programmed with Python using the Scikit-learn framework [16]. The novelty detector is trained and updated with the historic data, and then it analyses the current data to determine whether it is probably malfunctioning. Depending on the specific domain, one should properly select which input is preprocessed and if it needs preprocessing. For instance, it considers windows of certain input intervals and processes these as a time series. It can also consider input from other sensors or circumstances to identify inconsistency patterns.

In the case of using time series for the novelty detector, this approach recommends using the similarity based on dynamic time warping (DTW) [17]. The advantage of DTW is that it can associate different time patterns with similar shape regardless of different speeds or different starting points. In fact, we have previously successfully applied DTW for identifying patterns in IoT devices, such as smart cupboards, in the context of perception-layer attacks [18]. On this occasion, this approach is used for identifying failures for the self-management and making sustainable rehabilitation systems.

DTW is obtained with dynamic programming with an association of two time series in which the order in both series is kept, but the association does not need to be one-by-one paired. One point of each series can be associated with several of the other series as long as the orders of both time series are kept. The underlying algorithm selects the minimum distance association by generating a matrix of distances D, of size NxN N being the number of points of each series. The matrix of distances is calculated row by row based on previous values of D, as long as the corresponding values are inside the matrix, with the following equation:

$$D[i,j]=\left|t\right[i]-r[j\left]\right|+min\left(D\right[i-1,j],D[i-1,j-1],D[i,j-1\left]\right)$$

(1)

where t and r are the two series that are being compared (i.e., the new sequence and each sequence of the corpus).

Later on, the minimum distance of DTW is calculated by checking the last cell of distance, i.e., D[N − 1, N − 1], assuming the range of array indexes is from 0 to N − 1 as commonly in most programming languages.

In order to handle the self-management of malfunctioning IoT devices, this approach proposes to use expert system rules for avoiding disruptions. For this purpose, each IoT device can be associated with similar IoT devices that capture sensors with similar information. In particular, machine learning (ML) techniques are used to convert the substitute sensor data into the data of the IoT device that is failing. On some occasions, the IoT devices will be paired, but in other cases the information may be estimated from a group of sensors.

For each piece of necessary input information, the list of sensors is ordered as the most reliable for each information source. The selection of the sensor is the first one of the list that is not considered as having failure with a threshold in the corpus of valid historical patterns, as formalized in the equation below:

$$s=(mini:i\in [0,len\left(S\right)-1]\wedge (\exists c:c\in C:DTW\left(S\right[i],c)\le K):i)$$

(2)

where s is the selected index of the sensors list (S) for each necessary input; C is the corpus with all the existing previous time series; K is the threshold for considering novelty, len indicates the length of a list, and DTW calculates the distance as previously explained, and S[i] is the time series of the sensor in the ith position for the current time.

4. Case Study of Sustainable Rehabilitation System Using a Virtual Smart City

This case study is a rehabilitation system in a virtual smart city, which uses different exercises in common machines of the city. Figure 2 shows the rehabilitation scenario in two drinking machines. In this scenario, a person could practice moving any of the two upper limbs by getting drinks identified by colors. The program indicates a color of a button of the drinking machine and the user should move the arm to get these with the corresponding sensors attached to them. This figure represents a 3D representation developed with Unity 3D, and an avatar controlled by the user, in which the avatar can walk to different city locations for different rehabilitation exercises.

In the rehabilitation scenarios, there are several metrics for measuring the rehabilitation progress that respectively considers upper limb movements in different axes depending on the exercise. In particular, the first metric is calculated with the following equation:

$$m_y=100·\frac{{\sum }_{i=0}^{N-1}min((T{\left[i\right]}_y-O_y),(H{\left[i\right]}_y-O_y))}{{\sum }_{i=0}^{N-1}T{\left[i\right]}_y-O_y}$$

(3)

where T is the array of target positions of the buttons requests in the drinking machine, O is the origin position of the hand in a relaxed posture (i.e., hands down), and H is the array of the position of the hands considering the ones with highest vertical components. On this occasion, the measurement focuses on the vertical movements (i.e., y-axis component). The metric $m_y$ considers the movements in the y vertical axis considering a percentage. It obtains the highest percentage if the user reaches the target or higher positions in all the target buttons, as the movement from its origin position is always the maximum available. If the user does not reach the target, only the movement is considered in the summation. This summation is divided by the total summation of all the distances between the origin position and the target.

Each hand (left or right) can be evaluated with the corresponding drinking machine.

In the same 3D environment simulation, Figure 3 shows the upper limbs rehabilitation by taking the corresponding hose of gas. On this occasion, the user is asked to use a different hose. It can be used for both hands. It evaluates the horizontal (i.e., x-axis) movement. This metric is calculated in a similar way as the previous one but focuses on the x-axis component of the positions with the following formula:

$$m_x=100·\frac{{\sum }_{i=0}^{N-1}min((T{\left[i\right]}_x-O_x),(H{\left[i\right]}_x-O_x))}{{\sum }_{i=0}^{N-1}T{\left[i\right]}_x-O_x}$$

(4)

where $m_x$ is the metric of movement considering the horizontal (x-axis) component, T contains the targets, O is the origin considering horizontal component of the relaxed posture of the used hand, and H is the arrays with the positions furthest in the x-axis from the origin.

Regarding the lower limbs, this simple scenario can support rehabilitation by climbing five steps at the entrance of a house, as one can observe in Figure 4. In this scenario, it calculates the number of steps the user actually undertakes divided by the target number of steps.

In addition, the average time per exercise is considered to also track the progress of patients. In patients with low injuries that usually get high scores or similar scores, the reduction of time for performing exercises can show progress in the rehabilitation.

Figure 5 presents the network structure of the proposed approach. As one can observe, IoT devices and servers are connected through wireless antennas. IoT devices such as vending machines and smart bands provide streams of data to the servers, after some basic preprocessing to avoid network overload. The servers run novelty detectors for detecting malfunctioning devices. If any device is malfunctioning, the servers ask another similar device to send similar information about users so that this self-management improves the availability of the sustainable system. In addition, the user is warned through the cellphone so that the malfunctioning IoT device is repaired as soon as possible.

In order to simulate the failure of each sensor, we define the probabilities of failures per sensor. The failures of a sensor are simulated with the following equation:

$$v^{\prime }=\left\{\begin{array}{cc}v+F·(r_2-0.5),\hfill & \mathrm{if}{r}_1\le p\hfill \\ v,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

where v is the original value of the sensor, p is the probability of failure, $r_1$ and $r_2$ are real random numbers in the [0, 1] interval, and F is the range of failures.

The availability of each time is calculated with the following logic formula:

$$a\iff \left(\right|m_x^{\prime }-m_x|\le E_x)\wedge (|m_y^{\prime }-m_y|\le E_y)$$

(6)

where a indicates whether for each exercise the system was available, $m_x^{\prime }$ and $m_y^{\prime }$ are calculated with $v^{\prime }$ considering failures. It is reminded that $v^{\prime }$ refers to the value of the sensor considering the possible simulated malfunctioning. The percentage availability is calculated from the ratio of exercises in which this system was considered to be available according to the previous equation.

5. Experimentation

This experimentation focuses on the sustainability part concerning different failures. In order to experience this, we simulated several failure scenarios with a Python program. In the experimental setup, we used the 3D city environment for collecting the data. Then we simulated the failures by introducing random bias only in the devices that are simulated to fail. We needed to simulate the failures as it was difficult to have IoT devices that started malfunctioning during the experimentation.

In the detection of failure scenarios, we calculated the similarity between simulated malfunctioning scenarios and the corpus of normal usages.

Table 1. Parameters used for the simulations.

Parameter	Value (s)
Average distance threshold (events/h)	2.3
Numbers of simulations	1000
Simulated time in each simulation (h)	24
Probabilities of failures (%)	10; 20; 30

shows the parameters used in the experimentation. We run 1000 simulations in order to ameliorate the bias of the results due to the stochastic circumstances used for simulating the failures. We simulated a whole day of rehabilitation (i.e., 24 h) to make it possible to detect different daily patterns of rehabilitation and be able to identify malfunctioning IoT devices. We considered three different probability of errors (i.e., 10%, 20% and 30%) in order to show the utility of the current approach in different malfunctioning scenarios. The average distance threshold was tuned according to the experiments with the purpose of obtaining high accuracy.

We compared the simulated scenarios with the normal corpus. Figure 6 shows the distances in both cases. These are the DTW distances with the previous data assumed to be normal, considering the most similar pattern from the corpus. This is measured with the average number of events per exercise that were registered per hour, considering each event as the extreme position of the limb movement (e.g., highest position in the drinking machine exercise). We hypothesize that normal rehabilitation scenarios do not obtain an exact match with a previous normal case, but the similarity of distance is quite low because the sensors provide similar values for similar exercises considering a representative amount of simulated users processed for the training of the system. However, malfunctioning systems provide very different values, probably because of the differences of values introduced by the simulated malfunction devices.

Figure 7 shows the comparison of histograms of the frequencies of the corresponding event ranges. One can observe that the ranges are different, although there is some minor overlap in the range between 2.2 and 2.7 #events/h. This is the average number of events per exercise per hour. Notice that normally each perfectly done exercise takes two extreme positions (i.e., relaxed position and target position, e.g., pushing the button of the drinking machine). However, some users do not get to the relaxed position between exercises or ignore the exercise (this is why the value is lower than two), or they can do several movements (e.g., up and down) before reaching the target (this is why the average can get higher than two). Malfunctioning devices may provide random values and the number of events can substantially increase, as observed in the simulations.

As an example of DTW comparison, Figure 8 shows an example of the application of a DTW algorithm for comparing a malfunctioning rehabilitation system with a probability of failures of 20% per day and a normal system. This graph considers the total number of events per hour in a simulation of a day, considering some hours for performing exercises. The malfunctioning device registers activity even when the user is sleeping, which does not make sense for devices properly working. This kind of pattern and others are detected by identifying novelties from previous daily patterns.

In the first scenario, we simulated a scenario of error-prone devices with a probability of failures of 10% per day. Figure 9 shows the results of service availability in this scenario. The availability of the service depends on the capacity of the proposed approach in identifying the malfunctioning IoT devices, substituting the information and rapidly warning the user. Alternatively, in the control group, we also simulated that the user may notice the failures and replace the IoT components, although taking more time. One can observe that the proposed sustainability approach is better than a control experiment in which the user fixes the failures when observing them. The average improvement of service availability was 17.43%.

Moreover, we tested a scenario with more error-prone components with a 20% probability of failures, as shown in Figure 10. The improvement difference of service availability increases in this scenario. In fact, it reached an average difference of availability of 29.97%. Thus, it shows the difference in this middle error-prone scenario.

Furthermore, we performed this comparison in scenarios with highly error-prone scenarios with 30% probability. As one can observe in the corresponding results in Figure 11, the proposed sustainable approach obtains better results than the control approach, although the difference is not higher than the previous scenario. More concretely, it obtained an improvement of 23.31% in service availability. Thus, for highly error-prone scenarios, the ratio failures did not influence so much the improvement difference of the proposed approach, as it was observed in less error-prone scenarios.

The results have been compared with the other work about sustainable healthcare IoT systems called DITrust chain [19]. One can observe that the proposed approach increases the service availability over that work in the scenarios with malfunctioning IoT devices because the latter does not have any specific mechanism for ameliorating the impact of malfunctioning IoT devices.

6. Discussion

This article has presented a novel approach for integrating novelty detectors in rehabilitation systems with VR and IoT devices for making them more sustainable. Generally VR and IoT rehabilitation systems are usually innovative and lack enough maturity for addressing properly sustainability. This aligns with the previous study [20] that reveals that the maturity of system integration is strongly related to sustainability. In fact, our proposal is focused on increasing the maturity of the integration of IoT devices in a VR rehabilitation system. It uses an extra layer of predicting malfunctioning devices based on detecting different patterns from usual. This will not only detect clear malfunctioning, such as non-working devices or low-battery problems, but also more subtle malfunctioning.

This experimental setup has relied on simulation of realistic stochastic failures as there were no proper datasets about rehabilitation systems from known and reported malfunctioning IoT devices. Normally, rehabilitation datasets are based on systems with IoT devices that properly work. This aligns with the previous research about IoT malfunctioning devices [21], in which the failures are simulated with stochastic patterns in a realistic way. The current work has followed a similar approach for the experimental setup aligning with the existing state-of-the-art studies.

The performance of novelty detectors for detecting malfunctioning IoT devices has been supported by our previous work, as revealed in [18]. Novelty detectors are appropriate for identifying problems even if there are no data from these problems [22] and just require data about normal situations. In this particular context, VR rehabilitation systems with IoT are not so common or widespread to have enough data from malfunctioning IoT devices. Thus, novelty detectors have been considered as the best approach to address sustainability in this context.

System availability has been widely considered as one of the relevant properties in medical systems in general [23]. Even in rehabilitation, system availability is considered relevant, as one can observe for cardiac rehabilitation [24]. In this context, the current work shows the potential impact of our approach in the availability of rehabilitation systems with error-prone IoT devices.

Although novelty detectors have been considered the best approach considering the current available data, we think that certain ML techniques could be more appropriate in the future when there are more available data. ML techniques usually need a large corpus of balanced data with both positive and negative cases [25]. In fact, deep learning has proved to outperform other ML techniques with enough data [26]. With this purpose in mind, we plan to adapt our systems to collect data of error-prone IoT devices used in VR rehabilitation systems, to conform a corpus of data that allows future research in ML departing from both positive and negative cases, based on some human feedback for supervised learning. In this way, this work can provide a step toward sustainable IoT and VR rehabilitation systems, heading to further sustainable rehabilitation systems in the long-term future.

One of the most relevant and difficult challenges in the proposed approach is achieving a set of devices with enough redundant data to be robust to any IoT malfunctioning device without almost duplicating the cost of the IoT devices used in the system. This challenge depends on the rehabilitation domain and may be addressed by generic devices that can have approximate information similar to several other IoT devices without substantially increasing the price of the product. Another subsequent challenge is to commercialize the proposed approach despite the necessary cost for implementing it over other cheap solutions.

7. Conclusions and Future Work

This work has presented a novel sustainable approach based on the use of monitoring agents for cooperation and rapidly providing alternative services, and warning users to properly fix the malfunctioning IoT devices in the context of VR rehabilitation systems. The automatic identification of malfunctioning IoT devices is performed with machine learning based on the analysis of time series. The alternative data rely on properly using sets of IoT devices that can provide redundant data. In comparison with a control mechanism, the proposed approach obtained an improvement of 29.97% in service availability in the simulation of the middle range of error-prone devices (i.e., failures with a probability of 20% per day). The results advocate that this approach can be useful for achieving sustainable rehabilitation services.

In future work, we plan to integrate this approach in sets of bodily sensors managed through different types of smart watches, including both Apple Watches and Android smartwatches. It will also be integrated into more serious VR games designed for rehabilitation with IoT devices for performing long-term experimentation with users to assess their perception of these sustainable systems.