User-Centered Design and Evaluation of an Upper Limb Rehabilitation System with a Virtual Environment

Abstract

Virtual environments (VEs) and haptic devices increase patients’ motivation. Furthermore, they observe their performance during rehabilitation. However, some of these technologies present disadvantages because they do not consider therapists’ needs and experience. This research presents the development and usability evaluation of an upper limb rehabilitation system based on a user-centered design approach for patients with moderate or mild stroke that can perform active rehabilitation. The system consists of a virtual environment with four virtual scenarios and a developed haptic device with vibrotactile feedback, and it can be visualized using a monitor or a Head-Mounted Display (HMD). Two evaluations were carried out; in the first one, five therapists evaluated the system’s usability using a monitor through the System Usability Scale, the user experience with the AttrakDiff questionnaire, and the functionality with customized items. As a result of these tests, improvements were made to the system. The second evaluation was carried out by ten volunteers who evaluated the usability, user experience, and performance with a monitor and HMD. A comparison of the therapist and volunteer scores has shown an increase in the usability evaluation (from 78 to >85), the hedonic score rose from 0.6 to 2.23, the pragmatic qualities from 1.25 to 2.20, and the attractiveness from 1.3 to 2.95. Additionally, the haptic device and the VE showed no relevant difference between their performance when using a monitor or HMD. The results show that the proposed system has the characteristics to be a helpful tool for therapists and upper limb rehabilitation.

1. Introduction

A cerebrovascular accident, also known as stroke, refers to the alterations in blood supply to the brain that cause the loss of brain function. In general, stroke is related to intracranial or extra cranial vascular pathologies such as atherosclerosis or embolism [1].

Stroke is considered the sixth cause of disability worldwide, the third in developing countries [2], and the second leading cause of mortality [3]. To obtain a healthy condition after a stroke it is necessary to consider a biopsychosocial approach, which means considering the body structures and their functions and environmental and personal factors [2]. Activities of daily living (ADL) are considered as an indicator of the functional status describing the independence of a person according to fundamental skills, such as mobility and bathing. ADLs can be basic (BADLs) or instrumental (IADLs), BADLs are the necessary skills to cover basic physical needs, for instance, personal hygiene, ambulating, and eating; on the other hand, IADLs involve activities that allow an independent life and participation in the community [4]. About half of post-stroke patients have motor deficits, making them dependent on others for ADL. For instance, motor impairments in the upper limb (UL) remain for many stroke survivors; only 5% regain full function, and 20% remain nonfunctional [1,5,6].

In general, motor rehabilitation is focused on improving the function affected by pathologies or caused by injuries to allow patients to perform daily living activities [7,8].

To observe a change in the UL movement after a stroke, it is essential to take into account the repetition of exercises and the patient’s motivation in traditional rehabilitation [9,10]. However, sometimes the repetition of exercises is monotonous and boring, and patients may lose motivation, and this is finally reflected in the success of the rehabilitation [9,11]. Post-stroke rehabilitation exercises allow reorganization of neural networks in the brain and nervous system, which is known as neuroplasticity [12].

Up to now, the use of new technology, like virtual reality (VR) or virtual environments (VE), has been shown to have a positive effect in different medical areas, such as in the reduction of anxiety and stress [13], physical activity coaching for older adults [14], for presurgical tasks [15] and for rehabilitation [16,17].

VR is a simulation or model created with computer modeling. It allows the user to interact with a 3D environment [18]; on the other hand, VE refers to graphical scenarios where the users interact with virtual objects simulating reality [8]. This technology has benefits, for example, the frequency and intensity of exercise can increase compared with standard approaches, and the motivation and engagement increment for repetitive tasks [19].

To enhance the interaction between the system and the user, the use of a haptic device is useful, as it allows the reduction of mental effort and cognitive load, and it also decreases the time to perform a task and the number of collisions with a virtual object through the generation of haptic feedback [20]. This allows the user to interact with virtual objects through a feeling of touching and manipulating [21], according to the haptic type, the user can feel the texture, temperature, vibration, weight, or inertia of an object [22]. It is worth mentioning that a haptic device is accessible only for post-stroke patients with moderate or mild impairment, which means that the patient has sufficient motor and neurological functions to perform the activities of virtual scenarios [23].

To reduce the cost of systems that use haptic devices, some authors employ vibration motors to provide vibrotactile haptic feedback to touching virtual objects [24], in prosthetics with stimulation to the stump [25], and to improve standing balance [26]. Another advantage is the size of the motors, because they do not hinder user movements. That is why vibrotactile feedback is a solution for a low-cost system that gives cues about the virtual objects and VE to the user. With the addition of another feedback modality, such as vision, vibrotactile feedback is more effective [27], reflecting in the improvement of performance and in the reduction of real-time errors [28].

The development of a rehabilitation system with VR and haptic feedback seems an appropriate method to integrate strategies that maintain the patient’s motivation, that allow them to observe their performance in the therapies continuously, and improve their quality of life. In recent decades, researchers have developed different systems to enhance rehabilitation and integrate robotic devices with VE.

Game-based applications for rehabilitation use interactive games to maintain or increase a patient’s motivation or try to simulate daily living activities [29,30]. However, some of these systems use commercial video games, representing a disadvantage because they are designed for healthy users without considering the therapists’ needs, the participant’s opinions and the range of motion. Therefore, their feedback cannot reflect improvements in the patient’s motor control [3,31,32,33,34].

Another point to consider is that, in developing rehabilitation systems, the therapist’s experience must be part of developing the new technology in rehabilitation. According to therapists, barriers for a system’s inclusion in rehabilitation are the need for technical knowledge, the difficulty to use it, the time to set it up, and the support required by the system [35].

In most cases, characteristics like the range of motion [33,34] and users’ abilities and capacities [3,36] have been considered to develop these systems. It is not until the system is finished that User-Centered Design (UCD) is used to evaluate and improve it [37,38], or just to perform data analysis [33]. For instance, the evaluation of the system considers family members with post-stroke relatives [34] or stroke survivors [33], but does not consider therapists’ participation during the development, which is crucial because they provide feedback and ensure that the system fulfills its purpose. The main contribution of this article is that the VE for UL rehabilitation was developed, taking into account the requirements obtained from therapists using UCD, for instance, the desired movements. The VE proposed uses a haptic device and it is designed to be used in a clinical environment.

For the design of the system proposed, several factors were taken into account, such as: (a) the capacity of the patient (ranges of motion), (b) the psychological properties of the color for the Graphics User Interface (GUI), (c) motor learning principles, such as repetition and feedback [39], and (d) performance characteristics of the communication channels between the VE and the user (bandwidth and transmission delays for haptic feedback, visual and auditory feedback) [40].

The main difference in the proposed rehabilitation system’s development process from others available in the literature, is the therapists’ feedback in all its stages. Once the first version was ready, the usability and functionality were evaluated. Then, according to users’ results and feedback, necessary changes were identified and implemented. Finally, the usability, user experience, and functionality of the system were evaluated by volunteers. According to the therapists’ needs, four clinical scales were included in the GUI. It is hypothesized that implementing the User-Centered Design approach and therapists’ opinions will increase the usability, the hedonic and pragmatic qualities, and the attractiveness of an upper limb rehabilitation system with a virtual environment that features virtual scenarios focused on performing post-stroke rehabilitation exercises.

This paper is organized as follows: Section 2 presents the materials and methods used for the development of the rehabilitation system, which describe the collection of therapists’ requirements, the VE design, the integration of the haptic device and the VE, and the system evaluation by therapists. Section 3 describes the experimental study. Section 4 shows the results that describe the users’ experience and system usability, the performance obtained by the VE and the haptic device, and the comparison between the use of a monitor and Head Mounted Display (HMD), and Section 5 shows the discussion of these results. Lastly, Section 6 concludes and mentions future work.

2. Materials and Methods

The system’s development was based on user-centered design, consisting of four stages: therapists’ requirements analysis, VE design, integration of the VE with the haptic device, and system evaluation. From the beginning to the end of the development, therapists gave feedback regarding the system and its modifications. This feedback was complemented with information obtained from the literature.

2.1. Therapists and Therapy Requirements Analysis

For the development of the system using UCD, for the first step, a literature review in UL motor rehabilitation was conducted in order to design the first questionnaire. The aim of it was to obtain information regarding traditional rehabilitation characteristics and therapists’ requirements. The questionnaire consists of two parts (See

Table 1. Questions from the first therapists’ questionnaire.

N°	Question
Therapists and therapy requirements
1	Number of post-stroke patients attended
2	How many patients have upper limb (UL) problems?
3	Age range of the patients
4	Duration of a physical therapy session
5	Sessions per week
6	Number of sessions to discharge a patient
7	Are there breaks during the session?
8	Joints covered by rehabilitation exercises
9	Rehabilitation wrist, shoulder, and elbow movements
10	Correct position to perform therapy
11	Variables used to evaluate the patient’s progress
12	When is it decided to increase the difficulty of the exercises?
Technological tools in rehabilitation
13	Choose which of the following tools you have used and describe how you use them: computers, video-games, haptic devices, rehabilitation robots, apps.
14	Information for evaluating patient performance that can not be achieved with current techniques.

). The first one provides information related to the therapists and therapy requirements. The second one looks after information about the use of technological tools in rehabilitation, such as video-games or robots.

The questionnaire was applied to 16 therapists with experience in stroke rehabilitation. Six therapists belong to the Clínica Multidisciplinaria de la Salud from the Autonomous University of the State of Mexico, and the other 10 to the Centro Estatal de Rehabilitación y Educación Especial; both institutions are located in Toluca, Mexico.

From the answers, the movements performed by the patients in post-stroke rehabilitation were identified. Therapists emphasized that the evaluation of the patient’s rehabilitation progress should employ clinical scales, which became an essential requirement in developing this system. Additionally, they mentioned other metrics that could help them, such as time, range of motion, hits, and errors. With this questionnaire, a global idea about the clinical scales was obtained.

With the analysis of the questionnaire results, seven requirements (RQ1–RQ7) were identified, as shown in

Table 2. Therapists and therapy requirements.

Requirement	Description
Therapists’ requirements
RQ1	Therapists want to choose predetermined levels and not configure the characteristics of the virtual environment (VE).
RQ2	Therapists want to have medical scales included into the VE.
RQ3	Therapists would like to observe the changes in the patient´s performance.
Therapy requirements
RQ4	The VE must replicate the user movements.
RQ5	During the rehabilitation, the patient’s position must be sitting, standing, or lying down with a straight back.
RQ6	The joints involved in rehabilitation should be the shoulder, elbow, and wrist.
RQ7	The movements involved in rehabilitation are: extension/flexion, abduction/adduction, and rotation of shoulder; extension/flexion and pronation/supination of forearm and wrist; and ulnar/radial deviation for the wrist.

. Three requirements were related to the therapists’ needs, and the others related to the rehabilitation requirements.

To fulfill RQ1 (see Table 2), therapists should have predefined levels instead of modifying the parameters of virtual scenarios, such as object velocity, duration time, number of hits, etc. Therefore, each level should increase in difficulty, according to feedback provided by the therapists. To cover RQ2, a screen with medical scales was designed. It is worth mentioning that the scales integrated into the virtual environment do not add information according to the patient’s performance by themselves, and it is necessary that the therapists manually modify them (as explained in Section 2.2.2). For RQ3, the information related to the patient’s performance and medical scales were saved in a CSV file with the user’s ID. The ID was assigned to them after the therapists created an account for each patient.

The last four requirements, RQ4–RQ7, relate to therapy needs; these requirements allow the system to have similarities with traditional rehabilitation to achieve the same objectives. To cover them, the user should not perceive a delay between the haptic device and the VE (RQ4) [41]. Moreover, according to the questionnaire answers, the patient must be sitting or recumbent during the rehabilitation; therefore, the proposed rehabilitation system requires the user to sit in a correct posture (RQ5). Finally, the haptic device allows the free movement of the shoulder, elbow, and wrist of the patient (RQ6), including the identified movements involved in traditional rehabilitation (RQ7), these movements are focused on active rehabilitation, which is why the movements of the fingers were not considered.

According to the aforementioned requirements, an initial prototype of the system was developed. Figure 1 shows the architecture of this prototype; it consists of a haptic device, an electronic controller, and a virtual environment. In the next section, the design and development of the virtual environment are described; in Section 2.3, the integration of the haptic device and the VE is reported.

2.2. Virtual Environment Design

The virtual environment consists of four virtual scenarios and a Graphical User Interface (GUI). For its design the seven requirements identified in the first questionnaire were considered. The joints movements considered to satisfy RQ6 and RQ7 and performed by the user in each scenario are shown in

Table 3. Movements performed in each virtual scenario.

Virtual Scenario	Movements
	Shoulder	Elbow	Wrist
Shapes	External and internal rotation, extension, and flexion	Extension and flexion	Extension and flexion
Supermarket	External and internal rotation, extension, flexion, abduction, and adduction	Extension and flexion	Extension and flexion
Fruit	External and internal rotation, extension, flexion, abduction, and adduction	Extension and flexion	Extension and flexion
Building	External and internal rotation, extension, flexion, abduction, and adduction	Extension and flexion	Extension and flexion

. The difference between each level is the amplitude of the patient’s movement needed to interact with the scenario. The level will be selected according to the patient’s progress based on the planning of the rehabilitation sessions made and evaluated by the therapists. Not all the joint movements identified in RQ7 were integrated into the system; some of them cannot be displayed within the VE because the haptic device does not include orientation sensors on the pencil, such as wrist movements (see Section 2.3).

2.2.1. Graphical User Interface (GUI)

The GUI design was based on the psychological properties of color, ratio contrast, design guidelines provided by the user-centered development website [42], and the therapists’ requirements (RQ1–RQ3). In general, the user interface consists of the following screens: the main menu, create an account, account login, scenarios selection, levels selection, clinical scales selection, and a screen for each scale (Figure 2). The user’s account allows game data (time and hits) to be saved in a CSV file.

Using the clinical scales selection screen, the therapists can choose four stroke-assessment scales to record the progress of their patients; the scales are the Brunnstrom scale (stage of stroke recovery), Daniel’s scale (muscle strength), modified Ashworth scale (spasticity), and Barthel Index (activities of daily living). The therapist can select the grade/scale/stage related to the patient’s condition on each scale. To calculate the Barthel index, the therapist must choose the option and press the “Calculate” button. It is worth mentioning that the therapists must manually capture the necessary data for each scale.

2.2.2. Virtual Scenarios

Four virtual scenarios were designed as engaging games (Figure 3): Shapes, Supermarket, Fruit, and Building; all the scenarios have relaxing background music, auditory and vibrotactile feedback, which is provided when the virtual end-effector held by the patient collides with a virtual object. When the patients interact with the scenarios, they can see their score and a chronometer to know the remaining time and performance.

All the scenarios present clues to help users complete an activity successfully; however, the system does not force the user to correct their movements. When the user starts the system, first, the instructions appear, then the user has to press the button in the haptic device end-effector to start the game, then the virtual representation of it appears in the center of the screen. For all the scenarios, if the user ends the game, a screen appears showing the score.

In the Shapes scenario, the patient has to grasp and hold the real haptic device end-effector, and according to its movements, its virtual representation will replicate their movements. Then, to pick a virtual figure, the user needs to press the button (only once) located in the end-effector and move it to place it in the desired place. To drop the figure into the black hole with the corresponding shape, the user has to press the button one more time. The user gets an error when they put the figure over the black hole with a different shape. The difficulty of each level is different. In the first level, one of three different figures can randomly appear: square, triangle, or pentagon; for the second level, a hexagon is added, and the figures displayed are smaller than the first level.

The Supermarket scenario was designed as a game. The user moves the hand and presses the button to take a product; after that, the patient moves the hand over the supermarket cart and presses the button to drop it. Finally, the patient can move the supermarket cart across the hall by pressing the blue arrows that appear in the virtual scenario. According to the patient’s condition, the scenario has three levels. In the third level, patients have to raise their arms higher than in the first level, and these movements are related to the products’ positions on the shelves. In the first level, the products are on the middle shelf; in the second one, the patient has to take 12 products, one product of each type from the middle and high shelf, and finally, in the third level, the patient has to take products that appear in a checklist from all the shelf levels. The checklist appears in all the levels, but on the third level different products appear each time.

The Fruit scenario has four levels. In the first level, the patient can move a big basket over the ground (without raising the arm); the fruit falls slowly, and a red guide indicates where the fruit will fall. When the guide collides with the end-effector, the patient receives vibrotactile haptic feedback. In the second level, the patient must move a small basket, the fruits fall faster along with the guide, but it will not vibrate when it collides with the end-effector. In the third level, the basket must be moved over some rocks (raising and maintaining the arm up); if the end-effector collides for three seconds with the rocks, the game is over. Finally, in the fourth level, the patient should avoid colliding with the rocks and catch worms. The game ends when the patient catches three worms or collides for three seconds with rocks or after five minutes of playing it.

In the Building scenario, patients must move a hook on a block, and the button must be pressed to pick it up; after that, the patient moves the hook and block to the center of the scenario where there is a red guide, and the user presses the button to drop it. In the first level, the patient has to stack six big blocks; in the second, the patient has to stack ten small blocks. Finally, in the third level, the patient has to stack ten small blocks with more precision.

Table 4. General features of the virtual scenarios.

Features	Scenario
	Shapes	Supermarket	Fruit	Building
Objective	Put the figure that appears in the middle of a desk over the black hole with the same shape	Take products from shelves and put them inside a shopping cart	Move a basket and catch fruit falling from the sky	Move a hook, pick blocks and stack them
Virtual representation of the end-effector	Pencil	Hand	Basket	Hook
Number of levels	Two and a time trial (game against watch)	Three	Four	Three
Sounds	Two sounds: errors and hits	Two sounds: collisions with shelves and hits	Three sounds: collisions with fruit, worms and rocks	No sounds
Vibrotactile haptic feedback	Collisions with shapes	Collisions with products, shelves and arrows	Collisions with rocks, fruits and worms	Collisions with the guide where the blocks should be placed
Clues	When a wrong figure is chosen a sound is played	Each product is marked with its name	A virtual guide indicates the place where the fruit will fall	A virtual guide indicates where to put the block

shows some features of each aforementioned scenario.

2.3. Integration of the Haptic Device and the VE

A haptic device was developed and used to interact with the VE; it consists of three Degrees of Freedom (DoF), with one encoder on each joint for monitoring its angular position. Because the system has been designed for active rehabilitation, for the arm end-effector (third link), a 3D printed pencil was designed and attached employing a passive universal joint (without orientation sensors), which allows pencil rotation when the user moves their wrist and arm. The pencil internally has a mini DC motor that generates vibrotactile haptic feedback and a button that allows interaction between the user and the VE (see Figure 4). It is worth mentioning that the VE only shows the end-effector of the haptic device before the universal joint and the pencil, which do not affect the interaction with each virtual scenario.

Furthermore, the low- and high-level haptic device controls were implemented in the NUCLEO-STM32F767ZI embedded system board using the Simulink libraries for STM32 microcontrollers [43]. The low-level control is in charge of monitoring the encoders, user button, and the home button used to reset the encoders’ position and calibrate the haptic interface when it is in the home position; additionally, the variation of the pulse width modulation to control the vibration of the mini DC motor. In the high-level control, the forward kinematics and the end-effector velocity are programmed and estimated. It may be noted that the control loop frequency runs at 1 kHz.

The communication protocol is implemented using a high-speed USB UM232H module configured at 3 Mb/s to allow communication between the board and the host PC in real time. Finally, in the host PC, a GUI Simulink^® model, which runs in background mode, manages the haptic interface and VE data (end-effector position and velocity, the collision presented) that are exchanged with Unreal Engine 4 via UDP (at 100 Hz).

For the VE display, the system can use two different devices, the first is a monitor (laptop or PC), and the second is the HTC Vive Pro with a base station. In Figure 5 the components of the rehabilitation system are shown.

2.4. System Evaluation by Therapists

An evaluation study with therapists was carried out to evaluate the system’s functionality, usability, and user experience. For the usability evaluation, the Usability Scale (SUS) was used. This scale consists of 10 items with a five-point scale and scores the system from 0–100 [44]. This scale is a valuable tool to measure the usability of systems. It is considered a standardized questionnaire for evaluating perceived usability in different applications [45], and one of these advantages is that with the evaluation of 5–10 participants, 80% of usability problems can be identified [46]. The user experience was assessed with the AttrakDiff-Short questionnaire (available from www.attrakdiff.de (accessed on 30 September 2021)). This test consists of 10 pairs of words that evaluate the attractiveness, pragmatic and hedonic qualities of the system with the semantic differential scale using pairs of opposite adjectives with a seven-point scale [47].

For this evaluation, five therapists with UL motor rehabilitation experience participated. The therapists were working in Clínica Multidisciplinaria de la Salud from the Autonomous University of the State of Mexico.

During the evaluation, each therapist used the system, and after this, a second questionnaire was applied. The questionnaire consists of four sections. The first obtains information with the SUS about the use of the system in the clinical environment and its usability evaluation. In the second section, the user experience was evaluated with the AttrakDiff-Short test. In the third section, the therapists were consulted about the importance of clinical scales and, in the last section, the therapists were asked about the GUI and for additional comments about the system.

Table 5. Questions from second questionnaire.

N°	Question
Part 1: System Usability Scale (SUS) and use in a clinical environment
1	I think it would be easy to use the system in my clinical environment
2	I think my patients will be motivated using this system
3	The content is appropriate for my patients
4–13	SUS items [44]
Part 2: User experience
14–23	AttrakDiff-Short [47]
Part 3: Importance of clinical scales
24	Selection of the clinical scales that would be useful in the virtual environment
Part 4: Information related to GUI and additional comments
25	Do you prefer to choose the levels or the characteristics of the game?
26	Additional comments

shows the questions related to each part. Besides the questionnaire, the therapists also commented on some issues or gave advice during the systems use.

2.4.1. Results of System Evaluation by Therapists

From the first part of the second therapists’ questionnaire, therapists agreed that the VE content was appropriate for rehabilitation and would motivate their patients’ therapies. Regarding the systems use in a clinical environment, two therapists were undecided about its use in a clinical area; it meant they did not agree or disagree with its use in a clinical environment. Some reasons were that the system seemed complicated and some functions were unpredictable; also, the keyboard did not work to interact with the VE except to enter text.

In the usability test, the system got a score of 78/100, representing a system with good usability (see Section 4.1). These results showed that some therapists thought the system could be more straightforward and, consequently, the system functions (screens, buttons, instructions) needed to be integrated. On the other hand, therapists thought it was easy to use the system, and the assistance of a technician in all the sessions was not necessary, only at the beginning, when they learned how to use it. Regarding the user experience evaluation (AttrakDiff-Short), a diagram of the description of word pairs was obtained (see Section 4.2). In this questionnaire, ten pairs of words were presented. These pairs were grouped into three categories: pragmatic quality, hedonic quality, and attractiveness. Pragmatic qualities refer to the utility and usability of the system [48], hedonic qualities are related to emotional needs [49], and attractiveness refers the qualities that make a system aesthetically attractive. The average score in these areas was 0.6 for hedonic qualities, 1.25 for pragmatic qualities, and 1.3 for attractiveness. With this test, some characteristics were detected to improve the system. For example, therapists mentioned that the system was not very predictable, and that it looked a little cheap, which meant that the system did not look attractive enough to acquire. Furthermore, the system should be more straightforward and structured, with more user-friendly buttons, menus, and tools. Simultaneously, the system needed to be more creative to result in a captivating system, adding more colors or tools that make the system interesting for users.

2.4.2. System Modifications

After the therapist evaluation, and according to their feedback and comments, the following improvements were made in the system:

• In the initial prototype, if the users wanted to start interacting with a scenario, they needed to push a button with the name of the scenario, but the therapists suggested adding the button’s functionality to the images, so if the users pressed the images, the scenario would start.
• For the sign-in screen, the therapists mentioned that it would be better to use the keyboard to enter the data and not just the button.
• For better interaction between the user and the system, 3D elements (furniture, objects) were added for a more realistic environment; the letter size of the scenarios was also increased.

Once all these changes were introduced, subsequent usability and functionality tests were conducted with healthy volunteers, as described below. For this study, the protocol was approved by the research ethics committee of the Faculty of Medicine of the Autonomous University of the State of Mexico.

3. Experimental Study

3.1. Participants

Ten healthy subjects (nine male, one female) participated in this study to test the system, the mean age of the volunteers was 39.6 years old (range 26 to 55 years old) with a minimum of 9 years of education. All the subjects participated in the same tasks. This evaluation aimed to verify the proper operation of the system before a clinical trial.

3.2. Experimental Design and Task

According to the participant’s schedule, the experiment was conducted at two different sessions, which could be held on the same day or two different days. During the study, the participants answered the volunteers’ questionnaire, which consisted of three parts: (1) experience with computational systems, video games, virtual reality; (2) user experience, usability, and performance of the system; and (3) which display system the user prefers. The second part was conducted two times, one for each display system (monitor and HMD).

Table 6. Volunteers’ questionnaire.

N°	Question
Part 1: Experience with computational systems
1	Do you have experience with computers?
2	What do you use the computer for?
3	How many hours a week do you use the computer?
4	From 1 (lowest) to 10 (highest) How would you rate your computer skills?
5	Have you played videogames?
6	Do you know what virtual reality is?
7	Have you had any experience with virtual reality?
8	Have you suffered dizziness or discomfort when traveling, playing videogames, using the computer, or with virtual reality?
Part 2: User experience, usability and performance of the system
9	The information provided by the system is clear
10	The movements that are shown on the screen match with the ones I perform
11	I like the virtual scenarios
12	Tasks are right for me
13	I felt discomfort or pain while using the system
14	I felt some auditory or visual problems while using the system
15	I successfully completed the activities
16–25	SUS items
26–35	AttrakDiff-Short
36–50	Haptic device performance
51–64	Virtual environment performance
Part 3: The preferred display system
65–72	Which display system (monitor or head Mounted Display (HMD)) has a greater presence, clearer image, clearer sound, better interaction, less optical distortion, and better perception of distances?
73	If you had to participate in another session, which visualization system would you choose?

describes the parts and questions of this tool. Each stage of the experiment is described below:

• Explanation: The tasks to be carried out were explained to the volunteers by the authors. Including their purpose, the study activities, and their information would be used only for the study and not for any other purpose. After this explanation, if they wanted to continue their participation in the experiment, a questionnaire with questions about age, occupation, academic degree, and their participation consent was given.
• Familiarization: Approximately 10 min were given to the volunteers to understand how the system works. First, the haptic device’s movements were explained, also how they could see these movements replicated in the virtual scenarios. Volunteers played the Shapes scenario’s time trial and could ask questions about the operation of the system. This stage ended when the volunteer had no questions.
• First session. This stage took approximately 40 min to be completed and consisted of two parts. In the first part, an account was created for the volunteer, and then the volunteer had to play all the levels of the four virtual scenarios using the HMD, following the sequence presented. First, the subject played Building, then Shapes, after that Supermarket and finally the Fruit scenario (Figure 6 shows a volunteer using the system), and in case the volunteers felt sick or had discomfort from using the HMD, a break was given. In the second part, the volunteer had to fill the HMD fields in the second part of the volunteers’ questionnaire.
• Second session. In this stage, the volunteer had to perform the same activities that they had already done in the first session, this time using headphones and the monitor as a visualization system. After this, volunteers filled the monitor fields in the second part of the volunteers’ questionnaire and the third part of the questionnaire.

4. Results

General results obtained from the volunteers are described in this section. These evaluation findings of the evaluation were classified into five categories: system usability, user experience, performance of the haptic device, performance of the virtual environment, and display system preferences (HMD or monitor).

4.1. System Usability

The scale SUS was used for each display system when performing. The results are expressed as mean and standard deviation in

Table 7. Usability score from therapists and volunteers.

Individuals	Visualization System	Average Score	Standard Deviation
Therapists	Monitor	78.00	4.47
Volunteers	Monitor	85.50	6.50
	HMD	86.50	8.35

. According to these results, the HMD has a better score than the monitor, with 86.5 and 85.5, respectively, but the HMD presents a higher standard deviation (8.35 against 6.5). The most significant difference between the score of the sessions was in the SUS item related to learning many things to use the system correctly, which had a difference of two users more when using the HMD. In the other items, the answers were similar, and it was reflected in the final usability score.

4.2. User Experience

The AttrakDiff-Short questionnaire was applied after the two sessions to determine the user experience. Figure 7 shows the results of the therapists before the changes mentioned in Section 2.4.2 and of the volunteers after the changes. Volunteers gave better scores for all categories and word-pairs. The highest average score was attractiveness with 2.95, then hedonic qualities with 2.23, and finally pragmatic qualities with 2.20. These scores mean that the system is attractive to the user, it is usable but not very predictable and straightforward, and it is engaging but looks a little cheap according to the volunteers.

Table 8. Average score of the categories of the user experience evaluation.

Individuals	Hedonic Qualities	Pragmatic Qualities	Attractiveness
Therapists (before changes)	0.6	1.25	1.3
Volunteers (after changes)	2.23	2.20	2.95

shows the average scores obtained when therapists and volunteers were tested.

4.3. Haptic Device Performance

The haptic device performance was measured two times, one for each display system. The user had to qualify some system characteristics on a five-point scale (See

Table 9. Items used for the evaluation of the haptic device and virtual environment performance.

Number	Characteristics
	Haptic Device Performance	Virtual Environment Performance
1	General feeling	General feeling
2	Feedback	Feedback
3	Intuitive	Intuitive
4	Experimental learning	Experimental learning
5	Practical	Practical
6	Device limits the movements’ velocity	Remember functions
7	Reliability	Reliability
8	Precision of movements	Precision of movements
9	Position control	Velocity
10	Quick response	Movements’ flow
11	Error detection	Position’s control
12	Physical effort	Quick response
13	Mental effort	Error detection
14	——————————	Physical effort
15	——————————	Mental effort
16	——————————	HMD performance

); the zero was used for bad, and five for good. As shown in Figure 8, the characteristics two, five, eight, and thirteen got a better score with the HMD, on the contrary, the characteristic six got a better score with the monitor, and the others got the same score. When the volunteers wore the HMD, they used the haptic device with more confidence and executed the movements without worrying about moving it due to the immersion given by the HMD.

4.4. VE Performance

The VE performance was evaluated two times, one for each display system, and the user had to qualify the characteristics of the system shown in Table 9. In a five-point scale, the zero was used for bad, and five for good, and as for the evaluation of the haptic performance, just the display system was changed (monitor or HMD). As Figure 9 shows, there are three characteristics where the monitor got a higher score as follows: general feeling, error detection, and physical effort. In three characteristics, the volunteers could not perceive any differences between both display systems; these were feedback, the system was intuitive, and remembered the functions. Of the elements of the system with either of the displays, the other characteristics got a higher score when the volunteers wore the HMD (characteristics 4, 5, 7, 8, 9, 10, 11, 12, and 15). One of these characteristics was the mental effort, and the users mentioned that when they used this device, it was easier for them to understand the position of the virtual objects, including the representation of the end-effector of the haptic device.

4.5. Display System

For this category, the volunteers were asked about which display system they prefer, whether the HMD or the monitor. For this, seven characteristics were presented to volunteers, and they had to choose which system matched the characteristic the most, or they could choose both systems if they did not perceive any difference. Figure 10 shows these characteristics, and all the volunteers agreed that they felt more presence with the HMD. One user mentioned that they did not feel a difference between the HMD and monitor for the VE’s image clarity, sound, and sensation. The monitor was chosen as the best display system in the next three characteristics: a clearer sensation (one user), a lower optical distortion (three users), and a better distance perception (two users); the other volunteers chose the HMD for the other characteristics. Subsequently, the volunteers were asked: if you have to participate in a third session, what display system would you prefer? and nine of the ten volunteers chose the HMD.

The quantitative data of each level, such as time and hits, were analyzed. The number of hits in each level with each display system were compared. The level where the volunteer had a better performance was defined as the level with the biggest number of hits in less time. Figure 11 shows the number of levels with the highest performance with each display system; not all the volunteers finished the system’s 12 levels.

In this evaluation, the clinical scales were not considered and therefore were not evaluated because the volunteers were healthy subjects, and the evaluation was focused on the functionality and usability to detect improvements to the system.

5. Discussion

This work presents the development and usability evaluation of an upper limb rehabilitation system based on a UCD approach for active rehabilitation in post-stroke patients. One of the advantages of using UCD is the continuous feedback from therapists, which facilitates the acceptance of new technologies in clinical practice. This acceptance is necessary because the quality of rehabilitation can be affected by the rejection of this technology [50].

A guideline that helps develop new rehabilitation technologies is necessary for a successful transfer of rehabilitation research to the clinical environment. In [51], the authors detected that usability and clinical effectiveness are relevant points for the integration of rehabilitation systems. Another essential point related to the system, is to satisfy the requirements of therapists, such as comfort and safety, and if it is easy to set up and fun to use.

The objective of this work was to measure usability, user experience, and observe how feedback from therapists can increase the usability of a system. In addition, a general comparison was made between the use of a monitor and HMD as the visualization system; however, the system needs to be evaluated with post-stroke patients to observe its clinical effectiveness. For this evaluation, the use of specific scales for virtual environments in rehabilitation such as the Questionnaire for Satisfaction Evaluation of Virtual rehabilitation (USEQ) [52], and presence questionnaires will provide relevant information related to the performance of the system in a clinical environment, including which visualization system will be more effective in the clinic.

With this work, it is possible to identify some elements that make a rehabilitation system with a VE usable for therapists, such as the ease of using it, the integration of GUI elements that are easy to understand, and intuitive interaction, easy-to-learn functions, and engaging virtual scenarios for patients. In addition, with identifying the requirements shown in Table 2, the system integrates valuable features for the therapist, for instance, the selection of predefined levels, the medical scales used by therapists, the best position of the patient, and the movements involved in rehabilitation.

The therapists’ involvement in developing the proposed system is reflected in the scores of usability (see Table 7), pragmatic and hedonic qualities, and attractiveness (see Table 8), resulting in a system with high usability and attractiveness to the users.

On the other hand, the haptic device’s evaluation from the volunteers show similar average scores when they use the monitor or the HMD. The most significant difference is in characteristic number eight, where the volunteers feel greater precision when using the HMD. This could be related to the immersion that this device provides. This difference is also reflected in the VE evaluation, where the same characteristic obtains the highest difference.

In addition, the quantitative data obtained from the volunteers’ evaluation was useful to detect any difference between the use of an HMD and a monitor. The usability results showed that there is not a remarkable difference between the two display systems. With the improvements made after the therapists’ evaluation, the system obtains better results in the hedonic and pragmatic qualities, meaning that the changes made after the therapists’ feedback improved the system.

This research indicates that a VR rehabilitation system that includes the needs of the therapy and the therapist could be a helpful tool in UL rehabilitation; this allows acceptance in a clinical environment and reduces rejection of this type of technology by the therapist.

6. Conclusions

The presented work reports the development of a UL rehabilitation system focused on the VE. The design was based on the therapists’ and the therapy requirements by using user-centered design and theoretical concepts, such as color psychology, motor learning principles, and characteristics of the communication channels in VEs. For the virtual scenarios, interactive games, daily activities, and the characteristics of the rehabilitation were taken into account, and elements were introduced to improve motivation, such as the score or time and the possibility of visualization of the patient’s performance after each session.

This work shows the importance of therapist participation in the development of a rehabilitation system. The results of the experimental study showed an increase in usability, pragmatic and hedonic qualities, and in the system’s attractiveness after the system was modified according to the results of the therapists’ evaluation. The results suggest no significant differences in the perceived performance of the virtual environment and the haptic interface when interacting with either the monitor or the HMD, allowing a lower cost when the monitor is used.

Although the volunteers’ usability and user experience evaluation show positive results, future studies must test the system in a clinical environment with post-stroke patients to evaluate the clinical effectiveness. This study will be highly appropriate to evaluate the system with specific scales for rehabilitation systems such as the USEQ, and in order to obtain a more precise result for the difference between the use of a monitor and an HMD, the application of a presence questionnaire will be relevant.