A Study of Exergame System Using Hand Gestures for Wrist Flexibility Improvement for Tenosynovitis Prevention ^†

Abstract

Currently, as an increasing number of people have been addicted to using cellular phones, smartphone tenosynovitis has become common from long-term use of fingers for their operations. Hand exercise while playing video games, which is called exergame, can be a good solution to provide enjoyable daily exercise opportunities for its prevention, particularly, for young people. In this paper, we implemented a simple exergame system with a hand gesture recognition program made in Python using the Mediapipe library. We designed three sets of hand gestures to control the key operations to play the games as different exercises useful for tenosynovitis prevention. For evaluations, we prepared five video games running on a web browser and asked 10 students from Okayama and Hiroshima Universities, Japan, to play them and answer 10 questions in the questionnaire. Their playing results and System Usability Scale (SUS) scores confirmed the usability of the proposal, although we improved one gesture set to reduce its complexity. Moreover, by measuring the angles for maximum wrist movements, we found that the wrist flexibility was improved by playing the games, which verifies the effectiveness of the proposal.

1. Introduction

In recent years, smartphone tenosynovitis has become common as smartphones have penetrated the daily lives of people all over the world. Then, an increasing number of people are suffering from tenosynovitis [1,2] due to prolonged use of smartphones with input operations by thumbs. The prevalence of static postures while accessing smartphones, computers, or communication devices in many occasions can affect the cause of tenosynovitis. Moreover, playing a music instrument such as a piano or a guitar, or performing a grip sport such as a golf or a tennis for a long time can influence it.

It has been known that physical hand exercises can be helpful for preventing tenosynovitis and/or curing it. Stretching hands/fingers and exerting pressures at specific points on them are effective for alleviating tenosynovitis. However, hand exercises may be too simple and boring for a lot of people, particularly, for young ones. It can be difficult to continue them in daily lives for a long period, although they are important to create good effects.

For young people, tenosynovitis and hand discomfort can be easily triggered, as most of them lack exercises to prevent them. On the other hand, some casual, no-brainer mini-games are very popular among young people and even adults. To make hand exercises more attractive and established in their daily schedules, performing hand exercises while playing video games is an attractive approach to provide easy and enjoyable daily hand exercise opportunities for preventing tenosynovitis. Several hand gestures can be defined to control a video game where a user needs to properly change and show the hand gesture. A game system for exercises is called an exergame in general [3].

In this paper, we propose a simple exergame system for preventing tenosynovitis by improving wrist flexibility, which runs on a conventional PC for easy installation. Figure 1 illustrates an overview of the proposed system. A user can control a video game running on a web browser in the system using hand gestures that are captured by a PC camera and are recognized by the Python program. This program is implemented with the Mediapipe library [4]. The camera captures the image containing the hand of the user. Then, Mediapipe extracts the coordinates of the key points on the hand. The Python program calculates the distances between the specific key points and compares them with the given thresholds to recognize each hand gesture.

Three sets of hand gestures are defined to represent different hand exercises. Moreover, five video games are installed in the system for variations. The multiple hand gesture sets and video games offer various hand movements and experiences with different levels of difficulty for users [5].

For evaluations of the proposal, we implemented the system and prepared the user manual and the questionnaire with 10 questions on the system usability. Then, we asked 10 students from Okayama and Hiroshima Universities, Japan, to play the five video games using each of the three hand gesture sets one by one by following the manual and to answer the questionnaire. After all the students completed them, we compared the total game scores between the three sets to find the easiest one. We also calculated the System Usability Scale (SUS) scores [6] from their answers. The results confirmed the effectiveness of the system.

The game score comparison found that one gesture in the hand gesture set that was supposed to be the easiest should be improved, because as the exception, the score from this gesture for one game was worse than the score from the other gestures in the set. We analyzed the reason and changed the corresponding hand gesture to an easier one based on the estimated reason. Then, we asked the students to play the game using this improved gesture and verified the correctness from the score increase in this additional experiment.

The research methodology of this paper is summarized as follows. First, we focused on tenosynovitis and asked a physician how to prevent it. We also surveyed the literature on tenosynovitis and its prevention. Then, we discovered that increasing the wrist flexibility was effective for tenosynovitis prevention, and the wrist flexibility could be enhanced by repeating wrist exercises. A computer system motivating users to repeat wrist exercises in daily lives became important.

Based on this background, we proposed and implemented an exergame system for tenosynovitis prevention to achieve our goal. In this system, users practiced wrist exercises while playing video games using hand gestures. The required hand gestures were defined to control the video games and were recognized by a Python program with the Mediapipe library. A computer camera was used to capture an image of the user’s hand, and we extracted the coordinates of the key points of the hand using the Mediapipe library. By comparing the distances between the specified key points with the predefined threshold, each defined hand gesture was recognized. A hand exercise with frequently changing gestures to control a game can help users to correct bad postures and avoid overuse of hand muscles. At the same time, playing various video games can increase the engagement and pursuit of exercises through fun and interactive ways. In addition, the exergame system can provide instant feedback and adjustments to help users better master the correct movements.

The implemented system was applied to 10 subjects, and the effectiveness was verified through measurements of the maximum angle when bending the wrist and the questionnaire with System Usability Scale (SUS) scores. The flexion–extension motion of a wrist can be used for evaluating tenosynovitis, especially for observing any pain, the limitation of wrist motions, and the response to palpation. Thus, the wrist flexion ability was tested to demonstrate the system’s benefits on wrist health by measuring the maximum bending angle of the wrist [7].

Meanwhile, the architecture of Mediapipe was highly optimized to enable real-time gesture recognition on mobile and embedded devices. This is important for real-time applications that require low latency and high responsiveness. We adopted this framework to recognize the hand gestures to achieve better interactivity in gesture-controlled games. Unlike an expensive virtual reality (VR) system, our system used a low-cost PC and open-source software. This approach significantly lowered the technical and financial thresholds for its introduction, enabling more users to use the system in daily lives for tenosynovitis prevention.

The rest of this paper is organized as follows: Section 2 introduces works related to this study from the literature. Section 3 presents the design of hand gestures for controlling games. Section 4 presents the hand gesture recognition program. Section 5 introduces five video games installed in the exergame system. Section 6 evaluates the proposal by applying it to students. Section 7 concludes this paper with future works.

2. Related Works

In this section, we explore works related to this paper, focusing on tenosynovitis, exergame, and hand gesture recognition with Mediapipe.

2.1. Tenosynovitis

In [8], BenitesZapata et al. highlighted the link between smartphone overuse and symptoms of Dekelvin’s tenosynovitis among young people, calling for attention to hand health and suggesting measures to reduce the resulting problems on tendons. Frequent smartphone use will be a trigger of tenosynovitis, especially for young high-frequency users.

In [9], Ahari et al. found that tendinitis triggered by the use of smartphones was an increasingly common health problem, especially for young and frequent mobile phone users. Studies have shown that excessive typing is a significant trigger for tenosynovitis and that early identification and adjustment of usage habits is key to preventing such problems. Therefore, we learned that tenosynovitis was not limited to the elderly, but that young people were also at similar risk due to the frequent use of devices such as mobile phones.

In [10], Ferrara et al. analyzed the literature on studies dealing with the use of physical modalities in De Quervain disease, Dupuytren disease, and trigger finger, to obtain indications for everyday clinical practice.

In [11], Palmer et al. proposed compensations for occupational illnesses that could be problematic for disorders. They were not specific to work where there were no distinctive clinical features in occupationally related cases. However, attributions could be made on the balance of probabilities, if there was convincing evidence that the risk was at least doubled in an occupational group. Their review highlighted the relative lack of data to support attributions for tenosynovitis and epicondylitis and discussed the difficulty of compensating upper-limb disorders.

In [12], Liu et al. summarized research status and global trends on tenosynovitis from multiple angles, including countries, institutions, authors, and publications. These considerations were helpful to better understand the research hotspots and development trends in the field.

In [13], Rutkowski and Rutkowski provided a comprehensive discussion of the impact of tenosynovitis amongst practitioners in the beauty industry and how they managed it. Through that paper, we learned about some hand movements that were useful in preventing tenosynovitis.

2.2. Exergame

In [14], Sinclair et al. considered game designs for successful exergames. To do this, they reviewed the history of exergames and the current state of research in this field. They found that there existed some research aimed at evaluating the physical and health characteristics of exergames, but how to design exercises was still in the early stage. By drawing on established principles from sports science for the prescription of exercise programs, they attempted to identify success factors to guide designers of exergame systems.

In [15], Brox et al. provided a narrative review of how exergames could help to motivate the elderly to exercise more, focusing on possible social interactions in online exergaming and persuasive technologies. Finally, they discussed how social exergaming could be used both for preventing loneliness and encouraging physical activity.

In [16], Daniel et al. presented the conception of a framework to support the design of serious games controlled by wearable technologies that aimed to assist healthcare professionals with the rehabilitation of sensory-motor skills in their patients.

In [17], Zheng et al. found that NIVR (Non-Immersive Virtual Reality) exergames combined with motor and challenging cognitive tasks could promote the activation of the SMA (supplementary motor area), PMC (Premotor Cortex), and DLPFC (Dorsolateral Prefrontal Cortex) in healthy young people compared with resistance exercise alone, providing a compelling preliminary evidence of their power for the rehabilitation of motor and cognitive functions in patients with central nervous system diseases.

In [18], Pereira et al. carried out a systematic mapping to identify an overview of games aimed at rehabilitating fine motor skills. The objective was to identify the scientific literature that addressed activities that stimulated the player’s motor coordination and verified which ones were specific for fine motor skills and their characteristics.

In [19], Yu et al. analyzed the effects of somatic games on the physical health of middle-aged and elderly people in Taiwan and pointed out that somatic games not only improve physical function and mental health but also enhanced the opportunities for social participation of the elderly. They explored the acceptance of somatosensory games by middle-aged and older adults. Although some of the older people were unfamiliar with the high-tech equipment, they adapted more quickly and enjoyed the games because they were easy to learn. Therefore, we reduced the difficulty level of the games so that the elderly could use them.

In [20], Kim et al. analyzed the effects of different exercise types and game modes in exergames and emphasized that physical activity levels and health benefits could be significantly improved by combining high-intensity exercises and interactive modes. Thus, we combined games and hand exercises to increase the motivation of reluctant exercisers.

In [21], Wang et al. revealed the differences in interaction preferences between older and younger people in exergames, aiming to provide theoretical support for the design optimization of exergames in order to create game experiences that are more suitable for different user groups. From this we argue that exergame design should be personalized for different age groups. Therefore, we designed the game with a simpler interface and a lower difficulty level in order to take care of the elderly.

In [22], Drummond et al. reviewed important advances in serious games in health and looked at their future directions, highlighting how health-related learning, behavior changes, and rehabilitation could be enhanced through innovative game designs. Serious games have been widely used in physiotherapy and rehabilitation, especially for patients with balance and movement disorders, and the games have been designed with a series of tasks to help patients gradually improve their body coordination and strengths in an interactive way. This is in line with the aim of our study.

In [23], Finkelstein et al. demonstrated the enormous potential of virtual reality (VR) technology in promoting physical activities and provided innovative demonstrations of how gamification can be used to enhance motivations and experiences of exercises. It is important to consider how to demonstrate effects in terms of promoting physical activities, increasing exercises and user pleasure.

In [24], Reis et al. demonstrated the potential of exergames as exercise rehabilitation tools for older adults by integrating the results of existing studies, highlighting applications in improving balance, gait, and building muscle strength, as well as providing directions for future research. They also pointed out the shortcomings in existing studies, such as the small sample sizes, short intervention periods, and lack of long follow-up data. The problem of insufficient sample data is present in our study and should be addressed in subsequent studies.

In [25], Desai et al. demonstrated an innovative application of augmented reality (AR) technology in rehabilitation training, especially in the form of exergames, which could improve the fun and effectiveness of rehabilitation. Through gamified exercises, patients could improve their sense of balance, coordination, muscle strength, and range of motions in their joints.

2.3. Hand Gesture Recognition

In [26], Sung et al. presented an on-device real-time HGR (hand gesture recognition) system, which detected a set of predefined static gestures from a single RGB camera. This system consisted of two parts: a hand skeleton tracker and a gesture classifier. They used MediaPipe as the basis of the hand skeleton tracker, improved the key-point accuracy, and added the estimation of 3D key points in a world metric space.

In [27], Padhi and Das proposed a hybrid architecture for hand gesture recognition involving Mediapipe for the detection and tracking of the hand along with various neural network architectures for training and classifying the hand gestures. A sub-sample of the publicly available HaGRID (hand gesture recognition) dataset was considered that involved eighteen hand gestures and a no-gesture category.

In [28], Chen et al. introduced a hand gesture recognition system to recognize a continuous gesture over a stationary background. This system consisted of four modules for real-time hand tracking and extraction, feature extraction, hidden Markov model (HMM) training, and gesture recognition.

In [29], Chandwani et al. provided a review of vision-based hand gesture recognition algorithms reported in the last 16 years. The methods using RGB and RGB-D cameras were reviewed with quantitative and qualitative comparisons of algorithms. Quantitative comparisons of algorithms were conducted using a set of 13 measures chosen from different attributes of the algorithm and the experimental methodology adopted in the algorithm evaluations.

In [30], Vicente et al. aimed to evaluate the effectiveness and usability of an augmented reality (AR) pose estimation prototype intended for use as real-time visual feedback in ergonomic training at the workplace.

In [31], Sundar used the MediaPipe framework to extract hand key points and combined it with an LSTM model to achieve real-time recognition of American Sign Language letters. We learned from that paper how to design gestures based on key points.

2.4. Wrist Rehabilitation

In [32], Hsieh et al. used the Blobo Bluetooth ball in wrist physical therapy training. The preliminary results were encouraging. They thought that more diverse wrist or limb rehabilitation games should be developed to meet the needs of physical therapy training in the future.

In [33], Li et al. focused on the interrelationships between the complex movements of the wrist, which are important for understanding the motor function of the wrist and the associated biomechanics in rehabilitation medicine.

In [34], Hagert focused on the proprioception of the wrist and its importance in the rehabilitation process, which elucidates how motor control and functional recovery of the wrist can be improved through perceptual training. The paper mentioned JPS (active JPS is when the patient moves the wrist actively to the predetermined target position), which makes it easy to evaluate the results of the JPS exercise using a kinetic device to determine the accuracy of reproducing a particular joint’s openness. Thus, we measured angles to examine wrist flexibility to improve the effectiveness of the system in preventing tenosynovitis.

In [35], Ferreira explored the use of serious games in hand and wrist rehabilitation, highlighting the potential of gamified interactions in rehabilitation to enhance patient engagement and rehabilitation outcomes, as well as providing specific recommendations and future directions for the design of rehabilitation games. Through this study, we learned that games should be designed to incorporate rehabilitation goals, including training of specific functions such as finger dexterity, grip strength, and wrist stability. Data collection and evaluation can be accomplished through a functional assessment of the patient (e.g., hand grip test, dexterity test) and rehabilitation progress tracking (e.g., game score, movement completion).

In [36], Bouteraa et al. designed an IoT-integrated robotic system for wrist rehabilitation. They proposed a novel rehabilitation protocol that incorporated a dynamic wrist model to adjust the intensity of treatments during rehabilitation in real time based on the patient’s muscle condition and fatigue level. It is very important to test wrist flexibility with regard to wrist rehabilitation.

In [37], Farahanipad et al. generated a wrist rehabilitation system that combined the dynamic gesture recognition with a game-based environment. This system, called HandReha, allowed users to perform specific gestures and wrist movements that were automatically recognized in the system and were used to control 3D characters in the game. At the same time, the gestures were part of a rehabilitation exercise to help patients with wrist rehabilitation.

3. Design of Hand Gestures for Game Control

In this section, we present the design of three sets of hand gestures for game controls.

3.1. Hand Exercise for Tenosynovitis Prevention

To design the exergame system for playing video games using hand gestures on a PC, we consulted a pain physician at a hospital of Qilu University Hospital, Dezhou Branch, China on effective hand exercises. We found that clenching fists, bending fingers, putting thumbs up, and stretching wrist could be effective in preventing and relieving tenosynovitis. Figure 2 shows the hand exercises for bending fingers, clenching fists, stretching wrist, and putting thumbs up [38,39].

3.2. Three Hand Gesture Sets

Based on the hand exercises in Figure 2, we designed a series of various hand gestures that could help relieve hand fatigue and prevent tenosynovitis. In this paper, we defined the three sets of hand gestures that represented the different key press operations required to control a video game. The three sets were called the wrist exercise set, the thumb exercise set, and the finger exercise set.

In this system, each hand gesture gave a specific key operation so that a user could control the game by showing the corresponding hand gesture, instead of pressing the key on a keyboard. In addition, another hand gesture was defined to represent the release of any pressed key to stop the corresponding operation. This release hand gesture was also used for a user to keep their hand at the right position such that the PC camera could capture it.

3.2.1. Wrist Exercise Set

The first set was called the wrist exercise set and intended to exercise the wrist of the player. Figure 3 shows the six gestures in this set. This hand gesture required the player to keep the fingers together and control the game by turning the wrist to the four directions of left, right, up, and down. For example, for the “up” key press, the player needed to make the palm be perpendicular to the horizontal plane and all the fingers upward. For the “right” key press, the player needed to make the palm be parallel to the horizontal plane and all the fingers to the right direction.

3.2.2. Thumb Exercise Set

The second set was called the thumb exercise set. Figure 4 shows the gestures in that set, which gave exercises of the thumb. It required to show the thumb-up “Good” sign and controlled the game by turning the thumb to the four directions. This set could help to relax the muscle in the tendon sheath area and relieve pain.

3.2.3. Finger Exercise Set

The third set was called the finger exercise set. Figure 5 shows the gestures in that set, which gave exercises for the fingers. The user needed to raise different fingers depending on the keys. This set could help to relax finger muscles and avoid finger pain.

4. Hand Gesture Recognition Program

In this section, we present the implementation of the hand gesture recognition program for game controls.

4.1. Overview of Hand Gesture Recognition Program

In the system, each defined hand gesture was recognized by the Python program that processed the captured image of the PC camera. This program consisted of two main steps: (1) the hand landmark called key point was recognized from the hand image using the open-source library Mediapipe, and (2) the hand gesture was recognized from the key-point information.

4.2. Mediapipe Architecture

The architecture of Mediapipe is highly optimized to enable real-time gesture recognition on mobile and embedded devices, which is important in practical applications that require low latency and high responsiveness. It uses the “streaming data processing pipeline” architecture, which allows different computational modules to run in parallel, reducing overall processing latency. The pipeline allows input streaming data such as image frames to be passed through multiple processing nodes, each of which can perform a separate computational task. It also uses a multi-threading technology to reduce waiting time by processing image frames in parallel, allowing the hand detection and the gesture recognition to run independently in different threads. Therefore, Mediapipe can quickly and accurately recognize hand gestures for better interactivity of gesture-controlled games.

4.3. Key-Point Recognition by Mediapipe

First, Mediapipe extracted the coordinates of the 21 key points from the image for one hand. Figure 6 illustrates them [40]. It is noted that this library can handle both hands at the same time, if necessary.

4.4. Hand Gesture Recognition Algorithm

Then, using the coordinates of the key points, each of the defined hand gesture in the three sets was recognized.

4.4.1. Recognition of Wrist Exercise Set

The angle of the line between the two key points 0 and 12 on the mid-finger against the horizontal line was calculated and is compared with the given thresholds. When this angle was between $-{120}^{\circ }$ and $-{60}^{\circ }$, the hand gesture was recognized as “right”. When the angle was between ${150}^{\circ }$ and ${200}^{\circ }$, the gesture was “up”. When the angle was between ${60}^{\circ }$ and ${120}^{\circ }$, the gesture was “left”. When the angle was between $-{30}^{\circ }$ and ${30}^{\circ }$, the gesture was “down”.

4.4.2. Recognition of Thumb Exercise Set

The angle of the line between the two key points 2 and 4 on the thumb against the horizontal line was calculated and compared with the same given thresholds as those of the thumb exercise set.

4.4.3. Recognition of Finger Exercise Set

First, the module to determine whether one finger was bent or not in [41] was implemented for judging the position of the fingertip. Then, each gesture was recognized according to the number of bent fingers.

4.5. Hand Gesture Recognition for Additional Keys

The hand gesture for “key release” and that for “space” were additionally defined and recognized by the program to control the video game. The hand gesture for “key release” was necessary to halt the recognition of a key input by the gesture. It was also useful to keep the hand position so that the camera detected it correctly. The gesture for “space” was necessary because some games needed the “space” key input.

4.5.1. Recognition of Key Release

The distance between the two key points 4 and 8 was calculated to be compared with a given threshold. When it was smaller than the threshold, the stone gesture was recognized for releasing all the keys.

4.5.2. Recognition of Space Key

The angle of the line between the two key points 5 and 8 against the horizontal line was calculated and compared with the given threshold. When the angle was between ${140}^{\circ }$ and ${220}^{\circ }$, the hand gesture was recognized as pressing the space key.

4.6. Image Acquisition Step

For the image acquisition, the PC camera used in the implemented system had 920k pixels ($1920\times 1080$). To investigate the image processing speed of the Python program including Mediapipe, we added a function for showing the frame rate on the display, as shown in Figure 7. The measured frame rate was around 30 fps.

5. Five Video Games and System Operation

In this section, we introduce five video games that were installed in the exergame system and the system operation.

5.1. Overview of Video Games

These video games can be downloaded freely from the Internet. They have different difficulty levels to play with. They run on a web browser, and the programs were implemented with HTML, CSS, and JavaScript.

5.1.1. Cave

Cave [42] can be controlled by pressing the “space” key only (one key). In this game, the player needs to press the key properly to make the spaceship fly while avoiding collisions with walls through moving up and down. When the spaceship collides with a wall, the game is over.

In the wrist exercise set in our system, when the user extended his/her index finger, this hand gesture was recognized as pressing the “space” key, so that the ship moved upwards. When the user clenched their fist, this gesture was recognized as releasing the “space” key, so that the ship moved downward.

5.1.2. Pickup Fruits

Pickup Fruits [43] can be controlled by pressing the “left” key or the “right” key (two keys). In this game, fruits continuously fall from the upper side at random locations of the game screen. The player needs to control the location of the basket to receive fruits at the bottom side of the screen by pressing either key.

When the user extended his/her index finger representing the leftward movement, it was recognized as pressing the “left” key, so that the basket and the duck moved to the left. When the user extended their index finger representing the rightward movement, it was recognized as pressing the “right” key, so that the duck and the basket moved to the right.

5.1.3. Blocks Game

Blocks [42] can be controlled by pressing the “left” key, the “right” key, or the “space” key (three keys). In this game, a number of different colored blocks appear at the top of the game screen. The player can receive a point by controlling the transmitter at the bottom properly to hit the ball back so that it collides with the block. When it collides, the block disappears and a point can be gained. When it fails to hit back the ball twice, the game is over.

When the user extended his/her index finger into the upward direction, it was recognized as pressing the “space” key, so that the ball on the transmitter was launched. When the user extended their index finger toward the left (right) direction, the gesture was recognized as pressing the left (right) button, and the transmitter moved to the left (right).

5.1.4. SnakeBite

SnakeBite [42] can be controlled by pressing the “left” key, the “right” key, the ”up” key, or the “down” key (four keys). In this game, the player needs to control the moving direction of the snake to eat as much food inside the game screen as possible. Every time the snake bites a piece of food, the player receives a point, and the snake becomes longer. If the player can obtain the defined score within one minute, the game continues. Otherwise, the game is over.

When the user extended his/her index finger into the left, right, up, or down direction, the gesture was recognized as pressing the corresponding button, causing the snake to move toward that direction. For example, when the user extended their index finger into the left direction, the gesture was recognized as pressing the left button, and the snake moved to the left. Relaxation of the wrist and fingers was facilitated by intermittent hand movements.

5.1.5. 3DTank

3DTank [42] can be controlled by pressing the “left” key, the “right” key, the “up” key, the “down” key, or the “space” key (five keys). In this game, the player controls the view of the player’s tank and moves it forward, backward, or rotated while firing bullets to hit enemy tanks. On the top right of the game screen, there is a 2D plane view that shows the positions of the player’s tank and enemy tanks in real time. In the implemented system, it was the most difficult game among the five games to be controlled by hand gestures.

When the user separated their five fingers, the gesture was recognized as pressing the space key, so that the tank fired bullets. When the user extended their index finger toward the left, right, up, or down direction, the gesture was recognized as pressing the corresponding button, causing the tank to move toward that direction.

5.2. System Operation Procedure

The following procedure explains the operation procedure of the proposed exergame system.

1.

Set up the PC at a suitable distance from the player so that the whole hand of the player can be included in the camera image.

2.

Read the instruction manual to know about the installed video games, the hand gestures, and the programs to run on the system.

3.

Open the HTML file of the selected video game on a browser.

4.

Select one set of hand gestures to control the game.

5.

Run the Python hand gesture recognition program for the selected set.

6.

Play the video game using hand gestures.

7.

Copy the game scores and other data from the top of the page into a text file for later collation.

Figure 8 shows the flowchart for this operation procedure.

At the beginning, the game control using hand gestures could be difficult for many people. To be familiar with playing the game, it could be controlled by pressing the keys as in the conventional way. After the player mastered how to play the game, he/she could start playing the game using hand gestures.

The following procedure explains the procedure in the Python program for gesture detection using the Mediapipe framework.

1.

Capture video frames from a PC camera.

2.

Preprocess the captured image to adjust for input requirements.

3.

Use the hand detection model of Mediapipe to recognize the hand region and detect the key points.

4.

Extract the key-point coordinates from the hand detection results.

5.

Recognize the gesture from the key-point coordinates.

6.

Issue the key input corresponding to the detected gesture.

7.

Press the “ESC” key to exit the program.

Figure 9 shows the flowchart of the Python program for hand gesture recognition.

5.3. Hardware Platform

Table 1. Hardware platform specifications.

Item	Specification
PC	OS; Windows 10 Home 64 bit
	Memory: 16 GB
	Processor: Intel(R) Core(TM) i7-10750H CPU
Camera	Integrated	Webcam
Wireless mouse	Legion	R9000P

shows the hardware platform specifications used in the implementation of the proposed system. The hand of a user should be clearly captured by the camera so that he/she can control the game by showing the defined hand gestures properly. A mouse was used to select and start the game. A conventional laptop PC was used here instead of a smartphone or VR in this exergame system, because it had higher capabilities for software development, program execution, user interaction, and data collection. This could improve the gesture recognition accuracy, the real-time performance, the system scalability, and the user experience.

6. Evaluation

In this section, we evaluated the proposed exergame system through students from Okayama and Hiroshima Universities, Japan.

The subjects in the application experiments were randomly selected from members in certain laboratories. Thus, only young people joined the experiments. For more comprehensive evaluations of our proposal, people of different ages from different countries, including persons suffering from tenosynovitis, will be included in our next studies.

6.1. Comparison of Three Hand Gesture Sets

We compare the three hand gesture sets in terms of easiness in playing video games through their game scores.

6.1.1. Game Score Results

We asked 10 students to play the five games using the three hand gesture sets and compared their average scores.

Table 2. Comparison of normalized average score among gesture sets.

Game	Wrist Exercise	Thumb Exercise	Finger Exercise
Cave	0.449	0.343	0.192
Pick up fruits	0.267	0.226	0.152
Blocks	0.664	0.447	0.410
Snakebite	0.730	0.408	0.327
3DTank	0.253	0.271	0.085
Total	2.363	1.695	1.166

shows the results. It suggested that the wrist exercise set was the easiest whereas the finger exercise set was the most difficult.

Since each game had a different score range, we extracted the maximum and minimum values of each game score and normalized the scores for the three hand gesture sets by the students into the range of 0–1.

6.1.2. Space Key Problem in 3DTank

In Table 2, for 3DTank, the score for the wrist exercise set was smaller than that for the thumb exercise set. This game often needed the hand gesture for the space key to shoot missiles for destroying enemy tanks while moving the tank to four directions with other gestures. Because the current gesture for the space key needed to open the index finger only, the change to this gesture from the others of opening all five fingers could be hard for players.

6.1.3. Improved Hand Gesture for Space Key

To solve the abovementioned problem, the hand gesture for the space key in the wrist exercise set was modified. Figure 10 shows the new hand gesture. It opened the gap between any pair of adjacent opened fingers in the hand, so that the transition from the other hand gesture would be much easier and smoother by avoiding close fingers. The average game score for 3DTank with this improved hand gesture was $0.531$ from the same students, which was higher than that of the thumb exercise set.

The distance between two key points 6 and 10 for the index and middle fingers and that between two key points 10 and 14 for the middle and ring fingers were calculated and were compared with the given threshold. When both of them were larger than that threshold, it was recognized as the space key.

6.2. Questionnaire Results

Next, after a student had finished the five games with the three sets, we asked him/her to answer 10 questions on the system usability as a questionnaire and analyzed the results using the System Usability Scale (SUS) score [29].

6.2.1. Questions in the Questionnaire

Table 3. Questions in questionnaire for usability evaluations.

ID	Question
1	I think that patients with tenosynovitis can receive benefits from hand exercises.
2	It is difficult to play the game with hand gestures.
3	The exercise game is enjoyable to play.
4	The system is not capable of accurately recognizing hand gestures.
5	I think the wrist exercise set in the hand gesture set is good for relieving wrist.
6	The gestures are difficult to remember.
7	I think the thumb exercise set in the hand gesture set is good for relieving the muscles of thumb.
8	The finger exercise set in the hand gesture set is difficult to play.
9	I want to continue the exercise game using hand gestures.
10	I did not feel comfortable using the exercise game.

shows the 10 questions in the questionnaire. For each question, it was requested to answer with five grades (one, strongly agree; two, agree; three, neutral; four, disagree; five, strongly disagree).

6.2.2. System Usability Scale

Then, we calculated the System Usability Scale (SUS) score from the answers.

The SUS score can be a “quick and dirty” and reliable tool for measuring the usability and was created by John Brooke in 1986 [44]. It allows one to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites, and applications.

We used the SUS due to its efficiency, reliability, and focus on usability, which aligned well with the study’s goals of assessing and refining the exergame system for tenosynovitis prevention. It stroke a balance between thoroughness and simplicity, making it an ideal tool for this particular application.

The answers of each question were converted to a 0–100 number, by added them together and multiplying them with 2.5. A SUS score above 68 was considered above the average and anything below 68 was below average. A score above $80.3$ represented an A level or the top $10\%$ of all the scores.

6.2.3. SUS Score Results

Table 4. Questionnaire results.

User	Answer the Questions										SUS Raw Score	SUS Final Score
	1	2	3	4	5	6	7	8	9	10
1	4	2	4	2	4	4	5	2	4	1	30	75
2	5	2	4	2	4	3	4	2	4	2	30	75
3	5	2	4	2	4	2	5	2	4	2	32	80
4	4	2	4	2	4	2	4	1	4	2	31	77.5
5	4	4	4	2	4	2	4	2	4	2	28	70
6	4	2	4	2	5	3	5	2	5	1	33	82.5
7	4	2	4	3	5	2	4	3	3	2	29	72.5
8	5	2	5	4	4	2	4	2	4	1	33	82.5
9	5	2	5	2	5	1	5	1	5	1	38	95
10	5	4	5	1	4	1	5	1	5	1	36	90
Average											32.2	80.5

shows the answer results of the 10 students. The average SUS final score among them was $80.5$. For each student, the lowest one was 70 and the highest one was 95. It means that all the students positively evaluated the proposed exergame system.

However, the current evaluation lacked application results in real patients with tenosynovitis or elderly ones. In future works, we will apply the proposed system to tenosynovitis patients and elderly subjects who may suffer from it.

6.3. Wrist Flexibility Results

The flexibility of a wrist is very important to prevent tenosynovitis. Thus, to quantify its flexibility, we measured the wrist bending angle after playing the games in the exergame system.

6.3.1. Measurement Setup

Figure 11 shows the four hand gestures to measure the wrist bending angle. As this angle becomes smaller, the wrist can be considered more flexible. In our experiments, we asked 5 students among 10 to play the five video games using the wrist exercise set for five days. After playing one set of the five games each day, we measured the angles.

6.3.2. Results

Table 5. The test results.

Student ID	Times	Gesture 1	Gesture 2	Gesture 3	Gesture 4	Average
1	1	120	126	145	140	132.75
	2	120	115	150	135	130
	3	105	110	145	135	127.5
	4	115	115	155	125	127.5
	5	105	108	155	125	123.25
2	1	135	105	135	140	128.75
	2	108	106	138	118	117.5
	3	105	102	128	127	115.5
	4	108	105	139	128	120
	5	110	105	135	120	117.5
3	1	125	100	150	112	121.75
	2	118	105	150	105	119.5
	3	118	110	149	110	121.75
	4	117	98	150	115	120
	5	110	110	140	109	117.25
4	1	130	108	165	120	130.75
	2	120	103	155	120	124.5
	3	118	110	149	120	124.25
	4	117	110	150	115	123
	5	118	105	150	115	122
5	1	135	110	160	135	135
	2	135	108	155	135	133.25
	3	138	110	152	130	132.5
	4	135	105	155	130	131.25
	5	132	105	153	128	129.5

shows the angle measurement results for the five students with the four gestures. For all students, the average angle became smaller day by day. Thus, playing the games in the exergame system increased the wrist flexibility, which can prevent tenosynovitis.

The evaluation results in this paper demonstrated that the proposed system not only served as an effective tenosynovitis prevention tool but also provided a fun and interactive form of health education that could promote healthy hand use habits in wider populations by reducing wrist overuse.

7. Conclusions

This paper presented the implementation of an exergame system with a hand gesture recognition program in Python using the Mediapipe library. Three sets of hand gestures that could be useful for tenosynovitis prevention were designed to control the key operations for gameplay and compared. Five video games were installed in the system where each game needed a different number of hand gestures. For evaluations, 10 students from Okayama and Hiroshima Universities, Japan, were asked to play the games in the system and to answer the 10 questions in the usability questionnaire. The results including System Usability Scale (SUS) scores confirmed the effectiveness of the proposal. The wrist bending angles in four gestures were measured after playing the games in the exergame system to quantify the flexibility of the wrist. The results showed that the wrist flexibility was improved by playing the games. In future works, we will improve the system implementation, such as the user interface, and adopt different video games and hand gestures. We will ask persons of various ages from different countries, including tenosynovitis patients and elderly individuals, to use this system and evaluate its effectiveness and usability comprehensively.