Mobile Augmented Reality Serious Game for Improving Old Adults’ Working Memory

Abstract

As the aging population increases rapidly throughout the world, various approaches and studies are in progress to prevent age-related diseases. Among the diseases related to the elderly, dementia (in which cognitive function declines) is classified as a mental disorder. Since there is currently no therapeutic agent for dementia, early diagnosis and prophylactic approaches may be useful. In this study, a mobile-based augmented reality system for regular cognitive function training is proposed to minimize declines in cognitive function among the elderly. Using the characteristics of markerless augmented reality technology that can support physical activities, the foregoing system was developed in the form of a serious game based on an understanding of physical aging by the main users and inspired by existing psychological cognitive evaluation tools. The augmented reality system proposed in this study aims to induce the active participation of clients with goal setting and motivation using a gamified training system. In addition, it can ultimately be used as a self-assessment tool by recording an individual users’ performance ability. This proposed system must be used after receiving proper guidance from psychologists. The game protocol was designed together with experts in clinical psychology: therapists as well as neuropsychological assessors who were experienced in carrying out cognitive training sessions. The experts said that the system could help improve cognitive functions, such as working memory, attention concentration, and visual perception memory. However, this system has some limitations. This system was verified once with a small number of experts and could not be introduced to an actual elderly group to undergo verification of effectiveness. To compensate, we will conduct experiments to verify the effectiveness in order to avoid placebo effects. The effectiveness of program implementation will be verified by digitizing the correlations between the results of the neuropsychological assessment in the form of paper and pens and the results of signal data.

1. Introduction

As the lifespan of humans is prolonged due to the development of medical technology and increasing interest in health, the elderly population is also rapidly increasing. The proportion of the aging elderly among the total population is increasing rapidly throughout the world, not only in developed countries but also in emerging countries [1]. As the lifespan of the elderly is prolonged, age-related disorders are also on the rise, and dementia is one of the senile disorders classified as a mental disorder [2]. Due to the current absence of dementia-modifying drugs [3], methods that would delay the progression rate and early diagnosis are needed, and prophylactic approaches may be more efficient in promoting improvement of the quality of life.

Dementia incurs huge social costs [4]. When dementia occurs, there are many problems in the daily life of the patient because many difficulties accompany dementia, such as family members’ stress, limitations in social activities, psychological stress, financial burdens due to constraints on economic activities, and the declining health of caregivers. Dementia increases risks because cognitive functions, including memory and learning, problem solving, attention, and reasoning cannot play their roles as a result of brain damage from various causes [5].

Furthermore, declining cognitive function may lower the patient’s self-esteem and lead to depression, anxiety, and behavioral constraints, thereby, worsening the symptoms [6]. In a situation where countermeasures for social problems related to the elderly are urgently required, various approaches and studies are needed to prevent cognitive decline.

Physical activities are reported to have direct effects on cognitive function. Some studies attempted to prevent dementia and improve cognitive function in normal elderly persons with exercise [7] or applied exercise to the elderly with dementia to prevent the decline of cognitive function and improve the symptoms of psychological disorders, such as sleep disorders, depression, and behavioral disorders [8,9,10].

This study proposes a new type of working memory training system that enables physical activities that can prevent decline in cognitive function of the elderly using the characteristics of augmented reality technology that supports indoor and outdoor activities. The proposed system was developed in the form of a game inspired by the method of performing the Trail Making Test (TMT), a neuropsychological cognitive test tool, which can train multiple cognitive functions, such as attention ability and working memory.

Augmented reality was defined by Milgram and Kishino by classifying the continuum between the real world and the virtual world that can be experienced by users according to the degree of mixture with reality [11] Figure 1. The ultimate goal of augmented reality is to augment the user’s perception of and interaction with the real world [12]. According to Ronald Azuma, augmented reality should combine real-world elements with virtual information and enable interaction in real time, and should be augmented in a three-dimensional space.

Augmented reality can provide new user experiences by augmenting and providing information on the real world and enabling real-time interactions. Augmented reality is rapidly spreading to a wide range of industrial fields along with the development of technology and presents new possibilities as a technology that enables the organic fusion of technologies in various fields.

This paper proposes a mobile augmented reality (MAR)-based cognitive training system in the form of a serious game that can be easily performed at any time, which was inspired by the TMT neuropsychological test evaluation tool used in the field of psychology. An interface suitable for the main user base was designed to induce active participation in the 3D environment. This system aims to prevent declines in the cognitive functions of the elderly, and its major contributions to the relevant fields are as follows.

• Design of an appropriate interface based on an understanding of the physical and psychological processes of the main user base.
• Design of systematic game levels including tutorial sessions together with clinical psychology experts.
• Primary verification and feedback with a number of neuropsychological assessors and cognitive training experts.
• Promotion of the improvement of cognitive function with multiple tasks that combine low-intensity physical activities and cognitive tasks.
• A game-type mobile augmented reality-based cognitive training system that can be carried out at any time in one’s own space.

2. Literature Review

Among the symptoms of dementia, which is an age-related disease for which no clear treatment method is available, the decline of cognitive function hinders daily living activities. Therefore, early diagnosis and prevention are important for dementia. In the field of psychology, the Trail Making Test (TMT) [13] is used as a method for the early diagnosis of cognitive decline. The TMT is one of the most commonly used neuropsychological tests to evaluate short-term memory and concentration among cognitive functions, and relatively simple and reliable results can be derived from it.

The TMT is generally composed of type A and type B tests as follow as Figure 2. Type A is carried out by finding and connecting numbers arranged in a circle in order, and type B is carried out by alternately finding and connecting numbers and letters arranged in a circle in order. This test was found to be quite sensitive in the detection of minor brain dysfunctions, and its clinical usefulness as a representative attention or frontal lobe test was proven in a long study [14].

2.1. Trail Making Test

Cognitive factors in TMT performance were studied, and the results indicated that type A is related to visual and motor exploration functions and type B is related to reading outcome ability, visual exploration skills, the ability to mentally maintain two sequences at the same time, attention, and working memory [15]. Type B involves a more complex cognitive process as well as visual exploration compared with type A [16].

However, since the TMT was originally developed as an evaluation tool, it is not suitable for cognitive function training because the same stimulus is always used, and, in the process of performing the test, the performance speed is measured from the start to the end under the control of the tester. In particular, the TMT-A module, a tool for simply measuring attention, is not very useful as a stimulation for cognitive training because the task itself is too easy and simple. This study developed a cognitive training system by borrowing the existing TMT method while focusing on the training of not only attention (which is measured by the TMT) but also visuospatial functions and working memory.

2.2. Physical Exercise

Cognitive functions have been reported to be associated with physical activities, and studies have been attempted to prove the association between actual exercise and cognitive function [17,18,19,20]. In fact, various forms of cognitive training are employed for the purpose of improving cognitive function, and since cognitive decline due to age-related diseases is closely correlated with a decline in physical functioning, cognitive training and exercise should be applied in parallel to maximize the effects of prevention and treatment [21]. In a meta-analysis of the effects of exercise in the elderly with dementia, study findings indicated that exercise improved physical functions, health conditions, cognitive functions, and positive behavior [10].

2.3. Multitasking Exercise

In a recent study, it was found that a multitasking exercise that combines physical activity and cognitive tasks was effective in improving cognitive function in patients with mild cognitive impairment [22].

Regular physical activities have positive effects on cognitive function and can delay the aging of the brain by continuously stimulating it, thereby, helping the activation of the cognitive functions of the elderly. Cognitive functions require continuous practice from a long-term perspective, and the pleasure gained during training plays a great role in the user’s continuance of training for a long period of time [23]. Instead of a traditional treatment method that is repeatedly implemented, if the treatment method is provided in the form of a game with an element of pleasure, which requires an appropriate level of physical activity that can be implemented by the elderly alone without any burden or help by others, it can be practiced consistently.

In this context, studies on serious games (SG) made by applying the attractive characteristics of games to the field of healthcare, such as therapy sessions including physical and psychological therapies, have been actively performed.

The term serious game is only used for games with educational intent [24].

2.4. Serious Game

Although there are various definitions of this term, the most common one is that serious games are ones used for purposes other than entertainment and can have positive effects on the development of players [25]. Therefore, serious games refer to those games that contain sufficient game elements, have a special purpose other than fun, and mainly have educational effects, therapeutic effects, and training effects.

Many studies on using serious games for the improvement of the cognitive abilities of the elderly have been conducted in the past. Such games were mainly developed as video games that could be used in the user’s home or nursing room. After having users participate in a game for a certain period time, the results of evaluation were compared and analyzed with the existing results of neuropsychological measurement tests to evaluate the effects of participation in the game [26,27,28,29,30,31,32,33]. In addition, some studies have proposed the Eldergames project, in which four persons play the game simultaneously at a desk using table-top augmented reality [34].

The penetration rate of smartphones has increased, and therefore a mobile device application was developed. This study enabled self-management and tests with a personalized app at home based on working memory [35].

The aforementioned studies were not only faithful to the serious game functions but also conducted experiments with users.

2.5. Adopting AR/VR Technology

In a serious game, it is important to motivate the players to hold their attention so that they can focus on the game itself rather than their physical limitations. The use of interactive AR/VR technology can help them do this. There are a number of studies on serious games for pain relief and rehabilitation treatment using AR/VR technologies [36,37,38,39,40,41,42,43]. Among them, studies related to upper limb rehabilitation therapy for adult stroke patients have been actively conducted, and guidelines were presented in [44].

For example, in 2008, Ma and Bechkoum developed a game-based exercise to encourage physical exercise in virtual reality using a head-mounted display (HMD) for patients with upper limb motor disorder. In the case of VR movement therapy, the framework was designed by adjusting various factors so that the game could be played according to the patient’s ability [39].

In addition, an experiment was done to measure and evaluate the effects of natural interaction technology on the rehabilitation process [45]. Studies using AR/VR technologies have been performed not only in the field of rehabilitation for the improvement of physical functions but also in the field of psychotherapy. Studies aimed at the treatment of mental disorders, such as alcoholism, anorexia, anxiety disorder, autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), etc. have been actively conducted [45,46,47,48,49,50,51,52,53].

Whereas the purpose of the above-mentioned studies is psychological treatment, studies that proposed educational and auxiliary tools in relation to dementia and Alzheimer’s disease (AD), which are representative diseases of the elderly, were also conducted. Smartkuber designed a tablet-based game for measurement of the cognitive functions of the elderly using augmented reality and evaluated the effectiveness of the game [54], and, in a study by Nina et al., a multi-step action was constructed to support the daily activities of Alzheimer’s disease patients using the HoloLens from Microsoft [55].

However, in advanced studies on the improvement of cognitive function for the elderly, games were developed with an HMD device; however, their supply and activation are still in the beginning stages. Therefore, it is considered to be difficult to access and use the games in daily life, and continuous practice cannot be guaranteed.

This study proposes a cognitive training system for the elderly made by combining a working memory measurement test and mobile augmented reality with the topic of serious games applied to the field of rehabilitation. This system was developed with mobile augmented reality technology that can accompany the minimum physical activity, which can be practiced with a smartphone with high penetration rates in all age groups. In addition to the areas targeted by this system, it can help with training on cognitive functions in other areas, such as attention concentration and spatiotemporal cognitive ability.

3. Gamification

The personalized feedback given through gamification with game design elements and principles applied provides people with a sense of ownership, reinforces the achievement of task performance goals, and helps people easily start and understand the relevant system [56,57].

This study aims to achieve this purpose through the continuous practice of serious games by applying gamification elements and principles to give fun to users and induce immersion to perform actions, as well as by strongly causing motivation for achievement. For persistence and motivation, it is very important to allow users to feel good so that they will continue to participate. The combination of serious games and gamification elements and principles can lead to the positive result of inducing continuous actions through the stimulation of the user’s desire for fun, action and achievement, as well as immersion.

Many scholars have defined gamification frameworks and elements. Representatively, Chou developed the Octalysis Framework that organizes gamified elements. The Octalysis Framework is a human-centric gamification design framework that lays out eight core drives for human motivation [58]. Zichermann and Cunningham mentioned the eight elements, which are points, badges, levels, challenges, leaderboards, onboarding, social engagement loop, and feedback, and they defined gamification as a process of solving problems and attracting users by applying game-like thinking and game production techniques to objects that are not games [59]. Bunchball mentioned six elements: points, levels, challenges, virtual goods, leaderboards, and gifts and charity [60].

By categorizing points, levels, onboarding, and challenges, which are the representative elements of gamification mentioned above, points, levels, and access, excluding rewards, were applied as the measurement of the time from the start to the end of a game, the design of game levels, and the tutorial stage for the elderly, respectively, to prepare a game device.

3.1. Evaluation of MAR Serious Game Based on the Octalysis Framework Core Drives

In order to motivate the elderly and induce them to achieve their learning objectives, game mechanics and game design techniques were integrated. In this section, preliminary evaluations of the Octalysis core drives were carried out by a primary verification questionnaire with five experts, and the results were obtained. The experts evaluated the system using a 5-point Likert scale. The experts repeated the game three to four times and provided scores and brief comments on the Octalysis core drives. The average scores from the five experts were entered with an online tool, and graphs for individual core drives were generated as shown in Figure 3.

• Epic Meaning and Calling—The MAR serious game was designed to improve cognitive abilities, focusing on working memory. In addition, this system can be used as a tool for measuring cognitive abilities.
• Development and Accomplishment—A 30-day protocol is provided so that changes can be observed by checking the list of games played by date and records on My Page.
• Empowerment of Creativity and Feedback—The user can interact with virtual objects in the AR environment, and the system provides visual effects. The system also provides a mini-map, navigation bar, and zoom animation when searching for virtual objects.
• Ownership and Possession—The users log in with their own accounts and provide personalized data. The users can read data scored based on the game results.
• Unpredictability and Curiosity—The difficulty for users gradually increases for 30 days. The system randomly provides the forms and shapes of objects with or without numbers.
• Loss and Avoidance—There is a timer at the top right of the screen so that game can be played in real time. The game can only be played once a day, and only the first performance record is kept. If it is assumed that the game is played every day, the time to complete the game and the user’s cognitive abilities such as attention and working memory are closely related.

The average score of the Octalysis Tool in the MAR serious game was 240. Although the score seems to low, it is fairly balanced in both White Hat and Black Hat Core Drives. It also seems to have balance between Left Brain and Right Brain Core Drives, which refer to the Intrinsic and Extrinsic Motivation of the core drives, maintained in a proper balance.

3.2. Target User Context

Since the elderly have relatively low perceptual ability, learning ability, and motor ability compared to young people due to aging, the users should be understood accordingly, and the games should be designed to be easier to understand and more convenient. In order to develop games, there are components that must generally be taken into account for gamification. In this game, design elements with regard to game properties and functionality were set based on seven basic considerations: fairness, flexibility, simplicity, tolerance, perceptibility, minimization of physical effort, and accessibility.

Fairness means providing the same level of usability to all users. In particular, since the elderly have much lower literacy compared to young people, the number of letters should be minimized. Therefore, virtual objects were made with only figures and numbers whenever possible. In consideration of adaptability to users, flexibility was provided with a tutorial composed of three stages, and in consideration of simplicity, the game was made to be played with only simple touch inputs at any stage without additional equipment.

In consideration of the decline in the perceptual ability of the elderly, the number of letters was minimized, and large fonts were applied to virtual objects to improve readability. The use of parts of the body for game play other than the upper body was minimized to improve psychological and physical accessibility so that the game can be played easily on the user’s mobile device anytime and anywhere.

Augmented reality technology follows a procedure to explore and interact with virtual objects. Although this is relatively simple, it will be felt difficult by the elderly. For augmented reality, objects should be explored by scanning their top, bottom, left, and right sides with a mobile camera. Since the experiences, cognitive abilities, and physical abilities of the elderly who enjoy games are different from each other, top priority was given to the provision of standardized game operation methods so that the users could become familiar with the augmented reality technology-based game environment. In addition, a separate tutorial stage was designed so that the game could be played appropriately, reflecting the physical functions and game performance abilities of each person.

4. Proposed System

The mobile augmented reality cognitive function training system proposed in this study is a smartphone-based game-type training tool for the elderly as shown as Figure 4. This system was developed with inspiration from the TMT, which has been implemented in the form of a written evaluation in the field of psychology. Since the TMT was originally a simple tool for measuring attention concentration and executive functions, it was not suitable for application to the cognitive function training program because it always uses the same stimulus. We developed by borrowing the method of the existing TMT while focusing more on working memory than on the attentional concentration measured by the TMT.

In this paper, we consider low-intensity activities with behavior characterized by an energy expenditure of less than 3.0 METs. According to the World Health Organization (WHO), sedentary behavior is defined 1.5 METs or less and light-intensity physical activity is between 1.5 and 3 METs [61]. This proposed system also supports not only sitting but walking while operating it. Therefore, this system can provide multiple tasks to simultaneously carry out low-intensity physical activities (less than 3.0 METs) and working memory tests using mobile augmented reality technology.

We strove to reduce the psychological aversion to augmented reality technology by having users play the game using a handheld mobile device familiar to them. With the game-type cognitive function training system, users can freely play the game anytime in their own space without expert intervention or control. In addition, by inducing regular practice, the system aims to positively affect physical activities and the activation of cognitive functions.

Since the elderly persons targeted by this system have the problem of being easily exposed to danger even in an indoor environment, the user plays the game while stationary. A suitable user interface was designed to minimize dangerous situations around the user and to improve system convenience.

4.1. System Overview

The system was developed with an emphasis on maintaining the safety of users by entirely blocking risks in advance. It consists of an introduction part, which is a preparatory stage before game play, and a game play session. As shown in Figure 5, the introduction part means the time from the start of the application to the point immediately before the beginning of the game. When the application is started, a warning window is displayed first to help the user recognize risk factors.

Since the risk of safety accidents due to attention deficit exists during movements, securing safety is particularly important for programs targeting the elderly. The session to measure the range of a user’s activities can be said to be a function for elderly users, and the relevant function measures the range of activities, while the user only moves the upper body laterally in a stationary state. In addition, objects are augmented within this range when the game is played.

In the game play session, functions that should be helpful for the elderly to play the cognitive training game smoothly are implemented and provided.

4.2. Level Design

As mentioned earlier, since elderly persons are not only unfamiliar with augmented reality technology but also have a much lower ability to play games than that of general users, particular attention should be paid to game design. In the case of this game, a user tutorial was provided to help the elderly understand the applied technology and game, and the game levels were designed in detail with a number of difficulty adjustment variables.

The level of difficulty of training is adjusted with the number and shape of target objects. As for the number of target objects, two are presented in the tutorial stage, and three are presented at the first level of difficulty of this training, and the number of target objects gradually increases as the difficulty increases, so that eight or nine objects are presented at the highest level of difficulty. The number of objects was determined by taking notice that nine objects are used in the forward test and eight in the backward test of the Digit Span Test, which is a test used to measure working memory of the Seoul Neuropsychological Screening Battery (SNSB-II), which is used for the evaluation of old-age cognitive function [62].

4.2.1. Tutorial Stage

A tutorial, which is a practice game mode to help the understanding of mobile augmented reality and support appropriate game play, is provided. The tutorial is largely divided into three stages: interaction, searching, and memorizing, and the details are as follows.

First, in the interaction stage, the object is augmented at a fixed location, which is the center of the user device. A simple task of touching the augmented object is provided so that the user can learn how to interact with the device. In the second stage for searching, there are two detailed stages, and the user is helped to adapt to augmented reality technology in earnest by scanning the spaces around them. First, one object is augmented, and an arrow is displayed in the direction in which the object moved. The user moves the device in the direction indicated by the arrow to find the object.

Thereafter, two objects are augmented in the same way, and the device is moved in the direction indicated by the arrow to find the two objects. In the third stage for memorizing, there are two sub-stages, which guide the user to learn the game rules. Two objects are shown to the user, and the objects blink in a specified order and move in a specified direction. Next, the camera is moved to the location where the objects have moved to find the objects in order. While the user searches for the objects, an arrow is displayed on the screen in the direction of the location of the objects to give a hint. After that, two objects are provided in the same way, but no hint is given.

Providing a tutorial in the system in consideration of the user’s age group can help the user play the main game because they can learn about the system and the cognitive training method provided by it. In addition, the user’s understanding of the technology can be enhanced while they learn the basic method of interaction with the device and play the augmented reality game firsthand.

4.2.2. Main Game

The form of presentation of an object consists of a combination of three variables: whether there is a number, the shape of a figure, and the color. These variables are combined in the following ways:

• There are numbers (random order), different figures, and different colors.
• No numbers, different figures, and different colors.
• No numbers, the same figures, and different colors.
• No numbers, different figures, and the same colors.

Since numbered objects are the easiest to remember in general, the stage where objects are numbered is regarded as a low level of difficulty. The difficulty gradually increases, and the case where all the colors are the same and only the shapes of objects are different is placed at the highest level of difficulty of memorization. The levels of difficulty are organized by combining the number of objects and the shapes of target objects as follows.

• Three objects, and there are numbers (random order), different figures, and different colors.
• Four objects, and there are numbers (random order), different figures, and different colors.
• Three objects, no numbers, different figures, and different colors.
• Three objects, no numbers, the same figures, and different colors.
• Three objects, no numbers, different figures, and the same color.

The above five stages are carried out as one cycle, and the stages are repeated for up to six cycles. That is, a total of 30 levels of difficulty are presented, ranging from stage 1-1 to stage 6-5. The total number of stages is 31, because there is a tutorial stage. Two stages per day are given for training in the order shown in

Table 1. Order of the provision of stages.

Day	Training Program
1	Tutorial Stage + Stage 1-1
2	Stage 1-1 + Stage 1-2
3	Stage 1-2 + Stage 1-3
Omitted
29	Stage 6-3 + Stage 6-4
30	Stage 6-4 + Stage 6-5

.

The game was composed as shown above as a training program for a total of 30 days. Afterward, the user can set the level of difficulty and continue the training. The game was composed so that once a level of difficulty has been set (one to six cycles), training at the relevant level of difficulty will be provided for five days, and the level of difficulty can be set again thereafter.

4.2.3. Scoring

This system was designed for repetitive implementation even after completing the training program for 30 days, and the level of difficulty is maintained during the relevant time. While maintaining the same level of difficulty, continuous motivation and stimulation of the desire to accomplish will be helpful for consistent practice in daily life, and the scores will act as a device for continuing the game, such as providing the user with a sense of achievement.

The scores, which are the important data of this system, represent the time taken to complete the game from the start to the end. As mentioned above, since the decline in physical and cognitive functions makes it difficult to elicit an immediate response, the speed of playing games by the elderly is significantly lower compared to general users. In the case of the elderly, unexpected variables, such as a lack of understanding of the game and lack of ability to comprehend information can act as a hindrance to data collection and, thereby, affect the results of play.

The detailed stages of the game part were classified to minimize hindrances and measure game play time relatively accurately. The tutorial stage and the main game stage for 30 days were designed in consideration of the age group of users. Although the play time is recorded in the main game for 30 days, the data recorded after this time are considered reliable because the user should now be able to skillfully operate and play the game with repeated learning.

The game play result data can be used as a self-assessment tool based on the individual’s ability to play from the perspective of the game player. For the tester, systematic data can be accumulated in the evaluation of cognitive function and the measurement of the effect of the game to improve cognitive function. In addition, it is easy to obtain basic data for studies on the cognition of the elderly, and when sufficient data have been accumulated later, the system proposed in this study can be used for the diagnosis, treatment, and prevention of senile cognitive disorders.

4.3. Cognitive Training System

This system was developed in Unity 3D, a graphical game development platform and AR Core. AR Core is basically an augmented reality framework for smartphones with android operating system. It is used to detect horizontal surfaces, which is necessary for the correct positioning of virtual objects on physical environment. The virtual objects placed in the physical environment will remain stable despite having the camera move around them. Therefore, users can move freely without being hindered by the marker tracking.

This training system is an augmented reality-based game that requires the user to continuously input touches while looking at a smartphone screen. To run the game in an augmented reality environment, the recognition and exploration of the surrounding environments is essential. However, no matter how familiar with the environment they are, it is impossible for the user to consider various environmental factors around them when the game is started because they should play the working memory function training game, which requires instantaneous concentration. The best way to minimize dangerous situations around the user and support smooth game play when they play the game while moving, such as walking or running, is to systematically protect the user.

This system offers functions that can be used as auxiliary tools when playing games in consideration of changes in body functions due to aging. The various functions designed in this study not only lessen elderly persons’ sense of burden in relation to the game and the use of the device but can also help make the game easy to play.

4.3.1. Measurement of Range of Activities for the Elderly

Markerless augmented reality follows an interaction method accompanied by a user’s movements. However, it is important to reduce risks due to movements because the elderly have weakened physical functions and require particular attention to their daily living activities. Elderly persons, who are the main users of this system, have relatively large variations in body function. Rather than standardizing the user’s range of activities by reflecting the elderly’s physical functions, personalized interfaces set by the users themselves are more appropriate.

The system was designed so that the screen for measuring the range of activities appears first when the user starts the game. The user only moves their upper body according to the instructions on the screen to find the area where they can act while drawing circles in a stationary state Figure 6. When measuring the angle of activity in this way, since there is no separate object in the AR space presented on the device screen, the user is very likely to concentrate on the act of moving their body rather than the AR space in the device screen.

For the measurement of the range of activity, it is appropriate to support the universal posture taken by the user when they use the device. As shown in Figure 7a, when using the device, users typically tilt it while taking a posture in which the head is slightly bowed. If the range of activities is calculated in this state, the error between the actual range where the user can act, and the angle of activity measured by the system will be large. Therefore, a calculation process to compensate for the error that occurs just before calculating the final angle of activity as shown in Figure 7b should be undertaken.

In order to derive the angle of activity, 3D space coordinates are calculated based on the user’s location. In this process, the camera images of the device and the acceleration sensor can be used together to recognize user motions. A method to approximate a circle using the position values measured in units of one frame when the user moves according to the guide was used. The approximation derives the center of the measured position values and calculates the average distance between the values. The larger the average distance, the wider the user’s range of activities.

Figure 8a shows the positions of objects in an AR space projected onto an $x-y$ plane. When it is assumed that a camera is placed at point A and objects are placed at points J, F, and I, the direction vectors when the camera looks at each object are v, u, and u. Here, the angle of activity becomes each CAB, and the maximum angle of activity varies according to the value that limits the range where an object can break away within the AR space. Both objects F and I on vector u have the same direction vector and angle of motion. For instance, although objects J and F are placed at the same distance along the Z axis, F has a larger angle of activities. Figure 8b shows the positions measured in the sections (number; n) where the range of the user’s activities is measured expressed on the $x-y$ plane.

$P_i$ in Equation (1) means a point on the $z=1$ plane. Here, $P_c$ is the center of each point, which is the average of the sum of the coordinates of $p_i$. Each point $P_i$ is measured at regular angular intervals, and all are placed at the position of $z=1$. The vertical component, $p_{iy}$ is weighted so that it can react relatively insensitively in the vicinity of the center and sensitively at large angles.

$$P_{ix}=tan{x}_i,p_{iy}=f(tan{y}_i)$$

(1)

$$P_c=\frac{1}{n}\sum _{i=1}^n{P}_{ix},\frac{1}{n}\sum _{i=1}^n{P}_{iy},1)$$

(2)

D is the average distance between $P_i$ and $P_c$ and is used to calculate the angle of activity $\theta$. Since the average distance may be too large or too small, this is the process to correct the relevant error.

$$D=\frac{1}{n}\sum _{i=1}^{n}dist(P_i,P_c)$$

(3)

$$tan\theta =g(D,D_{max},D_{min})$$

(4)

Using the activity angle $\theta$ calculated through the process, the positions of the objects are created only within the AR space in the range where the user can act, set by them when they play the game.

With the above process, the range of the user’s activities is collected in real time, and the values are digitized and provided at the center of the screen Figure 9. The circular interface located at the center of each screen improves the user’s visibility with contrasting colors and can be identified while activities are measured. Gray was used for the maximum range of activities allowed by the system, and red was used for the measured values of the actual activities of the user.

4.3.2. Zoom-in Animation

Presbyopia is a condition in which the lens of the eye loses its elasticity, leading to difficulties in focusing on nearby objects. It is the first symptom experienced by most people as a result of aging, and magnifying glasses are worn to correct it [63]. In the case of the elderly, the ability to identify necessary information decreases due to failing eyesight and the reduction of the field of vision. This has a great effect on the use of smartphones, which are digital devices. Since smartphones contain a lot of information in a small screen, the text size must be small. In order to read the text, there must be a short distance between the smartphone and the eyes. For the elderly with presbyopia that has already progressed, reading small prints from smartphones is not only difficult but also increases the burden on their eyes. A simple way to solve this problem is to increase the font size displayed on the screen.

However, the text displayed during the game consists only of simple numbers. To increase the text size, the object size must also be increased due to the nature of the game. However, when all objects are enlarged, not many objects can be augmented on the smartphone screen. In addition, stages where texts should be displayed occupy a relatively small proportion compared to stages where no text is displayed.

This system provides an animation function to temporarily zoom in on information on the screen of a smartphone as if wearing magnifying glasses to view small print. The size of the object located at the center of the screen can be enlarged, as shown in Figure 10.

4.3.3. Game Guide for Users

Since the objects were placed within the range where the user can act, only some of them were exposed, so that it was impossible to identify all the objects in one screen. Various guides for the elderly to play augmented reality games have been designed and implemented to provide convenience. Some examples are presented in Figure 11 below.

First, in Figure 11a, a radar guide is provided to the user in the lower left corner. This guide is a sort of mini-map that gives object location information in the direction the camera is facing based on the user’s current location. It can help to identify the location of an object and find the object by reflecting the user’s movement.

Second, as shown in Figure 11b, a simple function in the form of a slide bar is used to inform the user of the existence of an object. The slide bar appears after determining whether any object exists in four directions, up, down, left, and right, based on the direction the camera is facing. For example, when objects exist only in two of four directions from the position of the user’s camera, red slide bars appear only in the relevant directions so that the existence of the objects can be intuitively identified.

Third, as shown in Figure 11c, the aforementioned radar guide and navigation bar are shown at the same time. The radar guide (mini-map) has the advantage of giving relatively detailed object location information, and intuitively gives only information on whether any navigation object exists. Since the two guides are provided together, the locations of objects can be easily grasped.

5. Expert Verification

As mentioned earlier, this study aimed to break away from the existing cognitive training programs that have been implemented under the supervision and control of experts and induce users to implement cognitive training programs by themselves in their daily lives. Therefore, the main purpose of the questionnaire survey with experts, which was an essential step, was to obtain opinions and feedback on the effectiveness of the proposed cognitive training program. In this test, experts were asked to install the system on their smartphone, use it and fill out the questionnaire afterward.

For the primary verification of the clinical psychological cognitive training system, five neuropsychological assessors and experts with experience in elderly cognitive evaluation were selected from among state-certified clinical psychologists. The participants who responded to the questionnaire were 37.4 years old on average and consisted of one male and four females. Their neuropsychological evaluation-related experience was 7.3 years on average, their elderly-related experience was 6.8 years on average, and their cognitive training-related experience was 7.6 years on average. The items and answer format used in the questionnaire were first prepared by selecting items to measure the user experience of the system and were revised twice after receiving feedback from two clinical psychologists.

The questionnaire was then classified into four major categories: learning motivation, ease of use, ease of learning, and the validity of the training stimulus, which is considered most important in this study. The score for each question was measured with a 5-point Likert scale ranging from 1 point for “do not agree at all” to 5 points for “agree very much.” The contents of the questionnaire are as follows

Table 2. Questionnaire for experts.

Category	Item No.	Item Content
Validity of training stimulation	1	With this training program, overall cognitive functions can be improved.
	2	With this training program, working memory can be improved.
	3	With this training program, attention concentration can be improved.
	4	With this training program, visual perception memory can be improved.
	5	The difficulty control protocol of this training program is suitable for optimizing cognitive training for elderly individuals.
	6	The zoom-in animation function (temporarily enlarging objects) supported by this training program will be useful to the elderly group.
	7	I think the user navigator (object location notification function) supported by this training program will help complete the task.
	8	I think the application of augmented reality technology is suitable for the programs for improving the cognitive functions mentioned above.
Learning motivation	1	Elderly groups will feel pleasure and fun when they use this training program.
	2	I think it is meaningful to implement this training program in the elderly group.
	3	The elderly group will continue to steadily use this training program.
Ease of use	1	The elderly group can easily recognize the intent of this training program and begin training.
	2	It was not difficult to find and touch objects in this program using augmented reality technology.
	3	This training program can be easily used by the elderly.
Ease of learning (Tutorial part)	1	The elderly, the primary users of this training program, will be able to quickly learn how to use it.
	2	It will be easier for the elderly group to understand the instructions for use.
	3	The elderly, the primary users of this training program, will be able to easily remember how to use it.
	4	The elderly, the primary users of this training program, will be able to easily learn how to use it.
	5	When the elderly group learns how to use it, they will be able learn well even without the help of experts.

.

Since this system supports all Android devices that can run the AR Core, the experts installed the program on devices used by them. The experiment environments of the five experts are as follows

Table 3. Experiment environments.

Participants	Testing Device	Software Version	Testing Time	Number of Test Runs	Testing Place
P1	Samsung Galaxy S10 5G	Android 10	40 min	3	Outdoors and Indoors
P2	Samsung Galaxy S9	Android 10	60 min	3	Indoors
P3	Samsung Galaxy Tab A	Android 8.1	50 min	5–6	Indoors
P4	Samsung Galaxy S9	Android 10	20 min	3	Indoors
P5	Samsung Galaxy A7	Android 10	20 min	2	Indoors

.

This questionnaire survey was conducted with a small number of experts, and the test results of individually digitized expert opinions in each major category are presented below.

As shown in Figure 12, the validity of training stimulation category received the highest score among the four categories. The experts said that, in addition to the cognitive functions mentioned in the questionnaire, the system may be suitable for improving spatiotemporal perception abilities such as visual perception discrimination and three-dimensional visual composition functions, because the elderly can be trained to recognize directions or location information based on augmented reality technology. There was positive feedback on using the proposed system as a tool for cognitive function training of the elderly.

However, the experts responded that the involvement of experts is essential when implementing the cognitive training program. Since the elderly not only shun complex and annoying things, even if they are accustomed to using smartphones, but also tend to avoid new and unfamiliar things, especially machines, it is important to make them aware of the necessity and importance of the program.

In addition, it was predicted that it would be difficult to successfully play the game before completely learning how to use the program, and as a result, there was an opinion that the active involvement of experts was required. To that end, an additional device is necessary so that the users can recognize the need to play the game by themselves and enhance their understanding of it.

Additionally, the respondents were requested to give opinions on the game guide presented in Figure 11. Their opinion was that the method of mixing the radar and navigation bar would be of the greatest help in finding objects. However, it was pointed out that education on how to use the relevant function should precede because its presence is not large when the program is implemented.

6. Discussion

As the proportion of the elderly population has been rapidly increasing worldwide in recent years, aging is emerging as a serious social problem that will be faced by us in the near future. However, as the era of an aging population is coming, the participation of the elderly is increasing socially and economically, and the perception of the elderly in society is also changing. In a social atmosphere where the independence of the elderly is gradually becoming more emphasized, the elderly must reduce their dependence on others to lead independent lives.

Age and the degree of physical decline are proportional in the elderly, and physical and cognitive functions decline according to the degree. Daily life activities are important for maintaining the health of the elderly, and slow the rate of physical decline due to aging. Many studies on the relationship between voluntary physical movement, periodic exercise or activity and quality of life, mental health, or psychological well-being have emphasized physical activities for the elderly.

In this paper, a system that enables the performance of cognitive tasks along with low-intensity physical activities using mobile augmented reality was developed. Game playing methods and game levels were designed together with experts to develop a serious game aimed to improve the cognitive function of the elderly. In addition, in a preliminary study, the potential inconvenience that may be felt by the elderly when they use digital devices was predicted, and efforts for smooth game play were made with various functions and elderly-friendly interfaces.

However, this system has limitations including that we present the experimental results from a small number of experts rather than a test group. Therefore, additional studies will be conducted later. Since this system was developed focusing on the weakening of physical functions due to aging, it may be regarded as a system for relatively healthy elderly people.

We worked on numerous beneficial functions for common elders with weak vision. However, there is still need of improvement in some of other functionalities, such as the screen size of smartphones. In the future, we will explore a barrier-free interface and worked with individuals who have other vision disabilities as well, such as color blindness and blurred vision.

Furthermore, this system was verified once with a small number of experts and was not introduced to an actual elderly group to undergo verification of the effectiveness.

When face-to-face experiments are possible, we will conduct them to verify effectiveness in order to avoid placebo effects. The experiments are to collect the subjects’ biosignal data. As with other studies, the subject will be asked to use the system proposed in this study for a certain period of time, and biosignal data will be collected through periodic visits. The effectiveness of program implementation will be verified by digitizing the correlations between the results of the neuropsychological assessment in the form of paper and pens and the results of signal data.

7. Conclusions

Up until now, it has been recognized that cognitive training should be conducted under the supervision of psychologists, and this study also considers this important. However, the final purpose of the system proposed in this study is to minimize the control and involvement of experts in order to induce continuous practice by the elderly. Prior to the introduction and application of the cognitive training program, primary verification was carried out with neuropsychological assessors and clinical psychologists with experience in elderly cognitive evaluation. As a result, this system obtained a higher score for validity than other aspects of training stimulation, and thus it can play the role of a cognitive training tool.

However, due to the nature of the elderly who may shun new things, it is difficult to completely eliminate the involvement of experts in game-type training programs using smartphones. However, it is necessary to adequately support training and education, such as helping the elderly to fully understand the importance and necessity of training with a video before training or by providing a method of playing the game. Therefore, it is important to provide customized education to the elderly regarding how to play and use the game. This will reduce the involvement of experts compared to the cognitive training programs previously implemented, thereby, enabling independent cognitive training of the elderly.

This system was designed by applying game elements to induce fun and interest when carrying out cognitive function training in a free environment without the direct intervention of experts. The purpose of the system is to positively affect the activation of cognitive function of the elderly by carrying out physical activities and cognitive training at the same time.

The system proposed in this study can systematically record and manage the results of a user’s cognitive training and can also be a cognitive measurement tool. To that end, data construction and standardization should precede. This requires work, such as increasing the number of game users to accumulate a large amount of game results and collecting data for users under various conditions. The collected data can be used to standardize user results. The standardized data can be used as important basic data for future studies on the elderly, cognitive ability diagnosis and evaluation, and the development and verification of cognitive training function improvement programs.