1. Introduction
Mild cognitive impairment (MCI) is a state of cognitive decline between normal aging and early dementia, characterized by the presence of cognitive impairment that does not affect normal daily and social life. Society has entered a stage of advanced aging, high morbidity, and high medical needs, and the risk of conversion from mild cognitive impairment to Alzheimer’s disease (AD) is high. Studies have shown that the early identification and intervention of MCI can reduce the risk of AD by 48% [
1,
2]. Therefore, early identification, intervention, and treatment measures for older adults with MCI are an important part of the effective prevention of AD occurrence. Assessment and rehabilitation products for older adults with MCI have also begun to receive increased attention from designers [
3].
Rehabilitation training products for MCI older adults were previously not given much attention, with commercially available cognitive training products being most common. With the development of science and technology, the popularization of computers, and the continuous advancement of software development techniques, specific products for MCI older adults have been developed. Computer-based cognitive training products have become the main cognitive rehabilitation diagnostic and treatment products [
4,
5]. In recent years, with the advancement of virtual reality (VR) technology, VR has been widely used in healthcare, education, and physical function rehabilitation [
6,
7,
8]. In rehabilitation healthcare, VR devices can be used not only to assess the subject’s condition in a timely and accurate manner, but also to create reasonable rehabilitation training tasks based on the patient’s condition to achieve better therapeutic results. VR products for the training of MCI patients mainly focus on situational memory training and assessment, cognitive rehabilitation and navigation training [
9]. Manera et al. [
10] developed a serious game, “Kitchen and Cooking”, and verified its usefulness in cognitive assessment and training in the MCI population. Sebastian and other scientists [
11] found that an immersive VR radial arm maze was more valuable for human spatial memory and learning than training on a tablet and could be a beneficial tool for early MCI–AD differentiation. Park [
12] showed that training using VR virtual shopping can improve the executive abilities and instrumental activities of MCI patients at daily living levels. Ruhong Ge et al. [
13] compared the performance of young and older adults in cognitive training games using both VR and mobile devices; their fNIRS test results showed that VR is a reasonable means of cognitive training for older adults. VR environmental scenes have a high ecological nature, which can provide multisensory stimulation for the user and drive multiple brain regions to participate in the task simultaneously, providing patients with a good immersive interactivity and a high level of concentration and participation, and helping to achieve better diagnostic and rehabilitation training results [
14]. As a result, VR-based product development is proliferating, and the effectiveness of these products in diagnosing and rehabilitating MCI seniors motivates designers to continually improve their products. This means that more accurate evaluation methods are required to judge the effectiveness of VR products.
Traditional scales are commonly used to assess VR-based products. Regarding the subject’s cognitive level, the Simple Mental State Examination (MMSE), the Montreal Cognitive Assessment (MoCA), the Wechsler Memory Scale (WMS), and the Instrumental Activities of Daily Living Scale (ALDs) are often used to diagnose the subject or determine whether it is possible to administer treatment by applying VR products. Chatterjee et al. [
15] conducted a task-training for stroke patients using a VR application, demonstrated the validity of the application using the MoCA scale, and determined the application’s usability through a questionnaire. From the perspective of the subjects’ user experience, questionnaires and interviews are often used to evaluate products by investigating user satisfaction, acceptability, usability, and tolerability. Mary Hassandra et al. [
16] conducted an experiment on VR-based dual task-training with young people and MCI older adults. This ended with questionnaires and semi-structured interviews. This ultimately determined the effective-ness of the VR product for the three aspects of acceptability, usability, and tolerability. These subjective scales are simple and quick to assess, but the influence of the subjects’ own conditions (education level, cultural background, emotions, etc.), the subjects’ experience with the scales, and the testing environment can make the scores biased. Therefore, using a qualitative basis, the researcher gradually added objective measures to enhance the accuracy of the measurement.
Objective measures oriented towards the cognitive domain focus on functional brain imaging techniques, such as positron emission tomography (PET), electroencephalography (EEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). The fNIRS reflects brain activity by collecting changes in hemoglobin concentration in the brain blood [
17], which are judged by changes in the concentration of hemoglobin oxyglobin (HbO) and hemoglobin deoxyglobin (HbR) in the blood. This has a high spatial resolution, low cost and portability, and no noise, and can provide different sensitivities during the experiment depending on the distance between the light source and the detector, obtaining clearer physiological information about brain activity [
18]. VR-based cognitive training products, mostly using the movement tasks plus cognitive unfolding, are more suitable for fNIRS intervention, and can be used to make timely and accurate assessment judgments based on objective measurements.
The Kinect motion sensor (Microsoft Corporation) provides a markerless, full-body 3D motion-tracker and enables the user to virtually interact with the computer system. It can also record the behavioral characteristics of a person when using their body in a natural way to interact with the game [
19]. The Kinect sensor is considered a viable and effective tool for training the physical and cognitive components that simultaneously occur in older adults [
20]. One of the major features of VR-based products is their relationship with movement. In this study, the subjects’ natural behavioral interaction during the task was recorded with the help of Kinect. This was used to objectively determine the relationship between behavior and cognition.
Although some studies have begun to evaluate products focusing on the cognitive and motor perspectives of brain function, many challenges remain. First, cognitive training products are still in their infancy, with a gradually increasing variety of training products appearing in various forms, such as physical and virtual products, and no standard training model. Second, the current assessment of training results produced by these products is not accurate enough, and the assessment tools that are used are not comprehensive. A cognitive assessment based on brain function is the current research trend, and due to the complexity of cognition, different scholars are trying to use richer multi-source data to evaluate cognitive training products, without forming a fixable evaluation system. Third, the training characteristics for MCI elderly are not clear, and targeted training cannot be conducted. Based on the above considerations, this study used fNIRS to identify changes in the brain hemoglobin concentration of test subjects while carrying out activities based on products in the VR environment with the help of the device’s multisensory features, combined with the Kinect device. This was used to obtain behavioral data from the test subjects. We aim to detect the cognitive and motor characteristics exhibited by MCI elderly people during the use of VR products, to determine whether the combination of multiple modalities, using fNIRS and Kinect as well as cognitive scales, can obtain a correlation between their data and some physiological representations of MCI elderly people while using VR products, as well as to enhance multiple forms of cognition with the use of these products.
We make the following predictions:
The fNIRS monitoring results reflect the effectiveness of using VR products to monitor results in MCI older adults.
The behavioral monitoring results of Kinect can reflect the performance of the MCI elderly when using VR products.
Multi-source data can be used to evaluate the product from multiple perspectives and improve the monitoring of the product.
This study proposes a new product evaluation method for a specific population, using multiple sources of data to accurately assess subjects’ responses to the product and validate the product’s use in the diagnosis and treatment of MCI older adults. We also aim to propose product improvement strategies to supplement the technology used to monitor patient recovery and assist in rehabilitation.
2. Materials and Methods
2.1. Participants
Twenty volunteers were recruited from the community, all of whom were older adults who no longer participated in the workforce. The basic personal information of the subjects, such as their age, height, and weight, was recorded before the test, and they were asked whether they had neurological disorders. In this study, a group of older adults without neurological disorders was selected, and only their cognition was judged; their cognitive abilities were quickly assessed using the MoCA scale.
Table 1 shows the basic information. According to the assessment results, volunteers ranged from 17 to 28 points. Using the score set by the MoCA scale as the standard, twenty-six or more points was considered normal cognition; volunteers with MoCA scale scores ≤ 26 were screened for the experiment, and a total of 17 subjects were retained.
Table 1 shows the individual data of the subjects. The incorporation standards were as follows: (1) absence of traumatic brain damage, (2) absence of any diseases of the nervous system, (3) no use of drugs with neurological impacts, (4) shortfall of visual hindrance, (5) sufficient rest during the week before the trial, and (6) no engine impedance. Each test was conducted after obtaining in-formed consent from the subjects. Experimental procedures were performed in accordance with the requirements of the Community Human Ethics Committee, and conformed with the ethical standards set forth in the 1975 Declaration of Helsinki (revised in 2008).
2.2. Experimental Equipment
fNIRS, VR and motion capture equipment were utilized in the tests.
The fNIRS sampling device used in the experiments was NirSmart (Danyang Huichuang Medical Equipment Co., Ltd., Beijing, China), which has a frequency of 10 Hz and uses wavelengths of 740 and 850 nm. The distance between the optical poles was 30 mm. According to the requirements of the internationally accepted 10/20 electrode distribution system, 10 signal transmitters and 8 detectors were set on the electrode caps and divided into 18 channels. The electrodes were arranged in the left prefrontal cortex (LPFC), right prefrontal cortex (RPFC), left motor cortex (LMC), right motor cortex (RMC), left occipital cortex (LOL), and right occipital cortex (ROL). A flexible headgear fixator was used to fix the distance between the transmitter and the scalp to control the ambient light during the experiment and reduce the effect of excessive light on the experimental data.
Figure 1 shows the channel position of the fNIRS device.
Oculus Mission 2 was used as a VR gadget. Oculus Mission 2 is a remote, across-the-board VR gadget weighing 503 grams with an inherent Android portion, controlled by the Qualcomm Snapdragon XR2 stage and a coordinated motor for visual investigation. It has a monocular goal of 1832 × 1920, upholds 60, 72, and 90 Hz revive rates, and empowers regulators with less signal-following. The gadget upholds an enormous number of games and contains various classifications. Through a panel discussion with six experts, the six categories of action, sports, music, puzzle, leisure, and education were selected; from these, Beat Saber was selected as the experimental task for the VR product.
Beat Saber belongs to the action, music, and leisure combination of VR products; the product is very popular, with 99% of player reviews being positives. The game is carried out using handle-control, with a two-color beam, and color blocks appear on the screen, forming a correspondence with a musical rhythm. The same color beam cuts through col-or blocks. To carry out the task, the blocks must be cut at the correct rate, and must be cut to complete the level. VR’s somatosensory control should be fully utilized to avoid incorrectly pressing the handle. The combination of music rhythm, flying colors, block locations and changing directions, and the relationship between these factors, makes the game interactive and rich in action. The game also has visual simplicity and clarity, and the sense of music and rhythm are very strong. This process requires a lot of continuous movement of the upper limbs, accurate judgment of the position of the red and blue blocks, accurate judgment of the direction of the split blocks, and accurate judgment of the speed of the blocks according to the rhythm. The subject’s attention, responsiveness, spatial orientation, and limb movements are all mobilized.
Microsoft’s Kinect V2 utilizes infrared light to follow various parts of the body continuously, and was chosen as the movement gadget. The Kinect V2 upholds a limit of 25 skeletal hubs and hubs numbered 1–11 and 20–24 were chosen to examine the furthest point information. Information object types were denoted as skeletal edges, and the most extreme results contained in each edge were noted.
2.3. Experimental Procedure
Prior to the test, it was determined that each subject was new to the VR product and each subject was provided with a basic understanding of the game to ensure they under-stood the requirements and were prepared for the test. Subjects were fitted with fNIRS acquisition devices, and electrodes were tuned until all channels showed green on the product page. The analysis started after the subject loosened up in their seat, and upper-extremity activity was perceived by the movement catch gear.
The experiment was separated into two segments, resting and mission states, with each lasting for 10 min.
For the first 10 min, the subject remained stationary in a characteristic, loose, nondozing state while sitting in a chair in the resting state. This procedure was performed by collecting cerebral blood oxygen concentration data through a near-infrared head cap. The surroundings were quiet during the experiment.
After 10 min, the subjects entered the task state. The VR device was added to the electrode cap and the electrodes were recaptured; then, the device was debugged again until all channels showed green on the software page, and the Kinect device was switched on to capture upper-extremity motion data.
Subjects played Beat Saber, a persistence game, in the VR climate for 10 min. To guarantee that the gathered information was free from influencing variables, the games were set to similar music and the same level of difficulty. There was also a programmed reboot mode in the event of disappointment, while the PC projected the screen and recorded the subject’s progress. At the end of the task, the completion status was used as the classification criterion, and the subjects were divided into two categories, complete task and incomplete task, to study their interaction response.
Figure 2 shows the experimental process.
2.4. fNIRS Data Preprocessing
The fNIRS signal contains some instrument noise, experimental noise, and physio-logical noise [
21,
22]. The instrument noise was mainly caused by signal interference from the surrounding environment or instrumentation [
23]. Experimental noise was mainly caused by interference from artifacts’ motion, caused by the subject’s head movements during the test. Physiological noise was mainly the extracerebral activity fNIRS signals collected while monitoring brain activity and is the most significant cause of interference in fNIRS [
24]. To reduce the influence of these factors, the raw signal of fNIRS needed to be preprocessed. We used TDDR [
25] to remove baseline shifts and spike artifacts and con-vert the raw intensity signal to optical density. A modified Butterworth bandpass filter [
26] was then used to obtain a 0.01–0.08 Hz-filtered signal with an improved signal-to-noise ratio, and obtain the preprocessed Delta [HbO2] signal.
2.5. Wavelet Transform
The quantization of the fNIRS signal can be estimated using the Hilbert transform, Fourier transform or wavelet transform, which are mathematically equivalent when applied to spectral analysis [
27]. However, the use of the wavelet transform provides an adjustable window, thus providing an intuitive visualization of the time-frequency domain as well the high resolution for the high- and low-frequency components [
28]. This transforms the time series
with a series of usually non-orthogonal basis functions generated from the mother wavelet. Its equation is as follows:
where
is the wavelet coefficient, Ψ is the Morlet wavelet, s is the scale factor, and the corresponding frequency is the Morlet wavelet, which is a complex sinusoidal curve modulated by a Gaussian function:
where
.
Using WT, we can derive the wavelet amplitude (WA) of each Delta [HbO2] signal. WA is the average result of WT in the time domain. WA reflects the amplitude of fluctuations in the original signal at a certain frequency, serving as a power index that can be used to describe the intensity of activity in cortical areas. The quintessential hemodynamic response to mental activity provides the basis for fNIRS measurements [
29]. When specific brain regions are activated, neurometabolism is supported by a local vascular response, leading to an influx of oxygen-rich blood into the active region and surrounding tissues. This impunity leads to an increase in [HbO2] and decrease in [dHb] in the active brain regions [
30]. The functional congestion mechanism adjusts the distribution of cerebral blood flow (CBF) according to the functional activity of different brain areas [
31]. Thus, when the activity in a cortical region increases or decreases, the blood flow to that region changes correspondingly. This change is reflected in the WA of the fNIRS signal [HbO2] during the game, which is used to characterize the difficulty of the activity in specific cortical regions. In the present study, brain activation represents the mission-induced cortical activity associated with the task.
2.6. Extraction of Kinect Data Metrics
The json file of the Kinect device was read using Python to obtain the points corresponding to the upper limb, and eight behavioral metrics were calculated: upper-limb overall speed, upper-limb velocity standard deviation, median upper-extremity speed, upper-limb global acceleration, standard deviation of overall upper-limb acceleration, mean angle of left arm movement, and discrete and continuous stability ratios.
2.7. Statistical Analysis
One-way ANOVA can provide a comparative analysis of N classifications, which uses the mean of the values for a particular characteristic to see if there is a significant difference between groups. The experiment involved six brain region indicators, eight behavioral indicators, and two task states, and the multi-source data needed to be compared between and within groups. For the analysis of variance of data from two groups and above, one-way ANOVA met the need for statistical analysis in this study. To determine the degree of activation, one-way ANOVA tests were performed on the data from the subjects’ task and resting states to explore the implication of different states on the degree of activation of brain regions during the game. For Kinect data metrics, a one-way ANOVA test was conducted, looking at within-group interaction and completion rates to analyze the effect of behavior on different states of task completion. Pearson correlation analyses were conducted for behavior and brain region activation. The statistical significance level (p) for all analyses was set at 0.05.
4. Discussion
In the present study, an fNIRS device was used to monitor the activation of brain regions in elderly people with MCI during the completion of a VR task. fNIRS is a reliable functional brain-imaging technique, and its stability improves the monitoring of changes in neural activity during the experiment [
32]. The analysis of experimental data in this paper was based on wavelet amplitudes, which can reflect frequency information regarding changes in neural activity in the cerebral cortex at any moment [
33], and is suitable for analysis and discussion in the context of measurement tasks [
34].
This study estimated that the cognition level of older adults affects their ability to respond to product interactions. The results showed that the cognitive level of the group that completed the task was significantly higher than that of the group who left the task incomplete. The WA of brain areas in the completion group was also significantly higher than that of the incompletion group. We believe that this change in wavelet amplitude is related to the level of awareness. Brain areas are significantly activated when an individual participates in cognizant visual movement or develops an extraordinary idea [
35]. This is consistent with previous studies that used EEG devices to monitor users’ cognitive workload during interactions with VR products [
36]. Neither group had been exposed to such products previously, and both completed the experiment during the same timeframe. The results for both groups clearly showed a high level of cognition, mastery, and comprehension in the group that completed the task. Cay Anderson-Hanley, Christos A. Frantzidis et al. [
37,
38] alluded to a correlation between the interactive body and cognitive level, and stated that the user’s cognitive level is reflected in their interaction with the product. Thus, users with a higher cognitive level have an interactive responsiveness that is strong enough to allow for them to complete the task within the required time, while the group that left the task incomplete, as shown in all aspects of the results, had relatively weak cognitive abilities and showed a low level of product understanding during the task.
VR-based products can stimulate the brains of MCI older adults [
39,
40], especially older adult groups with relatively low cognitive levels [
14]. The results showed that two regions, ROL and LOL, were significantly activated in the incomplete group, and that OL is crucial for processing visual information and plays an important role in the coordination of language, perception, and abstraction [
41]. The products in the experiment had high brightness and pure, clear colors, and were in the main field of vision; thus, they provided a more intense visual stimulus to the occipital lobe (OL) area.
The behavioral data of the uncompleted and completed bunches showed that those who completed the task had a fundamentally higher general upper-appendage speed compared to the group who did not complete the task, in addition to an average up-per-appendage speed increase. The correlation between RPFC, LPFC, and upper-limb movements in the task suggests that VR-based products act together through multimodal senses, and subjects form their behavioral responses through information integration, which requires coordination between attention and sensorimotor functions. This may ex-plain the association between cognitive areas and movements. The cognition level also affects sensorimotor function, and a decrease in cognitive ability leads to a decrease in motor ability [
42]. Long-term video-game training enhances functional integration between attention and sensory-motor networks [
43], and VR-based products with audiovisual multidimensional sensory and rich picture effects are can promote complex brain system constructs, which is consistent with the results obtained from previous studies [
44].
In addition, Pearson correlation analysis revealed significant correlations between activation level, motor ability, awareness level, and level of completion, verifying the reliability of the multimodal combination data.
In conclusion, this analysis suggests that the experience of MCI older adults using VR products can be assessed by looking at brain function and motor function. WA analysis using fNIRS data can determine the perceived impact of subjects’ interaction with VR products and the effectiveness of these products. At the same time, labeling motor metrics can assist in assessing the awareness of ground behavior that the product produces in the subjects. This may have a positive impact on the evaluation of the product.
The current study has several limitations. The sample size included in this study is small, which may impact its accuracy. Many factors influence the effectiveness of a users’ use of the product; gender may be one of them. The elderly people recruited in this study were mainly female, which may have some influence on the results. Regarding fNIRS data preprocessing, subjectivity when identifying artifacts may influence the results.
5. Conclusions
In this study, fNIRS was used to monitor subjects’ brain signals at rest and during a task, and Kincet was used to monitor subjects’ behavioral patterns during a task, to evaluate VR-based products, focusing on user–task interaction responsiveness. The fNIRS data extracted from fNIRS were analyzed to show that the OL region of the brain was significantly more activated in older adults during the Beat Saber task than in the resting state. The data extracted from fNIRS showed that the OL region of the brain was significantly activated during the Beat Saber task compared to the resting state, and the upper-limb behavioral data recorded by Kinect were significantly correlated with the subjects’ functional brain regions. Therefore, when evaluating VR products, multimodal data can be used to determine the products’ impact on users, focusing on multiple perspectives. The results indicate that VR products activate some brains regions in MCI older adults. When comparing the degree of interactive response task completion in MCI older adults, it was found that a decrease in cognitive level causes a decrease in motor ability, which can lead to changes in brain activation. This is a new assessment method that can help strengthen the assessment of the products’ effectiveness in helping diagnosis and treatment, and can be applied to the field of rehabilitation product design to assist rehabilitation medications.
In future research programs, based on the application of this method in rehabilitation medicine, the selected subjects can be expanded. Other signal processing methods could also be examined to improve the data accuracy. When studying VR products, a combination of multimodal data can be used to further experiment with the interface composition, context, and interactions with VR products to find VR products that are more suitable for the MCI elderly.