A Study of the Effectiveness Verification of Computer-Based Dementia Assessment Contents (Co-Wis): Non-Randomized Study

Featured Application

Computer-based dementia assessment content (Co-Wis) has been developed for the purpose of efficient and accurate dementia prevention and screening. In this study, we verified the reliability and validity of the developed Co-Wis which we anticipate will play an important role in future big data-based dementia management systems.

Abstract

Computer-based neuropsychological assessments have many advantages over traditional neuropsychological assessments. However, limited data are available on the validity and reliability of computer-based assessments. The purpose of this study was to examine the reliability and validity of computer-based dementia assessment contents (Co-Wis). This study recruited 113 participants from Yeungnam University Medical Center in Daegu from June 2019 to December 2019 and received ethical approval. Participants were evaluated using standard and objective dementia cognitive test tools such as the Korean version of the Mini-Mental State Examination (K-MMSE), the Clinical Dementia Rating Scale (CDR), and the Standardized Seoul Neuropsychological Screening Battery-II (SNSB-II). To verify the effectiveness of Co-Wis, the concurrent validity, test–retest reliability (Pearson’s correlation coefficients), construct validity (Factor analysis), and signal detection analysis (ROC curve) were used. In most of the Co-Wis subtests, the concurrent validity and test–retest reliability showed statistically significant correlations (p < 0.05, p < 0.01). The factor analysis showed that Co-Wis assessed the most major cognitive areas (Tucker–Lewis Index (TLI) = 0.876, Comparative Fit Index (CFI) = 0.897, RMSEA = 0.88). Thus, Co-Wis appears clinically applicable and with high reliability and validity. In the future, we should develop tests to evaluate both standard data and big data-based machine learning.

1. Introduction

Dementia is most commonly diagnosed in the elderly. In the Republic of Korea, more patients are being diagnosed with dementia due to the rapid aging of the population [1]. In 2017, it was estimated that the prevalence of dementia among those aged 65 or above was 9.94% and that approximately 700,000 individuals had dementia [2]. Along with the advance of pharmacological treatments or interventions, modifiable dementia risk factors need to be addressed to prevent this disease [3].

Dementia is a degenerative disease, its early diagnosis is critical, and considerable effort has been invested in preventing and curing dementia [4]. To receive a diagnosis of dementia, diverse medical assessment encompassing interpretation of neuropsychological tests, imaging or pathology, medical history, and assessment of activities of daily living are needed [5]. Neuropsychological assessments are both used to identify signs of cognitive impairment that are consistent with dementia or with neurological disorders [6] and include traditional paper-based and computer-based assessment methods [7].

Computer-based neuropsychological assessments have many advantages over traditional assessments including conservation of both costs and time, consistency of results, accurate recording of responses, and the ability to automatically store and compare a person’s performance between assessment sessions [8]. However, most computer-based neuropsychological assessments are limited to those that use a keyboard and mouse as the primary inputs and besides there are limited data on the validity and reliability of them [9,10,11]. Although previous studies on patients with brain injuries evaluated the usefulness of computerized neurocognitive tests [12,13], further research is needed to overcome the limitations reported in the ethics, number of subjects, and heterogeneity of research methods. Thus, the present study aimed to validate the effectiveness of computer-based dementia assessment content (Co-Wis).

2. Materials and Methods

2.1. Clinical Trial Ethics (Institutional Review Board Approval)

The study procedures were performed in accordance with the Declaration of Helsinki and were approved by the Institutional Review Board of Yeungnam University (IRB No.: YUMC 2019-04-058-001). Coded identifiers were assigned to study subjects who visited the testing agency (medical institution) and provided written informed consent for study participation. The descriptive writing included the general content of the study, and the participants participated after fully understanding the study (Figure 1).

2.2. Clinical Trial Design

The purpose of this clinical trial was to confirm the effectiveness of the Co-Wis for the early diagnosis and prevention of dementia symptoms. Subjects were recruited from the Division of Clinical Psychology of the Department of Psychiatry at Yeungnam University Hospital in Daegu, Korea, from June 2019 to December 2019. All individuals with a subjective cognitive decline who attended the unit to undergo neurocognitive assessment were screened and among them, 113 subjects with an objective cognitive decline were included in the study. Participants were evaluated using standard and objective dementia cognitive test tools such as the Korean version of the Mini-Mental State Examination (K-MMSE), the Clinical Dementia Rating Scale (CDR), and the Standardized Seoul Neuropsychological Screening Battery-II (SNSB-II). Additionally, participants underwent a comprehensive clinical evaluation including a detailed medical history, neurological examinations, and a neuropsychological evaluation. The overall clinical trial progression is depicted in Figure 2.

Subjects were included in the present study if aged between 45 and 90, having an expressed informed consent, a diagnosis of Neurocognitive Disorder according to DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition; APA, 2013) criteria, a total score of 10–26 on the K-MMSE completed within one year before screening or after screening, a score of 0.5–2 at the CDR, a completed SNSB-II within one year before screening or after screening, and not having alterations in the sensory perception and motricity which could interfere with the trial. Subjects were excluded if they needed hospitalization or outpatient care due to severe medical conditions, showed persistent behavioral problems requiring frequent hospitalizations or inpatient cares, medical re-access was needed due to rapid dementia symptoms presentation, having other medical illnesses, using drugs affecting cognitive function, participated in clinical trials, or other studies that are related to this study within 3 months before screening.

This was a non-randomized control single-group clinical trial. Only a single-blind method was applied hiding the purpose of the study from the subjects since the researcher’s blindness was difficult to assure due to the neuropsychological test behavioral characteristics [14]. A total of 11 researchers participated in the clinical trial. The SNSB-II was administered at the screening stage and the Co-Wis at Visit 1 and 2. The test–retest was done in a time interval ranging between two weeks and three months. All tests were conducted in a quiet and focused environment to accurately examine the subject’s cognitive function [15,16,17]. Two tablet PCs with a built-in Co-Wis application were prepared. Tests were conducted in a pre-specified order and, if they were discontinued, they were re-administered from the point of interruption. Clinical trials with the same procedure will be revisited within two weeks or three months after the first trial with Co-Wis to avoid learning biases. Contents and procedures implemented in this study were standardized and conducted per the maximum number of participants.

2.3. Computer-Based Dementia Assessment Contents (Co-Wis)

Co-Wis evaluated attention, memory, language, visuospatial, and executive function domains [18] (

Table 1. Cognition area construction of Co-Wis.

Cognition Domain	Subtest	Subtest Item
Attention	Trail Making Test	A, B type
Memory	Figure Drawing	Immediate, Recall
	Word Memory	Immediate, Delay, Recognition
Language	Picture Naming
	Verbal Fluency	Animal, Market, Initial Sound
Visuospatial	Clock Drawing
	Puzzle Construction
Executive Function	Four Fundamental Arithmetic	Add, Subtract, Multiply, Divide
	Digit Coding
	Stroop	Word, Color

).

To verify the effectiveness of the contents, the concurrent validity, test–retest reliability, construct validity, and signal detection analysis were used (Figure 3).

The Co-Wis was designed as a computer-based tool for dementia prevention with the aim to make its contents accessible for psychology and rehabilitation clinics worldwide. Furthermore, we improved the system to support multiple dementia prevention sessions simultaneously and patients used the application created with the Unity game engine. At the initial stage of the setup, the user was prompted to enter the identifier provided by the therapist. The identifier was used by the application to find the patient within the assessment database. Each therapist had access to his or her Co-Wis account and was able to accept new patients and track the historical session data. Co-Wis was designed to predict dementia disease by applying machine learning based on big data collected through content execution (Figure 4 and Figure 5).

2.4. Data Analysis

Data were analyzed using IBM Statistical Package for Social Sciences (SPSS) version 25.0. To examine the concurrent validity of the Co-Wis, Pearson correlation coefficients were calculated for Co-Wis and the SNSB-II. A value of p < 0.05 was considered statistically significant [19,20]. The test–retest reliability of Co-Wis was evaluated. Pearson correlation coefficients were used to test for test–retest reliability within the clinical trial process [21,22]. Factor analyses (exploratory and confirmatory) were conducted for each cognitive domain subtest to investigate the Co-Wis composition feasibility [23,24]. To assess model fit, we examined the CMIN/df, Tucker–Lewis Index (TLI), the Comparative Fit Index (CFI), and the Root Mean Error of Approximation (RMSEA), which are less sensitive to sample sizes and reflect the simplicity of the model [25]. To evaluate the diagnostic utility, we examined the Co-Wis sensitivity and specificity using a receiver operating characteristic (ROC) curve [26].

3. Results

3.1. Descriptive Statistics

Demographic information and screening data of the participants are presented in

Table 2. Demographic information of study subjects.

Demographics	Male	Female	Total
	N (%) or Mean (SD)	N (%) or Mean (SD)	N (%) or Mean (SD)
Sex	46 (40.71)	67 (59.29)	113 (100)
Age
Range	47–86	45–86	45–86
45–59	7 (15.22)	7 (10.45)	14 (12.39)
60–69	7 (15.22)	21 (31.34)	28 (24.78)
70–79	27 (58.70)	27 (40.30)	54 (47.79)
Over 80	5 (10.87)	12 (17.91)	17 (15.04)
Total	70.11 (10.29)	71.08 (8.63)	70.68 (9.31)
Education
0–10	17 (36.96)	52 (77.61)	69 (61.06)
11–20	29 (63.04)	15 (22.39)	44 (38.94)
Total	11.50 (4.35)	7.37 (4.12)	9.05 (4.67)
K-MMSE
10–18	6 (13.04)	40 (86.96)	18 (15.93)
19–26	12 (17.91)	55 (82.09)	95 (84.07)
Total	23.22 (3.58)	22.99 (3.63)	23.08 (3.60)
CDR
0.5	28 (60.87)	37 (55.22)	65 (57.52)
1	13 (28.26)	25 (37.31)	38 (33.63)
2	5 (10.87)	5 (7.46)	10 (8.85)
Total	0.80 (0.48)	0.80 (0.42)	0.80 (0.44)

¹ Total sample (n) = 113.

. The average age of the sample was 70.68 years (SD = 9.31), and the average years of education was 9.05 years (SD = 4.67). The mean K-MMSE score was 23.11 (SD = 3.62) and the mean CDR score was 0.80 (SD = 0.44) (Table 2).

3.2. Concurrent Validity

To verify the public validity of Co-Wis construction, the study subjects also completed the SNSB-II within a year from the first test administration to compare results. Pearson’s correlation coefficient between the raw scores of the Co-Wis subtests and the corresponding SNSB-II subtests scores were calculated. All the Co-Wis and the SNSB-II subsets were significantly correlated, with correlation coefficients that ranged from 0.409 to 0.798. Upon examination of convergent validity, we determined that the Co-Wis and SNSB-II results were highly correlated (

Table 3. Concurrent validity of Co-Wis.

Co-Wis		Traditional Assessment (SNSB-II)		r
Subtest	Mean (SD)	Subtest	Mean (SD)
Word memory (immediate)	11.39 (4.98)	SVLT-E (immediate)	13.87 (5.64)	0.607 **
Word memory (delay)	1.69 (2.20)	SVLT-E (delay)	2.99 (2.77)	0.534 **
Word memory (recognition)	5.88 (2.49)	SVLT-E (recognition)	8.53 (3.02)	0.409 **
Word memory (total)	18.96 (8.39)	SVLT-E (total)	25.39 (9.76)	0.654 **
Figure drawing (immediate)	14.81 (4.78)	RCFT (copy)	26.14 (10.45)	0.501 **
Figure drawing (recall)	4.65 (5.46)	RCFT (delay)	8.50 (8.03)	0.592 **
Clock drawing	2.24 (0.98)	Clock drawing	2.32 (0.91)	0.639 **
Puzzle construction	1.84 (1.82)	Clock drawing	2.32 (0.91)	0.468 **
Four fundamental arithmetic (total)	6.19 (3.30)	Calculation (total)	8.74 (3.18)	0.776 **
Digit coding	7.91 (6.83)	Digit coding	8.74 (3.18)	0.776 **
Stroop (word)	40.72 (17.15)	K-CWST-60 (word)	51.87 (26.93)	0.494 **
Stroop (color)	18.56 (14.97)	K-CWST-60 (color)	27.89 (16.78)	0.655 **
Picture naming	6.01 (1.99)	S-K-BNT	10.11 (2.98)	0.664 **
Verbal fluency (total)	31.06 (16.99)	COWAT (total)	38.47 (21.17)	0.793 **
Trail making (A type)	47.91 (56.66)	Trail making (A type)	56.05 (55.88)	0.422 **
Trail making (B type)	135.04 (115.67)	Trail making (B type)	145.04 (112.74)	0.798 **

¹ * p < 0.05, ** p < 0.01

).

3.3. Test–Retest Reliability

In this study, the correlation coefficient between Visit 1 and 2 trials was calculated to verify Co-Wis test–retest reliability. The test–retest was done in a time interval ranging between two weeks and three months. Most items showed significant test–retest reliability, with scores ranging from 0.529 to 0.872. Trail making (A type)’s test–retest low reliability results 0.251 were shown (

Table 4. Test–Retest reliability of Co-Wis.

Co-Wis Subtest	Co-Wis Visit 1 Mean (SD)	Co-Wis Visit 2 Mean (SD)	r
Word memory (immediately)	11.39 (4.95)	13.42 (6.22)	0.772 **
Word memory (delay)	1.69 (2.20)	2.72 (2.79	0.626 **
Word memory (recognition)	5.88 (2.49)	6.46 (2.42)	0.683 **
Word memory (total)	18.96 (8.39	22.60 (10.47)	0.816 **
Figure drawing (immediately)	14.81 (4.78)	15.48 (5.10)	0.564 **
Figure drawing (recall)	4.65 (5.46)	8.82 (6.87)	0.553 **
Clock drawing	2.24 (0.98)	2.38 (0.81)	0.588 **
Puzzle construction	1.84 (1.82)	1.90 (1.86)	0.529 **
Four fundamental arithmetic	6.19 (3.30)	6.48 (3.39)	0.759 **
Digit coding	7.91 (6.83)	9.02 (7.64)	0.872 **
Stroop (word)	40.72 (17.15)	41.49 (15.82)	0.587 **
Stroop (color)	18.56 (14.97)	21.39 (15.57)	0.765 **
Picture naming	6.01 (1.99)	6.47 (1.99)	0.754 **
Verbal fluency (total)	31.06 (16.99)	32.75 (17.05)	0.829 **
Trail making (A type)	47.91 (56.66)	39.53 (40.35)	0.251 **
Trail making (B type)	135.04 (115.67)	109.49 (105.60)	0.696 **

¹ * p < 0.05, ** p < 0.01.

).

3.4. Construct Validity

In this study, we explored the factor structure of the Co-Wis subtests. Factor analyses were conducted to see if the Co-Wis subtests had a factor structure that corresponded to the presumptive cognitive domain. We extracted Co-Wis factors using the principal component formula and SPSS version 25.0. Through the Kaiser–Meyer–Olkin (KMO) and Bartlett tests, the adequacy of the selection of the subtests and the suitability of the factorial analysis model were verified, as were results of the commonality (

Table 5. Appropriateness selection of Co-Wis subtest and suitability of factor analysis model.

Measure	Result Value
Kaiser–Meyer–Olkin	0.928
Bartlett	0.000**

¹ Kaiser–Meter–Olkin (correlation coefficient), Bartlett (probability value), ** p < 0.001.

).

The Kaiser–Meyer–Olkin (KMO) result was 0.928, meaning that the number of variables and cases used in the exploratory factor analysis was appropriate. Bartlett’s test results suggested that factor analysis was appropriate (p < 0.05). The five criteria in the cognitive domain had a commonality of 0.3 for all the Co-Wis subtests, indicating a high commonality among the items (

Table 6. Commonality of Co-Wis exploratory factor analysis.

Co-Wis Subtest	Initial	Extraction
Word memory (immediately)	1.000	0.636
Word memory (delay)	1.000	0.635
Word memory (recognition)	1.000	0.456
Figure drawing (immediately)	1.000	0.374
Figure drawing (recall)	1.000	0.502
Clock drawing	1.000	0.574
Puzzle construction	1.000	0.527
Four fundamental arithmetic	1.000	0.610
Digit coding	1.000	0.734
Stroop (word)	1.000	0.641
Stroop (color)	1.000	0.691
Picture naming	1.000	0.576
Verbal fluency (category: animal)	1.000	0.645
Verbal fluency (category: market)	1.000	0.636
Verbal fluency (initial word: ㄱ)	1.000	0.783
Verbal fluency (initial word: ㅅ)	1.000	0.748
Verbal fluency (initial word: ㅇ)	1.000	0.728
Trail making (A type)	1.000	0.577
Trail making (B type)	1.000	0.710

).

Subsequently, confirmatory factor analysis was performed to verify the five sub-factors assumed in this study. The result of confirmatory factor analysis and goodness-of-fit are shown in

Table 7. Confirmatory factor analysis model validity of Co-Wis.

Co-Wis Subtest	χ2	df	CMIN/df	TLI	CFI	RMSEA
5 factor	264.696	142	1.864	0.876	0.897	0.088

and the path model is shown in Figure 6 (TMT A: trail making A, B type, FD I: figure drawing immediate, FD R: figure drawing recall, WM I: word memory immediate, WM D: word memory delayed, M R: word memory recall, PN: picture naming, VF A: verbal fluency category: animal, VF M: verbal fluency category: market, VF ㄱ: verbal fluency initial word: ㄱ, VF ㅅ: verbal fluency initial word: ㅅ, VF ㅇ: verbal fluency initial word: ㅇ, CD: clock drawing, PZ: puzzle construction, AR: four fundamental arithmetic, DC: digit coding, ST W: stroop word, ST C: stroop color). The coincidence of the five-factor model of Co-Wis was CMIN/df = 1.864, TLI = 0.876, CFI = 0.897, and RMSEA = 0.088. If the CMIN/df was less than or equal to three, it was considered acceptable. TLI and CFI were judged to have a good agreement when they were greater than 0.90. The RMSEA was considered good when below 0.06 and with a good agreement when below 0.08.

3.5. Signal detection analysis

All the Co-Wis subtest were significantly value, with area under the curves that ranged from 0.694–0.860 (Figure 7,

Table 8. Receive operating characteristic curve of Co-Wis.

Co-Wis Result Subtest	Area	Standard Error	p Value	95% CI
				Lower	Upper
Trail making A type	0.800	0.043	0.001	0.715	0.885
Trail making B type	0.824	0.043	0.001	0.739	0.909
Figure drawing immediately	0.779	0.043	0.001	0.598	0.791
Figure drawing recall	0.694	0.049	0.001	0.770	0.912
Word memory total	0.860	0.035	0.001	0.791	0.929
Picture naming	0.702	0.051	0.001	0.602	0.801
Verbal fluency total	0.848	0.037	0.001	0.776	0.920
Four fundamental arithmetic	0.800	0.044	0.001	0.714	0.886
Digit coding	0.856	0.036	0.001	0.785	0.927
Stroop semantic	0.784	0.044	0.001	0.698	0.871
Stroop color	0.840	0.037	0.001	0.768	0.912
Clock drawing	0.767	0.049	0.001	0.672	0.862
Puzzle construction	0.753	0.047	0.001	0.661	0.845

). We also analyzed and presented the sensitivity and specificity of each subtest (

Table 9. Sensitivity and specificity of Co-Wis.

Co-Wis Result Subtest	Sensitivity (%)	Specificity (%)	Cuff-Off Score
Trail making A type	75.0	72.3	31
Trail making B type	77.1	78.5	89
Figure drawing immediately	70.8	70.8	16
Figure drawing recall	68.8	63.1	3
Word memory total	81.3	83.1	18
Picture naming	70.8	53.8	7
Verbal fluency total	77.1	72.3	29
Four fundamental arithmetic	79.2	72.3	7
Digit coding	83.3	76.9	7
Stroop semantic	70.8	72.3	43
Stroop color	79.2	73.8	16
Clock drawing	66.7	75.4	3
Puzzle construction	83.3	58.5	3

).

4. Discussion

Cognitive assessment could favor early detection of neurological disorders in comorbidity with psychiatry disorders and the necessity of computer-based neuropsychological tests rather than traditional paper-based ones is increasing [27]. The purpose of this clinical trial was to verify the validity and effectiveness of the Co-Wis compared with those of the SNSB-II, similar to a previous study on elderly participants comparing the prototype of a machine cognitive test with a traditional paper-based psychometric tool [28,29]. To confirm the consistency of the test results, we also checked the test–retest reliability between Visit 1 and 2 in a time interval between two weeks and three months to avoid fatigue and carryover effects linked to repeated measurements [30,31,32]. Exploratory and confirmatory factor analysis was conducted to investigate the factor structure of the Co-Wis subtests. We found that the Co-Wis was significantly correlated with the SNSB-II. Co-Wis has been tested and retested in two trials to ensure the stability of the test results. Our findings showed that Co-Wis results were significantly correlated with the SNSB-II scores. Test–retest reliability was assessed to ensure the consistency of the Co-Wis test results. The trail making test A type showed a test–retest reliability value of 0.251, which is considered as low. Given that the trail making test measured the time until test completion, this result seems to reflect the presence of a practice effect bias of the participants [32,33]. Therefore, it is reported that the participants showed a big difference in trail making test A between the first and repeated trials. Before using exploratory factor analysis, the KMO measure and Bartlett test were used to examine the suitability of the data for factor analysis. The KMO measure 0.928 and Bartlett test was 0.000. KMO measures have values greater than 0.90 (marvelous), 0.80–0.89 (meritorious), 0.70–0.79 (middling), and 0.60–0.69 (mediocre). A value between 0.50 and 0.59 was classified as poor and unacceptable if below 0.5. Bartlett’s test of sphericity tests the null hypothesis and the alternative hypothesis for the existence of commonality and small values (less than 0.05, rejecting the null hypothesis) of the significance level indicate that factor analysis may be useful for data analysis. Bartlett’s test of sphericity tests the null hypothesis and the alternative hypothesis for the existence of commonality and small values (less than 0.05, rejecting the null hypothesis) of the significance level indicate that factor analysis may be useful for data analysis [23]. Using the exploratory factor analysis results of the Co-Wis subtest, we confirmed a commonness of 0.3 over for all subtest items [23]. The cognitive domain was derived from the five components of attention, memory, language, visuospatial, and executive function. Confirmatory factor analysis showed that the five factors of the Co-Wis were acceptable. Considering the standardization coefficients of each factor, all factor coefficients showed statistically significant (p < 0.001), thus ensuring the stability of the structural model [24,34,35]. In the signal detection analysis, the overall scale showed good sensitivity and specificity, so that the Co-Wis contents could be identified as well as existing dementia assessment tools. Our results are in line with previous studies of Park and Heo (2017), Park et al. (2017), and Lee et al. (2019) on computerized neurocognitive test development and assessment of the elderly and patients with mild cognitive impairment or stroke. Additionally, Co-Wis showed high reliability and validity compared with the ones of paper-based assessments. This study confirmed the effectiveness of the Co-Wis and results will be of great value in the field, as the existing studies were conducted on small samples, without IRB approval and formal procedures [36,37,38]. The Co-Wis contents will direct guidelines for big data-based dementia care system development to solve challenging medical demands in dementia medical health experts. However, further research and data collection are needed. First, Co-Wis still requires a standardization score and big data collection in various environments. Secondly, to develop a computerized neurocognitive test that is effective in predicting and detecting dementia and multi-national data, the development of an English version and the verification of its validity and reliability are issues that need to be approached in the future [39]. Thirdly, in the future, it would be necessary to further verify the role of Co-Wis in dementia prevention and diagnosis by comparing patients with healthy controls (RCT). Co-Wis could play an important role in advancing dementia prevention, screening, and care. Thus, using big data collection and machine learning analysis, we may be able to reduce the social impact of dementia while also mitigating the symptoms of this disease.

5. Patents

Clupea, Inc. Co-Wis. KIPO Trademark application 40-2018-0112761, August 14, 2018, and Dementia testing management server using machine learning and dementia testing method using the same. KIPO Patent 10-2019-0171912, December 20, 2019.