Dementia Classification Using Deep Reinforcement Learning for Early Diagnosis

Abstract

Neurodegeneration and impaired neuronal transmission in the brain are at the root of Alzheimer’s disease (AD) and dementia. As of yet, no successful treatments for dementia or Alzheimer’s disease have indeed been found. Therefore, preventative measures such as early diagnosis are essential. This research aimed to evaluate the accuracy of the Open Access Series of Imaging Studies (OASIS) database for the purpose of identifying biomarkers of dementia using effective machine learning methods. In most parts of the world, AD is responsible for dementia. When the challenge level is high, it is nearly impossible to get anything done without assistance. This is increasing due to population growth and the diagnostic period. Two current approaches are the medical history and testing. The main challenge for dementia research is the imbalance of datasets and their impact on accuracy. A proposed system based on reinforcement learning and neural networks could generate and segment imbalanced classes. Making a precise diagnosis and taking into account dementia in all four stages will result in high-resolution sickness probability maps. It employs deep reinforcement learning to generate accurate and understandable representations of a person’s dementia sickness risk. To avoid an imbalance, classes should be evenly represented in the samples. There is a significant class imbalance in the MRI image. The Deep Reinforcement System improved trial accuracy by 6%, precision by 9%, recall by 13%, and F-score by 9–10%. The diagnosis efficiency has improved as well.

1. Introduction

Dementia refers to a collection of symptoms produced by brain abnormalities. There are currently around 46.8 million people living with dementia worldwide, with this number expected to increase to 131.5 million by 2050 [1,2]. There are numerous forms of dementia, the most prevalent of which is Alzheimer’s disease (AD). Although significant efforts have been made to decipher the process of dementia and to create better therapies, a precise diagnosis of dementia is still a challenging task. In an attempt to develop an effective method of detecting AD, several computer-aided (CAD) systems have been examined [3]. Such systems rely on machine learning and include magnetic resonance imaging (MRI), functional MRI (fMRI), structural MRI (sMRI), and positron emission tomography (PET) [4,5]. Dementia associated with Alzheimer’s disease [6,7] is typically classified as one of the following:

• Mild Cognitive Impairment: Many persons have a loss of memory as they become older, while others get dementia.
• Mild Dementia: Individuals with intermediate dementia frequently experience cognitive impairments that disrupt their daily life. Symptoms of dementia involve forgetfulness, insecurity, personality changes, disorientation, and difficulty doing everyday tasks.
• Moderate Dementia: The patient’s daily life becomes significantly more complicated, necessitating additional attention and support. The symptoms are comparable to mild-to-moderate dementia. Even combing one’s hair may require further assistance. Patients may also have substantial personality changes, such as becoming paranoid or irritable for no apparent reason. Sleep disturbances are also possible.
• Severe Dementia: Throughout this period, symptoms might worsen. Patients often lack communication skills, necessitating full-time treatment. One’s bladder function may be compromised, and even simple events such as holding one’s head up in a decent position and sitting down become difficult.

Deep learning [8,9] is a subset of artificial intelligence (AI) that uses numerous non-linear processing unit layers for performing tasks such as feature extraction and transformation. Each successive layer accepts as input the output of the previous layer [10]. Learning algorithms can be supervised (for example, classification) or unsupervised (for example, regression). Deep learning frameworks are critical for semantic video segmentation, image processing, object detection, and facial recognition, among other applications.

Deep learning (DL) appears to have a framework focused on learning several levels of data-based interpretations. Attributes are derived from relatively low qualities to construct a hierarchical representation. Typically, DL is primarily dependent on data representation learning. A vector of density-based values for every pixel, or features like custom shapes and edge clusters, can be used to represent a picture. A few of these characteristics help to depict data better. Instead of handmade features, deep learning techniques employ sophisticated algorithms for hierarchy-based feature extraction that best reflect data at this point. Rapid innovations in image analysis have aided the advancement of fast-changing advanced technologies [11,12,13,14,15,16]. Image processing has grown in prominence, particularly in the medical field. In fact, DL is a method that saves time and improves performance when compared to previous methods. While conventional approaches can only process single-layer photos, deep learning can process multi-layer images with outstanding performance. Deep learning’s most essential characteristic is that one can self-discover the variables that must be manually entered, allowing it to process images inside a single pass. This study aims to provide software for classifying dementia in MRI scans using deep learning [17,18,19].

1.1. Gap in Previous Work

Prior research found that the primary barrier to early dementia detection was an imbalance across classes, which resulted in an increase in the percentage of false positives and a decrease in precision. This is the main gap observed in previous research. Other reasons for the lack of precision include:

• Features overlapping as a result of the overlap of pixels and the rise in the difficulty of fitting in classifiers.
• Presence of noisy features, which enhance the error rate in early dementia detection.

With the help of noisy feature extraction, early dementia detection can become more accurate.

1.2. Contribution of the Present Research

The goal of this study is to reduce class disparities and the false positive rate of dementia stages in the early stages. Deep learning-based active reinforcement learning places a secondary emphasis on learning features and outlier samples. Such an emphasis improves the accuracy of the proposed method performance metrics and increases dementia stage detection in the early stages. In brief, the following contributions have been achieved.

I.

Class imbalance is improved through deep reinforcement learning and an iterative policy by balancing instances of each class.

II.

A convolution neural network with deep reinforcement learning is used to improve segmentation.

III.

Structure-based learning is used to improve the classification.

The main problem in medical imaging is the dataset. A class imbalance problem arises when the problem is exacerbated by the use of more expensive MRI images. Some researchers solved this problem using augmentation and smote methods, which increase noise, but our main contribution in the proposed work is to use reinforcement learning with a deep convolution network, which generates instances based on the learning process. The rest of this article is organized as follows: Section 2 contains the related work for the illustration of the Dementia classification scheme. Section 3 goes into great detail about the proposed methodology. Section 4 gives insight into the experimental findings and discussions, while Section 5 draws conclusions.

2. Related Work

Deep learning has received much interest because of its potential use in identifying Alzheimer’s disease. Numerous DL algorithms have recently been presented as Alzheimer’s disease diagnostics aids, supporting clinicians in making educated treatment decisions. In this area, we provide some works that seem directly connected to this study.

The author’s De and Chowdhury [20] used Diffusion Tensor Imaging (DTI) in 3D to distinguish Alzheimer’s disease, CN, EMCI, and Late Mild Cognitive Impairment (LMCI) for the first time in a direct four-class classification test. Individual VoxCNNs are trained on MD values, EPI intensities, and FA values extracted from a 3D DTI scan volume. Nawaz et al. [21] proposed Alzheimer’s stage detection based on in-depth features, using early layers imported from an AlexNet model and CNN deep features extracted. They applied ML techniques such as those of k-nearest neighbors (KNN), random forest (RF), and support-vector machines (SVM) to the recovered deep features. Evaluation results indicate that a feature-based model outperforms handmade and deep-learning approaches by 99.21%. Mohammed et al. [22] evaluated multiple machine-learning techniques for detecting dementia using a big, publicly available OASIS dataset. They evaluated AD detection using CNN models (ResNet-50 and AlexNet) and hybrid DL and ML techniques (ResNet-50+SVM and AlexNet+SVM). After correcting missing data in the OASIS database using t-SNE, i.e., the Stochastic Embedding method, they represented high-dimensional data in low-dimensional space using t-SNE. ML algorithms in every way you can think of. All these metrics were met or exceeded by the random forest method, which had an overall accuracy of 94%. Ghosh et al. [23] demonstrated the applicability of DL and its accompanying techniques, which could have a far-reaching impact on the diagnosis of dementia phases and lead to future treatments for image retrieval. In the study databases, MRI scans were categorized into four degrees of AD severity: no dementia, very mild dementia, mild dementia, and moderate dementia. Each patient’s dementia stage was classified via the CNN approach based on these four imaging criteria. Using the CNN-based model, the researchers could accurately diagnose and predict the different phases of Alzheimer’s disease. Abol Basher et al. [24] showed a way to diagnose AD by taking slice-wise volumetric characteristics from structural MRI data of the right and left brain hemispheres. The suggested method uses both CNN and DNN models. The classification networks were trained and tested using the image’s volumetric features. Using this proposed method, the researchers in [24] got a TA weighted average classification accuracy of 94.02 percent for the right and 94.82 percent for the left. Also, the AUC for the right was 90.62 percent, while 92.54 percent for the left. Herzog and Magoulas [25] focused on using transfer learning architectures for MRI image categorization. The basic model and fully connected layers or functions serve as the foundation. To look at changes in the brain, scientists use segmented images of brain asymmetry, which were recently shown to be a sign of early-onset dementia. The diagnostic strategy is made using these data. Murugan et al. [26] used a CNN to create a structure for recognizing AD features in MRI scans. By utilizing the four stages of dementia, the proposed framework achieves exceptional disease probability mapping from the local anatomical brain to an understanding of specific AD risk. To maintain a balanced distribution of instances, each class must include the same number of each kind. Class imbalance is a severe issue in the MRI picture database. Patients’ dementia stages are detected using an MRI-based technique. Al-Shoukry et al. [27] reviewed some of the most recent research on Alzheimer’s disease and explored how DL can aid in the early diagnosis of the condition. Pan et al. [28] suggested two effective sampling procedures for improving data distributions. Adaptive-SMOTE is a method that enhances the SMOTE approach by selecting risk and internal data groups from the minority class and constructing a new minority class based on the selected data. This is done to prevent the categorization boundary from expanding and to preserve the distributional properties of the original data. By contrast, Gaussian Oversampling combines the Gaussian distribution with a reduction in the number of dimensions. This shortens the tail of the Gaussian distribution. Chen et al. [29] analyzed three modern architectures: the VGG16, the ResNet-152, and the DenseNet-121. A total of 1315 participants were included. They utilized optimization techniques to train the neural network (NN) to deal with the limited data, including multiple dementia types. Their efforts resulted in a computerized and standardized evaluation of a person’s dementia test result and severity level. The work exceeded both the declared state-of-the-art and human accuracies, with screening levels of 96.65% accuracy and score accuracies as high as 98.54 percent.

Li et al. [30] analyzed MRI scans from 2146 patients (1,343 for verification and 803 for training). A deep learning algorithm was created and validated to predict the progression of MCI participants to Alzheimer’s disease in a time-to-event analysis setting.

Ucuzal et al. [31] employed deep learning to construct open-source software for the classification of dementia in MRI images. A framework for deep learning is used to develop a model capable of distinguishing between dementia patients and healthy individuals. The findings demonstrate that the proposed technique can identify individuals suspected of having dementia.

Tsang et al. [32] summarized the current state-of-the-art machine-learning approaches for dementia in healthcare informatics. The research analyzes and evaluates current scientific methodologies to identify significant challenges and obstacles when dealing with vast quantities of health data. While machine learning has demonstrated promise in data analysis for dementia care, there have been few attempts to utilize advanced machine learning techniques to leverage unified heterogeneous data. Raza et al. [33] presented a novel ML-based method for identifying and monitoring AD-like diseases. The DL analysis of MRI scans is used to diagnose AD-like disorders, together with an activity monitoring framework that records participants’ everyday tasks using body-worn inertial sensors. The activity surveillance system provides a foundation for assisting patients with daily tasks and gauges their sensitivity based on their activity level. Kaka and Prasad [34] proposed a new supervised ML technique for diagnosing Alzheimer’s disease using MRI. They used the OASIS and ADANI datasets. They obtained an image with improved contrast, applied Fuzzy C Mean clustering to segment tissues, and extracted features from segmented brain tissues. They used the Correlation-Based Feature Selection (CFS) techniques to reduce the dimensionality of features.

Table 1. Study of literature for Deep learning-based Dementia.

Ref	Year	Aim	Methodology	Feature	Dataset	Limitation
ADNI Dataset
[20]	(2021)	Alzheimer’s disease(AD) classification using diffusion tensor imaging	DTI processing in three dimensions using CNNs and RF Rank modulated decision fusion for merging the outputs of CNNs and RF.	Fractional Anisotropy (FA) Mean Diffusivity (MD) Echo Planar Imaging (EPI)	ADNI database	Study of literature for Deep learning-based Dementia
OASIS Dataset
[21]	(2021)	Alzheimer’s disease stages detection system in real time	Efficient framework for transfer learning	Deep Learning	OASIS dataset	Overfitting and class imbalance
[31]	(2019)	Deep-learning-based imaging classification	DLB and AD	Deep-learning-based imaging classification	OASIS-2 data	This investigation was conducted at a single facility. Despite the small sample size, the accuracy was deemed adequate.
OASIS AND ADNI Dataset
[25]	(2021)	Using DL technology, Alzheimer’s illness can be recognized automatically using MRI data.	3D brain MRI with DL	CNN Network Training	ADNI and OASIS database
[34]	(2022)	Alzheimer’s Disease Diagnosis and Monitoring Using Classical Machine Learning Techniques	A unique method for diagnosing and monitoring AD using ML	A correlation-based feature selection (CFS)	OASIS and ADNI dataset	A large dataset is a significant challenge.
Other Dataset
[24]	(2021)	The diagnosis of AD in (Mild Cognitive Impairment) MCI patients using DNN	Volumetric Features Extraction and DNN	Structural MRI features	GUARD	Individual Modalities
[29]	(2020)	Automatic detection and classification of dementia using DL	Transfer learning and Deep neural networks	Clockdrawing test’ (CDT)	From July 2018, a neuropsychiatric clinic will be located in Nürnberg, Germany.	Overfitting

summarizes important contributions to the literature on DL-based dementia.

Literature Review and Observation

The following analysis shows that there are problems in databases and claims that the proposed approach will significantly improve these problems. In this article, we use three datasets.

• Kaggle-based tiny dataset
• OASIS Imbalance dataset
• Balanced ADNI dataset

One of the problems that lead to poor performance in classification is the categorization of a dataset that has unbalanced categories. Except for the accuracy scale, the rating scales require that the categories be distributed evenly throughout the available slots. In this article, the OASIS dataset includes three classifications that are not evenly distributed [21,24].

The majority of medical datasets do not include enough images to fulfill the requirements for deep learning models, which need a large dataset. Consequently, the issue may be remedied by producing further images using the same information. In addition [24]

When the dataset is tiny, the issue of overfitting arises owing to a lack of data during the training phase. This problem is caused by a lack of data [21].

This article, https://www.kaggle.com/datasets/tourist55/alzheimers-dataset-4-classes-of-images, (accessed on 5 October 2022) uses this dataset to resolve problems using the proposed approach, as well as the OASIS dataset.

“Extensive preprocessing,” which often takes place in architectures other than CNNs, is applied to imaging features like texture, forms, or cortical thickness and regional characteristics, respectively [29].

3. Proposed System

3.1. Dataset

Figure 1 depicts in step 1 the input Dementia dataset. The data came from https://www.kaggle.com/datasets/tourist55/alzheimers-dataset-4-class-of-images (accessed on 5 October 2022). The data consists of MRI images. The data consists of four groups of images for training and testing: mildly deranged, moderately deranged, not deranged, and very mildly crazy. The ADNI dataset was gathered from over 40 radiology hubs, and it comprised 509 cases in total (137 AD cases, 76 MCIc cases, 134 MCInc cases, and 162 CN cases) [35]. The time frame to supervise the transition to AD was 18 months. ADNI was previously utilized in many studies to categorize AD and understand its transformation into AD. http://www.oasis-brains.org/longitudinal_facts.html (accessed on 5 October 2022) [36]. The Oasis longitudinal dataset contains 15 features of 374 patients aged 10, 28, or 5 of 20 between 60 and 90 years old, of whom 37 are converted, 190 are demented, and 147 are non-demented. There are a lot of different topics to choose from in each class. Figure 1 shows the main idea behind the DRLA technique that was suggested. A system for diagnosing the disease is trained using data consisting of samples with labels prior (i.e., (Xl, Yl)) and samples selected and labeled during reinforcement learning. During the learning process, an actor-network uses the learned policy and the current state to choose the most relevant dementia samples from unlabeled trained data (i.e., Xu). An annotator then assumes responsibility for making notes on the selected samples. Because of this, one can get more and more training data to improve the Xg-Boost classifier. Finally, a critic network can be taught to determine whether or not the selection of the actor-network improved the classifier’s performance. By using a deep reinforcement learning technique to train the critic and actor networks, one can choose and label the most specific example that seems to help train an effective classifier, thereby improving the classifier’s performance.

3.2. CNN Layer

Table 2. Details of CNN Layers.

Layer	Output	Kernel Size	Activation Layer
Conv2D-1	128	3 × 3	RELU
Conv2D-2	64	3 × 3	Sigmoid
Conv2D-3	32	3 × 3	RELU
Conv2D-4	16	2 × 2
		Pool size(2 × 2)
Dropout			0.2

discusses the CNN layers details in brief. With a kernel size of 45 × 45 × 45 and an image size of 8 blocks, convolution operations are carried out on the image. Strides 1, 2, and 3 have four convolutional layers, and the first filter has 32 3 × 3 kernels. The kernels act as feature finders and convolve with the picture to generate a set of features that have been convolved. Regional connection of the neurons to the previous volume is consequently enforced in neural networks because the size of the kernel denotes a neuron receptive field. Figure 1 of the CNN architecture shows the final arrangement.

3.3. Training the Classifier

In Figure 1, step 2, and step 3, classifiers are trained. Let Id serve as the variable of a classifier network. At the start of the learning phase, one can pre-train the classifier using a labeled training data set. Then, deep reinforcement learning is used to choose and annotate samples, and then training of the classifier is done further to improve its performance.

The Dementia training samples are $D_l={\left\{d_i\right\}}_{i=1}^{n_l}$and the labels $L_l={\left\{L_i\right\}}_{i=1}^{n_l}$, where $n_l$ expresses the number of labelled data. The cross-entropy loss is reduced by training the classifier network, which is characterized as:

$${ℒ}_{ce}=-{\displaystyle \sum }_{i=1}^{n_l}{\displaystyle \sum }_{j=1}^{M}I\left(L_i=i\right)logPr\left(l_i=i|x_i;{\theta }_d\right)$$

(1)

$I\left(y_i=i\right)$: determines whether the label of an ith sample is either j or not, M represents the classes number, and$I$ (·) expresses the indicating function.

$Pr\left(y_i=i|x_i;{\theta }_j\right)$ represents Xg-Boost output of a classifier provided xi for the jth class, denoting the likelihood of xi belonging to label j as determined by the classifiers with parameters ${\theta }_d$.

3.4. Extreme Gradient (XG) BOOST Learning with Deep Reinforcement

Figure 1, step 2, depicts a novel approach for developing a policy that leads the actor-network in the selection of samples for annotation. The next sections go over the suggested approach in depth.

$$X{G}_{ij}=Pr\left(y_i^u=j|n_i^u;{\theta }_d\right),$$

(2)

Here $y_i^u$ represents the uncertain label, and $n_i^u$ represents an unlabeled training sample.

Action: Define θa as the actor network’s parameters. Since the objective of an actor-network would be to annotate training samples out of unlabeled training data, we specify the action in terms of $\in {\left(0,1\right)}^{N_u}$, where every other component refers to an unlabeled training instance. Also, as an activation function for every element, the sigmoid function is used to produce a value amid 0 and 1.

A policy $\pi \left(S,{\theta }_a\right)$is learnt to produce action in response to a state S. After collecting the action vector; one can rate all candidate image samples, excluding those that have already been picked, and pick the very first specimens with the greatest values for annotations. The annotator’s several sorts of samples and labels are described by ${\left\{n_t^s,l_t^s\right\}}_{i=1}^{n_s}$ whereas the enhanced labeled training data are represented by

$$(X_l,Y_l)\text{:}=(L_l,Y_l) \cup {\left\{x_t^s,y_t^s\right\}}_{i=1}^{n_s}.$$

State Transition. One can upgrade the classifier using the enriched training data ($X_l,Y_l)$ by minimizing the cross-entropy loss in Equation (1). Once the selected samples have been annotated and incorporated to the labeled training data, the modified classifier is then used to generate the new state matrix S# depending on Equation (2).

Reward. We recommend that the actor emphasize more on those samples that are very likely to be misidentified by the classifier in terms of improving the classifier’s performance. To do this, we devise a novel incentive mechanism based mostly on the annotator’s true labels. We define ${\widehat{k}}_i$ as the predicted label received from the classifier, i.e., ${\widehat{k}}_i=ma{x}_{j}Pr\left(y_i^s={\widehat{k}}_i|x_i^s;{\theta }_d\right)$ for the sample selection $x_i^s$ and $k_i$as the true label gained from the annotator.

The reward is as follows:

$$r\left(S,a\right)=\frac{1}{n_s}{\displaystyle \sum }_{i=1}^{n_s}Pr\left(y_i^s={\widehat{k}}_i|x_i^s;{\theta }_d\right)-Pr\left(y_i^s=k_i|x_i^s;{\theta }_d\right)$$

(3)

If a sample $x_i^s$ is correctly classified, then $k_i={\widehat{k}}_i$, and

$$Pr\left(y_i^s={\widehat{k}}_i|x_i^s;{\theta }_d\right)-Pr\left(y_i^s=k_i|x_i^s;{\theta }_d\right).$$

A big reward, on the other hand, suggests that the selected samples have been erroneously categorized. This suggests that the classifier should pay more attention to the samples with incorrect predictions. As a result, the actor-network is incentivized to choose these image samples.

Reinforcement learning in state S tries to maximize the anticipated future reward, which would be represented like a Q-value function. The Q-value function is being used to assess the state-action pair (S, a), analogous to Q-learning in conventional reinforcement learning, and is described by the Bellman equation as

$$Q\left(S,a;{\theta }_c\right)=E\left[\gamma Q\left(S^{\prime },\pi \left(S^{\prime };{\theta }_a\right);{\theta }_c\right)+r\left(S,a\right)\right]$$

(4)

To estimate the Q-value function, we use a critic network with parameter ${\theta }_c$. By considering the following problem, we want to establish a greedy policy for actors via deep Q-Learning.

$$\underset{{\theta }_a}{\max }(Q\left(S,a\right);{\theta }_c)$$

We describe the estimated Q-value function

$$\widehat{Q}\left(S,a;{\theta }_c\right)=\gamma Q\left(\gamma Q\left(S^{\prime },\pi \left(S^{\prime };{\theta }_a\right);{\theta }_c\right)+r\left(S,a\right)\right)$$

(5)

to solve the following problem to train the critic network.

$$\underset{{\theta }_c}{max}{\left(\widehat{Q}\left(S,a;{\theta }_c\right)-Q\left(S,a;{\theta }_c\right)\right)}^2$$

Target Networks for Training: To stabilize the critic networks and training of an actor, we use a separate targeted system to compute $\widehat{Q}\left(S,a;{\theta }_c\right)$ as described in equation (4). The function $\widehat{Q}\left(S,a;{\theta }_c\right)$ is dependent on the proposed state S’, the actor to output action $\left(S^{\prime };{\theta }_a\right)$, and the critic to assess $\left(S^{\prime },\pi \left(S^{\prime };{\theta }_a\right)\right)$ in Equation (5).

To calculate $\widehat{Q}\left(S,a;{\theta }_c\right)$, we use a separate target actor network specified by ${\theta }_{a^{\prime }}$ and a separate target criticism network parameterized by ${\theta }_{c^{\prime }}$. As a consequence, Equation (5) is rewritten as:

$$\underset{{\theta }_c}{\min }\left(\gamma Q^{\prime }\left(S^{\prime },{\pi }^{\prime }\left(S^{\prime };{\theta }_{a^{\prime }}\right);{\theta }_{c^{\prime }}\right)+r\left(S,a\right)-Q(S,a;{\theta }_c\right){)}^2$$

(6)

Here, ${\pi }^{\prime }$(·; ${\theta }_{a^{\prime }}$) expresses the target policy assessed by target actor, and $Q^{\prime }$(·; ${\theta }_{c^{\prime }}$) represents the target critic’s function. The Deep deterministic policy gradient (DDPG) algorithm can be used to resolve this issue. The critic and the target actor are updated at the end of each era.

The critic and the target actor are updated at the end of each epoch given as:

$${\theta }_{a^{\prime }}:=\lambda {\theta }_a+\left(1-\lambda \right){\theta }_{a^{\prime }},{\theta }_{c^{\prime }}:=\lambda {\theta }_c+\left(1-\lambda \right){\theta }_{c^{\prime }}$$

(7)

A trade-off element is represented by $\lambda \in \left(0,1\right)$. Samples (S, a, S’, r) are placed in a replay buffer to train the critic and actor inside the mini-batch case. As a result, one can choose data from the replay buffer evenly to update the critic and actor networks. The Algorithm 1 describes our recommended approach.

Algorithm 1. DRLA (Deep learning Reinforcement Learning for Active learning)

Input: Training Data With Labels and testing data without labels

Output: Efficient Classified Dementia Stages

1. Initialize training dataset $\left({\mathit{X}}_{\mathit{l}},{\mathit{Y}}_{\mathit{l}}\right)$

2. Train Xg-Boost Classifier and obtain $\mathit{f}\left(\mathit{x},\mathit{\theta }\right)$

3. Determine the State S by Preprocessing of images based on Equation (2)

4. Pick unlabeled training Sample ${\mathit{n}}_{\mathit{s}}$ unlabeled training Sample ${\left\{{\mathit{x}}_{\mathit{i}}^{\mathit{s}}\right\}}_{\mathit{i}=\mathit{1}}^{{\mathit{n}}_{\mathit{s}}}$ In accordance with the actor (training phase) $\mathit{a}=\left(\mathit{S};{\theta }_a\right)$ is updated

5. To get ${\left\{{\mathit{x}}_{\mathit{i}}^{\mathit{s}},{\mathit{y}}_{\mathit{i}}^{\mathit{s}}\right\}}_{\mathit{i}=1}^{{\mathit{n}}_{\mathit{s}}} , \mathit{a}\mathit{n}\mathit{n}\mathit{o}\mathit{n}\mathit{a}\mathit{t}\mathit{e} {\left\{{\mathit{x}}_{\mathit{i}}^{\mathit{s}}\right\}}_{\mathit{i}=\mathit{1}}^{{\mathit{n}}_{\mathit{s}}}$

6. Using $\left({\mathit{X}}_{\mathit{l}},{\mathit{Y}}_{\mathit{l}}\right)=\left({\mathit{X}}_{\mathit{l}},{\mathit{Y}}_{\mathit{l}}\right) \mathit{U} {\left\{{\mathit{x}}_{\mathit{i}}^{\mathit{s}},{\mathit{y}}_{\mathit{t}}^{\mathit{s}}\right\}}_{\mathit{i}=\mathit{1}}^{{\mathit{n}}_{\mathit{s}}}$ update the classifier Parameters ${\mathit{\theta }}_{\mathit{n}}$,

7. Compute $\mathit{t}\mathit{h}\mathit{e} \mathit{S}\mathit{t}\mathit{a}\mathit{t}\mathit{e} {\mathit{S}}^{\prime }$ using Equation (2) and the reward r using Equation (3)

8. Save all instances ${\mathit{X}}_{\mathit{l}}<-\left(\mathit{S},\mathit{a},{\mathit{S}}^{\prime },\mathit{r}\right)$

9. Train by Xg-Boost

10. Complete analysis of performance metrics

4. Experimental Results and Discussion

This section demonstrates the experimental results of the proposed approach. We have performed the DRL-XGBOOST experiments on the datasets. The proposed method assesses results using accuracy, precision, recall, and F1-Score. To carry out the procedure, Windows 10 is installed, along with an NVIDIA GeForce graphics card, a CPU running at 2.70 GHz, and 32 GB of RAM. PyTorch and Keras API is used in Tensor Flow to implement DRL-XGBOOST.

4.1. Metrics Evaluation

The accuracy, precision, recall, and F1-score of the model are considered as follows [35,36,37]:

• Accuracy: It is the most important statistic for determining how effective the model is at forecasting true negative and positive outcomes. Equation (8) is used to calculate the accuracy.

$$Accuracy=\left(\frac{TP+TN}{TP+TN+FP+FN}\right)$$

(8)

• TP (true positives): Here, the model suggests that the image will be normal.
• TN (true negatives): Here, the model assumes an aberrant image and the actual label confirms the prediction.
• FP (false positives): In this, the model suggests a typical image, but the actual label is abnormal.
• FN (false negatives): Here, the model forecasts an aberrant image but the actual label is normal.

• Precision (PR), also called positive predictive value, is a value that belongs to [0, 1]. If the precision is equal to 1, the model is considered satisfactory. Precision is calculated using Equation (9).

$$PR=\left(\frac{TP}{TP+FP}\right)$$

(9)

The recall (REC), also known as sensitivity, measures the classifier’s ability to find all positive samples. Equation (10) is used to compute the recall.

$$REC=\left(\frac{TP}{TP+FN}\right)$$

(10)

According to Equation (11), the F1-score expresses the harmonic mean of precision and recall, which indicates how precision memory are matched.

$$F1-Score=\left(\frac{2\times PR\times REC}{PR+REC}\right)$$

(11)

Figure 2 is a graph that depicts the training and validation curves for the proposed accuracy and AUC of DRL-XGBOOST. Owing to the overfitting and class imbalance issues, the training accuracy of the model is around 95%, whereas the validation accuracy is 94%. On the other hand, the model has the training AUC of approximately 98% and validation accuracy of 90%, as depicted in the above figures.

Figure 3 is a graph that depicts the training and validation curves for the proposed DRL accuracy and AUC. Owing to the overfitting and class imbalance problem, the model’s training accuracy is around 28%, and its validation accuracy is 26%. On the other hand, the model has the training AUC of approximately 56% and a validation accuracy of approximately 55%, as depicted in the above figures.

Figure 4 depicts the confusion matrix (contingency table) that is utilized to differentiate the phases of dementia and predict AD. The confusion matrix displays the expected class alongside the classes of the four distinct groups of varying degrees of dementia. The confusion matrix depicts the training performance of the model. The computation is conducted for (i) 639 ND (Non-Demented) images, (ii) 635 VMD (Very Mild Demented) images, (iii) 662 MD (Mild Demented) images, and (iv) 624 MOD (Moderate Demented) images. The confusion matrix is utilized to generate the class metrics presented in

Table 3. Performance analysis with different approaches.

Approach	Accuracy	Precision	Recall	F1-Score
Reinforcement Learning (RL)	83.23	82.13	81.23	81.23
Deep Reinforcement Learning (DRL)	84.34	83.45	82.12	80.23
DRL-XGBOOST	90.23	92.34	95.45	96.12

, which are derived from the individual class metrics.

Table 3 displays the performance analysis of three different techniques, namely RL, DRL, and DRL-XGBOOST, based on their accuracy, precision, recall, and F1-score. DRL-XGBOOST with an accuracy of 90.23%, 92.34% of precision, 95.45% of recall, and 96.12% of F1-score outperforms the remaining two approaches. Figure 5 depicts the comparison of RL with the proposed approach graphically.

The performance indices of each class, including the ND, VMD, MD, and MOD classes, are presented in

Table 4. Performance indices of individual classes.

Class	Precision	Recall	F1-Score	Support
ND	90.12	91.23	82.34	356
VMD	92.34	89.34	85.67	379
MD	95.34	86.12	90.23	356
MOD	96.23	85.23	94.34	380

according to their precision, recall, F1-score, and support. where MOD yields the highest precision value, followed by MD, VMD, and ND, respectively. Whereas the maximum recall value was found for ND, followed by VMD, MD, and MOD, respectively. By contrast, MOD has the highest F1-score and Support value compared to the other classes.

Figure 6 depicts a class-by-class comparison of DRL-XGBOOST performance in terms of precision, recall, and F1-score, as shown in Table 3.

Table 5. Accuracy and precision of existing approaches.

Existing Approach	Accuracy	Precision	Recall	F-Score
RF, SVM [37]	73.23	72.12	65.56	70.12
Logistic regression with Lasso [38]	71.23	70.23	76.34	78.12
VGG16 [12]	82.13	85.34	84.34	80.12
Siamese network [39]	86.12	80.12	85.23	82.12
DRL-Xgboost	90.23	92.34	95.45	96.12

displays the precision and accuracy of available methods. DRL-XGBOOST outperforms the competition with an accuracy of 90.23% and a precision of 92.34% followed by the Siamese network [37], VGG16 [12], Logistic regression with Lasso [38], RF, and SVM [39].

Figure 7 shows the key metrics of the existing and proposed methodologies, with the proposed approach achieving a high performance, as shown in Table 5.

Table 6. OASIS dataset comparison.

Approach	Accuracy	Precision	Recall	F-Score
Reinforcement Learning (RL)	84.23	80.12	81.34	82
Deep Reinforcement Learning (DRL)	85.12	85	80.23	81.23
DRL-XGBOOST	91.23	92.3	94.56	95

compares OASIS datasets using three distinct techniques, namely RL, DRL, and DRL-XGBOOST. DRL-XGBOOST has the highest accuracy compared to other approaches (91.23%), followed by DRL (85.12%) and RL (84.23%). Similarly, DRL-XGBOOST has the highest precision (91.23%), followed by DRL (85%) and RL (80.12%). Similarly, DRL-XGBOOST (94.56%) has the highest recall value, followed by RL (81.34%) and DRL (80.22%). Consequently, DRL-XGBOOST (95%) has the highest F-score value, followed by RL (82%) and DRL (81.23%).

Table 7. ADNI dataset comparison.

Approach	Accuracy	Precision	Recall	F-Score
Reinforcement Learning (RL)	83.12	82.13	80.2	81
Deep Reinforcement Learning (DRL)	84	83	80	82.12
DRL-XGBOOST	93.23	94.5	96.34	92.12

above compares ADNI datasets using three distinct techniques, namely RL, DRL, and DRL-XGBOOST. DRL-XGBOOST has the highest accuracy compared to other approaches (93.23%), followed by DRL (84%) and RL (83.12%). Similarly, DRL-XGBOOST has the highest precision (94.5%), followed by DRL (83%) and RL (82.13%). Similarly, DRL-XGBOOST (96.34%) has the highest recall value, followed by RL (80.2%) and DRL (80%). Consequently, DRL-XGBOOST (92.12%) has the highest F-score value, followed by DRL (82.12%) and DRL (81%).

4.2. Observations about the Experiment

• Using the actor-network and the critics-network, we apply deep reinforcement learning to tackle the concerns of class imbalance in the proposed method. Both networks improve the learning design.
• Figure 6 depicts the proposed approach’s confusion matrix, which includes all four classes of dementia severity degrees and helps to eliminate false positives.
• In Table 3, a comparison is made between the proposed approach and some existing forms of deep reinforcement learning. The proposed method greatly improves all performance indicators considered.
• Thanks to deep reinforcement learning the proposed approach improves the iteration-based picture sample actor and critics approach and enhances performance measures.
• In the proposed approach, training is updated based on the test, and active learning continues throughout the testing phase.
• The proposed method improved accuracy by 6–7%, precision by 9–10%, recall by 13–14%, and the F1-score by 9–10% in a typical experiment. The improvement is due to reinforcement learning layer-wise feature mapping by an activation function and Xg-Boosting learning to optimize features.

5. Conclusions

Deep Reinforcement Architecture (DRA) is utilized in this study to classify dementia. The model used to classify dementia stages is trained and evaluated in terms of data obtained from Kaggle. The dataset’s most important problem is its class imbalance. This issue is resolved via the XgBoost Deep Reinforcement method. We were able to achieve an overall accuracy of 90.23% with a 96% F1-Score by comparing our new model to the current models. A comparison between machine learning and deep learning approaches is shown in Table 4. The proposed technique increases performance measures because features are repeatedly learned and taught through an active learning process.