Engagement Recognition Using a Multi-Domain Feature Extraction Method Based on Correlation-Based Common Spatial Patterns

Abstract

Engagement ability plays a fundamental role in allocating attentional resources and helps us perform daily tasks efficiently. Therefore, it is of great importance to recognize engagement level. Electroencephalography is frequently employed to recognize engagement for its objective and harmless nature. To fully exploit the information contained in EEG signals, an engagement recognition method integrating multi-domain information is proposed. The proposed method extracts frequency information by a filter bank. In order to utilize spatial information, the correlation-based common spatial patterns method is introduced and extended into three versions by replacing different correlation coefficients. In addition, the Hilbert transform helps to obtain both amplitude and phase information. Finally, features in three domains are combined and fed into a support vector machine to realize engagement recognition. The proposed method is experimentally validated on an open dataset composed of 29 subjects. In the comparison with six existing methods, it achieves the best accuracy of $87.74\pm 5.98\%$ in binary engagement recognition with an improvement of $4.03\%$, which proves its efficiency in the engagement recognition field.

1. Introduction

Engagement is the ability to spend attentional resources on task-related stimuli and ignore external interference [1]. As a fundamental aspect of cognition, engagement ability plays a crucially important role in the operational environment [2]. Engagement helps us to ignore interference and focus our awareness, which is necessary for learning, working, and building normal relationships [3,4,5]. Being a participant in numerous and heterogeneous daily tasks, everyone endeavors to improve their efficiency with high engagement [6,7,8]. In highly professional scenarios in particular, such as clinical operations [9], aircraft piloting [10] and aerial work [11], engagement level monitoring is very important and related to life safety [12]. The disability of being engaged in current tasks and dealing with mental workload may lead to severe accidents [13]. Moreover, measuring engagement is of great clinical application. For example, measuring engagement can help to diagnose psychological diseases, such as attention deficit hyperactivity disorder (ADHD), which is a common disease in children [14]. Furthermore, engagement level influences the reliability of psychological tests [15]. Monitoring the engagement level can measure the credibility of psychological tests and therefore serves as an auxiliary means in other psychological treatments. In this way, it becomes more and more significant to recognize and measure engagement level [16,17].

Psychologists have made attempts to design various self-reporting questionnaires and scales in order to reflect the engagement level of subjects more accurately [18,19,20,21]. However, these subjective tools are delayed and sometimes uncertain due to indistinct memory and dishonesty. Unlike subjective tools, physiological signals can be collected simultaneously with the execution of the current task and can reflect the engagement level more accurately without subjective interference [22,23]. Therefore, researchers have started to recognize engagement level and workload through physiological signals. In [24], physiological signals, including electrocardiogram (ECG), skin conductivity (SC), and respiration (RESP) signals, are collected and combined with the Self-Assessment Manikin to detect the cognitive load. Belle et al. [25] decomposed ECG signals using an S-transform algorithm to detect engagement and attention state. Zennifa et al. [22] introduced the implementation of a low-density hybrid system for engagement recognition, in which wireless electroencephalography (EEG), ECG, electrooculogram (EOG), and near-infrared spectroscopy (NIRS) are utilized.

This study focuses on EEG-based engagement recognition. Existing studies in neuropsychology have revealed the direct correlation between EEG signals and a variety of cognitive behaviors, including engagement level [26,27,28], and established EEG-based engagement recognition. The key to decoding engagement level using EEG signals is to extract the most representative features to realize engagement level classification. The most widely used feature in engagement level classification is spectral power. Booth et al. [29] predicted engagement utilizing power spectral density (PSD) features. The PSD features in $\delta$ (1–4 Hz), $\theta$ (4–8 Hz), $\alpha$ (8–13 Hz), and $\beta$ (13–30 Hz) were extracted as features to feed into classifiers to predict engagement level. Similarly, 1 Hz bin PSD features were utilized by Li et al. and fed into deep models for engagement assessment [23]. Relative band power of $\delta$, $\theta$, $\alpha$, and $\beta$ bands were calculated, and $P_{\beta }/(P_{\alpha }+P_{\theta })$ was taken as an EEG index by Rabbi et al. [30]. The four features were analyzed under different tasks. Researchers in [31] extracted relative PSDs from differential EEG channels and fed them into a linear DFA to realize binary engagement classification for subjects who ranged from 67 to 93 years old. Researchers in [22] extracted multi-modal physiological signals, including EEG, ECG, and NIRS signals, in which the maximum power, the power density integral power, and the relative power of $\delta$, $\theta$, $\alpha$, and $\beta$ bands were extracted as features. Then, the features were selected using a correlation-based feature selection (CFS) method and classified with the k-nearest neighbor (KNN) method. However, only information in the frequency domain is utilized in these methods, which fails to exploit the underlying information related to engagement level in EEG signals. Therefore, it is significant to improve the engagement recognition accuracy by integrating information from different domains and extracting representative features to realize classification.

For the spatial domain, the common spatial pattern (CSP) [32] method is one of the most popular and powerful spatial filtering methods in the motor imaginary (MI)-based brain–computer interface (BCI). The purpose of the CSP is to find spatial filters so that the variance of the filtered data is maximized for one class and minimized for the other class. However, due to the sensitivity of the CSP to noise, many extensions of CSP-based methods have emerged to improve its effectiveness. In [33], an L1 norm-based CSP is introduced, in which the L1 norm is in place of the L2 norm to figure out the spatial filter estimation in order to improve the robustness of the CSP. Lotte et al. [34] proposed an integrated theoretical framework of a regularized CSP (RCSP) and specifically designed four RCSP algorithms, among which the optimal RCSPs are a CSP with Tikhonov regularization and weighted Tikhonov regularization. Recently, a correlation-based CSP (CCSP) based on temporal correlation was designed to improve the performance of the CSP [35]. In this study, the CCSP algorithm was adopted to extract spatial features. For the frequency domain, the combination of a filter bank and a CSP is a simple way to combine the information in the frequency domain with that in the spatial domain [36]. A set of filter banks was designed, and their effectiveness was compared to select the optimal filter bank. In addition, for the phase domain, both amplitude and phase information were considered to utilize the potential information embedded in the EEG signals. Therefore, this paper proposes an engagement recognition using a multi-domain feature extraction method based on correlation-based common spatial patterns (MDCCSPs), which presents three contributions:

• Firstly, an engagement recognition method based on a correlation-based CSP is proposed, in which the temporal correlation is utilized as a prior to improve the effectiveness. Specifically, the original CCSP is extended into three versions by replacing various correlation coefficients (Pearson’s linear correlation coefficient, Kendall’s correlation coefficient [37], and Spearman’s correlation coefficient).
• Secondly, this study integrates information from spatial, frequency, and phase domains to fully exploit the potential of EEG data. A filter bank is combined with the original CCSP to extract features in the frequency domain and the spatial domain. Besides, the Hilbert transform is applied to obtain the amplitude and phase angle of EEG signals. Multi-domain features are integrated and fed into an SVM to realize engagement recognition.
• Thirdly, the proposed method is validated and compared with existing methods on an open dataset composed of 29 subjects. Experimental results show that it offers an efficient way to recognize the level of engagement, which is validated by its outperformance.

The rest of the article is arranged as follows. The proposed engagement recognition method using the multi-domain feature extraction method based on correlation-based common spatial patterns is described in detail in Section 2. In Section 3, the experimental setups are presented, and the results are analyzed. Then, a discussion is provided in Section 4. Finally, Section 5 summarizes the whole article.

2. Materials and Methods

2.1. CSP-Based Methods

2.1.1. Common Spatial Patterns

In BCI systems, especially MI-based BCI systems, the CSP is widely used to extract spatial features and achieves good performance. Considering ${\mathbf{X}}_i\in {\mathbb{R}}^{T\times C}$ is the EEG signal from class i, $i=1,2$, where T and C, respectively, denote the number of time samples and the number of channels, the target spatial filters in the CSP algorithm can be found by maximizing the objective function

$$J\left(w\right)=\frac{{\mathbf{w}}^T{\mathbf{X}}_1^T{\mathbf{X}}_1\mathbf{w}}{{\mathbf{w}}^T{\mathbf{X}}_2^T{\mathbf{X}}_2\mathbf{w}}=\frac{{\mathbf{w}}^T{\mathbf{V}}_1\mathbf{w}}{{\mathbf{w}}^T{\mathbf{V}}_2\mathbf{w}},$$

(1)

where ${\mathbf{V}}_i\in {\mathbb{R}}^{C\times C}$, $i=1,2$ represents the spatial covariance matrix of ${\mathbf{X}}_i$, and $\mathbf{w}$ represents the spatial filter. It should be noted that ${\ast }^T$ is the transpose operation for the matrix. This constrained optimization problem can be solved by using a Lagrange multiplier method:

$$L(\lambda ,\mathbf{w})={\mathbf{w}}^T{\mathbf{V}}_1\mathbf{w}-\lambda ({\mathbf{w}}^T{\mathbf{V}}_2\mathbf{w}-1),$$

(2)

where $\lambda$ stands for the Lagrange multiplier. The target spatial filter $\mathbf{w}$ maximizes L; thus the derivative of L with respect to $\mathbf{w}$ should be equal to zero:

$$\begin{array}{cc}\hfill \frac{\partial L}{\partial \mathbf{w}}& =2{\mathbf{w}}^T{\mathbf{V}}_1-2\lambda {\mathbf{w}}^T{\mathbf{V}}_2=0\hfill \\ \hfill & \iff {\mathbf{V}}_1\mathbf{w}=\lambda {\mathbf{V}}_2\mathbf{w}\hfill \\ \hfill & \iff {\mathbf{V}}_2^{-1}{\mathbf{V}}_1\mathbf{w}=\lambda \mathbf{w}.\hfill \end{array}$$

(3)

Next, the spatial filter $\mathbf{w}$ can be figured out by solving an eigenvalue problem. It can be obtained as the largest and smallest eigenvalues of matrix $\mathbf{M}={\mathbf{V}}_2^{-1}{\mathbf{V}}_1$.

2.1.2. Correlation-Based Common Spatial Patterns

In order to overcome the shortcomings of sensitivity to noise, instability, and overfitting of the original CSP, CCSPs are proposed in [35] to impose the temporal correlation between signals as a penalty term to obtain a regularization CSP. Therefore, the objective function becomes

$$J_1\left(w\right)=\frac{{\mathbf{w}}^T{\mathbf{V}}_1\mathbf{w}}{{\mathbf{w}}^T{\mathbf{V}}_2\mathbf{w}+\alpha {\mathbf{P}}_1\left(\mathbf{w}\right)},$$

(4)

where $\alpha$ stands for the regularization parameter ($\alpha$ > 0), and ${\mathbf{P}}_1\left(\mathbf{w}\right)$ is the penalty term in $J_1\left(w\right)$ based on the correlation coefficient

$${\mathbf{P}}_1\left(\mathbf{w}\right)={\mathbf{w}}^T{\mathbf{D}}_1\mathbf{w}$$

(5)

where ${\mathbf{D}}_1$ is a diagonal matrix in which the diagonal elements are correlations between the average signal $\overline{{\mathbf{X}}_i}$, $i=1,2$ in two classes

$$\overline{{\mathbf{X}}_i}=\frac{{\sum }_{l=1}^L{\mathbf{X}}_i^l}{L},i=1,2,$$

(6)

where L is the number of trials in each class. The correlation matrix $\mathbf{R}\in {\mathbb{R}}^{C\times C}$ is then constructed as

$$\mathbf{R}=\left(\begin{array}{cccc}{r}_{11}& r_{12}& \cdots & r_{1C}\\ r_{21}& r_{22}& \cdots & r_{2C}\\ ⋮& & \ddots & ⋮\\ r_{C1}& r_{C2}& \cdots & r_{CC}\end{array}\right).$$

(7)

In the correlation matrix $\mathbf{R}$, each element is defined as

$$r_{m,n}=corr(\overline{x_1^m},\overline{x_2^n}),m,n=1,2,\cdots ,C$$

(8)

where $\overline{x_1^m}$ and $\overline{x_2^n}$ are the m-th and the n-th columns of the matrices $\overline{{\mathbf{X}}_1}$ and $\overline{{\mathbf{X}}_2}$. Given the correlation matrix, ${\mathbf{D}}_1$ is formed as

$${\mathbf{D}}_1=\left(\begin{array}{cc}{a}_1& \\ & a_2& \\ & & \ddots \\ & & & a_C\end{array}\right),$$

(9)

where the diagonal element $a_m$ is calculated as

$$a_m=\frac{{\sum }_{n=1}^C\left|r_{mn}\right|}{C},m,n=1,2,\cdots ,C.$$

(10)

It should be noted that $|\ast |$ is the absolute operation. Next, ${\mathbf{D}}_2$ is calculated as

$${\mathbf{D}}_2=\left(\begin{array}{cc}{b}_1& \\ & b_2& \\ & & \ddots \\ & & & b_C\end{array}\right),$$

(11)

where $b_n$ is obtained following

$$b_n=\frac{{\sum }_{m=1}^C\left|r_{mn}\right|}{C},m,n=1,2,\cdots ,C.$$

(12)

Therefore, the Lagrange multiplier method is adopted to solve Equation (4):

$$L_1(\lambda ,\mathbf{w})={\mathbf{w}}^T{\mathbf{V}}_1\mathbf{w}-\lambda [{\mathbf{w}}^T({\mathbf{V}}_2+\alpha {\mathbf{D}}_1)\mathbf{w}-1].$$

(13)

Similar to Equation (3), the derivative in the following equation is equal to 0:

$$\begin{array}{cc}\hfill \frac{\partial L_1}{\partial \mathbf{w}}& =2{\mathbf{w}}^T{\mathbf{V}}_1-2\lambda {\mathbf{w}}^T({\mathbf{V}}_2+\alpha {\mathbf{D}}_1)=0\hfill \\ \hfill & \iff {\mathbf{V}}_1\mathbf{w}=\lambda ({\mathbf{V}}_2+\alpha {\mathbf{D}}_1)\mathbf{w}\hfill \\ \hfill & \iff {({\mathbf{V}}_2+\alpha {\mathbf{D}}_1)}^{-1}{\mathbf{V}}_1\mathbf{w}=\lambda \mathbf{w}\hfill \end{array}.$$

(14)

The target filters are the eigenvectors corresponding to the largest eigenvalues of matrix ${\mathbf{M}}_1={({\mathbf{V}}_2+\alpha {\mathbf{D}}_1)}^{-1}{\mathbf{V}}_1$. For the objective function

$$J_2\left(w\right)=\frac{{\mathbf{w}}^T{\mathbf{V}}_2\mathbf{w}}{{\mathbf{w}}^T{\mathbf{V}}_1\mathbf{w}+\alpha {\mathbf{P}}_2\left(\mathbf{w}\right)},$$

(15)

where

$${\mathbf{P}}_2\left(\mathbf{w}\right)={\mathbf{w}}^T{\mathbf{D}}_2\mathbf{w},$$

(16)

the spatial filters are solved following a similar process. Finally, the filters are the eigenvectors corresponding to the largest eigenvalues of matrix ${\mathbf{M}}_2={({\mathbf{V}}_1+\alpha {\mathbf{D}}_2)}^{-1}{\mathbf{V}}_2$.

In this study, the original CCSP is extended into 3 versions using different correlation coefficients, including Pearson’s linear correlation coefficient (P-CC), Kendall’s correlation coefficient (K-CC), and Spearman’s correlation coefficient (S-CC). And the effectiveness of the 3 proposed CCSPs is compared.

2.2. Amplitude and Phase Feature Extraction

To fully exploit the information in EEG signals, the features in the phase domain are integrated to improve engagement recognition accuracy. To extract the phase angle of EEG signals, the Hilbert transform [38] is employed to an EEG signal $\mathbf{X}\left(t\right)=[x_1\left(t\right),x_2\left(t\right),\cdots ,x_C\left(t\right)]$, in which C is the number of channels:

$$\widehat{x}\left(t\right)=\mathcal{H}\left[x\left(t\right)\right]=\frac{1}{\pi }{\int }_{-\infty }^{+\infty }\frac{x\left(\tau \right)}{\tau -t}d\tau .$$

(17)

The analytic signal is then formed as

$$\Theta \left(t\right)=x\left(t\right)+i·\widehat{x}\left(t\right)=A\left(t\right)·e^{i·\theta \left(t\right)},$$

(18)

where i denotes the imaginary unit. $A\left(t\right)$ and $\theta \left(t\right)$ stand for the amplitude and phase (AP), respectively:

$$\left\{\begin{array}{c}A\left(t\right)=\sqrt{x^2\left(t\right)+{\widehat{x}}^2\left(t\right)},\hfill \\ \theta \left(t\right)=arctan{\displaystyle \frac{\widehat{x}\left(t\right)}{x\left(t\right)}}.\hfill \end{array}\right.$$

(19)

2.3. Multi-Domain Feature Extraction Method Based on Correlation-Based Common Spatial Patterns

In this study, an engagement recognition using the multi-domain information extraction method based on correlation-based common spatial patterns (MDCCSP) is proposed. The framework of the MDCCSP is shown in Figure 1.

As shown, one trial of a multi-channel EEG signal $\mathbf{X}\left(t\right)=[x_1\left(t\right),x_2\left(t\right),\cdots ,x_C\left(t\right)]\in {\mathbb{R}}^{T\times C}$ is first input into the filter bank. The filter bank filters the EEG signal into several sub-bands to process the signal in each sub-band independently in order to extract frequency domain features from frequency bands. Therefore, the signals in the sub-bands are obtained as

$$\{{\mathbf{X}}_1\left(t\right),{\mathbf{X}}_2\left(t\right),\cdots ,{\mathbf{X}}_K\left(t\right)\}\in {\mathbb{R}}^{T\times C},$$

(20)

where K denotes the number of filters in the filter bank. Then, the Hilbert transform described before is conducted to the EEG signal in each frequency band to extract both pieces of AP information. Supposing ${\mathbf{X}}_k,k=1,2,\cdots ,K$ is the sub-band EEG signal filtered by the k-th filter, the AP information of ${\mathbf{X}}_k$ is extracted as

$$\{{\mathbf{X}}_k^A\left(t\right),{\mathbf{X}}_k^P\left(t\right)\}\in {\mathbb{R}}^{T\times C},k=1,2,\cdots ,K,$$

(21)

where ${\mathbf{X}}_k^A\left(t\right)$ and ${\mathbf{X}}_k^P\left(t\right)$ represent the EEG signal containing the amplitude information and the EEG signal containing the phase information, respectively. In all, the number of signals for all sub-bands is $2\times K$. For each sub-band, two CCSPs are applied to extract both AP features:

$$\{Featur{e}_k^A,Featur{e}_k^P\}\in {\mathbb{R}}^N,k=1,2,\cdots ,K,$$

(22)

where N is the number of features produced by a CCSP. $Featur{e}_k^A$ and $Featur{e}_k^P$ stand for the features obtained from ${\mathbf{X}}_k^A\left(t\right)$ and ${\mathbf{X}}_k^P\left(t\right)$. Furthermore, $2\times K\times N$ features are obtained in all. Finally, all the features are fed into a classifier to realize the output of the results. In this study, the classification is binary, and the results are either low engagement or high engagement.

2.4. Support Vector Machine

Support vector machine (SVM) is widely applied as a supervised classification model in BCI systems for its effectiveness [39]. It achieves binary classification with a main strategy to find a hyperplane that can separate the negative and positive samples in the training set to the utmost extent. Specifically, the hyperplane that maximizes the distances between the hyperplane and the closest data samples in two classes is found in the training stage. In the test stage, unlabeled test data are classified into one class by the relative position to the hyperplane obtained in the training stage.

In practice, the concept of soft margin is introduced to solve the problem that there is barely linearly separable data in reality. Given a set of training data and labels $T=\{(t_1,y_1),(t_2,y_2)$, ⋯, $(t_H,y_H)\}$, the process to find the hyperplane $y=w·t+d$ can be formed as an optimization problem:

$$\begin{array}{c}\underset{\omega ,b,{\xi }_i}{min}\frac{1}{2}{\parallel \omega \parallel }^2+c\sum _{i=1}^H{\xi }_i,\end{array}$$

(23)

$$\begin{array}{c}s.t.y_i(\omega ·t_i+d)\ge 1-{\xi }_i,{\xi }_i\ge 0,\end{array}$$

(24)

where ${\xi }_i=max[0,1-y_i(w·t_i+d)]$, $i=1,2,\cdots ,H,$ which is a hinge loss function, and c is the penalty parameter. In this study, a c-SVC model is utilized and a radial basis function (RBF) kernel is implemented in experiments. The penalty parameter c is set to 2.

3. Results

3.1. Dataset

In the experiments, an open dataset [40] is used to validate the effectiveness of the proposed MDCCSP method. It is a multi-modal dataset comprising NIRs and EEG data from 29 subjects, from which the EEG data are merely used in this study. Therefore, the EEG dataset is introduced in the following paragraph. The information (age and gender) of the subjects is represented in

Table 1. Information of subjects in the dataset where “M” and “F” stands for “male” and “female”.

Subject	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Age	28	25	26	23	27	34	28	29	27	32	31	29	33	27	25
Gender	F	F	M	F	M	M	M	F	F	M	F	M	M	F	F
Subject	16	17	18	19	20	21	22	23	24	25	26	27	28	29
Age	32	39	26	32	28	24	23	27	27	26	27	27	36	26
Gender	M	F	F	F	F	M	F	M	M	F	F	M	F	M

.

The EEG signals in the dataset are acquired using a BrainAmp EEG amplifier, which is produced by a German company named Brain Products GmbH. During data acquisition, the electrodes are placed according to the 10-5 international standard, and 30-channel EEG signals are recorded with a sampling rate of 1000 Hz. Subjects who participated in the experiments are ensured healthy conditions. They are instructed to complete two tasks, which are baseline tasks and mental arithmetic (MA) tasks. In the baseline tasks, subjects are supposed to be relaxed without thinking, while in the MA tasks, subjects need to carry on MA following the instructions on the screen. The EEG signals from baseline tasks and from MA tasks are labeled as low engagement data and high engagement data, respectively. The detailed process of experiments is represented in Figure 2.

In both tasks, subjects first take a 60 s pre-test rest and then receive an instruction on the screen. The instruction is either a cross or a subtraction formula such as “514 − 9”. After 2 s, the instruction disappears. A beep prompts the beginning of the task. In the baseline task, subjects only need to stare at the cross, while in the MA task, subjects need to repeatedly subtract the same value from the result of the last time in mind. For instance, “514 − 9 = 505, 505 − 9 = 496, 496 − 9 = 487, ⋯”. Each task period lasts for 10 s, and each subject takes both tasks 20 times. There is a rest with a random length from 15 s to 17 s between every two tasks. During the whole experiment, subjects are required to keep still to avoid motion artifacts to the utmost extent.

3.2. Data Pre-Processing

The original EEG data are first common average re-referenced and then band-pass filtered with a passband from 0.5 Hz to 50 Hz using a fourth-order Chebyshev II filter. The filtered data are downsampled to 200 Hz. A toolbox is used to remove the EOG artifacts automatically. The 10 s EEG data recorded during the tasks are segmented into 2 s pieces for the next steps, which is shown in Figure 2. The data from baseline tasks is labeled as low engagement, while that from MA tasks is labeled as high engagement. The whole experiment is carried out under the environment of MATLAB 2020b. Results are obtained using a 10-fold cross-validation.

3.3. Effect of Filter Banks

In this study, a filter bank is utilized to extract frequency-domain information to improve the accuracy of the proposed MDCCSP method. Here, a set of filter banks are designed with various bandwidths and various sizes of overlapping to find an optimal set of filter banks for experiments. The principle of the design of filter banks is to fully cover all the frequency range of 4–32 Hz that is considered. Therefore, the designs of filter banks are shown in

Table 2. Filter banks validated in the experiments.

Filter Bank
Index	1	2	3	4	5	6	7	8
Frequency Range (4–32 Hz)	4–6	4–8	4–8	4–12	4–16	4–20	4–24	4–8 (theta)
	6–8	6–10	8–12	8–16	8–20	8–24	8–28	8–13 (alpha)
	8–10	8–12	12–16	12–20	12–24	12–28	12–32	13–32 (beta)
	10–12	10–14	16–20	16–24	16–28	16–32
	12–14	12–16	20–24	20–28	20–32
	14–16	14–18	24–28	24–32
	⋯	⋯	28–32
	30–32	28–32
Number of filters	14	13	7	6	5	4	3	3

.

The number of filters included in the eight sets of filter banks is between 3 and 14. The effects of the filter bank combining traditional EEG rhythms (theta, alpha, and beta bands) are also compared in the experiments. In this experiment, $x\left(t\right)$ and $\Theta \left(t\right)$ are extracted to obtain AP information, and the S-CC is used in the proposed MDCCSP method. The regularization parameter in the MDCCSP is set to $\alpha =0.001$. The number of features generated by each CCSP N is set as two. The engagement recognition results using different sets of filter banks in Table 2 are shown in Figure 3. And the recognition accuracies are shown in

Table 3. Engagement recognition accuracy using MDCCSP with different filter banks where the best results are marked bold.

Filter Bank
Index	1	2	3	4	5	6	7	8
Accuracy (%)	90.33	93.67	93.33	92.00	92.00	93.67	92.00	93.67
	87.67	90.00	89.67	88.00	85.00	83.00	81.67	83.33
	94.00	94.67	95.67	95.00	94.33	92.00	91.67	96.00
	90.00	91.33	89.33	91.67	91.33	89.33	86.00	91.33
	75.00	77.00	76.67	77.33	75.00	75.00	73.67	76.00
	86.00	91.00	88.33	89.00	83.67	82.33	83.00	84.33
	94.00	92.33	93.00	92.67	91.67	91.67	92.67	93.33
	89.67	91.67	91.00	90.67	88.67	88.00	88.67	88.67
	86.33	87.33	86.33	85.67	85.00	85.33	86.00	87.67
	92.00	92.33	94.00	94.33	90.33	90.00	90.33	94.67
	76.67	80.33	79.00	76.67	75.33	72.67	74.33	76.00
	80.00	77.33	78.33	79.33	77.00	80.00	77.00	75.67
	93.00	94.33	92.67	93.33	92.00	90.33	90.67	90.33
	88.00	87.33	87.67	86.00	85.67	86.00	85.00	83.00
	80.67	82.00	82.00	80.00	75.67	74.00	70.67	70.67
	83.00	84.67	81.67	84.33	81.00	79.33	79.33	80.67
	87.33	89.00	86.67	85.33	84.00	83.33	78.00	82.00
	91.33	92.67	92.67	92.00	92.00	89.67	90.00	89.00
	79.67	85.67	86.00	82.33	78.67	78.33	79.33	79.67
	65.67	72.00	70.33	69.33	70.00	66.67	69.00	66.00
	87.00	89.67	90.33	89.67	89.00	85.67	88.67	87.67
	86.33	85.33	86.00	85.33	87.00	84.00	85.00	86.67
	82.00	88.00	87.33	87.33	83.33	82.33	82.00	82.67
	83.67	82.67	83.67	80.00	80.00	77.67	79.67	82.00
	89.00	90.33	89.33	92.33	94.00	91.00	90.33	90.33
	96.67	96.00	96.00	95.33	95.33	95.67	96.00	94.67
	79.33	82.67	82.33	81.67	80.67	83.33	77.67	81.33
	83.33	88.00	85.00	86.33	82.00	79.00	76.00	80.00
	93.33	95.00	94.67	92.33	93.67	92.00	90.67	92.00
Mean ± std	85.90 ± 6.77	87.74 ± 5.98	87.21 ± 6.17	86.74 ± 6.40	85.29 ± 6.88	84.18 ± 7.02	83.62 ± 7.22	84.80 ± 7.50
p-value	<0.01	/	<0.05	<0.01	<0.01	<0.01	<0.01	<0.01

.

As shown, the second filter bank achieves the highest recognition accuracy of $87.74\pm 5.98\%$ at the significance level of $5\%$. In the following experiments, the second filter bank with 13 filters is adopted.

3.4. Effects of Phase Information

Phase-domain information is utilized to enhance the proposed MDCCSP method. Thus it is supposed to validate the effects of phase information in engagement recognition. To extract phase information, the Hilbert transform [38] is employed following Equation (17). And therefore, the analytic signal is obtained in Equation (18). It should be noted that AP features are extracted from $x\left(t\right)$ and $\Theta \left(t\right)$ in the experiments. The optimal filter bank with 13 filters in Table 2 is adopted. And the S-CC is used in the proposed MDCCSP method. The regularization parameter in the MDCCSP is set to $\alpha =0.001$. The effects of extracting only amplitude information, extracting only phase information, and extracting both amplitude and phase information are compared. The results are shown in Figure 4 and

Table 4. The average engagement recognition results of MDCCSP extracting AP information, where the best result is marked in bold.

	Amplitude	Phase	AP
Mean ± std	81.28 ± 7.25%	76.57 ± 6.94%	87.74 ± 5.98%
p-value	<0.001	<0.001	/

, from which it can be observed that for all the subjects, the MDCCSP extracting only phase information achieves an average accuracy of $76.57\pm 6.94\%$, which is much higher than the chance level $50\%$ ($p<0.001$). These results prove that only phase information still matters in engagement recognition. Moreover, the MDCCSP extracting both pieces of AP information achieves the best performance $87.74\pm 5.98\%$ ($p<0.001$), which demonstrates that the phase information can enhance the effectiveness of the proposed engagement recognition method.

3.5. Effects of Correlation Coefficients

The correlation coefficient used in the original CCSP is P-CC. This study proposes the K-CCSP and the S-CCSP and applies them to the proposed MDCCSP recognition method. To determine the optimal CC, the recognition results of the MDCCSP utilizing the P-CC, K-CC, and S-CC are compared. It should be mentioned that the various CCs are only adopted in extracting features from $x\left(t\right)$ signals because $\Theta \left(t\right)$ is a complex signal to which the K-CC and the S-CC are not applicable. The optimal filter bank is used, and AP features are extracted in this experiment, and the regularization parameter in the MDCCSP is set to $\alpha =0.001$. The experimental results using the P-MDCCSP, K-MDCCSP, and S-MDCCSP are shown in

Table 5. The average engagement recognition results of MDCCSP using different CCSPs where the best result is marked bold.

	P-CC	K-CC	S-CC
Mean ± std	87.74 ± 5.98%	87.68 ± 6.40%	87.61 ± 6.52%
p-value	/	>0.1	>0.1

.

As shown in Table 5, there are no significant differences between the accuracies of the P-MDCCSP and K-MDCCSP methods, and between the accuracies of the P-MDCCSP and S-MDCCSP methods. In the following experiments, the P-MDCCSP is utilized.

3.6. Effects of Regularization Parameter $\alpha$

In order to improve the accuracy of the proposed engagement recognition method, we seek the optimal regularization parameter $\alpha$ used in Equations (4) and (15). The optimal filter bank is used and AP features are extracted in this experiment, and the S-CC is used in the MDCCSP. The results of the proposed MDCCSP using different regularization parameter $\alpha$ in $\{{10}^{-1}$, ${10}^{-2}$, ${10}^{-3}$, ${10}^{-4}$, ${10}^{-5}$, ${10}^{-6}\}$ are compared, which are presented in Figure 5. As shown, it achieves the best average accuracy when $\alpha ={10}^{-3}$; therefore, the regularization parameter $\alpha$ defaults to ${10}^{-3}$ in the following experiments to obtain the best recognition results.

3.7. Experimental Comparison

The CSP is the most popular method in BCI systems, and all CSP-based methods are on the basis of the CSP method. The proposed MDCCSP is based on the original CSP method. Therefore, it is first supposed to compare the results with CSP-based methods. The CSP [32], CCSP [35], and FBCSP [36] methods are chosen to validate the improvements of the proposed method. To compare with the CCSP and the FBCSP, the effectiveness of the main idea of the MDCCSP is validated, that is, integrating multi-domain information, including frequency information, spatial information, and phase information. The filter bank used in the FBCSP in the experiments is the second filter bank in Table 3. In addition, the proposed method is also compared with Barachant’s minimum distance to the Riemannian mean method (MDMR) [41], Shin’s method [40], and She’s method [42] to validate the effectiveness of the MDCCSP. Barachant et al. projected data onto the Riemannian space and extracted features in the space to realize classification. Shin et al. employed a shrinkage linear discriminant analysis to classify the features extracted by the CSP. An extreme learning machine was utilized to extract high-level features in She’s method. The recognition accuracies are presented in

Table 6. Engagement recognition accuracies of several existing methods compared with the proposed method where the best results are marked bold.

Comparison Results
Literature	Blankertz [32]	Ang [36]	Shin [40]	Ghanbar [35]	She [42]	Barachant [41]	Ours
Methods	CSP	FBCSP	CSP + ShrinkageLDA	CCSP	HSS-ELM	MDMR	MDCCSP
Accuracy (%)	88.33	89.33	84.33	87.67	88.67	78.33	93.67
	70.33	84.00	67.33	69.33	80.67	74.00	90.00
	82.33	88.67	82.67	84.33	86.67	94.00	94.67
	82.00	91.00	78.33	84.33	82.00	80.00	91.33
	71.67	73.67	67.33	74.33	76.00	75.33	77.00
	77.00	80.33	72.00	77.33	84.67	81.67	91.00
	88.67	93.33	86.33	90.00	89.00	93.00	92.33
	81.67	88.67	80.67	84.67	78.67	80.00	91.67
	79.33	82.33	76.33	79.00	78.00	73.67	87.33
	87.33	88.00	87.00	88.00	85.33	87.33	92.33
	64.67	80.67	68.67	68.67	75.00	72.33	80.33
	75.00	73.33	73.33	76.67	79.67	62.00	77.33
	81.00	87.33	74.33	83.00	86.67	84.33	94.33
	80.67	84.00	78.67	80.00	77.00	74.33	87.33
	71.67	75.33	68.00	71.67	65.00	65.33	82.00
	74.33	82.67	68.00	74.67	75.00	63.67	84.67
	74.00	81.33	60.33	77.00	83.67	83.33	89.00
	85.33	90.33	85.67	88.33	90.67	83.33	92.67
	74.67	82.00	71.67	76.33	81.67	78.00	85.67
	66.67	70.33	65.00	64.33	72.67	62.67	72.00
	84.67	86.00	76.00	81.67	79.67	84.67	89.67
	79.67	83.33	79.00	83.67	84.00	85.67	85.33
	75.67	84.33	74.00	77.33	84.67	76.00	88.00
	79.00	78.00	80.67	77.33	85.00	78.67	82.67
	83.00	86.33	85.00	90.00	88.33	77.33	90.33
	94.33	94.67	92.00	93.67	92.33	94.67	96.00
	68.67	73.67	71.00	68.67	72.00	71.00	82.67
	68.33	83.33	64.33	72.33	76.67	70.00	88.00
	78.00	91.33	71.33	78.67	88.33	91.00	95.00
Mean ± std	78.21 ± 7.07	83.71 ± 6.20	75.49 ± 7.93	79.41 ± 7.36	81.64 ± 6.38	78.47 ± 9.05	87.74 ± 5.98
p-value	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001	/

. It can be observed that the proposed MDCCSP method achieves the best performance compared with existing methods, with an average accuracy of $87.74\pm 5.98\%$, which provides a $4.03\%$ improvement on the second-best method. In detail, for 24 out of 29 subjects, the MDCCSP achieves the best accuracy, which validates the effectiveness of the proposed method.

4. Discussion

This article attempts to improve the effectiveness of engagement recognition by integrating multi-domain information and exploiting the potentiality of original EEG signals. An engagement recognition method using AP feature extraction based on filter bank correlation-based common spatial patterns (MDCCSPs) is proposed to realize more accurate engagement recognition. By integrating frequency, spatial, and phase information, the proposed method utilizes EEG signals more sufficiently and therefore improves the recognition accuracy, which is validated in experiments compared with existing methods.

The filter bank is used to extract features in different frequency sub-bands to exploit frequency-domain information. Therefore, it is supposed to validate the effectiveness of various filter banks and find the optimal set to improve the performance of the proposed methods. A total of 7 filter banks including 3 to 14 filters are validated and compared. The results show that the second filter bank in Table 3 including 13 filters achieves the best performance. Except for frequency-domain information, phase-domain information is also adopted. To validate its effect, the recognition results of extracting only amplitude, only phase, and both AP are compared. Only phase information reaches $76.57\pm 6.94\%$ accuracy, which is far higher than the chance opportunity. By integrating both pieces of AP information, an accuracy improvement of $6.46\%$ is achieved over the accuracy of using only amplitude information. The CCSP is introduced to the engagement recognition field to extract spatial features. In addition, the optimal settings for the CCSP are decided. The CCSP method is extended into three versions by replacing various correlation coefficients, and the results of these three methods are compared. As shown in Table 5, three methods achieve similar results, and there is no significant difference among them. Another setting is the regularization parameter, different settings of $\alpha$ are employed to seek the optimal value. After validation, $\alpha$ is set to ${10}^{-3}$ to obtain the best accuracy.

In addition, the effectiveness of the proposed method is analyzed by comparing it with the existing methods. Six methods, including the CSP, FBCSP, CSP+shrinkage LDA, CCSP, HSS-ELM, and MDMR, are compared with the proposed MDCCSP in engagement recognition. Results show that the proposed method outperforms existing methods with an accuracy of $87.74\pm 5.98\%$, which achieves an improvement of $4.03\%$ over the second-best method. The positive predictive value and negative predictive value are, respectively, $88.41\pm 5.90\%$ and $88.29\pm 5.60\%$. These satisfactory experimental results prove the effectiveness of the proposed method.

However, there are still some limitations in this study. First, although the proposed method integrates multi-domain information, including frequency information, spatial information, and phase information, and achieves satisfactory improvements, temporal information is not considered. In the future, integrating temporal-domain information should be further discussed in order to enhance the efficiency of engagement recognition. Second, the proposed method can only resolve binary engagement recognition. However, the realistic situation is often more complex, and multi-class recognition is more practical. Further study should focus on extending the engagement recognition method into a multi-class recognition method.

5. Conclusions

In this study, an engagement recognition method MDCCSP is proposed. In order to overcome the inadequate utilization of EEG signals, information from multi-domains is integrated, including frequency domain, spatial domain, and phase domain. Specifically, a filter bank is used to extract information from various frequency sub-bands. Phase features are acquired through the Hilbert transform, and spatial information is extracted by a correlation-based CSP. In addition, an SVM is utilized to obtain the results of binary engagement recognition. In comparison with existing methods, the proposed MDCCSP achieves the best performance of $87.74\pm 5.98\%$ with an improvement of $4.03\%$.