Deep Learning-Based Classification of Fine Hand Movements from Low Frequency EEG

Abstract

The classification of different fine hand movements from electroencephalogram (EEG) signals represents a relevant research challenge, e.g., in BCI applications for motor rehabilitation. Here, we analyzed two different datasets where fine hand movements (touch, grasp, palmar, and lateral grasp) were performed in a self-paced modality. We trained and tested a newly proposed CNN, and we compared its classification performance with two well-established machine learning models, namely, shrinkage-linear discriminant analysis (LDA) and Random Forest (RF). Compared to previous literature, we included neuroscientific evidence, and we trained our Convolutional Neural Network (CNN) model on the so-called movement-related cortical potentials (MRCPs). They are EEG amplitude modulations at low frequencies, i.e., $(0.3,3)$ Hz that have been proved to encode several properties of the movements, e.g., type of grasp, force level, and speed. We showed that CNN achieved good performance in both datasets (accuracy of $0.70\pm 0.11$ and $0.64\pm 0.10$, for the two datasets, respectively), and they were similar or superior to the baseline models (accuracy of $0.68\pm 0.10$ and $0.62\pm 0.07$ with sLDA; accuracy of $0.70\pm 0.15$ and $0.61\pm 0.07$ with RF, with comparable performance in precision and recall). In addition, compared to the baseline, our CNN requires a faster pre-processing procedure, paving the way for its possible use in online BCI applications.

1. Introduction

Several BCI systems for motor rehabilitation or motor control [1,2,3,4,5,6] and other basic neuroscience studies strongly rely on the ability to precisely and effectively distinguish different fine hand movements. One example is the investigation of the neural mechanisms underlying the writing and the music performance [7,8] or during real-life performance in ecologically valid situations outside the laboratory [9]. Movement-related cortical potentials (MRCPs) are amplitude modulations of the time-domain EEG signal that occur in the ($0.3,3$) Hz frequency band [10]. MRCPs can be detected during motor execution, motor attempt, and even during imagery of a movement, and they reflect the cortical processes involved in the planning and execution of a movement [11]. Previous literature [10] reports that the components of the MRCPs can be influenced by several factors, such as the preparatory state (self-paced or cue-based), the level of intention [12], the type of movement, the praxis, and the previous experience of the same movement. Nevertheless, it has been found that MRCPs can also encode several properties of the movements, such as the type of grasp [13], the force level [14], and the speed of the task [15]. In addition, MRCPs have been previously employed to discriminate hand movements in patients with severe manual impairments [5]. For this reason, MRCPs are considered valid signals to be used for BCI control [5,6]. Based on this well-established neuroscience background, our aim was to test if a deep learning (DL) approach can improve the performance of the classification of touch, grasp, palmar, and lateral grasp movements. Previous literature has already investigated the classification of different fine hand movements, including touch and different kinds of grasp [16,17,18]. Shrinkage linear discriminant analysis (sLDA) and random forest (RF) are well-established approaches for electroencephalography (EEG) classification, showing low complexity and good performance even with a limited amount of trials. However, they might score poorly in case of complex nonlinear EEG data [19]. However, deep DL has recently demonstrated promising results in decoding brain activity in several scenarios, e.g., sleep stage scoring [20], epileptic seizure detection [21], as well as hand movement classification [22]. Therefore, the aim of this work was to evaluate the performance of a newly proposed DL-based model, compared to the well-established sLDA and RF methods, in the classification of three different classes of movement, using two pre-recorded datasets. Given the small size of our datasets, we decided to adopt a CNN-based model to classify MRCPs, as it has been previously showed to be effective in small datasets [23,24,25].

The paper is organized as follows: in Section 2, we present the most relevant previous studies related to our work; in Section 3, we describe the experimental protocol, the common steps of pre-processing for all models, our proposed CNN model, and we briefly review the two baseline models chosen for comparison. In Section 4, we report and discuss all results, both from the qualitative analysis of the MRCPs and of the classification from different movements. Finally, Section 5 concludes the paper and mentions the possible impact of this work for other studies.

2. Related Works

The possibility of decoding touch and grasp actions from low-frequency EEG signals has been shown in other studies [16,17,18]. Well-established literature proved that brain activity in the frequency band 2–5 Hz contains relevant movement-related information. Thus, in [26], the authors proposed an interesting approach to classify speed in hand movements: they applied wavelet-CSP to MRCPs and were able to classify slow and fast movements with $83.71\%$ accuracy. In [18], Ofner et al. classified single upper limb movements with a binary classification approach, recording six different types of movements, both executed and imagined, and rest trials. For the executed movements, in the movements versus rest binary classification, the average accuracy reached the value of $87\%$, while, for the movementsversusmovements, the average accuracy dropped down to $55\%$. For the imagined movements, an accuracy of as less as $27\%$ and $73\%$ was obtained for movements versus movements and rest versus movements classification, respectively. In [16], palmar, lateral, and pincer grasps were recorded and classified, in a cue-based paradigm. A 4-class sLDA was used to classify the three movements and the rest data, obtaining a peak accuracy of $65.9\%$. Moreover, a binary classifier was trained in the same study, for each binary combination of classes. The palmar versus lateral grasp classification obtained a peak accuracy of $73.5\%$. In [17], MRCP was shown to significantly discriminate between unimanual (e.g., left hand vs. right hand grasps) as well unimanual vs. bimanual movements. Particularly, both unimanual and bimanual reach and grasp actions were classified with sLDA based on low-frequency time-domain EEG in the frequency range 0.3–3 Hz. Binary combinations of the different movements were also classified separately, leading to average accuracies for the movement classes between $66\%$ and $70\%$. The highest accuracies were obtained with the rest class versus the movement ones, with performance between $74\%$ to $90\%$. Recently, new approaches have been increasing. DL showed promising results in many different fields of application and has been successfully applied also in the BCI field [27]. It provided satisfactory outcomes in a variety of EEG analysis, ranging from channel selection to classification of motor imagery (MI). Among other models, CNN was particularly successful to extract spatial and frequency features from EEG for speech classification, as reported in [28], to detect artifacts in EEG [21], as well as to recognize MI for BCI [25]. In the works by Dose [23] and Lee [24], the possibility of MI-EEG classification using a CNN architecture was explored. In both studies, the input of the CNN was represented by the raw EEG signal, without any pre-processing. In [23], fist movements were classified and transfer learning was also used, reaching a subject-specific accuracy of $86.49\%$, and a standard accuracy of $80.38\%$. In [24], instead, four MI classes were analyzed, namely elbow extension, twisting, grasping, and rest. The average accuracy obtained was $84\%$. It is worth noting that these results were obtained with a relatively low amount of data: only 50 trials for each class were available, for a total of 150 trials for the training of the DL architecture.

3. Materials and Methods

For this work, we re-analyzed two datasets, which were recorded in the scope of the EU Horizon 2020 project “MoreGrasp” [6]. First, we describe the experimental protocol used to acquire the two datasets. Second, we describe the pre-processing pipeline that is employed in this study, both before the CNN and the baseline models. Then, we introduce our CNN-based model and the baseline models used for the performance comparison. Finally, we explain the cross-validation procedure and the metric we used for the evaluation of the performance.

3.1. Experimental Protocol

At the very beginning of the experimental protocol, the participants’ handedness was tested with the well-known hand dominance test of [29]. Then, they were asked to seat on a comfortable chair in a noise and electromagnetic shielded room. Their brain activity was acquired via EEG by means of 4 g. USBamp amplifiers (g.tec medical engineering GmbH, Austria) and a 64 gel-based channel EEG cap (g.GAMMAsys/g.LADYbird, g.tec medical engineering GmbH, Austria). Incidentally, 58 electrodes recorded the brain activity, while six of them were used to record the electrooculogram (EOG). The EEG electrodes locations were defined by a well-established modified version of the International 10–20 System [30]. All data were recorded using a 256 Hz sampling frequency. In the resting position, the participants’ right arm was placed, relaxed, upon a pressure button on a table in front of them. They were also recommended to avoid unnecessary body or eye movements, and to fix their gaze at a fixed point, for a few seconds, at the beginning of each repetition of the movement. All movements were self-initiated to ensure a more natural application scenario. Additionally, at the beginning, middle, and end of the experiment, 3 min rest was repeated 3 times.

3.1.1. Experiment 1—Touch and Grasp

In the first experiment, 11 healthy volunteers (11 males, ages 20–38) were included. The hand dominance test resulted in nine right-handed participants, one left-handed and one undefined. During the experiment, two glasses were on the table at the participant’s reaching distance. They were equipped with one pressure sensor each, in order to precisely detect the grasping onset. The participants were instructed either to grasp the first glass or to touch the second glass for a minimum time of 4 s. Thus, the total duration of each repetition was longer than 5 s. Four sessions of 20 repetitions of the same movement, i.e., grasping and touching, were included in the protocol. Thus, 80 touching and 80 grasping movements were performed by each participant at the end of the experiment. After each session, the participants could take a break and the glasses were switched. The same number of repetitions was performed in each glasses’ positions. On the computer screen in front of them, they could see the remaining number of trials to perform.

3.1.2. Experiment 2—Palmar and Lateral

In the second experiment, 15 right handed participants (10 males, ages 21–30) were involved. During the experiment, two jars were on the table at the participant’s reaching distance. The first one was empty, while the second had a spoon stuck in it. The participants were instructed either to reach-and-grasp the first jar or the second for a minimum time of 2 s. Thus, the total duration of each repetition was longer than 5 s. They freely decided which movement to perform. To interact with the empty jar, they had to perform a palmar grasp, while, for the jar with the spoon, they exploited a lateral grasp. Four sessions of 20 repetitions of the same movement, i.e., palmar or lateral grasp, were included in the protocol. After each session, the participants could take a break and the objects were switched. The same number of repetitions was performed in both objects’ positions. The full experimental description can be found in [17].

3.2. Pre-Processing

We adopted the same pre-processing pipeline for both EEG datasets used in this study. The pipeline is a well-established algorithm, previously implemented in [16,17]. The full data processing was implemented in Matlab 2020a [31]. A scheme of the pre-processing pipeline is displayed in Figure 1.

First, every EEG signal was band-pass filtered between $0.01$ Hz and 100 Hz (Chebyshev filter, order 8). Second, a notch filter was applied to suppress the power line noise at 50 Hz. Additionally, ICA was applied for classification with sLDA as well as RF, to identify and remove the artifacts due to eye movements, as in [32]. Third, a narrower band-pass filter (Butterworth filter, order 4) was applied to extract the signal low-frequency component in the band $(0.3,3)$ Hz [33]. All filters were implemented using the non-causal Matlab function filtfilt in order to compensate for the delay introduced by them. The full dataset, i.e., including all EEG signals, was transformed using the common average reference (CAR) filter [34], a spatial filter used to enhance the signal component due to the brain region under each individual EEG sensor (i.e., discarding components that are spread all around the scalp). Finally, every signal was downsampled to 16 Hz (using the Matlab function resample).

During the experimental sessions, a pressure sensor (either on the table or on the object to interact with, see Section 3.1) was exploited to identify the time instants when the individual initiated the movement, i.e., the movement onset. Therefore, proper segmentation of the continuous pre-processed EEG signals was ensured. Each segment (or trial) was defined as the signal period of time from $-2$ s to $+3$ s around the movement onset (i.e., time 0). Not only movement-related trials, but also 5 s rest trials have been obtained from the datasets: they were extracted from the 3 min rest periods (see Section 3.1).

In order to include only clean data in the datasets to analyze, we applied a well-established outlier rejection algorithm [35,36,37]. Briefly, it works as follows: a single trial was kept in the dataset if it simultaneously met the following conditions: (1) its absolute amplitude does not exceed 125 V, (2) and its kurtosis does not exceed its standard deviation by four times.

Finally, we obtained two different 3 class datasets: dataset 1 includes clean data from Experiment 1, while dataset 2 includes those from Experiment 2. Both datasets can be described as follows:

$${\mathbf{X}}^{\left(i\right)}=\left[\begin{array}{cccc}{x}_1^{\left(i\right)}\left(1\right)& x_1^{\left(i\right)}\left(2\right)& \cdots & x_1^{\left(i\right)}\left(N\right)\\ x_2^{\left(i\right)}\left(1\right)& \ddots & \cdots & ⋮\\ ⋮& ⋮& \ddots & ⋮\\ x_{58}^{\left(i\right)}\left(1\right)& \cdots & \cdots & x_{58}^{\left(i\right)}\left(N\right)\end{array}\right]$$

(1)

where i is the total number of trials in the dataset (despite the specific class of movement), and N the number of time samples available. At the end of the pre-processing procedure, the dataset 1 includes $64.9\pm 5.6$ Touch trials, $67.8\pm 7.8$ Grasp trials, and $69.6\pm 3.9$ Rest trials (average number across subjects), while the dataset 2 includes $68.7\pm 2.8$ Palmar trials, $69.6\pm 5.1$ Lateral trials, and $69.5\pm 2.8$ Rest trials (average number across subjects). Similarly, in the case in which (ICA) was used, the dataset 1 includes $63.9\pm 4.8$ Touch trials, $64.4\pm 7.7$ Grasp trials, and $67.1\pm 3.4$ Rest trials; while dataset 2 includes $67.1\pm 3.7$ Palmar trials, $67.4\pm 4.9$ Lateral trials, and $68.5\pm 4.1$ Rest trials.

To note, N varies depending on the learning model used to analyze the data (see Section 3.3 and Section 3.4). ${\mathbf{X}}^{\left(i\right)}$ can be interpreted as an EEG 2D image.

Moreover, the class of movements can be either touch, grasp, palmar, lateral or rest. Dataset 1 includes touch, grasp and rest classes, while dataset 2 includes palmar, lateral, and rest classes.

Both datasets were processed by adopting the same pre-processing pipeline reported in Figure 1, separately. For both datasets, preliminary results (not reported here) showed that sLDA and RF performed better if pre-processing includes an ICA step, while CNN works better without any ICA decomposition. For this reason, Figure 1 shows two different branches, depending on which classifier is used.

3.3. Classification with CNN

The CNN is a particular type of neural network that implements, in at least one of its layers, a convolutional operation [38]. In this study, the architecture of the CNN was adapted from [23,24]. As depicted in Figure 2, it consisted of five layers.

The first two were convolutional layers: the first one performed a temporal filtering (i.e., convolution along the time axis), while the second one a spatial filtering (i.e., convolution along the channel axis). Each convolutional layer was followed by a batch normalization and an exponential linear unit (eLu) activation function. Then, an average pooling layer, which flattened the input to a single dimension, and two fully connected layers were stacked on the top of the convolutional ones. Finally, a softmax activation function returned the probability of each sample to belong to each class. To note, since the kernel size at the output of the second convolutional layer was equal to the number of channels, this filter reduced the channel dimension to one. The input to this CNN was given by the EEG 2D images ${\mathbf{X}}^{\left(i\right)}$, for every available trial i, as computed in (1), which resulted in a three-dimensional tensor. To implement such architecture, several parameters had to be decided: specifically, the kernel size and the depth of the convolutional layers, and the size of the pooling and dense layers. Given each participant, we used a grid-search procedure to optimize such parameters over a priori selected ranges. Then, the optimal combination of parameter values was given by a majority vote strategy across all participants. As a result, the kernel size of the first convolutional layer was equal to 30, while, for the second convolutional layer, it corresponded to the number of channels, i.e., 58. Moreover, for both of them, the optimal depth was found to be 40 filters. The kernel size of the average pooling layer was equal to 15, the first fully connected layer had 80 neurons, while the second fully connected layer had three neurons, corresponding to the number of classes. The same CNN architecture was used for both datasets, while one CNN model was trained for each participant.

3.4. Classification with Baseline Models

Two state-of-the-art machine learning models were used for comparison with our proposed CNN: an sLDA and an RF. They both have the advantages to be simple in their implementation, to require a light computational burden, and to show good performance in EEG classification during hand movements, gesture recognition, and BCI experiments.

The linear discriminant analysis (LDA) is a supervised multi-class classification technique that aims at estimating the parameters of the linear multivariate model of the input data, via parametric density estimation procedure [19]. Here, the input to the sLDA is the vector $\mathbf{x}$ obtained by reshaping matrix $\mathbf{X}$ as follows:

$${\mathbf{x}}^{\left(i\right)}=[x_1^{\left(i\right)}\left(1\right),x_2^{\left(i\right)}\left(1\right),\dots ,x_{58}^{\left(i\right)}\left(1\right),x_1^{\left(i\right)}\left(2\right),x_2^{\left(i\right)}\left(2\right),\dots ,x_{58}^{\left(i\right)}\left(N\right)],$$

(2)

where i is the trial number, and N the number of time samples available in the sliding window. The shrinked LDA version, i.e., the sLDA, introduces a regularization strategy, especially useful with high-dimensional feature spaces, when only a few data points are available. For the regularization, we considered the pooled covariance matrix, computed from the three classes, and we optimized the regularization parameter as in [39]. A common approach to obtain the optimal sLDA model with time series, i.e., as in the EEG case, is to train several sLDA models, each one based on a different subset of the training set (e.g., given by a different observation window), and to select the one which yields the best training performance. Thus, here, for each single trial i, a sliding window is used to scan the entire EEG segment from $-2$ s to 3 s. Then, an sLDA model was obtained for each, every 2, time instant (i.e., one every 125 ms). For each participant, the time instant where the sLDA model resulted in the best classification performance was taken as the trained model. Moreover, three different window lengths were tested for each participant, specifically $\{0.6,0.8,1\}$ s, and the same model training was repeated for every length value.

The RF is a classifier that works as an ensemble of individual decision tree algorithms to reduce the risk of overfitting and, thus, to enhance the classification performance. Each tree is obtained by independently bootstrapping the samples from the input dataset, resulting in uncorrelated models whose predictions are more accurate than the ones we would obtain from a single one [40]. Then, a random set of predictors is used at each split to grow the tree [41]. To compute the predictions, a majority vote across the predictions of the individual decision trees is used. In this study, the vector in (2) was also used as the input to the RF. The number of trees was empirically set to 50, found as the best trade-off between the classification accuracy and the computational complexity.

3.5. Cross-Validation and Performance Evaluation

The performance of the classifiers were evaluated by means of the accuracy, computed as follows:

$$\mathrm{accuracy} = \frac{\mathrm{correctly} \mathrm{classified} \mathrm{instances}}{\mathrm{total} \mathrm{number} \mathrm{of} \mathrm{instances} \mathrm{to} \mathrm{classify}}.$$

(3)

The chance level was computed for each model and each participant by means of the Adjusted Wald Interval [42], with $\alpha$ set to $0.05$. For both datasets, we split each of them into a training set ($75\%$) and a validation set ($25\%$). During training, a 10 times repeated 5-fold cross validation procedure was adopted to ensure the robustness of the trained model. The validation set was used for testing the performance of the trained models on unseen data. All splits led to representative subset of the dataset, in order to have balanced classes for an unbiased classification.

4. Results and Discussion

In this section, we describe both the quality of our dataset after pre-processing and the results of the classification using the CNN model designed in Section 3.3, including the comparison with sLDA and RF.

4.1. Pre-Processing, Feature Extraction, and MRCPs

As a result of the pre-processing (see Section 3.2), 3 out of 11 participants (namely, $S002$, $S003$, $S005$) were rejected from the dataset 1, for the massive presence of artifacts in their EEG recordings. Then, the high quality of the cleaned EEG data after pre-processing is shown in Figure 3. It reports the subset of EEG segments, after synchronization to the movement onset, for different movement classes and for rest periods, in both datasets.

In Figure 3, we can notice that, in case of any movement, negative values are seen around time zero, i.e., the movement onset, which represent the negative peak of the MRCPs. Moreover, all panels show good repeatability across movement repetitions. On the contrary, as expected, in the rest condition, we cannot notice any clear pattern. We also observed (results not reported for space constraints) that a difference in the MRCP peak amplitude was especially noticeable at the EEG electrodes located in the contralateral side of the movement and that this spatial pattern is consistent across several participants, in line with other literature [43]. However, it is also clear that Dataset 1 is more affected by noise compared to Dataset 2, so that, e.g., the touch-related EEG data could show a less pronounced negative peak of the MRCPs (as seen in Figure 3a). We also observed that this behavior is consistent across most of the channels, with no specific spatial pattern (results not reported for space constraints).

4.2. Classification Results

Table 1. Comparison of classification performance (in terms of accuracy) in validation from Dataset 1.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
S000	0.62	0.60	0.64	0.64	0.58	0.58	0.58
S001	0.66	0.50	0.52	0.60	0.54	0.56	0.69
S004	0.74	0.75	0.73	0.75	0.69	0.73	0.69
S006	0.84	0.76	0.72	0.70	0.86	0.74	0.82
S007	0.68	0.76	0.80	0.76	0.78	0.73	0.80
S008	0.85	0.78	0.78	0.83	0.88	0.80	0.93
S009	0.61	0.62	0.67	0.50	0.62	0.58	0.62
S010	0.59	0.52	0.54	0.57	0.52	0.50	0.46
MEAN	0.70	0.66	0.68	0.67	0.68	0.65	0.70
STD	0.10	0.11	0.10	0.11	0.14	0.11	0.15

and

Table 2. Comparison of classification performance (in terms of accuracy) in validation from Dataset 2.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
G01	0.79	0.65	0.65	0.65	0.61	0.59	0.67
G02	0.43	0.61	0.55	0.53	0.49	0.63	0.47
G03	0.58	0.51	0.59	0.67	0.59	0.53	0.53
G04	0.58	0.67	0.63	0.61	0.53	0.59	0.55
G05	0.75	0.52	0.63	0.65	0.58	0.63	0.56
G06	0.55	0.51	0.60	0.64	0.68	0.53	0.49
G07	0.60	0.58	0.50	0.54	0.58	0.60	0.50
G08	0.72	0.78	0.75	0.78	0.67	0.76	0.73
G09	0.73	0.69	0.69	0.62	0.58	0.71	0.63
G10	0.65	0.65	0.54	0.54	0.63	0.61	0.63
G11	0.61	0.56	0.60	0.63	0.58	0.50	0.56
G12	0.80	0.56	0.58	0.58	0.66	0.66	0.52
G13	0.57	0.53	0.45	0.55	0.51	0.63	0.51
G14	0.60	0.64	0.68	0.75	0.55	0.59	0.55
G15	0.65	0.63	0.71	0.59	0.57	0.61	0.55
MEAN	0.64	0.61	0.61	0.62	0.59	0.61	0.56 *
STD	0.10	0.08	0.08	0.07	0.06	0.07	0.07

Note: * means p < 0.05 at the Mann–Whitney U test (α = 0.05).

report the comparison of the classification performance between the CNN and the baseline models over the unseen validation sets. They show the results of the classification in terms of accuracy. To achieve these performances, we used the CNN model with the best selection of hyperparameters, employing the same architecture for all participants. On the other hand, for sLDA and RF, we considered all possible choices of the sliding window length, with the best window time location, for each participant. The chance level was computed as in Section 3.5, and it was found to be $0.40$. Comparing the classification results among the three classifiers, similar accuracies were observed for all of them, with all values above the chance level. Furthermore, they achieved slightly better results in the Dataset 1, as expected from its higher repeatability across EEG segments (see Figure 3), compared to the Dataset 2. However, for both datasets, the CNN model reached the best average accuracy across all participants ($0.70$ for Dataset 1, $0.64$ for Dataset 2). Previous results on Dataset 2 were obtained using sLDA for the binary classification of right hand palmar vs. lateral movements [17]. There, a grand-average participant-specific peak accuracy of $66,3\%$ was reached (individual results were not available), in line with our classification outcomes. sLDA and RF achieved the best classification accuracy at the single-subject level in Dataset 1: thus, a particular configuration (i.e., an optimal choice of the window length and time location) can lead a baseline model to yield higher performance compared to CNN. Nevertheless, especially for the Dataset 2, the CNN showed higher variability in the individual participant accuracies, with some of them reaching very high values ($0.80$ for G12) and others slightly above the chance level ($0.43$ for G02). Finally, from the confusion matrices (not reported here for space constraints), we observed that the rest class was classified with the highest accuracy compared to the other movement classes (best accuracy among the two datasets: 78% for rest, 57% for touch, 62% for grasp, 55% for palmar, 52% for lateral), in line with previous literature [16,17,18]. Precision and recall metrics have also been computed for all subjects, in both dataset (results are reported in

Table A1. Comparison of classification performance (in terms of precision) in validation from Dataset 1.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
S000	0.62	0.60	0.64	0.64	0.58	0.58	0.58
S001	0.66	0.50	0.52	0.60	0.54	0.56	0.69
S004	0.74	0.75	0.73	0.75	0.69	0.73	0.69
S006	0.83	0.76	0.72	0.70	0.86	0.74	0.83
S007	0.68	0.75	0.79	0.75	0.77	0.73	0.80
S008	0.84	0.75	0.79	0.85	0.88	0.79	0.93
S009	0.61	0.61	0.67	0.50	0.61	0.57	0.61
S010	0.58	0.52	0.54	0.57	0.52	0.50	0.46
MEAN	0.70	0.66	0.68	0.67	0.68	0.65	0.70
STD	0.10	0.11	0.10	0.11	0.14	0.11	0.15

,

Table A2. Comparison of classification performance (in terms of recall) in validation from Dataset 1.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
S000	0.62	0.59	0.65	0.64	0.58	0.58	0.60
S001	0.65	0.49	0.54	0.62	0.53	0.55	0.68
S004	0.76	0.78	0.74	0.76	0.69	0.73	0.71
S006	0.83	0.77	0.73	0.70	0.88	0.73	0.87
S007	0.69	0.75	0.81	0.78	0.80	0.73	0.79
S008	0.84	0.77	0.79	0.85	0.88	0.82	0.94
S009	0.62	0.62	0.67	0.51	0.61	0.56	0.64
S010	0.59	0.51	0.54	0.57	0.53	0.55	0.54
MEAN	0.70	0.66	0.68	0.68	0.69	0.66	0.72
STD	0.10	0.12	0.10	0.12	0.15	0.11	0.14

,

Table A3. Comparison of classification performance (in terms of precision) in validation from Dataset 2.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
G01	0.79	0.64	0.65	0.65	0.60	0.59	0.66
G02	0.43	0.61	0.55	0.53	0.49	0.63	0.47
G03	0.59	0.51	0.59	0.68	0.59	0.53	0.53
G04	0.58	0.67	0.63	0.61	0.52	0.58	0.54
G05	0.75	0.52	0.62	0.64	0.58	0.62	0.56
G06	0.55	0.50	0.59	0.64	0.68	0.51	0.49
G07	0.59	0.57	0.49	0.53	0.56	0.59	0.50
G08	0.73	0.79	0.75	0.79	0.68	0.77	0.71
G09	0.74	0.69	0.69	0.62	0.57	0.71	0.63
G10	0.63	0.66	0.54	0.55	0.63	0.61	0.63
G11	0.61	0.55	0.59	0.63	0.56	0.49	0.55
G12	0.81	0.56	0.58	0.58	0.66	0.66	0.52
G13	0.58	0.53	0.45	0.56	0.51	0.63	0.51
G14	0.60	0.63	0.68	0.74	0.54	0.59	0.54
G15	0.65	0.63	0.71	0.59	0.57	0.61	0.55
MEAN	0.64	0.60	0.61	0.62	0.58	0.61	0.56 *
STD	0.10	0.08	0.08	0.07	0.06	0.07	0.07

Note. * means p < 0.05 at the Mann–Whitney U test (α = 0.05).

,

Table A4. Comparison of classification performance (in terms of recall) in validation from Dataset 2.

Subject	CNN	sLDA (0.6 s)	sLDA (0.8 s)	sLDA (1 s)	RF (0.6 s)	RF (0.8 s)	RF (1 s)
G01	0.80	0.64	0.64	0.65	0.60	0.58	0.67
G02	0.46	0.62	0.54	0.53	0.49	0.63	0.47
G03	0.58	0.53	0.60	0.68	0.58	0.55	0.54
G04	0.59	0.66	0.63	0.62	0.52	0.57	0.52
G05	0.74	0.54	0.62	0.64	0.57	0.62	0.56
G06	0.55	0.52	0.57	0.63	0.68	0.67	0.51
G07	0.60	0.58	0.45	0.53	0.57	0.58	0.49
G08	0.73	0.81	0.75	0.80	0.67	0.77	0.73
G09	0.75	0.69	0.69	0.64	0.59	0.72	0.64
G10	0.64	0.65	0.44	0.51	0.62	0.59	0.63
G11	0.61	0.55	0.62	0.64	0.56	0.48	0.53
G12	0.80	0.57	0.57	0.57	0.66	0.66	0.54
G13	0.59	0.53	0.46	0.54	0.51	0.64	0.50
G14	0.60	0.64	0.67	0.73	0.54	0.58	0.54
G15	0.67	0.63	0.72	0.59	0.57	0.62	0.55
MEAN	0.65	0.61	0.60	0.62	0.58 *	0.62	0.56 *
STD	0.10	0.08	0.10	0.08	0.06	0.07	0.07

Note. * means p < 0.05 at the Mann–Whitney U test (α = 0.05).

in the Appendix A). We obtained an average value of $0.68\pm 0.12$ both for precision and recall in Dataset 1 (average across subjects and models); similarly, we obtained an average value of $0.60\pm 0.08$ for precision and $0.61\pm 0.08$ for recall in Dataset 2.

As expected, a critical drawback of the CNN approach is its computational complexity, significantly higher with respect to sLDA and RF: the former use less time points as an input to train the model (either $0.6$ s, $0.8$ s and 1 s), while the latter took the entire 5 s EEG segment into account. Therefore, the training time for a single model is significantly lower compared to CNN: for a single subject, a single run of classification takes $0.46$ s for sLDA, $0.56$ s for RF, and $29.16$ s for CNN. However, we should notice that sLDA and RF require a significant pre-processing step to select the optimal location of the time window in the EEG segment for the classification. In addition, a 10 times 5-fold CV is applied. Then, overall, the time required to classify one single subject by sLDA increases to 3 min, to 10 min for RF, while the CNN requires 31 min. In addition, CNN implements a more complex architecture compared to sLDA and RF. However, it showed promising advantages over sLDA and RF: indeed, the latter exploit a semi-quantitative pre-processing pipeline, including ICA to clean data from eye movements artifacts. Moreover, for both sLDA and RF, we needed to train a classifier at each time point to select the one that led to the best performance. On the other hand, less pre-processing (i.e., without the need of running ICA) was needed to classify the datasets by means of the CNN; and it is completely automatic. Even if two relatively small datasets were available, we could show that our CNN model can achieve classification accuracies in line with two well-established baseline models. Moreover, we obtained similar performance with a simpler pre-processing pipeline, reducing it to those steps (e.g., filtering and automatic trial rejection) that could be performed in an online modality. This may be explained by the fact that the CNN can behave as an automatic feature extraction method as well as an efficient classifier. As mentioned earlier, the CNN showed the drawback of a higher variability in the individual participant accuracies compared to the baseline models. Therefore, further studies could investigate the inter-individual differences of the CNN performance. Furthermore, CNN needs to have 5 s EEG images as an input, while sLDA and RF could work with shorter windows: thus, further studies are needed to identify the minimal input length for the CNN architecture in order to produce acceptable classification outcomes. This could pave the way for its application in an online scenario, i.e., the assisted living or a BCI system. Finally, CNN could take larger advantage by the spatial information in the EEG dataset, by applying a spatial convolution at its second layer. On the other hand, sLDA and RF did not use this kind of information to enhance their predictions.

5. Conclusions

In this study, we evaluated the classification performance of a DL model, i.e., a CNN, on two different datasets including self-paced fine hand movements (touch, grasp, palmar, and lateral grasp). The classification results of the CNN were compared with two well-established machine learning models, i.e., sLDA and RF. The classification included three classes, i.e., two movements and the rest condition, and it was based on the components of the EEG signals in the 0.3–3 Hz low frequency band. We showed that CNN achieved good performance in both datasets (average accuracy of 0.70 in Dataset 1, $0.64$ in Dataset 2, with a chance level of $0.40$), with similar or superior results compared to the baseline models. All classifiers yielded better results in the first dataset (touch, grasp, and rest), reflecting neurophysiological observations of the MRCPs that were more pronounced in that dataset. We also highlighted that, compared to the baseline models, our CNN did not require strong pre-processing, e.g., ICA, or heavy burden and semi-quantitative pre-processing steps, paving the way for its possible use in online BCI applications.