Development and Preliminary Testing of a System for the Multimodal Analysis of Gait Training in a Virtual Reality Environment

Abstract

Gait training in a virtual reality (VR) environment is promising for children affected by different disorders. However, the efficacy of VR therapy is still under debate, and more research is needed to clarify its effects on clinical conditions. The combination of VR with neuroimaging methods, such as the electroencephalography (EEG), might help in answering this need. The aim of the present work was to set up and test a system for the multimodal analysis of the gait pattern during VR gait training of pediatric populations by analyzing the EEG correlates as well as the kinematic and kinetic parameters of the gait. An EEG system was integrated with the Gait Real-time Analysis Interactive Lab (GRAIL). We developed and validated, with healthy adults (n = 5) and children (n = 4, healthy or affected by cerebral palsy (CP)), the hardware and software integration of the two systems, which allowed the synchronization of the acquired signals and a reliable identification of the initial contact (IC) of each gait cycle, showing good sensitivity and critical success index values. Moreover, we tested the multimodal acquisition by successfully analyzing EEG data and kinematic and kinetic parameters of one healthy child and one child with CP. This system gives the possibility of monitoring the effect of the VR therapy and studying the neural correlates of gait.

1. Introduction

Virtual reality (VR) training is becoming increasingly popular in its applications for motor and cognitive rehabilitation of heterogeneous clinical populations. The spreading of this technique is mainly due to the possibility of combining controlled laboratory environment with a more ecological and interactive training that increases patient motivation and improves the rehabilitation outcome [1]. Moreover, an increasing number of studies support the hypothesis that VR interventions might foster a neuroplasticity effect, thus facilitating the functional recovery of patients affected by disorders related to neurological dysfunction or damage [2,3].

Among the implemented VR motor training, gait training takes great importance, since it is devoted to the recovery of a fundamental motor activity. In the pediatric field, gait training with VR has been tested with promising results in patients affected by cerebral palsy (CP) ([4,5], see [6] for a review), acquired brain injury (ABI) [7], and ataxia [8]. Furthermore, VR motor and cognitive training have been administered to children with attention-deficit/hyperactivity disorder (ADHD) [9] and developmental delay [10] with positive effects on their gait parameters. Nevertheless, the efficacy of VR therapies is still under debate. The existing literature reports mixed findings [2,11], with results often obtained from small groups and sometimes without an adequate control comparison. Thus, more research is needed to better understand the effects of VR-based rehabilitation on different clinical conditions.

In this framework, the combination of VR with portable and non-invasive neuroimaging methods, such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), represents a great advantage [2,3]. These systems might serve several purposes: (1) to monitor the effect of the VR therapy, (2) to develop more targeted interventions augmenting training benefits, and (3) to allow the study of the neural correlates of the physiological motor or cognitive activities under study. The combination of VR and neuroimaging methods is still relatively new, but several studies have already used portable neuroimaging systems during motor tasks and motor imagery in VR environments, demonstrating their efficacy in detecting task-specific changes of the brain activity (e.g., [12,13,14]). Few studies combined such methods with VR-based gait training. Maidan et al. [15] recorded fNIRS in patients with Parkinson’s diseases during treadmill training with or without VR. They aimed at investigating if the two types of training differentially affected prefrontal activation and if this was related to the better fall rates obtained after the intervention with VR. Maggio et al. [16] investigated the relation of body representation (BR) and motor function recovery by comparing the effects of a robot-aided gait training (RAGT) with and without VR targeted at BR in improving lower limb sensorimotor function, gait performance, and BR in stroke patients. They used the EEG, recorded during RAGT, to study the neurophysiological basis of the BR-based motor function recovery. The combination of the EEG with VR-RAGT was used by Calabrò et al. as well [17]. Their aim was to study the neurophysiological correlates of motor function recovery in patients with unilateral ischemic stroke induced by the association between RAGT and VR.

To the best of our knowledge, no studies recorded EEG during VR-based gait training in pediatric populations. This is probably due to the difficulties in conducting complex experimental protocols with children and to the technical complexity of setting up the combined experiment that often requires interfacing two or more different devices.

The present work aims at setting up and testing a system for the study of the electrophysiological correlates and the kinematic and kinetic parameters of the gait pattern during gait training and rehabilitation of children and adolescents in a VR environment.

To achieve this goal, we integrated a portable 32-channel EEG system with the Gait Real-time Analysis Interactive Lab (GRAIL), which is a platform specifically designed for gait analysis and training and characterized by an immersive VR environment, an instrumented dual-belt treadmill, and an optoelectronic system.

In the following, we will describe in detail the system integration, the validation of the step recognition algorithm used to synchronize the acquired signals, and the testing of the multimodal recording in one healthy child and one child with hemiplegic cerebral palsy (CP).

2. Materials and Methods

2.1. GRAIL System

The GRAIL (Motek Medical, Houten, The Netherlands) is a system developed for gait analysis and training in an immersive VR environment. It includes many different hardware components: a dual belt treadmill with two degrees of freedom, equipped with a multi-sensor force platform; the Vicon motion capture system (Oxford Metrics, Oxford, UK); a 180° semi-cylindrical screen where VR environments are projected on; and 3 video cameras. Furthermore, the GRAIL system produces multi-sensorial feedbacks (audio, video, or proprioceptive) that can be used for functional training and allows the integration of external components such as sensors or I/O boards. All these components are integrated, synchronized, and controlled by the D-Flow software, which allows the real-time calculation of the gait parameters and manages the relationship between the patient, the scenery, and the interactive feedbacks and stimulations. The D-flow is a visual programming software based on the concept of modules, each of which presents specific functionalities. Some modules directly control hardware devices, such as the treadmill, the motion platform, and the Vicon system, other modules manage eventual external devices that can be connected to the system, and finally, other modules are specific for Lua programming and give the possibility to customize the system functionality.

2.2. EEG System

The eegoTMmylab (ANT Neuro, Hengelo, The Netherlands) EEG system was used for the present study. A compact, portable amplifier battery-powered characterizes the system. The EEG signals were recorded using the 32 channels gel-based Waveguard™ EEG cap (ANT Neuro, The Netherlands) with a sampling rate of 500 Hz. EEG data are transferred to the acquisition laptop via USB for visualization and storing. The amplifier is able to receive triggers from external devices through either a parallel port or a BNC connector. For the latter, only binary synchronization is possible. Moreover, the system includes an input for recordings bipolar channels (i.e., electromyography (EMG), electrocardiography (ECG), electrooculography (EOG)), and a variety of physiological sensors (i.e., temperature sensor, respiration, skin conductance, and acceleration).

2.3. Hardware Setup

The physical connection between the GRAIL and the EEG system was performed through a 1018_2 Phidget I/O board (Phidgets Inc., Calgary, AB, Canada). The Phidget was connected to the D-Flow pc through a USB port and to the EEG amplifier through the BNC connector (Figure 1). The connection was aimed at synchronizing the signals acquired by the two systems using a trigger (5V pulse) that the Phidget sent to the EEG amplifier in correspondence to each initial contact (IC) of the gait cycle. Since the BCN connection allows only a binary synchronization, the setup included the acquisition of the EMG signals of both legs in order to identify the triggers corresponding to right and left steps. The Phidget was controlled by an ad hoc developed application module that will be described in the next section. To allow a safe and easy physical connection of the two systems, a custom-made support for the EEG amplifier was designed and produced with 3D-printing technology to place the amplifier on the treadmill handrail.

2.4. Software Setup

The software development of the EEG-GRAIL system integration was implemented by adding new functionalities in the Forest Road application, which was already included in the gait suite package provided by Motek, as proof of concept. In this application, gait patterns (kinetic and kinematic parameters) are detected during the walking along a road in a wooded VR environment. The additional functionalities were developed using the Script modules, which allows the implementation of new algorithms in Lua language. Specifically, three Script modules were realized (Figure 2).

A first Script module receives the left and right ground reaction forces from the Mocap module, which is the D-Flow module that collects data from Vicon system managing kinematic and kinetic data and re-samples the ground force values of both platforms according to the D-flow sampling rate of 250 Hz.

A second Script module detects in real-time detection the gait IC by analyzing the ground reaction force values. In particular, the IC is detected when the force of the contact side of the platform is increasing and overcomes a threshold value (min_threshold), while the force measured on the other side is decreasing. Once the IC is detected, this Script module is paused for a defined time interval and is deactivated as long as the force value of the contact side remains above a threshold value (max_threshold). These two conditions were set to avoid incorrect IC detections. The force range (min_threshold; max_threshold) was experimentally set to a default value of 40 N and 120 N, respectively, and manually tailored on each subject since it might vary according to the subject’s weight and his/her gait characteristics. The time interval was set to a default value of 0.06 s and adjusted considering the treadmill speed and the subject’s step cadence in an inversely proportional way. The tuning of the parameters ended when the gait IC detection was considered satisfactory by visual inspection. Both time and force parameters were selected for each subject after he/she performed a period of walking habituation on the treadmill. The Graphic User Interface (GUI), included in the Forest Road application, was specifically modified to easily enable the adjustment of these parameters.

The IC detection module is active only when the treadmill reaches a constant speed excluding from the analysis the acceleration and deceleration gait phases. Once the IC is detected, a signal is sent to the Phidget, which in turns triggers the EEG system. As previously described, the trigger is the same for the left and right ICs and the identification of the left and right steps was possible off-line by analyzing the EMG signal acquired on both legs.

Finally, a third Script module was developed to automatically change the VR environment using the VR wooded setting for the walking sessions, and a setting with a white cross on a gray background was selected as the baseline condition during stand phases. This module receives the treadmill speed values and displays the corresponding setting.

The GRAIL-EEG system integration and synchronization were respectively tested and validated with healthy participants and children with CP. The study was approved by the Ethics Committee of the Scientific Institute IRCCS E. Medea (Bosisio Parini, LC, Italy); written informed consent to participate was obtained from all participants and/or children’s legal guardians prior to inclusion in the protocol.

2.5. Validation of the Integrated System Synchronization

2.5.1. Participants

To validate the integrated system synchronization by means of the gait cycle identification, five healthy adult (HA) volunteers (mean age = 30.9 years; SD = 5.6 years; 2 males) and four children were recruited (mean age = 11.2 years, SD = 1.9 years, 1 male healthy child (HC), 3 children with a diagnosis of unilateral CP, 2 males).

2.5.2. Protocol for the Validation of the Gait Cycle Identification Procedure

The main purpose of this study phase was to test the reliability of the step identification performed by the Phidget triggering by comparing it with the step identification performed through the analysis of gastrocnemii EMG signals and the visual inspection of a video recording.

The protocol consisted in the execution of various walking sessions on the GRAIL dual-belt instrumented treadmill while the EMG signals of the gastrocnemius muscles of both legs and the Phidget triggers were acquired. The sessions were recorded by a webcam (C920 PRO HD Webcam, Logitech, Losanna, Switzerland) connected to the EEG system and synchronized with both the EMG signals and Phidget triggers. The EMG was recorded through bipolar electrodes placed on the medial head of the gastrocnemius of both legs and connected with the EEG amplifier. Phidget triggers were identified as explained in Section 2.4. The data coming from all these 3 systems (webcam, EMG, and Phidget) were separately analyzed to detect the gait steps.

The validation protocol was different for adults and children. Adult subjects carried out 3 walking sessions of 2 min at 3 different speeds: 0.5 m/s, 1 m/s, and each subject’s preferred speed, for a total duration of about 6 min. Whereas children carried out a walking session at a constant speed, which was set according to the child’s preference. One minute of walking was alternated with 30 s of rest, during which the child had to stand still. Five trials (walking phase + rest phase) were performed. Thus, the session lasted about 7.5 min.

2.5.3. Gait Cycle Identification Algorithms and Validation

Matlab software (The Mathworks, Natick, MA, USA) was used to analyze the step identification data.

The EMG signals were detrended, high-pass filtered (5th order Butterworth filter, cut-off frequency = 30 Hz), full-wave rectified, and finally low-pass filtered (5th order Butterworth filter, cut-off frequency = 3 Hz) to extract the linear envelopes. A semi-automatic algorithm was implemented to detect the walking pattern by the left and right gastrocnemius EMG envelopes, thus identifying the left and right steps, respectively. The algorithm looks for EMG signals peaks that exceed an amplitude threshold, which were adjusted empirically according to the maximum signal peak of each subject (healthy adults’ threshold range: 20–25% of EMG maximum; children’s threshold range: 15–25% of EMG maximum). Additionally, two further constraints had to be satisfied to identify the muscle activation corresponding to the step execution: a minimum interval of 0.7 s between two successive EMG peaks of the same leg and a minimum interval of 0.25 s between consecutive left and right activation. These thresholds aimed at avoiding the detection of pre-contact muscle activation.

An off-line analysis of the Phidget triggers was also performed, imposing a minimum interval of 0.25 s between two successive triggers (corresponding to consecutive left and right ICs) and excluding eventual triggers that occurred with a time interval shorter than that value.

The webcam video was visually inspected to detect and count right and left steps. The analysis of the webcam video highlighted that as expected, the Phidget trigger occurred during the rearfoot strike, while the EMG peaks were identified in the pre-swing phase, which was the instant of maximum muscle activation. Thus, in one gait cycle, the Phidget trigger (rear foot strike) precedes the EMG peak (pre-swing). Figure 3 shows Phidget and EMG trigger signals and the corresponding phase of the gait cycle detected by the video recording.

The video was considered as the gold-standard to validate the system. Thus, the count of right and left steps obtained by visually inspecting the video was compared to the number of steps found by both the EMG and Phidget trigger signals. This allows the identification of eventual ‘double’ triggers (false positive (FP), wrongly identified steps) and ‘missing’ triggers (false negative (FN), missing steps), as well as the number of correctly identified steps (true positive (TP)). To evaluate the system reliability, the following indexes were computed for both EMG and Phidget trigger signals: sensitivity, which is defined as the ratio between the number of TP triggers and the total number of steps:

$$Sensitivity(\%)=\frac{TP}{TP+FN}\times 100$$

(1)

and critical success index (CSI), which is defined as the ratio between TP triggers and the sum of TP, FP, and FN triggers:

$$CSI(\%)=\frac{TP}{TP+FN+FP}\times 100.$$

(2)

The CSI was selected as a measure representative of the overall performance of the triggering systems (Phidgeht and EMG) in the step detection without being influenced by the correct rejection values (i.e., true negative values). Sensitivity and CSI were computed for adults and children separately.

Since we expected that both the Phidget and the EMG signal might show some limitations in the step detection, we created a new trigger signal that combined the Phidget and the EMG information. Specifically, we developed an algorithm that analyzes Phidget triggers automatically identifying FN triggers. The missing trigger position is identified considering a fixed distance from the first subsequent EMG trigger. This distance is computed by averaging time intervals between every Phidget trigger (rear foot strike) and the following EMG trigger (gastrocnemius contraction in the pre-swing phase). Successively, right and left steps are differentiated. Once the new trigger signal was generated TP, FP and FN were identified, and sensitivity and CSI were calculated.

2.6. Testing of Multimodal Acquisition

2.6.1. Participants

To test the feasibility of a multimodal acquisition with the GRAIL-EEG integrated system, the healthy child (HC, 8.8 years old, male) and one child with unilateral CP (CP01, 11.4 years old, male, right hemiparesis), already enrolled for the validation protocol, were recruited for a further acquisition.

2.6.2. Multimodal Acquisition Protocol

Children carried out a walking session identical to the one performed in the validation phase consisting of 5 trials that alternated 30 s of rest (stand position looking at a white fixation cross) and one minute of walking at the preferred speed in the VR wooden environment. During the session, the following data were acquired: gait kinematics and kinetics, EEG, EMG, Phidget triggers and video. For the acquisition of the kinematic parameters, the Vicon motion capture system (Oxford Metrics, Oxford, UK) integrated into the GRAIL was used. The Vicon captured in real time the coordinates of spherical passive markers placed in specific landmarks on children’s body following the Human Body Model 2 (HBM2) model, which is provided by Motek.

2.6.3. EEG Data Processing

EEG signals were analyzed off-line in the Matlab environment using the open source EEGLab signal processing Toolbox [18]. Data were firstly pre-processed to remove artefacts arising from extra-cerebral sources. Subsequently, time-frequency analysis of the signal during the gait cycle was performed. The pre-processing consisted of the following steps.

Data were band-pass filtered between 0.5 and 45 Hz, thus removing both slow drifts and high-frequency components, such as the power line noise. Flat and “bad” channels were removed for further analysis. A flat channel was identified if it showed amplitude around zero (flatline) for more than 5 s consecutively, whereas a channel was identified as “bad” if it was noisy for more than 90% of acquisition duration. An artefacts subspace reconstruction (ASR) algorithm was applied to all kept channels to identify and remove transient or high-amplitude artefacts, e.g., eye movement artefacts [19]. The ASR algorithm automatically identifies clean periods of the EEG data using a sliding window. An artefact rejection threshold, based on the obtained calibration signal distribution of variance and a user-defined cut-off parameter, is calculated. Data portions, whose variance is larger than this threshold, are removed and reconstructed from neighboring channels using a covariance matrix computed from the calibration data. According to the cut-off parameter k chosen by the user, the method can be more or less aggressive in removing EEG portions of data. In this work, a k = 15 was employed, accordingly to the range suggested by Chang et al., who validated the ASR method on real EEG data [20]. Furthermore, the independent component analysis (ICA) was applied to the ASR-cleaned EEG recordings exploiting the RUNICA Infomax algorithm as implemented in EEGLab [21]. Then, the independent components (ICs) accounting for nonbrain (artifactual) activities were identified using the ICLabel plugin [22]. Specifically, all ICs with less than 40% contribution attributed to the brain were rejected. Finally, the cleaned reconstructed EEG data were re-referenced to a common average reference.

To study the cerebral activity during the gait cycle, the cleaned EEG data were segmented into epochs exploiting the foot strike triggers provided by the integrated system. Each epoch lasted 1.8 s, from −200 to 1600 ms around each right foot strike, to comprehend a complete cycle (right–left–right foot strike) even for different subjects’ preferred speed. Few epochs that did not include the complete stride were removed. A final visual check of the quality of the cleaned epoched data was performed, and eventually, still noisy epochs were removed. At the end, 188 and 174 epochs were kept for CP01 and HC subjects, respectively.

Coherently with the walking phase, the initial baseline rest EEG acquisition was segmented in epochs of 1.8 s length after the pre-processing procedure and visually inspected for noise. Then, 32 and 55 epochs were kept for further analysis for CP01 and HC subjects, respectively.

The time-frequency analysis was performed exploiting the EEGLab function for the calculation of event-related spectral perturbation (ERSP) maps. The time-frequency decomposition and each resting and walking epoch was computed with the Fast Fourier Transform using a Hanning window tapering. The decomposition was performed for each EEG channel between 0.3 and 40 Hz, with a frequency resolution of 0.3 Hz and a time window of 400 ms. Walking epochs were time-warped to the median foot strikes events latencies. The final ERSP maps were computed by subtracting the mean resting state spectral power from the walking epochs spectrograms.

2.6.4. Gait Data Processing

All gait parameters such as the kinetics and kinematics of the ankle, knee, hip, trunk, and pelvis and spatiotemporal gait parameters were recorded step-by-step by the GRAIL system and analyzed with the Gait Offline Analysis Tool software (GOAT, Motek Medical, The Netherlands). GOAT Software allowed for the manual deletion of individual missteps, such as gait cycles that had a foot placement on only one of the force platforms or when passive markers were not visible. The total number of steps correctly recorded are reported in the results.

Then, data were exported and analyzed in Matlab with an ad hoc algorithm to compute the following parameters.

Spatiotemporal gait parameters:

• Percentage of stance phase with respect to the whole step;
• Length and width of the step;
• Length and duration of the stride.

Kinetics:

• Power of hip, knee, and ankle in the sagittal plane.

Kinematics:

• Flexo-extension of hip, knee, and ankle in the sagittal plane;
• Tilt and obliquity of pelvis computed in the sagittal and frontal plane, respectively;
• Trunk flexion in the sagittal plane.

The mean and standard deviation of all the selected steps were computed for each child (separately for the left and for the right side for CP01, as the mean value of the right and left leg for HC).

3. Results

3.1. Results of the Validation of the Gait Cycle Identification Procedure

The table below (

Table 1. Phidget performance in five healthy adults (HA) at three different treadmill speeds in three children with unilateral CP and one healthy child (HC). * preferred speed.

Speed	Subject	Step# (Camera)	TP	FN	FP	Sensitivity (%)	CSI (%)
0.5 m/s	HA1	169	169	0	0	100	100
	HA2	189	189	0	0	100	100
	HA3	133	133	0	0	100	100
	HA4	150	149	1	3	99.33	97.38
	HA5	178	178	0	0	100	100
1 m/s	HA1	216	212	4	0	98.14	98.14
	HA2	218	214	4	0	98.16	98.16
	HA3	211	206	5	0	97.63	97.63
	HA4	199	199	0	5	100	97.54
	HA5	248	242	6	0	97.58	97.58
0.9 m/s *	HA1	189	186	3	1	98.41	97.89
0.9 m/s *	HA2	212	208	4	0	98.11	98.11
1.2 m/s *	HA3	222	216	6	0	97.29	97.29
1.2 m/s *	HA4	211	210	1	0	99.52	99.52
0.8 m/s *	HA5	232	227	5	0	97.84	97.84
0.72 m/s *	CP01	489	312	177	0	63.80	63.80
0.55 m/s *	CP02	486	464	22	0	95.47	95.47
0.55 m/s *	CP03	515	505	10	0	98.05	98.05
0.8 m/s *	HC	654	622	32	0	95.10	95.10

) shows the Phidget triggering system performance in comparison to the detection performed with the video inspection at the different speeds tested. The average sensitivity of healthy adults was 98.68%, and the average CSI was 98.39%. The average sensitivity of the four children was 90.97%, and the average CSI was 90.95%.

Table 2. EMG performance in five healthy adults (HA) at three different treadmill speeds in three children with unilateral CP and one healthy child (HC). * preferred speed.

Speed	Subject	Step# (Camera)	TP	FN	FP	Sensitivity (%)	CSI (%)
0.5 m/s	HA1	169	159	10	4	94.08	91.90
	HA2	189	182	7	0	96.29	96.29
	HA3	133	133	0	0	100	100
	HA4	150	143	7	1	95.33	94.70
	HA5	178	178	0	0	100	100
1 m/s	HA1	216	216	0	0	100	100
	HA2	218	218	0	0	100	100
	HA3	211	211	0	0	100	100
	HA4	199	199	0	0	100	100
	HA5	248	248	0	0	100	100
0.9 m/s *	HA1	189	189	0	0	100	100
0.9 m/s *	HA2	212	211	1	0	99.52	99.52
1.2 m/s *	HA3	222	220	2	0	99.09	99.09
1.2 m/s *	HA4	211	211	0	0	100	100
0.8 m/s *	HA5	232	232	0	0	100	100
0.72 m/s *	CP01	489	477	12	0	97.54	97.54
0.55 m/s *	CP02	486	471	16	0	96.71	96.71
0.55 m/s *	CP03	515	398	117	24	77.28	73.84
0.8 m/s *	HC	654	642	12	0	98.16	98.16

reports the result of the EMG triggering system performance validation at the different speeds tested. The average sensitivity was 99.09%, and the average CSI was 98.92% on healthy adults. The average sensitivity on children was 92.68%, and the average CSI was 91.65%.

Both triggering systems show high values of sensitivity and CSI for all subjects. For the sample of children, in general, the EMG trigger signal was more reliable. However, for both triggering systems, we found two subjects for which one of the two systems showed sub-optimal sensitivity and CSI values (subject CP01 and CP03, respectively). Moreover, EMG measures could be problematic in the clinical scenario (e.g., in patients with hemiparesis) failing in the step detection.

For this reason, as previously explained, a new trigger signal was created combining both the EMG and the Phidget triggers. The sensitivity and CSI of the new signal were computed and are presented in

Table 3. Combined trigger signal: five healthy adults (HA) at three different treadmill speeds, in three children with unilateral CP and one healthy child (HC). * preferred speed.

Speed	Subject	Step# (Camera)	TP	FN	FP	Sensitivity (%)	CSI (%)
0.5 m/s	HA1	169	169	0	0	100	100
	HA2	189	189	0	0	100	100
	HA3	133	133	0	0	100	100
	HA4	150	150	0	3	100	98.03
	HA5	178	178	0	0	100	100
1 m/s	HA1	216	216	0	1	100	99.53
	HA2	218	218	0	0	100	100
	HA3	211	211	0	0	100	100
	HA4	199	199	0	5	100	97.54
	HA5	248	248	1	0	99.59	99.59
0.9 m/s *	HA1	189	189	0	1	100	99.47
0.9 m/s *	HA2	212	212	0	0	100	100
1.2 m/s *	HA3	222	222	0	0	100	100
1.2 m/s *	HA4	211	211	0	0	100	100
0.8 m/s *	HA5	232	232	0	0	100	100
0.72 m/s *	CP01	489	481	8	2	98.36	97.96
0.55 m/s *	CP02	486	486	0	0	100	100
0.55 m/s *	CP03	515	514	1	0	99.80	99.80
0.8 m/s *	HC	654	653	1	1	99.84	99.69

. The average sensitivity in the adult group was 99.96%, and the average CSI was 99.63%. Whereas in the children group, the average sensitivity was 99.39%, and the average CSI was 99.26%.

Using the combined signals, CSI and sensitivity increased for all participants and were around 98% or higher for both adults and children.

3.2. Multimodal Acquisition Results

Figure 4 shows a child performing the acquisition protocol used to test the feasibility of the multimodal recording. The child was walking on the treadmill in the wooden VR environment wearing the EEG cap for the EEG signal recording, the EMG electrodes for the EMG signals acquisition, and the passive marker according to the HBM2 model for the kinematic parameters recording. Moreover, kinetic data were acquired by the force platform installed on the treadmill. The EEG amplifier was placed on the treadmill handrail with the custom 3D-printed support.

3.2.1. EEG Data Results

The more accurate gait cycle identification obtained from the combination of Phidget triggers and EMG signal was used for the time-frequency analysis of the EEG signal to extract gait-related ERSP for both the healthy child (HC) and the child with CP (CP01).

Since we are focusing on the effect of the walking task over brain oscillations, we expected that fronto-central electrodes, placed on the scalp in correspondence to the motor and pre-motor cortex, would be the most sensitive to show a correct match between the triggering system and the cerebral rhythms variations locked to the gait cycle.

In Figure 5a,b, ERSP maps with respect to the resting-state condition for electrode FC5 and FC6, on left and right hemisphere respectively, of both subjects, are reported.

In both children (HC and CP01), we observe a desynchronization of alpha (8–12 Hz) and low beta (13–21 Hz) rhythms during the swing phase of the gait cycle, which is followed by a resynchronization to baseline values during the foot strike. The alpha desynchronization in the left ipsilesional hemisphere of the child with CP (FC5 electrode in Figure 5a) is stronger than the one occurring in the right hemisphere (FC6 electrode in Figure 5a). Indeed, on the left side, the desynchronization reaches −1.6 dB with respect to the baseline, while on the right side, −0.97 dB is reached. However, the desynchronization appears to be less influenced by the gait cycle as compared to the HC’s response. In addition, the low beta desynchronization in the patient with CP is less pronounced than the HC one. On the other hand, low-frequency synchronization locked to the foot strike is enhanced in the patient with CP, while higher frequency synchronization is accentuated in the HC one. In fact, in the theta band (4–7 Hz), we find a variation up to 3 dB with respect to baseline for the child with CP, whereas it remains below 2 dB in the HC. On the contrary, synchronization in the beta band (13–30 Hz) is predominantly observed for the HC during the foot strikes in both hemispheres, whereas the child with CP shows a less symmetric behavior: beta synchronization can be observed only in the right unaffected hemisphere (FC6 electrode), and no variations with respect to baseline are observed in the left hemisphere.

3.2.2. Gait Data Results

The recording of gait data were successfully performed. A total of 229 right steps and 237 left steps were collected for CP01; 256 right steps and 244 left steps were acquired for HC.

Table 4. Spatiotemporal parameters. For CP01, both healthy and paretic legs are shown. For HC, the mean value between legs is reported. Mean and standard deviation values are computed over the total number of steps. L: left; R right.

	CP01		HC
	Healthy Leg (L)	Impaired Leg (R)	Mean LR
Stance%	69.36 ± 1.22	68.58 ± 1.8	65.95 ± 1.65
Stride length (m)	0.88 ± 0.04	0.88 ± 0.05	0.85 ± 0.06
Stride time (s)	1.23 ± 0.06	1.23 ± 0.06	1.05 ± 0.06
Step Length (m)	0.45 ± 0.03	0.43 ± 0.02	0.42 ± 0.04
Step width (m)	0.16 ± 0.02	0.16 ± 0.02	0.17 ± 0.03

shows mean spatiotemporal parameters for the two children. CP01 showed values of stride length, step length, and step width comparable to HC. CP01 had a longer stride time, due to a lower gait speed (0.72 m/s with respect to 0.8 m/s for HC) and a higher percentage of stance over the whole duration of the gait cycle with respect to HC.

Figure 6 shows kinematic and kinetic data for HC (green) and CP01 (blue). The latter showed a more tilted pelvis than HC, while pelvic obliquity and trunk flexion were comparable. Furthermore, CP01 had both hemiplegic and healthy hips more flexed and hemiplegic ankle more dorsiflexed than HC. In contrast, knee flexion and healthy ankle were comparable to HC. Finally, CP01 showed a reduced power at toe off (about 60% of gait cycle) for the hip and knee of healthy side but a comparable power of ankle with respect to HC.

4. Discussion

The purpose of this work was to set up and test a system for the multimodal analysis of gait training and rehabilitation of children and adolescents in a VR environment, thus allowing the study of the electrophysiological correlates and the kinematic and kinetic parameters of the gait pattern. To reach this aim, we integrated a portable 32-channel EEG system with the GRAIL allowing the synchronization of the signals acquired by both systems. This was possible by using a 1018_2 Phidget I/O board, which physically connected the two systems. Specific application software was developed to control the Phidget, which generated a triggering signal for the synchronization of the two systems. Specifically, the triggers marked each IC during the subject’s locomotion so that the signals acquired by the EEG system can be aligned to the gait cycle and to the kinematic and kinetic signals recorded by the GRAIL, allowing the multimodal analysis of the gait pattern.

The implemented triggering system was validated comparing its performance in detecting the IC of both adults and children with a video recording visually inspected for counting the occurred steps. Results showed a good performance, with an average sensitivity of 96.55% and an average CSI of 96.29%. These values were higher for the adult group (96.68% and 98.39% respectively) than for the children group (90.97% and 90.95% respectively); this was not surprising taking into account that most of the enrolled children were affected by CP and adults generally have a more reliable gait pattern then children [23]. Since the Phidget triggers did not allow the identification of right and left steps, the gastrocnemii EMG signal was simultaneously acquired to allow this identification, and it was also proposed as an alternative off-line trigger signal for the step recognition. Thus, its performance was evaluated in comparison with the video recording data. Results showed a good performance of the EMG signal as well, with an average sensitivity of 99.09% and an average CSI of 98.92% on healthy adults and lower but still good values for children (92.68% and 91.65%, respectively).

However, both EMG and Phidget triggers showed some limitations due to missed steps (FN) and “double steps” (FP). In the case of the EMG signal, FNs occurred when the EMG peak detection algorithm did not identify the peak because of the low EMG signal amplitude. Whereas in the case of Phidget, they occurred when the IC detection failed because of incorrect placement of the subject’s feet on the treadmill platforms or because of an inaccurate selection of force range and time interval parameters used in the algorithm of real-time IC detection. On the other hand, the EMG FP values were due to the identification of “double” triggers for the same step caused by the activation of the gastrocnemius in the terminal swing phase or by its prolonged contraction in the pre-swing phase. In the case of the Phidget, they were probably due to a hesitant step of the subject or to an incorrect setting of the force range and time interval parameters used in the algorithm of real-time IC detection. FP and FN might increase when considering pathological subjects, since they can show abnormal muscle contraction as well as altered gait pattern [24], thus reducing the reliability of both triggering systems. Indeed, for both triggering systems, our data showed sub-optimal sensitivity and CSI values in one participant with CP. This limitation can be overcome by integrating the information coming from both the EMG and the Phidget, as demonstrated by the validation of the combined trigger signal we obtained by recovering Phidegt FN values from the EMG signal. This newly generated triggering signal showed a very high value of CSI and sensitivity for both children and adults, with an average sensitivity in the adult group of 99.96% and an average CSI of 99.63% and similar values in the children group, 99.39% and 99.26%, respectively. These values are higher than those previously reported in [25] and [26]. Thus, our analysis confirmed an overall goodness of the proposed on-line synchronization system (Phidget trigger), which are enriched by the possibility of recovering off-line the missing information thanks to the integration of the EMG signal trigger. Therefore, the combined systems provide a tool that facilitate the multimodal analysis of the pathological gait pattern, opening new possibilities for gait analysis and rehabilitation.

This approach presents some limitations. First of all, it needs an off-line processing, and as a consequence, it cannot be used in applications that foresee online data analysis, such as brain computer interface approaches, which are often proposed in VR environments. Nevertheless, if online data analysis is required, it is possible to count on the online Phidget trigger. Secondly, the trigger recovered by the EMG does not have a precise temporal information, since the temporal occurrence of this trigger is reconstructed based on the information of the other steps detected with the Phidget trigger. However, we expect that this error is small enough to not affect the successive analysis.

Once we had validated our synchronization mechanism, we tested the feasibility of performing a multimodal analysis of the gait pattern with the integrated GRAIL-EEG system. To our knowledge, no other studies proposed a system, which allows a real-time synchronized acquisition of multimodal signals (i.e., EEG, kinematics and kinetics) during gait rehabilitation in a VR environment. Calabrò et al. [27] used the combination of the EEG and a wireless inertial sensor to evaluate the effects of rhythmic auditory stimulation during treadmill gait rehabilitation in Parkinson’s patients. However, this combination did not include a VR system; it allowed the study of the EEG merely with the gait spatiotemporal parameters, without kinematic and kinetic data; moreover, an online signal synchronization system was not presented. The same group [17] also combined EEG and gait parameters during VR RAGT performed by unilateral ischemic stroke patients. In this case, the EEG was directly synchronized with the VR RAGT system. However, the gait parameters analyzed were limited to the hip and knee flexion/extension forces. Other studies (e.g., [28,29]) combined the acquisition of EEG and optoelectronic systems for the investigation of the cortical activity associated with the locomotion, in both healthy subjects and clinical populations. Thus, spatiotemporal or kinematic data were analyzed in combination with the EEG. Nevertheless, a real-time synchronization of the two systems was not always implemented, and VR environments were not included in these studies. Our results show the capability of the integrated system to detect EEG, kinematic, and kinetic data related to the gait cycle in both a healthy child and a child with CP. All the acquired signals were successfully analyzed, as previously done in the literature [5,30,31], showing reliable EEG activity modulated by the gait cycle as well as gait parameters in accordance with the literature both for the healthy child and the child with CP [32]. Thus, even if an extensive test of the integrated system with a larger sample of both healthy and pathological children is still needed, our results are promising and show the capability of the system to allow a multimodal analysis of the gait pattern in VR environment. This can be extremely useful to deepen the study of the gait neural correlates in pathological pediatric populations and to monitor the effect of the VR gait rehabilitation evaluating both the motor and the related neural recovery.

It should be noted that the simultaneous and synchronous acquisition of other biological signals could be easily integrated in our system (i.e., respiratory signal, the skin conductance, the temperature and acceleration measured by the sensors included in the EEG system), enriching the multimodal recording. Moreover, the introduction of an online analysis of some of the acquired signals might be used to create fully personalized rehabilitation protocols that adapts in real time to the patient activity.