Air Force Pilot Expertise Assessment during Unusual Attitude Recovery Flight

Abstract

Pilot training and expertise are key aspects in aviation. A traditional way of evaluating pilot expertise is to measure performance output. However, this approach provides a narrow view of the pilot’s capacity, especially with regard to mental and emotional profile. The aim of this study is hence to investigate whether neurophysiological data can be employed as an additional objective measure to assess the expertise of pilots. In this regard, it has been demonstrated that mental effort can be used as an indirect measure of operator expertise and capacity. An increase in mental effort, for instance, can automatically result in a decrease in the remaining capacity of the operator. To better investigate this aspect, we ask two groups of Italian Air Force pilots, experienced (Experts) and unexperienced (Novices), to undergo unusual attitude recovery flight training simulations. Their behavioral (unusual attitude recovery time), subjective (mental effort demand perception) and neurophysiological data (Electroencephalogram, EEG; Electrocardiogram, ECG) are collected during the entire flight simulations. Although the two groups do not exhibit differences in terms of unusual attitude recovery time and mental effort demand perception, the EEG-based mental effort index shows how Novices request significantly higher mental effort during unusual conditions.

1. Introduction

When flight training simulations are used to evaluate and compare the utility of tactical operating procedures, the competence and expertise of the pilots are essential. Military pilots are often required to constantly make quick decisions in a rapidly changing environment and to perform cognitively challenging tasks under immense temporal pressure [1,2,3]. Flight training simulations are therefore crucial to their ability to perform and deal with emergencies and unexpected situations. Flight training is characterized by a mixture of factors, such as training content, training environment, use of tools, procedures and decision making, which increase the complexity of training and entail many cognitive processes [4,5,6,7]. Pilot expertise can be measured through several tools.

Conventional methods to gather information about pilots’ psychophysical and operational status and to evaluate their expertise are typically based on expert supervision (briefing and de-briefing), self-reports or performance statistics [8,9]. For example, a conventional metric for evaluating pilot experience and capacity is the number of hours flown (NHF). This is a good and valuable parameter for understanding these aspects, but it does not provide any insight about the mental effort requested by the pilot while performing specific flight phases. For example, a pilot can have a high NHF, but in a specific moment or situation, due to intense work shift or schedule or personal reasons (e.g., bad arguments or stressful events right before the flight), a standard and simple flight may be very demanding. From this perspective, it is clear how the NHF cannot provide any information about a pilot’s capacity. Furthermore, these measurements are highly operator-dependent (who may be prone to personal experiences and cognitive and emotional biases), require interrupting the execution of tasks (invasiveness and low temporal resolution) and do not include information related to teamwork (paucity of user’s insights). It is therefore clear how these measurements alone cannot be used to accurately and properly assess a pilot’s expertise. Neurophysiological measures (e.g., Electroencephalogram, EEG; Electrocardiogram, ECG) have gained momentum in different research and operative areas and represent an objective, unobtrusive and powerful tool to determine a user’s affective–cognitive state on the basis of mind–body relations [10,11].

In this regard, Human Factors (HFs) and Crew Resource Management (CRM) refer to psychological concepts linked to cognitive and emotional processes, such as Mental Effort, characterizing and affecting individual behavior [12]. Mental effort represents the amount of cognitive resources involved in performing a task [13,14,15,16,17,18]. It is seen as a combination of perceived demand characteristics, depth of information processing and personal expertise [13,15]. Perceived demand characteristics mainly depend on the inherent complexity of the task content, which is related to the degree of interaction between various information elements [15]. In other words, mental effort can be used as an indirect measure of operator capacity. An increase in mental effort, for instance, can automatically result in a decrease in the remaining cognitive capacity of an operator (spare cognitive capacity) [19]. In this regard, recent advances in neuroimaging techniques, such as functional Magnetic Resonance Imaging (fMRI), Regional Cerebral Blood Flow (rCBF), Positron Emission Tomography (PET), Magnetoencephalogram (MEG) and Electroencephalogram (EEG) allow cognitive processing and its neural correlates to be studied noninvasively in humans. For example, efforts have been made to use these techniques to examine brain regions involved in calculation [20,21], long term potentiation [22], encoding of new information [23,24], successful maintenance of information in memory [25], performance monitoring [26,27,28,29], attention [30,31,32] and working memory [33,34,35,36,37,38,39,40]. All these aspects are linked to mental effort. These studies suggest and discuss the roles of the dominant brain hemisphere and of different cortical areas in these mental functions. In particular, rCBF, fMRI and PET studies showed increased blood flow in the prefrontal cortex, specifically in the Anterior Cingulate Cortex (ACC) [41,42,43,44]. EEG results showed increased frontal EEG activity in theta band (4–8 Hz) associated with the performance of various mental tasks [3,27,29,36,38,45,46]. In other words, increases in EEG theta activity with task difficulty suggest that this brain activity is related to the mental load allocated to task performance. This effect is present regardless of task modality [47]. Furthermore, MEG studies localized the electrical current dipoles responsible for the magnetic frontal theta wave [42,48]. The current dipole of the frontal theta wave is found not necessarily in the midline part but is distributed in relatively wide areas of the frontal lobes of both left and right hemispheres. Moreover, it was always found that the current dipoles for the frontal theta wave dynamically move around in the frontal lobes of both left and right hemispheres during mental activities. Such findings can be explained by a vector summation of the multiple electrical current dipoles scattered in the left and right frontal lobes. Possibly many current dipoles successively occur in wide areas of the frontal lobes on both sides, and some of them may be synchronized in phase [42,48]. These studies have hence shown that the frontal brain areas and theta activity are reliable indicators of mental effort elicited by tasks of varying complexity. Among the mentioned techniques, in our study, we employed the EEG because of its time resolution, portability and usability in real settings. Moreover, other physiological measures have been shown to be sensitive to mental effort. Heart rate variability (HRV) in studies with military populations, for instance, was found to increase with an increase in mental effort [49,50]. Higher levels of resting vagally mediated HRV are linked to the performance of executive functions, such as attention and emotional processing by the prefrontal cortex [51]. Afferent information processing by the intrinsic cardiac nervous system can modulate frontal-cortical activity and impact higher-level functions [52].

The computation of a synthetic neurophysiological index can therefore radically change the entire field of pilot expertise assessment and comparison with respect to a population of experienced ones. In fact, the capability of having a reference value (threshold) that estimates how much each pilot deviates would be very useful for training program management.

With these challenges in mind, this work aimed at investigating the benefit of employing neurophysiological measures (EEG and ECG), in combination with the conventional ones, to provide an additional and objective measure for more accurate pilot expertise assessment and comparison. In this regard, as mentioned above, the number of hours flown (NHF) represents overall information about the flight experience of a pilot, but it does not provide the instructor with any online information while dealing with flight simulations in order to, for example, identify mental effort peaks corresponding to particular procedures, maneuvers or conditions so the instructor can better tailor the training program, nor does the NHF provide any insight related to the amount of cognitive resources requested by the pilots to handle such flight situations. In this regard, we gathered neurophysiological, behavioral and subjective data from Expert and Novice pilots while performing the same flight training simulation, evaluated the results derived from the different measures and finally proposed a metric by which to quantify the expertise of Novices with respect to Experts regarding mental effort requested.

2. Materials and Methods

2.1. Experimental Subjects

A total of 13 pilots from the 61° Stormo of the Italian Air Force base in Galatina (Lecce, Italy) took part in the study. The pilots were selected on the basis of their rank and flight experience, and then they were divided into two groups according to the latter parameters. A group of 6 Expert pilots (mean age: 34 ± 3 y, mean flight experience: 1450 NHF, 467 h on the MB339) and a group of 7 Novice pilots (mean age: 26 ± 1 y, mean flight experience: 157 NHF, 37 h on the MB339 airplane). The flight simulations were conducted with the flight simulator of the MB339, an alpha airplane. Figure 1 shows the experimental setup in the Italian Air Force simulator facility. The experiments were conducted following the principles outlined in the Declaration of Helsinki of 1975, as revised in 2000. The experiments were approved by the Ethical Committee of the Sapienza University of Rome (protocol code 1211/2014). Informed consent was obtained from each subject on paper after the study explanation, and all the data were pseudonymized to prevent any association with subject identity.

2.2. Flight Simulation

The flight simulations consisted of training on recovery from unusual attitudes. The pilots took off from the 61° Stormo Base in Galatina (TOFF), climbed to the flight level specified (CLIMB) and then reached the training area. Once there, the pilots were asked to read some information on the Flight Information Publication (FIP) book while the instructor was taking the airplane to the unusual attitude from the remote-control station (Figure 1a). In particular, the three unusual attitude conditions were Nose-Down (UPSET_1), Nose-Up (UPSET_2) and Spin (UPSET_3), and they were initiated only after the pilots recovered the airplane attitude. The time for recovering each of these unusual attitudes (URT) was considered a behavioral measure. The flight simulation scenario also included two failures. The first one (FAIL_E) occurred during the taxi phase (TAXI) and consisted of an electrical problem with the gear, and the second one was the AC Essential Bus failure (FAIL_AC), which was induced a couple of minutes after the end of the recovery training phase. After the pilots solved the FAIL_AC, they headed back to the base for landing (LAND).

2.3. Neurophysiological Signal Recording

For this experiment, a standard 10–20 EEG cap with 16 gel-based passive Ag/AgCl electrodes and a g.USBAMP EEG amplifier (g-tec GmbH, Schiedlberg, Austria) were used to acquire the pilots’ neurophysiological signals. In particular, the EEG signals were recorded with a sampling frequency of 256 Hz through 15 channels referenced to both the earlobes, and the investigated scalp positions were Fpz, AF3, AF4, F3, Fz, F4, C3, Cz, C4, T7, T8, P3, Pz, P4 and POz. The pilots’ heart activity (ECG signal) was recorded synchronously with the EEG and with the same sampling frequency from the 16th electrode placed on the pilots’ chest. The impedances of the electrodes were kept below 10 kΩ. Due to technical issues during the experiments, two of the Novice datasets were discarded. After equipment and pilot preparation, the flight simulation started, and it lasted about 1 h and 30 min on average.

2.4. Subjective Assessment: Likert Scale

At the end of the flight simulation, the pilots filled in a 5-point Likert scale to obtain a subjective measure of the pilots’ mental effort demand perception throughout the different phases of the flight simulation [53,54]. The Likert scale consisted of a 5-point scale, where the value “1” means “very easy” and “5” is “very demanding”. The question “how demanding was the X phase”, where X = READY; TAXI; FAIL_E; TOFF; CLIMB; UPSET#1; UPSET#2; UPSET#3; FAIL_AC; LAND, was listed for each flight phase on a paper sheet, and the pilots and SME had to provide their score (from 1 to 5) next to the different items. The Likert scale was filled at the end of the simulation to avoid any interference during the experiment and to keep the flight simulation as realistic as possible. The instructor (Subject Matter Expert, SME) also provided a unique subjective assessment of the different flight phases.

2.5. Mental Effort (MEF) Index Estimation

The EEG signal was firstly band-pass filtered with a 5th-order Butterworth filter in an interval of 2–30 Hz. The blink artifacts were detected by the Reblinca method [55] and were corrected by leveraging the ocular component estimated through a multi-channel Wiener Filter (MWF) [56]. Since the EEG is a non-stationary signal [57,58] for the estimation of any temporal (mean, median, standard deviation) or frequency (Power Spectral Density, PSD; Global Field Power, GFP) parameter, it is necessary to satisfy some preliminary criteria of stationarity. One of the most common ways for making the EEG stationary is to segment it into epochs of 1 or 2 s. In fact, it has been widely demonstrated that, within such a period of time, the EEG can be considered stationary [58,59]; therefore, the estimation of the temporal and frequency parameters are reliable. From this perspective, the EEG signals were then segmented into epochs of 1 s. From this perspective, the EEG signals were then segmented into epochs of 1 s, and if the EEG signal amplitude exceeded ±80 μV, the corresponding epoch was marked as an artifact (threshold criterion). From the artifact-free EEG, the Global Field Power (GFP) [60,61] was calculated for the EEG frequency band of interest, which is the theta band.

This band was defined accordingly with the Individual Alpha Frequency (IAF) value [24]. Since the alpha peak is mainly prominent during rest conditions, the pilots were asked to rest and keep their eyes open for a minute before starting the flight simulation (EC condition). Such a condition was then used to estimate the IAF value specifically for each pilot. Consequently, the EEG theta band was defined as [(IAF−6) ÷ (IAF−2)] Hz. Among the EEG channels and based on the objective of this work, only the frontal ones (AF3, AF4, Fz, F3 and F4) were considered. In fact, as described previously, it has been widely demonstrated that brain activity in the theta (ϑ) frequency band estimated over the frontal channels is linked to mental effort changes. We have therefore calculated the GFP in the theta band over the frontal channels, and then we averaged these values to provide a synthetic Mental Effort Index (MEF) for each pilot and within each flight phase. In particular, the MEF index was defined according to the literature and previous results as:

$$MEF=GF{P}_{\vartheta }\left(AF3, AF4, Fz, F3, F4\right)=\frac{1}{N} {\displaystyle \sum }_1^N{x}_{\vartheta , i}^2\left(t\right)$$

where the average of the squared EEG signals (x) was filtered in the theta band (ϑ) over the frontal brain areas (N = 5 electrodes) [62]. As demonstrated by the studies mentioned previously, we expected to find an increase in the pilots’ MEF index, with an increase in the mental effort demand of the flight phases.

2.6. Heart Rate Variability (HRV) Estimation

The ECG was filtered using a 5th-order Butterworth band-pass filter (high-pass filter: cut-off frequency fc = 5 Hz; low-pass filter: cut-off frequency fc = 15 Hz) to reject the continuous component and the high-frequency interferences, such as that which was related to the main power source.

The following step consisted of measuring the distance between consecutive R peaks (each R peak corresponds to a heartbeat) of the ECG signal to estimate heart rate (HR) values and thus the tachogram. In this regard, the Pan–Tompkins algorithm [63] was employed. Other artifacts of the HR signal were automatically corrected using the HRVAS suite [64]. Finally, a spectral analysis of the HR signal was performed to estimate the HRV using the Lomb–Scargle periodogram. This method has been demonstrated to produce much more accurate estimates of the Power Spectral Density (PSD) than Fast Fourier Transform (FFT) methods for typical HR data [65]. Since the HR data are unevenly sampled, another advantage of the Lomb–Scargle method is that it can be used without the need to resample and de-trend the RR data [66] in contrast with FFT-based methods.

Thirty-second windows were considered to obtain a frequency resolution of 0.033 Hz and to allow the analysis of the characteristic HRV frequency sub-bands. In particular, in line with the scientific literature [67], the PSD of the HR signal was computed. Among the frequency components, low (LF: 0.04 ÷ 0.15 Hz) and high frequencies (HF: 0.15 ÷ 0.4 Hz) were considered, and the LF–HF index was defined according to the literature as:

$$\mathrm{L}\mathrm{F}\text{–}\mathrm{H}{\mathrm{F}}^{}=\frac{\mathrm{L}\mathrm{F}}{\mathrm{H}\mathrm{F}}$$

The LF–HF estimates the ratio between sympathetic nervous system (SNS) and parasympathetic nervous system (PNS) activity under controlled conditions. The assumptions underlying the LF/HF ratio are that LF power may be generated by the SNS and that HF power is produced by the PNS. In this model, a low LF/HF ratio reflects parasympathetic dominance. This is seen when we conserve energy and engage in tend-and-befriend behaviors. In contrast, a high LF/HF ratio indicates sympathetic dominance, which occurs when we engage in fight-or-flight behaviors or parasympathetic withdrawal [68,69]. In other words, we expected to see an increase in the LF–HF index when the mental effort demand increased.

2.7. Statistical and Correlation Analyses

The MEF, LF–HF and LIK scores were normalized for each pilot to obtain comparable values and to perform group statistics. In particular, the MEF and LF–HF of each pilot was normalized by using the mean and standard deviation of the corresponding resting condition (EC) distribution. These parameters were then used to calculate the z-score [70,71,72] values of the remaining experimental conditions.

The LIK score values were normalized within the [0 ÷ 1] range by the following formula:

$$Normalised LIK scor{e}^{}=\frac{ LIK score-\min \left(LIK score\right)}{(\max \left(LIK score\right)-\min \left(LIK score\right)},$$

where min(LIK score) is the minimum, and max(LIK score) is the maximum value of the LIK score distribution of the pilot that was considered. Group statistics were performed on the normalized parameters. In particular, the Wilcoxon signed-rank test was employed to compare the different parameters of the two groups with a statistical significance of α = 0.05. Due to the small sample size, we could not perform repeated-measures analysis. However, we wanted to perform statistics to provide preliminary and useful evidence to assess pilot expertise. We therefore employed only pairwise tests on the average values of each specific flight condition between the two groups (Experts vs. Novices), where no multiple comparison correction (i.e., Bonferroni correction) was necessary.

Furthermore, Pearson’s correlation analyses were performed among the LIK scores of the Experts, Novices and SME to investigate how mental effort perception may differ based on expertise and user perspective (user dealing with the task vs. external supervisor).

Finally, Pearson’s correlation analyses were performed between the LIK score of the pilots and the corresponding EEG-based MEF index values to evaluate coherence between subjective and neurophysiological measures. Since the degrees of freedom (df) for correlation analysis are the total number of score pairs (N: number of flight phases = 10) minus 2, for our analyses, the degrees of freedom of the correlation analyses performed in our study are df = 10 − 2 = 8.

3. Results

3.1. Recovery Time of the Unusual Attitude Conditions

The Wilcoxon signed-rank test between the unusual attitude recovery time (URT) of the Experts (blue bar) and Novices (orange bar) did not report any significant (p = 0.73) difference (Figure 2). In other words, the behavioral measures of the two groups during the unusual attitude conditions, which are the time to realize that the aircraft had entered an unusual attitude, take back the aircraft control, understand the situation and apply the right procedure to recover aircraft attitude, did not differ significantly. In particular, the median URT to deal with the unusual attitude conditions for the Experts was 78 s, whereas the median URT for the Novices was 76 s.

3.2. Self-Reported Mental Effort

Similar to the previous result, the Wilcoxon signed-rank test on the mental demand perception (LIK score) of the Experts (blue bar) and Novices (orange bar) did not differ significantly (p = 0.73). This means that the two groups perceived the mental effort demand of the different flight phases, failures and unusual attitude recovery conditions in a similar way (Figure 3).

However, Pearson’s correlation analysis between the LIK scores of the two groups returned a low (R = 0.35) and not significant (p = 0.32) correlation. In other words, the ratings of the Experts and the Novices do not have the same trend; therefore, they perceived mental effort demand differently across the different flight conditions. Pearson’s correlation analyses were also performed between the LIK scores of each group (Experts: blue line, Novices: orange line) and the ones provided by the SME (green line in Figure 4). The results show moderate and not significant correlations for both the Experts (R = 0.47; p = 0.16) and the Novices (R = 0.61; p = 0.06) with the SME’s mental effort demand perception.

Table 1. Results of the Pearson’s correlations analyses on subjective mental effort perceptions.

Comparison	R	p
Experts–SME	0.47	0.16
Novices–SME	0.61	0.06
Experts–Novices	0.34	0.32

reports the results of the different Pearson’s correlation analyses.

3.3. HRV Results

The result derived from the Wilcoxon signed-rank test on the LF–HF values of the Experts (blue bar) and Novices (orange bar) did not exhibit any significant (p = 0.6) difference between the two groups (Figure 5). This evidence shows that the information processing of frontal-cortical brain activity modulated by the intrinsic cardiac nervous system does not differ between the two pilot groups.

3.4. EEG-Based Mental Effort (MEF) Index

The results derived from the Wilcoxon signed-rank test on the EEG-based MEF index of the Experts (blue bar) and Novices (orange bar) report a significant difference (p = 0.035) between the two groups (Figure 6). This result shows how the Novices required significantly higher mental effort to complete the flight simulation.

Furthermore, replicating the statistics on the MEF index within each different flight phase of the simulation, the results highlight a very interesting aspect (Figure 7). Regardless of whether the flight phase was a standard one such as taxi (TAXI), takeoff (TOFF) or landing (LAND), the two groups required a similar (p > 0.05) amount of mental effort. However, during unusual conditions such as the resolution of the first failure (FAIL_E) and the recoveries from the unusual attitudes (UPSET_1, UPSET_2, UPSET_3), the Novices (orange line) exhibited significantly higher mental effort requests than the Experts (blue line).

The Pearson’s correlation analyses between the MEF index of each group, their mental effort demand perceptions (LIK score) and the SME’s subjective measure (ILIKSA score) returned the results in

Table 2. Correlations between the MEF and LIK values of the pilots.

Group	R	p
Experts	0.23	0.52
Novices	0.41	0.23
Experts–SME	0.47	0.16
Novices–SME	0.61	0.06

. Only a moderate (R = 0.61) but not significant (p = 0.06) correlation can be noticed for the Novices.

3.5. Comparison with Respect to the Expert Population

Since the Experts and Novices reported the same operational behavior (no statistical differences from the behavioral and subjective measures), we wanted to propose a method by which to quantify how much of the expertise of each Novice deviates from the Expert population with respect to mental effort requested. In this regard, we applied a z-score transformation by firstly calculating the mean (mu_Experts) and standard deviation (std_Experts) of the Expert MEF value population and then used these parameters to calculate the z-score values of each Novice’s MEF index (named MEF’) over the different flight simulation conditions, as reported in the following formula:

$$ME{F}^{\prime }\left(i,j\right)=\frac{MEF\left(i,j\right)\text{–} mu\text{\_}Expert{s}_{}}{std\text{\_}Expert{s}_{}},i=1÷5; j=1÷10$$

where the normalized mental effort index (MEF’) for each ith Novice and jth flight simulation phase was calculated with respect to the Expert population.

Based on the MEF’ values of each Novice assumed during the flight training simulation, we were able to quantify how much each Novice deviated from the Expert population with respect to the overall mental effort requested. For example, in Figure 8, it is possible to see how all the Novices requested higher overall mental effort with respect to the Experts (all z-score values are positive). In addition, this analysis highlights how Novice #5 had an overall mental effort request that was higher than those of the other Novices.

Finally, we also reported the MEF’ index values throughout each flight phase (Figure 9) to identify the conditions where each Novice deviated more, both from the other Novices and from the Expert population. For example, the “FAIL_E” failure resolution and “UPSET_3” unusual attitude recovery conditions should be further trained for Novice #2, Novice #4 and Novice #5, as they exhibited higher MEF’ values than the others.

4. Discussion

The objective of our work was to demonstrate how neurophysiological measurements can provide objective and useful information in addition to conventional ones (self-reports and external observations) for more accurate assessments of pilot expertise and skill. From this perspective, we recruited two groups of air force pilots, namely Experts and Novices, having different expertise with respect to rank and number of hours flown (NHF). The two groups were asked to perform the same unusual attitude recovery training simulations, where a couple of electrical failures were inserted without informing them.

The time to recover the aircraft from unusual attitudes (URT), the subjective perception of the mental effort demand (LIK score) and the pilots’ brain and heart activity were gathered throughout the entire simulation, and they finally analyzed to compare the two groups (Figure 1).

The results derived from the behavioral data (URT: Figure 2) and subjective perception (LIK score: Figure 3) do not indicate any significant difference between the Experts and Novices (all p > 0.05). In other words, by the conventional approach of evaluating how well the pilots dealt with the different flight conditions, the two groups did not differ. Therefore, these subjective measures [73] did not return any significant differences.

On the other hand, the EEG-based MEF indexes revealed that the Novices requested significantly higher (p = 0.035) mental effort than the Experts (Figure 6). This difference was particularly evident during unusual flight phases, such as the resolution of the first failure (FAIL_E) and the three unusual attitude recoveries (Figure 7). This evidence highlights how, during standard flight phases such as takeoff (TOFF), climbing (CLIMB) or landing (LAND), Novices and Experts did not differ significantly. However, when something unexpected happened (electrical failure) or during unusual conditions (unusual attitude recovery), the Novices required higher mental effort.

Looking in detail at the MEF values throughout the flight phases of the two groups (Figure 7), especially the dashed lines representing the linear trend of the MEFs, it is quite clear how the mental effort experienced by the Experts (blue line) was almost the same across the entire flight mission. However, we noticed an increase during the second failure (FAIL_AC). This can be explained by the fact that FAIL_AC is an electrical failure that may cause the loss of all the electrical equipment of the aircraft; therefore, they had to analyze the situation carefully and apply corresponding procedures to solve it. The trend of the overall Novices’ MEF (orange line) increased during the flight simulations (as reported by the LIK scores), particularly in some phases such as FAIL_E, UPSET_1, UPSET_2, UPSET_3 and FAIL_AC with respect to standard phases such as TAXI or TOFF. Additionally, there is a consideration for the Novices in the FINAL_CK condition. As described previously, the FAIL_AC failure can have important consequences on the electrical system, such as the eventual loss of the aircraft control. For this reason and due to poor experience in facing and dealing with this kind of situation, the Novices kept checking the instruments to ensure that nothing was becoming worse before the landing; therefore, their MEF increased up to the final checklist (FIANL_CK) and decreased only when landing.

In other words, during the resolution of failures and unusual attitude recoveries, the spare cognitive capacity of the Novices was lower than that of the Experts, which indicates that there were less cognitive resources available for dealing with more unexpected or demanding events. In this regard, we considered the Experts’ MEF indexes as a reference population from which to normalize the Novices’ MEF. We z-score normalized the Novices’ MEF values by using the mean and standard deviation of the Experts’ MEF value population to define a metric (MEF’) to quantify the deviation of each Novice regarding the mental effort requested, with respect to the Expert population. This metric (MEF’) allowed us to identify the Novices and flight phases which deviated more from the Expert population (Figure 8 and Figure 9).

The capability to objectively measure a Novice’s mental effort during flight simulation training is therefore very important. In fact, this aspect can allow for better assessment of a Novice’s progress throughout the training sessions, for identifying which flight phases were particularly critical for each of them and finally for tailoring training sessions based on each Novice’s attitude and skill [74,75,76].

Regarding Pearson’s correlation analysis, we aimed at:

• Evaluating the similarity between the Experts and Novices with respect to mental effort perception (LIK score).
• Evaluating the similarity between the EEG-based mental effort index (MEF) and the subjective mental effort perception (LIK score) between the Experts and Novices.

The main results derived from these correlation analyses show that there is no correlation between the Experts and the Novices and no correlation between the EEG-based MEF index and the LIK score. In other words, these results demonstrate that (1) the mental effort perception of experienced operators (Experts) differ from non-experienced operators (Novices), and this aspect may likely depend on the different skills and experiences of the pilots; and that (2) subjective measures (self-reports) represent the perception of what the user is feeling and not an objective measure of what the user is doing or experiencing regarding cognitive demand. The non-significant correlation between the MEF and LIK score shows therefore that neurophysiological-based measures can provide a more objective and direct measurement of what a user requires with respect to mental effort demand. To summarize, the results without a correlation emphasize the objective of our work, which is demonstrating how the employment of neurophysiological measures can provide an additional and useful measure to assess pilot expertise and skill. Furthermore, there are only moderate correlations between the Novices’ MEF and the LIK score provided by them and the SME. This result highlights also how subjective measures can potentially differ if provided as an external supervisor (SME) or as the user dealing with the task (pilots). In this regard, Veltman and Gaillard [77] already demonstrated how participants may have difficulty in distinguishing task demands from invested effort, and hence this may be a potential limitation of the subjective measures.

Although we were able to perform realistic flight simulations, recruit qualified personnel and provide interesting results, the evidence described in this work must be considered as a suggestion and preliminary result for further studies. In this regard, some limitations must be considered. Firstly, the number of pilots within each group should be increased [78]. In fact, due to the pilots’ scheduling and time available for running the experiments, we could not collect a larger sample size. In the next study, we will therefore enlarge the number of both Expert and Novice pilots to obtain a proper sample size, as demonstrated by Vozzi et al. [78], for making our results and conclusions more robust. Secondly, the subjective rating should be performed along with the flight simulation, for example, by asking both the pilots and the SME to rate the phenomenon considered (mental effort) via radio, to have more reliable scores rather than at the end of the simulation. In this regard, we preferred to keep the flight simulation as realistic as possible, and we did not interfere in any way and time with the main task execution. Thirdly, since we wanted to investigate whether neurophysiological measures can provide benefits to assess pilot expertise in combination with the conventional ones, i.e., behavioral (URT), subjective (ISA) and flight experience (NHF), we may have performed correlation analyses between these measurements. Due to limited points of measurement (three recoveries) and a unique value of NHF per pilot, we could not address this aspect. In fact, for accurately investigating the correlations among these measurements, a repeated-measures correlation analysis [79] between pairs of different parameters (MEF vs. NHF vs. URT) must be performed; therefore, we can assess if high and significant correlations occur at the single-subject level (as we are interested in evaluating each pilot’s expertise) and not at the group level. In this regard, a higher number of recovery maneuvers will be inserted in the next experimental protocol to obtain a proper number of observations and to be able to evaluate correlations between URTs and other parameters, such as the MEF (neurophysiological data) and LIK scores (subjective data). Finally, additional neurophysiological signals, such as a pilot’s Electrodermal activity (EDA), can be collected for considering more mental (Mental Fatigue [80,81,82,83], Stress [68], Attention [32], cognitive control behavior [76]) and emotional aspects [84,85] when comparing Expert and Novice pilots.

5. Conclusions

This study involved qualified military pilots, Experts and Novices, who performed the same flight simulations. Due to some limitations in recruiting the pilots, the final sample size was not very big, and therefore the results described in this work should be considered as suggestions and preliminary evidence for further investigation.

This study describes how the employment of diverse and complementary measures can provide a more accurate approach to compare and assess Novice pilots’ expertise and competencies. In fact, by considering only conventional measures such as operational behavior (how well the user performed the task considered), and subjective data (difficulty perception), we did not find any significant differences between the two pilot groups.

On the other hand, the EEG-based Mental Effort index (objective measure) demonstrates how the Novices requested a significantly higher mental effort than the Experts, especially during unusual flight conditions.

We can therefore state that neurophysiological measures can be employed as additional and objective measures to support the instructors in better assessing pilots’ progress with respect to a reference (Expert population), identifying flight phases which may require additional training for each different Novice, and tailoring training sessions based on the different outcomes of each Novice.

In this regard, as a result of the progress of wearable technology, neurophysiological signals can already be potentially used in operational environments as real-time measures of monitoring pilot mental states without requiring filling in questionnaires (secondary task) and interrupting the execution of the flight simulation.