Wind Shear Operation-Based Competency Assessment Model for Civil Aviation Pilots

Abstract

Undesirable meteorological conditions are typical aviation flight safety threats. Although most meteorological radar and flight augmentation computers have avionic system wind shear alarms, the preferred approach is that pilots avoid unsafe wind shear events. Therefore, effective pilot competency evaluations are needed to assess pilots’ abilities to deal with these events. This study developed a wind shear operation competency model that includes observable behavior indicators, sub-task decomposition, and competency check items. An adapted competency model and a quantitative data-driven competency evaluation criteria optimization method were then developed using three-dimensional competency feature modeling, after which wind shear simulation flight training data were used to verify the effectiveness of the proposed method. The competency assessment grades were significantly correlated with the results from experienced examiners with a 93.33% evaluation accuracy. The proposed quantitative data-driven competency assessment method can provide effective pilot competency assessments for a range of aviation meteorological threats.

1. Introduction

Wind shear, which is a complex weather phenomenon characterized by sudden horizontal or vertical changes in wind speed, seriously threatens flight safety. When an aircraft encounters wind shear, its airspeed, climb rate, angle of attack, lift coefficient, and other parameters can suddenly change, which can seriously affect maneuvering and altitude control, and, if not properly handled, can result in an aviation accident. Low-level wind shear, in particular, which can cause drastic horizontal or vertical changes in wind vector directions below 600 m [1,2], is a significant takeoff and approach risk factor. Low-level wind shear usually occurs at low altitudes because of severe convection, frontal systems, radiative inversions, and special airspace, all of which can result in a drastic loss of aircraft energy [3,4]. Therefore, if the crew does not take effective, timely responses, accidents may occur. These low-level wind shear hazards have been defined by the International Civil Aviation Organization (ICAO) as unsafe events that require additional monitoring. The International Air Transport Association (IATA) reported that 31% of all accidents in 2022 were the result of adverse weather conditions, with the most often cited contributing factors (18% of the accidents) being wind shear and thunderstorms [5]. In China, the flight quality-monitoring big data platform reported that 21.31% of the red events in 2022 were also wind shear alarm events. Recent typical unsafe wind shear-related events are shown in

Table 1. Typical unsafe wind shear incidents in recent years.

Time	Flight Number	Location—Phase	Type	Characteristics	Result	Cause
2016	EK 521	OMDB—approach	reactive wind shear	warning alert	unable to recover	Change in configuration after landing; the crew did not push the throttle to the TOGA.
2018	JD 5759	VMMC—landing	reactive wind shear	without warning alert	a severe hard landing	Wind shear leads to higher vertical acceleration during landing and landing bounces occur (POGO landing).
2020	CZ 6960	ZLXY—takeoff roll	predictive wind shear	warning alert	recovered	Anomalous airspeed growth rate was detected, but with no corresponding operations; the crew made an improper decision.
2020	A6 7288	ZLLL—takeoff	predictive wind shear	warning alert	recovered	Wind shear occurred several times during the takeoff, and the flight crew did not reject takeoff.

. Although flight crews are given meteorological information in advance and through onboard forward-looking radars (predictive wind shear systems, or PWSs), automatic terminal information services (ATISs) and flight augmentation computers (FACs) can provide some level of detection and warning [6], and human factors can have a significant impact on the occurrence of unsafe events. Faced with unexpected aviation meteorological threats, such as wind shear events, flight crews need to be able to effectively manage the complex information and make timely decisions that test their knowledge, skills, attitudes, behavioral performances, and threat and error management abilities [7].

Aviation meteorological situations focused on aircraft movement and wind vectors, divide wind shear events into four types: downwind, upwind, side, and vertical [8]. Each of these requires different pilot operations. When downwind shear is encountered, there is a sudden decrease in airspeed, which causes the airplane to lose altitude due to the reduction in lift. When encountered during takeoff, the roll is characterized by a slow increase in speed and an inability to leave the ground. If encountered at a high altitude, the airplane loses altitude after passing through the shear line, but the crew has a chance to recover. However, if the plane is at a low altitude or action is delayed, the plane may crash or be forced to touch down outside the runway. When upwind shear is encountered, the airplane may depart from its normal climb or descent path because of sudden increases in airspeed and lift, especially if it is passing through an upwind wind shear area during the approach phase, at which point the crew must deal with the airplane’s higher altitude or speed. Lateral wind shear can cause the airplane to slip, undergo a slope change, and deviate from its original intended trajectory, which can cause it to land on a slope, land with yaw, or even run off the runway during landing. Depending on the landing distance available (LDA), a strong tailwind can adversely affect the landing; therefore, if this risk is not quickly dealt with, the landing could be dangerous. Finally, the most dangerous of all wind shear types is vertical wind shear. When an airplane is subjected to a sudden and strong downdraft, it can suddenly fall abnormally and deviate from its original trajectory. In general, because unexpected meteorological conditions, such as wind shear, can threaten flight safety, knowing how to handle and manage such threats is vital to ensuring flight safety. Based on many incident investigations and line operation safety assessment (LOSA) data, the ICAO developed a pilot competency framework to ensure that pilots have the skills to effectively respond to aviation threats. The competencies include the application of knowledge, procedural operations, regulatory compliance, automated flight path management, and manual control flight path management, as well as non-technical competencies, such as communication, leadership and teamwork, problem solving and decision making, situation awareness, and workload management [9]. It can be seen that the non-technical abilities are mainly the capacities of Crew Resource Management (CRM). In fact, the ability to handle complex weather threats such as wind shear depends more on their performance of specific behaviors in CRM.

Consequently, the effective assessment of a pilot’s abilities to handle these meteorological threats in flight has attracted widespread attention and is also the core of the data-driven evidence-based training and assessment training proposed by the ICAO, the aims of which are to effectively identify pilot skill deficiencies and reduce the potential risk of unsafe events [9,10]. Current pilot competency assessment methods comprise two categories. The first is a comprehensive evaluation method that assesses pilot competencies using an assessment index system, an analytical hierarchy process, and fuzzy comprehensive evaluations [11,12,13]. However, because the determination of the indicator system weights is somewhat subjective, the assessment consistency is limited. The second type is a data-driven assessment method that exploits flight parameter data or pilot behavior data to construct a prediction model [14,15] and utilizes machine learning algorithms to predict the pilot’s competency scores [16]. Although this method can improve the prediction accuracy when the machine learning models are combined, it has poorer interpretability in practical applications and is unable to provide a specific pilot competency evaluation standard. In this study, we focused on the three-dimensional competency assessment (3DCA) criteria in the competency assessment framework model. To assess pilot competency in a wind shear scenario, we propose a method that optimizes the assessment criteria using existing assessment data, which allows the assessment criteria to be automatically generated. Competency assessments based on effectively constructing data-driven quantitative assessment criteria using competency feature representation have rarely been examined.

To assess a pilot’s competency in managing aviation meteorology threats, we propose an adapted competency model that has observable behaviors (OBs) based on wind shear operations and corresponding competency check items. We developed a data-driven competency pilot assessment optimization model based on three-dimensional competency feature modeling and an optimization algorithm to ensure that the model meets the competency quantification assessment standards. Finally, we conducted experiments using simulated wind shear flight training data to verify the effectiveness of the proposed method.

The remainder of this paper is organized as follows. Section 2 describes the characteristics of different wind shear events and the competency assessment criteria for dealing with these meteorological threats. Section 3 introduces the wind shear operation-based competency model and the competency feature modeling for the competency assessment method. Section 4 details the wind shear simulated flight training evaluation experiments and provides the results of a comprehensive analysis to verify the performance of the proposed data-driven competency assessment method, and Section 5 presents the conclusions.

2. Problem Statements

In actual flight, wind shear is categorized into two types based on the warning methods: predictive wind shear (PWS) and reactive wind shear (RWS) [6,17]. PWS refers to the transmission of coherent impulses by the airborne meteorological radar to a certain wind field range in front of the aircraft and then receiving echoes to detect the wet air wind vectors and their distance; that is, PWS detects an imminent wind shear in the plane’s flight path. RWS refers to a wind shear that exists at the aircraft’s current location, the presence of which is determined by the aircraft’s FAC, which collects system and environmental signals from the flight environment.

Table 2. Distribution of wind shear alarm events in China (from January to June 2020).

Altitude/ft	Takeoff and Climb		Approach Landing		Proportion
	PWS	RWS	PWS	RWS
0~50	9	4	2	19	25.96%
50~200	25	9	8	38	61.07%
200~400	0	1	1	4	4. 58%
400~600	0	0	0	1	0.76%
600~800	0	0	0	0	0.00%
800~1000	0	0	0	1	0.76%
1000~7000	0	0	0	9	6.87%
Proportion	25.95%	10.69%	8.40%	54.96%

shows the 131 wind shear flight alert events that occurred in China from January 2020 to June 2020, the data for which were extracted from a Flight Operational Quality Assurance system [18]. As can be seen, there are significant distribution differences in the altitude and flight phases for the PWS and RWS alert events.

Of the 131 wind shear events shown in Table 2, 86 were RWS events, accounting for 65.65%, and 45 were PWS events, accounting for 34.35%. As there were no correlations between them, more than two-thirds of these actual wind shear events could not be detected in advance by the airborne meteorological radars. The flight phase data for the wind shear events show that there were 48 takeoff and climb phase events, most of which were PWS (34 events), and 83 approach and landing phase events, 72 of which were RWS. The reasons for these differences may be because of the parameters obtained when the wind shear threat factor was being assessed. If the aircraft was already in the wind shear region when the RWS alarm was triggered, the aircraft altitude at the time that the RWS alarm was triggered was the height of the wind shear event. Table 2 also indicates that 80% of the wind shear events occurred below 200 ft, that is, at a low altitude, which, if improperly handled by the pilots, has a stronger possibility of a dangerous crash.

However, regardless of the wind shear threat and the wind shear alarm, pilots must initiate timely and effective operations [19]. Pilots can use different methods to identify RWS wind shear events, such as predicting them in advance; observing the wind direction, speed and other characteristics during the phases when the FAC is unable to provide wind shear detection; and paying attention to changes in the avionic system’s flight parameters when the FAC provides wind shear detection information. When an RWS warning is triggered, the flight crew must immediately initiate procedural operations. Airborne meteorological radars can detect PWS events; however, pilots need to acknowledge PWS warnings before takeoff, during takeoff, and when climbing and landing, and they must execute the correct operations to avoid entering the wind shear area. The choice of wind shear procedure indicates a pilot’s competence in managing these types of threats. Effectively assessing these wind shear pilot competencies, therefore, requires an analysis of the entire threat and error management process and assessments of the pilot’s competence behavioral indicators for the various wind shear handling sub-tasks.

At present, the IATA has a 3DCA criteria guideline for assessing pilot competency. The pilot competency assessment needs to measure three competency feature dimensions: the number of OBs demonstrated by pilots (how many); the frequency of the demonstrated OBs (how often); and the threat and error management outcomes (TEM outcomes) [20]. Thus, as the 3DCA criteria provide three dimensions of competency assessment characteristics, they are sufficient. However, because the grading standard descriptions are uncertain, examiners vary in their subjective understanding of the assessment guidelines, which can lead to assessment misalignments and inconsistencies. Therefore, a competency behavioral feature-based quantitative assessment model based on real flight training data can be used to evaluate pilot competency effectively when there are undesirable weather conditions, such as wind shear.

3. Methodology

3.1. Wind Shear Operation-Based Competency Model

The corresponding dispositions for the two wind shear events are different [21,22]. RWS wind shear identification requires pilots to pay attention to the airborne avionic system alarms; if the RWS warning is triggered, the pilot flying (PF) should immediately activate the RWS procedure operations while coordinating with the pilot monitoring (PM). If the wind shear warning occurs during the takeoff phase before reaching the takeoff decision V1 speed, it is necessary to perform a rejected takeoff procedure. If wind shear is encountered before V1, there may not be sufficient runway remaining to stop if a rejected takeoff is initiated at V1. Therefore, before making this decision, flight crews must assess whether the remaining runway is sufficient to safely perform the rejected takeoff. If the remaining runway is judged to be insufficient, the takeoff must continue to avoid a runway overrun. If the RWS alarm is triggered after reaching the takeoff decision V1 speed, the VR speed must be sufficient to lift the wheels and takeoff. However, in special cases when the remaining runway is insufficient to accelerate to normal takeoff speed or perform an interrupted takeoff, the pilot may need to take the wheels off below the VR speed to maximize aircraft safety. The checking criteria for handling an RWS encountered after takeoff and departure or during the approach and landing phase are as follows: (1) decisively set the TOGA (Take Off/Go Around) thrust to ensure aircraft performance; (2) maintain the current configuration until recovery to minimize the aircraft altitude and speed loss caused by the wind shear; and (3) adjust the pitch attitude as required.

For PWS, it is first necessary to combine the wind shear information displayed on the PFD (primary flight display), the ND (navigation display), and the audio warnings to confirm the PWS warning trigger level and judge the wind shear authenticity and the corresponding risk, after which the PWS operations are performed. The PWS handling procedures are as follows: (1) if a PWS warning is triggered before takeoff, the crew should delay takeoff or choose the most favorable runway for the takeoff; (2) if a PWS warning is triggered during the takeoff roll, the crew should execute a rejected takeoff; (3) if a PWS warning is triggered during the approach or landing phase, the crew should perform a go-around maneuver to avoid the aircraft being unable to recover due to the low altitude; and (4) if a PWS warning is triggered during flight and the aircraft does not enter the wind shear, the crew should adjust the thrust and speed reference system (SRS) settings and improve the aircraft’s performance by changing the wing flaps to counteract the wind shear effect on the aircraft. The wind shear procedural operation check items for the different wind shear modes and flight phases are shown in

Table 3. Wind shear operation check items.

Subjects (Sub)	Check Items (No)
PWS after takeoff lift front wheel	Set thrust
	Monitor variation trend of speed
	Select flight path
	Set the flaps
PWS during takeoff roll	Monitor variation trend of speed
	Implement rejected takeoff decision
	Select a runway for takeoff
	Set thrust if continuing takeoff
RWS after takeoff lift front wheel	Set thrust
	Maintain aircraft configuration
	Follow SRS
	Call and report
RWS during takeoff	Monitor variation trend of speed
	Rejected takeoff decision
	Set thrust
	Follow SRS
PWS during approach	Delay approach
	Select a runway
	Set the flaps for landing
	Manage speed
	Set thrust when go-around
	Manage configuration and follow SRS
RWS on approach	Set thrust
	Implement go-around procedure operations
	Follow SRS
	Remain aircraft configuration
	Call and report

, which does not specify specific aircraft types but gives the required capability check items for different representative wind shear conditions.

Traditional training wind shear scenario flight assessments are conducted by experienced examiners who rate the check items in Table 3 and subjectively rate the pilots; however, because this assessment is subjective, examiners are not required to consider the wind shear response competencies. Analyses of actual flight wind shear events and decompositions of the key behavioral performances for wind shear handling conditions reveal that wind shear handling requires five main competencies: problem solving and decision making, procedural application, flight path management, workload management, and leadership and teamwork. Problem-solving and decision-making skills refer to the ability to accurately recognize and deal with problems using appropriate decision-making processes. In wind shear conditions, the pilot’s problem-solving and decision-making skills need to be centered on the establishment of situational awareness of wind shear timeliness and accurate wind shear handling operations. Procedural application refers to a comprehensive application of knowledge to execute standard operating procedures (SOPs). The flight path management capability refers to the ability to control the aircraft’s path using automated or manual controls, which for wind shear events involves maintaining the aircraft’s altitude. Workload management refers to effective information processing and the correct prioritization and assignment of tasks. When wind shear is encountered, there is an increase in the quantity of information that needs to be processed, which means that it is important that pilots appropriately process the meteorological and flight state information. Leadership and teamwork refer to the ability to be receptive to suggestions and feedback from the team and ensure effective communication by considering the opinions of others and allowing others to participate in developing relevant plans. When confronted with wind shear events, pilots are expected to actively communicate with and provide feedback to the controllers and collaborate with the crew to overcome the wind shear threat. Although the adapted competency model that we provide for wind shear scenarios is based on expert opinions, the formulation methodology for the proposed competency quantitative assessment criteria is based on a data-driven algorithm. The adapted competency model based on these wind shear operations is shown in Figure 1.

3.2. Competency Feature Representation with Wind Shear Operations

The competency model developed in Section 3.1 is then correlated with the check items or observations, which involves correlating the check items corresponding to the capability items in Figure 1 with the required observable behaviors. During wind shear operation training, an examiner uses a checklist to record the student’s completion of the observation items for each subject and then scores the results. Wind shear flight training scoring is usually conducted on a 5-point scale, with higher scores representing higher performances. The obtained wind shear operation-based subject check worksheet data item scores can be denoted as $\mathbf{A}={\left(a_i\right)}_{m\times 1}={\left(a_1,a_2,\cdots ,a_m\right)}^T$, $i=1,2,\dots ,m$, where $m$ is the total number of check items, $a_i$ is the score for the $i$th check item evaluated, and the corresponding $a_i^{max}$ is the full value of the actual score; therefore, the upper bound of the scores for all observations can be denoted as ${\mathbf{A}}^{max}={\left(a_1^{max},a_2^{max},\cdots ,a_m^{max}\right)}^T$. The TEM outcome wind shear operation performance on the check worksheet can be denoted as $\mathbf{C}={\left(c_i\right)}_{k\times 1}={\left(c_1,c_2,\cdots ,c_k\right)}^T,i=1,2,\dots ,k$, where $k$ is the total number of observations on the checklist, $c_i$ is the score for the $i$th observation, the maximum value $c_i^{max}$ is the upper bound of the score for that item, and the observation vector when all check items are taken as full scores is ${\mathbf{C}}^{max}={\left(c_1^{max},c_2^{max},\cdots ,c_k^{max}\right)}^T$.

After the competency behavioral indicator observation, each check item corresponds to an OB in the wind shear competency model; that is, each check item reflects the performance of a corresponding OB. The association between any check item $i$ and competency $O{B}_j$ is constructed using the Delphi survey method to solicit flight experts’ opinions. In the development process, we obtained suggestions from twenty pilot competency assessment experts about the correlations between the competency check items and the observable behaviors, with the corresponding elements in the correlation matrix being 0 or 1. If more than half of the experts believed there was a correlation, the $b_{ij}$ was 1, that is, there are correlations between the competence check items and the observable behaviors. Therefore, to assess the wind shear operational performance, a check item and OB association matrix was constructed, notated as $\mathbf{B}$, which is determined using Equation (1):

$$\mathbf{B}=\left[{\mathbf{B}}_1,{\mathbf{B}}_2,\cdots {\mathbf{B}}_n\right]=\left[\begin{array}{cccc}{b}_{11}& b_{12}& \dots & b_{1n}\\ b_{21}& b_{22}& \cdots & b_{2n}\\ ⋮& ⋮& ⋮& ⋮\\ b_{m1}& b_{m2}& \cdots & b_{mn}\end{array}\right]$$

(1)

Similarly, the observation term and OB association matrix D for the TEM result worksheet was as shown in Equation (2):

$$\mathbf{D}=\left[{\mathbf{D}}_1,{\mathbf{D}}_2,\cdots {\mathbf{D}}_p\right]=\left[\begin{array}{cccc}{d}_{11}& d_{12}& \dots & d_{1p}\\ d_{21}& d_{22}& \cdots & d_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ d_{k1}& d_{k2}& \cdots & d_{kp}\end{array}\right]$$

(2)

where n denotes the number of observable behaviors in the wind shear operation competency model, p denotes the number of observable behaviors for threat and error management on the TEM checklist, and $b_{ij}$ and $d_{ij}$ denote the association attributes of the $i$th observation with the $j$th OB, with $b_{ij}$ or $d_{ij}=1$ indicating that the $i$th observation contains the $j$th OB performance; otherwise 0 is taken.

$$\mathbf{X}=\left[{\mathbf{X}}_1,{\mathbf{X}}_2,\cdots ,{\mathbf{X}}_n\right]=\left[\begin{array}{cccc}{a}_1{b}_{11}& a_1{b}_{12}& \cdots & a_1{b}_{1n}\\ a_2{b}_{21}& a_2{b}_{22}& \cdots & a_2{b}_{2n}\\ ⋮& ⋮& ⋮& ⋮\\ a_m{b}_{m1}& a_m{b}_{m2}& \dots & a_m{b}_{mn}\end{array}\right]$$

(3)

Similarly, to obtain the competency assessment matrix$\mathbf{Y}$ based on the TEM outcome for the relevant OB competency composition, the TEM score vector$\mathbf{C}$ outcome is multiplied with each correlation matrix $\mathbf{D}$ column, as shown in Equation (4):

$$\mathbf{Y}=\left[{\mathbf{Y}}_1,{\mathbf{Y}}_2,\cdots ,{\mathbf{Y}}_p\right]=\left[\begin{array}{cccc}{c}_1{d}_{11}& c_1{d}_{12}& \cdots & c_1{d}_{1p}\\ c_2{d}_{21}& c_2{d}_{22}& \cdots & c_2{d}_{2p}\\ ⋮& ⋮& ⋮& ⋮\\ c_k{d}_{k1}& c_k{d}_{k2}& \dots & c_k{d}_{kp}\end{array}\right]$$

(4)

In (3) and (4), $a_i{b}_{ij}$ is the contribution value of the $j$th OB to the $i$th required wind shear check item, and $c_i{d}_{ij}$ is the contribution value of the $j$th OB to the $i$th TEM outcome. Because each wind shear scenario or sub-task involves multiple competency behavioral indicators, extracting the competency-based quantitative assessment criteria from historical check data requires the modeling of multiple competency feature dimensions [23]. Based on the 3DCA criteria, $f_{many}$, $f_{often}$, and $f_{tem}$ are the competency behavioral performance feature dimensions for the wind shear events, which are, respectively, how many, how often, and the TEM outcome. As the contribution value for the observed OB demonstration in the competency assessment matrix was higher than 50% of the maximum, it has not yet been assessed. Therefore, the OB observed quantity was modeled using the vector norms in Equation (5):

$$f_{many}=count\left\{\begin{array}{l}{\mathbf{X}}_j,\left|\right|{\mathbf{X}}_j|{|}_1\ge (50\%){‖{\mathbf{A}}^{max}\odot {\mathbf{B}}_j‖}_1\\ \forall j=1,2,\dots ,n\end{array}\right\}$$

(5)

where $\odot$ is the Hadamard product.

To determine the ‘how often’ information in the OBs, the actual number of occurrences is expressed as the sum of the non-zero elements in the competency assessment matrix using Equation (6):

$$f_{\mathit{often}}=\left|\right|\mathbf{X}|{|}_1={\sum }_{i=1}^m{\sum }_{j=1}^n\left|a_i{b}_{ij}\right|$$

(6)

To determine the TEM outcome, which represents the security management margin for the threat disposal operations, the lowest score of all competency OBs for each check item is taken, with the OB-based threat and error outcomes determined using Equation (7):

$$f_{tem}=\min (c_i{d}_{\mathrm{i}1}+{\mathrm{c}}_i{d}_{\mathrm{i}2}+\cdots +c_i{d}_{\mathrm{ij}}|i=1,\dots ,r)$$

(7)

3.3. Wind Shear Competency Assessment Optimization Model

With $P$ denoting the competency level, a five-level grade is usually used for competency assessments. Therefore, the ratings for each check item level rating in $P_{exa}$ are as follows:

$$P_{exa}=\left\{\left.\begin{array}{l}\mathrm{e}\mathrm{x}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{y}, \mathrm{e}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{y}, \mathrm{a}\mathrm{d}\mathrm{e}\mathrm{q}\mathrm{u}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{l}\mathrm{y},\\ \mathrm{m}\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{l} \mathrm{a}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{a}\mathrm{b}\mathrm{l}\mathrm{e}, \mathrm{i}\mathrm{n}\mathrm{e}\mathrm{f}\mathrm{f}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{y}\end{array}\right\}\right.=\left\{\left.5, 4, 3, 2, 1\right\}\right..$$

Because the performance expectations and completion criteria settings for the operational check items are different in different training scenarios, the three-dimensional competency features information is standardized. The maximum values for $f_{many}$, $f_{often}$ and $f_{tem}$ are then obtained based on the competency assessment matrices shown in Equations (8)–(10):

$$f_{many}^{max}=cou\mathrm{n}\mathrm{t}\left\{\begin{array}{l}{\mathbf{X}}_j,{‖{\mathbf{X}}_j^{max}‖}_0>0\\ \forall j=1,2,\dots ,n\end{array}\right\}$$

(8)

$$f_{often}^{max}={\sum }_{i=1}^m{\sum }_{j=1}^n\left|a_{ij}^{max}{b}_{ij}\right|$$

(9)

$$f_{tem}^{max}=\min \left\{{\sum }_{j=1}^k{c}_i^{max}{d}_{ij}\left|i=1\dots k\right.\right\}$$

(10)

The normalizations of the three-dimensional features from the 3DCA criteria are expressed as follows: $\overline{f_{many}}=\frac{f_{many}}{f_{many}^{max}}$, $\overline{f_{often}}=\frac{f_{often}}{f_{often}^{max}}$, and $\overline{f_{tem}}=\frac{f_{tem}}{f_{tem}^{max}}$, where $\overline{f_{often}},\overline{f_{many}},\overline{f_{tem}}\in \left[\mathrm{0,1}\right]$.

The aim is to minimize any expected deviations between the competency assessment score and the competency rating given by the experienced flight examiners. $P_{often}$, $P_{many}$, and $P_{tem}$ are the competency scores under different dimensions, and $P_{OB}$ is the competency rating based on the 3DCA criteria competency features modeling; therefore, the competency assessment optimization model can be expressed as shown in (11) and (12):

$$\mathit{min\; E}\left|P_{\mathrm{OB}}-{\mathrm{P}}_{\mathrm{e}\mathrm{xa}}\right|$$

(11)

$$P_{{\mathrm{d}}_i}=\left\{\begin{array}{l}\mathrm{1,0}\le \overline{f_{d_i}}<{\alpha }_1^{\left(i\right)}\\ 2,{\alpha }_1^{\left(i\right)}\le \overline{f_{{\mathrm{d}}_i}}<{\alpha }_2^{\left(i\right)}\\ 3,{\alpha }_2^{\left(i\right)}\le \overline{f_{{\mathrm{d}}_i}}<{\alpha }_3^{\left(i\right)}\\ 4,{\alpha }_3^{\left(i\right)}\le \overline{f_{d_i}}\le {\alpha }_4^{\left(i\right)}\\ 5,{\alpha }_4^{\left(i\right)}\le \overline{f_{{\mathrm{d}}_i}}\le 1\end{array}\right.$$

(12)

$$P_{OB}=\mathit{min}({P_d}_i)$$

(13)

The original optimization model in (11) to (13) is a nonlinear stochastic optimization problem, which is difficult to solve directly. Therefore, the original stochastic optimization objective is replaced by the sample mean deviation using empirical error minimization; that is, the optimization objective is replaced with $\mathit{min}{\sum }_{l=1}^L|{P_{OB}}^{\left(l\right)}-{P_{exa}}^{\left(l\right)}|$.

The acquired data samples were combined, and a distribution approximation method was developed to solve the approximate optimization problem.

Let L be the number of check item samples, 85% of which are randomly selected for the training set. The wind shear competency assessment optimization solution algorithm is then implemented as follows.

Step 1: For each of the three competency dimensions, the score frequencies in the training sample are counted $q^{\left(i\right)}$, $i=1,2,3$, as shown in (14):

$$q^{\left(i\right)}=\frac{count(P_{di}=q)}{M},q=1,2,\dots ,5$$

(14)

Step 2: For each of the three competency dimensions $\left\{{\mathrm{d}}_i,i=1,2,3\right\}$ in the 3DCA model, the initial value of the decision variable $\overline{{\alpha }_q^{\left(i\right)}}$ is recursively calculated using (15):

$$\left\{\begin{array}{l}{q}^{(i)}=\frac{1}{2}(\overline{{\alpha }_q^{(i)}}-\overline{{\alpha }_{q-1}^{(i)}})(\overline{{\alpha }_q^{(i)}}+\overline{{\alpha }_{q-1}^{(i)}}),\mathit{IF}q<0.5\\ \overline{{\alpha }_5^{\left(i\right)}}=1,\overline{{\alpha }_0^{\left(i\right)}}=0,\overline{{\alpha }_q^{\left(i\right)}}\ge 0,\\ q=1,2,3,4,5\end{array}\right.$$

(15)

Step 3: A step size for h is set at 0.01, and the search is activated based on the variable update in (16):

$${\alpha }_q^{(i)}(t)=\overline{{\alpha }_q^{\left(i\right)}}\pm kh$$

(16)

Step 4: When the approximate optimization objective reaches a minimum, the search stops, and the approximate optimal threshold is obtained.

4. Experiment

To assess the pilots’ wind shear operation competencies, an experiment was conducted in an upset prevention and recovery training course, which is a transitional training course pilots need to complete before being able to fly an aircraft. The pilots selected for this experiment were licensed and all had 250 h of logged flight time. The selected pilots, all young males, were currently undergoing the transitional phase course to become co-pilots. The aircraft they performed well in was an SR-20 or a C172, and the simulator was the BOEING 737NG. Similar competency assessments using simulators are part of the main ICAO- and IATA-recommended EBT retraining programs. The experiment was conducted with the consent of the pilots. Six typical wind shear scenarios or sub-tasks at different flight phases were used to evaluate the pilots’ wind shear procedural operational performances. To verify the performance of the proposed competency assessment model, evaluation data from 200 pilots were obtained from a flight training institution for simulated flight training for wind shear disposition operational assessment 5.1.

4.1. Competency Assessment Standard Setting

Different from traditional methods where instructors directly score pilot competencies based on the OB descriptions, the following steps were followed to construct the competency assessment optimization model and determine the quantitative evaluation criteria for the competency assessment.

(1)

Construct observation vectors

First, based on typical wind shear conditions, a flight training evaluation comprising six subjects and twenty-seven check items was developed, which included the TEM checklist data, and the 3DCA model of the ‘how many’ and ‘how often’ information. The TEM inspection data corresponded to the ‘TEM outcome’ for the different subjects. The different check items in the inspection worksheet corresponded to the evaluation standards; for the wind shear checklist, there were twenty-seven check items or observations, an example of which is shown in

Table 4. Wind shear operation evaluation checklist.

Subject (Sub)	Observation (No)	Evaluation Criteria	Levels
Sub 1: PWS after takeoff lift front wheel	No. 1: Set thrust	5: Set and verified TOGA in a timely manner 4: Set TOGA in a timely manner 3: Reached TOGA 2: Did not set TOGA in place 1: Forgot to set TOGA	4
	No. 2: Monitor speed	5: Discovered multilevel change in a timely manner 4: Discovered change in a timely manner 3: Discovered change 2: Did not discover change in time 1: Forgot to monitor	3
…	…	…	…
Sub 6: RWS on approach	No. 26: Maintain aircraft configuration	5: Accurately and conditionally maintained configuration 4: Maintained configuration in a timely manner 3: Maintained configuration 2: Failed to maintained configuration 1: Put down landing gear	3
	No. 27: Call and report	5: Efficiently and accurately conveyed information 4: Conveyed information in a timely manner 3: Conveyed information 2: Failed to convey information 1: Forgot to convey information	3

.

Based on the wind shear operation evaluation checklist data, a vector of scores for the check items is obtained, such as $\mathbf{A}={\left(a_i\right)}_{27\times 1}={(4,3,\cdots 3,2,4)}^T$. Similarly, based on the TEM outcomes, a score vector $\mathbf{C}$ is obtained for the corresponding TEM results.

(2)

Construct the correlation matrix for check items and OBs

We used a Delphi survey to solicit the opinions of flight experts and instructors and construct the correlation matrix for the twenty-seven check item observations, the six TEM outcome observations, and the eleven wind shear operation competency OBs, with the value of the corresponding elements being 1 if there was an influential relationship, and 0 if there was not. The final obtained observations OB competency correlation matrix constructed from the flight expert information is shown below, where the wind shear operation-based competency OBs demonstrated are concentrated from OB1.1 to OB5.2.

$$\mathbf{B}=\left[{\mathbf{B}}_1,{\mathbf{B}}_2,\cdots {\mathbf{B}}_{11}\right]=\left[\begin{array}{ccccccc}0& 1& 0& 0& \dots & 1& 0\\ 1& 0& 0& 1& \dots & 0& 1\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 1& 1& 0& 0& \dots & 0& 0\\ 0& 0& 0& 1& \dots & 1& 0\\ 1& 0& 1& 0& \dots & 0& 0\end{array}\right]$$

The observation term–OB correlation matrix for the corresponding TEM outcomes was similarly derived.

(3)

Build model of competency assessment feature representation and optimization

The observation vector $\mathbf{A}$ and the correlation matrix $\mathbf{B}$ were combined to construct the OB-based competency assessment matrix $\mathbf{X}$ as follows:

$$\mathbf{X}=\left[{\mathbf{X}}_1,{\mathbf{X}}_2,\cdots {\mathbf{X}}_{11}\right]=\left[\begin{array}{ccccccc}0& 4& 0& 0& \dots & 4& 0\\ 3& 0& 0& 3& \dots & 0& 3\\ ⋮& ⋮& ⋮& ⋮& ⋮& ⋮& ⋮\\ 3& 3& 0& 0& \dots & 0& 0\\ 0& 0& 0& 2& \dots & 2& 0\\ 4& 0& 4& 0& \dots & 0& 0\end{array}\right]$$

Based on the competency feature representations in the 3DCA criteria-based competency assessment model framework, for each pilot evaluation sample, the how many, how often, and TEM outcome information for the OB performances was determined using Equations (5)–(10). For instance, when $f_{many}=5$, $f_{\mathit{often}}=363$, and $f_{tem}=4$, the three-dimensional features after normalization were as follows:

$$\overline{f_{many}}=\frac{5}{5}=1,\overline{f_{often}}=\frac{363}{535}=0.68,\overline{f_{\mathit{tem}}}=\frac{4}{5}=0.8$$

4.2. Competency Assessment Model Solution

The historical evaluation checklist data for 200 pilots were divided into a training set (85%) and a test set (15%), after which it was easy to determine the wind shear operation-based competency assessment matrices for all pilots. Based on the assessment method flow, the optimization problem in the previous steps needed to be solved. As the optimization approximation objective for the training set was small (in this case, the training set sample K = N * 85% = 170), the problem was then transformed into twelve non-negative real numbers (${{\alpha }_1}^{(i)},{{\alpha }_2}^{(i)},{{\alpha }_3}^{(i)},{{\alpha }_4}^{(i)}$) for the solution, which made the approximate optimization objective extremely small.

Based on the actual training set score data in the training set, the score distribution from 1 to 5 for the ‘how often’ characteristic dimension is shown in Figure 2.

In Figure 2, the 3DCA criteria-based competence behavior indicators show a single-peak distribution for the ‘how often’ characteristic dimension grades, of which there were 1 with 1 point, 2 with 2 points, 56 with 3 points, 103 with 4 points, and 8 with 5 points.

In Figure 3, the 3DCA model of competence assessment indicators still shows a single-peak distribution for the ‘how many’ characteristic dimension grades, of which there were 0 with 1 point, 1 with 2 points, 2 with 3 points, 157 with 4 points, and 10 with 5 points.

In Figure 4, the 3DCA model competency behavior indicators for the ‘TEM outcome’ dimensional grades were more concentrated, for which there were a 1-point count of 0, a 2-point count of 2, a 3-point count of 3, a 4-point count of 156, and a 5-point count of 9.

Figure 2, Figure 3 and Figure 4 show that the competency feature distributions from the actual flight training sample across all three dimensions are unimodal. The distribution characteristics are representative, which indicates that most pilot competency levels are concentrated at a specific level.

The grid search method was first used to obtain the initial values for the quantitative assessment criteria threshold decision variables for each dimension: $\overline{{\alpha }_1^{\left(1\right)}}=0.1$, $\overline{{\alpha }_2^{\left(1\right)}}=0.17$, $\overline{{\alpha }_3^{\left(1\right)}}=0.82$, and $\overline{{\alpha }_4^{\left(1\right)}}=0.93$; $\overline{{\alpha }_1^{\left(2\right)}}=0$, $\overline{{\alpha }_2^{\left(2\right)}}=0.1$, $\overline{{\alpha }_3^{\left(2\right)}}=0.17$, and $\overline{{\alpha }_4^{\left(2\right)}}=0.92$; and $\overline{{\alpha }_1^{\left(3\right)}}=0$, $\overline{{\alpha }_2^{\left(3\right)}}=0.14$, $\overline{{\alpha }_3^{\left(3\right)}}=0.22$, and $\overline{{\alpha }_4^{\left(3\right)}}=0.93$.

The search was then carried out in steps of h = 0.01 until the approximate optimization objective was minimized, after which the search was stopped. The obtained approximate optimal thresholds were: $\widehat{{\alpha }_1^{\left(1\right)}}=0.48$, $\widehat{{\alpha }_2^{\left(1\right)}}=0.53$, $\widehat{{\alpha }_3^{\left(1\right)}}=0.71$, and $\widehat{{\alpha }_4^{\left(1\right)}}=0.84$;$\widehat{{\alpha }_1^{\left(2\right)}}=0$, $\widehat{{\alpha }_2^{\left(2\right)}}=0.6$, $\widehat{{\alpha }_3^{\left(2\right)}}=0.83$, and $\widehat{{\alpha }_4^{\left(2\right)}}=0.92$; and $\widehat{{\alpha }_1^{\left(3\right)}}=0$, $\widehat{{\alpha }_2^{\left(3\right)}}=0.76$, $\widehat{{\alpha }_3^{\left(3\right)}}=0.8$, and $\widehat{{\alpha }_4^{\left(3\right)}}=0.93$. The final competency assessment criteria based on the training sample are shown in

Table 5. Wind shear operation competency assessment criteria based on three dimension features.

Grading	1	2	3	4	5
$\mathrm{OB}\mathrm{frequency}\mathrm{classification}\mathrm{interval}\left(\overline{f_{often}}\right)$	[0, 0.48)	[0.48, 0.53)	[0.53, 0.71)	[0.71, 0.84)	[0.84, 1]
$\mathrm{OB}\mathrm{number}\mathrm{of}\mathrm{classification}\mathrm{intervals}\left(\overline{f_{many}}\right)$	/	[0, 0.6)	[0.6, 0.83)	[0.83, 0.92)	[0.92, 1]
$\mathrm{TEM}\mathrm{outcome}\mathrm{of}\mathrm{classification}\mathrm{interval}\left(\overline{f_{\mathrm{tem}}}\right)$	/	[0, 0.76)	[0.76, 0.8)	[0.8, 0.93)	[0.93, 1]
Number of levels evaluated	1	2	56	103	8
Number of real levels	1	2	61	96	10

.

4.3. Assessment Result Analysis

4.3.1. Consistency Analysis

The wind shear operation-based competency assessment levels for the test sample data were validated against the real levels given by the flight instructors. Using the obtained competency assessment criteria in Table 5, the assessment results are given in Appendix A,

Table A1. The scores on three dimensions with the test samples.

Sample Serial No.	$\overline{{\mathit{f}}_{\mathit{o}\mathit{f}\mathit{t}\mathit{e}\mathit{n}}}$	${\mathit{P}}_{\mathit{o}\mathit{f}\mathit{t}\mathit{e}\mathit{n}}$	$\overline{{\mathit{f}}_{\mathit{m}\mathit{a}\mathit{n}\mathit{y}}}$	${\mathit{P}}_{\mathit{m}\mathit{a}\mathit{n}\mathit{y}}$	$\overline{{\mathit{f}}_{\mathit{t}\mathit{e}\mathit{m}}}$	${\mathit{P}}_{\mathit{t}\mathit{e}\mathit{m}}$	${\mathit{P}}_{\mathit{O}\mathit{B}}$	${\mathit{P}}_{\mathit{e}\mathit{x}\mathit{a}}$
1	0.5190	2	0.8	3	0.77	3	2	2
2	0.5705	3	0.82	3	0.8	4	3	3
3	0.6040	3	0.86	4	0.82	4	3	3
4	0.6197	3	0.87	4	0.85	4	3	3
5	0.6331	3	0.89	4	0.86	4	3	3
6	0.6376	3	0.91	4	0.89	4	3	3
7	0.6421	3	1	5	0.92	4	3	3
8	0.6555	3	1	5	0.96	5	3	3
9	0.6600	3	1	5	0.97	5	3	3
10	0.6734	3	1	5	0.99	5	3	3
11	0.6756	3	1	5	1	5	3	3
12	0.6890	3	1	5	1	5	3	4
13	0.6980	3	1	5	1	5	3	3
14	0.7025	3	1	5	1	5	3	3
15	0.7047	3	1	5	1	5	3	3
16	0.7069	3	1	5	1	5	3	3
17	0.7092	3	1	5	1	5	3	3
18	0.7136	4	1	5	1	5	4	4
19	0.7226	4	1	5	1	5	4	4
20	0.7383	4	1	5	1	5	4	4
21	0.7393	4	1	5	1	5	4	4
22	0.7450	4	1	5	1	5	4	4
23	0.7494	4	1	5	1	5	4	4
24	0.7517	4	1	5	1	5	4	4
25	0.7673	4	1	5	1	5	4	4
26	0.7740	4	1	5	1	5	4	4
27	0.7760	4	1	5	1	5	4	4
28	0.8054	4	1	5	1	5	4	4
29	0.8479	5	1	5	1	5	5	4
30	0.8814	5	1	5	1	5	5	5

. The smallest of the three dimensions for each sample is the model assessment level. Consistency analyses often use nonparametric hypothesis testing to calculate the Spearman rank correlation coefficient [24], with a Spearman correlation coefficient range of [−1, +1]. A Spearman correlation coefficient less than 0 represents a negative correlation, a Spearman correlation coefficient greater than 0 represents a positive correlation, and a Spearman correlation coefficient equal to 0 represents no correlation; the closer the correlation coefficient is to 0, the weaker the correlation, and the closer the correlation coefficient is to −1 or +1, the stronger the correlation. The correlations between the proposed model-based evaluation levels and the real levels were analyzed (

Table 6. Spearman correlation analysis results. (** indicates the p-value is less than 0.01).

Correlation Analysis	Levels Evaluated from Model	Real Levels
Correlation coefficient	1.000	0.854 **
Significance	-	0.000
Number	30	30

). Spearman’s correlation coefficient between the levels evaluated from the model and the real levels was r = 0.854, p < 0.01 (**), which indicates that the proposed model and real levels were significantly correlated.

4.3.2. Accuracy Analysis

To verify the accuracy of the results for the competency assessment model, the test set evaluation data were used for validation by calculating the $\overline{f_{many}},\overline{f_{often}},\mathrm{a}\mathrm{n}\mathrm{d}\overline{f_{tem}}$ for each pilot, from which the $P_{often},P_{many},\mathrm{a}\mathrm{n}\mathrm{d}{P}_{tem}$, based on the competency assessment model criterion thresholds, were obtained by taking the minimum of the three as the pilot’s levels. Model assessment level comparisons for the samples and the deviations from the actual levels are shown in Figure 5.

The analysis based on the proposed model found that 93.33% of the sample agreed with the actual levels, and the other level deviations were all within 1 level of 6.67%, which was an acceptable deviation range because of the possible subjectivity in the instructor’s original checklist observations. Therefore, the proposed competency assessment optimization model based on historical data information provides good results within acceptable rating deviations.

5. Conclusions

Complex and unexpected aviation meteorological conditions are typical flight safety threats. Therefore, an effective pilot competency evaluation method is needed to enhance pilots’ comprehensive skills in dealing with meteorological aviation threats. The 3DCA criteria are a general criterion framework for assessing the competency of the pilots. Because of the uncertain grading descriptions and subjective evaluator judgment, the consistency of the assessment results is not ensured when using the 3DCA criteria directly. In this study, we developed an evaluation method that automatically generates quantitative assessment criteria based on the existing assessment data. With this focus, we constructed an adapted competency model and observable behavior indicators, conducted competency check item and sub-task decomposition, and then, with simulated flight training evaluation data under wind shear conditions, developed a competency assessment criteria optimization model based on three-dimensional competency feature modeling and an algorithm to determine the competency evaluation criteria solution. The consistency and accuracy of the proposed competency evaluation method were validated on actual wind shear simulation flight training sample data.

While pilots with different preferred aircraft types behave differently in wind shear restrictions, they have behavioral performance commonalities. Our approach is based on available evaluation data, and the obtained competency quantitative assessment criteria are not affected by the operational wind shear differences for various aircraft types. Meanwhile, because different airlines have their own checklist data for these different training subjects, the quantitative competency assessment criteria need to be generated based on appropriate evaluation data. Therefore, the proposed data-driven competency assessment method can be applied to competency assessment models for other conditions; that is, it can be used for pilot competency assessments under various training stages and to improve pilot abilities to deal with sudden aviation threats. Further, due to the large number of complex conditions that must be simulated for competency assessments, except for the skills that need to be verified in real airplanes, training is generally conducted in simulators. Although the conditions in a real aircraft and a simulator are not the same, the observable behavior data from the simulator training scenarios provide good information about whether a pilot has the competencies to deal with the situation in a real aircraft. However, further verification could be the direction of future research.