Performance, Emotion, Presence: Investigation of an Augmented Reality-Supported Concept for Flight Training

Abstract

Augmented reality (AR) could be a means for a more sustainable education of the next generation of pilots. This study aims to assess an AR-supported training concept for approach to landing, which is the riskiest phase of flying an aircraft and the most difficult to learn. The evaluation was conducted with 59 participants (28 women and 31 men) in a pretest–post-test control group design. No significant effect of the AR-supported training was observed when comparing the experimental and the control groups. However, the results show that for the experimental group that trained with AR, higher performance in post-test was associated with higher AR presence and comfort with AR during training. Although both gender groups improved their approach quality after training, the improvement was larger in women as compared to men. Trainees’ workload, fear of failure, and negative emotions decreased in post-test as compared to pre-test, but the decrease was significantly larger in women than in men. The experimental group who used AR support during training showed improved performance despite the absence of AR support in post-test. However, the AR-based training concept had a similar effect to conventional simulator training. Although more research is necessary to explore the training opportunities in AR and mixed reality, the results of this study indicate that such an application would be beneficial to bridge the gap between theoretical and practical instruction.

1. Introduction

1.1. Augmented Reality (AR) Support for Pilot Education

Currently, the aviation industry is facing a shortage of pilots and is exploring new ways to attract and train talented candidates, both women and men [1]. Women account for less than 10 percent of the pilot population worldwide [2], and there are only a few studies that have addressed gender differences in flight training [3,4,5,6,7].

Augmented reality (AR) has the potential to improve flight instruction and to bridge the gap between theoretical and practical training [8]. Surveys with pilots and flight instructors have identified gender-specific preferences regarding training contents and potential augmented reality (AR) benefits both in ab initio pilot training [9] and in advanced pilot courses [10]. However, there were also many similarities between genders [10]. A survey on gaming concepts in relation to flight training showed that most of the pilots, both women and men, liked achieving a target to finish a task, receiving feedback for correct actions, and receiving points when completing a task [11]. Interestingly, more women than men considered it satisfying to answer questions during the game and to collect assets or information during the game [11].

The research on AR related to flight training is based mainly on use case assessments with small numbers of participants, surveys, and expert workshops [8,9,10,12]. The qualitative assessment conducted in case studies allows researchers/developers to detect and address issues early on in development [12]. Although assessments with controlled experiments are rare, they are necessary for a comprehensive understanding of the quantitative AR effects on trainees’ performance and experience. Recently, an AR application for training traffic procedures in accordance with visual flight rules (VFR) was assessed with an experimental and a control group [13]. The results showed a performance benefit for trainees attending an AR-guided visual scanning training as compared to trainees that used conventional classroom means. However, not every AR application for training had a significant effect on performance. During simulated flight, when the trainees were not supported by AR, experimental and control groups showed similar scores in the accuracy of collision estimations and the application of right-of-way rules [13]. No negative effect of AR-supported training was found when assessing a large number of variables such as performance, workload, situation awareness, emotion, and motivation. Positive effects were found in both groups irrespective of having trained with AR or conventional means. The accuracy of collision detection and trainees’ emotions and motivational factors were positively affected [13]. Interestingly, there were some gender effects on trainees’ subjective experience related to training (e.g., emotion, motivation, engagement with AR, and preferences for AR features), but not on performance. Specifically for the AR applications used in Moesl et al. [13], women preferred voice interaction with AR and the orientation cue (i.e., compass hologram) while men liked the traffic holograms and the AR projection field more than women. In summary, the literature on educational effects of AR is dominated by qualitative research, and research on the statistical effects of AR-based training is rare. Nevertheless, AR is a modern technology that has the potential to make flight training more attractive to the new generation, and to address the needs and preferences of both women and men. The present study implements an experimental quantitative approach to explore the effects of AR-based training for approach to landing of an aircraft. Trainees of both genders were each assigned to an experimental or a control group.

1.2. Issues with the Approach and Landing of an Aircraft

Approach and landing are considered the most difficult and risky phases of a flight. A report of the Aircraft Owner and Pilot Association (AOPA) which analyzed aviation accidents occurring during a period of ten years found that the approach phase had the highest lethality index (33.6%), followed by go-around (18.6%), takeoff/initial climb (17.2%), and landing (3.5%) [14]. Most frequent causes of the approach, landing, and go-around accidents were control issues with the altitude and airspeed (27.2%), and loss of control (not due to wind) (17.1%) [14].

A survey with flight instructors and female and male pilots showed that the approach and landing of an aircraft were the most difficult parts of the ab initio pilot training. Interestingly, almost 45% of female pilots and 21% of male pilots considered that not only the nature of the content but also the learning conditions contributed to the difficulty of flight training. When asked to make suggestions for improvement, the female pilots addressed elements such as clear communication of learning objectives, use of multiple ways to explain, and more precise feedback. Male pilots suggested that the availability of more instructors and of simulator training would be useful. The flight instructors also pointed out that more time to learn would be beneficial for trainees’ performance [9].

The availability of instructors and more time to practice would also involve higher costs in flight training. However, the quality of the training and the training conditions could be improved by using innovative technology such as AR. After a video familiarization with different AR use cases, both instructors and pilots considered that AR had the potential to improve the ab initio training for a number of syllabus components including approach and landing [9]. These opinions inspired the development of the concept for AR-supported training evaluated in this study.

1.3. Applications of AR in Education

Research shows that augmented reality (AR) could be a vehicle to address social sustainability aspects of pilot education such as gender diversity and inclusivity [10]. Augmented reality (AR)-supported education benefits from the augmentation of the real world with computer-generated information and an interactive experience with the learning material [15,16]. AR-supported science games have been shown to increase students’ interest and their engagement with the content, as well as their feeling of discovery and desire for better performance [17]. Immersion, the “subjective impression that one is participating in a comprehensive, realistic experience” ([18], p. 66) is considered to be a key player in the interaction with AR contents. Research shows that immersion is facilitated by the experience of flow [19]. Flow was defined as “a state of optimal experience arising from intense involvement in an activity that is enjoyable, such as playing a sport, performing a musical passage, or writing a creative piece. Flow arises when one’s skills are fully utilized yet equal to the demands of the task, intrinsic motivation is at a peak, one loses self-consciousness and temporal awareness, and one has a sense of total control, effortlessness, and complete concentration on the immediate situation (the here and now)” [20] proposed by [21]. Flow is more about a complete focus on an activity whereas immersion is more about being fully absorbed in an environment or content. In addition, in the context of AR location-aware learning environments, immersion was also associated with presence [19]. The concept of presence was originally used in the context of virtual reality [22,23], where it was defined as a strong “experience of being present in a virtual environment” as opposed to the real environment [23]. The concept of presence has been adopted and redefined for the context of AR, where virtual cues are embedded in a real environment [19]. In this study, immersion is seen in line with [19] who proposed an “operationalization of immersion as a continuum towards flow and presence” [p. 26].

The influence of emotions has been investigated in relation to motor behavior and sport performance [24]. High demands related to the conditions and performance requirements related to flight training can be experienced as stressful by a number of students [10]. Research shows that trainees who experienced more intensive negative emotions performed worse in flight training experiments [25]. However, flight training can influence both the negative and positive emotions of participants. Simulator training [26] improved not only the performance and positive emotions, but also reduced the intensity of negative emotions in student pilots. Therefore, trainees’ subjective experience is an important factor to be considered in flight training. Coaching sessions, held as group discussions with professional pilots, addressing, among other aspects, challenges and difficulties encountered and managed during the flight training also had a positive effect on reducing trainees’ fear of failure [27]. In this study, the use of interactive AR-based training means and the application of gaming features is also expected to improve trainees’ subjective experience of the training.

1.4. Aspects of Gender Diversity

Initial interpersonal differences are considered responsible for different learning outcomes of women and men attending flight training courses [3]. Studies of differential psychology showed that, generally, women were better in verbal tasks, but men were better in visuospatial tasks [28]. However, research shows that initial gender differences in performance can be addressed by training that is designed in a gender-sensitive and gender-neutral manner without modifying the number of repetitions or the duration of the training. For example, the evaluation of an upset recovery training [26] showed that in the post-test the performance of female and male trainees was similar despite significant gender differences in the pre-test prior to training.

The ability to control an aircraft‘s approach to the runway for landing involves perception of whether a contact or collision would occur and perception of the time-to-contact [29] or time-to-collision [30]. Research showed significant gender differences in perception of the time-to-collision in pilots [30] and in the estimation of relative distance to other vehicles in drivers [30]. Since these estimations are critical for performance and safety, various measures of support have been studied. Experiments on a feedback-based simulator training for student pilots have shown positive effects on improving the estimation accuracy of the time-to-collision and relative distance [31]. Feedback-based simulator training was also effective at improving the situation awareness and reducing student pilots’ workload when approaching an airport with complex traffic [4]. Technical cockpit aids such as the flight guidance information presented in a head-up display can contribute to better flight path control [32]. The use of a velocity flight path vector in the head-up display diminishes lateral error during landing and is currently used by pilots for approach and landing in various visibility conditions [33].

In this study, technical aids and training are combined with the intention to develop a method for preparation of the students in bridging the gap between theoretical and practical instruction for approach to landing. Part of the AR support for the training concept addressed in this study is the holographic visualization of the flight path vector, speed, and altitude indications. These augmented features are expected to support trainees in developing a mental model of the approach flight path, altitude, speed, and the effect of their actions. The flight training concept developed for this study includes gaming features that meet diverse gender preferences such as answering self-evaluation questions and receiving qualitative and quantitative feedback when finishing the task [11]. In order to determine the feedback elements, the cognitive and associative stages of skill acquisition described in Fitts and Posner [34] were considered. In the cognitive stage, an initial cognitive encoding of the skill takes place. In the associative stage, the trainees are able to detect their errors and deviations. This is a precondition to improvement, but it can also be discouraging, and thus, it is important to highlight the benefits of this phase for performance. The autonomous stage [34], which involves a certain degree of automation, is not addressed in this short experiment.

1.5. Research Question

The purpose of this exploratory study is to compare the effects of a conventional and an AR-supported concept for final approach training on the performance, situation awareness, workload, motivation, and emotion of the participants, by also addressing gender effects. In addition, this study investigates the relationship between learning performance and various aspects of immersion, motivation, emotion, situation awareness, and workload. The AR application is assessed in terms of features, immersion, and gender preferences.

2. Materials and Methods

2.1. Participants

Volunteers for the experiment were recruited using university announcements. Twenty-eight women participating in the experiment were randomly assigned to an experimental group (N = 14, M = 22.64 years, SD = 2.47 years) or a control group (N = 14, M = 24.21 years, SD = 6.09 years). Thirty-one male participants were also assigned to an experimental group (N = 15, M = 24 years, SD = 3.80 years) and a control group (N = 16, M = 22.13 years, SD = 2.47 years). Informed consent was obtained from all subjects involved in the study, who were also offered a compensation of EUR 50 for participation. The study was conducted in accordance with the ethical guidelines of the Declaration of Helsinki and the General Data Protection Regulation of the European Union.

2.2. Equipment

2.2.1. AR Headsets

For the experiment, a Microsoft HoloLens (1st gen) was used. It is a wireless stand-alone device with six degrees of freedom. This stereoscopic see-through AR headset offers a 30${}^{\circ }$× 17.5${}^{\circ }$ field of view. Using a Windows 10 operating system, the AR headset enables users to pin virtual objects into the real-world space of the flight simulator. To interact with the holograms’ special gestures, speech control could also be used [35].

2.2.2. Flight Simulator

A generic light aircraft flight simulator with glass cockpit and genuine cockpit controls (e.g. flight stick, rudders, flaps, engine controls) similar to those of a light aircraft was used for the experiment. It was a fixed-base aircraft simulator with a cabin in the original size of a light aircraft. The visual scenery was projected on a cylindrical screen with 7 m diameter providing 190-degrees horizontal and 40-degrees vertical vision angle. The instructor station contained a set of predefined landing scenarios, a data logger, and a data processor. Therefore, the instructor station was used to set the landing scenarios, and to save and process the kinematic data used for performance evaluation.

2.3. AR System

The implemented AR system [36], already introduced in a case study by [12], is described there in detail. The system consists of three parts: the HoloLens (1st gen) application, the server application, and the generic light aircraft flight simulator. The HoloLens application was responsible for the visual AR cues such as flight path vector (FPV), airspeed, altitude, simulation time, and a self-evaluation feedback form (see Figure 1). The server application was used as proxy between the HoloLens application and the flight simulator to exchange data, e.g., for the orientation of the FPV.

The flight simulator was used to generate the landing scenarios and the data for the AR cues shown in the HoloLens. Communication between the flight simulator and the server application was implemented using user datagram protocol (UDP) messages, and communication between the server application and the HoloLens application used the transmission control protocol (TCP). After an initial calibration of the HoloLens application, an instructor started the simulation for the trainee. The flight simulator logged the position and orientation of the simulated aircraft, and these kinematic data were sent to the HoloLens application, which used them to display the FPV. Furthermore, the deviation from the ideal landing approach was analyzed to evaluate the flight performance. In case of an unsafe approach defined by exceeding certain thresholds, the simulation was stopped. Only safe approaches could be continued for landing. After each approach (regardless of whether successful or not), the trainee completed a holographic self-evaluation form. Thereafter, the AR system provided two forms of feedback (again, after each approach whether successful or not): a quantitative score (0—poor to 100—very good) calculated for the landing performance, and a qualitative accuracy feedback of the self-evaluation. The feedback of the self-evaluation (e.g., too short, too far, too slow, too fast) was displayed as green check marks (correct) and red cross symbols (incorrect) directly at the specified answers, after the trainee submitted the self-evaluation form [12,36].

2.4. Procedure

For the experiment, a pre-test–post-test control group design [37] was used.

Initially, both groups received a written briefing and a familiarization session with the flight simulator and with the procedure for the final approach. Both groups conducted a pretest without AR in the flight simulator consisting of three final approaches with the scenarios A (see Figure 2). The treatment of the groups differed in the training phase. The experimental group trained half of the scenarios with the AR application that was connected to the flight simulator, but half of the scenarios without the AR application. The control group conducted the training with the same final approach scenarios, but used the flight simulator only. The six approach scenarios differed in altitude and lateral offset. The scenarios “A” started at 2300 ft above ground (3400 ft AMSL), and “B” at 2700 ft (3800 ft AMSL) above ground. Regarding the lateral offset, there were three versions: “center” without offset, and “left” and “right” with offset. Training was conducted with the scenarios “A” and “B”. After each training scenario, the participants answered a self-evaluation form and they received formative feedback about it, as well as a performance score. For the experimental group, this was realized by the AR application for all flights, including the ones only using the flight simulator for training. For the control group, a paper form was used. In the post-test without AR, both groups conducted three final approach scenarios in the variant “B” in the flight simulator. This procedure is summarized in

Table 1. Procedure for the experimental and control group.

Phase	Content	Experimental Group	Control Group
Introduction	Briefing	written instructions
	Familiarization	with simulator
Pre-test (three scenarios)	Approach maneuvers	in the simulator without AR
Training 1	Familiarization	with HoloLens and the AR application	—
(12 scenarios)	Approach training in the simulator	six with AR and six without AR	12 without AR
	Self-evaluation and feedback	with AR	with pen and paper
	Break
Training 2	Approach training in the simulator	six with AR and six without AR	12 without AR
(12 scenarios)	Self evaluation and feedback	with AR	with pen and paper
Post-test (three scenarios)	Approach maneuvers	in the simulator without AR

.

2.5. Dependent Measures

The final approach performance was assessed in terms of deviation, self-assessment of the deviations, and the approach quality. The approach quality was calculated as an objective score based on recorded kinematic data of the simulator. Kinematic data (e.g., altitude, speed, direction) were recorded for all test and training flights during the experiment. When deviation of airspeed and estimated touchdown point exceeded a certain threshold, a cumulative penalty was calculated during the approach. The approach quality was calculated by subtracting the penalty from the initial value of 100 (excellent approach). Therefore, the lower limit of the approach quality score was 0 (extremely poor approach).

In addition, an objective deviation score was calculated for each approach scenario by using kinematic data of the simulator. One point for each type of deviation: too fast, too slow, too short, and too far. Lower objective deviation scores reflect better performance.

The self-assessment of the deviations was calculated for each scenario as the sum of correct answers received from the trainee. Therefore, a higher self-assessment score reflects superior situation awareness/performance because this is required in the associative stage of skill acquisition [34]. For the statistical analysis, final pre-test and post-test scores were calculated by adding the values obtained for each of the three approach scenarios per test.

Trainees’ situation awareness was self-rated after each test using the Situation Awareness Rating Technique (SART) [38], with item scales ranging from 1 (low) to 7 (high). The final subjective score of SART was used, as well as the scores of the subscales: demands on attention resources, supply of attention resources, and understanding of the situation. Workload was self-assessed using the Task Load Index (NASA-TLX) [39] containing six dimensions: mental demand, physical demand, and temporal demand of the task, effort, performance, and frustration. The NASA-TLX scales ranged from 1 (very low) to 7 (very high). In order to calculate the total workload score, the performance scale was inverted. After each test, the trainees self-rated the emotions experienced during the test using the Positive and Negative Affect Schedule (PANAS) [40]. In PANAS, the item scales ranged from 1 (very slightly or not at all) to 6 (extremely). Motivation was self-assessed by the trainees after each test using the Questionnaire on Current Motivation (QCM) [41] consisting of 18 items measuring four factors: challenge, interest, probability of success, and anxiety/fear of failure. Each QCM item had a scale ranging from 1 (disagree) to 7 (agree). The trainees from the experimental group assessed various features of the AR application after using it. The quiz, the gesture and voice interaction used for feedback, the holograms, the FPV, indicated airspeed and altitude, and the projection field were assessed on a scale ranging from 1 (very poor interaction) to 5 (very good interaction). Comfort and trust in the AR application were rated from 1 (very low) to 5 (very high). After using the AR application, the trainees assessed their interaction with it using the Augmented Reality Immersion (ARI) questionnaire [19] with 42 items. Each ARI item was self-rated on a scale ranging from 1 (totally disagree) to 7 (totally agree). The ARI items were grouped on three main scales: AR engagement, AR engrossment, and AR total immersion. The composition of the main scales according to [19] is presented below:

• Engagement = Attraction + Time investment + Usability;
• Engrossment = Emotional attachment + Focus of attention;
• Total immersion = Presence + Flow.

2.6. Independent Variables

Independent variables were the test (pre-test vs. post-test), the group treatment (experimental vs. control group), and the gender group (female vs. male).

2.7. Data Analysis

The repeated measures analysis of variance with one within-subjects factor (test) and two between-subjects factors (group treatment and gender) was used to assess the training effects. The sample size was chosen to meet the statistical requirements [42]. The Bonferroni correction was applied to pairwise comparisons. One-way analyses of variance were used to calculate differences between the gender groups in the assessment of AR immersion and in the preferences for AR features. Pearson’s r was used to calculate correlations between performance in post-test and various aspects of immersion, motivation, emotion, situation awareness, and workload. Alpha was set at 0.05. The template from [43] was used for figures.

3. Results

3.1. Training Effects

Descriptive data for the pre-test and post-test are presented in

Table 2. Descriptive data for the pre-test and post-test across all groups. (*) indicates statistically significant variables.

Test	Pretest		Posttest
Variable	Mean	SE	Mean	SE
Performance
Objective deviation *	7.71	0.22	4.24	0.31
Deviation self-assessment *	7.98	0.23	9.15	0.25
Approach quality *	158.79	11.47	261.86	9.22
Situation awareness SART
Understanding *	12.62	0.39	14.35	0.46
Demands	10.48	0.42	10.63	0.49
Supply	20.59	0.39	20.35	0.51
Total SA score	22.72	0.83	24.07	0.93
Workload NASA-TLX
Workload	10.83	0.66	10.02	0.6
PANAS
Positive Emotion	36.33	0.75	35.49	0.97
Negative Emotion *	15.37	0.66	12.09	0.41
Motivation
Challenge *	17.87	0.46	16.57	0.53
Interest	24.47	0.45	24.16	0.55
Success probability	8.56	0.22	8.34	0.25
Fear of failure *	10.07	0.54	7.61	0.32

. Results show that the training, irrespective of the group treatment, had a significant effect on improving all performance parameters. The deviation from the optimal approach was significantly reduced after training [F(1,50) = 122.94, p < 0.0001, ${\eta }^2$ = 0.71]. The trainees also significantly improved their deviation self-assessment [F(1,50) = 13.68, p < 0.001, ${\eta }^2$ = 0.22], and approach quality [F(1,50) = 64.47, p < 0.0001, ${\eta }^2$ = 0.56] in post-test as compared to pre-test. Training also had a significant effect on improving trainees’ understanding of the situation [F(1,54) = 17.56, p < 0.0001, ${\eta }^2$ = 0.25], reducing the negative emotions [F(1,54) = 36.52, p < 0.0001, ${\eta }^2$ = 0.40], the challenge [F(1,54) = 15.64, p < 0.0001, ${\eta }^2$ = 0.23], and fear of failure [F(1,54) = 39.41, p < 0.0001, ${\eta }^2$ = 0.42]. No significant training effects were found on the demand and supply of attentional resources, total subjective scores of situation awareness, workload, positive emotion, interest, or success probability.

3.2. Analysis of the Specific AR Training

Descriptive data for the scores of the experimental and of the control groups in pre-test and post-test are presented in

Table 3. Descriptive data of the experimental group (EG) and control group (CG) for the pre-test and post-test.

Variable	Group	Pretest		Posttest
		Mean	SE	Mean	SE
Performance
Objective deviation	EG	7.57	0.33	4.55	0.46
	CG	7.84	0.29	3.94	0.41
Deviation self-assessment	EG	8.17	0.34	9.01	0.37
	CG	7.78	0.30	9.29	0.33
Approach quality	EG	149.78	17.11	250.37	13.76
	CG	167.89	15.28	273.35	12.29
Situation awareness SART
Understanding	EG	12.75	0.55	14.29	0.66
	CG	12.49	0.54	14.42	0.63
Demands	EG	10.89	0.60	11.25	0.71
	CG	10.08	0.58	10.00	0.69
Supply	EG	20.29	0.56	19.79	0.73
	CG	20.89	0.55	20.91	0.71
Total SA score	EG	22.14	1.19	22.82	1.34
	CG	23.30	1.15	25.33	1.29
Workload NASA-TLX
Workload	EG	11.32	0.95	9.79	0.86
	CG	10.34	0.92	10.25	0.83
PANAS
Positive Emotion	EG	36.89	1.08	35.43	1.40
	CG	35.77	1.05	35.56	1.35
Negative Emotion	EG	15.93	0.95	12.96	0.59
	CG	14.81	0.92	11.21	0.57
Motivation
Challenge	EG	18.11	0.67	16.79	0.76
	CG	17.63	0.64	16.36	0.74
Interest	EG	25.18	0.65	24.25	0.78
	CG	23.76	0.63	24.08	0.76
Success probability	EG	8.79	0.32	8.18	0.35
	CG	8.34	0.31	8.51	0.34
Fear of failure	EG	10.96	0.77	8.21	0.47
	CG	9.18	0.75	7.01	0.45

. Statistical analyses did not show significant effects of the type of training, whether AR-supported or conventional flight simulator training, either on trainees’ performance, workload, situation awareness, emotion, or on their motivation.

The results for trainees’ AR immersion and their assessment of AR features are presented descriptively in

Table 4. Descriptive data of AR immersion and appreciation of AR features of the female and male trainees from the experimental group.

Variable	Women		Men
	Mean	SE	Mean	SE
AR immersion
Attraction	13.25	0.65	12.17	0.65
Time investment	12.67	0.64	12.00	0.64
Usability	6.00	1.24	6.50	1.24
Emotional attachment	13.33	1.15	12.25	1.15
Focus of attention	17.33	0.88	17.08	0.88
Presence	12.58	1.35	14.33	1.35
Flow	17.83	0.98	16.17	0.98
Engagement	31.92	2.29	30.67	2.29
Engrossment	30.67	1.67	29.33	1.67
Total immersion	30.42	1.84	30.50	1.84
AR features and interactions
AR comfort	2.50	0.37	3.00	0.37
AR trust	2.35	0.34	2.92	0.34
Gesture interaction	3.25	0.36	3.25	0.36
Voice interaction	0.92	0.36	0.08	0.36
Quiz	3.67	0.43	3.00	0.43
Holograms	3.08	0.50	3.58	0.50
AR projection field	3.42	0.39	3.58	0.39

. There were no significant gender differences.

3.3. Gender Effects

The statistical analysis of gender effects was focused on the interaction term between test and gender, and the interaction term gender and group. The workload reported in pre-test was higher in women (M = 13.11, SE = 0.95) than in men (M = 8.55, SE = 0.92), but in post-test smaller differences were found between women (M = 10.64, SE = 0.86) and men (M = 9.40, SE = 0.83). The interaction term between test and gender had a significant effect on trainees’ workload [F(1,54) = 6.32, p < 0.02, ${\eta }^2$ = 0.11]. In pre-test, women experienced more intensive negative emotions (M = 17.75, SE = 0.95) than men (M = 12.99, SE = 0.92), but in post-test smaller differences were found between women (M = 12.68, SE = 0.59) and men (M = 11.49, SE = 0.57). The interaction term between test and gender had a significant effect on trainees’ negative emotion [F(1,54) = 10.83, p < 0.002, ${\eta }^2$ = 0,17] and fear of failure [F(1,54) = 5.21, p < 0.026, ${\eta }^2$ = 0.09]. In pre-test, the fear of failure was rated higher by women (M = 11.32, SE = 0.77) than by men (M = 8.83, SE = 0.92), but in post-test similar levels were reported by both women (M = 7.96, SE = 0.47) and men (M = 7.26, SE = 0.45).

As Figure 3 shows, in pre-test the approach quality was better in men (M = 220.20, SE = 15.59) than in women (M = 97.37, SE = 16.83), but in post-test smaller differences were found between women (M = 244.84, SE = 13.53) and men (M = 278.88, SE = 12.54). Also as illustrated in Figure 4, the improvement of the approach quality in post-test as compared to pre-test was larger in women than in men.

The approach quality was significantly influenced by the interaction term between test and gender [F(1,50) = 11.96, p < 0.001, ${\eta }^2$ = 0.19], as illustrated in Figure 3. Nevertheless, the interaction term between test and gender neither had a significant effect on the objective deviation score, deviation self-assessment, understanding of the situation, demands on attentional resources, nor on the supply of attentional resources, total SA score, positive emotion, challenge, interest, or success probability.

3.4. Correlations between Performance and Subjective Measures

Results of the correlations between performance in post-test (landing deviation, self-assessment, and approach quality) and various motivational factors, emotion, immersion, situation awareness, and workload, are presented in

Table 5. Correlation tests between performance in post-test and motivational, emotion, immersion, situation awareness, and workload (* results significant at p < 0.05).

Variable	Landing Deviation			Self-Assessment			Approach Quality
Statistical Test	r	p	N	r	p	N	r	p	N
Situation awareness SART
Understanding	0.154 *	0.02	57	−0.078	0.28	57	0.154	0.13	55
Demands	0.249 *	0.03	57	−0.098	0.23	57	−0.249 *	0.03	55
Supply	−0.153	0.13	57	−0.181	0.09	57	0.136	0.16	55
Total SA score	−0.338 *	0.01	57	−0.073	0.30	57	0.275 *	0.02	55
Workload NASA-TLX
Workload	0.227 *	0.05	57	0.014	0.46	57	−0.239 *	0.04	55
PANAS
Positive Emotion	−0.186	0.08	57	−0.088	0.26	57	0.196	0.08	55
Negative Emotion	0.206	0.06	57	−0.059	0.33	57	−0.466 *	0.01	55
Motivation
Challenge	0.060	0.33	57	−0.106	0.22	57	0.076	0.29	55
Interest	−0.033	0.41	57	−0.158	0.12	57	0.079	0.28	55
Success probability	−0.284 *	0.02	57	0.083	0.27	57	0.306 *	0.01	55
Fear of failure	0.241 *	0.04	57	−0.156	0.12	57	−0.366 *	0.01	55
AR immersion
Attraction	0.076	0.35	28	−0.255	0.10	28	−0.200	0.17	26
Time investment	0.080	0.34	28	−0.222	0.13	28	−0.130	0.26	26
Usability	−0.228	0.12	28	0.135	0.25	28	0.154	0.23	26
Emotional attachment	−0.104	0.30	28	−0.050	0.40	28	−0.030	0.45	26
Focus of attention	0.061	0.38	28	−0.483 *	0.01	28	−0.100	0.32	26
Presence	−0.367 *	0.03	28	0.031	0.44	28	0.078	0.35	26
Flow	0.193	0.16	28	−0.328 *	0.04	28	−0.040	0.43	26
Engagement	−0.08	0.34	28	−0.063	0.38	28	−0.010	0.48	26
Engrossment	−0.028	0.44	28	−0.315 *	0.05	28	−0.070	0.36	26
Total Immersion	−0.163	0.20	28	−0.158	0.21	28	0.036	0.43	26
AR Features
Comfort	−0.441 *	0.01	28	0.086	0.33	28	0.378 *	0.03	26
Trust	−0.125	0.26	28	−0.032	0.44	28	−0.160	0.22	26

.

4. Discussion

This study explored three main areas related to a new concept for flight training. The effects of an AR-supported flight training application were compared with those of a conventional simulator training. In addition, the relationship between learning performance and various subjective variables was explored. Furthermore, trainees’ interaction with the AR application was assessed.

4.1. Effects of the Flight Training

Results show that the training, irrespective of the group treatment, significantly improved trainees’ performance as indicated by smaller deviations from the required final approach parameters and a higher approach quality. Furthermore, the trainees improved their self-assessment of deviation, indicating progress towards the associative stage of skill development as described in Fitts and Posner [34]. Training also significantly improved trainees’ understanding of the situation, reduced the perceived challenge related to the task, the intensity of their negative emotions, and the motivational factor fear of failure. No significant changes in the demand and supply of attention resources, total situation awareness, or workload were found. This means that the trainees were not yet progressing towards the autonomous phase of skill development [34], which was hardly expected, considering the small number of repetitions during the experiment. Although self-awareness of deviations could be demotivating for some trainees, this was not the case in the present study, as positive emotion, interest, and perceived success probability did not change significantly in the course of the training.

Both the experimental group using AR-supported training and the control group that only used the flight simulator for training showed similar results in terms of performance, workload, situation awareness, emotion, and motivation. No statistically significant effect of the AR-supported training was observed. This is in line with other experimental results [13]. Although the presentation of flight guidance information in a head-up display resulted in better approach performance in pilots [32,33], this result is not directly comparable to our study in terms of procedure or target group. Here, the guidance was only presented to the trainees from the experimental group and only during training, not during tests. Therefore, the expected effect was on building a more accurate mental model of a correct approach by using AR cues for training and the transfer of this ability to a situation that was not supported by AR. However, the statistical results do not indicate this effect. Maybe the test scenarios were too simple. It could also be that motivational factors [41,44,45] compensated for the differences in the training concepts, or possibly other factors interacted with the training effect. Furthermore, the objective formative feedback concept is not typical for conventional training, but it was included here for a better comparison with the AR procedure. Therefore, more research is needed to clarify which features can provide an advantage of AR training for approach to landing as compared to conventional flight training. The expected benefit is not to replace conventional training, but to facilitate the use of AR or other mixed-reality applications as a preparation for training, and to bridge the gap between the theoretical and practical instruction, thus addressing a number of issues (e.g., time pressure) mentioned in research [10] that are related to flight training. As indicated by the gender effects, AR-based training means could also contribute to more socially sustainable conditions for pilot training and hopefully influence the gender gap in the pilot profession [1,2]. However, although the sample was relatively small, the pre-test–post-test control group design [37] was considered most appropriate to determine whether a change takes place as an AR training effect. This type of design also “controls for many potential threats to internal validity” ([37], p. 216). Multiple significant interactions between test and gender show that the training had different effects depending on the gender group. The decrease in trainees’ workload, fear of failure, and negative emotions in post-test as compared to pre-test was significantly larger in women than in men Although both gender groups improved their approach quality in post-test as compared to pre-test, the improvement was larger for women as compared to men (Figure 4). These results are in line with previous research [13,26], showing that initial gender differences can diminish after training.

No significant interactions between gender and test were found when comparing objective deviation scores, deviation self-assessments, understanding of the situation, demands on attentional resources, supply of attentional resources, total SA scores, positive emotion, challenge, interest, or success probability.

4.2. The Relationship between Performance and Subjective Variables

The results show that in post-test, higher approach quality was associated with higher situation awareness, lower workload, and lower perceived demands on attentional resources. Similarly, lower deviation scores were associated with higher situation awareness, lower workload, and demands on attentional resources. Interestingly, there was also a significant positive correlation between the perceived understanding of the situation and the deviation score. Maybe the trainees were aware of the deviations, but excessive corrections may have induced more deviations.

The results of correlation tests also highlight the importance of trainees’ subjective experience for performance. Arousal and emotion regulation when managing the cognitive, emotional, and physical demands of a flight task are not only important for the training phase, but also for expert performance [46]. In the present study, there were significant correlations between performance and the emotions experienced by trainees and their motivational factors in post-test. Higher self-estimated success probability and less intense fear of failure were associated with lower deviation scores during the approach to landing. A better approach quality was associated with higher perceived success probability, and less intensive negative emotion and fear of failure. This is in line with previous findings [25] indicating that student pilots who experienced more intensive negative emotions performed worse in flight training experiments.

4.3. Trainees’ Interaction with the AR Application

Main features of the AR application were the presentation of the flight guidance cues, and the interactive feedback with elements of gamification. Trainees’ subjective experience in interaction with the AR-supported training concept was analyzed with regard to performance in post-test, when the experimental group did not use AR. Results show that higher AR presence was associated with lower objective deviation scores in post-test. Presence is considered to be an essential contributor to total immersion [19,47] and is facilitated by multimedia features and game-based challenges [48]. Higher comfort with AR during training was also associated with lower deviation scores and a better approach quality in post-test. Thus, these results indicate that there is an interdependence between the trainees’ learning performance, subjective experience, and interaction with the AR-supported training application. Interestingly, better self-assessment in post-test was associated with less focused attention and lower flow scores during the AR training. Although focus of attention was assessed as an element of AR immersion [19], these findings need to be interpreted with care because the approach task required attention sharing among multiple control parameters such as descend rate, speed, and heading.

There were no significant differences between women and men in the AR immersion and their assessment of AR features. Thus, the findings of previous research [13] were not replicated, indicating that gender preferences may be rather application-specific and cannot be generalized to the interaction with other AR applications. Nevertheless, more research with larger numbers of participants is needed to gain clarity on gender preferences related to AR-supported training concepts.

4.4. Limitations

As we are in the early stages of exploring the effects of AR in flight training, more experimental research is needed for a comprehensive evaluation of the benefits and limitations of this technology, as well as for the identification of the most valuable areas of implementation. Although this study presents one of the few experimental evaluations of AR in flight training, there are also a number of limitations to mention such as the relatively small sample size. In addition, for addressing reliability aspects, future replication studies would be needed to verify whether the results could be reproduced.

5. Conclusions

Overall, this exploratory study indicates a positive effect of both conventional and AR-supported training on improving trainees’ performance, their understanding of the situation and self-assessment, and also on reducing the perceived challenge related to the approach task, the intensity of negative emotions, and fear of failure. The performance of the experimental group was improved despite the absence of AR support during post-test. Initial gender differences in performance, workload, fear of failure, negative emotions, and fear of failure were diminished after the training, thus confirming previous research on gender diversity in flight training [4,26,28]. Higher performance in post-test was associated with higher situation awareness, lower workload, and lower perceived demands on attentional resources. Higher AR presence and comfort with AR during training were associated with higher performance in post-test. Therefore, these results indicate that there is an interdependence between the trainees’ learning performance and their emotions, motivations, and interaction with the AR-supported training application. The results show that such a training concept, whether implemented in AR or in another mixed-reality environment, would benefit the trainees in transition from theoretical to practical flight instruction, thus supporting the results of qualitative research [8]. The methods and results presented in this study are important in the current context of rethinking the methods of education [1] with the aim of attracting talented women and men to technical domains in general and aviation in particular.