4.1.2. Experimental Setup
The mentioned “gaming” workshop, which was the chosen method for performing the first validation cycle, was conducted on 21 to 23 November 2017 and led by DLR at their premises in Cologne, Germany. Several operational experts were invited to take part in this workshop:
Two pilots of commercial aviation (in the following referred to as “conventional pilots”);
Two pilots of military unmanned aircraft (in the following referred to as “RPAS pilots”);
One civil aerodrome controller;
One military aerodrome controller with RPAS experiences;
One ATM expert from an Air Navigation Service Provider.
The last three participants of this list are in the following referred to as air traffic controllers (ATCOs).
The workshop comprised several steps:
- (1)
Introduction and E-OCVM basics,
- (2)
Gaming session Baseline 1 and Baseline 2,
- (3)
Detailed presentation of the SSTR concept and related procedures,
- (4)
Discussion and questions,
- (5)
Gaming session “RPAS arrival using the SSTR procedure”,
- (6)
Gaming session “RPAS departure using the SSTR procedure”,
- (7)
Gaming sessions “non-nominal situations” (lost communication, lost command and control link),
- (8)
Summary and closure.
Every gaming session was conducted in two phases: the gaming phase itself and the subsequent assessment phase. During the gaming phase, the first author of this article supervised the play through every single step of the RPAS movement, as well as the analysis and discussion by the workshop participants. This was done by presenting an airfield layout, while the position of the remotely piloted aircraft (RPA), ATC actions and possible effects on the surrounding traffic were animated. Sample screenshots are shown in
Figure 2. Different traffic constellations were also taken into account.
The assessment phase was done by interviewing the participants and by using standard and tailor-made questionnaires to determine several indicators. More information is given in the next section.
4.1.4. Results
Safety
(a) Tailor-Made Questionnaire Answered by ATCOs
The following tables display the individual questions of the tailor-made questionnaire and the average answers given by ATCOs regarding nominal (
Table 1) and non-nominal scenarios (
Table 2). The following Likert Scale was used for all questions: (1) = strongly disagree, (2) = disagree, (3) = neutral, (4) = agree and (5) = strongly agree. The term “this procedure” refers to the new SSTR concept.
(b) Tailor-Made Questionnaire Answered by Pilots
The following tables display the individual questions of the tailor-made questionnaire and the average answers given by conventional pilots regarding nominal (
Table 3) and non-nominal scenarios (
Table 4). The following Likert Scale was used for all questions: (1) = strongly disagree, (2) = disagree, (3) = neutral, (4) = agree and (5) = strongly agree. The term “this procedure” refers to the new SSTR concept.
(c) Tailor-Made Questionnaire Answered by RPAS Pilots
The following tables display the individual questions of the tailor-made questionnaire and the average answers given by RPAS pilots regarding nominal (
Table 5) and non-nominal scenarios (
Table 6). The following Likert Scale was used for all questions: (1) = strongly disagree, (2) = disagree, (3) = neutral, (4) = agree and (5) = strongly agree. The term “this procedure” refers to the new SSTR concept.
(d) Estimation of Safety Level
The results of this inquiry have been comprehensively published in Reference [
8]. Summarized, the level of safety was estimated on the average as very high for all scenarios, i.e., Baseline 1, Baseline 2 and SSTR (average value 9.05 on a scale from 1 to 10). All ratings from ATCOs, RPAS pilots and conventional pilots differ marginally.
(e) Collected Qualitative Feedback
The workshop participants found that the current safety risks, which already exist in manned aviation, will still exist in an integrated manned–unmanned traffic, but no additional safety risks are introduced when applying the SSTR procedure.
From the ATC perspective, it would be beneficial to have an “emergency stop” button for all taxiing RPA in case an unmanned flight does not comply with a mandatory stop point.
Regarding non-nominal situations, one ATCO had safety concerns for the case that the communication failure is caused on ATC side and not on aircraft side. In this case, the pilots of the manned aircraft cannot be informed about taxiing unmanned aircraft accordingly. Another safety risk would be that the RPAS experiencing a communication loss has no possibility to communicate other additional emergencies besides the communication loss. The same situation would occur in case of manned aircraft, so this was not considered an additional safety risk. In any case, no matter if a non-nominal situation occurs with a manned or an unmanned aircraft, these flights shall be handled with extreme care, and detailed information and instructions should, in any case, be contained in the local aerodrome emergency plan.
One conventional pilot saw an additional safety risk when the RPAS experiences a loss of the Command and Control (C2) link when already lined up on the runway. This RPA would then block this runway, meaning that it would not be available for several minutes for a possible emergency landing of a manned aircraft. However, this problem is not specific to the SSTR procedure.
RPAS pilots did not see any additional safety risk in nominal or non-nominal conditions. Due to the possibility to use a landline between ATC and the remote pilot as backup communication channel, which is not available for manned aircraft, safety would even be improved in case of lost communication situations.
There was no doubt that safety was guaranteed at all times for Baseline 1 and Baseline 2 scenarios by all participants.
Interoperability
(a) Estimation of Interoperability Level
The results of this inquiry have been comprehensively published in Reference [
8]. Summarized, the level of interoperability was estimated to be very high and almost equal for Baseline 2 and SSTR scenarios (average value 9.71 resp. 8.76 on a scale from 1 to 10). In contrast to this, Baseline 1 was rated to provide a very low level of interoperability (average value 2.86 on a scale from 1 to 10).
(b) Collected Qualitative Feedback
As a general comment, without a direct link to the SSTR procedure, RPAS pilots highlighted that the RPAS, including the remote pilot, shall always be considered as one working system. Following this approach, it is not necessarily required that specific abilities must be reproduced by a technical system when it is also possible to enable the remote pilot to perform this task with some technical support.
Regarding the SSTR procedure, the concrete communication channel between ATC and the RPAS still must be defined. When restricting this communication to a simple “go” command, as foreseen by the SSTR concept, other aerodrome traffic participants may lose their traffic situational awareness, which is currently provided by the open voice radio communication. However, as the same problem is also introduced with the increasing use of data link technology like Controller-Pilot Data Link Communication (CPDLC), a change of this paradigm is ongoing from fully transparent communication toward individual communication with full trust in ATC. In any case, the ATC–RPAS communication must be clear and unambiguous, which is fully guaranteed by the SSTR procedure.
The PRAS pilots stated that the idea of standard taxi routes is very compatible with the way how RPAS taxi movements are conducted from the remote pilot’s side in the future, i.e., following a defined sequence of waypoints on ground. In addition, it eases flight preparation and orientation on an airport’s surface.
The Baseline 1 scenario is seen by the tower controllers as very comparable to a state visit, e.g., the arrival of the US president at a civil airport. Therefore, the same serious impact on airport capacity, delays and airline schedules can be expected.
The Baseline 2 scenario, which is towing operations, is already a standard procedure for military RPAS ground movements, as well as for civil aircraft staying overnight or similar situations.
Access and Equity
(a) Estimation of Access and Equity Level
The results of this inquiry have been comprehensively published in Reference [
8]. Summarized, the level of access and equity was estimated to be very high and almost equal for Baseline 2 and SSTR scenarios, (average value 8.71 resp. 8.79 on a scale from 1 to 10). In contrast to this, Baseline 1 was rated to provide a very low level of access and equity (average value 1.86 on a scale from 1 to 10).
(b) Collected Qualitative Feedback
Regarding the Access and Equity KPA, ATCOs were concerned about the fact that, especially shortly after the SSTR procedure has become operational, it will take some time until ATC staff gets used to it. RPAS ground movements using the SSTR procedure might unintentionally be advantaged over manned aircraft, and additional safety buffers might be implemented, which may cause a little delay for manned aircraft during this initial period.
In general, it must be clarified with which level of priority RPAS flights will be treated in which situation, but as there are no humans on board of these flights, it would be natural and can be expected that unmanned aircraft will have the lowest priority of all air traffic participants.
The Baseline 1 scenario was seen by all participants as the worst case, showing the maximum imbalance between manned and unmanned flights in terms of access and equity, and should therefore not be understood as a realistic option for operational implementation any more.
The Baseline 2 scenario was rated slightly better by the RPAS pilots compared against the SSTR procedure, as towing operations still provide a totally free access to any position on the aerodrome at any time for an unmanned aircraft, while the lateral movement is more restricted for the SSTR procedure.
Human Performance
(a) Impact on Mental Workload (AIM-s)
The AIM-s questionnaire is a standard questionnaire specific to ATCO workload. It contains a standardized set of 16 questions, using a seven-point Likert Scale from 0 (=no impact) to 6 (=extreme impact). The final result of this questionnaire is determined by calculating the average over all questions [
18]. The lower the average score, the lower the overall impact on mental workload is.
As this questionnaire is specific to ATC, it was just filled by all participating ATCOs per scenario. The results are displayed in
Figure 3.
(b) Situational Awareness (SASHA)
The SASHA questionnaire is a standard questionnaire specific to situational awareness. It contains a standardized set of six questions, using a seven-point Likert scale from 0 (=never) to 6 (=always). The final result of this questionnaire is determined by calculating the average over all questions [
19]. The higher the average score, the higher the level of situational awareness is.
This questionnaire was filled by all participants, per scenario, from their professional perspective. The results are displayed in
Figure 4.
(c) Trust in Automation (SATI)
The SATI questionnaire is a standard questionnaire specific to trust in automation. It contains a standardized set of six questions, using a seven-point Likert scale from 0 (=never) to 6 (=always). The final result of this questionnaire is determined by calculating the average over all questions [
20]. The higher the average score, the higher the trust in the procedure is.
This questionnaire was filled by all participants, per scenario, from their professional perspective. The results are displayed in
Figure 5.
(d) Estimation of Human Performance Level
The results of this inquiry were comprehensively published in Reference [
8]. Summarized, the demand on human factors was rated almost equal for all three scenarios, on average, with only slight differences. Nevertheless, the Baseline 1 scenario was rated most demanding for ATCOs and conventional pilots, but least demanding for remote pilots. For all participants, the Baseline 2 scenario was rated slightly better than the SSTR procedure (average value 8.71 against 8.00 on a scale from 1 to 10).
(e) Collected Qualitative Feedback
All participants did not express any concerns regarding their mental workload, situational awareness or other human factors aspects, both for arrival or departure operations, as well as in nominal or non-nominal conditions.
For ATC, an increased workload is to be expected in any case, including the Baseline 2 and the SSTR scenario. An appropriate training is recommended, as this can help to get familiar with the new situation and procedures.
The two conventional pilots just highlighted that it might cause some level of frustration, when manned aircraft are required to hold on a taxiway because of an RPAS movement, no matter if it is towed or using the SSTR procedure. In general, they still see the participation of unmanned aircraft in commercial airport traffic in a skeptical way.
4.1.5. V1 Validation Summary
In this section, the results presented above are summarized.
Safety
All three scenarios are considered by the participants to ensure safety on an almost equal level. This conclusion is underlined by all applied assessment methods in the same way. Safety is seen as maintained for all three scenarios, compared to pure manned aviation.
Interoperability
The Baseline 2 and SSTR scenarios provide a high level of interoperability. The Baseline 2 scenario was rated even higher than the new SSTR procedure, because towing operations are basically daily airport business, implying no change to the existing ATM system and standard procedures at all. In strong contrast, the Baseline 1 scenario achieved very low ratings, as high impact on current aerodrome capacity and traffic flow is expected.
As a conclusion, the Baseline 1 scenario is not considered to be a realistic option at medium size or larger airports. However, the Baseline 2 and SSTR scenarios are seen as equally suited. The only point to be regulated is the detailed communication means between ATC and the RPAS.
Access and Equity
For this KPA, the results are very similar to the Interoperability KPA. The Baseline 2 and SSTR scenarios provide a high and very comparable level of access and equity. The Baseline 1 scenario achieved very low ratings, as a free access to the aerodrome infrastructure is not possible for an RPAS when manned aircraft are taxiing, and vice versa. As the traffic share between manned and unmanned aircraft is not balanced (it can be expected that in the near future there will always be much more manned traffic movements than unmanned ones), there will often be the situation that several manned aircraft will have to wait for one single RPAS movement when applying the Baseline 1 scenario. For the Baseline 2 and SSTR scenarios, it must be regulated which level of priority will be applied when handling RPAS flights.
As a conclusion, the Baseline 1 scenario is again not considered to be a realistic option at medium-size or larger airports. Baseline 2 and SSTR scenarios are seen as equally suited here, too.
Human Performance
From an ATCO’s perspective, all three scenarios (Baseline 1, Baseline 2 and SSTR) show a small to very small impact on mental workload. The highest impact is expected for Baseline 2 (towing operations), as the towing vehicles are additional traffic participants moving on the taxiways and subject to aerodrome control. However, the ATCO workload is not seen as an issue.
Moreover, for the ATCO’s situational awareness, no difficulties are expected for any of the three scenarios. All of them were rated to provide “very often” to “always full” situational awareness about the traffic situation. The highest rating was given for Baseline 1, which is because the traffic is clearly separated here and therefore most orderly.
In terms of trust, again all three scenarios were rated high. The Baseline 2 scenario was rated marginally lower than Baseline 1 and SSTR, because drivers of towing trucks sometimes get only basic trainings on how to communicate with ATC, and how to behave as an aerodrome traffic participant on the maneuvering area.
Summarizing the ATCO’s perspective, no human performance issue is seen for all three scenarios. It is recommended that ATCOs receive an appropriate training, to further reduce the impact of RPAS integration.
From the conventional pilots’ perspective, the situational awareness ratings are not as high as the ATCO ratings, but still positive for all three scenarios. An explanation could be that the demand for situational awareness about the current traffic situation is lower for conventional pilots, especially on controlled aerodromes, compared to ATCOs. The lowest rating was given for the Baseline 1 scenario. Baseline 2 and SSTR are rated almost equal and higher than Baseline 1.
The same results were obtained for the trust index. Again, Baseline 1 got the worst (but still a positive) rating, which leads to the conclusion that the scenario works from a pilot’s perspective, but there are still doubts if this will lead to a smooth flow of air traffic.
From the perspective of the RPAS pilots, the situational awareness was rated the highest for the Baseline 2 scenario, as the ground movement is conducted by the driver of the towing truck and therefore is less complex for the RPAS pilot. The SSTR and Baseline 1 scenarios were rated almost equally high. All three scenarios were rated positively, so no issue could be found here.
The same results were obtained again for the trust index. Here, the ratings are in principle only marginally higher than for situational awareness. Similar to the situational awareness assessment, the SSTR and Baseline 1 scenarios were rated almost equally high. All three scenarios were rated positively for trust, too.