Editorial for Special Issue: Test and Evaluation Methods for Human-Machine Interfaces of Automated Vehicles

Abstract

Today, OEMs and suppliers can rely on commonly agreed and standardized test and evaluation methods for in-vehicle human–machine interfaces (HMIs). These have traditionally focused on the context of manually driven vehicles and put the evaluation of minimizing distraction effects and enhancing usability at their core (e.g., AAM guidelines or NHTSA visual-manual distraction guidelines). However, advances in automated driving systems (ADS) have already begun to change the driver’s role from actively driving the vehicle to monitoring the driving situation and being ready to intervene in partially automated driving (SAE L2). Higher levels of vehicle automation will likely only require the driver to act as a fallback ready user in case of system limits and malfunctions (SAE L3) or could even act without any fallback within their operational design domain (SAE L4). During the same trip, different levels of automation might be available to the driver (e.g., L2 in urban environments, L3 on highways). These developments require new test and evaluation methods for ADS, as available test methods cannot be easily transferred and adapted. The shift towards higher levels of vehicle automation has also moved the discussion towards the interaction between automated and non-automated road users using exterior HMIs. This Special Issue includes theoretical papers a well as empirical studies that deal with these new challenges by proposing new and innovative test methods in the evaluation of ADS HMIs in different areas.

1. Introduction

The human–machine interface (HMI) will play a crucial role in the safe, comfortable and efficient use of automated vehicles. For example, the automated driving system (ADS) HMI should be capable of informing the user about the current mode and minimize confusion about the status of the ADS and the user’s current responsibilities (e.g., whether the ADS is functioning properly, ready for use, unavailable for use or requesting a transition of control from the ADS to the user). While ADS might allow new and more comfortable seating positions and engagement in nondriving-related tasks that were not allowed in manual driving, these might lower the user’s availability for a transfer of control or generate motion sickness. As the driving task is no longer actively fulfilled by the driver, distraction by nondriving-related tasks might turn into controlled engagement by activating activities that prevent fatigue, generating the need to advance assessment methods for nondriving-related tasks. Furthermore, when interacting with other vehicles, ADS might behave differently than manually driven vehicles, which might generate a need for external HMIs or standardized motion patterns for an adequate interaction with non-automated traffic participants. This is only a small proportion of the new challenges for test and evaluation methods of HMIs that arise from the introduction of ADS. The articles of this Special Issue analyze the developments and new challenges by introducing new test methods about the topics outlined above. Among the submissions received, all of which went through a rigorous peer-review process, 21 papers have been selected for publication. The contributions all stem from well-known research institutes and leading practitioners in the field of ADS research. The papers, which will be described in the following, deal with a broad selection of relevant topics such as the evaluation of the relationship of automated vehicles and surrounding non-automated traffic, external as well as interior human–machine interfaces of automated vehicles and the influence of driver state, driver availability and situational factors on control transitions and comfort of automated driving.

• Assessing the relationship of automated vehicles and surrounding non-automated traffic

ADS will very likely be introduced into a mixed traffic environment, which means that some road users will be automated while others will be driven manually. The following papers focus on the impact of automated vehicles on surrounding, non-automated traffic such as pedestrians or cyclists. The first paper “Comparison of Methods to Evaluate the Influence of an Automated Vehicle’s Driving Behavior on Pedestrians: Wizard of Oz, Virtual Reality, and Video” by Fuest, Schmidt and Bengler [1] investigates four different methods regarding the communication between automated vehicles and pedestrians. Hence the same study design in four different settings was used. Two video, one virtual reality, and one Wizard of Oz setup was replicated. An automated vehicle approached from the left, using different driving profiles characterized by changing speed to communicate its intention to let the pedestrians cross the road. Participants were asked to recognize the intention of the automated vehicle and to press a button as soon as they realized its intention.

The second paper “Effects of Marking Automated Vehicles on Human Drivers on Highways” by Fuest, Feierle, Schmidt and Bengler [2] presents a simulation study with different highway scenarios each with and without a marked automated vehicle. Common to all scenarios was that the automated vehicles strictly adhered to German highway regulations, and therefore moved in road traffic somewhat differently to human drivers. After each trial, the participants were asked to rate how appropriate and disturbing the automated vehicle’s driving behavior was. In addition, objective data, such as the time of a lane change and the time headway were measured.

The third paper “Multi-Vehicle Simulation in Urban Automated Driving: Technical Implementation and Added Benefit” by Feierle, Rettenmaier, Zeitlmeir and Bengler [3] investigates the simultaneous interaction between an automated vehicle (AV) and its passenger, and between the same AV and a human driver of another vehicle. For this purpose a multi-vehicle simulation consisting of two driving simulators, one for the AV and one for the manual vehicle was implemented. This paper analyzes the effect of an automation failure, where the AV first communicates to yield the right of way and then changes its strategy and passes through the bottleneck first, despite oncoming traffic. The research questions the study aims to answer are what methods should be used for the implementation of multi-vehicle simulations with one AV, and is there an added benefit of this multi-vehicle simulation compared to single-driver simulator studies?

The next paper focuses on the communication of surrounding traffic conditions to users of automated vehicles. The paper “Feeling Uncertain—Effects of a Vibrotactile Belt that Communicates Vehicle Sensor Uncertainty” by Krüger, Driessen, Wiebel-Herboth, de Winter and Wersing [4] deals with the design and evaluation of a vibrotactile interface that communicates spatiotemporal information about surrounding vehicles and encodes a representation of spatial uncertainty in a novel way. For the measure of subjective understanding and benefit, a questionnaire, ratings and scores were used, for the objective benefit, the minimum time-to-contact as a measure of safety and gaze distributions as an indicator for attention guidance were computed.

• Designing and evaluating external human–machine interfaces (eHMIs)

Automated cars may be equipped with eHMIs for communication with other unequipped road users such as pedestrians. Their potential benefits and drawbacks are discussed in the technical and scientific community, but there are currently no available standards for their implementation. Therefore the first paper “Standardized Test Procedure for External Human-Machine Interfaces of Automated Vehicles”, by Kaß, Schoch, Naujoks, Hergeth, Keinath and Neukum [5] presents a standardized test procedure that enables the effective usability evaluation of eHMIs from the perspective of multiple road users. The paper includes a methodological approach to deduce relevant use cases as well as specific usability requirements that should be fulfilled by an eHMI to be effective, efficient, and satisfying. To prove whether an eHMI meets these requirements, a test protocol for the empirical evaluation of an eHMI with a participant study is demonstrated.

To be effective, any message displayed by an automated vehicle to other road users must satisfy legibility requirements based on the dynamics of the road traffic and the time required by the human to process the respective message. Therefore the second paper “How Much Space Is Required? Effect of Distance, Content, and Color on External Human–Machine Interface Size” by Rettenmaier, Schulze and Bengler [6] examines the size requirements of displayed text or symbols regarding eHMIs for ensuring the legibility of a message. Based on a developed eHMI prototype, the influence of content type on content size to ensure legibility from a constant distance, as well as the influence of content type and content color on the human detection range, was investigated.

The third paper “How Do eHMIs Affect Pedestrians’ Crossing Behavior? A Study Using a Head-Mounted Display Combined with a Motion Suit” by Kooijmann, Happee and de Winter [7] focuses on the investigation of the effects of eHMIs on participants’ crossing behavior. For this purpose, the participants were immersed in a virtual urban environment using a head-mounted display coupled to a motion-tracking suit. The approaching vehicles’ behavior (yielding, or nonyielding) and eHMI type (None, Text or Front Brake Lights) were manipulated and the participants could cross the road whenever they felt safe enough to do so. The study shows that the motion suit allows investigating pedestrian behaviors related to bodily attention and hesitation in the context of interacting with automated vehicles.

The fourth paper “External Human–Machine Interfaces: The Effect of Display Location on Crossing Intentions and Eye Movements” by Eisma, van Bergen, Brake, Hensen, Tempelaar and de Winter [8] addresses the effects of the position of the eHMI on the feeling of safety to cross the street. The eHMI showed “Waiting” combined with a walking symbol 1.2 s before the car started to slow down, or “Driving” while the car continued driving. Participants had to press and hold the spacebar when they felt it was safe to cross. After that, the percentages of spacebar presses and the eye-tracking analyses were evaluated.

The last paper regarding the concept of eHMIs “Efficient Paradigm to Measure Street-Crossing Onset Time of Pedestrians in Video-Based Interactions with Vehicles” by Faas, Mattes, Kao and Baumann [9] introduces a methodology to compare eHMI concepts from a pedestrian’s viewpoint. Therefore a quantifiable concept that allows participants to naturally step off a sidewalk to cross the street was developed. Hidden force-sensitive resistor sensors recorded their crossing onset time (COT) in response to real-life videos of approaching vehicles in an immersive crosswalk simulation environment.

• Evaluating interior HMIs of automated vehicles

As long as vehicles can be driven manually or require manual intervention by their users, the interior HMI will still play a crucial part in their safe and efficient usage. However, guidelines and test methods are only slowly being adapted from those of manual and assisted driving. The next three papers investigate methods regarding the assessments of interior HMIs of automated vehicles. The first one “Usability Evaluation—Advances in Experimental Design in the Context of Automated Driving Human–Machine Interfaces” by Albers, Radlmayr, Löw, Hergeth, Naujoks, Keinath and Bengler [10] aggregates common research methods and findings based on an extensive literature review. These methods and findings are discussed critically, taking into consideration requirements for usability assessments of HMIs in the context of conditional automated driving. The paper concludes with a derivation of recommended study characteristics framing best practice advice for the design of experiments.

The second paper “Checklist for Expert Evaluation of HMIs of Automated Vehicles—Discussions on Its Value and Adaptions of the Method within an Expert Workshop” by Schömig, Wiedemann, Hergeth, Forster, Muttart, Eriksson, Mitropulos-Rundus, Grove, Krems, Keinath, Neukum and Naujoks [11] summarizes the results of a workshop about a checklist method for the evaluation of automated vehicles’ HMIs. Within this workshop, members of the human factors community were brought together to discuss the method and to further promote the development of HMI guidelines and assessment methods for the design of HMIs of automated driving systems (ADS). The results will be used to further improve the checklist method and make the process available to the scientific community.

The paper “Human–Vehicle Integration in the Code of Practice for Automated Driving” by Wolter, Dominioni, Hergeth, Tango, Whitehouse and Naujoks [12] deals with a new Code of Practice for automated driving (CoP-AD) as part of the publicly funded European project L3Pilot. It provides developers with a comprehensive guideline on how to design and test automated driving functions, with a focus on highway driving and parking. This paper focuses on the human factors aspects addressed in the CoP-AD, which includes, inter alia, general human factors-related guidelines, mode awareness, trust, and misuse, driver monitoring together with the topic of controllability and the execution of customer clinics, as well as the training and variability of users.

• Evaluating the influence of driver state, driver availability and situational factors on control transitions and comfort of automated driving

A crucial human factor in the use of automated driving functions is the driver’s state, such as the readiness to take over manual driving, mode awareness, fatigue or motion sickness. The driver’s state can have an impact both on the safety of control transitions as well as the perceived comfort and acceptance of automated driving. The following papers provide empirical studies as well as theoretical analyses and test protocols on this issue.

The first one “Sleep Inertia Countermeasures in Automated Driving: A Concept of Cognitive Stimulation” by Wörle, Kenntner-Mabiala, Metz, Fritzsch, Purucker, Befelein and Prill [13] shows the concept and evaluation of a reactive countermeasure against sleep inertia, which could be useful with regard to dual-mode vehicles that allow both manual and automated driving. The so called “sleep inertia counter-procedure for drivers” (SICD), has been developed with the aim to activate and motivate the driver as well as to measure the driver’s alertness level. The SICD is evaluated in a study with drivers in a driving simulator.

The second paper “Methodological Approach towards Evaluating the Effects of Non-Driving Related Tasks during Partially Automated Driving” by Hollander, Rauh, Naujoks, Hergeth, Krems and Keinath [14] shows the development of a test protocol for systematically evaluating non driving-related tasks’ (NDRT) effects during partially automated driving (PAD). Two generic take-over situations addressing system limits of a given PAD regarding longitudinal and lateral control were implemented to evaluate drivers’ supervisory and take-over capabilities while engaging in different NDRTs (e.g., manual radio tuning task). The test protocol was evaluated and refined across the three studies (two simulator and one test track).

The third paper “Mode Awareness and Automated Driving—What Is It and How Can It Be Measured?” by Kurpiers, Biebl, Mejia Hernandez and Raisch [15] introduces a measurement method to assess mode awareness when using automated vehicles. The background of this study is the different responsibility allocation in different automation modes that requires the driver to always be aware of the currently active system and its limits to ensure a safe drive. For that reason, current research focuses on identifying factors that might promote mode awareness. In the method presented by the authors, the behavior aspect is represented by the relational attention ratio in manual, Level 2 and Level 3 driving as well as the controllability of a system limit in Level 2. The knowledge aspect of mode awareness is operationalized by a questionnaire on the mental model for the automation systems after an initial instruction as well as an extensive enquiry following the driving sequence.

The fourth paper “Engagement in Non-Driving Related Tasks as a Non-Intrusive Measure for Mode Awareness: A Simulator Study” by Forster, Geisel, Hergeth, Naujoks and Keinath [16] describes a driving simulator study, based on the expectation that HMI design and practice with different levels of driving automation influence NDRT engagement. Therefore the participants completed several transitions of control and could engage in an NDRT if they felt safe and comfortable to do so. The NDRT was the Surrogate Reference Task (SuRT) as a representative of a wide range of visual-manual NDRTs. Engagement (i.e., number of inputs on the NDRT interface) was assessed at the onset of a respective episode of automated driving (i.e., after transition) and during ongoing automation (i.e., before subsequent transition).

The fifth paper “Methodological Considerations Concerning Motion Sickness Investigations during Automated Driving” by Mühlbacher, Tomzig, Reinmüller and Rittger [17] discusses methodological aspects for investigating motion sickness in the context of automated driving including measurement tools, test environments, sample, and ethical restrictions. Additionally, methodological considerations guided by different underlying research questions and hypotheses are provided. Selected results from the authors’ own studies concerning motion sickness during automated driving which were conducted in a motion-based driving simulation and a real vehicle are used to support the discussion.

The sixth paper “Supporting Drivers of Partially Automated Cars through an Adaptive Digital In-Car Tutor” by Boelhouwer, van den Beukel, van der Voort, Verwey and Martens [18] investigates the effects of a Digital In-Car Tutor (DIT) prototype on appropriate automation use and take-over quality during a driving simulator study. A DIT is proposed to support drivers in learning about, and trying out, their car automation during regular drives. Participants needed to use the automation when they thought that it was safe, and turn it off if they did not. The control group read an information brochure before driving, while the experiment group received the DIT during the first driving session.

The seventh paper “The Impact of Situational Complexity and Familiarity on Takeover Quality in Uncritical Highly Automated Driving Scenarios” by Scharfe, Zeeb and Russwinkel [19] differentiates between the objective complexity and the subjectively perceived complexity of a traffic situation. The aim of the present study was to examine the impact of objective complexity and familiarity on the subjectively perceived complexity and the resulting takeover quality. In a driving simulator study, participants were requested to take over vehicle control in an uncritical situation. Familiarity and objective complexity were varied by the number of surrounding vehicles and scenario repetitions. Subjective complexity was measured using the NASA-TLX; the takeover quality was gathered using the take-over controllability rating (TOC-Rating).

The eighth paper “Repeated Usage of an L3 Motorway Chauffeur: Change of Evaluation and Usage” by Metz, Wörle, Hanig, Schmitt and Lutz [20] investigates changes in drivers’ evaluation, in function usage and in drivers’ reactions to take-over situations with repeated usage of automated driving functions. Therefore, drivers used a level 3 (L3) automated driving function for motorways during six experimental sessions in a driving simulator study. They were free to activate/deactivate the system as they liked and to spend driving time on self-chosen side tasks. After that the experienced trust and safety, the time spent on side tasks, attention directed to the road and behavioral adaptation was analyzed.

The last paper “Measuring Drivers’ Physiological Response to Different Vehicle Controllers in Highly Automated Driving (HAD): Opportunities for Establishing Real-Time Values of Driver Discomfort” by Radhakrishnan, Merat, Louw, Lenné, Romano, Paschalidis, Hajiseyedjavadi, Wei and Boer [21] investigates how driver discomfort was influenced by different types of automated vehicle (AV) controllers, compared to manual driving, and whether this response changed in different road environments, using heart-rate variability and electrodermal activity. The drivers were subjected to manual driving and four AV controllers: two modelled to depict “human-like” driving behavior, one conventional lane-keeping assist controller, and a replay of their own manual drive.

2. Conclusions

This Special Issue brings together research from well-known human factors experts in the field of automated driving. The impressive number of published papers covering a wide range of research topics on test and evaluation methods for automated vehicles HMIs shows the high relevance of this Special Issue. The Special Issue has thus contributed to the promotion and dissemination of these methods within the scientific community and will hopefully stimulate further research on these topics.