Evaluation of Autonomous Driving Safety by Operational Design Domains (ODD) in Mixed Traffic

Abstract

This study derived effective driving behavior indicators to assess the driving safety of autonomous vehicles (AV). A variety of operation design domains (ODD) in urban road networks, which include intersections, illegal parking, bus stop, bicycle lanes, and pedestrian crossings, were taken into consideration in traffic simulation analyses. Both longitudinal and interaction driving indicators were investigated to identify the driving performance of AVs in terms of traffic safety in mixed traffic stream based on simulation experiments. As a result of identifying the appropriate evaluation indicator, time-varying stochastic volatility (VF) headway time was selected as a representative evaluation indicator for left turn and straight through signalized intersections among ODDs related to intersection types. VF headway time is suitable for evaluating driving ability by measuring the variation in driving safety in terms of interaction with the leading vehicle. In addition to ODDs associated with intersection type, U-turns, additional lane segments, illegal parking, bus stops, and merging lane have common characteristics that increase the likelihood of interactions with neighboring vehicles. The VF headway time for these ODDs was derived as driving safety in terms of interaction between vehicles. The results of this study would be valuable in establishing a guideline for driving performance evaluation of AVs. The study found that unsignalized left turns, signalized right turns, and roundabouts had the highest risk scores of 0.554, 0.525, and 0.501, respectively, indicating these as the most vulnerable ODDs for AVs. Additionally, intersection and mid-block crosswalks, as well as bicycle lanes, showed high risk scores due to frequent interactions with pedestrians and cyclists. These areas are particularly risky because they involve unpredictable movements from non-vehicular road users, which require AVs to make rapid adjustments in speed and trajectory. These findings provide a foundation for improving AV algorithms to enhance safety and establishing objective criteria for AV policy-making.

1. Introduction

The National Traffic Safety Administration (NHTSA) reports that 94% of vehicle accidents are caused by human error. Additionally, alcohol is involved in about 30% fatalities. Autonomous vehicles (AVs) have the potential to effectively reduce accidents caused by human factors [1]. In other words, the implementation of AVs has the potential to improve traffic safety fundamentally by eliminating traffic accidents caused by human error. However, AVs with imperfect autonomous driving systems (ADS) are limited in their ability to improve traffic safety. In addition, the roadway will continue to have mixed traffic with vehicles of different ADS levels for a considerable period [2,3,4,5,6,7]. Several literatures have shown that unstable traffic flows can occur in mixed traffic environments [8,9]. In real-world road environments, various static and dynamic factors present in the environment and their interactions with vehicles can amplify unstable traffic flow [10]. Consequently, this study defines ODD as an environmental domain that influences driving stability. Numerous studies have concluded that unstable traffic flow can arise from differences in cognitive and judgment abilities between AVs and non-AVs, as well as from driver behavior characteristics influenced by ODD [11,12,13].

The California AV-related accident report shows that the number of accidents by road type is higher on urban roads than on freeways [14]. The reason for this result is that compared to roads such as freeways and expressways, urban roads are more likely to have a combination of various ODDs that reduce the driving safety of AVs [15]. Thus, it is important to analyze the ODDs that affect the driving safety of AVs and to determine the extent of the impact. However, several studies have been published on the performance of ADS-mounted sensors, but research analyzing the level of impact on the driving safety of AVs by ODD is still insufficient [16,17,18,19]. In particular, since the factors affecting AVs differ according to the characteristics of ODDs, appropriate evaluation indicators should be selected to evaluate the driving safety vulnerability of AVs by ODD, ranking the ODDs with vulnerable driving safety is essential.

Current research has mainly focused on the capability of AV sensors under specific conditions, without fully reflecting the variety of situations AVs will face in the real world, including complex road geometries and transportation infrastructure. Therefore, the objective of this study is to evaluate the driving safety of AVs in mixed traffic conditions by analyzing the impact of various ODDs and vehicle behavioral characteristics, and to establish a ranking of ODDs based on their vulnerability to compromising driving safety. This study suggests a methodology for determining evaluation indicators to evaluate the driving safety of AVs in mixed traffic conditions by ODD, considering the characteristics of ODDs that affect AVs and the behavioral characteristics of vehicles in ODDs. For evaluating the driving safety of AVs, this study defines promising indicators as those deemed suitable for assessing the performance and safety of AVs within specific ODD environments. The findings of this study can contribute to the evaluation of AV safety and the enhancement of ADS algorithms, particularly for ODDs with reduced driving safety. These results can serve as a foundation for determining AV operational permissions in specific ODDs, providing objective criteria for policy-making regarding the phased introduction of AVs. Furthermore, this research is expected to contribute to the development of performance verification standards for various ODDs and scenarios, essential for comprehensively assessing AV driving capabilities.

This research is organized as follows Section 2 reviews the research that has been conducted to evaluate the driving safety of AVs and describes the differences of this research. In Section 3, a methodology was described to rank ODDs that are vulnerable to driving safety by implementing the behavior of AVs in a simulation environment through real-world data and deriving promising indicators for each ODD through principal component analysis (PCA). In Section 4, the results of the promising indicators and vulnerability ranking analysis for each ODD are presented, and in Section 5, the findings of this study are summarized, and applications and limitations are presented.

2. Literature Review

This study intends to derive a ranking of vulnerable ODDs by evaluating the driving safety of AVs by ODD. Therefore, literature on the analysis of driving safety of AVs according to ODDs is reviewed and the differences and implications of this study are described.

The fact that AVs are driven by ADS has led to a considerable amount of research on sensor and vehicle behavior control performance. The California department of motor vehicle (CA DMV) in the U.S. is providing open-source data on the causes of failures of AVs on real world, as well as accident rates and accidents per accumulated mileage. Using open-source data, a model for estimating the minimum mileage required to improve the reliability of AVs and a deep neural network were developed to analyze the causes of defects in AVs [20]. In addition, several studies have been conducted on reaction times for take-over [21,22,23]. Leledakis et al. (2021) estimated the effectiveness of collision avoidance technologies to assess the crashworthiness of AVs [24]. Accident avoidance rates of advanced emergency braking systems (AEBS) in AVs were analyzed by performing model-in-the-loop (MIL) simulations based on actual traffic accident data. According to the analysis, 61–92% of accidents in discontinuous flows and 56% in continuous flows could have been prevented by the AEBS capability. Hou (2023) analyzed the traffic efficiency and safety of roads with a mixed traffic of connected and autonomous vehicles (CAVs) and human-driven vehicles (HDVs) under adverse weather conditions [25]. The weather conditions were categorized into clear, rainy, and snowy, then analyzing the changes in driving safety according to market penetration rate (MPR). The findings showed that as MPR increases, traffic volume increases in all three weather conditions. In particular, the MPR of 100% was found to be collision-free in all weather conditions. Wang et al. (2022) proposed a Cooperative Adaptive Cruise Control Lane Change (CACCLC) controller for CACC platoons to change lanes in dense traffic [26]. The developed CACCLC exhibited marked enhancements in lane-change competence and efficiency relative to conventional methods, with improvements of 29–43% in competence and 88–92% in efficiency across diverse road types. Hu et al. (2024) proposed an optimal-control-based parking motion planner for automated parking assist systems [27]. The study demonstrated improved parking performance, with the proposed planner enhancing parking success rates and reducing completion time compared to conventional methods across various parking scenarios. Wang et al. (2023) proposed a trajectory planning method for automatic lane-changing based on a dynamic safety domain model [28]. The approach was validated through both MATLAB simulations and real-world vehicle tests. The study showed that the proposed method improved the safety and efficiency of lane-changing compared to traditional static planning approaches. Hu et al. (2024) proposed a next-generation simulation platform for truck platooning evaluation in an interactive traffic environment [29]. The study focused on meeting the evaluation needs of various stakeholders. The results proved the platform’s ability to evaluate truck platoon performance and assess traffic impact. Liu et al. (2023) developed a comprehensive approach to enhance the agility and path tracking capabilities of autonomous trucks equipped with dual 4WIS4WID modular chassis [30]. According to the results, the proposed communication framework, reference path generation strategy and adaptive model predictive control (AMPC) were effective in achieving accurate path tracking over a range of speeds and path types.

Regarding the safety aspect, studies are divided into those that analyze AV-side safety in mixed traffic environments and those that analyze MV-side safety affected by AVs. Niroumand et al. (2022), conducting a study to analyze the driving safety of AVs, analyzed the impact of AV driving behavior on intersection performance and safety in a mixed AV and MV environment [31]. Data collected from 12 AVs were used to develop a driving behavior model of AVs to analyze changes in intersection safety due to driving behavior. The results of the analysis showed that the performance and safety of the intersection decreases when the AV performs a sudden stop at the intersection entry point. In addition, the time-to-collision (TTC) decreases, and the acceleration variation is higher when the AV exhibits aggressive driving behavior. Sinha et al. (2020) used VISSIM to study the impact of CAVs on four-way intersections, dividing the analysis into CAV-MV pairs and CAV-CAV pairs where the leading vehicle is a CAV, and MV-CAV pairs and MV-MV pairs where the leading vehicle is an MV [32]. At the signalized intersection, the conflict rate increases until the MPR of the CAV reaches 60%, after which it decreases. Hu et al. (2024) developed a Safety-Aware Human-Lead Cooperative Adaptive Cruise Control (SDHL-CACC) controller based on Stochastic Model Predictive Control (SMPC) [33]. The proposed approach was evaluated through PreScan and Simulink coupled simulations using real vehicle trajectory data. The study proved that the SDHL-CACC controller improved perceived safety in oscillating traffic by 19.17% and actual safety against hard braking by 7.76% compared to conventional methods, while maintaining string stability. Li et al. (2023) developed a potential-field-driven model predictive controller for shared control in AV [34]. The study focused on driving safety and obstacle avoidance, evaluating the controller’s performance in various scenarios. Results demonstrated improved safety and reduced driver-automation conflicts compared to conventional approaches.

Lee et al. (2023), evaluating the driving safety of MVs, utilized a multi-agent driving simulator (MADS) based on the SCANeRTM STUDIO program to analyze the driving safety of AV pairs (AV-AV), mixed pairs (AV-MV, MV-AV), and MV pairs (MV-MV) on interrupted flow [35]. The findings of the analysis revealed that lateral driving safety is weak in the longer curve sections, roundabouts, and u-turns. Longitudinal driving safety was analyzed to be poorer at roundabouts in the case of AV pairs and mixed pairs, and at u-turns in the case of MV pairs. Jung et al. (2023) used MADS to analyze the driving safety of four leading- following vehicle pairs in an uninterrupted flow [36]. Lateral safety vulnerable sections were analyzed as right-curve curves and downhill sections, and longitudinal safety vulnerable sections were analyzed as short curve lengths. Wen et al. (2022) used the waymo open dataset to analyze the driving behavior variation of MV drivers in leading and following situations [37]. They found that MV drivers have lower driving variation, shorter travel time, and higher TTC when following an AV. This means that the existence of AVs in traffic increases safety because speed deviations are reduced. Zhao et al. (2020) used an AV data set collected at the Weishui campus CAV test track with a total length of 2.4 km to analyze the differences in longitudinal driving behavior characteristics between human vehicle (HV) drivers following AVs and HV drivers following AVs [38]. The subjects were divided into two groups: drivers who trusted the ADS and drivers who did not. The evaluation indicators utilized are speed, spacing, headway, and standard deviation of speed. The results revealed that drivers who trusted ADS reduced spacing, reduced travel time, and avoided cutting in front of other vehicles when following an AV. Conversely, the analysis showed that drivers who did not trust ADS increased their spacing when following an AV. This means that the driving behavior of the MV driver of the following vehicle is affected by the driver’s subjective trust in the ADS.

Recently, a study was conducted to analyze the conflicts between AVs and pedestrians and cyclists on urban roads to define ODD. Alozi and Hussein (2023) evaluated the safety of interactions between AVs and road users [39]. Utilizing AV sensor data collected in Canada, the United States, and Singapore, AV collisions between AVs and road users were quantified using post-encroachment time (PET). Analysis of 1255 crashes involving pedestrians and 434 crashes involving cyclists found that AVs performing right turns and pedestrian conflicts are most hazardous. Analysis revealed that bicyclists are at the most increased hazard in conflicts with AV left turns. Kim et al. (2020) defined ODD areas that reflect the actual situation of Korean roads based on the ODD areas proposed by NHTSA and conducted an evaluation of the operation design area, including the feasibility of autonomous driving on urban roads [40]. Standards were set for the demonstration test of autonomous driving shuttles in Korea by determining whether AVs can be driven according to the infrastructure support of each node, traffic volume, weather, etc.

Several studies have suggested and analyzed various methods to evaluate the performance of AVs, however, the previous studies have focused on the performance of the sensors and vehicle behavior control equipped with ADS. Verification of the performance of ADS under various ODD conditions is important, however, it is also important to verify the safety of the traffic flow aspect by analyzing the impact on mixed traffic depending on the perception and performance of AVs. In particular, driving safety indicators for AVs should be derived by considering the safety of mixed traffic, affected by various ODD characteristics in the case of interrupted flow, rather than uninterrupted flow. In addition, the studies analyzing the driving safety aspects of AVs do not provide validation or validity of the indicators used in the analysis. Therefore, this study developed a methodology to verify the driving safety vulnerabilities of AVs in a traffic simulation environment with various ODD. Promising indicators suitable for evaluating the driving safety vulnerability of AVs by ODD were identified. The significance is that the ranking of ODD with vulnerable driving safety of AVs was derived by comparing the risk between ODD.

3. Methodology

This study derived promising indicators according to specific ODD characteristics through PCA to evaluate the driving safety of AVs and presented a rank of ODDs with vulnerable driving safety. The analysis was divided into three steps as shown in Figure 1. Step 1 is to build a simulation environment based on real roads. The Seoul autonomous mobility testbed, where AVs are driving in the real world, was set as a section to be analyzed, and a road network was constructed in the VISSIM environment, a traffic simulation. In addition, the traffic volume and vehicle types were based on traffic data provided by View-T, which is a national traffic DB-based traffic measurement system managed by the Korean Ministry of Land, Infrastructure, and Transport. Automated vehicle data (AVD), which is the real-world driving data, was used to implement the driving behavior of AVs and MVs in the analysis zone. The driving mode information included in the AVD is divided into autonomous driving mode (AD mode) and manual driving mode (MD mode). The velocity, acceleration, and deceleration were determined, and the parameters were calibrated to implement the behavior of each driving mode in the VISSIM environment. Step 2 was the selection of evaluation indicators and driving safety analysis, and the evaluation indicators utilized 7 longitudinal evaluation indicators of the target vehicle and 9 evaluation indicators in the interaction between vehicles. In step 3, the methodology for deriving promising indicators for each ODD was described by utilizing PCA, which is used for identifying key factors, reducing and structuring variables. In this study, PCA was used to identify the indicators that can most effectively evaluate the driving safety of AVs among the evaluation indicators applicable to each ODD. Furthermore, the results of the promising indicators derived for each ODD were normalized and a risk score was generated to determine the ranking of ODDs with vulnerable driving safety. Furthermore, the results of the promising indicators derived for each ODD were normalized and a risk score was calculated to determine the ranking of ODDs with vulnerable driving safety.

3.1. Methodology of PCA

The driving safety vulnerability of an AV in an ODD is determined by the characteristics of the ODD and the AV’s driving behavior. Various evaluation indicators can reflect these characteristics. PCA is used for variable reduction and structuring, which is utilized to more easily identify important information in complex data sets, identification of the most important factors within the data to derive patterns or relationships. In this study, PCA was utilized to select appropriate evaluation indicators that reflect the unique characteristics of each ODD, acknowledging that different ODDs require specific safety assessment indicators. Therefore, this study devised a ranking method for ODDs with vulnerable driving safety by utilizing PCA to select and quantify evaluation indicators that can reflect the characteristics of each ODD.

The algorithm for performing PCA and the process of deriving promising indicators is illustrated in Figure 2. The kaiser-meyer-olkin (KMO) measure and Bartlett’s test of sphericity were checked to verify that the results of the indicator-specific analyses calculated in the interval where a specific ODD A was present were suitable for performing PCA. PCA was considered feasible if the KMO measure was greater than 0.6 and the Bartlett’s test of sphericity p-value was less than 0.05 [41,42,43,44]. In PCA, commonality refers to the variance shared by the observed and latent variables. A commonality above 0.5 can be assumed to be a significant variable [45]. Therefore, driving safety indicators with a commonality value of less than 0.5 were removed from the analysis. Various indicators were reduced and structured into core indicators for ODD evaluation, and the indicators included in the principal components with the highest explained total variance were identified as promising indicators.

3.2. Evaluating Autonomous Driving Safety by ODDs

The promising indicators and ranking of vulnerable ODDs were derived for each ODD to identify ODDs that decrease driving safety in terms of traffic safety in mixed traffic conditions. The indicators identified as promising indicators by ODD were normalized using the expression shown in Equation (1), the risk score was derived as the average of the normalized promising indicator values as shown in Equation (2). The higher the value of risk score, the higher the risk ranking of ODDs in terms of traffic safety.

X_n = (x − x_min)/(x_max − x_min)

(1)

X_n means normalized value of evaluation indicators n. x is value of evaluation indicator. x_max is maximum value of evaluation indicator and x_min is minimum value of evaluation indicator.

Risk Score_i = (x₁ⁱ+ x₂ⁱ +⋯+ x_nⁱ)/n

(2)

x_nⁱ is normalized value of promising evaluation indicator n for ODD_i.

3.3. Implementing a Real-World Based Simulation Environment

The analysis area was set as the Seoul autonomous mobility testbed. The driving behavior, road network, traffic volume, and vehicle types of AVs and MVs were implemented to establish the VISSIM environment. The behavior of AD and MD mode were implemented using AVD to analyze driving behavior by driving mode. The results were used to optimize the behavior parameters and traffic speed distribution. AVD were collected from a total of five AVs between 10 February 2022, and 31 October 2022, as presented in

Table 1. Characteristics of the AVD.

Route	Number of AVs	Operating Time	Number of AD Mode Data		Number of MD Mode Data		Total
A	3	Mon.~Sat. 09:00~16:00 (Break time 12:00~13:30)	2,119,706		2,047,418		4,167,124
B	2	Mon.~Fri. 09:30~17:00 (Break time 12:00~13:30)	1,119,177		957,571		2,076,748
Total			3,238,883		3,004,989		6,243,872
AVD sample
Terminal_id			9272			6075
gps_dt [yyyymmddhhmmss]			20220210103000			20220210093001
Latitude			37.57634458			37.57647618
Longitude			126.8938359			126.8989613
Speed [km/h]			30.7			35.4
Driving_mode [1: MD mode/2: AD mode]			2			1

. Analysis of the driving behavior in the intersection affected area and non-affected area by driving mode adjusted the distribution parameters as shown in

Table 2. Parameter adjustment of each driving mode.

AVD-Based Parameter Adjustment for Each Driving Mode
Parameter Adjustment for Each Driving Mode Based on Existing Research Review
Division	Parameter		Value
			MD Mode	AD Mode
Following	Maximum look ahead distance	m	250	300
	Number of interaction objects	-	2	10
	Number of interaction vehicles	-	1	8
Car following model	Gap time distribution (CC1)	s	0.9	0.6
	Threshold for entering “Following” (CC3)	s	8.00	6.00
	Acceleration from standstill (CC8)	m/s2	3.50	4.00
	Acceleration at 80 km/h (CC9)	m/s2	1.50	2.00
Signal control	Reaction after end of green	-	One decision	One decision

. The adjusted parameters correspond to the desired speed distribution, desired acceleration functions, and desired deceleration functions. The intersection affect area is set to 30 m upstream from the point where the stop line is located according to the intersection affect area calculation criteria. The rest of the intersection was set as the intersection non-affect zones. In addition, the CoEXist parameters included in VISSIM were used for parameters where it was not possible to implement vehicle behavior based on AVD. Traffic volumes were obtained from a real-time traffic monitoring system during peak hours. The main road traffic was set to 1180 vph and the minor road traffic was calibrated by adjusting the left and right turn ratio. The driving behavior of AVs was implemented in the simulation to replicate the operational characteristics of real-world AVs. All vehicle types are set to passenger vehicles except for city buses for MVs because there is almost no freight vehicle traffic in the analysis area. Bus arrival interval was implemented in the simulation environment by investigating the arrival time interval of buses at each stop in the real world. The average bus arrival interval is 176 s.

The road network was constructed to serve as the Sangam AV testbed. The two routes measure 5.3 km and 4.0 km in length. For this study, the analysis area was segmented into 30-m units, resulting in a total of 219 sections. The urban road network ODDs selected in this study were carefully chosen based on a thorough review of Korea’s Road Traffic Act, existing literature, consultations with numerous traffic experts, and analysis of AV accident data from California, USA. This study aimed to address critical scenarios where AVs could encounter hazardous situations in urban environments. Furthermore, ODDs potentially affecting AV driving safety were selected for analysis based on road geometry and facility characteristics implementable in a simulation environment. The study focused on the most common ODD scenarios to ensure general applicability and relevance, avoiding atypical or rare configurations. A total of 16 ODDs were identified through field surveys, including straight, left, and right turns at both signalized and unsignalized intersections. The spatial distribution of these ODDs within the road network is illustrated in Figure 3. Segments with illegal parking were defined as those where parking violations were observed in the same location for more than 80% of the time across 10 field surveys. In the simulation environment, these segments were modeled with parking lots, and the ‘duration for the parking lot’ parameter was adjusted to replicate illegal parking conditions. Pedestrian and bicycle traffic volumes were collected through one-hour field surveys, with observed flows of 80 bicycles per hour and 180 pedestrians per hour. The behavior of pedestrians and cyclists in the simulation was implemented using the default parameters provided by VISSIM for adult male and female walking speeds and bicycle parameters.

3.4. Identify Indicators and Evaluate Driving Safety

The simulation was run for 2 h, excluding 30 min for warm-up time to sufficiently distribute the vehicles on the road network. In addition, the simulation data was collected 10 times by applying different random seeds of the same scenario to increase the reliability of the simulation results. The driving safety of AVs was evaluated using 7 longitudinal indicators and 9 interaction indicators. The formulas for each indicator are summarized in

Table 3. Lists of Driving Safety Indicators.

No	Variable	Indicator		Equation
1	STD_spd.	Standard deviation	$\mathrm{Speed} \left(v\right)$	$\sigma \left(x\right)=\sqrt{{\displaystyle \frac{\sum _t^T(x_t-\overline{x}{)}^2}{n}}}$
2	STD_acc.		$\mathrm{Acceleration} \left(a\right)$	$x$	Measurement	$T$	Analysis time unit
3	STD_spc.		$\mathrm{Spacing} \left(s\right)$	$x_t$	$x$$\mathrm{at}\mathrm{time}t$	$t$	Sampling interval
4	STD_hdwy.		$\mathrm{Headway} \left(h\right)$	$\overline{x}$	$\mathrm{Average}x$	$n$	Number of data
5	VF_spd.	Time-varying stochastic volatility	$\mathrm{Speed} \left(v\right)$	${VF}_x=\sqrt{{\displaystyle \frac{\sum _t^T(r_t-\overline{r}{)}^2}{n-1}}}$
6	VF_acc.		$\mathrm{Acceleration} \left(a\right)$	$r_t=\ln \left({\displaystyle \frac{x_t}{x_{t-1}}}\right)\times 100\mathrm{\%}$
7	VF_jerk		$\mathrm{Jerk} \left(j\right)$	$x$	Measurement	$T$	Analysis time unit
8	VF_spc.		$\mathrm{Spacing} \left(s\right)$	$x_t$	$x$$\mathrm{at}\mathrm{time}t$	$t$	Sampling interval
9	VF_hdwy.		$\mathrm{Headway} \left(h\right)$	$\overline{r}$	$\mathrm{Average}r$	$n$	Number of data
				$r_t$	$\mathrm{Relative}\mathrm{change}\mathrm{at}\mathrm{time}t\left(\mathrm{\%}\right)$
10	Avg. DRAC	Deceleration rate to avoid a crash		${DRAC}_{FV,t+1}={\displaystyle \frac{{(v_t^{FV}-v_t^{LV})}^2}{\left(P_t^{LV}-P_t^{FV}\right)-L_{LV}}}$
				$v_t^{FV}$	$\mathrm{Speed}\mathrm{of}\mathrm{the}\mathrm{following}\mathrm{vehicle}\mathrm{at}\mathrm{time}t$
				$v_t^{LV}$	$\mathrm{Speed}\mathrm{of}\mathrm{the}\mathrm{leading}\mathrm{vehicle}\mathrm{at}\mathrm{time}t$
				$P_{FV,t}$	$\mathrm{Position}\mathrm{of}\mathrm{the}\mathrm{following}\mathrm{vehicle}\mathrm{at}\mathrm{time}t$
				$P_{LV,t}$	$\mathrm{Position}\mathrm{of}\mathrm{the}\mathrm{leading}\mathrm{vehicle}\mathrm{at}\mathrm{time}t$
				$L_{LV}$	Length of the leading vehicle
11	NC_SDI	Number of conflicts by SDI		$d_p^{LV}-d_p^{FV}IfSDI<0,then conflict$
				${NC}_{SDI}=\sum {conflict}_{SDI}$
				$d_p^{LV}$	$\mathrm{Stop}\mathrm{distance}\mathrm{of}\mathrm{the}\mathrm{leading}\mathrm{vehicle}\mathrm{at}\mathrm{point}p$
				$d_p^{FV}$	$\mathrm{Stop}\mathrm{distance}\mathrm{of}\mathrm{the}\mathrm{following}\mathrm{vehicle}\mathrm{at}\mathrm{point}p$
12	NC_TTC	Number of conflicts by TTC		${\displaystyle \frac{s_t}{v_t^{LV}-v_t^{FV}}}IfTTC<1.5 \mathrm{s},then conflict$
				${NC}_{TTC}=\sum {conflict}_{TTC}$
				$s_t$	$\mathrm{Spacing}\mathrm{between}\mathrm{vehicles}\mathrm{at}\mathrm{time}t$
13	NC_DRAC	Number of conflicts by DRAC		$IfDRAC<3.35 \mathrm{m}/{\mathrm{s}}^2,thenconflict$
				${NC}_{DRAC}=\sum {conflict}_{DRAC}$
14	NC_RDE	Number of conflicts by rapid deceleration events		$IfRDE>-7.35 \mathrm{m}/{\mathrm{s}}^2,then conflict$
				${NC}_{RDE}=\sum {conflict}_{RDE}$
15	CPI	Crash potential index		$CPI={\displaystyle \frac{\sum _{t=0}^N\mathrm{P}\mathrm{r}(MADR<{DRAC}_t)\times t\times b}{T}}$
				$MADR$	Maximum available deceleration rate
				$b$	Binary state variable (0: non-interaction/1: interaction)
16	P2P jerk	$\mathrm{Peak}\mathrm{to}\mathrm{peak}\mathrm{jerk}\left(j\right)$		$P2Pjerk=Max\left(j\right)-Min\left(j\right)$

. The driving safety evaluation indicators in terms of longitudinal and interaction of vehicles were commonly selected as standard deviation and time-varying stochastic volatility (VF), which can quantitatively estimate changes in driving behavior. VF is capable of quantitatively analyzing irregular changes in the indicators over time and has been reported to be effectively used for driving behavior analysis [46,47].

The vehicle interaction-based driving safety indicators utilized in addition to standard deviation and VF are TTC, stopping distance index (SDI), deceleration rate to avoid a crash (DRAC), and crash potential index (CPI), which are surrogate safety measure (SSM) based indicators that can quantitatively indicate the likelihood of a rear-end collision. The TTC considers a serious collision to have occurred if the remaining time before a collision occurs is less than 1.5 s if both vehicles are driving in the same direction and at the same speed as they are currently driving [48]. SDI is a conflict measurement indicator based on the difference between the minimum stopping distance of the leading and following vehicles. If the stopping distance of the trailing vehicle is longer than the leading vehicle, it can be estimated that a conflict has occurred [49]. The DRAC is defined as a conflict if the collision avoidance deceleration exceeds 3.35 m/s² when the following vehicle recognizes the hazard ahead and begins to slow down [50]. In addition, CPI is the probability that the DRAC of the following vehicle is greater than the vehicles specific maximum deceleration in a vehicle following situation [51]. CPI’s effectiveness in identifying road hazard points and segments in C-ITS environments has been validated [52,53].

The longitudinal indicators utilized in addition to standard deviation and VF are rapid deceleration events (RDE) and peak to peak jerk (P2P jerk) [54,55]. Therefore, the longitudinal indicators utilized in this study were velocity standard deviation, acceleration standard deviation, peak-to-peak jerk, RDE, and VF-based velocity, acceleration, and jerk. Longitudinal indicators can be interpreted as increasing accident probability with higher values [56]. The indicators based on vehicle interaction were selected as spacing standard deviation, headway standard deviation, SDI conflicts, TTC conflicts, DRAC conflicts, average DRAC, CPI, VF-based spacing, and headway.

4. Results

This study derived promising indicators and driving safety vulnerability rankings for each ODD to analyze the level of impact of ODDs on AVs in terms of driving safety. The results of the promising indicators for each ODD derived from the PCA were categorized into intersections-related ODDs, ODDs with interactions between vehicles, and ODDs with interactions with non-vehicular road users.

The results of ODD’s analysis for intersection types and ODD’s risk ranking based on risk score derived using promising indicators are shown in

Table 4. The results of the intersection type ODD.

Total Variance Explained			Rotated Component Matrix		Component Score	Normalized Values	Risk Score
ODD Name	Total	Variance (%)	Indicator	Component Matrix Coefficient
Unsignalized intersection (left turn)	3.11	31.15	STD_spd.	0.96	0.32	0.552	0.554
			STD_acc.	0.92	0.31	0.574
			VF_spd.	0.92	0.30	0.376
			VF_jerk	0.67	0.21	0.714
Signalized intersection (right turn)	3.73	31.08	STD_spd.	0.87	0.29	0.629	0.525
			VF_jerk	0.86	0.24	0.760
			VF_spd.	0.81	0.17	0.448
			STD_acc.	0.77	0.18	0.509
			STD_spc.	0.77	0.32	0.281
Roundabout	4.16	37.77	P2P jerk	0.94	0.27	0.446	0.501
			VF_spd.	0.91	0.23	0.415
			STD_acc.	0.89	0.20	0.469
			NC_RDE	0.85	0.29	0.581
			STD_spd.	0.80	0.15	0.541
Unsignalized intersection (right turn)	3.51	38.99	STD_acc.	0.95	0.27	0.469	0.485
			STD_spd.	0.91	0.26	0.291
			VF_jerk	0.88	0.25	0.543
			VF_spd.	0.75	0.23	0.562
			VF_acc.	0.63	0.16	0.562
Unsignalized intersection (through)	3.98	38.44	P2P jerk	0.96	0.26	0.288	0.372
			VF_spd.	0.93	0.24	0.220
			STD_acc.	0.89	0.22	0.490
			STD_spd.	0.82	0.18	0.463
			NC_RDE	0.71	0.24	0.400
Signalized intersection (left turn)	3.43	34.28	VF_hdwy.	0.97	0.28	0.370	0.369
			VF_spc.	0.96	0.27	0.383
			CPI	0.96	0.27	0.579
			Avg. DRAC	0.76	0.24	0.143
Signalized intersection (through)	2.90	28.74	VF_hdwy.	0.97	0.52	0.148	0.135
			VF_spc.	0.96	0.51	0.122

. Promising indicators for left turns at unsignalized intersections were STD spd., STD acc., VF spd., and VF acc. The driver’s performance is important because there is no signal control at unsignalized intersections. In particular, the scenario of performing a left turn has a relatively large number of conflict points, which increases the possibility of conflicts. Speed and acceleration-related indicators were derived as promising indicators because it is important for AVs to accurately perceive intersecting vehicles and to control stable speed and acceleration without impact on following and opposing vehicles. The promising indicators for right turns at unsignalized intersections were analyzed, including STD spd., STD acc., VF spd., VF acc., and VF jerk. Similar to left turns, accurate situation assessment by AVs is required. Stable speed and acceleration control is necessary to minimize the impact on the leading and following vehicles by showing predictable driving to other vehicles. ODDs corresponding to going straight through unsignalized intersections, STD spd., STD acc., P2P jerk, VF spd., and NC RDE were found to be promising indicators. In the case of going straight, drivers pass through the intersection at a relatively high speed compared to left and right turns. Accident avoidance requires the ability to judge the priority of the road being traveled and to respond quickly to sudden actions of MVs. The risk scores for left turns, right turns, and straight turns at unsignalized intersections were calculated to be 0.554, 0.485, and 0.372, respectively. The risk score of a straight right turn, which has relatively few conflicting points and priority, is lower than the average risk score of 0.404 for all ODDs, but the risk scores of left and right turns at unsignalized intersections are higher. In particular, the risk score of a left turn at unsignalized intersections is more than three times higher than the risk score of a straight right turn at the lowest risk signalized intersection, which is 0.135.

The ODDs associated with unsignalized intersections are right turns at signalized intersections and roundabouts. The promising indicators for right turns at signalized intersections were STD spd., STD acc., VF spd., VF jerk, and STD spc. Right turns at signalized intersections involve complex interactions with other vehicles, as well as a relatively increased potential for conflicts with pedestrians. AVs require appropriate deceleration and acceleration while maintaining a safe distance. For roundabouts, STD spd., STD acc., P2P jerk, VF spd., and NC RDE were found to be promising indicators. In complex traffic flow situations with vehicles entering and turning from multiple directions, AVs require continuous decision-making. In addition, the circular structure of the intersection creates a different driving pattern than a regular intersection. Through the derived indicators, the assessment of stability and driving comfort can be performed due to the rapid change in speed, whether it affects other vehicles, and the rapid change in acceleration when passing through the intersection. The risk scores for right turns at signalized intersections and roundabouts were found to be 0.525 and 0.501, respectively, which is higher than the average risk score of 0.408 for all ODDs. Ultimately, for ODDs that require safe passage by the driver’s judgment, AVs should be able to detect vehicles on the connected road or on the opposite lane and make appropriate deceleration and yield decisions to improve driving safety.

The ODDs of intersection types controlled by signals include straight through and left turns at signalized intersections. The promising indicators of the straight through signalized intersection were analyzed as VF spc. and VF hdwy. and the promising indicators of the left turn signalized intersection were derived as Avg. DRAC, CPI, VF spc. and VF hdwy. ODD requires stop-and-go operations according to signals, so AVs must accurately recognize signals and perform deceleration and acceleration while detecting the distance to the vehicle in front. In particular, when performing a left turn, the likelihood of a collision with a vehicle making a right turn from the oncoming lane increases. Therefore, DRAC and CPI, which are directly related to collision accidents, as well as the variability of vehicle following distance over time, should be added to verify driving safety. The risk scores for straight through and left turns at signalized intersections are 0.135 and 0.369, respectively. These risk scores are lower than the average risk score, indicating that driving safety is not significantly compromised compared to other ODDs. However, this ODD is considered essential for the evaluation of AVs as it represents the most common scenario in interrupted flow conditions.

The results for ODDs with vehicle interactions other than intersection type are shown in

Table 5. The result of a vehicle interaction ODD.

Total Variance Explained			Rotated Component Matrix		Component Score	Normalized Values	Risk Score
ODD Name	Total	Variance (%)	Indicator	Component Matrix Coefficient
Merge from side road	3.57	29.71	NC_TTC	0.98	0.28	0.507	0.493
			NC_DRAC	0.97	0.27	0.582
			CPI	0.95	0.27	0.494
			STD_spc.	0.70	0.19	0.390
Illegal parking	2.99	38.49	STD_spc.	0.83	0.43	0.274	0.457
			STD_spd.	0.76	0.37	0.523
			NC_DRAC	0.65	0.42	0.573
U-turn	3.90	35.45	STD_spd.	0.89	0.23	0.377	0.364
			STD_spc.	0.88	0.22	0.380
			VF_spc.	0.88	0.22	0.414
			STD_hdwy.	0.86	0.25	0.277
			VF_hdwy.	0.85	0.23	0.370
Speed limit change zone	3.90	41.48	STD_acc.	0.96	0.37	0.332	0.306
			P2P jerk	0.94	0.36	0.301
			STD_spd.	0.93	0.36	0.284
Bus stop	3.07	31.92	VF_spc.	0.92	0.32	0.160	0.247
			VF_hdwy.	0.90	0.31	0.194
			NC_RDE	0.80	0.27	0.285
			STD_acc.	0.68	0.21	0.400
			P2P jerk	0.52	0.14	0.196
Additional lane	2.82	33.48	VF_hdwy.	0.91	0.33	0.204	0.213
			NC_SDI	0.90	0.32	0.127
			VF_spc.	0.83	0.28	0.198
			STD_spc.	0.67	0.25	0.322

. Among the ODDs involving vehicle interactions, those with higher-than-average risk scores include sections with minor road merging and illegal parking. The most promising indicators for the minor road merge segments were NC TTC, NC DRAC, STD spc., and CPI. AVs are required to predict the movement of vehicles entering from minor road merges and take appropriate evasive actions to avoid collisions. For example, they may adjust their speed to maintain a safe distance from the merging vehicle or change lanes if necessary. In this context, the derived indicators can be employed to assess the accuracy of situation recognition and response performance of AVs. The promising indicators for the ODD involving illegal parking were identified as STD spd., NC DRAC, and STD spc. AVs are required to decelerate appropriately to assess the reason for the stopped vehicle ahead. It is also necessary to evaluate the frequency of sudden stops performed by AVs to avoid illegally parked vehicles. The risk scores for the minor road merging section and the section with illegal parking were calculated to be 0.493 and 0.457, respectively, which are higher than the average risk score. This ODD requires evaluation of how safely AVs navigate interactions with other vehicles. Moreover, in this ODD, earlier situation recognition by the AV facilitates smoother lane changes and more stable evasive maneuvers. The development of algorithms that enhance sensor performance and predict object movement is crucial for improving the driving safety of AVs in these scenarios.

Among the ODDs involving vehicle interactions, those with lower-than-average risk scores were identified as U-turn sections, bus stops, lane addition sections, and speed limit transition zones. For U-turn sections, STD spd., STD spc., STD hdwy., VF spc., and VF hdwy. were identified as promising indicators. U-turn sections require a large turning radius, which can reduce vehicle stability compared to standard straight or turning maneuvers. Additionally, interactions with oncoming vehicles occur in the opposing lane, requiring AVs to accurately assess the speed and distance of oncoming vehicles and accelerate to merge at an appropriate time. In sections with bus stops, STD acc., P2P jerk, NC RDE, VF spc., and VF hdwy. were found to be promising indicators. It is necessary to evaluate AVs’ rapid speed changes and sudden braking in response to bus stops and departures. Moreover, the variability of interaction patterns should be assessed to evaluate the irregular movement of vehicles around buses. The promising indicators for lane addition sections were identified as NC SDI, STD spc., VF spc., and VF hdwy. This ODD is characterized by frequent cut-in maneuvers from neighboring vehicles. The analyzed indicators are suitable for evaluating AV driving safety in cut-in situations. For speed limit transition zones, the promising indicators were STD spd., STD acc., and P2P jerk. These sections require vehicles to adjust their speed accordingly. AVs need to execute speed adjustments that minimize impact on following vehicles, and this capability should be evaluated. While the risk scores for the aforementioned ODDs are relatively low, the risk may increase significantly if ODD-specific high-probability events occur.

The analysis of ODDs with interactions with non-vehicular road users is presented in

Table 6. The result of a road user interaction ODD.

Total Variance Explained			Rotated Component Matrix		Component Score	Normalized Values	Risk Score
ODD Name	Total	Variance (%)	Indicator	Component Matrix Coefficient
Crosswalk at intersection	3.46	28.81	VF_spd.	0.87	0.28	0.393	0.552
			STD_spd.	0.84	0.25	0.629
			STD_acc.	0.81	0.22	0.509
			VF_jerk	0.77	0.21	0.760
			P2P jerk	0.77	0.23	0.316
Mid-block crosswalk	3.24	29.47	P2P jerk	0.85	0.28	0.402	0.485
			NC_RDE	0.82	0.27	0.520
			STD_spd.	0.81	0.25	0.435
			VF_acc.	0.74	0.24	0.584
Bicycle lane	2.70	29.19	STD_spd.	0.88	0.42	0.594	0.473
			STD_acc.	0.77	0.35	0.380
			VF_jerk	0.65	0.36	0.592
			VF_spd.	0.64	0.21	0.325

. The risk scores for intersection crosswalks, mid-block crosswalks, and bicycle lanes, which are ODDs involving interactions with non-vehicular road users, are 0.552, 0.485, and 0.473, respectively, and have higher risk rankings than the average risk score. The promising indicators for intersection crosswalks were analyzed as STD spd., STD acc., P2P jerk, VF spd., and VF jerk. In addition, promising indicators for mid-block crosswalks were identified as STD spd., P2P jerk, and VF acc., NC RDE. Both intersection crossings and mid-block crossings are ODDs where frequent interactions with pedestrians occur. When pedestrians are crossing, AVs should detect them and decelerate smoothly to avoid compromising the safety of following vehicles and pedestrians. Additionally, when a pedestrian suddenly enters the roadway, AVs must be evaluated for their ability to react quickly and avoid accidents. For bicycle lanes, STD spd., STD acc., VF jerk, and VF spd. were identified as promising indicators. Bicycles travel at higher speeds compared to pedestrians and exhibit more erratic behavior than motor vehicles. AVs should rapidly detect cyclists and predict their movements, maintaining stable speed control and deceleration to avoid creating hazardous situations for both cyclists and following vehicles.

5. Discussion

This study identified promising indicators for ODDs affecting AVs and developed a methodology to rank ODDs based on driving safety vulnerability. A simulation environment was established using the Seoul autonomous mobility testbed, with driving behaviors in AD and MD modes implemented through AVD, which represents real-world driving data of AVs. The evaluation indicators used to assess the driving safety vulnerability of AVs include 16 indicators that evaluate longitudinal driving stability and vehicle interaction safety. Based on PCA results for each ODD, indicators included in the component with the largest explained total variance were defined as promising indicators for that ODD. A risk score was derived by averaging the normalized values of promising indicators for each ODD. Higher risk scores indicate ODDs where AV driving safety is more vulnerable.

The analysis revealed that driving stability indicators were identified as promising for ODDs at unsignalized intersections. Notably, unsignalized left turns, signalized right turns, and roundabouts yielded risk scores of 0.554, 0.525, and 0.501, respectively. These scores are higher than the average risk score for all ODDs analyzed in this study, indicating unsafe conditions leading to greater potential of accident occurrence. Such high-risk scores suggest these ODDs are particularly vulnerable in terms of driving safety. This implies that there is an opportunity for improving AV algorithms. In addition, These ODDs should be prioritized for infrastructure safety improvements to reduce risks.

While unsignalized intersections are typically established for low-traffic segments, left turns at these intersections present a higher number of conflict points. Consequently, AVs must detect vehicles on conflicting approaches and make appropriate yielding decisions to enhance driving safety. For ODDs involving vehicle interactions, minor road merging and illegal parking sections showed risk scores of 0.493 and 0.457, respectively. Vehicle interaction indicators were identified as promising for segments with minor road merges. AVs are expected to improve safety by detecting vehicles entering from minor roads and executing appropriate evasive maneuvers to prevent collisions. Regarding ODDs with illegal parking, STD_spd., NC_DRAC, and STD_spc. were identified as promising indicators. These indicators are essential for evaluating AV’s performance in detecting illegally parked vehicles and executing appropriate deceleration while maintaining safe spacing with surrounding vehicles. The STD_spd. and NC_DRAC are able to evaluate the stability of speed control and collision risk during deceleration, while STD_spc. evaluates the spacing management capability when AVs attempt to change lanes to avoid illegally parked vehicles. The STD_spd. and NC_DRAC evaluate the stability of speed control and collision risk during deceleration, while STD_spc. evaluates the spacing management capability when AVs need to change lanes to avoid illegally parked vehicles. Additionally, lane changes should be executed without affecting approaching vehicles in adjacent lanes. Risk scores for intersection crosswalks, mid-block crosswalks, and bike lanes were 0.552, 0.485, and 0.473, respectively. Intersection and mid-block crosswalks are characterized by frequent pedestrian conflicts, which may cause AVs to decelerate abruptly. Such sudden deceleration can lead to rear-end collisions. Therefore, deceleration should be implemented to minimize impact on following vehicles. Furthermore, when pedestrians are crossing, AVs should stop with appropriate deceleration to avoid creating a sense of threat.

The results of this study are expected to provide an important guideline for AV technology development and policy decisions. The methodology presented for ranking ODDs based on their vulnerability can serve as a foundational tool for establishing evaluation criteria for AV performance. In terms of technological development, AVs can improve driving safety by promoting the development of sensors or algorithms that are designed for particular ODDs, where driving safety vulnerabilities are particularly high. Additionally, infrastructure design can be optimized based on these findings, such as enhancing visibility at high-risk intersections or implementing dedicated AV lanes in certain areas. In terms of improving traffic safety and regulatory policies, ODDs that significantly affect the driving safety of AVs are monitored to enable effective emergency response. In terms of traffic system management, the traffic management system can be optimized for mixed traffic environments by analyzing real-time data and evaluating AV driving safety vulnerabilities by ODD. The promising indicators of each ODD and the ranking methodology of ODDs with vulnerable driving safety provided in this study can be used as a foundation for establishing driving performance evaluation factors and evaluation environment for AVs in the future. Furthermore, the results of this study’s ODD vulnerability analysis emphasise the importance of risk-aware planning and vehicle-signal coupled coordination, which are part of the future development directions for CDA technology proposed by Wang et al. (2024) [57]. The ODD vulnerability ranking highlights the need for risk-aware planning in mixed CAV/HV environments. High risk scores for intersections emphasise the importance of developing vehicle-signal coordination technologies.

This study proposed a new approach that evaluates the driving safety vulnerability of AVs by comprehensively analyzing each ODD’s characteristics. Although the results can be used as a fundamental foundation for various industries such as traffic safety, road infrastructure design, and transportation system management, further research is needed to address the limitations of this study. First, the simulation environment in this study is based on the real world, however, real-world conditions have more unexpected variables. Accordingly, future research should be conducted to validate and supplement the results of this study with AVs driving data that can generate various evaluation indicators. The second limitation is that it only considers the driving safety aspect of AVs. In the real world, the driving behavior required of AVs is not only about individual safety, but also about the safety of neighboring vehicles, pedestrians, and cyclists. In future research, the weights of evaluation indicators derived from the PCA-based method in this study will be utilized to develop a more comprehensive assessment framework for AV safety across different ODDs. Additionally, when a controllable experimental environment is established, future research should be conducted to analyze the safety of road users under the driving algorithms of AVs. Finally, the ODDs selected in this study were analyzed based on local characteristics of traffic conditions in the Seoul autonomous mobility testbed. Future research should expand to include a wider range of ODD environments and scenarios. This includes varying traffic volumes and vehicle types, as well as additional complex interactions that may not have been fully captured in the current study.