The Evaluation of Experimental Variables for Sustainable Virtual Road Safety Audits

Abstract

This paper evaluates experimental variables for virtual road safety audits (VRSAs) through practical experiments to promote sustainable road safety. VRSAs perform road safety audits using driving simulators (DSs), and all objects in the road environment cannot be experimental variables because of realistic constraints. Therefore, the study evaluates the likelihood of recommendation of VRSA experimental variables by comparing DSs experiments and field reviews to secure sustainable road safety conditions. The net promoter score results evaluated “Tunnel”, “Bridge”, “Underpass”, “Footbridge”, “Traffic island”, “Sign”, “Lane”, “Road marking”, “Traffic light”, “Median barrier”, “Road furniture”, and “Traffic condition” as recommended variables. On the contrary, the “Road pavement”, “Drainage”, “Lighting”, “Vehicle”, “Pedestrian”, “Bicycle”, “Accident”, and “Hazard event” variables were not recommended. The study can be used for decision making in VRSA scenario development as an initial effort to evaluate its experimental variables.

1. Introduction

Traditional road safety audits (RSAs) officially began in the UK in the 1980s and have continued to expand worldwide [1]. The RSA is a formal and proactive examination of road safety by expert teams [2,3]. It is a process of ensuring safety issues of an existing or future road and suggests remedial measures for enhancing safety [2].

In recent years, there have been several attempts to apply driving simulators (DSs) to RSAs as driving simulators become more advanced [4,5]. Virtual road safety audits (VRSAs) are newly proposed approaches to practice design decisions and safety reviews on an existing or future road using a DS [6,7,8,9]. The DS, a technology used in VRSAs, simulates a real traffic environment with a visual display and vehicle motions [4,5]. The DS has three levels of fidelity depending on the simulator configurations: A desktop with a single monitor, a desktop with three monitors, and a vehicle cab with a wide display [4,5]. Previous research shows VRSA experiment fidelity as an experimental requisite: from lower end to higher end [6]; the higher fidelity is a full-scale DS, while the lower fidelity is a dynamic survey that simulates the driving scene, including videos of 3D rendered models [6]. If the project requirements are simple, the low-fidelity single-screen DS is suitable, whereas if the project requirements are complex, the high-fidelity multi-screen DS is [6]. Experiments for VRSAs depend on the research question and project complexity; therefore, the driving simulator requisites may vary.

As mentioned above, VRSAs are a newly proposed concept, so there is little research on the topic. Some studies have suggested the framework, scenario creation methodologies, and experimental variables of VRSAs, and introduced practical applications of VRSAs [6,7,8,9]. Santiago-Chaparro et al. (2011) suggested the VRSA framework and criteria: testability and feasibility [7]. They also show a simple scenario creation of VRSAs by creating a 3D model [7]. Moreover, Noyce et al. (2018) propose the whole process of VRSAs: candidate site identification, collection of supplemental data, and scenario creation [6]. They also completed a practical VRSA experiment at Highway 45 and Watertown Plank Road interchange in the United States [6]. Similarly, VRSAs are practiced at the Seoul–Sejong Expressway in Korea [9].

However, related studies did not research the experimental variables for VRSAs. All RSA items cannot be experimental variables of VRSAs because of the gap between DSs environment and real road environment. It is necessary to determine whether a specific road object can be tested using DSs for VRSAs. Therefore, in a previous study, Jun et al. (2021) evaluated the priority of experimental variables based on testability and feasibility by using AHP [8]. However, this study did not evaluate the VRSA variables based on practical experiments and, instead, qualitative analysis was used.

Therefore, this study focuses on evaluating experimental variables for sustainable VRSAs by comparing practical DS experiments and field reviews. Then, it presents the expert survey results on the net promoter score (NPS) of experimental variables for VRSAs. This study also suggests proper experimental variables for VRSAs. Although it is in early-stage research, it can help decision making in scenario creation and experiments for VRSAs. Furthermore, it is also worth noting that the VRSA is a new and alternative approach that applies technologies (i.e., DSs) to traditional RSAs.

2. Literature Review

In this study, the literature review serves to identify the available resources related to the VRSA variables. Traditional RSAs suggest checklists, which can determine items that may affect road safety; in addition, the local dynamic map (LDM) can help identify the experimental variables of VRSAs. Undoubtedly, since the LDM is a database embedded in ITS stations, its relevance to this paper may be lower. However, the LDM is a data model that describes, classifies, and standardizes road environment objects as data. Therefore, this paper uses information in the LDM to identify and classify experimental variables of VRSAs.

2.1. Categories and Content of Road Safety Audits

RSAs are formal road safety examinations by an independent team of experts in various disciplines [2] and are conducted in pre-construction, construction, and post-construction phases [3]. For increasing the safety of all road users, they judge potential hazards and propose remedial measures to eliminate the risk [2,3,10,11]. The advantages of RSAs include proactive problem solving, low-cost, and high-value remedial work by promoting a safety culture [2,3,12]. In particular, RSAs can break the links of three contributing factors (human, vehicle, and road environment factors) to accidents [12].

RSAs have been widespread worldwide since they were first developed in the UK; accordingly, the knowledge and experience of RSAs are being updated [2,8,12]. Many countries have adopted RSAs procedures and concepts, but the actual content is dependent on country characteristics—specifically, the road type, audit project stage, audit team, ordering body (the project owner), audit checklist items, and audit process (method) are different for each country [8].

The discussion on RSAs in Korea began in 1994 and was enacted with the revision of related laws in 2006 [13]. RSA phases are divided into design, pre-construction, and existing stages by covering all road types. The auditing process consists of planning, ordering for audits, selecting an audit team, conducting audit analysis, and reporting results. The audit analysis can be conducted through field visits, a checklist, a review of project data, and drawings. The checklist categorizes the road environment, as shown in

Table 1. Road safety audits categories and content. Source: [14].

Categories	Content
General topics	Road type, Speed, etc.
Cross-section elements	Median barrier, Roadway layout, Shoulder, etc.
Alignment	Horizontal alignment, Vertical alignment, Sight distance, Transition, etc.
Intersection	Traffic light, Traffic island, Roundabout, etc.
Interchange	Ramp, Rail crossing, etc.
Pedestrian and Bicycle	Sidewalk, Bicycle route, Related facilities, etc.
Pavement and Drainage	Road pavement, Drainage, Tunnel, Bridge, etc.
Road furniture	Fence, Delineator, Cushion, Barrier, Hump, Glare screening, Parking area, Rest area, Bus bay, Soundproofing, etc.
Sign and Marking	Sign, Road marking, Pole, etc.
Lighting	Illumination, Pole, etc.
Work zone and Crash handling	Detour, Work zone, Crash handling of debris, etc.
Weather and Natural features	Weather, Animal, Landscaping, etc.

, and each checklist content, such as visibility and clearance, must be reviewed [14].

2.2. Road Environment Definition by the Local Dynamic Map

The LDM is a conceptual database embedded in an intelligent transport system (ITS) station and includes vehicle sensor data, map data, and status information related to the ITS station [15,16,17]. The LDM is a crucial element of co-operative ITS and supports various ITS applications [16]; it also serves as an integrated platform that combines dynamic objects based on a static digital road [18]. The primary purpose of the LDM is to provide consistent situational awareness and interoperability between distributed applications as the common model of the world [16,19].

The LDM was first introduced in the SAFESPOT project, which designed a system for improving road safety; it has then been continuously developed through diverse projects such as CVIS, DRIVE-C2X, and eCoMove [19]. The concept of the LDM has been standardized at the initial stage by ETSI (European Telecommunications Standards Institute) and ISO technical recommendations [16,18,20,21]. The LDM contains information on the real world and conceptual objects that can affect traffic safety and efficiency [16]. As shown in

Table 2. The four layers and example data of the LDM. Source: [17,18].

The Four Layers		Examples
The first layer (Bottom)	Permanent static data	GIS map provided by map data suppliers. It includes intersections, lane precise local road topography, and statutory speed limit.
The second layer	Transient static data	Traffic signs, landmarks, intersection features, and roadside infrastructures (ITS stations)
The third layer	Transient dynamic data	Hazard information, traffic congestion, traffic conditions, weather situation, slippery road, traffic signal phase, road works information, temporary speed limit, temporary restriction changes, and parking lots’ current status.
The fourth layer (Top)	Highly dynamic data	Automotive sensor information, dynamic traffic signs, pedestrians, ego vehicles, and vehicles (V2X messages such as GPS position and speed)

, the LDM consists of four different layers with categorizing data that describe real-world objects [16,17,18].

2.3. Identifying and Categorizing Experimental Variables Based on the Literature Review

This study suggests VRSA experimental variables based on the literature review of RSAs and the LDM. The road environment can be identified and categorized as shown in

Table 3. Categorizing experimental variables for VRSAs. Source: Compiled by the authors based on related studies.

Category	Experimental Variables
Static road environments	Roadway layout *, Sidewalk, Bicycle route, Shoulder (roadside)
	Tunnel, Bridge, Underpass, Footbridge
	Other road types (Rail track, etc.)
	Road pavement
	Drainage
	Traffic light
	Traffic island
	Median barrier
	Landscaping
	Sign
	Lane, Road marking
	Lighting
	Road furniture (Fence, Delineator, Cushion, Barrier, Hump, Glare screening, Parking area, Rest area, Bus bay, Soundproofing, etc.)
Dynamic road environments	Vehicle
	Pedestrian
	Bicycle
	Traffic condition
	Accident
	Work zone
	Weather
	Hazard event (Disaster, Animal, etc.)

* Roadway layout variable includes road geography elements such as alignment and transitions because a 3-dimensional scenario is based on road blueprint.

. According to LDM standards, experimental variables are classified into static and dynamic factors that include the checklist items of the traditional RSA. With the advancement of computer graphics and simulator technology, DSs can develop all road environment factors such as drivers, vehicles, roads, and environments [22].

The variables presented in Table 3 have a relationship with road safety because they are derived from the items of RSAs. According to the FHWA RSA guidelines, a RSA prompt list (i.e., RSA items) is presented to help identify potential road safety problems at the design or operation stage of the road [3]. Therefore, the experimental variables presented in this study have an inevitable relationship with road safety.

In addition, previous studies also show the correlation between road safety with road environment items presented as experimental variables [13,23,24]. There is a study that analyzed the traffic accident density based on RSA items such as lane width, number of traffic lights, length of bicycle path, tunnel dummy, and number of safety signs [13]. In the review of research on spatial road safety studies, study design characteristics were traffic (speed, traffic volume, etc.), road user (age, modal distinction), road environment (speed limit, gradient, lane width, lane number, etc.), and spatial aggregation approach (regional level, zonal level, and link-level) [23]. Speed, congestion, and road horizontal curvature variables were notably presented as factors to improve road safety in a study that analyzed traffic and road-related factors as the leading causes of car accidents [24].

2.4. Comparison with Other Research in This Field

The experimental variables presented in this study can be compared with the scenario creation of VRSA research. DS scenario creation develops a 3D road site model to perform VRSAs [6,7]. First, a 2D CAD drawing of the VRSA candidate site is created and then converted to a 3D model that reproduces the site’s geometry, which contains latitude, longitude, and elevation [6,7]. When converting to a 3D model, the roadway surface, sidewalks, and bike path, pavement marking, and building outline are vital factors among the road environment factors [6]. In addition, in the 3D model, texture, traffic signals, and signs are expressed, metadata of roads are defined, and behaviors and events should be coded in simulation according to the need for research [6]. A similar study conducted RSAs through 3D design visualization. The 3D model of this study was developed using digital terrain models, surface models, 2D or 3D CAD files, and details for rendering [25]. In the 3D model, specific road elements include traffic signs, pavement markings, lanes, landscaping, roadside furniture, traffic signals, and right-of-way [25].

Previous VRSAs studies only wrote that VRSA scenarios create road elements such as terrain, surface, signs, signals, buildings, and events. In addition, an RSA study with 3D visualization lists the elements that need to be built into a 3D road environment, such as traffic signs. Therefore, previous studies suggested that the road environment should be implemented as a 3D model but did not specifically suggest what road elements should be implemented in practice. On the other hand, this study revealed what road environment factors are related to road safety and suggested which road environments should be expressed in 3D for VRSAs. In addition, the experimental variables of VRSAs identified in this study were evaluated through a DS experiment and field review.

3. Methods

3.1. Materials and Methods

3.1.1. Participants

The participants were four experts in transportation, all between the ages of 40 and 59 and with more than ten years of work experience. Participants had been driving for at least 20 years and had experience with driving simulators. Participants’ vision was 1.0 (both eyes) and one person had a vision of 0.3 in both eyes. None of the participants had any disease and experienced discomfort in the experiment condition. The reason for selecting only professionals is that a team of only experts performs the traditional RSA. Since VRSAs perform RSA based on a DS, it is necessary to follow the conventions of the existing RSA.

3.1.2. Driving Simulator and Field Reviews

The experiment aimed to evaluate the recommendation of experimental variables; therefore, DS experiments and field reviews were sequentially performed on 4 November 2020, as shown in Figure 1.

The DS used was the advanced full-scale DS from the Korea Expressway Corporation, shown in Figure 1. The Korea Expressway Corporation’s DS was built about three years after 2015 as part of Korea’s national R&D [26]. The DS was developed and validated as part of the national R&D project, and it is currently used for various research projects [26,27,28]. Specifically, it comprises image generation and display systems, sound systems, real-time simulation systems, motion systems, simulation monitoring, and a control system [26,27]. It includes a spherical screen with a car cabin and a six-axis motion system. The resolution scale of visual scenes is 2560 × 1600 pixels at a refresh rate of 50–60 Hz [28]. The X-Y rail, 6DOF motion platform, Yaw Table, 4-Axis vibrator, and 0.7G acceleration can reproduce the vehicle tilting occurring in the inclined or curved section [28]. The system records data on brakes, accelerator pedals, and steering wheels, allowing analysis of vehicle barycenter position, speed, and acceleration [26].

In this research, the reason for choosing a high-fidelity DS is that VRSA demanded a high sense of realism and complexity as a replacement for the traditional RSA in reality. In addition, VRSA case studies also used high-fidelity DS [6,9]. The pursuit of high-fidelity is also based on the assumption that higher fidelity improves the validity of performance and improves transfer to actual car driving [4].

After the DS experiments, experts conduct field reviews through actual vehicle driving in the same area and route as the DS scenario.

3.1.3. Net Promoter Score

The likelihood of recommendation for VRSAs is evaluated based on the net promoter score (NPS), and expert interviews specifically demonstrate the reason for the evaluation. The NPS is a methodology to measure customers’ willingness to recommend a product or service to their friends and was published in 2003 by Frederick F. Reichheld [29]. The traditional customer satisfaction scores did not fit the observations of actual customer actions; thus, the NPS was developed as an alternative to the conventional method [30]. Calculating the NPS uses the 11-point scale from a 0 to 10 rating in survey responses; then, the result is calculated by subtracting the ratio of the detractor from the promoter ratio. “Promoters” are respondents with 9–10 points, and “detractors” refer to respondents with 0–6 points. Respondents with a score of 7 or 8 are “passively satisfied” [29,30], and the final NPS scores mathematically range from −100 to +100 [31].

In general, the NPS is widely used by companies as a tool to measure customer loyalty [30,31,32,33]. Recently, it has been expanding its research field of application, such as inpatient experience and healthcare service [31,34,35]. The NPS has the advantage of being simpler and quicker to produce results than other survey techniques [29,32]. This method uses a likelihood-to-recommend questionnaire; thus, it is more suitable for this study, which aims to evaluate the recommendation of experimental variables for VRSAs by calculating the NPS.

Table A1. Questionnaire.

Q. How Likely Is It That You Would Recommend the VRSA Experimental Variables to a Colleague?
Experimental Variables		Not at All Likely						Extremely Likely
Static road environments	Roadway layout, Sidewalk, Bicycle route, Shoulder (roadside)	0	1	2	3	4	5		6	7	8	9	10
	Tunnel, Bridge, Underpass, Footbridge	0	1	2	3	4	5		6	7	8	9	10
	Other road types (Rail track, etc.)	0	1	2	3	4	5		6	7	8	9	10
	Road pavement	0	1	2	3	4	5		6	7	8	9	10
	Drainage	0	1	2	3	4	5		6	7	8	9	10
	Traffic light	0	1	2	3	4	5		6	7	8	9	10
	Traffic island	0	1	2	3	4	5		6	7	8	9	10
	Median barrier	0	1	2	3	4	5		6	7	8	9	10
	Landscaping	0	1	2	3	4	5		6	7	8	9	10
	Sign	0	1	2	3	4	5		6	7	8	9	10
	Lane, Road marking	0	1	2	3	4	5		6	7	8	9	10
	Lighting	0	1	2	3	4	5		6	7	8	9	10
	Road furniture (Fence, Delineator, Barrier, Hump, Glare screening, Parking area, Rest area, Bus bay, etc.)	0	1	2	3	4	5		6	7	8	9	10
Dynamic road environments	Vehicle	0	1	2	3	4	5		6	7	8	9	10
	Pedestrian	0	1	2	3	4	5		6	7	8	9	10
	Bicycle	0	1	2	3	4	5		6	7	8	9	10
	Traffic condition	0	1	2	3	4	5		6	7	8	9	10
	Accident	0	1	2	3	4	5		6	7	8	9	10
	Work zone	0	1	2	3	4	5		6	7	8	9	10
	Weather	0	1	2	3	4	5		6	7	8	9	10
	Hazard event (Disaster, Animal, etc.)	0	1	2	3	4	5		6	7	8	9	10

shows the questionnaire developed according to the previous study [29,30,31]. The experimental variables in the survey are all road environments included in the DS scenario and are compiled by the authors based on a literature review (i.e., Section 2.3).

3.1.4. Case Study

The experiment area is Sangam-dong, Seoul, Korea. The case area starts with a five-lane road and transfers to two and three-lane roads, with a total of about 3 km. As shown in Figure 2, the 3D scenario was developed around Sangam-dong based on the picture, road blueprints, case site investigations, and SEOUL T-GIS that mark traffic facilities on the GIS map. In Figure 2, there is also an experimental driving route. The DS experiment is conducted through a scenario that includes all VRSA variables for the actual existing road. It means that all road environmental features were implemented in the scenario. The experimental scenario consists of two situations, day and night, and each takes about 10 min. It is similar to traditional RSAs that conduct a day review and a night review [3].

As in Figure 3, the scenario changes weather conditions (rain, fog, clear) and includes all experimental variables identified in this paper: roadway layout of Sangam-dong case area, tunnel, road pavement, drainage, traffic light, traffic island, median barrier, landscaping, sign, lane and road marking, lighting, road furniture, vehicle, pedestrian, bicycle, traffic condition, accident, work zone, weather, hazard event (disaster, animal, etc.).

The surrounding vehicles were developed with various vehicle types (trucks, compact cars, etc.), and the vehicle speed was randomly specified in compliance with the road speed limit of 60 km/h. Traffic volume was implemented similarly to the reality of the case area through field visits before scenario development.

3.2. Analysis Procedures

Upon arrival at the laboratory, participants were briefed and the experiment and NPS survey were explained. To adapt to the DS, participants conducted test driving on different roads before the experiment. Participants then conducted a DS experiment in day and night scenarios for a total of 20 min. The scenario is in the order of a day and then a night. The vehicle started from the first lane, and the driving route was guided, but the drivers can drive autonomously and, for instance, change lanes and speed. Because of the purpose of this study, data such as speed and acceleration collected from DS were not used separately for analysis.

Secondly, participants move to the case site for a field review by driving a real car. They drive freely with the same driving route as the DS experiment.

Then, participants responded to the survey and were interviewed. The DS experiment and field review are compared to determine how similar to reality the DSs scenarios developed for VRSAs are; afterward, experts recommend VRSA experimental variables based on comparing the real-world and DS scenarios.

4. Results and Discussion

4.1. Results

4.1.1. Survey Results

This study evaluates the likelihood of recommendation for each road environment variable identified through previous studies. Since each variable is presented as a single statement, individual items cannot be calculated as Cronbach’s alpha for the internal consistency reliability [36].

Table 4. NPS results.

Experimental Variables	NPS Score	Experimental Variables	NPS Score
Roadway layout, Sidewalk, Bicycle route, Shoulder (Roadside)	0	Lighting	−50
Tunnel, Bridge, Underpass, Footbridge	50	Road furniture (Fence, Delineator, etc.)	25
Other road types (Rail track, etc.)	0	Vehicle	−25
Road pavement	−50	Pedestrian	−50
Drainage	−100	Bicycle	−50
Traffic light	25	Traffic condition	25
Traffic island	50	Accident	−50
Median barrier	25	Work zone	0
Landscaping	0	Weather	0
Sign	50	Hazard event (Disaster, Animal, etc.)	−25
Lane, Road marking	50

shows the results of the NPS analysis. The NPS score of “Tunnel”, “Bridge”, “Underpass”, “Footbridge”, “Traffic island”, “Sign”, “Lane and Road marking” is the highest. “Traffic light”, “Median barrier”, “Road furniture”, and “Traffic condition” are also evaluated as recommended variables. On the other hand, some variables such as “Road pavement”, “Drainage”, “Lighting”, “Vehicle”, “Pedestrian”, “Bicycle”, “Accident”, and “Hazard event” are not recommended.

4.1.2. Expert Interviews

The reasons for the NPS scores are discussed through expert interviews. “Tunnel”, “Bridge”, “Underpass”, and “Footbridge” must be displayed and evaluated because they are the surrounding road environment. This recommended variable can evaluate the transition areas’ safety, illumination inside the tunnel, and visibility. “Traffic island” can directly evaluate behavior at roundabouts, junctions, and in diverging areas as a part of the road layout. In addition, assessments of the traffic island visibility and necessity can improve pedestrian safety and minimize vehicle conflict. Similarly, the “traffic light” variable should be included in the evaluation of VRSAs as it is essential to evaluate the adequacy of signal operation, driver perception, and behavior at signal intersections.

“Sign”, “Lane”, “Road marking”, and “Median barrier” are recommended because they impact a vehicle’s driving route in VRSAs. In other words, these variables with various colors and layouts are vital safety components to distinguish traffic flow directions and affect the driving route. “Road furniture” can be easily installed by the DS software; moreover, it makes the road environment of DS scenarios more realistic. In particular, when VRSAs are performed on existing roads, road furniture can be suggested as remedial measures with the process of test and retest by easily placing the object. “Traffic condition” is an essential variable for evaluating the impact of traffic congestion, and it will provide the participants with a more realistic scenario by developing surrounding traffic conditions similar to the real world.

On the other hand, the results of the interview on non-recommended variables are as follows: “Road pavement” is visually good, but it is challenging to experience realistic driving due to simulation limitations of vibration and steering. “Drainage” and “Lighting” show poor fidelity, and it is difficult to evaluate the drainage function and illumination with DSs. “Drainage” is difficult to render with traditional DSs software, and there are limitations in evaluating drainage capacity, efficiency, and urban flooding through DSs. “Lighting” requires separate adjustment for streetlight effects, and it is difficult to display the illuminance precisely as it is in reality. In addition, there is a limit on the maximum number of street lights in the DSs software.

Likewise, the realism of “Vehicle”, “Pedestrian”, and “Bicycle” is insufficient. Vehicle variables have headlights and less realistic driving behavior. Since there is no data such as speed, detailed trajectory information, and data collected for a long period [6], the DS scenario is developed by assuming such data. Thus, the DS scenario with assuming data contributes to the decline in the realism of VRSA experiments. Besides, the pedestrian and drivers of traditional DSs usually have exotic appearances and specific behavior. The “Accident” and “Hazard event” variables are significantly time- and budget-consuming for developing a three-dimensional model. Moreover, given the wide variability of time, traffic, and weather conditions in the natural world, actual events cannot be imitated by assuming individual scenarios for all situations.

Lastly, the experimental variables examined should be sustainable under the VRSA conditions. That being said, data accumulated in the VRSAs might be analyzed in the future to identify road safety hazards that would never be understood in traditional RSAs. That is why the authors tried to identify experimental variables that could be used in the form of sustainable VRSAs.

4.2. Comparison of These Results with Other Researches

The results of this study must be compared with previous VRSA research. Previous VRSA studies have limitations that do not explain experimental variables of VRSAs, as described in Section 2.4. Therefore, the previous study that is most relevant is Jun et al. (2021), which evaluated the experimental variables of VRSAs. Jun et al. (2021) presented VRSA experimental variables based on RSA checklists and prioritized them using the analytic hierarchy process (AHP) [8]. The hierarchical criteria are “testability” and “feasibility”; The alternative levels of AHP are the experimental variables [8]. As a result, it suggests that “work zone and crash handling” and “sign and marking” alternatives have high priority, and “lighting” and “pavement and drainage” variable have the lowest [8]. In this way, the results of this study (i.e., the recommendation of experimental variables) are similar to the previous study results. However, in this study, the “accident” variable is different from Jun et al. (2021). According to Jun et al. (2021), the “accident” variable has a high “testability” criterion due to many previous DS experiments, and the “feasibility” criterion is high because it can be developed through typical design tools of the DS software [8]. In this study, the “accident” was evaluated as a non-recommended variable for the following reasons: Even if the accident situation is implemented with typical DS software, there is a gap between the simulation and actual events. It also takes time and cost to develop a traffic accident scenario suitable for the reality of a VRSA candidate site [6]. The difference in research results stems from the fact that the evaluation method and target are different. Moreover, the previous study performed only AHP, a qualitative analysis.

The results of this study must also be compared with related research using DSs. The recommended variables of this study have been widely used as variables or scenario backgrounds in studies using DSs according to the purpose of each study [22]. In general, the DSs implement topographic 3D data such as alignment and cross-section, and also develop infrastructure elements such as traffic lights, and inputs traffic data [22,37].

The non-recommended variables have been presented as limitations of the use of DS in previous studies. The “Road pavement” variable and the lower number and sizes of textures can be implemented because the game engine and hardware that DS depends on are not powerful [6]. In pavement rumble strips, it is also necessary to develop visual, vibratory, and acoustic feedback separately as warning audio-tactile systems [37]. However, recently, there have been efforts to develop the road surface such as rutting, cracks, and undulations in the DS in accordance with visual and actual road surface characteristics [38]. The “Drainage” variable is challenging to develop with typical DS software and has poor fidelity [8]. “Lighting” has limitations in light resolution, luminosity, colorimetry, rendering frequency, and light distribution effects because of the DSs limitations based on graphic engines and rendering devices [39]. In addition, the research conducted on the DS experiment under nighttime lighting conditions does not include the lights sources behind the drivers [40]. In other words, there is a difficulty in representing the reflection of interior lighting [40]. Recently, there are improvements for accurately simulating lighting through HDR rendering and display [37]. In the case of the “Vehicle” variable, it can be developed with typical DS software, but it can have a wide range of vehicle types that do not match the reality of the country of VRSAs candidate site [6]. “Pedestrian” and “Bicycle” also have the same limitations. In addition, interaction agents (e.g., pedestrians) must act realistically, and it requires a separate field investigation [37]. In the case of “Accident” and “Hazard event”, since crash types and many variables may be related, separating the overall event may be impossible from an experimental design and cost perspectives [6]. In other words, a whole accident or hazard event must be developed as one scenario; thus, there are limitations in terms of time, cost, and data collection. Previous studies conduct only one hazardous situation like the elderly pedestrian situation, fog situation, and movement of vehicles around the area [37,41,42].

4.3. Discussion

Previous VRSA research mainly consisted of framework development and practical case studies. Therefore, there is a limitation in that there is no effort to determine what elements of the road environment can be items of VRSAs and which road environment factors are appropriate to audit with VRSAs. Therefore, this study identified and categorized the road environment components as VRSA experimental variables. The recommendation of these variables for VRSAs was evaluated through a DS experiment and field review. In conclusion, static road environments (tunnel, bridge, underpass, footbridge, traffic island, sign, lane, road marking, traffic light, median barrier, road furniture) and dynamic traffic conditions are recommended as the VRSA variables. On the other hand, as non-recommended variables, there are static road environments (road pavement, drainage, lighting) and dynamic road environments (vehicle, pedestrian, bicycle, accident, hazard event). The result of this study is similar to that of the study evaluating VRSA experimental variables using the AHP method. In addition, previous studies have mentioned non-recommended variables as a limitation of the DS experiment or a problem to be overcome in the previous study.

The participants were four experts; therefore, the experiment results are challenging to generalize, and their scientific value is less significant. Nevertheless, this study was the first attempt to conduct a DS experiment and field review of VRSAs. Moreover, it can evaluate the visual experiential validity of performing VRSA using the DS simulator. Experiential validity refers to whether it produces a sense similar to what can be experienced in real situations [37]. In other words, this study is valuable for evaluating the empirical similarity of experimental variables between driving in a virtual environment and driving in the real world. In addition, this study specifically presented the variables of VRSAs by categorizing them into static and dynamic road environments, which may be meaningful because the study evaluates them.

4.4. Limitations and Future Researches

The response’s representativeness may be a problem because only a few experts answered the questions [33]. In addition, the evaluation method of the NPS itself also has a limit. Since NPS relies on a single customer metric, it may be necessary to apply a multidimensional approach to predict customer behavior and measure their loyalty [43,44].

The following are possible future research directions to overcome these limitations. The number of participants in the experiment should be expanded to ensure representativeness and the experimental participants should be expanded to experts and general road users. In addition, a follow-up study should perform a survey on the same participants and calculate the test-retest reliability coefficient to measure the internal consistency reliability. In addition, a multidimensional approach survey analysis should be performed.

5. Conclusions

In this study, experts evaluated the willingness to recommend experimental variables by comparing DS experiments and field reviews for sustainable road safety conditions. The willingness to recommend is based on the NPS evaluation, and interviews with experts identified the evaluation reasons. The results are summarized as follows.

Firstly, “Tunnel”, “Bridge”, “Underpass”, “Footbridge”, “Traffic island”, “Sign”, “Lane”, “Road marking”, “Traffic light”, “Median barrier”, “Road furniture”, and “Traffic condition” are the recommended variables. These can be realistically developed on a DS display; in addition, they have the advantage of being installed and changed quickly by the traditional DS software. Road characteristics variables are essential for vehicle behavior and driving path analysis at junctions and diverging areas, curved sections, and intersections; similarly, traffic condition variables can assess the impact of traffic congestion.

Secondly, the non-recommended variables are as follows: “Road pavement” is well developed visually in DSs, but its steering and vibration realism is poor. “Drainage” and “Lighting” have poor fidelity and are also challenging to render precisely with traditional DS software to reflect the real world. “Vehicle”, “Pedestrian”, “Bicycle”, “Accident”, and “Hazard event” are not recommended either. These dynamic variables require much time and cost in developing a scenario; moreover, it is not easy to develop a realistic scenario because data for the simulator scenario do not exist.

Finally, the study suggests the recommended variables and decision-making considerations for scenario development in conducting sustainable VRSAs in the future.