Vulnerable Road User Safety Using Mobile Phones with Vehicle-to-VRU Communication

Abstract

Pedestrians, bicyclists, and scooterists are Vulnerable Road Users (VRUs) in traffic accidents. The number of fatalities and injuries in traffic accidents involving vulnerable road users has been steadily increasing in the last two decades in the U.S., even though road vehicles now have perception sensors like cameras to detect risk and issue collision warnings or apply emergency braking. Perception sensors like cameras are highly affected by lighting and weather conditions. Cameras, radar, and lidar cannot detect vulnerable road users in partially occluded and occluded situations. This paper proposes the use of Vehicle-to-VRU communication to inform nearby vehicles of VRUs on trajectories with a potential collision risk. An Android smartphone app with low-energy Bluetooth (BLE) advertising is developed and used for this communication. The same app is also used to collect motion data of VRUs for training. VRU motion data are smoothed using a Kalman filter, and an LSTM neural network is used for future motion prediction. This information is used in an algorithm comparing Time-To-collision-Zone (TTZ) for the vehicle and VRU, and issues driver warnings with different severity levels. The warning severity level is based on the analysis of real data from a smart intersection for close vehicle and VRU interactions. The resulting driver warning system is demonstrated using proof-of-concept experiments. The method can easily be extended to a VRU collision-mitigation system.

1. Introduction

A significant number of traffic accidents occur each year and result in a large number of injuries and deaths [1]. To reduce these incidents, researchers have introduced new and better active and passive protection measures each year to improve passenger and other road user safety [2,3,4]. In traffic, pedestrians are among the most vulnerable road users (VRUs) [5,6,7], and pedestrian safety is a priority for the U.S. Federal Highway Administration’s (FHWA) Office of Safety [8]. The City of New York, for example, is testing new technologies to improve the safety of pedestrians [9]. According to National Highway Traffic Safety Administration (NHTSA) statistics, more than 6000 pedestrians are killed in traffic accidents every year in the U.S. Although pedestrians are the most common VRU type, other VRUs, such as bicyclists [10] and scooter users [11], are also very susceptible to high-collision-risk scenarios and crashes.

To improve the safety of VRUs as they interact with autonomous and non-autonomous vehicles, an important tool that can be utilized is vehicle-to-everything (V2X) communication [12,13,14,15]. This can be achieved on an individual basis, where each road user carries a message-broadcasting device, and on an infrastructure basis, where an intersection is designed as a smart intersection that detects VRUs as well as vehicles, and publishes information for them as if they are all connected [16,17], or through the over-the-air sharing of relevant Connected Vehicle (CV) data in centralized servers. Although an automated vehicle can obtain more accurate information about the VRUs using its own sensors, there are conditions in which this is not reliable or possible, such as bad weather (heavy fog, rain) or a no-line-of-sight (NLOS) situation in which the target of interest is blocked by an obstacle or another vehicle [18]. When the information is transferred via wireless communication, these types of situations can be remedied. Since Bluetooth advertising mode and Bluetooth 5.0 technology are now more widely available in current mobile phones [19,20], it is now a feasible communication method between VRUs and other road users [21]. VRUs can carry their mobile phones in their hands or in their pockets, which affects the range of communication. Current mobile phones possess a large number of sensors that can be used for tracking or predicting the motion and behavior of pedestrians, bicyclists, or scooters [22,23]. Using, for example, the Android Bluetooth Low Energy (BLE) application programming interface (API) [24], it is possible to develop an Android app that can be used for this purpose. It is possible to achieve different benefits when using this approach, depending on the different modes of Bluetooth communication [25].

Pedestrian safety systems need knowledge of the possible future path of nearby pedestrians to determine the possibility of a near-future collision risk [26,27,28]. Even with the correct information transfer from a pedestrian’s phone using Bluetooth or other communication means, predicting the path of a VRU can be tricky, especially for pedestrians [29]. Different types of methods, such as grid-based ones [30], constant velocity models [31], and neural networks [32], have been utilized in the literature to predict pedestrian paths. Generative Adversarial Networks (GANs) [33] and Long Short-Term Memory (LSTM) networks [34] have been used and observed in the literature to provide reliable results for prediction. The memory-based feedback nature of the LSTM network helps with the motion patterns of pedestrian movements, especially at intersections. The prediction method can be combined with the data obtained from sensors or wireless communication. One problem posed here could be the low-frequency update rate of normal GPS sensors of 1 Hz if a pedestrian is the source of information. However, the GPS, acceleration, and gyro sensors available on a phone can be utilized with a Kalman Filter [35] for higher-frequency location-tracking. Utilizing the sensors in a pedestrian’s mobile phone is very beneficial when combined with Bluetooth advertisement capabilities. This approach helps ensure information transfer between road users and, most importantly, allows us to implement a safety application that can be installed conveniently on mobile phones and can be used by both the driver of a vehicle and a VRU.

This paper focuses on VRU safety using V2P communication. The methods and results presented in the paper are applicable to all VRUs, but the experimental results reported here are for pedestrians. The contributions of the paper are as follows:

• The development of an Android app for collecting mobile phone sensor data of VRUs;
• Using the collected data to predict the future motion of a VRU using Kalman filtering and LSTM-based motion prediction;
• Converting the future motion and localization data of the VRU to a Personal Safety Message (PSM);
• Broadcasting the PSM using Bluetooth advertising or over-the-air using internet connectivity to nearby vehicles;
• Further developing and using the same Android app to receive PSM data in a vehicle;
• Developing and implementing programs in a Bluetooth board to similarly broadcast and receive PSM data if needed;
• Developing and implementing a pedestrian collision warning system using the PSM data;
• Analyzing available vehicle and pedestrian interactions in a smart intersection to calibrate the pedestrian collision warning system;
• Experimentally developing and demonstrating a full pedestrian collision avoidance system.

In this paper, V2P communication is used to improve the safety of VRUs, especially pedestrians, using a driver warning system. The VRU mobile phone sensors are used to predict future motion to better assess the risk of collision with nearby vehicles and issue warning messages. These warnings can also be used to begin the automated slowing down and braking of the vehicle. The organization of the rest of the paper is as follows. The V2P app’s development and implementation are presented in Section 2. V2P communication using the developed app is investigated in Section 3. The V2P-communication-based pedestrian safety system is developed in Section 4. Simulation and experimental results are presented and discussed in Section 5. The paper ends with conclusions and recommendations in Section 6.

2. V2P App Development and Implementation

To the determine collision risk and execute the necessary safety precautions, pedestrian motion information needs to be obtained by the vehicle. By transferring this over a wireless link, we also ensure that, even if there is NLOS and perception sensors are not able to detect the pedestrian, the vehicle would still be able to detect the pedestrian and track their location, speed and direction. Therefore, the main enablers of this implementation are its relatively wide availability and the Bluetooth 5.0 capabilities of recent Android phone models. However, it should be noted that the algorithms and methods developed in this paper for vehicle and pedestrian interaction analyses, driver alerts and warnings, pedestrian motion and intent prediction, and collision risk determination are independent the communication method that is used.

Bluetooth communication technology was experimented with to analyze the practical abilities of its implementation in smartphones. First, communication experiments were carried out on the nRF52840-DK Bluetooth development board [36]. Afterwards, an Android application was coded using the BLE module in Android API 28. This software has the ability to encode, transmit, receive, and decode messages via Bluetooth 5.0 through extended advertisement. PSM from the Society of Automotive Engineers (SAE) J2735 standard [37] was used to establish the pedestrian information message structure. Experiments were first carried out using the extended advertisement mode to achieve a long-range and large data-carrying capacity, and very good results were obtained. The mode of communication was later switched to normal advertisement, as this is available in more smartphones. As well as using the location, speed, acceleration and heading fields to calculate the collision possibility and issue a warning, PSM also has fields such as user type, device use state, cross request, cluster size, and attachment. These fields will be useful for improving the algorithm in the future for pedestrian group clustering, pedestrian behavior prediction, and more accurate collision predictions for a VRU with pets, carts, or a wheelchair. The standard is designed to allow for this message to be published at 10 Hz, which is a reasonable frequency for our calculations. The pedestrian transmits the PSM, and the vehicle receives it via the driver warning implementation. The PSM communication between the VRU and vehicle(s) is illustrated in Figure 1. The PSM-populated fields are the location (latitude and longitude), heading, speed, and acceleration, as shown in Figure 2.

An Android phone can be used in the vehicle to obtain the pedestrian’s information, handle the warning calculations, and show a warning and the recommended speed profile (if desired) to the driver. If the vehicle is autonomous, a Bluetooth receiver board such as the nRF52840-DK can be used to obtain the messages and transfer the information to computing unit(s) for comfortable braking and stopping, as well as for collision avoidance. This board has a serial interface. This was combined with a W5500 Ethernet shield [38] to transfer the information received via Bluetooth through the Ethernet connection to achieve better results. A picture of the combined BLE-receiving board can be seen in Figure 3.

3. V2P Communication Experiments

After the implementation of V2P into the smartphone, some initial testing for determining communication range was conducted. This testing involved both nRF52840-DK Bluetooth boards and Android smartphones in obstructed and non-obstructed cases along with normal and extended Bluetooth advertisement modes. The results obtained are presented in

Table 1. Bluetooth communication range experiments.

Device	Condition	Distance in Normal Advertisement (m)	Distance in Extended Advertisement (m)
nRF52840-DK	No obstructions	191	253
Android	No obstructions	78	114
Android	Intentionally obstructed by pedestrian body	27	55

. The Bluetooth board has the longest communication range, of more than 250 m, in the extended advertising mode. This corresponds to the first row of the table. The obstruction of the smartphone by the pedestrian body results in the lowest communication range in normal Bluetooth advertising mode when communicating with another smartphone in the vehicle. This corresponds to the last row of the table. Data transfer over WebSocket [39] functionality was also implemented in the application. However, this was changed later to internal storage during the experiments because the WebSocket connection required constant internet access. Moreover, due to GPS sensor updates being paused when the phone screen is locked, GPS callback was tied to a background process to ensure regular updates. The process flow diagram of the application is shown in Figure 4.

Along with the V2P broadcasting mobile application discussed above, a data collection application was developed and implemented in the Android phone. This application was developed separately from the previous app to support older model Android phones which do not have Bluetooth 5.0. The application, running on an Android phone, is displayed in Figure 5. This application receives data from several sensors built in the Android phone and records them locally with timestamps in the form of csv files. The sensors used are acceleration, gyro, step counter, GPS, orientation, proximity, and light sensor. A feature for uploading the data into a remote server was also implemented. Data plots after initial testing of the app are shown in Figure 6.

4. V2P-Communication-Based Pedestrian Safety Warnings

The two applications that were previously discussed regarding V2P communication and data recording were later combined into a single application built for recent Android phones with Bluetooth 5.0. The user interface was also improved. Then, a custom J2735 library was prepared to encode and decode PSM messages. Several quality-of-life changes were implemented later. These include manual or automatic user selection, manual or random PSM ID, easier file access for recorded data, a debug mode for developers, and a user mode for end-users. The new app screen is shown in Figure 7. PSM decoding, encoding, broadcasting and data collection were combined in this final application. The user interface was simplified and improved. Icons were added to warn the driver in case of a collision risk. The warning design, parameters and the user interface are discussed in more detail later in the paper.

4.1. Safety Approach

The overall architecture of the V2P-communication-based pedestrian safety approach of this paper is illustrated in Figure 8. The vehicle is approaching a crosswalk that a pedestrian is intending to cross in Figure 8. The elements that make up this architecture are explained. The first element in the approach is pedestrian path prediction and tracking. This is utilized to predict the pedestrian’s location in the near future with LSTM neural networks, as well as to track his/her motion at a higher sampling rate utilizing the Kalman filter. This is used to make safety warnings and action decisions. These include how to warn the driver or how to adjust the speed of the vehicle if it is autonomous.

The second element in the architecture, communication, or information transfer focuses on wireless communication to provide information about the pedestrian, especially in cases involving reduced visibility or NLOS, to the nearby vehicle. Mobile phone Bluetooth or extended Bluetooth are used in this paper due to the enhanced support they provide when carrying larger chunks of information over longer distances. On top of that, with the dual-frequency GPS technology that is recently starting to be implemented in mobile devices, localization accuracies of a few decimeters can be achieved [40]. Most importantly, mobile phones are very common and can be utilized in non-autonomous vehicles, as well to warn the driver. Because of this, the means of wireless communication, as a way to discuss and demonstrate the VRU safety approach, is chosen as Bluetooth in this paper. However, it should be noted that the VRU collision warning and avoidance braking methods of this paper do not depend on the communication method used.

The next two elements of the architecture, real-world data analysis and driver warnings, are intertwined. The warning system was designed to allow for non-autonomous vehicles to utilize this safety system by showing the driver warnings, of varying degrees of severity, of a collision risk situation. Real-world data analysis plays a very important role in determining the severity of these warnings. An analysis of real interactions between vehicles and pedestrians at an intersection was used to achieve a naturalistic slowing down and stopping of the vehicle by providing the driver with the correct warnings at the right times.

4.2. Pedestrian Path-Tracking and Prediction

The path-tracking and prediction system is designed to be a combination of Kalman filtering and neural-network-based path prediction. While neural-network-based prediction predicts motion information multiple seconds into the future, Kalman filtering increases the frequency of location to above the 1 Hz GPS update rate in mobile phones. The diagram of the neural network and the Kalman filter path-tracking and prediction algorithm is shown in Figure 9. The mobile phone GPS sensor is utilized at its maximum rate of 1 Hz to obtain location and velocity information, which are then fed into the LSTM neural network that predicts the future location of the pedestrian. Although a single neural network can be used to predict multiple locations of the pedestrian path, multiple neural networks were trained separately to ensure a better prediction accuracy, along with a smaller network size. Therefore, the algorithm uses multiple small neural networks to predict future location within a timeframe of a few seconds. Naturally, the further in the future the algorithm tries to predict, the less accurate it becomes. Along with the neural network prediction, the Kalman filter uses GPS information at 1 Hz, as well as other higher update rate sensors such as orientation and acceleration, to track the pedestrian location at higher sampling than 1 Hz.

The LSTM type recurrent neural network was used because of its memory, since pedestrian movement patterns highly benefit from both short- and long-term memory, especially at an intersection. LSTM was also tested in other research in the field [34] and shown to perform better for pedestrian path prediction compared to other options. These networks were trained with both real-world data collected from mobile phones and synthetically generated pedestrian data from the Vissim environment. Some of these samples are presented in Figure 10. The trained neural network can be transferred into mobile devices using the TensorFlow Lite library for runtime operation [41].

4.3. Pedestrian Behavior Prediction

Android phones have numerous sensors that provide high-frequency data. Some of these sensors were utilized to predict pedestrian behavior. An LSTM-based neural network structure was created, with these sensor readings used as multiple inputs. The first structuret that was experimented with had a single output with values ranging between 0 and 1. Different intervals within this range were trained as different behaviors. This structure, along with the inputs and outputs, is illustrated in Figure 11.

To train the neural network of Figure 11, real-world data were collected with the data collection application that was explained earlier. Afterwards, the data were manually labeled by utilizing a custom data-labeling tool, developed in MATLAB version 2022 specifically for this purpose. Using the sensor information visualization features of the labeling tool, the labeling process was significantly shortened. The network was trained using the labeled data and then tested. The first designed structure, as mentioned before, had a single output with different value intervals indicating different behavior. With repeated experiments, the sensors that resulted in unnecessary noise or inconsistent measurements and that provided minimal benefit to the detection, such as acceleration and step count sensors, were removed from the input for better accuracy. Figure 12 shows the testing results of the trained neural network structure.

As indicated in Figure 12, different colored areas indicate different behavior. When the output is between 0.5 and 0.75, the pedestrian is making a right turn. If the output is between 0.5 and 0.25, the pedestrian is making a left turn. Finally, an output between 0 and 0.25 means the pedestrian is moving forward. The graph in Figure 12 shows that the prediction results (green line) are very close to the ground truth (blue line), and always fall within the range of correct behavior. The second structure that was experimented with was created by modifying the first structure to incorporate multiple outputs to increase learning capacity, scalability, and compatibility with the literature. These outputs were then assigned to different modes of behavior, and interpreted as confidence levels. The network was trained with same data as before, although these data were now labeled for multiple outputs. The test results shown in Figure 13 have three different outputs, as opposed to one output in the previous figure. Each output has a confidence level between 0 and 100. The predicted behavior has the highest confidence level value. It can be seen that the prediction (orange line) follows the ground truth (blue line) closely. Although there is some noise in certain sections, the predictions were correct in almost all the test data.

4.4. Pedestrian–Vehicle Interaction

The base scenario geometry for pedestrian–vehicle interaction assumes that the vehicle and the pedestrian are moving straight along paths that are perpendicular to each other. This is similar to one of the configurations in a recent National Highway Traffic Safety Administration (NHTSA) report on the causes of accidents [8]. The base scenario geometry and variables are illustrated in Figure 14. The collision risk zone is defined at the crosswalk as an area of potential conflict, where vehicle speed is $v_v$, pedestrian speed $v_p$, and distances to the collision zone $d_v,$ and $d_p$, for the vehicle and pedestrian, respectively, are defined as the significant variables. This configuration was created as a base scenario to be used throughout the paper for driver warning discussions. The scenario geometry can easily be modified to include left and right turns, as well with similar variables, where distance will be circular rather than linear. Using the variables mentioned above, Time to Zone (TTZ) can be calculated as:

$$TT{Z}_v=\frac{d_v}{v_v}\hspace{1em}TT{Z}_p=\frac{d_p}{v_p},$$

(1)

where ${TTZ}_v$, and ${TTZ}_p$ are vehicle and pedestrian time to zones, respectively. The calculations for pedestrians can be combined with the tracking and prediction algorithm to calculate a more accurate ${TTZ}_p$, especially if the pedestrian is about to turn onto the crosswalk. Depending on how many seconds into the future the location is predicted, ${TTZ}_p$ can be completely replaced or extrapolated using a constant-velocity or other, similar motion model [31].

After the TTZ values are obtained, we can state the high collision risk condition as

$$TT{Z}_p-t_s<TT{Z}_v<TT{Z}_p+t_s,$$

(2)

or

$$\left|TT{Z}_v-TT{Z}_p\right|<t_s.$$

(3)

$t_s$ is the safety margin time, and is either constant or a dynamic value based on GPS accuracy and driver preferences. If the conditions in Equations (2) or (3) are satisfied, there is a possible collision in the future. This condition is further investigated in the next subsection where this correspondence is visualized, and parameters are defined and determined using real-world interaction data. TTZ can be calculated and used to determine a possible collision risk in the future and its severity. For each second from the start of the interaction (depicted as interaction time), there will be ${TTZ}_v$ and ${TTZ}_p$ values. An interaction diagram, such as that presented in Figure 15, can be created using the correspondence of TTZ values for the pedestrian and the vehicle.

TTZ values for the vehicle and pedestrian are shown as small colored circles in the example in Figure 15, where the color of the circle determines the interaction time, with the legend provided on the right-hand-side vertical axis. Since both TTZ values are equal to zero, i.e., an interaction at the origin means collision, the closer the correspondence for the ordered pair (${TTZ}_v$, ${TTZ}_p$) is to the origin, the more severe the collision risk. Severity margins can be defined as circular areas around the origin indicating different levels of severity. Defining the severity margins is highly important for non-autonomous vehicles, since the severity of the situation should be properly conveyed to the human drivers to allow for them to react correctly in a timely manner. Therefore, values for these margins were used as the main design parameters for the warning calculation. Another important parameter is the safety margin time, which represents the difference between the TTZ values for the vehicle and the pedestrian, like $t_s$, defined in the previous subsection. If the TTZ values are reasonably different from each other, either the vehicle or the pedestrian would cross the zone first, without any risk of collision. Therefore, the warning system only considers the correspondences that occur within this safety margin time region in Figure 15.

To better visualize how the interaction graph would look with corresponding vehicle and pedestrian behavior, Figure 16 was created as an illustration. A vehicle approaching the crosswalk at the same time as a pedestrian is illustrated in Figure 16. Steps 1, 2, and 3 represent the vehicle and pedestrian locations as the vehicle is slowing down. The vehicle TTZ is shown to be moving towards zero at first, but because the vehicle is slowing down, the values start to increase and the corresponding points move beyond the critical part of the diagram as the vehicle stops. The final image at the right side of Figure 16 shows the overall TTZ correspondence trajectory.

4.5. Real World Data Processing

To gain insight from real-world interactions, data from a real-world intersection at Marysville, Ohio, U.S., were collected and processed. This intersection is a smart intersection where four cameras were set up [16] to detect vehicles and pedestrians and send this information to the roadside unit (RSU) installed at the intersection. This RSU broadcasted basic safety messages (BSM) for each pedestrian as well as for each vehicle, along with their respective location and speed information, as if they were carrying a dedicated short-range communication (DSRC) broadcasting device. Although these data are very large and would not be practical to physically listen to and record, the City of Marysville made them available for public use through the Smart Columbus Operating System (SCOS) database [40]. The data were downloaded and processed to extract the vehicle and pedestrian interactions that fit the base scenario. This data contained eight months of BSM messages; some months contained more interactions than others.

Extracted interactions were processed to calculate TTZ for interacting vehicles and pedestrians. Then, interaction graphs were created showing all these interactions combined. To understand the similarities within all the interaction data, a heatmap graph was also created. These graphs can be seen in Figure 17. The interaction graph on the left side of Figure 17 shows each different interaction with a different color. Since the number of interactions is very high, it is difficult to interpret this, and a vehicle–pedestrian interaction density heatmap is used for better readability on the right-hand side of Figure 17. The yellow area shows the densest points in the graph. When the vehicle is slowing down, vehicle TTZ stays at around similar values for a while as pedestrian TTZ steadily decreases. The dense area in the heatmap graph corresponds to the area in which vehicles are slowing down when they see a pedestrian close to the intersection. Using this information, captured from real-world data, allows for us to design our warning system to mimic the natural behavior of drivers when they encounter nearby pedestrians.

Interactions were divided according to the sunset time for each respective month, and are plotted again in Figure 18 for a comparison of night and daytime interactions. As seen in this figure, drivers at this intersection behaved more cautiously at nighttime, resulting in an approximately 0.5 s shift in the dense area in the heatmap. This information can also be incorporated into safety algorithms, either as a preference or to provide a more comfortable and cautious experience during the evening and night.

Using the design parameters and real-world behavior information discussed in the previous section, driver warnings were designed for a safe stop. The warning system was designed with different severity values to convey the level of danger to the driver. This will help the driver decide how to slow down and stop. The main criteria used to determine the severity of the warning are the severity margins discussed in the previous section. With the current approach, which uses the TTZ as the main criteria to determine the current severity, we incorporate speed along with the remaining distance, as well as the TTZ correspondence for the pedestrian and vehicle. This prevents us from warning the driver unnecessarily in cases where either the pedestrian or the vehicle will be able to pass the crosswalk without the need to stop. Three margins were defined for the driver warning—Inform, Warn, and Emergency—with increasing degree of severity. The color-coding and explanations of these margins are shown in Figure 19.

Initially, there are no warnings, even though there may be the possibility of a collision in the distant future. As the risk of a future collision becomes higher, and as the value of ${TTZ}_v$ reduces below a pre-determined threshold value, the warning system enters the Inform state. The Inform state corresponds to the lightest severity, where a future collision could easily be avoided by a comfortable speed reduction. As the risk of future collision increases, and as the ${TTZ}_v$ value decreases below another pre-determined threshold, the system enters the Warn state, which means that the vehicle should start slowing down as soon as possible, using a relatively uncomfortable amount of deceleration to stop safely. These two margins can be chosen as constant values, and can later be modified adaptively or manually by the driver. To determine the value of these margins, the processed real-world behavior data that were discussed above were used. A histogram graph of the data was used to determine a base value ranging from 1 to 3.3 s for the Inform margin. After determining the base value for this first margin, a Vissim simulation environment with the base scenario was constructed to further test and adjust this value in simulations. An attention lag was also added to incorporate the driver reaction time. The test was conducted with a very comfortable deceleration to obtain a value that results in a conservative driver warning. The general flow of the simulations is illustrated in Figure 20.

As stated above, the simulations were conducted at a constant start speed and time where the only changing values were the random attention lag, and the Inform margin value. Interaction diagrams for multiple simulations, run across different Inform margin start values, are shown in Figure 21. When the diagram data were shifted by 1 s (diagram on left), there appear to be certain cases when the vehicle cannot stop in time with comfortable braking and needs to resort to emergency braking. When the data were shifted by 1.5 s (graph in the middle of Figure 21), although there were no accidents and vehicles were able to slow down and stop comfortably, an unexpected increase in reaction time still resulted in some unwanted situations. Therefore, the value shifted by 2 s (diagram on the right side of Figure 21) was selected, as this is the most conservative value, while still being close to the base value. This adjustment brought the Inform margin into the range of from 3 to 5.3 s.

After the Inform margin is determined with both lower and upper threshold values, the Warn margin (pink area) does not need a range determination, since this becomes the area between the dynamically calculated Emergency margin and the constant Inform margin. The third and final severity margin is the red Emergency margin. In this case, the vehicle must perform an emergency braking to stop safely. The threshold value that is used will be dynamic. It will depend on the vehicle speed and distance. It should be computed at each step to issue a warning to the driver, if necessary. The minimum amount of deceleration $a_{min}$ that is needed to stop the vehicle at a given distance from the conflict zone at the given vehicle speed can be calculated as:

$$a_{\min }=\frac{v_v^2}{2d_v}$$

(4)

When the minimum deceleration is below the braking deceleration limit, the driver is required to apply full braking to stop the vehicle. This will result in the red Emergency severity zone.

5. Simulation and Experimental Results and Discussion

This section discusses the evaluation and testing of the VRU safety warning system that was developed. The evaluations include pedestrian path prediction evaluation, Vissim simulations, and real-world testing.

5.1. Kalman Filter and LSTM Pedestrian Path Prediction Testing

As discussed previously, a system with LSTM neural networks in combination with Kalman filtering was developed for pedestrian path prediction and tracking. The test results and discussion are presented here, starting with the Kalman filter. The Kalman filter was first implemented in MATLAB and tested using real-world data recorded from an Android mobile device. Android location data from GPS, and acceleration data from its accelerometer sensor, were integrated together to achieve higher-resolution tracking. Since the mobile device GPS updates at 1 Hz, the use of the Kalman filter is highly beneficial for the calculations and for broadcasting messages, since the SAE J2735 standard messages are published at 10 Hz. Initial testing was conducted offline by replaying the recorded data through the algorithm. Some of the test results are plotted in Figure 22. GPS points are represented by blue circles, while Kalman filter updates are represented by red cross signs. As seen in Figure 22, the filter can successfully track the pedestrian at a higher frequency with fewer errors. Error calculations were also carried out to determine the accuracy of the algorithm, and the results were found to be satisfactory. The values are presented in

Table 2. Tracking error for Kalman filter testing.

Test	ADE (m)
A	0.2866
B	0.2112
C	0.3109

as the Average Displacement Error (ADE) in meters. After initial testing, the Kalman filtering algorithm was implemented in the Android device and was tested in a real-world setting. The results are plotted on top of a satellite image and are shown in Figure 23. As seen in the figure, the algorithm can provide higher-resolution location information compared to the Android GPS sensor. More importantly, it prevents sudden jumps in location, which can be seen in the top right corner of the plots.

The results in Figure 23 show that future motion prediction has problems when the pedestrian makes sudden turns at a relatively high speed. A constant turn or another motion model may be more useful in those cases than a constant-velocity model during Kalman filtering. Errors in phone-GPS-based location also cause problems and will influence the TTZ computations, which may result in an inadequate warning. Tuning the TTZ based warnings based on real data will help improve this. It is also expected that the localization accuracy of mobile phones will keep improving, meaning that this will be a smaller problem in the future. The possibility of these errors needs to be considered in real-world deployments.

The LSTM neural network for pedestrian future path prediction was trained and tested offline using data collected with an Android mobile device. Pedestrian location and speed data were collected and fed to the network to test 1 s future prediction first. The validation data were not included in the training dataset. The results for some of the tests are plotted in Figure 24 and Figure 25. Collected pedestrian path test data are plotted as a blue line, while the neural network predictions for each step are plotted on top of these data as an orange line. The predictions of 1 s into the future are very accurate for both test cases: A and B. Along with the path prediction plots, the calculated prediction error is presented in

Table 3. Prediction error for LSTM.

Test	ADE (m)
A	0.1132
B	0.1556

. The error is smaller than that obtained with the Kalman filter.

5.2. Vissim Simulation Testing

A Vissim simulation environment was constructed with the base scenario geometry discussed above. As the vehicle speed is controllable in the simulation, several aspects of the driver warning design during slowing down and stopping were tested. First, speed information from real-world data was extracted and fed into the simulation to test the safe stop algorithm. The main idea of this simulation test was to see if the interactions obtained from the simulation were like the real-world interactions. Some differences were expected, since the design of the timings was more conservative, and deceleration was set to a very comfortable, constant, 0.15 g value in the simulations. Finally, the simulation structure was created with the flow illustrated in Figure 26. Vehicles start with a random speed profile selected from a random real-world vehicle in the data, where the Inform state is issued at 5.3 s. This is followed by a random attention lag. At the end, the vehicle starts to slow down and stops with 0.15 g deceleration.

Real-world speed data, along with measurement noise, helped to ensure the observed behavior was similar to the real-world data-processing. Extracted interactions from the simulation results are shown in Figure 27. One main feature is obvious when we look at the graph on the left compared to the previously discussed real-world data displayed in Figure 17. Vehicle TTZ values do not come as close to zero as those in the graph, obtained from real-world data. This behavior indicates more conservative behavior. On the right side, in the heatmap graph, the area of dense points is like the real-world counterpart, while they are also farther from the origin point than the real data, indicating more conservative vehicle behavior. The more conservative but similar behavior observed in these simulations shows that the fundamental approach is correct. This method is useful for capturing the interactions, and can be modified to obtain a better fit later, if desired.

The Vissim simulations and real-world data are qualitatively compared in Figure 17 and Figure 27. The results were observed to be similar. Quantitatively, the vehicle–pedestrian interactions never entered the red emergency zone in Vissim, while this happened many times in the experimental data. The yellow area of high density in the real data showed that vehicles kept about 2 s of TTZ when a pedestrian was at the intersection. In contrast, Vissim vehicles kept a TTZ of about 4.5 s in the same situation, showing the conservativeness of the Vissim simulations. This should be adjusted in future work. It is also recommended to use real data in Vissim simulations in the future for the better replication of real-world experiments.

All three states of Inform, Warn and Emergency were enabled in the simulation to test the design parameters at different reaction times, as shown in Figure 28. The three different driver reaction times of 0.75 s, 1.25 s, and 2.5 s were tested with a random starting speed and random deceleration within a determined range, to see how reaction times affect the safe approach speed and what the limits are. The results are shown in Figure 29, where each graph shows multiple simulations with the vehicle starting at different speeds (y axis). The circles correspond to the severity state that caused the vehicle to stop. If the circle is gray, this means the vehicle comfortably stopped in the Inform state. If it is pink, stopping was a bit harder and the vehicle stopped in the Warn state. Lastly, if there is a red circle, this means the vehicle had to perform an emergency brake and stopped suddenly in order to not hit the pedestrian. With the current values for the design parameters, in terms of the maximum approach speed for safe stopping, without resorting to emergency braking, a 0.75 s reaction time results in approximately 100 km/h, a 1.25 s reaction time results in approximately 85 km/h, and a 2.5 s reaction time results in approximately 50 km/h when approaching the collision risk zone. Given that 1.25 s is a reasonable value for the average reaction time of a driver, the maximum approach speed for the current safe stop design is seen to be reasonable.

5.3. Real-World Testing of Driver Warning System for Pedestrian Collision Avoidance

The mobile phone app was used to test the driver warnings in the real world. We created an NLOS situation at the uncontrolled intersection behind our lab building, illustrated in Figure 30, to demonstrate the efficacy of generating driver warnings for pedestrian collision avoidance. The blue arrow in Figure 30 shows the vehicle’s motion towards the intersection while the green arrow shows the pedestrian’s motion. The orange rectangle in Figure 30 shows the collision risk zone, which is used to calculate the TTZ values for the vehicle and the pedestrian. The vehicle had one phone and the pedestrian also had a similar phone, with both running the application during the testing. The app was used for communication between the phones, for driver warning computations, and for recording data. The forward-looking camera inside the vehicle was used to record the driver’s view and the heads-up display (HUD), which was used to display the pedestrian safety warnings. Figure 31 shows different instances of the test, as recorded by the forward-looking camera.

The HUD is shown in Figure 31, with the phone screen facing upward and reflecting the driver warning generated by the app. The numbers from 1 to 4 correspond to different instances of the vehicle moving towards the intersection. The vehicle starts its forward motion while there is no warning, as shown in the first frame. The phone in the vehicle keeps receiving PSM over V2P, even during NLOS, as in frame 2, and a yellow-colored warning corresponding to the Inform state is displayed on the HUD. This indicates that there is a pedestrian approaching the intersection. The vehicle is closer to the intersection without slowing down in frame 3, when the warning becomes orange, corresponding to the Warn state. The vehicle stops in a safe manner as the crossing pedestrian is seen in frame 4. The HUD does not display a warning, as the vehicle has stopped, which successfully completes the real-world testing of the VRU safety application.

The data recorded by the phone app are summarized visually in Figure 32. The garage building forming the obstruction for NLOS is the gray rectangle in the middle of Figure 32. The pedestrian motion is shown with light blue circles. The vehicle positions are shown, with the color-coding corresponding to the warnings that were issued. Dark blue represents no warning. Yellow represents Inform and orange represents Warn. The vehicle starts with no warning while the pedestrian is in an NLOS position. The vehicle locations that are shown are its GPS measurements. The yellow warning is issued while the pedestrian is still behind the garage building. The orange warning is then issued as the vehicle moves closer to the collision zone without slowing down. The vehicle stops safely at the end, and warnings are disabled at the dark blue circle in Figure 32. The pedestrian continues their motion in a cross-path and passes in front of the stopped vehicle.

The experiment in Figure 32 had only one VRU, a pedestrian. Both mobile phones used, the one in the vehicle and the one held by the pedestrian, had some GPS localization errors, which were not large enough to affect the collision warning system in this experiment. VRUs carrying their mobile devices in their pockets have a shorter communication range, which would affect this NLOS experiment and reduce its performance. The vehicle would determine the presence of a dangerously moving pedestrian later in that case, and the driver would have to slow down faster. Multiple pedestrians and a combination of multiple pedestrians and bicyclists are also difficult situations as there will be more data. Currently, the VRU safety app calculates TTZ-based warnings simultaneously for each VRU, and the worst-case warning is used. Considering the combined dynamics of multiple VRUs may be useful in such cases to allow for the better prediction of future collision risk and reduce the computation time.

The Human–Machine Interface (HMI) used in Figure 32 was finalized for our proof-of-concept testing after many iterations, using different choices to improve the attentiveness of the driver. It was finally decided to use a very simple flashing graphical warning with no text information. The experiment shown in Figure 32 and other similar experiments were repeated many times and the driver reaction times were seen to be adequate. Future pilots and larger-scale deployments will enable us to obtain a better understanding of driver reaction times.

6. Conclusions and Recommendations

A V2P-communication-based driver warning system for pedestrian safety was developed and evaluated in this paper. The paper also considered possible problems and developed methods to remedy these problems. These methods were evaluated in several ways, including realistic simulation environments and real-world testing. By utilizing mobile phone sensors and Bluetooth communication on the Android smartphones, a V2P safety application was developed to broadcast information for pedestrian motion while providing warnings to the human driver, when necessary, using the developed driver warning system. This application allows for easy access to a pedestrian safety system for public use. The developed application was also used for data collection from the sensors on the Android phone. With these sensors, the capabilities of the data collection application, and the virtual environments created in Vissim, neural network structures were trained and experimented for pedestrian path and behavior predictions. Kalman Filtering was also utilized for higher-frequency pedestrian motion-tracking.

One of the key contributions of the paper was the utilization of real-world data to create a baseline for driver safety warning system design parameters, with the goal of achieving more natural interaction behavior. In addition to this, some interesting observations were made while comparing several aspects of driver behavior, such as approach direction or the time of day. It was observed that drivers were more cautious while approaching an intersection with a pedestrian at times when sunlight is limited or not available. The data that were analyzed and used had a span of several months and included a large amount of BSM messages that were published for all vehicles and pedestrians approaching or crossing the street at the chosen smart intersection, as well as signal phase and timing (SPaT) messages.

Among the current limitations that need to be further explored in detail is how to treat crowds of pedestrians (VRUs). Although the way in which the application is currently designed to work will not create problems in terms of providing a warning when there are multiple pedestrians, problems may arise due to the highly increased processing load and congestion in the Bluetooth communication band. Another limitation of the system that should be mentioned is the GPS accuracy of the Android phones. Although the accuracy was satisfactory for the provision of reliable data and warnings in many data collection and warning system experiments, this might not always be the case. The dynamics of the vehicle [41] on a road may also affect GPS accuracy. However, the Kalman Filter in the current implementation helps with this issue, especially for irregular jumps in the GPS location. One of the expected improvements is dual-frequency GPS. As dual-frequency GPS smartphones become more common, the GPS accuracy issue will be reduced. On top of this, RTK GPS chips are getting smaller and cheaper over time, while providers are building stations to publish RTK correction messages over the internet for better accessibility. With the RTK GPS technology becoming more widely available, the GPS accuracy for small devices like smartphones will be significantly improved. This will help to successfully implement the V2P-communication-based pedestrian safety system presented here.

As the work reported here is a proof-of-concept, only Android phones with Bluetooth connectivity were considered. The authors experimented with the different phones available to them, ranging from relatively old and simple ones to more recent ones with more enhanced capabilities. The Android application was developed and checked to make sure that it worked on different versions of Android and for the different mobile phones that were available to the authors. For real-world deployments with many users, an iOS implementation of the app is also necessary, considering the large number of iPhone users. There are also many VRUs who do not have a mobile phone or who use basic mobile phones, on which the app cannot be installed. It is recommended to use low-cost wearable devices that only act as a beacon broadcasting the presence of the VRU in such cases. Nearby vehicles should also use their perception sensors to detect and track VRUs in cases where the VRU does not have a mobile phone, or where the app cannot be installed on their mobile phone.

Communication delays between the VRU and vehicle mobile phones (communication devices) are also an important factor that needs to be considered. Here, the communication occurred at a 10 Hz rate, and the communication delay in the experiments reported here and the other tests that the authors conducted showed that this was not a serious problem for the Vehicle-to-VRU-communication-based VRU safety approach presented in this paper. All PSM messages are time-stamped such that the communication delay value is known when the message is received on the vehicle side. The use of a Kalman filter and the neural-network-based prediction of the future motion of the VRU and vehicle also help in alleviating the effect of the communication delay.

While this paper focused on a driver warning system, a collision avoidance system with collision-free trajectory computations and trajectory-tracking control [3] are also needed and will be the focus of future work. The parameter-space-based robust control methods [42,43] can be used for robust trajectory-tracking control for a collision mitigation system to reject disturbances in the presence of model uncertainty. Future work can also focus on the fused use of V2P communication and AV perception sensors [44], and the use of socially acceptable VRU collision avoidance in low-speed campus-like deployments [45].

Future work should focus on improving the applicability of this Vehicle-to-VRU-communication-based VRU safety system. An iOS implementation of the app should also be developed for iPhone users. Pilot and larger-scale deployments and further data analyses are necessary to improve the app for practical use and user acceptance. It will then be possible to make the app freely available to users. Android Auto [46] and Apple CarPlay [47] allow for mobile phones to be connected to the displays of current vehicles. It will be useful to use this connectivity to display VRU safety warnings using the vehicle display, instead of using an HUD. It would be best to further improve this method for automated braking, and braking and steering, to establish a more comprehensive VRU safety system in the future. The addition of communication devices that can communicate with both the vehicles and the VRUs at intersections will be helpful to improve the range of communication and reduce latency problems due to multiple simultaneous actors. In addition to Bluetooth and direct and over-the-cloud internet connectivity, ultra-wide band (UWB) connectivity can also be used [48].