Does the Condition of the Road Markings Have a Direct Impact on the Performance of Machine Vision during the Day on Dry Roads?

Abstract

The forthcoming arrival of automated vehicles (AV) on the roads requires the re-evaluation or even adaptation of existing infrastructures as they are currently designed on the basis of human perception. Indeed, advanced driver-assistance systems (ADAS) do not necessarily have the same needs as drivers to detect road markings. One of the main challenges related to AV is the optimisation of the vehicle–infrastructure pair in order to guarantee the safety of all users. In this context, we compared the performance of a vehicle equipped with an ADAS machine-vision system with a dynamic retroreflectometer during the daytime on a road section. Our results questioned the reliability of the literature thresholds of the luminance contrast ratio on a dry road under sunny conditions. Despite the presence of old and worn road markings, the ADAS camera was able to detect the edge lines in more than 90% of the cases. The non-detections were not related to the poor condition of the markings but to the environmental conditions or the complexity of the infrastructure.

1. Introduction

1.1. Context

Road safety is a key issue for all countries in terms of social and financial cost. Road markings are not expensive, easy to install and offer added value to users. In particular, they increase both visibility and the legibility of the road. Road markings are composed of a main layer usually made of paint or thermo or cold plastic tape. To allow the road markings to be visible at night on unlit roads, the main layer is usually covered with glass beads, which reflect the light from the car’s headlights onto the marking and then towards the driver. The characterization of road markings (visibility, colour, etc.) is currently defined according to the perception of human drivers. With the technological developments of automated vehicle (AV) sensors, are these characteristics still valid?

In order to deal with this issue, the French project SAM (Safety and Acceptability of Automated Mobility) is in progress. This project consists of developing knowledge to build a technical and regulatory framework to facilitate the circulation of automated vehicles on the French road network. One of the tasks of this project is to evaluate the detection of road markings by camera-based driving assistance systems through varying different parameters—both on the state of wear of the markings and also on the state of the road.

This paper presents a detailed analysis of the performance of markings, performed in conjunction with automated driver-assistance systems (ADAS) sensors on an experiment conducted during the day on a dry circulated road.

1.2. Road-Marking Visibility According to Standards

To be efficient, road markings must be visible by day and night and in various sunlight and weather conditions. The performance of road markings is controlled with some requirements given by the standards EN1436 [1] and ASTM2005 [2]. In particular, the visibility of the road markings is defined for human perception: it is based on the photopic visual sensibility curve.

By day, the visibility of road markings is mostly characterized by the luminance coefficient under diffused daylight $Q_d$ (expressed in mcd·m${}^{-2}$·lx${}^{-1}$ ). This corresponds to the ratio of the luminance of the diffused natural light reflected from the road marking at an angle of 2.29${}^{\circ }$ over the horizontal illuminance due to the overcast sky. This geometry represents a driver whose eyes are conventionally 1.2 m from the ground and who is looking 30 m away.

At night, the visibility of the road markings is no longer ensured by natural light but by the reflection of the car headlights on the surface of the marking. Night visibility is characterized by the retroreflection coefficient $R_L$, which is defined as the reflection of the headlights on a marking located 30 m from the driver. It is the quotient of the luminance L of the field of the road marking in the direction of observation by the illuminance $E_{\perp }$ at the field perpendicular to the direction of the incident light [1]. In standard measuring conditions, the observation angle is 2.29${}^{\circ }$, and the illumination angle is 1.24${}^{\circ }$.

In the standards, all these marking visibility indicators are expressed independently of the surrounding pavement characteristics.

Although it concerns only night-time visibility, the most widely used indicator of road-marking performance is the retroreflection $R_L$. Several studies [3,4,5] have shown that the number of accidents at night decreases when the $R_L$ is higher. Moreover, since the 1990s, the literature introduced maintenance models mostly based on retroreflectivity as a variable but also considering the age of the marking, initial $R_L$, traffic volume, material type, position of the marking etc. [6,7].

A strategy of preventive renewal with respect to age and considering budgetary constraints has been proposed by Redondin et al. [8] and Najeh et al. [9], using the Weibull model based on retroreflection measurements. Moreover, in the Eurorap 2011 report [10], a renewal of the markings was recommended when the retroreflection is lower than 150 mcd·m${}^{-2}$·lx${}^{-1}$. A review on the impact of road markings on driver behaviour and road safety was recently performed by Babic [11] and also confirmed the importance of higher retroreflection and marking maintenance for road safety.

The daytime visibility indicators (e.g., $Q_d$) could also be used to define maintenance policy [12] but are not used in practice because, contrary to the $R_L$ factor, this cannot be measured dynamically.

Although not standardised and dependent on ambient light conditions, the luminance is sometimes measured dynamically with a viewing angle of 2.29${}^{\circ }$. This indicator is notably used to calculate a luminance contrast between the road marking and the pavement as seen in [13,14].

1.3. Road-Marking Visibility According to Machine-Vision Systems

Driving assistance technologies or ADAS are becoming common in new vehicles with the increasing automatization of driving and the future arrival of AVs. These vehicles are equipped with machine-vision (MV) systems composed of artificial-vision systems, which act as a type of “automated eye” associated with algorithms and software. To understand the needs of these systems, research must be conducted to find a relationship between the standardized factors (particularly $R_L$ and $Q_d$) and the MV’s algorithm response. In this paper, road-marking recognition by cameras is considered.

The recognition of road markings from images recorded by on-board cameras can use different procedures of image processing: classical segmentation techniques, machine or deep learning and the use of proprietary algorithms.

Bar et al. [15] proposed a state-of-the-art use of classic image processing for the detection of road marking. The road markings are first extracted from the pavement surface by applying a binarisation method based on a threshold on a grey-scale image, such as the Otsu [16] method. Then, a road marking line is fitted from the different segmented objects [17]. Hough’s transform is also often used [18]. The major drawback of these approaches is that a relevant threshold value is needed to correctly segment the marking elements.

Since the reference paper of Bar et al. [15], many alternatives for road-marking detection based on convolutional neural networks (CNN) have been developed. The training phase of the CNN requires a huge number of images. To constitute a reliable ground truth, an operator carefully indicates where the road marking elements are located. The image annotation phase is greatly time consuming. Once the training phase is operational, the algorithm is able to automatically detect the road markings and then to model the marking line [19,20]. To our knowledge, there is no paper that presents a correlation between the algorithms’ answers and the standardized indicators characterizing the road-marking performance.

The last type of procedure uses proprietary algorithms. The characteristics of the used camera and of the implemented algorithm are often unknown. In addition, there is no access to the raw data. Most of the time, these systems provide a score indicating the quality of the road marking line detection and sometimes a view range [14,21,22,23].

For example, Babic et al. [23] investigated the differences in the detection quality and view range of a “Mobileye system” in dry daytime and night-time conditions. With this device, the detection of the road markings was slightly better in the night conditions. Pappalardo et al. [24] proposed a generalized estimating equation model to estimate the fault probability among various effects of road features, such as the retroreflection of markings. The authors set a maximum fault probability of $10\%$ as suggested by Reddy et al. [25]. Different classes of retroreflection were explored to define situations respecting this threshold. Using their study case, they found that the recommendations of the European Commission [10] (a minimum of 150 mcd·m${}^{-2}$·lx${}^{-1}$ in dry weather and 35 mcd·m${}^{-2}$·lx${}^{-1}$ in rainy weather) are adequate to fulfill the 10% fault probability.

Some studies [21,26,27,28,29,30] have attempted to find a relationship between ADAS performance and the characteristics of road markings. By testing different experimental conditions, they recommended threshold values of the standardized indicators to obtain good detection of the road marking lines by ADAS or a machine-vision system.

Table 1. Pavement marking standard requirements for MV according to the literature on a dry road.

	Nighttime Visibility	Daytime Visibility
Standard Indicators	${\mathit{R}}_{\mathit{L}}$ (mcd·m${}^{-\mathbf{2}}$·lx${}^{-\mathbf{1}}$)	Qd (mcd·m${}^{-\mathbf{2}}$·lx${}^{-\mathbf{1}}$)
(Lundkvist and Fors 2010) [26]	$R_L\ge 70$	$Q_d\ge 85$
(Pike et al. 2018) [21]	$R_L\ge 34$	-
(Somers 2019) [28]	$R_L\ge 100$	-
(Stacy 2019) [22]	$R_L\ge 200$	No correlation
(Pappalardo et al. 2021) [29]	-	$Q_d\ge 153$
(Babić et al. 2022) [30]	$R_L\ge 55$	-

gives a summary of the threshold values given in the literature.

For a given standardized indicator “X” ($X=R_L$, $Q_d$ or L), a contrast of “X” (labelled $C_X$) between the road marking element and its surrounding pavement is sometimes considered and calculated:

$$C_X=\frac{\overline{X}\left(marking\right)}{\overline{X}\left(pavement\right)},$$

(1)

where $\overline{X}\left(marking\right)$ is traditionally the mean value of X on the considered scale of the marking element and $\overline{X}\left(pavement\right)$ is the mean value of X of the pavement surface surrounding the marking element. The pavement surface area considered for the calculation is rarely indicated in the literature. The threshold values of the contrast of several indicators recommended by the literature are given in

Table 2. Contrast requirements for MV according to the literature on a dry road.

	Nighttime Visibility	Daytime Visibility
Contrast Ratio	${\mathit{C}}_{{\mathit{R}}_{\mathit{L}}}$	${\mathit{C}}_{\mathit{L}}$	${\mathit{C}}_{{\mathit{Q}}_{\mathit{d}}}$
(Lundkvist and Fors 2010) [26]	-	-	marking 5 mcd·m${}^{-2}$·lx${}^{-1}$ higher than the road surface
(Carlson and Poorsartep 2017) [27]	-	-	$C_{Q_d}\ge 2$
(Pike et al. 2018) [21]	$C_{R_L}\ge 2.5$	-	-
(Marr et al. 2020) [31]	$5\le C_{R_L}\le 10$	-	$C_{Q_d}\ge 3$
(Burghadt et al. 2021) [13]	-	$C_L\ge 3$	-

.

In [32], Davies introduced the luminance contrast as an alternative to the $Q_d$ contrast. In [31], Marr et al. found that a minimal value of 3 for the $Q_d$ contrast provided a reasonable confidence that MV would detect the line. It is noticeable that, despite the difference between the $Q_d$ and the luminance, similar contrast threshold values (between 2 and 3) of these two factors are recommended in the literature.

As shown in Table 1 and Table 2, there is a large variability in the threshold values from one study to another. This may be due to the fact that the MV systems (and the algorithms used) as well as the experimental conditions are not necessarily the same. Indeed, it is difficult to make a comparison because, most of the time, there is no information about these MV systems. Burghardt et al. [13] emphasized that there is no real collaboration between MV developers and the researchers working on horizontal signalisation. Moreover, the areas considered (both for the marking element and its surrounding pavement) for the calculation are rarely provided, which can lead to discrepancies according to the state of the road.

1.4. Objectives of the Work

The objective of this work is to propose a more appropriate characterisation of the marking–pavement pair in relation to what is perceived by the MV systems on a dry road by day and, therefore, to propose a characterisation that would be more relevant for AV. We attempted to find a relationship between the performance of the tested machine-vision system and the luminance contrast between the road markings and the surrounding pavement.

To do that, we used a dynamic retroreflectometer and a vehicle equipped with a commercialised “real world” camera associated with its proprietary software. We conducted an experiment by day on a section of a circulated road around the city of Rouen to answer the following questions:

• Do the conditions of the road markings have a direct impact on the performance of a machine-vision system on dry roads?
• Is the luminance contrast a reliable indicator of machine-vision performance by day?
• What is the reliability of a machine-vision system by day? Furthermore, in what cases is it less reliable?

After a description of the experiment and used vehicles, we present our statistical analysis and results separately for each device. We then compare the performances of both devices. A discussion of these results is finally conducted allowing a comparison with those of the literature and suggesting some perspectives.

2. Materials and Methods

2.1. Itinerary

The experiment was conducted by day on a circulated road about 14 km long near the city of Rouen. This itinerary was composed of different types of road (departmental road 13 “DR13”, national road 138 “NR138” and junction road), and therefore the speed of traffic could vary up to 90 km·h${}^{-1}$ (see Figure 1). A first analysis of this experiment was conducted for the axial line, considering only one run [14]. In the current study, the edge marking line of the slow lane was analysed for three runs.

According to the French regulations [33], all the road marking lines were white, and the edge marking line was composed of a T2 dashed line (a 3 m long skip followed by a 3.5 m long void), a T4 dashed line (a 39 m long skip followed by a 13 m long void) or a continuous line. Visually, the markings showed highly variable degrees of wear and tear. The road surface was a bituminous pavement of different brightness characteristics along the itinerary.

All measurements were performed by day in September 2020, on a dry road pavement, and the weather conditions were sunny with the potential presence of clouds. The three runs were conducted by two instrumented vehicles (see Figure 2): a mobile reflectometer (labelled ECODYN3) and a vehicle equipped with a machine-vision system (labelled MOOVE). They followed each other in order to have exactly the same environmental conditions. The first run began at 12:40, the second at 13:40 and the last one at 14:50. The three passages were conducted in strict compliance with French traffic regulations.

2.2. Mobile Measurement of Marking Performance

2.2.1. Presentation of the ECODYN3 Device

The Cerema vehicle equipped with the mobile retroreflectometer ECODYN3 (see Figure 2) measured both the retroreflected luminance coefficient $R_L$ and the luminance L of a road marking according to the standard geometry [1]. Since the head unit of ECODYN3 was placed on the vehicle structure, the measurement was conducted at 6 m instead of 30 m (see Figure 3). The vehicle was also equipped with an environmental camera, an odometer, a GPS sensor and an illumination cell.

The dimensions of the measurement area of the ECODYN3 are 0.5 m long and 1 m wide (see Figure 4). This area is composed of 32 measurement channels of approximately 0.03 × 0.5 m. In this area, a measurement channel might be located partly on the road marking and partly on the roadway as illustrated in Figure 4. The channels that overlap between the road and the markings are not used in statistical studies. An image of the ECODYN3 environmental camera is saved every 10 m and can help to understand the detection context.

2.2.2. Nature of the Data

With the ECODYN3, since we have access to the raw data, the outputs are not averaged values on 50 or 100 m as with the other retroreflectometers. The data are treated in the form of digital images by considering each measurement channel as a pixel according to the patent from [34]. It is then possible to generate two images, one for the retroreflection signal and the other for the luminance signal with each acquisition forming one line of the generated image (see Figure 4). Each line of the image is composed of 32 columns (matching the 32 measurement channels), and the distance between two lines corresponds to a longitudinal distance of 0.4 m.

From the recorded data, an extraction of markings was performed using a segmentation method. All statistical analyses were then conducted on the basis of the segmented images. It was then possible to characterize the markings at different scales, either at the longitudinal resolution of the ECODYN3 device or over a defined length (for instance, the length of a marking strip).

In this paper, the edge line is first analysed with a 0.4 m scale of analysis, corresponding to the distance of two consecutive acquisitions. The two indicators measured are the retroreflection level $R_L$ and the luminance contrast $C_L$. According to Davies [32], by day, the luminance contrast is the most important factor for machine-vision performance. In this paper, the luminance contrast ratio is calculated with the measurement of the luminance of the marking and its surrounding road, considering both the right and left side. This is defined by:

$$C_L=\frac{L_{median}\left(marking\right)}{L_{median}\left(pavement\right)},$$

(2)

where $L_{median}\left(marking\right)$ is the median luminance value of the measurement channels over the marking element and $L_{median}\left(pavement\right)$ is the median luminance value of the measurement channels over the pavement surrounding the marking element on both the left and the right of the marking element. We considered the median luminance value rather than the mean value (as in Equation (2)) because it is more robust than the average value [14].

2.3. Mobile Measurement with a Machine-Vision System

2.3.1. Presentation of the MOOVE Device

The MOOVE project is the result of a collaboration between the VEDECOM institute and French car manufacturers. Its main objective is to drive on European roads and acquire a large amount of data to create a database of the parameters defining real-life driving situations. A MOOVE vehicle is instrumented for ${360}^{\circ }$ of vehicle perception (see Figure 2 on the right). The main standard sensors are two on-board cameras, two LiDARs, one long-range front radar, four short-range radar corners and a Global Navigation Satellite System (GNSS). In this study, we only focused on the data collected by the proprietary “Real World” camera (associated with the GNSS), which is the only sensor of this vehicle dedicated to road marking detection.

2.3.2. Nature of the Data

The Real World camera used is a proprietary commercial system whose specifications and algorithms are not available. Only exploited results related to the detection of marking lines are available and could be used in this study. When it circulates, the MOOVE device can simultaneously analyse four road marking lines in its traffic lane. The Real World camera records images and provides a result file with a set of data at 25 Hz composed of:

• A quality level of the marking line detection. This is a rating with four different graduations: Very Low, Low, High or Very High.
• A confidence level in the detection quality, between 0% (no line detection) and 100% (certain to detect the line).
• A polynomial model describing the curve of the marking line on a range distance. This range distance corresponds to the area where the interpolation of the road marking line is considered valid and is updated at each acquisition with the polynomial coefficients.
• Other characteristics of the marking, such as the type, colour and width of the line.

For each dataset, the processing results are geolocated and saved for each marking line. The images analysed by this system are, unfortunately, not accessible.

2.4. Statistical Analysis

This subchapter describes the different statistical analyses conducted on the measurements collected in the Rouen circuit by the ECODYN3 and MOOVE vehicles. The results are first presented separately for the two vehicles and then compared.

2.4.1. Analysis of the Retroreflectometer Data for the Three Runs

A boxplot representation was chosen to represent the distribution of the $R_L$ values and luminance contrast over the three runs. To determine whether they could be considered identical between the successive runs, several analyses of variance (ANOVA) were performed. Since we have three runs, the one-way analysis of variance was used after checking its validity hypothesis: Levene’s test was used to check the homogeneity of variance and the Shapiro–Wilk test was applied to ensure the normality of the ANOVA residuals. If one assumption was not respected, a non parametric alternative of the ANOVA was used with the Kruskal–Wallis test by ranks.

2.4.2. Analysis of the Machine-Vision-System Data for the Three Runs

As presented in the Section 2.3.2, the detection of the road marking line is described by a quality level of detection (Very Low, Low, High or Very High) associated with a confidence rate of detection. The Very High rate (i.e., the frequency of data with a Very High quality level of detection over all the data) was first calculated all along the edge marking line of each run. In a second step, a classification of the circuit was established from the Very High rate changes between the different runs.

2.4.3. Comparison between the Mobile Retroreflectometer and Machine-Vision System

In the literature, authors have attempted to find a relationship between the standardized indicators and MV performance. Since this study was conducted by day, the indicator used for the comparison is the luminance contrast ratio. The retroreflection levels, which represent visibility during night-time, are given to assess the state of the marking lines.

A correlation analysis and comparison between the data collected by the ECODYN3 and the MOOVE vehicles was conducted for all the data as well as for a selection of road sections according to the MV classification. The idea is to segment the data according to the Very High rate observed during the three runs. A focus was also conducted on representative sections of the different classes.

3. Results

This chapter presents the results of the different statistical analyses conducted on the measurements collected in the Rouen circuit by the ECODYN3 and MOOVE vehicles.

3.1. Results for the Retroreflectometer

This paragraph presents the results of the pavement and marking characterisation performed by the mobile ECODYN3 device. For each run, the retroreflection values of the edge line are shown in Figure 5 on the left with a boxplot corresponding to each run. The three boxplots show a similar distribution of the retroreflection level over each passage. The first quartile values of the retroreflection for the three runs were very low at about 25 mcd·m${}^{-2}$·lx${}^{-1}$. The median values were between 40 and 56 mcd·m${}^{-2}$·lx${}^{-1}$. The third quartiles were between 105 and 125 mcd·m${}^{-2}$·lx${}^{-1}$. Concerning the road surface surrounding the marking, the median retroreflection was 12 mcd·m${}^{-2}$·lx${}^{-1}$. Globally, it appears that the $R_L$ values are low, suggesting that most of the markings of the itinerary are worn out and should be renewed.

The boxplot of the luminance contrast is presented in Figure 5 on the right. The first quartile values of the luminance contrast of the three runs were around 1.30. The median values were between 1.49 and 1.75, and the third quartiles were between 2.41 and 3.75. The third run had the highest values and less outliers compared with the other two. This is likely due to different environmental conditions for this passage (more shadows, etc.). It suggests a non-similarly distribution of the luminance contrast over each run.

To test the reproducibility of the measurements, several ANOVA were performed after a validation of their assumptions.

Table 3. Levene, Shapiro–Wilk and ANOVA tests of the retroreflection values for the three runs.

Test	Levene Test		Shapiro–Wilk Test		Anova Test
	F(2, 43531)	p-Value	W(2, 43531)	p-Value	F(2, 43531)	p-Value
$R_L$	6.49	0.32	2.37	0.15	5.92	0.62

presents the results of the tests conducted for the retroreflection ($R_L$) between the three runs. All p-values are above the 0.05 significance levels. According to the Levene test, the variances are statistically identical. According to the Shapiro–Wilk test, the hypothesis of normality of the data is confirmed. Therefore, it is possible to conduct an ANOVA analysis. Since the p-value is higher than the 0.05 level of significance, the hypothesis $H_0$ is fulfilled: the retroreflection ($R_L$) average values of the different passages are identical.

For the luminance contrast values and according to

Table 4. Levene, Shapiro–Wilk and Kruskal–Wallis tests of the luminance contrast values between the three runs.

Test	Levene Test		Shapiro–Wilk Test		Kruskal–Wallis Test
	F(2, 43531)	p-Value	W(2, 43531)	p-Value	${\mathit{\chi }}^2$(2, 43531)	p-Value
Luminance contrast	395.12	<0.05	112.57	<0.05	238.60	<0.05

, the Levene test p-value is below the 0.05 significance levels; thus, the variance between the runs is different. Regarding the Shapiro–Wilk normality test, the results show that the residuals do not follow a normal distribution. Therefore, an ANOVA test cannot be applied, and consequently the non-parametric Kruskal–Wallis test by ranks was applied instead. As the corresponding p-value is below the significance level of 0.05, we conclude that there are significant differences between the average luminance contrast values of each passage.

These tests confirm the results presented in Figure 5. The retroreflection values are identical on all three runs, which is consistent with the fact that $R_L$ does not depend on sunlight. In contrast, the luminance contrast values are not similarly distributed, and this is likely due to different environmental conditions for each run (changes in the ambient luminosity and different inclinations of the sun leading to different shadows on the pavement). This sensitivity to ambient brightness explains why the luminance ratio is not used to characterise road markings for maintenance policies. Knowing these results, we have to consider each run independently for the comparison with the data collected by MOOVE.

This was possible in our study because the two vehicles followed each other and had exactly the same experimental conditions. In the following, and since the study was conducted during the day, we compare only the performance of the machine-vision camera with the luminance contrast.

3.2. Results for the Machine-Vision System

The marking detection performance of the machine-vision system of MOOVE is presented for the edge line in Figure 6 and Figure 7 for the three runs:

• The left of Figure 6 presents the percentage of Very High rates observed at each run. This rate is between $90\%$ and $92\%$.
• The boxplots in Figure 6 on the right show the range distance where we applied the polynomial model describing the curve of the edge marking line. The first quartile is between 29 and 33 m, and the third quartile is between 59 and 63 m.
• Figure 7 presents the histogram of the confidence rates at each run. The median confidence level for the three runs is 100%. In 80% of the cases, the confidence level in the marking detection is 100% for the three runs, and in 89% of the cases, this confidence level is greater than or equal to 80%.

Figure 6. Very High quality rate (left) and boxplot of the range distance (right) during the three runs.

Figure 6. Very High quality rate (left) and boxplot of the range distance (right) during the three runs.

The edge line was almost always detected with a Very High confidence level all along the itinerary. Figure 8 (on the left) presents a map projection of each detection quality measurement observed during the first run. Very Low areas ($9.5\%$ of the data collected) are mainly located on sections where there is a particular feature of the infrastructure, such as roundabouts, insertion lanes or a working area.

Using a comparative analysis of the Very High rates of the three runs, a classification of the itinerary was defined according to the following criteria:

• “Perfect Very High”: identical behaviour for the three runs with 100% Very High rates (colour code green).
• “Imperfect Very High”: Very High rates higher than 90% for the three runs (colour code blue).
• “Variable Quality Level”: Very High rates between 30% and 90% and differences between the three runs (colour code yellow).
• “Worst Very High”: less than 50% Very High rates for each run (colour code red).

Figure 8 (on the right) and Table 5 show, respectively, the position of 17 sections identified on the Rouen circuit and the Very High rates observed for each run:

• There are six sections of Perfect Very High. They constitute 29% of the Rouen circuit.
• There are three sections of Imperfect Very High that constitute 58% of the circuit.
• There are four sections of Variable Quality Level that constitute 7% of the circuit and are mainly represented in section n°3.
• There are four sections of Worst Very High that constitute 6% of the circuit and are mainly represented by sections where it is more difficult to circulate because of the infrastructure complexity (roundabout or working area).

Figure 7. Histogram of the confidence rate observed during the three runs.

Figure 7. Histogram of the confidence rate observed during the three runs.

Figure 8. Map projection of the results obtained with the MOOVE vehicle. On the left, edge lane analysis during the first run. On the right, classification according to the quality level observed over the three runs. The sections names of the number 1–17 are given in the Table 5.

Figure 8. Map projection of the results obtained with the MOOVE vehicle. On the left, edge lane analysis during the first run. On the right, classification according to the quality level observed over the three runs. The sections names of the number 1–17 are given in the Table 5.

Table 5. Classification of the different sections according to the quality levels observed over the three runs. The green color correspond to a “Very High sections”, the blue to “Imperfect Very High sections”, the yellow for “Variable Quality Level sections” and the red for “Worst Very High sections”.

	Section	Length (m)	Very High Rate			Comment
			Run 1	Run 2	Run 3
1	2 × 1 to 2 × 2 lane	69	100%	47%	71%
2	DR13—Dir. 1—Part 1	884	100%	100%	100%
3	The long curve	614	88%	79%	74%
4	DR13—Dir. 1—Part 2	797	100%	100%	99%
5	The First Roundabout	102	6%	25%	6%	No marking
6	Between Two Roundabout	144	100%	100%	100%
7	The Second Roundabout	136	22%	12%	23%	Few markings
8	DR13 to NR138—Part 1	135	100%	100%	100%
9	Gaz Station Entrance	21	100%	29%	100%
10	DR13 to NR138—Part 2	101	100%	100%	100%
11	Gaz Station Exit	37	33%	31%	15%
12	DR13 to NR138—Part 3	1098	100%	100%	100%
13	NR138 Entrance	125	78%	84%	59%
14	NR138—Direction 1	1217	100%	100%	100%
15	Working Area	467	24%	50%	48%	Disrupted traffic
16	Direction 1 to 2	627	89%	90%	99%
17	Direction 2	5759	99%	98%	100%

Table 5. Classification of the different sections according to the quality levels observed over the three runs. The green color correspond to a “Very High sections”, the blue to “Imperfect Very High sections”, the yellow for “Variable Quality Level sections” and the red for “Worst Very High sections”.

	Section	Length (m)	Very High Rate			Comment
			Run 1	Run 2	Run 3
1	2 × 1 to 2 × 2 lane	69	100%	47%	71%
2	DR13—Dir. 1—Part 1	884	100%	100%	100%
3	The long curve	614	88%	79%	74%
4	DR13—Dir. 1—Part 2	797	100%	100%	99%
5	The First Roundabout	102	6%	25%	6%	No marking
6	Between Two Roundabout	144	100%	100%	100%
7	The Second Roundabout	136	22%	12%	23%	Few markings
8	DR13 to NR138—Part 1	135	100%	100%	100%
9	Gaz Station Entrance	21	100%	29%	100%
10	DR13 to NR138—Part 2	101	100%	100%	100%
11	Gaz Station Exit	37	33%	31%	15%
12	DR13 to NR138—Part 3	1098	100%	100%	100%
13	NR138 Entrance	125	78%	84%	59%
14	NR138—Direction 1	1217	100%	100%	100%
15	Working Area	467	24%	50%	48%	Disrupted traffic
16	Direction 1 to 2	627	89%	90%	99%
17	Direction 2	5759	99%	98%	100%

A Very High rate of 90–92% with a median confidence rate of 100% was observed on the whole Rouen circuit (see Figure 6 left).

In the next parts, comparisons between the marking luminance contrast observed with the retroreflectometer and the marking detection revealed by the machine vision are conducted. To this purpose, each run of the Rouen circuit was arbitrary segmented into 30 m segments. This distance corresponds to the first quartile of the range observed by the MOOVE vehicle (see Figure 6 right) during the three runs.

A total of 1212 road segments of 30 m long were available. For each 30 m segment, a median luminance contrast was calculated as well as the Very High rate of the Real World camera.

3.3. Analysis of Correlation

To study the relationship between two variables, the correlation coefficient is a specific measure that quantifies the strength of the linear relationship between two variables. The correlation coefficient between the marking luminance contrast and marking detection by machine vision was r = 0.01, thus, suggesting no correlation of the data along the itinerary for the three runs.

The Very High rate function of the luminance contrast is presented on a scatterplot (see Figure 9). It shows that most of the markings were detected with a rate of 100% regardless of the luminance contrast (in more than 92% of the data). This confirms the weak relationship between the two variables, accentuated by the existence of outliers with a luminance contrast higher than 4 and a detection rate lower than 50%.

In order to refine our analysis, we decided to use the classification established from the data of the MOOVE vehicle (see Figure 8 on the right), and we calculated the correlation coefficient for each section of Table 5. The results are provided in Figure 10. This figure shows huge variability of the correlation coefficient: the coefficient can be positive or negative independently of the class section. Moreover, most of the coefficients were very low—below 0.5. Thus, it is difficult to extract any correlation between the luminance contrast and the Very High rate of detection.

Figure 10 shows that the most significant correlation coefficients found were obtained for the sections n°7, n°13 and n°15. These sections are coloured in red or orange, with bad MV performance. Some sections with a Perfect Very High performance of the MV have extremely low correlation—for example, section n°2.

3.4. Focus on Certain Road Sections According to the MV Quality Level

Seventeen road sections had been previously identified on the whole circuit with the classification established from the measurement of the MOOVE vehicle (see Table 5). In this part, we focus on several specific sections that provide some explanation of the results:

• A Perfect Very High section with markings in bad condition: the section n°2 called DR13—Dir. 1—Part 1.
• A Variable Quality Level section with different performance of the Real World camera between the runs: the section n°3 called Long Curve.
• Examples of Worst Very High sections corresponding to bad performance.

3.4.1. Perfect Very High Road Section n°2

Images of the Perfect Very High road section n°2 (called DR13—Dir.1—Part 1) are given in Figure 11 at the same location for the three runs. In this section, the marking is worn, and the median retroreflectivity is very low—below 30 mcd·m${}^{-2}$·lx${}^{-1}$, which corresponds to a road marking without retroreflectivity.

Table 6. ECODYN3 and MOOVE results on the road section n°2—DR13-Dir.1-Part1.

	Median Luminance Contrast Ratio	Very High Rate	Confidence Rate
Run 1	1.31	100%	98%
Run 2	1.27	100%	97%
Run 3	1.35	100%	96%

presents a cross analysis between the median luminance contrast measured by ECODYN3 and the Very High rate given by MOOVE.

The median luminance contrasts on this section are between 1.27 and 1.35 for the three runs, which are below the thresholds of the literature (cf. Table 2). This road section has a perfect reproducibility with 100% Very High rates on the three runs with an average confidence rate equal to 97%. Despite the worn nature of the road marking line (according to the $R_L$ values), the edge line of the section n°2 was perfectly detected by the machine-vision system.

From Table 6, it seems that a luminance contrast around $1.31\pm 0.04$ on a dry road is able to give a Very High rate of 100%.

3.4.2. Variable Quality Level Road Section n°3 Called the Long Curve Section

The section n°3 with the Variable Quality Level class has the greatest variation of the Very High rate between the runs, despite having close detection quality levels between the three runs (Very High rates of 90–92% with a median confidence rate of $100\%$ on the whole Rouen circuit; see Figure 6 and Figure 7). Figure 12 presents three maps of the quality detection levels of the road markings located in section n°3 (one map per run).

On the first run, one Very Low subsection was observed at the beginning of the curve. On the second run, several other Very Low subsections were observed in the curvature. On the third run, several Very Low subsections were observed at other locations on the curve. To better understand these differences of reproducibility, the Long Curve section was divided into five subsections. Each one is defined by one non-occurrence of the Very High quality level on at least one run.

• Subsection 1 has a Very High rate of 100% for runs 2 and 3 and 61% for run 1.
• Subsection 2 has a Very High rate of 100% in the first run, while it is only 9% and 48% respectively in runs 2 and 3. This subsection has the lowest Very High Rate.
• Subsection 3 has a Very High rate of 100% for runs 1 and 2 and 80% for run 3.
• Subsection 4 has a Very High rate of 100% for the first run and no line detection at all during the second run.
• Subsection 5 has a Very High rate of 100% for runs 1 and 2 and 44% for run 3.

Figure 12. Comparison of the Very High rate on road section n°3—Long Curve between the three runs.

Figure 12. Comparison of the Very High rate on road section n°3—Long Curve between the three runs.

Visual scenes of road subsections 3, 4 and 5 are given in Figure 13 at the same location for the three runs and shows that the shaded areas are not the same for the different runs.

An analysis of the subsections was conducted to investigate the differences between the runs:

• The median luminance contrast was always above 1, meaning that the road marking was always brighter than its surrounding pavement.
• The subsection 3 had the highest variation of the median luminance contrast (between 1.18 and 1.47). The Very High rate was at 100% for runs 1 and 2 and decreased to 80% for the third passage. This is likely due to the appearance of the shadow created by the trees on the edge marking line during the third run (see Figure 13 top). This misdetection caused by shadows was also observed by Kim [35].
• Despite having the same median luminance contrast between runs 1 and 2 in subsection 4, the Very High rate was 100% for the first run and 0% for the second one;
• During the third run, subsection 4 and 5 showed similar median luminance contrasts (1.39 and 1.40); however, the Very High rates were, respectively, 100% and 44%.

The Long Curve section illustrates both the variability of the luminance contrast values and strong variations in Very High rates between runs. From

Table 7. ECODYN3 and MOOVE results on road section n°3—Long Curve for each subsection.

Passage	Sub-Sections	Median Luminance Contrast Ratio	Very High Rate	Confidence Rate
Run 1	1	1.36	61%	92%
	2	1.52	100%	100%
	3	1.36	100%	99%
	4	1.32	100%	95%
	5	1.22	100%	95%
Run 2	1	1.33	100%	79%
	2	1.42	9%	63%
	3	1.47	100%	96%
	4	1.32	0%	10%
	5	1.21	100%	100%
Run 3	1	1.15	100%	87%
	2	1.06	48%	60%
	3	1.18	80%	63%
	4	1.39	100%	82%
	5	1.40	44%	50%

, it is difficult to find a relationship between the luminance contrast and the Very High rate. On run 2, for example, a luminance contrast of 1.21 (for subsection 5) or 1.33 (for subsection 1) could allow for a 100% Very High rate. Considering such a threshold would mean that higher luminance contrast values would provide a similar Very High rate. Here, it is shown that a luminance contrast of 1.42 (for subsection 2) leads to a Very High rate equal to 9%. Thus, a minimal luminance contrast does not seem to be a good indicator to ensure a 100% Very High rate.

Figure 13. Representative visual scenes taken with the ECODYN3 environment camera for subsections 3 (top), 4 (middle) and 5 (bottom) of the road section n°3—Long Curve at the same location for each run.

Figure 13. Representative visual scenes taken with the ECODYN3 environment camera for subsections 3 (top), 4 (middle) and 5 (bottom) of the road section n°3—Long Curve at the same location for each run.

The presence of shadows seems to have an impact on the road-marking detection by the MV system. This impact seems more important when the shadow partially covers the road marking and its surrounding pavement (as on the subsection 3 of the third run with a Very High rate equal to 80%) than when it covers the whole measured surface (as in the subsection 4 of the third run with a Very High rate equal to 100%). However, the shadows cannot explain all the variability of the performances obtained over the long curve with the tested MV system. As shown in Figure 13, the road markings for the edge lines were not shaded during the first and second runs, and at the same time, the Very High rate fluctuated from 0 to 100%, while the the wear and tear of the markings was similar on the curve. We were not able to explain this result for from our study.

Since the experiment was conducted in optimal conditions for the visibility of the road marking (by day on a dry road without too much glare), it questions the reliability of the system when the marking is worn and has a low luminance contrast (less than 2).

3.4.3. Worst Very High Road Sections

This section illustrates three Worst Very High sections obtained from the classification. They correspond to the road sections where the MV system was not able to detect the edge marking line. The correlation coefficient was sometimes better in this class than on the others but remained rather weak. These sections with poor marking-line-detection performance all had a specific infrastructure.

Figure 14 presents three Worst Very High road sections:

• Section n°5 (see Figure 14 on the left) features a roundabout with a large radius of curvature where it was impossible for the reflectometer ECODYN3 to measure the performance of the road marking under good conditions. Moreover, the MOOVE vehicle did not detect any edge line on this section, while road markings were present.
• Section n°11 (see Figure 14 on the middle) presents a small entry section to the national road. The width of the road was substantial, and the marking line in the curve was beginning to fade.
• Section n°15 (see Figure 14 on the right) presents a road section with a working area marked by cones. Despite the good performance of the road marking lines (the median $R_L$ value was around 105 mcd·m${}^{-2}$·lx${}^{-1}$, and the median luminance contrast was around 2.74), the MV system of the MOOVE vehicle did not detect the marking line. The worksite cones may have disturbed the system.

Figure 14. Representative visual scenes taken with the ECODYN3 environment camera of the Worst Very High road sections: n°5 (left), n°11 (middle) and n°15 (right).

Figure 14. Representative visual scenes taken with the ECODYN3 environment camera of the Worst Very High road sections: n°5 (left), n°11 (middle) and n°15 (right).

The non-detection of road markings identified in this experiment mainly correspond to the absence of markings or a work area. Work areas are identified as tricky for autonomous vehicles and usually the control is given back to the driver. Consequently, the infrastructure and the geometry of the road appear to have major impacts on the performance of the MV system.

4. Discussion

Our objective was to check whether the luminance contrast between the road markings and the surrounding pavement was relevant to explain the response of an MV system by day. To do this, an experiment was conducted during the day on a small high-traffic road section, using a mobile retroreflectometer and a vehicle equipped with an AV camera. The two vehicles followed each other and made three successive runs in the middle of the day. The road-marking performance was evaluated according to the standard specifications [1] using retroreflection and a luminance contrast ratio considering the marking and its surrounding road.

According to the EN 1436 standard [1], the visibility of the road marking is usually assessed by the indicator $Q_d$. However, as mentioned in Section 1.2, the $Q_d$ is not dynamically measurable. Therefore, we measured the luminance at 2.29° instead. The characterization of the markings was conducted with medians, which are a more robust indicator than the average (less sensitive to extreme values) and, thus, more suitable for the characterisation of heterogeneous markings [14].

4.1. Findings

In this paper, the edge line results are presented for all the three runs. According to the retroreflection factor, the overall condition of this line was very worn. More than 40% of the road markings (over the whole itinerary) had a median $R_L$ value below 60 mcd·m${}^{-2}$·lx${}^{-1}$, and only 7% percent of the circuit had a retroreflectivity better than 150 mcd·m${}^{-2}$·lx${}^{-1}$, which is a threshold commonly considered to define good marking performance [10].

Concerning the luminance contrast ratio, 42% was below 1.75, while a luminance Weber contrast of 2 was recommended in [13] for good machine-vision performance (it corresponds to a luminance ratio of 3). In other studies where they considered the $Q_d$ contrast, they recommended a minimum ratio of 2 [27] or 3 [31]. Similarly to Stacy [22], who studied the relationship between the $Q_d$ contrast and the detection of the markings by an AV camera, we did not find a correlation between MV performance and daylight luminance contrast.

Our study also questioned the reliability of the literature thresholds for the luminance contrast ratio on a dry road under sunny conditions. A threshold that seems relevant in one case may be questioned in other experimental or environmental conditions. These observations suggest that there is no relationship between the luminance contrast and road-marking detection performance in dry daytime conditions.

Despite the presence of an old and worn marking in the circuit, the used machine-vision system was able to detect the edge road marking line on the majority of the circuit (Very High rates of 90–92% with a median confidence rate of 100% on the whole circuit). Even with a luminance contrast ratio below 1.3, a Very High rate of 95% was observed on average. In this situation, the current literature predicts a low probability of observing a Very High level of detection. We found that the Very Low detections of the road marking by the MV system generally occurred in specific cases, such as roundabouts or in the presence of shadows. Measuring the influence of shadows or working site cones on road-marking detection would be interesting but requires a separate experimental protocol.

4.2. Limitations and Perspectives

Since our results are based on a single experiment, conducted by day, we shall expand them with more data based on several experimental conditions. We found that the indicator of the luminance contrast was not correlated with the daytime performance of the machine-vision system; thus, it is necessary to look for new relevant indicators by day. Such indicators are not easy to find, because we do not have any access to the measurement process of the MV system, and consequently it is difficult to understand what does really matter for the algorithms of the camera.

The luminance measurements were conducted with a symbolic observation angle of 2.29°; however, the correlation between the luminance contrast and the detection quality may change if the angle is different. It could be necessary to conduct such experiments under night-time conditions because it is during this specific period that the $R_L$ factor is relevant. Under these conditions, the influence of glare due to low traffic lights on the marking detection could also be studied. A second interesting situation to experiment in would be wet road conditions.

However, in view of what is happening with the luminance contrast, one can wonder if there is really a relevant $R_L$ threshold value allowing a perfect detection of the marking line by the camera. A minimal value is likely necessary; however, this is not the only factor to be considered.

Finally, we conducted this experiment with only one type of MV system. Thus, it would be interesting to test other vehicles equipped with MV systems and other methods of characterizing the marking/roadway pair.

In the SAM project, we will create a marking pattern on a test track with different types of pavements. It will thus be possible to have several different controlled conditions, such as day, night and wet conditions. It will be easier to measure both the $R_L$ and $Q_d$ with a static retroreflectometer. We will also use other vehicles able to provide different characterizations of road-marking performance.

5. Conclusions

In dry daylight conditions, it seems that a classic retroreflectometer, which measures both the retroreflection and the luminance of the marking and its surrounding pavement, does not alone explain the operation of the tested machine-vision system. The detection of the marking line by the camera is impacted by qualitative factors, such as the sunlight and the infrastructure (shadow, lane changes and intersections), as already revealed by [20,31].

During driving, there is a continuity of the preview, which helps MV in determining trajectory. This could be an explanation for some of the noted inconsistencies. One of the problems is the absence of information on the machine-vision systems, which are proprietary devices. In particular, the area considered by the MV device for road-marking detection is unknown. This is why we made an assumption on the range distance to establish our classification.

However, the methods and algorithms used by automated vehicles are becoming very efficient, particularly during the day on dry roads, which is rather reassuring for the safety of road users. It seems that it is mainly the complexity of the infrastructure that best explains the poor performance, which was also identified by Zhang in their meta-analysis [20].