Repeatability and Reproducibility of Pavement Density Profiling Systems

Abstract

The work conducted in this study was designed to establish achievable testing tolerances for non-destructive pavement density measurements using Density Profiling Systems (DPSs). Nine and six sensors were used to determine the precision of repeatability and reproducibility in the laboratory and the field, respectively. A minimum of six sensors (considered in this study as independent laboratories) were needed to comply with the minimum number of participants required in the current ASTM standard practice (ASTM E691). The methodology included the development of laboratory precision evaluation with a total of nine sensors and two different mixtures (9.5 mm fine-graded mix, 19.0 mm coarse-graded mix) compacted at four density levels (97%, 94%, 91%, and 88% of G_mm). For the field portion of this study, pavement sections built at the National Center for Asphalt Technology (NCAT) Test Track in 2021 served as experimental variables. These sections were built with fine-graded asphalt mixtures and open-graded mixes as wearing courses. Additionally, the pavement sections included three underlying materials: new asphalt (binder layer), milled asphalt surface, and granular base, with thicknesses ranging from 3.8 to 13.9 cm. Density profile testing was conducted at two locations: within the mat (center of the lane) and along the joint. Computed precision statements regarding dielectric values within and between laboratories were about double for field results compared to laboratory results. However, when converted to density, the statements were significantly below the reported statements for Bulk Specific Gravity and Vacuum Sealing in the laboratory and Nuclear and Electromagnetic density gauges in the field.

1. Introduction

After several years of work, the pooled fund named “Continuous Asphalt Mixture Compaction Assessment Using Density Profiling System (DPS) [TPF-5(443)]” has developed and refined methods for effectively collecting data using DPSs, along with a variety of support materials such as documents and training videos, which explain in detail the required steps. Hundreds of miles of DPS data collected during this effort have been shared among fund members, consultants, vendors, and other industry stakeholders, fostering discussions and knowledge exchange during various events like project updates, peer exchanges, training sessions, and technical meetings [1]. The experiences contributed to developing draft specifications, which were submitted to the American Association of State Highway and Transportation Officials (AASHTO) in 2019, with another following in 2023 [2]. This draft marks a significant milestone thanks to the introduction of a standard procedure to accurately convert dielectric data to density measurements, eliminating the need for cutting field cores.

The first AASHTO Standard, PP 98-19: “Standard Practice for Asphalt Surface Dielectric Profiling System using Ground Penetrating Radar”, specifies the equipment and software requirements for a dielectric profiling system (DPS), along with calibration and verification procedures. Although it introduces several operating requirements, these criteria were developed based on limited experience. Thus, further research is needed to enhance these requirements with exact precision statements, improving the draft standard practice.

ASTM E691, “Standard Practice for Conducting an Inter-laboratory Study to Determine the Precision of a Test Method”, provides a standard procedure for determining the precision of a test method. Precision is expressed in terms of two measurement concepts: repeatability and reproducibility. Repeatability refers to the consistency of test results under stable conditions, where factors such as the operator, equipment, equipment calibration, and the testing environment (including temperature, humidity, and air pollution) remain reasonably constant. These factors usually have a minimal impact on the variability of the test results. Reproducibility, on the other hand, considers the variability of results when the testing conditions change from one laboratory to another. This includes differences in operators, equipment, and environmental conditions, which usually contribute appreciably to the variability of the test results. Thus, repeatability and reproducibility are two practical extremes of precision [3].

When conducting an interlaboratory study to determine repeatability and reproducibility precision, it is crucial to be aware of excessively “clean” data. Such data could arise from only involving the best operators or a laboratory taking unusual steps to receive “good” results. In this study, individuals with varying degrees of experience, including graduate students, technicians, and researchers, were involved to ensure a broad representation of capabilities. It is also essential to recognize and consider how to deal with “poor” results that have unacceptable assignable causes. Including such results in the final precision estimates might be questioned. In this case, it has been determined that samples compacted at low density (high air void) levels tend to add significant error in predicting air voids both in laboratory settings and potentially in the field [4].

Collecting applicable and consistent data requires careful planning and execution of the study. Questions concerning the number of laboratories required and the number of test results per laboratory affect the confidence in the precision statements derived from the study. According to ASTM E691, a minimum of six laboratories should participate in an interlaboratory study. Additional considerations include the number, range, and types of materials to be selected for the study and the need for a well-written test method accompanied by detailed instructions for the participating laboratories.

To evaluate the consistency of data in an interlaboratory study, ASTM E691 recommends using two key statistics: the “k-value” and the “h-value”. The “k-value” measures how a laboratory’s variability for a given set of samples compares with that of all the other laboratories. The “h-value” is a measure of how the average results for one laboratory compare with the average of the other laboratories. These test statistics are crucial for understanding the reliability and error margins of each laboratory’s performance. Laboratories identified with discrepancies or poor performance through these metrics can conduct an internal investigation and take corrective actions to address these deficiencies [5,6].

Most standard practices will not report k- and h-values. Instead, the repeatability standard deviation, S_r (single operator), and reproducibility standard deviation, S_R (multi-laboratory), are typically reported.

Table 1. Density single- and multi-laboratory standard deviations and statements, kg/m³.

	Laboratory Bulk Specific Gravity ASTM D2726 [7]	Laboratory Vacuum Sealing ASTM D6752 [8]	Nuclear Gauge ASTM D2950 [9]	Electromagnetic Surface Contact ASTM D7113 [10]
Single operator—Sr	13.1	12.3	25.1	20.4
Multi-laboratory—SR	15.1	13.5	28.0	23.5
Single operator—r	36.8	34.6	70.4	57.3
Multi-laboratory—R	42.0	37.6	78.4	65.8

shows the standard deviation and precision statement values for density measurements of asphalt pavements in both laboratory and field settings. The ASTM standard practices further define repeatability (r) and reproducibility (R) limits or statements, which are calculated as 2.8 times the S_r and S_R values, respectively.

The repeatability limit is defined in ASTM E691 as “the value below which the absolute difference between two individual test results obtained under repeatability conditions may be expected to occur with a probability of approximately 0.95 (95%)”. Similarly, the reproducibility limit is defined as “the value below which the absolute difference between two test results obtained under reproducibility conditions may be expected to occur with a probability of approximately 0.95 (95%)”.

An example of developing precision standards for field equipment used in pavement evaluation can be found in a friction study conducted by the Florida Department of Transportation (FDOT). This study was part of an effort to evaluate the potential use of the international friction index as a standard for friction measurement reporting. To achieve this, FDOT assessed the precision of two test methods: the dynamic friction tester (DFT) and circular track meter (CTM). This study involved a comparison between FDOT’s CTM and DFT equipment and similar portable equipment owned and operated by NCAT to quantify the repeatability and reproducibility of the test data. FDOT’s portable equipment was transported to Auburn, Alabama, for side-by-side comparison tests with the NCAT equipment [11].

In contrast to the guidelines set forth in ASTM E691, this study utilized only two laboratories to establish precision statements. The focus was on analyzing the data’s repeatability and reproducibility by examining the range, standard deviation, and coefficient of variance (COV). Here, the range served as a convenient measure of data dispersion, whereas the standard deviation provided a measure of the deviation around the mean. The COV is a normalized measure of variability, calculated as the standard deviation divided by the mean, expressed as a percentage. It helps to understand the extent of variability relative to the mean of the dataset. Additionally, paired t-tests were performed to compare the differences between the means of data sets from the two laboratories. In this case, the acceptance of the null hypothesis was used to indicate that the means of the data sets were not statistically different between laboratories [11].

The precision of the DFT and CTM equipment was assessed using ten different test surfaces at the NCAT Test Track. Both devices were used in five different locations within each of the ten test sections, employing equipment from NCAT and FDOT. The results of this comparative analysis led to the proposal of precision statements for both repeatability and reproducibility for each piece of equipment. Correlation relationships were also developed between the portable and full-scale equipment currently used by FDOT and others [11]. Although this approach deviated slightly from ASTM E691, it has proven to be an effective strategy for determining preliminary precision statements or verifying existing ones when limited laboratories are available. To complement the efforts described in this section, the research team conducted a laboratory and field experiments to establish achievable testing tolerances for non-destructive pavement density measurements using Density Profiling Systems (DPSs). A detailed description of the methodology, along with discussions and analyses are provided in the next section.

2. Materials and Methods

Based on the procedure described in ASTM E691 and the friction study conducted by FDOT, the experimental plans for laboratory and field testing were adjusted to reflect the conditions and limitations found in this study. In this context, each sensor or antenna was treated as an independent laboratory to measure dielectric values of laboratory compacted samples and measure dielectric profiles in the field. Each sensor was developed by the company GSSI to collect dielectric values at 2 GHz frequency from the top 1 to 2 inches of the asphalt concrete layer. Every sensor or antenna is built to work independently and requires individual calibration/verification by the manufacturer.

The methodology involved evaluating laboratory precision with a total of nine sensors across two mixture types: a 9.5 mm fine-graded mix and a 19.0 mm coarse-graded mix. These mixtures were compacted with a Superpave Gyratory compactor at four density levels—High (H) = 97%; Medium High (MH) = 94%; Medium Low (ML) = 91%; and Low (L) = 88% of the maximum theoretical density (G_mm). Each compacted sample was compacted to reach the Superpave design size of 115 mm (height) by 150 mm (diameter). The amount of asphalt mixture was adjusted (reduced or increased) from the design samples (compacted at 96% density level at 75 gyrations) to achieve the four levels of density used in this study.

Each density level had three replicates and three readings per replicate (as per AASHTO Standard, PP 98-19) [12]. Figure 1 shows the laboratory test set up where a sample is placed in the compaction orientation and the is rotated 90 degrees and finally flipped upside down with respect to the first test or centered configuration.

For the field portion of this study, only six sensors were utilized, operated by a single individual. The study covered eight different asphalt pavement sections, two different locations within each section, and three replicates per location. The selected field pavement sections included the following variables:

• Mix type: fine-graded and open-graded mixes (Sections E9 and E10);
• Underlying surface: new asphalt layer, milled asphalt surface, granular base;
• Asphalt layer thickness: ranging from 3.8 cm to 13.9 cm;
• Testing location: within the mat and along the joint.

Table 2. Properties of field sections.

Section	E9	E10	N8	N6	N9	N1, N2, N7
AC thickness [cm]	3.8	3.8	5.7	5.9	8.1	13.9
Base material	Old AC	Old AC	New AC	Old AC	Old AC	Granular base
%Gmm (cores)	NA	94.0	95.3	94.4	94.5	94.5 to 96.1

AC: Asphalt concrete. NA: Not available.

shows more details on the tested sections, including their thickness, base material, and, when available, compaction level based on quality control data. It should be noted that not all construction information was available at the time of preparing this report.

3. Results

3.1. Laboratory Dielectric Test Results and Precision Statements

Figure 2 shows examples of Superpave gyratory compacted specimens. The 19.0 mm specimens exhibited a rougher texture due to their larger particle sizes and coarser gradation compared to the 9.5 mm specimens. Figure 2 shows average dielectric measurements obtained in the laboratory, leveraging nine different sensors for both mixture types across all four air void levels. The sensors captured the variation of the results with air void content, while the dielectric values within each air void level were consistent. Each of the number/colors (195, 206, etc.) shown in Figure 3 are the serial numbers of the sensors used in this study. Figure 3 shows the standard deviations in dielectric measurements at each air void level, indicating the highest variability in measurements at increased air void percentages. Furthermore, the 9.5 mm specimens exhibited significantly less variability in dielectric values than the 19.0 mm specimens. As observed in Figure 1, 19.0 mm specimens had higher surface texture and overall higher density variability that contributed to the higher dispersion in Figure 4.

Figure 5 and Figure 6 show the laboratory results for dielectric consistency, displaying the h- and k-values, respectively. The critical h- and k-values were obtained from ASTM E691. These critical values vary depending on the number of laboratories used (or, in this case, the number of sensors) and the number of replicates. For a setup involving nine sensors and three replicates, the critical h-value is set at 2.23, while the critical k-value is 2.09. For all sensors, air void levels, and mixtures, the observed h- and k-values remained below these critical values. Therefore, it was concluded that the mean dielectric results are accurate and within acceptable ranges.

Figure 7 shows the relationship between standard deviations and the average values of the dielectric measurements. It was observed that 19.0 mm samples had significantly higher reproducibility standard deviations at lower dielectric values. This increase in variability can be attributed to both the overall higher variability in the results from the 19.0 mm samples and the increased variability observed at high air void levels. These results also indicate that there is no discernible trend in the standard deviations across the average results, indicating that there is no need to use a different statistical indicator for variability.

Table 3. Laboratory precision statements.

Variable	Parameter	All the Results
		Within Lab (r)	Between Lab (R)
Dielectric	Average	0.045	0.065
	Max	0.111	0.125
	Min	0.013	0.025
* Density [kg/m3]	Average	12.3	18.6
	Max	26.7	34.2
	Min	3.1	7.8

* Estimated from dielectric density equations (appendix of [13]).

shows computed repeatability and reproducibility statements concerning dielectric values and estimated density from dielectric to Gmb regression equations. These limits indicate that, within the same laboratory and under the same operator, the maximum expected difference in the measured dielectric values should be 0.111. Moreover, when comparing identical test specimens in different laboratories by different operators, the maximum expected difference in the measured dielectric values should be 0.125. On the other hand, statements for the estimated density indicate that the maximum expected difference in density measurements performed in the same laboratory by the same operator should be 12.3 kg/m³, and the maximum expected difference in density for identical test specimens measured in different laboratories with different operators should be 18.6 kg/m³. These values are significantly lower than the ones stated in ASTM D2726 (Laboratory Bulk Specific Gravity) and ASTM D6752 (Laboratory Vacuum Sealing), as shown in Table 1.

3.2. Field Dielectric Test Results and Precision Statements

Before testing, equipment consistency verification (standard High-Density Polyethylene—HDPE and line) was performed to check the adequacy of each sensor. The length of each section was measured, and metal plates were positioned at the beginning and the end of each test section (Figure 8) as reference points for each measured profile. One sensor was positioned approximately 15.2 cm from the longitudinal joint, while another was placed at the opposite end of the cart’s transverse bar (this should be close to the center of the lane). The operator collected dielectric values in distance mode, starting about 60 cm from the metal plate, with careful monitoring to maintain the sensor near the longitudinal joint about 15.2 cm away from it throughout the entire section. This process was repeated three times for each pair of sensors at both designated positions (center and joint). Due to the absence of a marked line for the sensor’s laser to follow, some variability was expected from one profile to another. However, this exercise was conducted to reproduce a typical dielectric profile test at any location within a pavement section and near the joint.

Figure 9 shows the average dielectric values measured at two locations: along the joint (six inches away from it) and within the mat, near the center of the lane. This measurement was performed for eight different sections. The results showed that for six of these sections, there were no significant differences in the average dielectric values between the joint and the within-mat test locations. Additionally, the average dielectric values obtained from all six sensors were not significantly different.

Figure 10 shows the standard deviations of dielectric values measured along the joint and within the mat near the center of the lane for eight different sections. Sensor 207 exhibited higher variability in seven out of sixteen cases compared to other sensors, showing the highest overall standard deviation. Despite this, the higher variability obtained with sensor 207 did not affect the consistency of the repeatability and reproducibility (h- and k-values), and the results were considered acceptable.

An evaluation of the height from the bottom of each sensor to the pavement surface was conducted to determine any distinguishing characteristics of sensor 207.

Table 4. Evaluation of sensor height during testing.

Location	Sensor	268	271	195	207	206	273
Center	Average Ht. [cm]	22.0	22.0	22.1	22.0	22.2	22.0
	Stand. Dev.	0.25	0.28	0.25	0.28	0.25	0.28
	Max	23.1	23.5	23.1	23.8	23.6	23.3
	Min	21.1	20.8	21.1	20.8	21.2	21.0
Joint	Average Ht. [cm]	22.3	21.8	22.4	21.8	22.4	21.8
	Stand. Dev.	0.28	0.23	0.28	0.23	0.25	0.23
	Max	24.1	22.9	23.9	23.0	23.7	22.9
	Min	21.2	20.7	21.1	20.9	21.3	20.7

shows the average, standard deviations, and maximum and minimum recorded heights when sensors were positioned near the center of the pavement section and the joint. These results do not reveal any significant difference in the height of sensor 207 relative to the others. However, it is essential to mention that, on average, all sensors were positioned slightly lower than the target height of 22.8 cm. This indicates a potential need for adjustments to correct any sagging in the device.

Figure 11 and Figure 12 show field dielectric consistency h- and k-values, respectively. The critical h- and k-values were obtained from ASTM E691 based on six laboratories (sensors) and three replicates. For this setup, the critical h-value is 1.92, and the critical k-value is 1.98. For all sensors, air void levels, and mixture types, the h- and k-values did not exceed these critical thresholds. Therefore, the mean dielectric results can be considered accurate and acceptable.

The potential impact of conducting dielectric tests near a joint was evaluated by performing two-sample t-tests of Sr and S_R values. Figure 13a,b show bar plots of S_r and S_R values, respectively, computed for locations near the joint and within the mat. Significant differences between S_r and S_R values can be observed within the same section. However, in some cases, these values were higher at the joint location and, in other cases, lower. To further investigate, a two-sample paired t-test was conducted for each parameter (S_r and S_R) at a 0.05 confidence level. The results showed no significant statistical differences (Sr: p-value = 0.292; SR: p-value = 0.486), indicating that separate precision statements for testing at the joint versus other mat locations are not required. Figure 13c shows the relationship between the standard deviations and the averages of dielectric values, where no specific trends were observed. This suggests that using the standard deviation is a valid approach for determining precision statements.

Table 5. Field precision statements.

Variable		Within Lab (r)	Between Lab (R)
Dielectric	Average	0.071	0.129
	Max	0.131	0.215
	Min	0.032	0.054
* Density [kg/m3]	Average	22.8	36.8
	Max	39.8	60.6
	Min	13.5	16.6

* Estimated from dielectric density equations (appendix of Leiva, et al., 2022 [13]).

shows the computed field repeatability and reproducibility statements for measured dielectric values and estimated density values. These limits indicate that the maximum expected difference in the average dielectric values at the same pavement profile or location by the same operator should be 0.071. Additionally, the maximum expected difference in the average dielectric values for identical test locations performed with different sensors should be 0.129. In this case, statements for estimated density were 22.8 kg/m³ for a single laboratory and 36.8 kg/m³ for multiple laboratories. These are significantly lower statements than those shown in ASTM D2950 (Nuclear Gauge) and ASTM D7113 (Electromagnetic Surface Contact), as shown in Table 1. However, the statements for ASTM D2950 and ASTM D7113 were obtained at single locations, whereas the dielectric density statements were derived from continuous measurement locations.

The sections evaluated at the NCAT Test Track varied in length, ranging from 30 m to about 67 m. To standardize the process, it was decided to evaluate precision statements for the 30 m sections. Therefore, all current and future evaluations of precision statements will be performed using the same length (30 m). Additionally, each 30 m section was further divided into subsections measuring 15.2, 6.1, 3.0, 1.5, and 0.9 m, allowing for the computation of precision statements for each subsection.

Table 6. Field dielectric standard deviations for all scenarios.

Section Size [m]	Within Lab—Sr			Between Labs—SR
	Average	Maximum	Minimum	Average	Maximum	Minimum
30 to 67	0.025	0.047	0.012	0.046	0.077	0.019
30	0.030	0.059	0.013	0.044	0.082	0.021
15.2	0.032	0.072	0.010	0.048	0.080	0.021
6.1	0.035	0.076	0.012	0.052	0.097	0.022
3.0	0.041	0.090	0.012	0.065	0.118	0.033
1.5	0.045	0.099	0.014	0.069	0.142	0.032
0.9	0.050	0.119	0.014	0.074	0.170	0.032

and

Table 7. Field dielectric precision statements of all scenarios.

Section Size [m]	Within Lab—r			Between Labs—R
	Average	Maximum	Minimum	Average	Maximum	Minimum
30 to 67	0.071	0.131	0.032	0.129	0.215	0.054
30	0.083	0.165	0.035	0.125	0.228	0.058
15.2	0.089	0.201	0.028	0.133	0.224	0.058
6.1	0.098	0.212	0.034	0.146	0.273	0.061
3.0	0.114	0.253	0.033	0.182	0.330	0.092
1.5	0.125	0.278	0.040	0.192	0.398	0.088
0.9	0.141	0.334	0.040	0.208	0.475	0.088

show the field dielectric standard deviations and precision statements for all scenarios. The data from these tables indicate that averaging dielectrics over a greater distance reduces variability and provides lower precision statements. It is essential to note that when performing these tests, no lines were marked on the pavement sections for operators to follow. This approach aimed to mimic field testing conditions on a newly paved surface, where an operator would test near a joint and follow an unmarked swerve or line pattern. As expected, due to the location variability of the sensor along the joint, the chances of matching profiles over a short distance were minimal, negatively affecting the repeatability of the results.

Table 8. Field estimated density precision statements of all scenarios.

Section Size [m]	Within Lab—r [kg/m3]			Between Labs—R [kg/m3]
	Average	Maximum	Minimum	Average	Maximum	Minimum
30	27.4	46.8	14.7	36.4	65.5	14.7
15.2	28.0	54.5	14.7	37.7	66.4	14.7
6.1	27.5	57.4	15.4	41.9	78.2	17.6
3.0	31.7	68.3	15.0	52.2	94.5	26.3
1.5	34.9	71.8	16.0	57.2	106.6	27.5
0.9	37.3	126.9	20.2	63.2	136.2	30.1

shows field repeatability and reproducibility statements for estimated density considering subsections of 30 m or less. Consistent with the results in Table 7, lower precision statements were achieved as the section length decreased. As previously mentioned, this is likely due to the challenge of obtaining profiles at the exact same location, especially with short-length profiles where there may be lateral and longitudinal offsets. To better evaluate variability on short distances (less than 6.1 m), it is recommended to draw a line on the pavement being studied and obtain at least three profiles per sensor.

Given that 30 m sections are now considered the standard size, the maximum expected difference in the average dielectric values within the same pavement profile or location by the same operator should be 0.165. Additionally, the maximum expected difference in the average dielectric values for identical test locations performed with different sensors should be 0.228. In this case, statements for estimated density were revised to 46.8 kg/m³ for a single laboratory and 65.5 kg/m³ for multiple laboratories.

4. Conclusions

The following conclusions and recommendations have been derived from the results of this work:

• Following field testing, it was determined that for all sensors and field variables, h- and k-values remained below the respective critical values. Therefore, the mean dielectric results are deemed accurate and acceptable.
• Preliminary laboratory conversions of dielectric values to air voids reveal that the maximum expected difference in the measured dielectric values in the same laboratory by the same operator should be 0.138. Additionally, the maximum expected difference in the measured dielectric values for identical test specimens in different laboratories with different operators should be 0.140.
• Preliminary field test precision statements indicated that the maximum expected difference in the measured dielectric values at the exact same pavement location by the same operator should be 0.083. Additionally, the maximum expected difference in the measured dielectric values for identical test locations performed with different sensors should be 0.125.
• All the computed precision statements were significantly below the reported statements for Bulk Specific Gravity and Vacuum Sealing in the laboratory and Nuclear and Electromagnetic density gauges in the field.
• Further evaluation and verification of the proposed precision statements are highly recommended with new mixtures incorporating different materials and gradations. This is especially important for evaluating and determining precision statements over shorter distances where, under this experiment, the sensor was not guaranteed to be reading precisely over the same line.