Accuracy Assessment of Low-Cost Lidar Scanners: An Analysis of the Velodyne HDL–32E and Livox Mid–40’s Temporal Stability

Abstract

Identifying and mitigating sources of measurement error is a critical task in geomatics research and the geospatial industry as a whole. In pursuit of such error, accuracy assessments of lidar data have revealed a range bias in low-cost scanners. This phenomenon is a temporally correlated instability in the lidar scanner where the measured distance between target and sensor changes over time while both are held stationary. This research presents an assessment of two low-cost lidar scanners, the Velodyne^® HDL–32E and Livox^® Mid–40, in which their temporal stability is analyzed and methods to mitigate systematic error are implemented. By immobilizing each scanner as it observes a stationary target surface over the course of multiple hours, trends in scanner precision are identified. Scanner accuracy is then determined using a terrestrial lidar scanner, the Riegl^® VZ-400, to observe both subject scanner and target, and extracting the distances between scanner origin and observed surface. Patterns identified in each scanner’s distance measurements indicate temporal autocorrelation, and, by exploiting the high linear correlation between scanner internal temperature and measured distance in the HDL–32E, it is possible to mitigate the resulting error. Application of the proposed solution lowers the Velodyne^® scanner’s measurement RMSE by over 60%, providing levels of measurement accuracy comparable to more expensive lidar systems.

1. Introduction

While accuracy assessment requirements may differ between industries and data applications, the need for understanding a measurement’s accuracy is universal; without understanding an instrument’s capabilities and limitations, users are limited in their ability to use it to their own ends. A measuring instrument’s accuracy is optimized (or restricted) by a variety of factors: the experience and competence of users, the conditions of its use, and the process by which its data is processed all contribute to the overall uncertainty and must be considered when determining the accuracy of the end products.

An accuracy assessment is a deliberate process by which the closeness of a measured or computed value is compared to an accepted significantly higher accuracy (often termed “true”) value of a particular quantity across the entire area of interest being observed [1]. This informs the users of the quality of the instrument and the derived product, and underwrites decisions made regarding the data’s applications. In order to report the accuracy of geospatial data, accuracy assessments are generated and published in conjunction with the collected data and its associated products.

While they are useful evaluations of an instrument’s performance, there are deliberate distinctions between precision and accuracy. Precision is defined as the “closeness with which measurements agree with each other”, whether or not the measurements “contain a systematic bias” [1]. Accuracy takes this one step further by comparing collected measurements to the location and orientation of target features within a real-world mapping frame through the use of deliberately surveyed check points or other “truth” data [1]. The accuracy of a dataset informs users of how well it agrees with the “ground truth” and informs appropriate use of the data with other collections or measurements of features within the captured area of interest. Considering stability takes performance evaluation one step further by assessing the instrument’s ability “to maintain its metrological properties constant with time” [2].

Parameters for assessing lidar system suitability for applications across different environments include the stability, precision (repeatability), and accuracy of their observations. This paper presents an accuracy assessment of two lidar scanners found in the topographic mapping community, and, using the assessment findings, proposes a methodology for deliberately mitigating error in their distance measurements.

1.1. Background

The research presented in this paper was first inspired by the results from Lassiter et al. [3], which sought to present a method to detect and measure both the absolute horizontal and vertical errors of UAS lidar surveys. This study was conducted using the Phoenix Lidar Systems^® Scout-32 mobile mapping system, which includes (among other sub-systems) a Velodyne^® HDL–32E lidar scanner. Through their analysis, the authors identified that there was a “positive bias in the height dimension” for flights conducted earlier in the study (first hour of flight time), versus those conducted later in the day. The temporal nature of the bias indicated that the scanner itself had a previously unaccounted error source, which further investigation suggested could be “a result of the scanner not being properly warmed up” (ibid.).

This finding is supported by existing literature, where Velodyne^® lidar scanners have been found to consistently exhibit temporal instability within the first hour of operation. Glennie et al. [4] found that there were “long term variations in the scanner range” in their Velodyne^® HDL-64E S2, though they were unable to attribute this error to one specific source. Chan et al. [5] found that “an approximate warm-up time of 2000 s was found for most lasers”, with each individual laser (of the HDL–32E’s 32 scanners) exhibiting a unique pattern of instability (range walk) when laser incidence and reflection angles were held constant for all observations. Glennie et al. [6] designed and implemented a methodology for measuring the stability of a low-cost Velodyne^® lidar scanner, the VLP-16. Like their previous research, Glennie et al. found evidence of a range walk in their scanners through the duration of their experiments, citing it as the reason “that the geometric calibration of the unit is not consistent” [6]. Less consistent, however, have been the causes to which this instability has been attributed, and minimal research exists where this instability’s impact on the absolute accuracy of the scanner’s distance measurements is analyzed and presented.

Less established in existing literature is the stability and UAS-mapping utility of the ultra-low cost Risley-prism based laser scanner from Livox^®, the Mid–40. First available on the market in 2019 for the cost of just $600, the Mid–40 was initially designed as an automotive lidar solution capable of detecting targets at up to 260 m within its circular scan area [7]. In their analysis of the scanner and the point clouds it produced, Ortiz Arteaga et al. [8] identified a “ripple effect” in close distance, indoor tests when observing a planar object that was found to have a significant “influence on range and angular measurements”. Glennie et al. [9] found that “the Livox^® Mid–40 sensor performance appears to be in conformance with the product specifications, with a ranging accuracy of approximately 2 cm” but, due to the beam divergence across the scan area, was “unable to properly model sharp edges and small features such as poles”. Studies of a similar Livox^® scanner, the Avia, have identified similar shortcomings. Teppati Losè et al. found that, while integrated as part of the DJI^® Zenmuse L1, the Avia failed to capture “a satisfactory level of detail” for objects defined by sharp edges (emergency stairs on the side of a building) [10]. Brazeal et al. [11], as part of the inclusion of the Mid–40 in the OpenMMS project [12], found that while “the angular observations for the Mid–40 sensor are more accurate than the manufacturer’s specification of <0.1°”, there was a systematic error in the “Cartesian coordinate representation of the observations” as “a result of the emergent beam not coinciding with the origin of the sensor’s reference coordinate system” [11]. As a result of this error, corrections to the calculated Y and Z coordinates were required by “upwards of 4.5 mm” [11]. While potential sources of systemic error in the Mid–40’s observations have been identified in literature, there has been limited discussion on the impact of these factors on the scanner’s absolute accuracy.

1.2. Research Questions

Given that there are a handful of published studies regarding the accuracy and precision of small form laser scanners, an important question remains: can the stability and absolute accuracy of distance measurements be directly observed in a robust, repeatable manner? Based on the demand for rigorous accuracy assessments of low-cost lidar scanners, the research presented here seeks to answer this question for the benefit of both the members of academia and the greater laser scanning community. As a whole, this study introduces methods to assess and increase the absolute accuracy of low-cost, sUAS-mountable laser scanners, informing their use in applications from rigorous scientific studies to practical utilizations. More specifically, this study presents:

• A methodology to directly observe and analyze the precision and absolute accuracy of low-cost lidar scanners;
• The identified range walk in low-cost lidar scanners and an analysis of its correlated scan parameters;
• Measures to correct, adjust out, or otherwise mitigate systematic errors in observations for high-accuracy sUAS surveys;
• A comparison between like-tier lidar scanners in terms of precision, accuracy, and temporal stability.

Through the execution of the outlined methodology, this study shows that low-cost laser scanners exhibit a temporal distance bias that is highly correlated with internal scanner temperature, and that survey accuracy can be improved through application of a per-laser linear regression in multi-laser scanners. Additionally, it is demonstrated that scan conditions, such as fluctuations in solar radiation on target surface and increased beam incident angle, have a measurable impact on scanner performance, which in turn can be effectively mitigated through deliberate selection of survey execution conditions and flight parameters.

2. Materials and Methods

2.1. Scanners Assessed

2.1.1. Velodyne^® HDL–32E

The HDL–32E is a lightweight lidar scanner first introduced by Velodyne^® in 2011 as an “ultra-compact and cost-effective optical sensor for many unmanned robotic vehicles [5]. Since its introduction, it has been popularized in use in the robotics, autonomous vehicle, and mobile mapping industries, and has been included as a keyhole system for multiple mobile lidar scanning systems [5,13,14]. For this study, the HDL–32E used was a sub-component of Phoenix Lidar Systems^® Scout-32 (Figure 1A), a pre-made laser scanning system featuring a navigation system and IMU (providing six degree of freedom inertial sensor measurements), RGB camera, and purpose-built housing for sUAS operations [14].

The HDL–32E functions by rotating a vertical array of 32 individual laser emitter/detector pairs about a central axis, with each scanner deliberately aimed with a vertical offset angle creating a fan-like scanning pattern (Figure 1B). As the scanner runs, its continuous spinning allows it to capture 32 distinct scan lines in a 360° ring around its central axis which, when coupled with the scanner’s physical motion through 3D space, enables a thorough scan of topographic areas within a moderate distance. The scanner is capable of collecting distance and calibrated reflectivity measurements, rotation angles, and synchronized time stamps (μs resolution) at a rate of up to 695,000 points per second (single return mode) or 1,390,000 points per second (dual return mode) [13]. Further specifications are listed in

Table 1. Velodyne^® HDL–32E Select Specifications [13].

Platform
Overall Dimensions	24.6 cm × 11.6 cm × 11.6 cm
Operating Temperature	−10 °C to +40 °C
Weight	2.40 kg
Sensor
Laser Wavelength	903 nm
Laser Classification	Class 1 Eye Safe
Channels	32
Measurement Range (min, max)	1 m, 100 m
Measurement Range (resolution)	0.002 m
Manufacturer’s Reported Range Accuracy	±0.02 m
Measured Returns	1 or 2
Field of View (Vertical)	+10.67° to −30.67° (41.33°)
Field of View (Horizontal)	360°
Angular Resolution (Vertical)	1.33°
Angular Resolution (Horizontal/Azimuth)	0.08°–0.33°
Beam Divergence	3 mrad
Rotation Rate	5 Hz to 20 Hz
3D Lidar Data Points Generated:
Single/Dual Return Mode	695,000/1,390,000 points per second

.

2.1.2. Livox^® Mid–40

The Livox^® Mid–40 is a recent addition to the lidar scanner industry, introduced by Livox^® (a subsidiary of DJI^®) in 2019 at a price point significantly below the automotive lidar standard price point ($599 at time of writing) [8]. A small form sensor, the Mid–40 is advertised by its manufacturer as a low-cost alternative for “autonomous driving, robotics, mapping, security, and other areas from small batch testing to large-scale applications” [7]. The scanner, like the Velodyne^® HDL–32E, has been incorporated in to both proprietary and open-source mobile mapping systems available on the open market. For this research however, the scanner was as pictured in Figure 2, separate from any other mapping system component.

The Mid–40 itself is capable of collecting up to 100,000 points per second within a circular field of view directly in front of the scanner’s glass window. Smaller than 10 cm³, it has been rated to collect points on objects at standoff distances of up to 260 m with precisions of up to 0.02 m (further specifications are listed in

Table 2. Livox^® Mid–40 Select Specifications [15].

Platform
Overall Dimensions	8.8 cm × 7.6 cm × 6.9 cm
Operating Temperature	−20 °C to +65 °C
Weight	760 g
Sensor
Laser Wavelength	905 nm
Laser Classification	Class 1 (IEC 60825-1:2014) eye safe
Channels	1
Measurement Range (min)	1 m
Measurement Range (max)	90 m at 10% reflectivity
	130 m at 20% reflectivity
	260 m at 80% reflectivity
Manufacturer’s Reported Measurement Range (precision) (1 σ at 20 m)	2 cm
Beam Divergence	0.28° (vertical) × 0.03° (horizontal)
Measured Returns	1
Field of View (Circular)	38.4°
Angular Accuracy	<0.1°
Point Generation Rate	100,000 points per second

). However, what sets the Mid–40 apart from other low-cost laser scanners is its scanning mechanism. Rather than rotate the laser scanner/receiver pair themselves (as is done in a multi-line scanner like the HDL–32E), the Mid–40 holds the two in a “solid state”, immobilized inside the scanner’s housing while the outbound and inbound light is directed through a pair of Risley prisms [11]. These wedge-shaped glass lenses act as optical steering mechanisms as they rotate in opposite directions, directing the beam in a rosette-style scan pattern in the circular scan area (Figure 3).

2.2. Research Methodology

2.2.1. Experiment Design and Execution

In order to assess the performance (precision, accuracy, and stability) of the Velodyne^® HDL–32E and the Livox^® Mid–40, it was necessary to identify parameters that may influence the scanner performance over time. In multiple iterations of experiments, the two scanners were immobilized, oriented towards a vertical wall (painted drywall or exposed concrete) between 2 and 30 m away from the scanner, and run for a multiple-hour collection period. A precision assessment was executed by comparing each scanner’s measurements of distinct subsections of the stationary target surface over time. Then, using a highly accurate terrestrial lidar scanner (the Riegl^® VZ-400), both scanner and target surface were mapped in a dense point cloud, enabling the standoff distance between scanner origin and target surface to be measured for comparison (Note that throughout the rest of this manuscript, these higher-accuracy distances are termed “true/truth” with the understanding that they are appropriate surrogates for actual true values used in accuracy assessment). The comparison of this “true” standoff distance to the measurements recorded by both the HDL–32E and Mid–40 provided an absolute accuracy assessment of each scanner, as well as a detailed understanding of their respective range walks, and how to mitigate any resulting error.

The subject scanners are positioned to maximize the target surface’s exposure to emitted lasers. Because the HDL–32E and Mid–40 have dissimilar scan patterns, the two scanners do not share the same specific orientation of each scan axis (X, Y, Z). The HDL–32E, being a line-type scanner, is laid on its side so its Z-axis is horizontal (rather than vertical), and its Y-axis is vertical (rather than horizontal) as shown in Figure 4. The end result of this rotation is that all 32 scan lines are approximately vertical, with a direct relationship between target height and scan angle. For the Mid–40, no rotation is required. As shown in Figure 4, the scanner is maintained in an upright position with the window parallel to the scanning surface.

Each experiment was executed in the same fashion: the scanner was immobilized on an adjustable, leveled tripod and oriented approximately perpendicular to the target surface. When conditions were set, the scanner was powered on and, while operating continuously for the entire experiment, observations were recorded for the first two out of every ten seconds. This two-out-of-ten collection method ensured that the lidar point clouds were manageable in size and enabled them to be easily isolated during analysis by timestamp, all while capturing any overall temporal trends in the data. The HDL–32E data were accompanied by a record of scanner diagnostic information, retrieved every 30 s of scanner operation, consisting of scanner internal temperature (°C) and power consumption (V). The Mid–40 lacks the capability to output this information, so no record of its internal temperature or power consumption was available for analysis.

Incident Angle

While both the HDL–32E and Mid–40 record horizontal and vertical scan angles, the incident angle of the scanner’s laser beam on the target surface potentially varies between experiments and must be calculated after the conclusion of each study. The incident angle is the difference between the laser’s angle of emission and the normal of the target surface, and is calculated using Equation (1), where (X, Y, Z) is the vector between the scanner’s origin and a point in the point cloud, and (N_X, N_Y, N_Z) is the normal vector to the target surface. The normal of the target surface is determined by extracting the scanner’s observations of the target surface and fitting them to a plane, thereby ensuring both vectors are in the same coordinate system.

$$\mathrm{Incident} \mathrm{Angle}={\cos }^{-1}\left(\frac{ \overrightarrow{\mathrm{A}}· \overrightarrow{\mathrm{B}}}{\Vert \overrightarrow{\mathrm{A}}\Vert ·\Vert \overrightarrow{\mathrm{A}}\Vert }\right)={\cos }^{-1}\left(\frac{\left({\mathrm{XN}}_{\mathrm{X}}\right)+\left({\mathrm{YN}}_{\mathrm{Y}}\right)+\left({\mathrm{ZN}}_{\mathrm{Z}}\right)}{\sqrt{{\mathrm{X}}^2+{\mathrm{Y}}^2+{\mathrm{Z}}^2}\times \sqrt{{\mathrm{N}}_{\mathrm{X}}{}^2+{\mathrm{N}}_{\mathrm{Y}}{}^2+{\mathrm{N}}_{\mathrm{Z}}{}^2}}\right)$$

(1)

2.2.2. Phase 1: Indoor Experiments (Precision)

Study Site

Indoor experiments, while not necessarily able to replicate the conditions of sUAS topographic surveys in both distances between scanner and target and environmental conditions, enabled a baseline of performance and stability of the laser scanners over time. Additionally, their repeatability allowed the identification of trends across datasets, reinforcing findings of scanner precision (or lack thereof) and associated stability. For these experiments, a temperature-controlled office with 2-m standoff distance from a blank, painted drywall wall was used. Maintaining a room temperature of approximately 22 °C and no sunlight exposure, the scanners were oriented towards the target wall, leveled, and immobilized (Figure 4) and run for three uninterrupted hours. This phase of the study consisted of a total of four separate experiments, two per scanner.

In order to ensure the scanners only collected measurements from uniform sections of the target surface, the HDL–32E was restricted to scan azimuths (α) ±8.00° relative to the target surface normal for all 32 individual laser channels. The Mid–40 required no such restriction so long as it was carefully aimed to collect data from empty portions of the target surface. Because the target surface was a generally planar wall in a controlled environment, observations from each scanner at like scan angles should have been consistent in measured distance, intensity, and incident angle over time. Deviations in any of these measurements indicated a lack (or loss) of scanner stability.

HDL–32E

After the completion of each experiment iteration, the two-second observations from the HDL–32E (Figure 5) were consolidated for analysis. This process was accomplished using Phoenix Lidar Systems^® Spatial Explorer [16] and Kitware^® VeloView [17]. Each observation file wassplit by individual channel (channel 00 through 31), from which measured distances were averaged to create a mean observed distance per channel.

By repeating this process for all observation files, a list of average distances per channel at each two-second epoch was generated. This simplified and isolated systematic phenomena for each individual channel. These lists were, in turn, consolidated and used to generate basic summary statistics (mean, maximum, minimum, range, standard deviation) for measured distances for each laser and, when combined, for the scanner as a whole (Figure 6).

Diagnostic data collected for the HDL–32E was downloaded and processed in accordance with the guidance published in the Velodyne^® HDL–32E User’s Manual, which contains step-by-step instructions for interpretation of the raw data provided by the scanner [13]. Once converted to °C, the measured internal temperatures could be compared, by timestamp, to the HDL–32E’s distance measurements. If a suspected correlation existed between any observed parameter and the scanner’s observations, the correlation of determination (R²) could be calculated, which would indicate the amount of variation in measured distance that could be explained by the scanner’s internal temperature or power consumption.

Mid–40

The raw observations from the Livox^® Mid–40 were recorded in a .lvx file format, which is a proprietary file type used by Livox^®. As with the HDL–32E, observations from every two-second epoch were saved and converted to a .csv file for analysis. Unlike the HDL–32E, this process could be done using just the Livox^® software package Livox Viewer, “convert to .csv” function [18].

Because the Mid–40 only has one scanner/receiver pair (unlike the 32 in the HDL–32E) and due to its rosette-style scan pattern, its observation files cannot be easily divided by individual scan lines. Instead, the observations were simplified for analysis by dividing the scan area into an array using the range of observed Y and Z coordinates (Figure 7). The distance measurements of each point within each cell were then averaged, and this calculated mean distance was assigned as each cell’s value. This process was repeated for every observation file, and the resulting series of arrays were combined to generate summary statistics of the scanner’s observation over the course of the three-hour experiment.

2.2.3. Phase 2: Outdoor Experiments (Precision and Accuracy)

Study Site

Outdoor experiments, used to more closely replicate the conditions of sUAS surveys, were conducted on the permanent parking infrastructure on the University of Florida campus. In order to maintain uninterrupted observation of a relatively flat target surface, the exposed concrete side of one of the university’s parking garages was used as the target surface, which was clearly visible from the top of the neighboring (shorter) garage (Figure 8). In this phase, a series of independent experiments were conducted per scanner, 30 m from the target surface during both night and day conditions. While a total of nine separate experiments were conducted under a variety of distances and experiment conditions, four were used for this analysis due to the similar collection parameters between scanners and iterations (

Table 3. Experiment List and Conditions.

Lidar Scanner	Standoff Distance	Day/Night	Experiment Date
HDL–32E	30 m	Day	4 June 2021
HDL–32E	30 m	Night	9 July 2021
HDL–32E	30 m	Day	13 July 2021
HDL–32E	60 m	Night	15 July 2021
Mid–40	30 m	Night	4 November 2021
HDL–32E	30 m	Night	21 November 2021 *
Mid–40	30 m	Night	21 November 2021 *
HDL–32E	30 m	Day	21 November 2021 *
Mid–40	30 m	Day	21 November 2021 *

* Experiment used in final accuracy assessment.

).

By setting the scanners approximately 30 m away from the target wall, four distinct horizontal surfaces were visible in each scan (Figure 8). This separation between sections allowed a comparison of like features between the two scanners, while also enabling the deliberate placement of targets across various scan angles (as described later in this section). Like the indoor experiments, scan azimuths for the HDL–32E were restricted to those that capture the target for extraction and later analysis. A similar restriction was also applied to the Mid–40’s measurements as its field of view included more than the desired target surface.

As previously described, there is a known degree of error associated with edge effects while using scanners such as the Mid–40 and HDL–32E [8,9,13,19]. Therefore, after collection was complete, edges of the target surface subsections were cropped out of the observations prior to any assessments. (These sections were easily isolated by using the scan angles of each point as recorded by their respective scanner). Additionally, for all outdoor experiments, a detailed weather record was downloaded from WeatherSTEM portal for Alachua County, Florida [20], which actively mines and stores meteorological data from the University of Florida weather stations at Ben Hill Griffin Stadium and Weimer Hall, approximately 1 km from the study site. The data used for this study included air temperature, atmospheric pressure, and solar radiation, all measured and recorded every 60 s.

The design of the outdoor experiments also included the placement of alignment targets along the target surface (Figure 9) and the inclusion of a Riegl^® VZ-400 Terrestrial Lidar Scanner (TLS), which, according to the manufacturer, has a distance measurement accuracy of ±0.005 m and a precision of ±0.003 m. The VZ-400 scanner was used to capture the target surface and assessed Velodyne^® or Livox^® scanner in a highly accurate and dense point cloud. By comparing measurements from the HDL–32E and Mid–40 to an independent and more accurate source (e.g., those from the VZ-400), this method of assessing data accuracy met the intent of the ASPRS Positional Accuracy Standards [1]. Similar methods of capturing truth data for scanner accuracy assessments can be found in the existing literature, such as [9].

During the HDL–32E data collection, alignment targets (Figure 9) were placed along the top of each target wall in line with the vertical scan lines being generated by the scanner. Alignment was made possible through use of an IR camera, which was used to detect the flash of individual laser pulses and, in turn, ensured the scan lines impacted on the desired targets. As the targets were placed outside of the identified sections of the walls that were being extracted for analysis, their placement and alignment had no impact on the collected measurements (Figure 9). As the HDL–32E and Mid–40 collections were conducted sequentially to ensure collection conditions were similar, the same alignment targets were used for both scanners despite their independent workflows.

Following the conclusion of each scanner’s collection window, the VZ-400 was used to capture the scanner and target surface in the same reference frame in a highly dense and accurate point cloud (Figure 10). These point clouds were used to estimate the “true” distance between each scanner’s origin and their observed points, which, when compared to the recorded Velodyne^® or Livox^® scanner observations, enabled the absolute accuracy assessment of each target scanner.

HDL–32E

The observations generated by the HDL–32E were subject to the same workflow as described in Section 2.1.2. with the additional step of separating the target surface by planar subsection. The VZ-400 data were added to the process through the mensuration of the HDL–32E’s scanner origins and focal points (Figure 11) in the VZ-400’s point cloud and the projection of its scan lines to extract the truth data for measured distances. Prior to data collection, black and white targets were attached to the top and base of the HDL–32E’s cylinder shape, centered as to mark the top and bottom of the central axis shown in Figure 11. These two points were identified in CloudCompare^® and used as inputs for the scan line projection script [21].

Using the published dimensions of the HDL–32E [13], the locations of all 32 individual diode/receiver pairs were calculated and plotted within the scanner’s cylinder. Additionally, the focal point for all diodes (appearing as a white circle shown in Figure 1B) was plotted as it rotates around the central axis for every 0.01° of rotation (Figure 11).

Points representing the scan lines for each individual laser were then plotted along vectors originating at the laser’s origin out to the approximate distance of the target surface (Figure 12). Where these vectors intersected the target surface (as scanned by the VZ-400) they were interpolated as an additional scalar field for the target surface point cloud with a width approximately equal to that of the laser’s footprint at scanner distance. Subsections of the target surface were cropped to match the HDL–32E observations, and distances between each observed point and scanner origin were used as the “true” distance between scanner and target. The accuracy assessment was then conducted by comparing the “true” distance (as determined by the VZ-400) and measured distance (as collected by the HDL–32E) for each of the four subsections of the target surface.

Mid–40

Like the HDL–32E, the workflow for processing observations made by the Mid–40 in this phase was identical to that from the precision assessments of phase two. However, additional steps were required to align and extract the truth data for the absolute accuracy assessment. First, as per the recommendations of [22], each of the Mid–40’s observation files were cropped by its measured intensity and radius values to eliminate erroneous measurements and isolate the target surface. The resulting target surface point cloud was aligned with the VZ-400 target surface point cloud using the alignment targets at CloudCompare’s^® “align two entities by picking equivalent points” tool (Figure 13) [21]. Following alignment, the Mid–40’s field of view was interpolated as additional scalar fields in to the VZ-400’s point cloud, identifying where the Mid–40’s laser hit the target surface. Finally, the distance between each identified point and the Mid–40’s origin was calculated using the scans of the Mid–40 in the VZ-400’s reference frame, which in turn acted as the truth value for the absolute accuracy assessment.

3. Results and Analysis

3.1. Velodyne^® HDL–32E

3.1.1. Phase 1: Indoor Experiments (Precision)

To assess the performance of the HDL–32E as a whole, its measurements of the target surface (±8°) from all channels were averaged per epoch (2 out of every 10 s for a 3-h experiment duration) for a series of approximately 1080 average measurements. This series of means was plotted over time (Figure 14) to visualize a number of temporal trends in scanner performance. First, there was a distinct decrease in measured distance over time, most noticeably in the first 30 min of scanner operation; in both iterations of indoor experiments, distance measurements decrease approximately 0.014 m at approximately the same rate over the same period of time. Second, during this initial period of operation, the scanner exhibited a higher standard deviation in measurements–an order of magnitude (or more) higher than in successive 30-min increments, due almost entirely to the observed temporal bias. Finally, after the initial range walk, there was minimal variation in the overall measured distance; scanner stability appears to increase over time.

These overall average distance measurements can be broken down by individual scanner, and in doing so it is possible to plot the observations of each of the 32 channels within the HDL–32E over time and compare their stability at an individual level. While all 32 exhibited similar curves in their measurements to the overall plot shown in Figure 14, there was a degree of discrepancy in some of the plots compared to others (Figure 15). To quantify this difference, the range and standard deviation of measured distances over the three-hour experiment for each individual scanner are shown in

Table 4. Summary of Velodyne^® HDL–32E indoor experiment indoor measurements (average of all points from each scanner within the −8° to +8° scan window).

Scanner	Offset	Range (m)	St Dev (m)	Average (m)
31	10.67°	0.011	0.001	2.226
29	9.33°	0.013	0.001	2.217
27	8.00°	0.011	0.002	2.196
25	6.67°	0.016	0.001	2.180
23	5.33°	0.012	0.002	2.172
21	4.00°	0.018	0.002	2.162
19	2.67°	0.016	0.002	2.143
17	1.33°	0.024	0.003	2.158
15	0.00°	0.021	0.002	2.124
13	−1.33°	0.015	0.001	2.119
11	−2.67°	0.015	0.001	2.111
9	−4.00°	0.018	0.002	2.107
7	−5.33°	0.026	0.004	2.105
5	−6.67°	0.021	0.002	2.101
3	−8.00°	0.027	0.003	2.099
1	−9.33°	0.014	0.001	2.093
30	−10.67°	0.017	0.002	2.103
28	−12.00°	0.019	0.002	2.105
26	−13.33°	0.013	0.002	2.105
24	−14.67°	0.019	0.002	2.114
22	−16.00°	0.019	0.002	2.120
20	−17.33°	0.011	0.001	2.131
18	−18.67°	0.017	0.002	2.128
16	−20.00°	0.013	0.002	2.144
14	−21.33°	0.022	0.003	2.150
12	−22.67°	0.021	0.003	2.160
10	−24.00°	0.018	0.002	2.168
8	−25.33°	0.015	0.002	2.192
6	−26.67°	0.017	0.002	2.207
4	−28.00°	0.010	0.001	2.225
2	−29.33°	0.015	0.002	2.250
0	−30.67°	0.012	0.001	2.253
Average		0.017	0.002	2.151

.

The distribution of scanners with relatively high and low measurement range or standard deviation did not appear to be correlated with position along central axis (and resulting offset angle). Variations in measured range were only as high as approximately 0.02 m between the most- and least-precise channels, but this, along with the minimal range in standard deviations did not appear to be statistically significant. As these results were consistent across all three indoor experiments, no one individual channel appeared to negatively influence the HDL–32E’s overall precision to a significant degree.

Due to the controlled nature of the indoor experiments, one of the few parameters that changed over the course of the experiment was the internal temperature of the HDL–32E itself. This data, extracted from the self-diagnostic output of the instrument, was converted to °C and plotted, over time, with the overall average measured distances from Figure 14 (Figure 16).

At a glance, there appears to be a correlation between the two curves (average measured distance and internal scanner temperature) that is generally consistent between both experiments. With the exception of a minor deviation in temperature curve in Control 2, the experiment results were almost identical: while operating in an indoor office kept at 22 °C, the HDL–32E took approximately 30 min to reach a steady state of 52.6 °C, at which time the distance measurements stabilized. Both internal temperature and measured distance remained consistent after reaching this steady state.

To confirm the suspected correlation between the two parameters, the HDL–32E’s internal temperature and overall average measured distance were plotted together and, using least squares to determine the slope and intercept coefficients, a linear trendline was fitted to the data (Figure 17). The resulting correlation coefficient (R) shows that the correlation between internal temperature and average measured distance is 0.9903 (very high), while coefficient of determination (R²) of 0.9807 shows that 98.07% of the variation in the measured distance can be explained by the internal temperature. This informs an assumption that the HDL–32E’s stability (or lack thereof) is highly correlated with its internal temperature, but this finding must be confirmed in survey-like conditions before it can be applied to UAS-lidar datasets.

The relationship between internal scanner temperature and measured distance was further explored by plotting the average measured distances (m) for each individual channel with the scanner’s internal temperature (°C). The resulting R² values for the overall scanner and each individual channel are shown in Figure 18, with the highest and lowest respectively shown in Figure 19. Of note, no individual channel has a higher R² value than the scanner as a whole (all scanner observations averaged together). Therefore, while it could be possible to generate and apply a linear regression for each individual scanner to potentially maximize the HDL–32E’s accuracy, it may not be as effective as applying one linear regression to all individual scanners. As with the measures of precision, the coefficients of determination were not distributed evenly across the HDL–32E’s central axis; there was no clear correlation between the linear measurement/temperature instability and offset angle of the channel. Additionally, there did not appear to be any relationship between channel precision and the calculated R² values per channel.

3.1.2. Phase 2: Outdoor Experiments (Precision and Accuracy)

In the second phase, both the HDL–32E’s precision and absolute accuracy were assessed through use of measurements collected from approximately 30 m under both day and night conditions, which were then compared to the observations of a Riegl^® VZ-400 terrestrial laser scanner. Both day and night iterations of the absolute accuracy assessment exhibit range walks similar to those that were observed in all previous iterations. Because of this, the scanner’s absolute accuracy varied over time; for approximately the first five minutes, the scanner’s average measurements were too “long” (greater than the “true” distance), and after that point they were (generally) shorter than the calculated “true” distance.

Figure 20 shows the observed (HDL–32E) and “true” (VZ-400) measurement curves plotted with scanner internal temperature for the average distance between the target surface and scanner. These curves show that there was a loss of precision when day conditions are introduced to the scan, which in turn undermine the overall accuracy of the scanner. Additionally, while the average distance curves again exhibited an apparent correlation with scanner temperature, there were anomalies in the day scan’s line indicating that additional factors are influencing the scanner’s stability.

Between the night and day iterations (Figure 21A,B, respectively), there were several trends in scanner accuracy that may inform conclusions as to its optimization. the scanner’s performance appears to optimize when the scan angle is closest to nadir. The full tables of calculated error (m) per individual scanner, per subsections of target surface for both the night and day experiments are available in Appendix A.

While offset angle did not appear to have a significant impact on scanner precision, the accuracy assessment experiments demonstrated that offset angle appeared to have a potentially significant impact on scanner accuracy. While individual scanner error is generally well within 0.015 m for most channels with offset angles less than 15°, individual channels with greater offset angles tend to exhibit greater error across all scan surfaces. With the exception of channel 4 during the day iteration, there were few examples of individual channels with offset angles greater than 15° with residuals that were regularly and significantly below the overall scanner RMSE.

Finally, while the individual channels that were the most accurate could vary between night and day iterations and even target surface subsections, the least accurate individual channels were much more consistent across experiment iterations. Channels 0, 2, 6, 8, and 10 showed the highest RMSE for all garage walls combined, and for all but one individual garage wall (channel 14 has an error of −0.037 m for the day scan of garage wall 4). This implies that scanner accuracy could be improved by simply removing these channels from the data—a course of action that is possible with both Velodyne^® and Phoenix Lidar Systems^® software. While doing so may not necessarily improve the HDL–32E’s stability (channels 9, 19, 26, and 29 were the least precise during the night iteration and channels 11, 13, and 15 were least precise during the day iteration), the potential improvements to RMSE could justify the loss of collected data.

As expected, RMSE in the HDL–32E’s observations was not consistent over time. Across the two hours of data collected, the RMSE was shown to generally increase from the beginning to the end of the experiments. This is in line with the results shown in Figure 20; while the scanner’s observations could start relatively close to the “true” distance between scanner and target surface, it stabilized at a distance measurement that was potentially centimeters off of truth. This trend is further captured in Figure 22, where the RMSE of each observation file of the HDL–32E is plotted over the time elapsed in the experiment. The fact that the RMSE was generally stable while the scanner’s measurements were stable implies that a correction could be more easily applied to reduce this error and increase the HDL–32E’s accuracy. This RMSE is further broken down by time and target surface subsection in Appendix B.

With the absolute accuracy of the scanner determined, it was possible to compare experiment parameters to observations and identify any potential impacts made by the environment. Again, three environmental conditions were used for this analysis: air temperature, atmospheric pressure, and solar radiation, shown in Figure 23.

The higher air temperature of the day iteration versus the night iteration was expected (Figure 23A), and was associated with the higher scanner temperature of the day iteration shown in Figure 20A. The solar radiation results identify a phenomenon where a loss of stability in measured solar radiation coincided with a loss of scanner measurement stability. Figure 23D shows that where there was inconsistent solar radiation, there were inconsistent measurements made by the HDL–32E.

This loss of stability correlated with an experiment factor other than internal scanner temperature potentially undermines the observed correlation between scanner temperature and measured distance. This theory is confirmed in Figure 24, where the R² between scanner internal temperature and measured distance increases from 0.7305 to a more significant 0.8650 with the observations affected by solar radiation removed.

3.2. Livox^® Mid–40

Following the same series of indoor test protocols as the Velodyne^® HDL–32E, the Livox^® Mid–40 was subjected to a series of indoor experiments to establish baseline of scanner behavior under controlled conditions. Because the Mid–40’s field of view is limited to ±19.2° in all directions from the normal of the scanner face, no restriction of scan angle was applied and the entire field of view was included in the analysis.

3.2.1. Phase 1: Indoor Experiments (Precision)

Taking the average distance measured across the field of view per epoch and plotting it over the duration of the experiment, it was possible to identify overarching trends in the scanner’s distance measurements. Figure 25 illustrates the range walk and autocorrection exhibited by the Mid–40, with an initial increase in measured distance of just under 0.020 m followed by a stabilization in both range and standard deviation after approximately 2000 s. Breaking the field of view down in to a 100 × 100 cell array, it is possible to visualize the trends in instability by both vertical and horizontal scan angle. The colorized raster is shown in Figure 26, and highlights, by percentile, the range of average measurements recorded per cell over time. The raster of calculated ranges shows that the across the entire field of view, the average range of distance measurements across the three hours observed was slightly higher than the change in measured distance found in the first 2000 s of the experiment (0.021 m versus the initial 0.017 m increase). This implies that, even after the scanner reached a relatively stable state, there was still a degree of instability in its measurements.

As the Mid–40 does not contain the function of internal scanner temperature measurement, it was not possible to determine what (if any) degree of correlation existed between the observed range walk and the operating temperature of the laser.

The field of view raster in Figure 26 illustrates a trend in the scanner’s precision that is mentioned in [8], where a “ripple” effect is observed and reported. Rather than presenting a strong correlation between beam incident angle and precision, the pattern in precision change appears to demonstrate a loose correlation with the distribution on point density across the point cloud, at least in the center of the scan area (Figure 27). Because the rosette-style scan pattern cannot guarantee an even distribution of points across the target surface, there is a possibility that there is a corresponding loss of precision in range of measurements where there are consistently fewer observations (where the rosette has the least overlap).

3.2.2. Phase 2: Outdoor Experiments (Precision and Accuracy)

To determine which trends observed in the indoor experiments carry over to survey-like conditions, a precision assessment was conducted at a scanner distance of approximately 30 m. As some obstructions blocked portions of the Mid–40’s field of view (Figure 28), the observations were limited to the four planar subsections of the target surface as indicated by Figure 8.

The final phase of the Mid–40 assessment added the observations of the Riegl^® VZ-400 to the outdoor experiments, which were repeated in both night and day conditions. The only significant change to workflow was the addition of processing the differences between observed measurements (from the Mid–40) and truth measurements (from the Riegl^® VZ-400).

As shown in Figure 29, when the average measured distance per epoch of the Mid–40 is plotted over time, there is no significant difference between night and day experiments in general line shape. Both day and night experiments had a series of apparently randomly distributed outliers (Figure 29A), which were removed when filtered out using a 60-s running average low-pass style filter (Figure 29B). Both experiments had a drop in average measurements of <1 cm over the duration of the experiments, matching the results from the Phase 2 outdoor precision assessment.

Of note in the outdoor experiments was the observed range walk of a lesser magnitude and in the opposite direction compared to the indoor experiments. While it is possible that this change in direction is dictated by the operating temperature, it is difficult to establish a correlation between the two. The study area temperature for indoor experiments was an average of 22.2 °C while the study area temperature for this outdoor experiment was an average of 14.2 °C. The calculated R² between the precision (range of measurements) and incident angle was found to be minimal with a value of 0.16.

When the range of measurements across the field of view are shown in a colorized raster (Figure 30), there is no significant difference in the Mid–40’s distribution of precision between day and night experiments. Additionally, the “ripple” effect observed in the indoor experiment was lost at increased distance; instead of point density having an apparent correlation with instability, the incident angle of the beam appeared to have a moderate to strong correlation with measurement precision, confirming findings of [8].

In order to collect “truth” measurements, the points on the target surface as observed by the Mid–40 must be identified in the VZ-400’s point cloud. This was accomplished using the alignment targets placed on the target surface observed by both scanners (Figure 31), yielding a sub-centimeter RMSE for both night and day alignments. The “true” distance between each point and the Mid–40’s origin was assigned as a value to each identified point in the Mid–40’s field of view.

Subtracting the “true” distance between point and Mid–40 origin for each point yielded the residual arrays shown in Figure 32, colorized by percentile. While there was minimal difference between night and day experiment precision, the day experiment had significantly more error in the middle-left portion of the field of view (−0.005 m for night, −0.013 m during the day). However, the average error for both experiments had a difference of less than a centimeter, with the majority of error in both experiments being in the negative (measured distances are shorter than “true” distances). The distribution of the error was also unique to this measure of scanner performance; there was no clear correlation between error and incident angle or point density.

Calculating the RMSE for each experiment showed both night and day experiments had like levels of accuracy, with the RMSE of the night experiment at 0.004 m and the day at 0.009 m.

Parameter Correlations

As there was no clear significant difference between the results from night and day experiments, and there was likely no impact from observed experiment conditions. This observation was consistent for atmospheric pressure, solar radiation, and air temperature (Figure 33). Additionally, the apparent loss of range walk observed between indoor and outdoor experiments did not appear to be related to experiment temperature, as the indoor experiments were conducted at an average of approximately 22.2 °C, and the accuracy assessment experiments were conducted at an average of 15.8 °C and 24.1 °C respectively.

The calculated R² between error and incident angle was found to be minimal at less than 0.1 for both and night experiments.

4. Discussion

4.1. Scanner Performance

4.1.1. Temporal Instability

Both the HDL–32E and Mid–40 exhibited a degree of temporal instability and autocorrelation in their indoor control experiments (Figure 34). At a standoff distance of approximately 2 m, each experiment had a range walk of just under 2 cm until the scanner reached a state of relative stability after 30 min of operation. This trend was repeated for the HDL–32E during all outdoor experiments, but was lost for the Mid–40 once standoff between scanner and target surface was increased.

These findings confirm that is there a temporal bias in both scanners, though this bias is not uniform between the two scanners. The HDL–32E’s range walk being highly correlated with its internal temperature further suggests that this instability in measured distance is the result of the heat generated by its continued operation. As its operating temperature cannot be directly measured in the Mid–40’s current configuration, a like finding of correlation could not be confirmed in the Mid–40. However, given that both scanners have moving components, operate within similar voltage ranges, and demonstrate a range walk under like indoor conditions, it is possible that the Mid–40 is also subject to a similar internal temperature-correlated distance bias.

The loss of observed range walk at increased scanner standoff distance for the Mid–40 may be attributed to the minimal calibration of the scanner. The HDL–32E used in this study has been deliberately calibrated by the distributor, Phoenix Lidar Systems^®, for high accuracy mobile mapping applications, whereas the Mid–40 only has the manufacturer’s factory calibration. While the effectiveness of calibration methods is outside the scope of this study, the overall increase of noise visible in the Mid–40’s point cloud at study distance versus the HDL–32E (and the HDL–32E’s general lack of significant outlier measurements) implies that Mid–40 lacks the same capabilities as the HDL–32E to compensate for potential variability in its observations. This lack of calibration may have removed any impact of range walk in the Mid–40’s outdoor experiments, but instrument bias being lost due to an increase in noise is hardly a viable solution to solving for scanner instability.

Regardless of why the range walk exists, the fact that it can be replicated under both controlled and survey conditions implies that it is an active source of error in the subject scanners, and must therefore be accounted for when executing surveys, topographic studies, or other high accuracy lidar applications.

4.1.2. Precision

The precision (repeatability) of the distance measurements made by the HDL–32E and Mid–40 were shown to have a moderate correlation with two separate scan parameters: the duration of scanner operation and the angle of beam emission relative to the scanner’s X-axis.

When range walk could be observed, the precision of both the HDL–32E and the Mid–40 had a direct relationship to the duration of their operation; the longer the scanner operates, the more precise its measurements became. During the first 30 min of use, the range walk resulted in a 66% and 75% reduction in standard deviations and 93% and 88% reduction in range of average measurements for the HDL–32E and Mid–40, respectively (Figure 35). A like trend was observed in the HDL–32E during outdoor experiments, whereas the Mid–40 exhibited overall higher calculated standard deviation and range in average measurements across the duration of the experiment.

While range walk was shown to impact scan precision over extended observation periods, the scan angle was found to impact measurement precision regardless of duration of scanner operation; measurement precision generally decreased as scan angle increases from nadir (Figure 36). In the HDL–32E, the horizontal scan angle (about the scanner’s Z-axis) consistently had a higher correlation with loss of precision than vertical offset angle (about the Y-axis) and beam incident angle (angle of beam relative to planar normal of target surface). In the Mid–40, this relationship was further complicated by the varying results from indoor to outdoor experiments. The results from indoor experiments indicate that precision is related to point density on target surface, which is a product of beam emission angle and the scanner’s rosette-style scan pattern. In outdoor experiments, this point density-precision relationship was mostly lost in favor of moderate correlations with scan angle and beam incident angle.

Given the findings regarding emission angle impact on scanner precision, it can be concluded that the most precise surveys were executed when the scanner (HDL–32E or Mid–40) was observing targets near-nadir and after having been allowed to stabilize in its operating environment.

4.1.3. Accuracy

Scanner error, defined in this research by the residual between observed and “true” measurements, was shown to be influenced by the aforementioned factors that correlate to changes in temporal stability and precision. As any change in the scanner’s measurements while held stationary changes the amount of observed error, this is a practical conclusion. However, this research shows that the presence of relatively high or low temporal stability or precision do not necessarily indicate the collection of the most accurate data.

As a whole, the HDL–32E exhibited an overall RMSE at or within the manufacturer’s specifications (0.020 m) for both night and day experiments. In all accuracy assessments conducted with the HDL–32E, the initial overall measurements were regularly too far (positive error) before decreasing to and past the “true” distance and stabilizing at a consistent level measurement that was shorter than the “true” distance (negative error) (Figure 37). Furthermore, the relationship between each individual scanner’s measurements and the “true” distance between scanner and observed point was not consistent across the array of individual laser/receiver pairs within the HDL–32E; there are individual scanners whose measurement curves, whether their measurements be low or high, never cross the “line” of “true” distance. The distribution of error across all 32 scanners was also not uniform; the largest error was found in scanners with offset angles greater than 15°, with the six scanners with the highest offset angle (scanners 0, 2, 4, 6, 8, and 10) having a level of error greater than 0.020 m.

The Mid–40 also demonstrated an overall RMSE that was within manufacturer specifications (0.020 m) in its accuracy assessments. While not consistent in its spatial distribution or magnitude, the measured error (after filtering potential outliers in the data) stayed on average within a centimeter of the “true” distance. The temporal distribution of error was shown to be relatively stable with no clear trends regarding level of error and scan conditions over time. However, the amount of error during the day experiment was more than double that of the night, implying that the addition of daytime conditions limits the average scanner accuracy (Figure 38).

4.2. Maximizing Accuracy

4.2.1. Solving for Error and Reducing RMSE

This study identified two main sources of error in the HDL–32E: scanner temperature and inaccurate individual laser emitter/detector pairs. Using the two successful accuracy assessments and the derived relationships between parameters presented in this study, three potential solutions were implemented to maximize scanner accuracy: removal of channels with high error, application of linear regressions developed from the same accuracy assessment (adjusting night measurements with regressions developed from the night accuracy assessment iteration, vice versa), and application of linear regressions developed from a separate, independent accuracy assessment (adjusting night dataset with day accuracy assessment results, vice versa). The resulting levels of RMSE before and after each of these techniques were implemented are shown in Figure 39.

The effectiveness of the linear regressions was further highlighted by applying them to each subsection of the target surface, extracted by the HDL–32E’s scan azimuth. While accuracy and precision were shown to decrease as scan azimuth increases, Figure 40 shows that measurements of target subsections that have a higher measured RMSE than published standards could be corrected to within the published 0.020 m maximum.

Removal of the least accurate channels from the dataset, while a relatively simple solution, did not demonstrate a large enough impact on the HDL–32E’s overall RMSE relative to other error mitigation techniques to justify loss of 20% or more of observed points. Instead, given the distribution of error over time, establishing and exploiting the relationship between error and scanner temperature demonstrated the most potential in maximizing measurement accuracy in a survey environment. Additionally, the net improvement in scanner RMSE using the regressions developed from alternate conditions (day regressions used for night data, night regressions for day data) implied that solutions developed under experiment conditions can be applied to observations from a variety of survey and mapping missions.

As the Mid–40 has no observed parameters with as high of a correlation with measured error as internal temperature in the HDL–32E, the findings regarding its precision and accuracy do not lend themselves to as straightforward an approach as a linear regression. Applying the recommendations of [22] to filter the data in terms of reflectance and scan angle are effective at filtering noise and reducing error, and are recommended for all topographic uses of the scanner. However, it is likely that both a rigorous calibration and the installation of diagnostic tools (a thermometer) are required to identify and mitigate systemic errors in the scanner’s distance measurements.

4.2.2. Impact of Scan Parameters and Conditions

Internal Scanner Temperature

While only observed in the HDL–32E, the internal scanner temperature is the parameter that has the highest observed correlation with average distance measurement with, in indoor experiments, repeated examples of a R² value in excess of 0.98. This correlation was observed in all experiments where the scanner internal temperature was recorded, with the lowest R² observed still being greater than 0.73. Though not consistent between individual scanners (R² ranging from 0.13 to 0.88), this study demonstrates that it is possible to build and implement linear regressions to reduce overall observed error. Given the range walk observed in the indoor tests with the Mid–40, it is implied that a similar relationship between internal scanner temperature and distance measurement may exist and could potentially be solved for in a like manner.

Solar Radiation

The disparity between night and day experiments for both the HDL–32E and Mid–40 in terms of precision and accuracy implies that the addition of sunlight on the target surface negatively impacts the overall performance of the scanners. Given that sunlight contains wavelengths of 905 nm (the wavelength used by both the HDL–32E and Mid–40), this is not unexpected. However, the fluctuations in observed solar radiation and the corresponding loss of precision observed in the HDL–32E implies that accounting for just the presence of sunlight during day experiments is insufficient for mitigating its effects. If collection at night is not possible due to safety concerns, local airspace restrictions (a lack of a waiver from the Federal Aviation Administration (FAA Part 107.29(b): operation at night) in the United States, for example), or survey requirements, it is recommended that consistent levels of solar radiation be prioritized during surveys to ensure consistent results.

Scan Angle

The HDL–32E and Mid–40 exhibited a moderate degree of correlation between laser scan angle and both their respective levels of precision and accuracy. Generally, as emission angle (and incident angle) increased, precision and accuracy decreased, with the lowest error and range of measurements being found near the instrument’s nadir. However, no correlation between performance and scan angle was found to meet or exceed that of other parameters (e.g., scanner temperature), and therefore developing a solution to account for identified error has not proven to be effective. Rather than retroactively solving for increased uncertainty at higher scan angles, it is recommended that data collection be executed with the highest feasible swath overlap to maximize the percentage of the study area observed near nadir. The resulting increase in redundant observations will likely mitigate errors from other sources present in the study area or instrument.

5. Conclusions

By conducting a deliberate assessment of the Velodyne^® HDL–32E’s and Livox^® Mid–40’s distance measurements in controlled and survey-like conditions, a holistic understanding of each scanner’s spatial and temporal trends in accuracy and precision were developed. It was shown that, under the reported conditions and limitations, the HDL–32E and the Mid–40 each me and exceeded their published manufacturer’s accuracy specifications, demonstrating potential for use in high-accuracy topographic surveys for a fraction the cost of their high-tier counterparts.

Each scanner exhibited a clear range walk of just under 0.020 m during the first 30 min of indoor conditions, which in the HDL–32E was found to be highly correlated with scanner internal temperature. This range walk and like correlation with temperature was found again in outdoor experiments with the HDL–32E, but was lost in the noise of the observations in the Mid–40. Allowing the scanners to operate for 30 min and stabilize in collection conditions prior to use effectively mitigated this instability.

Beyond the range walk, scanner precision and accuracy were found to be directly impacted by scan angle; as the angle of emission increased from the scanner’s nadir, there was a moderately correlated loss of both precision and accuracy in both the HDL–32E and Mid–40. However, this angular offset was not equal in all axes, and the resulting loss of performance did not lend itself to a correction model. In order to mitigate this loss of precision and accuracy, it is recommended to condense flight lines for sUAS surveys and restrict scan angles to ±15° when observing generally flat or planar environments.

The presence of solar radiation on the target surface was found to decrease the precision and accuracy of scanner measurements to a minor degree, while fluctuations in solar radiation were found to be strongly correlated with potentially significant losses of precision. Limiting lidar collections to hours of night was found to reduce RMSE by up to 5% in the HDL–32E and 50% in the Mid–40.

By taking advantage of the relationship between scanner temperature and measured distance by each individual scanner of the HDL–32E, linear regressions were developed and used to adjust the reported distance measurements, resulting in a decrease in RMSE of over 88% when applied to like experiments. More importantly, a RMSE reduction of over 33% was found when regressions from one experiment were applied to the other, indicating that solutions developed in test conditions can be applied to datasets collected under a variety of conditions.

This research demonstrates a methodology to assess the temporal stability of lidar scanners that can be implemented by most lidar users. However, it is limited to just the two subject scanners. Future research should explore the performance of other low-cost lidar scanners from both Velodyne^® and Livox^®, as well as other manufacturers in the mobile mapping industry. Additionally, the impact of calibrations on the mapping of scanner stability (and its resulting mitigation) should be explored in future applications.