Evaluation of Uncrewed Aircraft Systems’ Lidar Data Quality

Abstract

Uncrewed aircraft systems (UASs) with integrated light detection and ranging (lidar) technology are becoming an increasingly popular and efficient remote sensing method for mapping. Due to its quick deployment and comparatively inexpensive cost, uncrewed laser scanning (ULS) can be a desirable solution to conduct topographic surveys for areas sized on the order of square kilometers compared to the more prevalent and mature method of airborne laser scanning (ALS) used to map larger areas. This paper rigorously assesses the accuracy and quality of a ULS system with comparisons to terrestrial laser scanning (TLS) data, total station (TS) measurements, and Global Navigation Satellite System (GNSS) check points. Both the TLS and TS technologies are ideal for this assessment due to their high accuracy and precision. Data for this analysis were collected over a period of two days to map a landslide complex in Mulino, Oregon. Results show that the digital elevation model (DEM) produced from the ULS had overall vertical accuracies of approximately 6 and 13 cm at 95% confidence when compared to the TS cross-sections for the road surface only and road and vegetated surfaces, respectively. When compared to the TLS data, overall biases of −2.4, 1.1, and −2.7 cm were observed in X, Y, and Z with a 3D RMS difference of 8.8 cm. Additional qualitative and quantitative assessments discussed in this paper show that ULS can provide highly accurate topographic data, which can be used for a wide variety of applications. However, further research could improve the overall accuracy and efficiency of the cloud-to-cloud swath adjustment and calibration processes for georeferencing the ULS point cloud.

1. Introduction

Highly detailed and accurate digital elevation models (DEMs) can provide valuable insights into important topographic characteristics, such as terrain steepness and roughness [1]. Roughness in particular has proven to be an integral characteristic when discriminating geomorphological features, such as landslides [2]. With the advent of successive advances in light detection and ranging (lidar) technology, detailed DEMs have become increasingly simple to produce for nearly any terrain type. Further, as uncrewed aircraft systems (UASs) technology becomes more common and accessible, they have emerged as a popular platform for laser scanners, thus commencing a new phase of lidar data acquisition. Due to its small size, quick deployment, and low cost [3], uncrewed laser scanning (ULS) is often considered a desirable alternative to its counterparts: Airborne laser scanning (ALS) and terrestrial laser scanning (TLS). In comparison to ALS, ULS can be more cost effective for areas that are a few square kilometers in size. Although TLS provides a higher point density (typically few cm-level compared with several cm to dm-level) and accuracy (mm-level versus cm-level in the relative sense), it is not efficient to collect data for an area of substantial size, especially in forested or rugged environments. The point density of TLS varies substantially with the distance from the scanner. In contrast, the point density from ULS is similar to mobile laser scanning, where the point density is relatively high within a scanline (cm-level), but the spacing is larger between scanlines (dm-level). Given its vantage point, ULS also generally provides fewer data gaps for topographic surveys. Recently, Resop et al. [4] illustrated these accuracy and point density advantages when using ULS for a relatively small project area of a concrete bridge.

1.1. ULS Applications

Nonetheless, despite these promising results, ULS is still in its infancy; hence, researchers and practitioners have begun testing the capabilities of ULS to reliably support a variety of applications. One of the earliest examples comes from Wallace et al. [5], who developed a low-cost ULS system on an OktoKopter platform for forest management applications. Additional examples include canopy height modeling (CHM) and diameter at breast height (DBH) [6], and erosion monitoring on Alpine slopes [7]. For instance, Brede et al. [6] compared canopy height models derived from ULS-based lidar point clouds to TLS-based lidar point clouds. Wallace et al. [8] compared two separate technologies using the UAS platform, structure from motion (SfM) and ULS, in order to assess each respective technique’s ability to model forest structure. A somewhat similar study was performed by Salach et al. [9], who compared ULS to both UAS SfM and ALS. The work by Mayr et al. [7] provides an example of the capabilities of ULS to successfully monitor erosion through the use of multi-temporal datasets. Part of their work also included performing a comparative analysis of ULS and TLS. They stressed that further studies into the effects of the scan angles and ranges of the two systems would provide additional insight into understanding the differences between these technologies.

1.2. ULS Accuracy and Data Quality

In such evaluations, it is important to examine the accuracy and resolution (point spacing and density) of the acquired point cloud. Both of these characteristics have significant impact on the overall accuracy and quality of the dataset as well as the ability to describe features in the area of interest. Model accuracy is impacted due to the fact that higher point density decreases the amount of uncertainty between measurements as surface interpolation occurs across shorter distances. The ability to physically describe features is also impacted as higher point density not only provides greater detail of the captured features but also allows smaller features to be resolved due to the increase in detail and probability of capturing the said features. A recent study [10] comparing ALS, TLS, and ULS concluded that ULS fills the gap between TLS and ALS by enabling both extensive coverage of a project area that is difficult to achieve with TLS as well as the acquisition of sufficient point density for describing small morphologies missed in ALS, which is vital for landslide monitoring applications. Additionally, Resop et al. [4] also compared ULS to ALS in order to show the ability of ULS to capture important topographic features. In the USGS Lidar Base Specification (v1.3), for the highest quality level (QL0), the USGS suggests an aggregate nominal point spacing of ≤ 0.35 m, and an aggregate nominal point density of ≥ 8 points/m² [11]. While these values are specific to ALS, they provide a useful standard to compare ULS results against.

1.3. ULS Point Cloud Calibration and Georeferencing

ALS and ULS georeferencing procedures are similar in nature as they both typically require highly accurate observations of the lidar sensor’s position (X, Y, Z) and attitude (roll, pitch, yaw) with respect to time. To acquire this information, a Global Navigation Satellite System (GNSS) aided inertial navigation system (INS) comprised of an inertial measurement unit (IMU) and a GNSS receiver are used. The GNSS-aided INS is rigidly mounted to the lidar system, with the offsets (distance and angular) between the origins of these sensors known. The distance offsets, often referred to as lever arms, can sometimes be determined through direct measurements or obtained from the design drawings produced by the manufacturer. Due to the inability to measure these lever arms exactly, calibration processes [12] implement a software solution through adjustments based on overlapping geometric features that can refine these measured lever arms while also determining the boresight angles, which cannot be measured. The boresight angles are angular misalignments between the INS and scanner [13] and are also determined through calibration. Oftentimes, in addition to the calibration, a mission-specific swath adjustment is implemented once the point cloud has been generated to account for offsets between individual swaths. In most cases, the swath adjustment is only applied to improve the internal consistency of the point cloud but is not utilized to recompute the trajectory (which is a much more rigorous solution as discussed in Olsen et al. [14]). Some manufactures utilize a software-computed lever arm (SCLA) through performing a calibration per flight to identify offsets from the iterative closest point (ICP), linear feature, or planar registration of preliminary point clouds of adjacent swaths (herein, a flight path is defined as the path from take-off and landing while a swath is a linear flightline segment of the overall flight path). The process is meant to minimize GNSS errors between flights and improve consistency from other error sources; however, it applies those differences to the lever arm offsets (which should be constant given the rigid platform) per flight for simplicity in the computations to generate the point cloud for each flight. The second process, a swath adjustment, is then an additional software correction applied to minimize differences between adjacent swaths after the initial direct georeferencing. While a variety of approaches exist for swath adjustments of ALS data, to date, little research has focused on the challenges of implementing a swath adjustment for ULS data, particularly in natural environments where well-defined objects are limited. Li et al. [13] presented a combined boresight calibration and swath adjustment procedure for ULS in an urban environment. Additionally, Glira et al. [15] presented swath adjustment through the use of a spline trajectory correction model, which saw moderate success at improving both the absolute and relative orientation of the swath. Li et al. offered another example of an innovative registration technique using a coarse-to-fine method to first perform a coarse registration through trajectory correction, and then performing fine registration by iteratively minimizing the depth map discrepancy from SfM and the raw laser scans [16].

1.4. Objectives

While each of these recent studies and efforts show different assessments of the capabilities, accuracy, and resolution of ULS, the objective of this paper was to perform a more rigorous and detailed assessment beyond the work to date to analyze both quantitatively and qualitatively the accuracy, quality, and internal consistency of direct georeferencing with a ULS, particularly during key steps in the georeferencing stage, such as the flight calibration or swath adjustment. From the previously cited studies, it is difficult to fully infer the current capabilities of ULS independent of the chosen applications. For example, the method in [15] was only tested on a single pair of swaths in a well-defined urban setting, whereas this paper’s dataset incorporates over a dozen such swaths in heavily forested terrain. Therefore, this analysis more holistically considers the overall accuracy and data quality to inform the reader of the current general capabilities of ULS in order to facilitate informed decisions about its use. The analysis was accomplished through a series of experiments for ULS data collected for a landslide complex:

• Experiment #1—Flightpath calibration evaluation. A key component of ALS data processing is to perform a calibration and/or swath adjustment between adjacent swaths to minimize differences [17]. This study investigates challenges with field calibration processes in application to ULS data given the much shorter swaths resulting in less well-defined objects (e.g., planar features for evaluation) per swath combined with relatively high GNSS and laser measurement uncertainty compared to the ability to directly measure the lever arm offsets or at a dedicated calibration site.
• Experiment #2—Quantitative accuracy assessment. The second analysis described herein addresses the overall accuracy (bias and precision) of ULS point clouds and derived DEMs. Comparisons are made to (A) elevations of ground check targets predominately from GNSS, (B) elevations of a combination of reflectorless and prism-based total station measurements in road, gravel, and vegetated areas, and (C) TLS point clouds, which provide a 3D accuracy assessment.
• Experiment #3—Point density assessment. The third experiment provides a detailed analysis of how well ULS captures and represents the ground surface through statistical analysis of the ground point density and coverage within the study area. This assessment is performed by computing summary statistics of ground points on a cell-by-cell basis within a 1 m × 1 m grid cell. Evaluation of the ground point density is a common assessment of lidar data quality as it often relates to a higher accuracy and increased ability to discern captured features in the topography.
• Experiment #4—Change analysis. The chosen study area provides an excellent opportunity to perform a practical assessment of ULS accuracy and quality through change detection and analysis. Historical ALS data from 2009 serves as a topographic baseline to compare the more recently acquired ULS data against. This comparison was performed by direct cell-to-cell subtraction of the bare earth DEMs of the respective epochs. The purpose of this experiment was twofold: (1) To assess topographic change in the area as part of ongoing monitoring of an active landslide complex, and (2) to ascertain whethre ULS data can provide high enough spatial detail and accuracy for change analysis to be successful and provide important information on landslide kinematics.

2. Study Area

The study area (Figure 1) encompasses an active landslide complex in Mulino, Oregon, USA approximately 22 miles southeast of the city of Portland. The Oregon Department of Transportation (ODOT) is interested in this site due to its slow and steady change over time, which is the primary reason it was chosen as a site to test the capabilities of ULS, with the real-world application of change detection as a backdrop. Additionally, the site is an ideal candidate for testing ULS data due to its variation in topography and landcover. With a paved highway running through the center of it, the area of interest (AOI) also contains relatively high relief and areas with varying degrees of vegetation, ranging from open fields to moderately dense forests, with a mix of both coniferous and deciduous trees. The “desired survey extents” boundary in Figure 1 represents the entire AOI set as an overall objective (if time permitted) before data collection began, while the “priority boundary” represents the area of a specific landslide, which was the main focus in the landslide complex. Due to time constraints, however, the “actual survey extents” show what was actually surveyed over the two-day campaign.

3. Data Collection and Processing

A flowchart showing the overall workflow beginning with data collection and ending with change analysis (Experiment #4) can be seen in Figure 2.

3.1. Supporting Data Collection

All survey data were collected over a two-day period (

Table 1. Overview of supporting data collection effort.

Data Type	Equipment Used	Number of Measurements/Setups
GCT	Leica GS14/GS18	38
Total Station (Reflectorless)	Leica TS15P (1”)	245
Total Station (360° Prism)	Leica TS15P (1”)	240
TLS	Leica P40	24 setups
TLS	Riegl VZ400	33 setups

and Figure 3). Near the UAS command station, a Leica GS18 GNSS Receiver was set up as a GNSS base station to send real-time kinematic (RTK) corrections to roving Leica GS18 and GS14 GNSS receivers to survey ground check targets (GCTs) and to two additional Leica GS14 GNSS receivers mounted atop each of the TLS systems for positioning information. Static data were also logged at each rover simultaneously for post-processing to provide improved coordinates.

For GCTs, static GNSS occupations were performed by occupying a nail in the center of a 1.0 by 1.0 m and 0.3 by 0.3 m black and white checkerboard ground targets. A Leica TS15P 1” Total Station collected both reflectorless and Leica 360° prism measurements at two separate setup locations within the project area. The survey incorporated two separate TLS systems: A Leica P40 and a Riegl VZ-400, scanning at angular resolutions of 0.011°, and 0.025° to 0.035°, respectively. The TLS scans primarily covered the area along the main highway and some arterial roads in the AOI. All the scans were registered and geo-referenced using the GNSS data for the scanner origin combined with cloud-to-cloud surface matching.

3.2. Survey Data Processing

The GNSS base station occupations were processed in an OPUS static processing engine by the National Geodetic Survey. The base coordinates were computed as a weighted average of the earth-centered, earth-fixed (ECEF) coordinates from a total of four occupations with an average of 3.75 h each. A weight of unity divided by the standard deviation squared was used for each observation. OPUS static reported an average uncertainty of 0.041 m in the orthometric heights for the GNSS solutions; however, the maximum discrepancy between the different occupations in orthometric height was 0.021 m. The same base station was used for the ULS and the GNSS control network. The GNSS coordinates were projected from ECEF coordinates in NAD83(2011) Epoch 2010.00 to OCRS Portland Zone coordinates with the same datum. Geoid 12B was used to obtain orthometric heights from the ellipsoid heights.

The total station data were georeferenced using a resection process to GCTs followed by a least squares adjustment of the total station observations with GNSS baselines. The TLS scans were registered in Leica Cyclone. The scans were constrained to be level since high-quality level compensators were employed. The GNSS coordinates representing the scanner origin were also used as constraints, such that the primary transformation parameter solved for in the registration was the orientation (yaw) of the scanner.

3.3. ULS Platform and Processing

The ULS platform used for this survey was a Phoenix Lidar Systems MiniRanger ULS (Figure 4). This platform comprised of a DJI Matrice M600 Pro UAV equipped with a Riegl miniVUX laser scanner, Northrop Grumman uIMU, custom GNSS unit, and a Basler acA4096gc aerial camera, which concurrently collected photographic RGB imagery with the lidar data. Figure 4 and

Table 2. ULS acquisition parameters based on manufacturer specifications and assumed values during planning. Actual parameters of overlap and spacing will vary somewhat in flight.

Flight Parameters	RGB Imagery	Lidar System
Altitude: 100 m	Camera Sensor Presets: Basler/acA4096-11gc	Lidar Model: Riegl miniVUX
Forward Velocity: 6 m/s	Sensor Size: 25.40 mm across/13.44 mm along	Pulse Repetition Rate: miniVUX1-UAV 100 KHz
Lateral Overlap: 70%	Pixel Count: 4096.00 across/2168.00 along	Spin Velocity: 27.78 LPS
Forward Overlap: 80%	Focal Length: 14.58 mm	Angular Step: 0.1000°
Lidar Field of View*: 60°	Pixel Pitch: 3.44 µm pitch	Angular Range: 15° to 345°
Swath (flightline) Spacing: 52.87 m
# Flights: 14

summarize this platform’s specifications and flight parameters. Due to the significant slope of the terrain, substantial area of the tall tree canopy, and other topographic factors, a flight altitude of 100 m above ground level (AGL) was used, with a forward velocity of 6 m/s. These two flight parameters together have a direct effect on the resulting data product in terms of point spacing and density. It is important to note that the implemented acquisition altitude of 100 m AGL is the maximum altitude recommended by the ULS manufacturer but was required to clear tree canopy and efficiently survey the area of interest. Significant lateral and side overlap were desired at 70% and 80%, respectively, in order to limit data gaps from the dense vegetation and facilitate the registration process. Note that the 80% sidelap assumes the field of view (FOV) of the ULS sensor is 60° rather than the actual value of 360° in planning to account for blockage from trees and the fact that more dense/accurate data will be collected within a much smaller field of view. As the FOV increases it is expected that the total number of point returns to the sensor will begin to decrease due to the increase in range to the surface. Based on these flight parameters the distance between successive swaths was approximately 50 m and the resulting point cloud spacing was estimated to be 24.20 points per square meter.

Manufacturer standard operating procedures were followed during the flights. Static GNSS data were logged for at least 5 min before and after each flight. Kinematic alignment was typically obtained through at least two “figure 8” flight patterns on the front and back of the flight unless there was insufficient battery life remaining. Individual swaths were limited to 1 km in length, and the collection time was limited to 14 min in addition to the time required for the in-flight calibration procedure.

The ULS trajectory combining the INS and GNSS observations was processed in Inertial Explorer 8.70 software. Inertial Explorer provides built-in profiles for aerial, UAV, pedestrian, and ground vehicle and marine applications, of which the UAV profile was used for this project. The GNSS and INS data were processed simultaneously in the forward and reverse directions in time using differential GNSS measurements between the base and ULS GNSS unit. The ULS survey data were processed against the same GNSS base station used for the TLS and total station surveys.

Both the loosely coupled and tightly coupled solutions were analyzed in multiple iterations. For the loosely coupled solution, the PPK GNSS data were processed first and separately. The updated position and velocity information were then processed with the INS data, and a corrected trajectory (updated position, velocity, and attitude) was produced. The tightly coupled solution processes the GNSS data with the INS data to produce the corrected trajectory. For both solutions, the static and kinematic calibrations performed during data acquisition were considered, but more weight can be given to one or the other—with an extreme case being the elimination of the static or kinematic calibration data—to help reduce the influence of noise or poor-quality data. Adjusting the weight given to the kinematic and static calibration in the Kalman filtering, and considering both tightly and loosely coupled solutions, processed trajectories were iteratively analyzed until a best solution was achieved, with best defined by the solution with the least amount of deviation between the forward and reverse solutions, and lowest relative error. The minimum standard was to produce a solution with a combined positional separation of less than +/−2 cm and combined attitude separation of less than +/−2 arcmins in time.

The point cloud was then constructed from the corrected trajectory combined with the lidar range and angular observations. The captured images were processed using the Phoenix LiDAR Systems processing software, Spatial Explorer 4.0.3. The lidar data was fused with RGB values typically within 250 m of the ULS location, allowing color from photographs taken in adjacent swaths to be considered in coloring the point cloud. Lidar data was then split by swath and exported in ASPRS las format, and extraneous data collected before or after the swaths were completed, or in between swaths when the ULS was turning, were separately exported for more rigorous quality control.

Initially, data were processed in Inertial Explorer and Spatial Explorer using per flight computed lever arms (representing the offsets between the ULS IMU center and ULS scanner origin) followed by a swath adjustment in Terrasolid TerraMatch software. However, current swath adjustment algorithms are based on ALS data collected from large swaths where many intersecting planar features are used to extract matching tie lines. When attempting to perform this on ULS data, it was found that there were relatively few, if any, well-defined matching features between swaths. As a result, the swath adjustment results did not help improve consistency and thus were not used. Instead, the manufacturer calibrated lever arms (MCLA) were used, along with a –90-degree rotation about the Z-axis to complete the coordinate transform. The lever arm values used were 0.000, –0.105, and 0.395 m in the X, Y, and Z directions, respectively.

3.4. Ground Filtering and DEM Creation

The data were ground filtered using a custom ground filter developed by the research team. This ground filter performs progressive refinement by starting with a coarse cell size (10.0 m) and iteratively refines the model to the desired cell size (0.5 m). At the onset, a coarse raster is created by finding the median value of the points within each cell. A focal operator returning the median value of cells within a window (3 by 3) surrounding a center cell is then applied to each cell to minimize noise from artificially low points. A normal vector is then estimated for each cell and points within a threshold distance (0.5 multiplied by the cell size of the current iteration) of a plane derived from that normal vector and the centroid of the cell are then kept for the next iteration. Points within the threshold are classified as ground. Points further than twice the threshold are not considered in subsequent refinement stages. Points in between are marked as unclassified. The process is iterated by linearly interpolating between the coarse and fine cell sizes, median window sizes, and thresholds. For estimating the median value for each cell, only points classified as ground points are used.

Following this operation, a preliminary DEM was created as a Delaunay triangulation of the centroids of all ground points located within each cell. Only points within 5 cm of this model were then given a final classification as ground and everything else was unclassified. Next, minor manual cleanup was performed at locations such as the edges of the study area where the ground filter did not work as well. A final DEM (Figure 5) was created as the rasterization of the average Z coordinates of all of the ground points within each cell. This process helps minimize noise in the final model through the averaging process. In the end, a 0.5 m bare earth DEM was produced. This overall ground filtering process was iteratively refined in order to find the optimal cell size for this particular study area. While other studies have achieved higher resolution DEMs with ULS, it can be challenging to capture sufficient ground points in the dense forests on steep terrain, which is common in the Pacific Northwest of North America. Other studies achieving higher resolution results focus on smaller study areas, less terrain relief, and/or have much less tree canopy and lower story vegetation overall.

4. Experiment

Following the data processing and DEM generation, the aforementioned experiments were conducted. This section will discuss the results of each experiment. Figure 3 shows the locations of all data surveyed on the ground for this study. The link between these datasets and the experiments is shown in the flowchart in Figure 3. These data provided the basis for the following experiments.

4.1. Experiment #1: Individual Flight Path and Swath Calibration

4.1.1. Lever Arm Comparison

The SCLA values computed per flight were compared to the MCLA (

Table 3. Differences between software-computed lever arms (SCLA) for individual flights and manufacturer-calibrated lever arms (MCLA).

Statistic	ΔX (m)	ΔY (m)	ΔZ (m)
Min	−0.001	−0.027	0.042
Max	0.016	0.007	0.158
Average	0.006	0.000	0.056
Std. Dev.	0.006	0.008	0.030

). However, when comparing the computed lever arms for the flights collected at this site, a bias of +5.6 cm with a standard deviation of 3.0 cm in Z for the computed lever arm (ranging from 4.2 to 15.8 cm) was detected. Note that much smaller differences were generally observed in the lever arm computation in X and Y between the SCLA and MCLA values. In comparison, the MCLA values agreed within 1 cm to validation measurements of the offset performed by the research team.

4.1.2. Swath Adjustment Comparison

To further evaluate the consistency between individual flights and swaths, a simplified swath adjustment constrained to translation in the Z axis only based on a local point to plane variant of ICP was implemented. Adjustments were completed for both individual flights (IFs) and swaths (ISs) for both the manufacturer (MCLA) and software-calibrated (SCLA) lever arms. The offsets determined through the swath adjustment process for the SCLA presented in

Table 4. Statistical analysis of constrained Z translations in an adjustment between flight paths and swaths comparing processed swaths using a constant MCLA.

	ΔZ (m)
	MCLA		SCLA
Statistic	Ind. Flight	Ind. Swaths	Ind. Flight	Ind. Swaths
Min	−0.012	−0.028	−0.024	−0.032
Max	0.014	0.029	0.098	0.125
Mean	0.000	0.000	0.000	0.003
Std Dev.	0.008	0.011	0.031	0.026
RMS	0.008	0.011	0.030	0.026
95% Conf.	0.015	0.022	0.058	0.051
# Swaths	14	81	14	86

are very similar to the observed differences in the lever arms determined in Table 3. Hence, it is clear that the software calibration process was actually introducing additional error between swaths rather than reducing it. The MCLA overall provided much more consistency between swaths (Table 4) for both the individual flight and individual swath adjustments. Note that individual swaths have more flexibility to adjust given that there are more, smaller segments.

For testing purposes, additional adjustments were completed based on individual swath segments, allowing for translation in X, Y, and Z, and/or rotation in X, Y, and Z; however, given the limited GCTs available for constraints, it was unclear if the translations and rotations computed for each swath segment with the additional degrees of freedom in the transformation introduced more drift or actually improved the results.

Ultimately, based on these analyses, the MCLA solutions were determined to be substantially more consistent with one another between flight paths and swaths. In one case, use of the SCLA lowered an individual swath by approximately 12.5 cm. Hence, for the final models and remainder of the analysis, the results are based on the data processed with the MCLA only.

Consistency between swaths was also evaluated qualitatively (Figure 6) through cross sections, where each swath segment was given a different color. Measured offsets of 3 to 7 cm between swaths were observed in the cross sections, but these were not systematic. These differences correlate well with the results of the quantitative accuracy assessment in experiment #2.

4.2. Experiment #2: Quantitative Accuracy Assessment

The accuracy assessment was conducted through three different approaches. First, a series of GCTs were placed across the site. Second, two total station topographic surveys were completed in different locations to determine overall biases in the data and evaluate the accuracy and precision of the data. Note that normally one would remove the overall bias by translating the data based on a GCT or set of GCTs. For the purposes of evaluating the ULS direct georeferencing solution quality, we do not apply this bias correction. These comparisons were made both to the point cloud (ground points only) and the DEM. Lastly, qualitative visual assessments of cross sections combined with iterative closest point (ICP) quantitative comparisons were made to TLS point clouds captured at strategic locations throughout the study area.

4.2.1. Ground Check Target Comparison I

A total of 36 1 m × 1 m (Figure 7a) and 0.3 m × 0.3 m photogrammetric black and white targets were strategically placed at locations across the site (Figure 3) in accessible areas with the goal of serving as GCTs for 3D accuracy assessments. These targets were placed flush with the ground surface to avoid a bias in height, and a survey nail was driven at the center to provide a common reference point for the GNSS and total station surveys. Coordinates for the center of each target were obtained with 10-min static GNSS occupations post-processed against the GNSS observations at the base station. Some targets were measured with the total station, which were combined with the GNSS baselines in a least squares adjustment in Leica Infinity software for improved coordinates. Unfortunately, most of the study area had poor GNSS visibility conditions due to tall trees and other obstructions. Further, given the site characteristics, most of these targets could only be placed on sloping road shoulders consisting of gravel or grassy areas. In some cases, the targets were placed over frozen grass in the morning that likely shifted in position as temperatures warmed throughout the day. Regardless of these issues, the targets themselves could not clearly be distinguished (Figure 7) within a single swath with the ULS and only to a limited degree within the merged dataset given the flight height of 100 m. As a result, these targets were determined to be unreliable for any adjustment of the data and only a vertical accuracy assessment could be conducted using the GCTs.

Statistical comparisons to these targets were made using all acquired GCTs, only those that had total station observations, and were used in the least squares adjustment (for improved relative accuracy), for gravel points only, and high confidence gravel points, which exclude stations where GNSS observations were of poor quality. Note that the bias and other differences are larger for the point cloud comparisons in gravel points excluding the poor GNSS points. This occurs because of the scatter in the point cloud data that was minimized with the median filtering in the terrain model creation process. Further, the large errors observed when comparing to all GCTs are likely more the result of the GNSS than the ULS data. Several GCTs were found to have poor GNSS conditions and high errors clustered together at the north end of the site (

Table 5. Statistical vertical accuracy assessment of ULS data to GNSS GCTs based on (a) the bare-earth DEM and (b) point cloud comparisons. Units: meters.

a. Model Comparison		All Points	Adjusted Least Squares	Gravel Points	High Confidence Gravel Points
	Ave	0.0616	0.0251	0.0502	0.0031
	Std Dev	0.0766	0.0417	0.0852	0.0478
	Min	−0.0925	–0.0473	–0.0925	–0.0925
	Max	0.2231	0.0847	0.2231	0.0847
	RMS	0.0974	0.0464	0.0974	0.0464
	95%conf	0.1909	0.0910	0.1909	0.0910
	Count	36	8	25	17
b. Point Comparison		All Points	Adjusted Least Squares	Gravel Points	High Confidence Gravel Points
	Ave	0.0709	0.0356	0.0734	0.0305
	Std Dev	0.0613	0.0197	0.0669	0.0165
	Min	0.0040	0.0040	0.0040	0.0040
	Max	0.1990	0.0640	0.1900	0.0660
	RMS	0.0931	0.0401	0.0984	0.0345
	95%conf	0.1825	0.0786	0.1929	0.0676
	Count	36	8	25	17

).

4.2.2. Total Station Ground Check Point Comparison II

The next evaluation compared the ground model as well as the point cloud to total station observations. Two total station setups (Figure 8) were completed (day 1 and day 2). Statistical results (

Table 6. Statistical analysis of total station observations based on different categories of observations for day 1, day 2, and combined for the digital elevation model. Units: meters.

	Model Comparison
a. Day 1		All Points	Road Only	Road Reflectorless	Road Prism	Grass Only	Gravel Only
	Ave	0.0675	0.0381	0.0388	0.0355	0.1139	−
	Std Dev	0.0629	0.0144	0.0143	0.0145	0.0803	−
	Min	−0.0977	−0.0057	−0.0057	0.0063	−0.0977	−
	Max	0.3323	0.0766	0.0750	0.0766	0.3323	−
	RMS	0.0922	0.0407	0.0414	0.0383	0.1391	−
	95%conf	0.1808	0.0798	0.0811	0.0751	0.2727
	Count	235	144	112	32	90	0
b. Day 2	Ave	−0.0179	−0.0167	−0.0172	−0.0186	−	−0.0205
	Std Dev	0.0193	0.0198	0.0198	0.0188	−	0.0150
	Min	−0.0793	−0.0793	−0.0793	−0.0600	−	−0.0441
	Max	0.0624	0.0624	0.0413	0.0624	−	0.0190
	RMS	0.0263	0.0259	0.0262	0.0264	−	0.0253
	95%conf	0.0515	0.0507	0.0514	0.0516	−	0.0496
	Count	250	202	133	117	0	44
c. Combined	Ave	0.0235	0.0061	0.0084	−0.0070	0.1139	−0.0205
	Std Dev	0.0627	0.0323	0.0330	0.0286	0.0803	0.0150
	Min	−0.0977	−0.0793	−0.0793	−0.0600	−0.0977	−0.0441
	Max	0.3323	0.0766	0.0750	0.0766	0.3323	0.0190
	RMS	0.0669	0.0329	0.0340	0.0293	0.1391	0.0253
	95%conf	0.1312	0.0644	0.0666	0.0575	0.2727	0.0496
	Count	485	346	245	149	90	44

and

Table 7. Statistical analysis of total station observations based on different categories of observations for (a) day 1, (b) day 2, and (c) combined for the point cloud (ground classified only). Units: meters.

	Point Comparison
a. Day 1		All Points	Road Only	Road Reflectorless	Road Prism	Grass Only	Gravel Only
	Ave	0.0488	0.0298	0.0330	0.0185	0.0784	−
	Std Dev	0.0439	0.0192	0.0181	0.0191	0.0544	−
	Min	−0.0229	−0.0229	−0.0090	−0.0229	0.0043	−
	Max	0.3422	0.0693	0.0693	0.0515	0.3422	−
	RMS	0.0656	0.0354	0.0376	0.0264	0.0952	−
	95%conf	0.1286	0.0694	0.0737	0.0517	0.1867	−
	Count	235	144	112	32	90	0
b. Day 2	Ave	−0.0221	−0.0244	−0.0258	−0.0180	−	−0.0152
	Std Dev	0.0210	0.0200	0.0200	0.0215	−	0.0197
	Min	−0.0763	−0.0763	−0.0763	−0.0743	−	−0.0554
	Max	0.0558	0.0447	0.0558	0.0447	−	0.0273
	RMS	0.0305	0.0316	0.0325	0.0280	−	0.0247
	95%conf	0.0598	0.0619	0.0638	0.0549	−	0.0484
	Count	250	202	133	117	0	44
c. Combined	Ave	0.0122	−0.0019	0.0011	−0.0102	0.0784	−0.0152
	Std Dev	0.0492	0.0332	0.0350	0.0258	0.0544	0.0197
	Min	−0.0763	−0.0763	−0.0763	−0.0743	0.0043	−0.0554
	Max	0.3422	0.0693	0.0693	0.0515	0.3422	0.0273
	RMS	0.0506	0.0332	0.0349	0.0277	0.0952	0.0247
	95%conf	0.0992	0.0651	0.0685	0.0542	0.1867	0.0484
	Count	485	346	245	149	90	44

) are presented for (a) day 1 only, (b) day 2 only), and (c) both days combined. Day 1 contains a mix of land cover types (e.g., road, grass, and gravel) while day 2 only contains road and gravel measurements acquired along the roadway and shoulder. A flat shoe was used on the pole for prism observations in the grass area to avoid penetration of the pole tip into the soil. Day 1 contains a mix of topographic shots and profiles while day 2 consists of four profiles along the roadway. The total station measurements were obtained both using a Lecia 360-degree prism as well as reflectorlessly across the road surface. To ensure consistency between the prism measurements and reflectorless measurements, statistics were computed for each. Both show similar bias levels within a few mm of one another for each setup.

Note that for road only points day 1 shows a bias of +0.0381 m (i.e., model is higher) compared to a bias of −0.0167 m for day 2. These biases result from GNSS errors at the GCTs used for the resection and adjustments of the total station data. They are within typical ranges expected for 10-min static occupations with substantial canopy nearby. A larger bias of +0.1139 m is observed in the grassy areas, indicating that the laser did not fully penetrate the grass and/or the ground filter did not adequately remove grass points. Results between the model and point comparisons are generally consistent except for in the grassy areas, where the point cloud results are better (likely a result of the fact that the point offsets are computed based on nearest neighbors, so it is biased towards the lower points in the point cloud).

4.2.3. Comparison to TLS

Two strategies were employed to compare the ULS point cloud to the TLS point cloud. First, cross sections were extracted and visually examined. Second, an iterative closest point algorithm was used to determine global offsets between the ULS and TLS data.

Cross Section Evaluation

We first compared the geo-referenced TLS and ULS point clouds qualitatively by visualizing the horizontal and vertical cross sections throughout the study area. Representative cross sections are shown in Figure 9. A building façade was used for creating a horizontal cross section (Figure 9a) to evaluate the horizontal quality and accuracy of the ULS point clouds. Although the ULS point clouds are significantly sparser than TLS data, the ULS tends to provide better coverage on the building façade comparing against typical ALS data given the lower flight altitude and more flexibility in flight planning. In addition, the ULS points are distributed on both sides of the TLS point cloud, which indicates the ULS data has high uncertainty (scatter) but limited bias horizontally. The vertical cross section (Figure 9b) was extracted on the shoulder of the highway to avoid noise resulting from moving vehicles. The vertical cross section behaves similarly with the horizontal profile in terms of point density, uncertainty, and bias.

ICP Evaluation

To further compare ULS and TLS point clouds quantitatively, we ran an iterative closest point (ICP)-based cloud-to-cloud registration between the ULS and TLS point clouds in Leica Cyclone software. The transformation parameters, including translation and rotation, provide a quantitative indication of the overall accuracy (bias) of the ULS data compared with the TLS point clouds, while the resulting RMS differences provides a quantitative indication of precision (e.g., standard deviation)

For the ICP algorithm, there are three main parameters (

Table 8. Parameters used in the ICP registration to compare the ULS and TLS point clouds.

Parameter	Value
Max Search Distance	0.200 m
Subsampling Percentage	1%
Subsampling Spacing	0.05 m

). The “max search distance” provides the maximum search range to detect the potential match points between TLS and ULS point clouds. We used 0.2 m based on the measurements made in the horizontal and vertical cross sections to ensure the algorithm can find the matching points effectively to solve for the transformation parameters iteratively. In addition, since the searching process also considers the normal vector at each point, the “subsampling percentage” was set at 1% to help improve the robustness in normal estimation by avoiding normal computed from points too close together. To cope with the variable point density for both ULS and TLS point cloud, a “subsampling spacing” of 0.05 m was used in the process. This subsampling also significantly reduces the processing time.

In the ICP algorithm, the point clouds can be constrained to be leveled or unleveled. In this case, we held the TLS scans as leveled given the high quality level compensators employed on the scanners while ULS point cloud were tested under both scenarios ((

Table 9. Results of quantitative comparison between ULS and TLS point clouds.

Comparison	Translation (m)			Rotation (°)			RMS (m)	Overlap (pts)
	Tx	Ty	Tz	Roll	Pitch	Yaw
ULS (leveled)	−0.024	0.011	−0.027	0.0000	0.0000	−0.0012	0.0759	85,900
ULS (unleveled)	−0.027	0.012	−0.029	−0.0019	0.0011	−0.0006	0.0753	93,000

). The translation in each dimension is less than 0.03 m, which indicates a limited bias in the ULS point cloud. These values are also consistent with the comparisons to the total station measurement and GCTs. The rotation parameters of the leveled and unleveled ULS point clouds are negligible, thus the ULS point cloud does not appear to have significant errors in orientation. The computed RMS (< 0.08 m) agrees relatively well with the accuracy assessments completed in the previous sections in comparison with the total station and GNSS measurements.

4.3. Experiment #3: Lidar Point Density Assessment

The average ground point density of this dataset is 38.59 points/m², with a nominal point spacing of 0.16 m. Figure 10 shows the spatial pattern of point density within the project site. Clear patterns emerge upon inspection. The three areas of highest point density coincide with the three take-off locations of the ULS. This makes sense due to the fact numerous swaths would pass over these areas upon take-off and landing, resulting in higher density. Additionally, like ALS, ULS with a near infrared (905 nm) sensor has difficulty capturing areas of water; in this case, a stream that passes through in the southern portion of the AOI. NIR systems cannot penetrate the water like their green laser counterparts (typically used in topobathymetric lidar applications), and a mix of lower power compared with ALS lasers and water characteristics caused zero pulse returns (data voids) over this stream. Laser pulses also cannot “see through” buildings or dense vegetation, further adding to areas of zero ground returns, although this is also a limitation of lidar technology itself, and not unique to the ULS platform. However, a less obvious pattern is the generally low density of points on roads throughout the AOI. This is partially due to the fact that the ULS was not flown directly over the road for safety reasons [18], resulting in longer ranges and only off-nadir returns on a black asphalt surface, which is not conducive to laser returns. This also results from a relatively high flying height (100 m) where it can be difficult to capture darker surfaces. One of the main reasons why the ULS obtained such comparatively high point density compared to ALS is due to the much lower AGL with which the project was flown. That being said, it is important to note that the USGS base specifications mentioned earlier relate to all first returns to compute nominal point density, while the point density of the ULS discussed in this paper relate only to ground returns, meaning that the overall point density for this site would actually be significantly higher when considering the first returns. Given that our focus was to produce a bare earth model, we felt that the ground returns is a more meaningful metric to describe the point density for this application.

Figure 11 presents a cumulative probability plot of the project area’s point density along with the summary statistics. The high standard deviation can be explained by the large disparity in the ground point density between take-off sites and the remainder of the project area. This results in an incredibly irregular spread of points throughout the site; although this spread is to be expected with ULS data because of constraints in the flight time and the limited locations used for take-off sites. For this project’s acquisition, the size and geography of the site restricted where the ULS could take-off, and limited battery supply also contributed to numerous swaths crossing over the same area, thus creating a disproportionately high point count around the take-off sites. As far as overall coverage is concerned, however, the ULS performed well, as seen in the computed median (11 ground points/m²) and the cumulative probability graph. This graph shows that when disregarding grid cells with zero ground returns, the ULS successfully mapped 50% of all remaining grid cells with at least 17 ground points/m². Ground density was considered this way because, as mentioned earlier, it is quite common to have numerous areas of zero ground returns when buildings, water, and dense vegetation are present, all three of which can be found at this site.

4.4. Experiment #4: Change Analysis

This section explores the capabilities of the ULS data to be compared to ALS datasets for change analysis. The change analysis (Figure 12) of the ULS lidar DEM was performed with respect to an ALS dataset collected in 2009. The ALS data was ground filtered and processed into a bare earth DEM with a 1.0 m cell size using similar techniques as used to process the ULS data (although a bare earth DEM was already available through the Oregon Lidar Consortium, that DEM was created for a much larger geographic area. Hence, we reprocessed the data to improve the amount of ground points available across the site).

The change analysis yielded several important observations regarding the landslide. First, a graben is apparent from the change detection results and represented as a red section (subsidence) traversing the road. A graben is a block that subsides at the head of a landslide, wedged between the landslide body and headscarp. Thus, the vertical settlement is notable at the head of the landslide. The road appears as accretion (blue) owing to repaving and reconstruction at the onset of slope failure. Second, throughout the landslide body, a region of accretion (i.e., heave/advance) is observed, represented in blue. The observed accretion from change detection occurs as the generally translational movements of the landslide body, resulting in an apparent increase in the elevation of the downslope embankment. Third, most of the displacement is occurring at the northwest portion of the slide, likely due to increased groundwater levels from a recently installed pond at the bench west of the headscarp. Lastly, the DEM highlights numerous similar failures within the overall landslide complex. The current failure is likely the next calving block from a loss of lateral support from adjacent failures to the northwest as well as a relatively high ground water table.

Inclinometers and piezometers (Figure 12 and Figure 13) were installed on site by ODOT during the mitigation of the slide and provide additional context/interpretation of the landslide mechanics. The elevation from the ULS DEM were used as a reference from which depths on the inclinometer readings were measured to determine the water elevations and depth of shear. Using the inclinometer data, several failure mechanisms were characterized across the slide. Buckling is readily apparent in the graben at the hangwall, while rotation is observed at the base of the graben. The graben also shows significant depression and rotation. A toppling mechanism is observed to the southeast end of the graben. When evaluating a cross section (Figure 12) through the landslide body, an overall translational failure mechanism of the slide is readily apparent. The rupture surface of the slide daylights at approximately 81 to 85 m in elevation, suggesting a maximum depth of approximately 25 m. The groundwater also follows the elevation reasonably well. The observations from change detection support those from in-situ monitoring, suggesting that repeat collection of lidar can be a valuable supplement to traditional landslide monitoring techniques. It highlights that the poor spatial resolution of in-situ monitoring may be supplemented by change analysis using ULS lidar. Likewise, the in-situ monitoring can provide high temporal resolution, which is not usually cost effective to achieve with ULS lidar.

5. Conclusions and Lessons Learned

Several valuable lessons were learned in this research that could facilitate future ULS surveys. One major lesson involves the use of the SCLA, which did not work well in this area of limited well-defined features and artificially shifted the data in elevation by approximately 5 cm. The MCLA values were ultimately chosen due to significantly better agreement of the reprocessed ULS data compared with the validation survey data. Although the lever arm values from the manufacturer provide the best results in this case, this may not be the case in urban settings with well-defined objects. Therefore, it is sufficient to say that caution should be exercised when performing an SCLA given the inherent noise level of the data, which can provide unsatisfactory results.

Additionally, it is important to note that ULS datasets may not be able to undergo the same processing procedures as ALS datasets. Specifically, typical ALS swath adjustment procedures may be difficult to implement on ULS swaths given their shorter length and the limited well-defined geometric features captured within the said swaths. Further research is necessary to adapt and improve the overall swath adjustment workflow for ULS data.

For this study, we utilized manufacturer recommendations for operating procedures as a baseline to determine the capabilities and limitations of ULS for landslide assessment. Future work should evaluate and perhaps modify these procedures with more rigor given the relative infancy of this technology and potential for optimization. Another limitation of the study was that the flight height was at the maximum of the manufacturer’s recommend value range, which resulted in minimal returns on the dark pavement surface from signal degradation. While the flying height of 100 m was chosen in order to maximize the coverage given the limited time window for collection and large extent of the study area, future surveys can be optimized by including a specific swath at a lower height near the roadway to improve the data quality in this area.

Overall accuracy assessments revealed expected results based on system specifications and appear to agree well with other literature. The accuracy assessment using the GNSS GCTs supports the abilities of ULS. With a vertical accuracy of about 9 cm at 95% confidence (compared to high confidence gravel points), the ULS data agrees well with the GCTs. This value is slightly higher than that of the 95% confidence value seen in the total station comparison, but this is expected given that the surveyed area included many areas of tall, dense tree canopy, which undoubtedly disturbed the GNSS measurements. In fact, the site contained limited locations with sufficient sky visibility. This is often an issue in GNSS data collection, whereas ULS surveys have the benefit of flying above tree canopies with improved PDOP as well as supporting measurements with the INS that can improve the trajectory. Lastly, care is necessary when formulating a flight strategy for ULS surveys with GCTs. Flight altitudes that are too high and/or flight speed speeds that are too fast can render photogrammetric targets useless, as insufficient point density can make these targets nearly inscrutable.

When comparing ULS to the total station control points, we found that they agreed reasonably well on flat, open surfaces. This was seen in the approximately 5 to 7 cm vertical accuracy of the ULS data at 95% confidence; however, this number jumps to nearly 20 cm at 95% confidence in grassy areas. Such a high value indicates that either more robust ground-filtering techniques could improve the ULS ground model, and/or simply that ULS technology struggles to penetrate thick grass sufficiently enough to accurately model ground surfaces in such areas. Because the total station check points were acquired in the grassy areas using a prism atop a pole, it was expected that these points would sit lower than the ULS ground model due to the prism pole’s ability to reach the ground beneath the grass. However, since lidar technology has difficulty with thick vegetation in general, the total station control points still show that ULS is able to produce highly accurate ground models.

The assessment between ULS and TLS data provided additional insights into each technology’s capabilities. Both technologies were proven to collect highly accurate data, although the TLS coupled with a GNSS receiver tends to provide results of higher accuracy and precision, especially when considering the amount of noise seen in the ULS data. Ground and building profiles were more clearly defined in the TLS data while ULS data had a level of noise that can potentially prove problematic depending on the ground filtering used. We found that some ground-filtering approaches tended to be biased towards the lower points in this noise, meaning the ground model was slightly shifted down, producing an overall lower vertical accuracy. The minimal noise in the TLS data would therefore be less of a problem for such ground-filtering techniques. While the density and accuracy may not quite be on par with TLS, ULS has the ability to cover a significantly larger area (few square kilometers) with higher efficiency compared to TLS data collection.

Finally, the ULS DEM was compared to an ALS to analyze change across the site. The observed changes in the DEMs correlated well with observations of landslide kinematics from the data obtained with in-situ inclinometers. Hence, it is a viable tool to augment the relatively sparse measurements from in-situ monitoring sensors.