Comparison of UAS-Based Structure-from-Motion and LiDAR for Structural Characterization of Short Broadacre Crops

Abstract

The use of small unmanned aerial system (UAS)-based structure-from-motion (SfM; photogrammetry) and LiDAR point clouds has been widely discussed in the remote sensing community. Here, we compared multiple aspects of the SfM and the LiDAR point clouds, collected concurrently in five UAS flights experimental fields of a short crop (snap bean), in order to explore how well the SfM approach performs compared with LiDAR for crop phenotyping. The main methods include calculating the cloud-to-mesh distance (C2M) maps between the preprocessed point clouds, as well as computing a multiscale model-to-model cloud comparison (M3C2) distance maps between the derived digital elevation models (DEMs) and crop height models (CHMs). We also evaluated the crop height and the row width from the CHMs and compared them with field measurements for one of the data sets. Both SfM and LiDAR point clouds achieved an average RMSE of ~0.02 m for crop height and an average RMSE of ~0.05 m for row width. The qualitative and quantitative analyses provided proof that the SfM approach is comparable to LiDAR under the same UAS flight settings. However, its altimetric accuracy largely relied on the number and distribution of the ground control points.

1. Introduction

Unmanned aerial systems/vehicles (UASs/UAVs), paired with structure-from-motion (SfM) image processing workflows, have lately emerged as a popular strategy for various geoscience applications [1,2]. By collecting overlapping image sequences using a high-resolution camera mounted on a UAS and then inputting those images to an SfM algorithm, users effectively can create a three-dimensional (3D) point cloud of a scanned field. Particularly in precision agriculture, the SfM point cloud approach can be used to derive structural parameters of crops, such as plant height [3,4], canopy volume [5,6], and leaf area coverage [7,8], all of which could significantly help farmers to enhance agricultural management decisions [9].

Another widely used approach to generating 3D point clouds is light detection and ranging (LiDAR). LiDAR works by actively emitting high-frequency laser pulses toward the object and recording the reflected responses and transmitted time, from which range measurements can be computed [10]. Over the past decade, LiDAR successfully has been applied in forest inventory as either airborne laser scanners (ALS) [11,12,13] or terrestrial laser scanners (TLS) [14,15,16]. However, as UAS-LiDAR platforms have become more widely used, this approach has proven to be effective and accurate for the structural characterization of shorter vegetation/crops [17,18,19,20]. LiDAR point clouds have obvious advantages when compared with SfM point clouds, including high point density, robustness to illumination changes, and the ability to obtain below-canopy information, because of its multiple return capability from single pulses. However, LiDAR has its own disadvantages: The cost of LiDAR is usually much higher than that of a typical high-resolution color camera used for SfM [6,21]. While SfM algorithms have successfully been integrated into commercial software such as Pix4D, Agisoft Photoscan, or open-source software including OpenDroneMap, the preprocessing of a LiDAR point cloud is relatively more time-consuming and requires more effort, such as trajectory calculation, denoising, point cloud registration, and ground filtering [22,23].

Many studies used LiDAR as a baseline method to evaluate the relative accuracy of SfM point clouds. Some early studies have compared UAS-based SfM point clouds with data collected by TLS [3,4,18,24] and ALS [25,26,27,28]. For example, Holman et al. [3] compared UAS-SfM-modeled wheat plant height with those derived from TLS and field measurement of crop height (CH). Both SfM-derived and TLS-derived models achieve a root mean squared error (RMSE) of 0.03 m. Li et al. [26] used ALS and UAS-SfM point clouds to derive structural metrics of a maize and then used metrics to evaluate the leaf area index (LAI). The results showed that ALS achieve higher accuracy than SfM in evaluating the LAI, with $R_{adj}^2$ = 0.85 and $rRMSE=7.16\%$ versus $R_{adj}^2$ = 0.74 and $rRMSE=7.72\%$. Such comparisons with TLS and ALS have proved the effectiveness and accuracy of UAS-SfM systems, while highlighting its limitations. However, another important question has yet to be answered: Is a UAS-SfM point cloud as accurate/precise as a LiDAR point cloud under the same flight settings, i.e., when an imaging camera and a LiDAR are installed on the same UAS?

To the best of our knowledge, few studies directly compare UAS-LiDAR point clouds with UAS-SfM point clouds, especially in precision agriculture applications. Cao et al. [29] compared UAS-LiDAR and SfM data in a subtropical coastal planted forest of East China. They found strong correlations (r > 0.9) between metrics such as height percentiles and canopy cover, derived from the two modalities’ point clouds. The two kinds of data, however, were collected on different days and at different flight altitudes (80 m for LiDAR and 500 m for SfM). Lin et al. [30] used a customized UAS-based mobile mapping system to simultaneously collect LiDAR and imagery data over coastal environments. Their results suggested that UAS LiDAR and SfM point clouds are comparable within a 5–10 cm range, and both provided high-resolution and high-quality topographic data. In the context of precision agriculture, Sofonia et al. [31] employed a side-by-side comparison of a Hovermap LiDAR and a RedEdge multispectral camera, mounted on one UAS platform, for monitoring sugarcane growth response. They found that both systems demonstrated similar capabilities for accurate CH measurements ($R_{adj}^2=0.85–0.95$), while LiDAR provided relatively more consistent and significant correlations. Shendryk et al. [32] predicted biomass in sugarcane using a UAS-based LiDAR and multispectral imagery system. The authors found a similarity between the results of biomass prediction (${\overline{R}}^2<0.57).$ However, both modalities provided a limited perspective in terms of preliminary comparisons of the point clouds and focused more on the end-level comparisons.

This study focused on the above question and is based on a comparison of UAS-LiDAR and UAS-SfM point clouds, collected at five different growth stages of a snap bean field. Moreover, while most of the previous studies involved relatively high-stature crops, such as rice [33,34], barley [35,36], wheat [3,37], sugarcane [31,32], maize [26,38], and sorghum [39,40], which can grow to be taller than 1–3 m when mature, snap bean represents a short crop that only grows up to 0.6 m at most. It thus can be stated that snap bean results in more stringent requirements when attempting to assess limited vertical-range structural characteristics.

Our study was executed as follows: (i) First, we preprocessed and geospatially registered LiDAR and SfM point clouds; we estimated the primitive absolute accuracies of the checkpoints in both point clouds by utilizing the recordings of ground control points (GCPs); (ii) we then directly compared the two kinds of point clouds in terms of a point density map, histograms of z coordinates, and SfM point cloud to LiDAR-derived mesh (cloud-to-mesh (C2M)) distance maps; (iii) after implementing rasterization and interpolation on the point clouds, we next compared their derived products, including digital elevation models (DEMs) and crop height models (CHMs), by calculating the difference of the models and the multiscale model-to-model cloud comparison (M3C2) distance; (iv) as a complement to the above comparisons, we made qualitative comparisons between the cross-sections of the two point clouds; (v) finally, we compared the CH and row width (RW) of sampled rows, derived from both point clouds, with in-situ measurements in one of the data sets. Based on all the analyses above, we hope to provide a definitive comparison of UAS-based SfM vs. LiDAR point clouds in the context of the structural characterization of short broadacre crops, with snap bean being our proxy crop.

2. Materials and Methods

In this section, we first introduced the study site and the collected data (Section 2.1). We then explained the data processing steps (Section 2.2), followed by details on the methods and evaluation metrics that were applied to compare the data derived from the LiDAR and SfM approaches (Section 2.3).

2.1. Study Site and Data Collection

Our study site is located in Geneva, NY, USA (42°52′0.00″ N, 77° 1′43.00″ W; Figure 1a). The field consisted of two plots that were 100 m apart in the north–south direction. The north plot had three replications of six snap bean cultivars: Venture, Huntington, Colter, Cabot, Flavor Sweet, and Blevet. Each cultivar in each replication had four rows, resulting in 72 rows in total. The south plot contained only the Huntington cultivar, with 40 row segments treated as experimental plants and half of them inoculated with white mold after the first flowering stage. The sizes of the two plots were 1512 and 1416 m², respectively (Figure 1a).

The UAS system used a DJI Matrice 600 Pro hexacopter as a base platform. It consisted of a global navigation satellite system (GNSS)/inertial measurement unit (IMU; Trimble APX-15 UAS V2), a Velodyne VLP-16 Puck^TM LiDAR (Velodyne, San Jose, CA, USA), and a MicaSense RedEdge^TM multispectral camera (MicaSense, Seattle, WA, USA). The GNSS/IMU unit provided recordings of the geolocation and the GPS time during flights. The VLP-16 LiDAR generated up to ~600,000 points/second in a dual return mode. It operated in a 360° horizontal field of view and a ±15° vertical field of view. The claimed range accuracy was ±3 cm, and the laser wavelength was 903 nm. The MicaSense RedEdge captured images of five spectral bands: blue, green, red, red edge, and near-infrared, centered at 475, 560, 668, 717, and 842 nm, respectively. We implemented five flights across five days to collect data on the snap bean crop at different growth stages. Figure 1b shows an example of the flight trajectory, along with a LiDAR point cloud. The flight information and the data set specifications are listed in

Table 1. Unmanned aerial system (UAS)-light detection and ranging (LiDAR) and multispectral imagery data sets’ specifications.

Data Set Number	1	2	3	4	5
Date	1 July 2020	28 July 2020	6 August 2020	14 August 2020	24 August 2020
Flight altitude (m)	25	25	22	25	25
Flight speed (m/s)	3	2	2	2	2
Flight line space (m)	6.6	5	5	5	5
Ground sample distance (m/pixel)	0.017	0.017	0.015	0.017	0.017
Snap bean growth stage	Bare ground	Budding	Eight days before full blooming	Full blooming; 10 days ahead of harvest	Ready for harvesting
Number of images for structure-from-motion (SfM)	671	590	566	606	617

. For data set #3, we took ground measurements of the CH and the RW by sampling three times per row in the northern plot and two times per row segment in the southern plot by using a measuring tape.

2.2. Processing of the UAS-Based 3D Point Cloud Data

The processing of the SfM and LiDAR point clouds mainly included three steps: (1) preparing the data for comparison (data preprocessing); (2) deriving gridded models from the point clouds, including DEMs, digital surface models (DSMs), and CHMs; and (3) extracting segments of the vegetation points, calculating the CH and the RW and comparing them with ground measurements.

2.2.1. SfM and LiDAR Point Cloud Preprocessing

We input the multispectral images to a Pix4Dmapper (v.4.6.4, Pix4D S.A., Prilly, Switzerland) to generate SfM point clouds. While we mostly used it as an end-to-end tool, the basic principles of the SfM algorithm have been well-explained in previous studies [1,31,41,42]. First, the algorithm computes the keypoints and descriptors for all the images toward automatic image matching. In this step, at least 75% cross-track and along-track overlapping is recommended for the Micasense RedEdge camera [43] to ensure enough common features are identified. The algorithm next applies a bundle adjustment and a 3D scene reconstruction to generate a sparse point cloud, which then is densified to create a dense point cloud. The GCPs next are imported to georeference the point cloud. In our study, we used AeroPoints^TM (Propeller, NSW, Australia) checkerboards as GCPs and evenly distributed them in the field during the flights. With a proper set-up, the global accuracy of the GCPs is claimed to be 10 mm + 1 ppm horizontally and 20 mm + 1 ppm vertically [44].

The SfM point cloud preprocessing was quite direct—we visualized the SfM point cloud in the software CloudCompare (v.2.12 alpha [45]) and cropped it spatially to valid scene extents by the manual image interpretation of the field color patterns. However, it is worth noting that in our study, the default georeferencing system (NAD 83/NAVD 88) for the AeroPoints was different from that of the APX-15 (WGS 84/EGM 96), which was used in generating coordinates for the LiDAR point clouds, while the essence of the z coordinates also differed. In AeroPoints, it implies the orthometric height (height above geoid), while in APX-15, it refers to the ellipsoidal height (height above ellipsoid). For example, the difference between the ellipsoidal and the orthometric height was −35.00 m at the location of our study site in the WGS 84/EGM 96 geographic coordinate system. We therefore converted the recording of the GCPs’ coordinates using the software VDatum (v.4.0.1, NOAA, Washington, DC, USA [46]) to geospatially register the SfM point clouds with the LiDAR point clouds.

The preprocessing of LiDAR point clouds included a few more steps: (1) retrieving the flight trajectory from the IMU file, which focused on the transformation of geographical coordinate and time units; (2) temporal and spatial cropping according to field boundaries and flight trajectories; (3) removing outliers and noise using a statistical outlier removal (SOR) filter; (4) removing second return points and duplicate points; and (5) computing the scan angles of the points by using the IMU flight trajectory; and then filtering out the points with large scan angles (>20°), which theoretically had larger errors. Our LiDAR data preprocessing steps were similar to those by Sofonia et al. [47], except that in step 4, we did not need the second return points. This was because the VLP-16 LiDAR can only distinguish between two returns per pulse when the distance between the two returns is >1 m [48], while the heights of all the snap bean plants are <1 m. We filtered the points with too large scan angles in step 5, since larger scan angles resulted in the increased off-nadir distances of the laser beams, thus leading to significant errors, due to the footprint size of a laser beam increasing at the square of the transmitting distance. For steps 2–4, we used LAStools [49] and CloudCompare; for steps 1 and 5, we used Python 3 programming.

2.2.2. Derivation of Digital Models

As many previous UAS-based studies proposed [50,51,52], the derivation of the CHM involves deriving the DEM and the DSM, which was calculated as:

$$\mathrm{CHM} = \mathrm{DSM} - \mathrm{DEM}.$$

(1)

The DEMs were generated from the preprocessed SfM and LiDAR point clouds from data set #1, which was collected when the field contained only bare ground (Table 1). We set the grid spacing of the DEMs to 0.05 m, similar to the study of Lin and Habib [20]. We chose this value due to the following reasons: (1) it is small enough to retain plant structural details, and (2) it is large enough to ensure that most cells would include more than one point, i.e., cell values could be decided using actual data, instead of interpolation. For both the SfM and LiDAR point clouds, we first rasterized them based on point clouds with a regular spacing, by calculating the average z value in each cell. We then filled the empty cells using ordinary kriging interpolation in Surfer (v.15.3.307, Golden, CO, USA). We derived the DSMs from data sets #2–5 using rasterization and interpolation, such as the DEM generation process except for one difference—in rasterization, we used the highest z value, instead of the average value as the representative for each cell. Finally, we computed the CHMs using Equation (1).

2.2.3. Calculating CH and RW from Vegetation Sample Points

To evaluate the relative accuracy of the SfM-derived and LiDAR-derived CHMs, we calculated the CH and the RW from row segments in the CHMs from data set #3 and compared them with ground measurements. The extraction of segments of vegetation points is shown in Figure 2. We filtered out all the points with a height lower than 10 cm to ensure ground points did not impact the calculation. Within each segment, we calculated CH by finding an optimized top percentile of the z values. Similarly, we calculated the RW by finding an optimized difference between a well-above average percentile and a well-below average percentile of y values. The optimal values were determined by finding the smallest RMSE between the derived metrics and the field measurements [23]. Specifically for the north plot, since the original direction of the rows were at an angle of $6.6°$ due east (measured in CloudCompare), we manually applied a rigid transformation to rotate the point clouds to make the rows horizontal.

2.3. Comparing Methods and Evaluation Metrics

The sparsity and non-uniform point density of 3D point clouds introduce a level of complexity, which necessitates a multi-pronged approach to assessing the data quality. We thus evaluated the characteristics of the UAS-based SfM and LiDAR point clouds during different processing stages. We first made preliminary comparisons among the “raw” data, such as the accuracy of GCPs’ centers, average density, and z-histograms. We then compared the preprocessed point clouds and their derived DEMs and CHMs. After that, we observed the cross-sections (side-view) of small segment samples in the preprocessed point clouds versus the CHMs in data set #3, to serve as a complement to the previous top-view comparisons. Finally, we selected data set #3 to compare the CH and the RW, derived from the CHMs, with ground measurements.

2.3.1. Preliminary Comparisons

The distribution of the valid GCPs for each data set is shown in Figure 3. In fact, we placed nine GCPs in the field during all flights, although a couple of them failed to operate properly due to equipment aging. Similar to Lin and Habib [20], the absolute accuracy of the SfM and LiDAR point clouds was assessed against the AeroPoints recordings of the GCPs deployed at the site. We manually identified them based on the patterns, in order to determine the coordinates of the center of AeroPoints GCPs from an SfM point cloud. Due to the uncertainty of the LiDAR detection range, the determination of the GCP centers in the LiDAR point cloud required a more complex strategy. LiDAR points of a planar panel had a significant vertical variance and exhibited “multiple layers” as opposed to the “single-layer” counterpart in an SfM point cloud. We applied the approach proposed in Lin and Habib [20] to identify the centers: (1) manually selecting initial centers based on the intensity patterns; (2) finding all points that lay within a spherical neighborhood with a radius of 0.25 m and are centered at the initial points; and (3) fitting a plane among the neighboring points and then projecting the initial points to the fitted plane. The projection points on the fitted planes were finally identified as the GCP centers from the LiDAR point clouds. It should be noted that we selected 0.25 m as the radius in step (2), because this radius ensured that all the points within the neighborhood were from the checkerboard, given that an AeroPoints GCP’s dimensions are 0.544 m × 0.544 m. Finally, we compared the average surface point densities of the SfM point clouds and LiDAR point clouds in all data sets and the histograms of their z coordinates.

Finally, we also calculated the average surface point densities and the histograms of z-coordinates for each data set.

2.3.2. Derived Products Comparison

Four point cloud comparison methods are commonly used to compare 3D point clouds: DEM of difference (DoD) [53,54,55,56], cloud-to-cloud (C2C) distance [57,58,59,60], C2M distance [47,61,62,63], and M3C2 distance [64,65,66,67,68,69]. The DoD approach subtracts two DEMs or DSMs using a top view and is fast/easy to implement and interpret. However, the disadvantages are that it requires gridded representation, only estimates vertical distance and is sensitive to errors caused by misregistration [70]. In the C2C approach, two point clouds are set as the reference point cloud and the compared point cloud, respectively. For each point in the compared point cloud, the closest point can be determined in the comparison point cloud. This is the fastest and simplest 3D comparison method [64]. The measured distance can be assessed as the x/y/z directions but is sensitive to the cloud roughness, outliers, and point spacing [68]. As the SfM point cloud had a significantly lower point density than the LiDAR point cloud, it resulted in a larger point spacing for the former. The DoD and C2C methods were not suitable for a direct comparison between the two modalities, so the C2M approach was used for a robust comparison.

We used CloudCompare to calculate the C2M distance between the SfM and LiDAR point cloud-generated meshes. First, a mesh was generated using the Delaunay 2.5 D (best fitting plane) mesh tool from the LiDAR point cloud. Then, a C2M distance map was computed for the SfM point clouds, relative to the LiDAR-generated meshes. The output was saved as a scalar field in the SfM point clouds. Thus, the point density difference between the two types of point clouds did not impact the results.

The M3C2 is a relatively recent 3D comparison approach that calculates both distances and uncertainties [64]. It has advantages: (1) taking into account local normals, roughness, and registration errors, (2) allowing direct application to two un-rasterized point clouds (although it is still not ideal for the direct comparison of SfM with LiDAR, since SfM point clouds have a much lower density and even empty holes), and (3) resulting in distance calculations on the normal directions [64,68]. Therefore, we used the M3C2 to compare the derived digital models (DEMs and CHMs) from the point clouds. The algorithm includes four main steps:

• Find the “core” points, which are essentially a sub-sampled version from the original point cloud;
• For each core point, a normal vector is defined by fitting a plane to its neighbors, enclosed by a user-defined diameter $D$, which is named “normal scale”;
• Given every core point and its normal vector, a cylinder can be defined via the user-defined projection scale, $d$, and the cylinder depth, $h$, oriented along the normal direction. Thus, the intercept of the two clouds with the cylinder defines two subsets of points; and
• Project both subsets to the orientation axis of the cylinder, i.e., the normal vector, to generate two distributions of distances. The distance between the means or medians of the two distributions is the local distance, $L_{\mathrm{M}3\mathrm{C}2}$.

We used CloudCompare to implement the M3C2 in our study. Apart from the $L_{\mathrm{M}3\mathrm{C}2}$ distance map, CloudCompare also provided two more outputs: the distance uncertainty map and the change significance map. The distance uncertainty measure (at a level-of-detection of 95%, i.e., $L_{95\%}$) was calculated by:

$$L_{95\%}=\pm 1.96\ast \left(\sqrt{\frac{{\sigma }_1{\left(d\right)}^2}{n_1}+\frac{{\sigma }_2{\left(d\right)}^2}{n_2}}+r\right),$$

(2)

where ${\sigma }_1$ and ${\sigma }_2$ are the variances of point positions for each subset in step 3, $n_1$ and $n_2$ are the number of points in the subsets, and r is the co-registration error between two point clouds. These parameters are determined locally by the algorithm itself. The change significance map consists of binary values that indicate whether the distance corresponds to a real change or not [71].

As described above, three key parameters, namely the projection scale ($d$), the cylinder depth ($h)$, and the normal scale ($D$), need to be manually defined. We set $d$ as 0.15 m, because the point spacings for the DEMs and the CHMs were 0.05 m and thus at least six points would be included in each sub-cloud, which was more than the four points required by the M3C2 algorithm [65]. Considering the scale of snap bean—CH (up to ~0.65 m) and RW (up to ~0.70 m), we set $h$ = 0.80 m, so that when the normals were calculated on one side of a row, the cylinder in each local neighborhood was large enough to enclose sufficient points and small enough not to enclose unwanted points from the other side of the row. Previous research based on landslide [66], open-pit mines [65], and canyon studies [64] generally set $D$ as 20–25 times the average roughness in each point cloud. However, in our application, when $D$ varied from 0.15 to 0.30 m, the average roughness of the digital models ranged from 0.01 to 0.03 m, i.e., $D$ were 10–15 times the roughness. Based on our preliminary knowledge of snap bean crops, we considered the size of the snap beans as also limiting the value of D. More specifically, a proper $D$ should fall within 0.15–0.40 m, so that abundant points can be included to calculate the normals and to ensure that the normals reflect the genuine local features of the crops. We therefore fixed the normal scale $D$ at 0.30 m, which was twice the projection scale ($d$), a ratio that was close to the settings used in the aforementioned studies [64,65,66,68].

We used three standard accuracy measures to evaluate the differences calculated by C2M and M3C2 approaches, namely the mean difference (MD), the standard deviation of the difference (St. Dev), and the RMSE.

3. Results

3.1. Preliminary Comparison of the Point Clouds

3.1.1. Absolute Accuracy of GCPs

Figure 4 shows the box and whisker plots of the RMSEs of the differences between the 3D point cloud-derived coordinates of the GCPs centers against the AeroPoints measurements. The results included all the GCPs in the five data sets. We found that for both SfM and LiDAR point clouds, the average uncertainties (RMSE) in the x/y/z directions were about 0.01–0.03 m. Thus, both SfM and LiDAR point clouds achieved high accuracies.

3.1.2. Average Density and Histograms of the z Coordinate

The average point densities of each SfM point cloud and each LiDAR point cloud are shown in Figure 5. The SfM point density ranged between 140 and 840 pts/m², while the LiDAR point density ranged between 1400 and 2800 pts/m². It was not surprising that the point densities of both plots had a slight difference, since the data of the south plot and the north plot were collected during the same flights, with the same flight altitude, flight speed, flight line space, and even number of flight lines. This furthermore proved that, despite the difference of the snap bean variabilities in the two plots, the surface point density was mainly determined by the flight settings.

Figure 6 shows the histograms of all the point clouds. We found that SfM and LiDAR had similar distributions of bins for all data sets. The south plot histograms were more symmetric than the north plot histograms, while the latter usually had a long tail (more bins of small z-values). This was attributed to the south plot containing only one snap bean cultivar; these single-cultivar plants grew more uniformly than in the north plot, which had six cultivars. By comparing the first column with the third column, we observed that the bins of the small values in the LiDAR histograms were higher than in the SfM histograms, which indicated that more near-ground information was retained. This was likely due to the low-altitude points originating from stems near the ground and below leaves; LiDAR could actively scan them when its beam hit the stems at the right off-nadir angle, while the SfM point cloud was generated “passively” from imagery, on which the stems tended to show up as unlit/dim pixels, given that they were either in shadow or visually fully occluded.

3.2. C2M Distance Map Derived from the Preprocessed Point Clouds

The C2M distance maps were computed for both the north and south plots for all data sets (Figure 7). We found that the errors from the vegetation points were generally greater than those from the ground points. Most of the C2M maps exhibited uniformly distributed differences over the whole plot, except for the south plot on 28 July 2020 and the north plot on 24 August 2020; both dates presented relatively more significant errors in the top left corner. This was attributed to the unevenly distributed valid GCPs on these two dates (Figure 3), i.e., there was no valid GCP near those corners.

Table 2. The statistical metrics of the C2M distance maps of all the datasets.

Date	MD (m)		Standard Deviation of the Difference (St. Dev; m)		RMSE (m)
	South	North	South	North	South	North
1 July 2020	−0.01	−0.03	0.02	0.03	0.02	0.04
28 July 2020	0.01	0.01	0.04	0.03	0.04	0.03
6 August 2020	0.02	0.02	0.04	0.04	0.05	0.05
14 August 2020	0.01	0	0.03	0.03	0.03	0.03
24 August 2020	0.01	0.03	0.03	0.07	0.03	0.08

shows the statistics of the C2M maps. Most of the differences were below 0.05 m (RMSE = 0.03–0.05 m), except for the north plot in the last data set (RMSE = 0.08 m).

3.3. M3C2 Distance Map Derived from the DEMs and the CHMs

The results of the M3C2 maps are shown in Figure 8. We eliminated the outlier points with significant uncertainties (above 0.15 m) and large absolute errors (far beyond the prominent peak of the histogram). The remainder of the points are shown below.

Table 3. The statistical metrics of the M3C2 distance maps of all the datasets.

Date	MD (m)		St. Dev (m)		RMSE (m)
	South	North	South	North	South	North
1 July 2020	0.08	0.05	0.02	0.01	0.08	0.05
28 July 2020	0.12	0.10	0.13	0.06	0.18	0.12
6 August 2020	0.04	0.05	0.05	0.05	0.07	0.07
14 August 2020	0.07	0.04	0.06	0.05	0.09	0.06
24 August 2020	0.07	0.14	0.07	0.11	0.10	0.17

shows the statistical metrics of the M3C2 distance maps. By comparing Figure 8 with Figure 7, we found that the M3C2 maps better reflected the spatial distribution of the differences than the C2M maps by providing the following: (1) stronger contrast between warmer colors (taller) and colder colors (shorter); and (2) distance histograms that exhibited larger ranges. Table 3 shows that, except for the data sets from 28 July 2020 to 24 August 2020, which exhibited large errors/differences (RMSE = 0.10–0.18 m), the other three data sets showed distance differences ranging between 0.05 and 0.10 m.

3.4. Comparison of Sampled Cross-Sections in Point Clouds

We extracted two types of cross-sections by segmenting out thin slices of points across rows (Figure 9a,b,e,f) and along a row (Figure 9c,d,g,h) from a top-down view; this was performed, since the snap bean field exhibited repetitive patterns. The SfM point cloud was a “single-layer” point cloud that envelopes the snap bean plants, while the LiDAR point cloud presented “multiple layers” with much denser points in the 3D space. Although both point clouds could detect the snap bean rows and alleys (bare ground area between rows), the LiDAR point clouds retained more detail of the valleys than did the SfM point clouds. The SfM point clouds were generally higher than the LiDAR point clouds, and so were the derived CHMs. By comparing the left column (Figure 9a,c,e,g) with the right column (Figure 9b,d,f,h), we also observed that the CHM models did provide a clearer view of the geometric shape of the field.

3.5. Comparison of the Sampled CH and RW

As described in Section 2.2.3, different top percentiles of z-values resulted in different predictions of CH and thus yielded different errors when compared to ground truth measurements. The same was true for the central percentiles of y-values in terms of RW. Figure 10 shows the absolute errors of the predicted CH and RW from the SfM-CHM versus the corresponding ground measurements in data set #3, as per different percentiles. We could find the best SfM results by locating the percentile, which generated the lowest RMSE, by observing the curves of the RMSE vs. percentile. We applied the same approach to the LiDAR point clouds and obtained the statistical metrics listed in

Table 4. The best percentiles that determined the crop height and the row width, derived from the SfM-CHM and the LiDAR-CHM in data set #3 and the statistical metrics of the errors of the predicted values versus ground truth measurements.

		Best Percentile (%)		Mean (m)		St. Dev (m)		RMSE (m)
		South	North	South	North	South	North	South	North	Average
CH	SfM	86.8	91.5	0.002	−0.001	0.013	0.023	0.012	0.022	0.017
	LiDAR	98.4	99.2	−0.001	−0.003	0.014	0.03	0.013	0.029	0.021
RW	SfM	91.2	88	0.004	−0.014	0.041	0.056	0.038	0.056	0.047
	LiDAR	95.7	95.7	−0.001	−0.007	0.062	0.051	0.058	0.05	0.054

. We found that the errors for the north plot, which had a larger cultivar variety, were larger than those for the south plot. For both point clouds, the CH calculation turned out to be more accurate than the RW calculation. We found that the SfM approach matched LiDAR as modality and even performed slightly better when evaluating CH, by comparing the RMSEs from both SfM and LiDAR. The 0.1 interval on the x-axis resulted in the curves for the SfM approach “oscillating” more obviously than those from LiDAR. This was attributed to the fact that many points in the SfM CHM were generated by interpolation, while most of the points in the LiDAR CHM were generated by selecting representative samples among real points. In LiDAR-CHMs, the best top percentile of z-values for calculating CH was $98.8\%\pm 0.4\%$, and for calculating RW, it was $95.7\%\pm 0$; for the SfM-CHMs, the best central percentile of y-values for calculating CH was $89.2\%\pm 2.4\%$, and that for calculating RW was $89.6\%\pm 1.6\%$.

4. Discussion

4.1. Qualitative and Quantitative Comparisons of the Two Modalities

The qualitative comparison between the SfM and LiDAR point clouds in our study included observations of the z-histograms, the C2M distance maps, the M3C2 distance map, and the cross-sections projected on two vertical planes. Each of the methods included results from both the north plot and the south plot in the field. We concluded that the differences between the two kinds of point clouds were not discernable in the z-histograms and were slight in the C2M distance maps. The M3C2 approach, in contrast, which calculates the difference in normal directions, revealed more obvious spatial variability.

Most parallel studies that used the M3C2 algorithm to compare UAS-SfM with LiDAR were in geomorphological applications, and their LiDAR systems varied. Cook [72] compared the UAS-SfM point cloud of a bedrock gorge with terrestrial LiDAR data during the same survey period using the M3C2 algorithm. While the dominant GSD was ~1.8–2.6 cm, the author reported an RMSE of 0.3–0.4 m; however, that study also found that that the SfM accuracy varied by different surface features, with vegetation, water, and other textures causing more errors. Bash et al. [73] analyzed the precision and accuracy of a UAS SfM point cloud versus airborne LiDAR data over a spring snow surface at Haig Glacier; with a GSD of 2.4 cm, most of the errors were within a range of $-0.049\pm 0.111$ m. Our M3C2 distance between UAS-based LiDAR and SfM point clouds resulted in distance RMSEs of 0.05–0.18 m, which seemed to be slightly better than those from previous studies. However, this also could be attributed to the smaller GSD of 0.017 m in our study, while our study field only contained vegetation and ground surfaces, which were relatively more homogeneous than the sites from previous studies.

From the SfM point cloud in data set #3, we obtained RMSEs of 0.01 m and 0.02 m by directly comparing the calculated CH with the observations of the south plot and the north plot, respectively. The accuracies (in terms of the scales of RMSE and the relative RMSE) surpassed many related studies. Becirevic et al. [69] tested the effectiveness of UAS-based SfM point clouds for measuring the CH of winter wheat (up to 1 m when mature) and obtained an RMSE of 0.022 m in a linear regression model between the observed and calculated values. De Souza et al. [51] evaluated the CH of sugarcane (up to 3.2 m when mature) using SfM point clouds generated on a fixed-wing UAS platform and reported an RMSE of 0.40 m. Chang et al. [50], in turn, monitored the CH of sorghum (Sorghum bicolor; grew up to 2.7 m) using an SfM point cloud and achieved an RMSE of 0.33 m. Finally, Ziliani et al. [18] compared the CH of a maize (up to 2 m when mature), derived from fixed-wing UAS-based SfM point clouds, with corresponding field measurements and obtained an RMSE of 0.21 m during the flowering growth stage. This was corroborated by Chu et al. [74], who also reported similar accuracies for a maize.

4.2. Importance of the GCPs

The number and distribution of the GCPs turned out to have a significant impact on the SfM point cloud accuracies. We found that the sections of the study area that contained no or limited GCPs tended to exhibit significant positive or negative differences, especially when we compared the M3C2 distance map with the locations of the GCPs for the south plot in data set #2 (28 July 2020) and the north plot in data set #5. Sanz-Ablanedo et al. [75] discussed the critical impact of the number and location of the GCPs on the accuracy of SfM point clouds. As shown in Figure 11, they found that fewer GCPs would cause the RMSE in the selected evaluation locations to be ±5 times the average GSD of the study. As the number of GCPs increased to >2 GCPs per 100 photos, the altimetric accuracy would converge to approximately twice the GSD. Our results for the C2M distance maps nearly matched this plot. For example, as shown in data set #5 in Table 2, while both plots contained nearly 300 images, the north plot had an RMSE of 0.08 m (about five times of GSD), because it only had two GCPs (0.67 GCPs per 100 photos), while the south plot had an RMSE of 0.03 m (about two times the GSD), because it had 5 GCPs (1.6 GCPs per 100 photos). Sanz-Ablanedo et al. also recommended that GCPs should be evenly distributed, ideally in a triangular mesh grid [75]. This claim was born out by the apparent patterns of significant errors in the south plot for data set #3 and the north plot for data set #5.

4.3. Limitations and Future Research

It is worth noting that we implemented all our flights in a “striped” pattern, i.e., sensors scanned in only one direction, which was similar to previous studies [18,50,51]. A number of other relevant studies, however, applied “grids” (cross-flight) patterns that covered two perpendicular scanning directions [19,31,47]. The gridded pattern arguably enables the sensors to generate improved point clouds, since the flight design covers more scan angles. Therefore, we contend that the accuracies of the calculated CH and RW could be improved by adjusting the flight pattern similarly.

We set the flight altitude at around 25 m and the flight speed at around 2 m/s in order to ensure consistency in data sets #2–5. It seems reasonable that if flights were executed at higher altitudes and speeds, the accuracies may decrease. Because the point density of LiDAR point clouds would be lower, the overlapping area among images for the SfM point clouds would decrease and the features in the images may be coarser.

Finally, we evaluated and compared the CH and the RW only in data set #3. The transferability of the optimized percentiles applied in the CHM models therefore has not been fully tested. This aspect should be addressed in future research. From a fundamental view, it is challenging to make concrete measurements of the actual CH and RW, due to the complexity of the structural characteristics of plants, i.e., leaves and stems consisting of irregular and discontinuous shapes, and their structural properties could easily change according to environmental conditions, such as temperature, time of the day, and wind speed/direction. As is widely applied in other research efforts [19,35,76,77], the most direct way to approximate the true values of CH was using a ruler to sample several times per unit (row or segment of a row) and then averaging the discrete sampled values as a representative of the unit. However, through the point clouds, we had access to the entire unit or plot. What we did was essentially averaging the nearly continuous measurements (tens or hundreds of points per unit) in the CHMs and then comparing these values with field measurements. This “continuous” versus “discrete” principle inevitably will lead to some degree of deviation in the comparison of results. Furthermore, it also presents a challenge related to our current methodology—do we need a more reliable method to evaluate the accuracy of point clouds according to not just discrete values, but rather based on their continuous nature? Future research may consider dealing with this challenge.

5. Conclusions

This study performed comprehensive comparisons between SfM and LiDAR point clouds, collected concurrently for a snap bean field in Geneva, NY, USA. Both qualitative and quantitative comparisons were presented to verify the effectiveness of the two modalities’ point clouds for generating DEMs and CHMs and their accuracies when evaluating CH and RW. The results revealed that the SfM point clouds, despite their relatively low point density, could provide high-quality DEMs and CHMs, which were comparable to their LiDAR counterparts. We also found that both SfM and LiDAR point clouds achieved a high accuracy for assessment of CH and RW—we obtained RMSEs of ~0.02 m for CH and ~0.05 m for RW.

As crop sustainability and management efficiency become the trends in precision agriculture [78,79], our findings could help farmers or third-party companies to select a proper remote sensing modality with better trade-offs between cost and accuracy, when it comes to the structural characterization of crops. Since snap beans are shorter than many broadacre crops such as rice, sugarcane, and maize, we contend that the evaluation results would likely be even better for these taller crops, should the same settings and methods be applied. Our methods and results thus should be extensible to other crops, such as soybeans and beets, which structurally resemble snap beans. These results thus bode well for the eventual use of SfM-based point clouds, which are significantly cheaper to collect than LiDAR-based 3D assessments, for the extensive assessment of crop structure and eventually models based on such structures, e.g., growth-and-yield models.