Remote Sensing of Vegetation Structure Using Computer Vision

Abstract

High spatial resolution measurements of vegetation structure in three-dimensions (3D) are essential for accurate estimation of vegetation biomass, carbon accounting, forestry, fire hazard evaluation and other land management and scientific applications. Light Detection and Ranging (LiDAR) is the current standard for these measurements but requires bulky instruments mounted on commercial aircraft. Here we demonstrate that high spatial resolution 3D measurements of vegetation structure and spectral characteristics can be produced by applying open-source computer vision algorithms to ordinary digital photographs acquired using inexpensive hobbyist aerial platforms. Digital photographs were acquired using a kite aerial platform across two 2.25 ha test sites in Baltimore, MD, USA. An open-source computer vision algorithm generated 3D point cloud datasets with RGB spectral attributes from the photographs and these were geocorrected to a horizontal precision of <1.5 m (root mean square error; RMSE) using ground control points (GCPs) obtained from local orthophotographs and public domain digital terrain models (DTM). Point cloud vertical precisions ranged from 0.6 to 4.3 m RMSE depending on the precision of GCP elevations used for geocorrection. Tree canopy height models (CHMs) generated from both computer vision and LiDAR point clouds across sites adequately predicted field-measured tree heights, though LiDAR showed greater precision (R² > 0.82) than computer vision (R² > 0.64), primarily because of difficulties observing terrain under closed canopy forest. Results confirm that computer vision can support ultra-low-cost, user-deployed high spatial resolution 3D remote sensing of vegetation structure.

1. Introduction

High spatial resolution measurements of vegetation structure in three-dimensions (3D) are critical for measuring vegetation biomass [1], carbon [2,3], habitat quality [4] and biodiversity [5] across landscapes. Demand for these measurements has grown rapidly in support of national and local carbon accounting efforts [1,2,3], and these measurements are also increasingly used to support fire spread and risk models [6,7], commercial and scientific forestry [8], ecosystem process modeling [9], and for quantitative assessments of habitat suitability and biodiversity [4,5].

Canopy height measurements are the foundation for most estimates of forest biomass and carbon stocks, fire spread potential, and habitat quality [10]. For decades, manual and semi-automated photogrammetric methods have been used to estimate vegetation canopy heights by interpreting 3D surfaces from high spatial resolution two-dimensional (2D) image stereo-pairs obtained from passive optical remote sensing platforms, both aerial and satellite [11]. Today, Light Detection and Ranging (LiDAR) is the preferred tool for making accurate remotely-sensed 3D measurements of vegetation structure for forestry and other land management and environmental science applications [4,10].

To generate accurate 3D geometry, LiDAR sensors are coupled with high precision Global Positioning Systems (GPS) and inertial monitoring (IMU) equipment [12]. While research has shown that the size and cost of LiDAR sensor systems can be reduced substantially [13], high precision GPS and IMU systems remain relatively bulky and expensive, making it unlikely that LiDAR systems capable of supporting high spatial resolution vegetation structure measurements will be deployed on aerial platforms other than commercial aircraft (manned or un-manned) for the foreseeable future.

Inexpensive hobbyist aircraft (IHA), including radio-controlled fixed wing airplanes, helicopters, balloons and kites, are increasingly being used for low-altitude aerial photography (≤120 m) in part because it has become a simple matter to deploy consumer-grade digital cameras on these platforms [14,15,16]. It has also become relatively simple to generate 3D geometry from sets of overlapping digital photographs acquired using uncalibrated consumer-grade cameras using newly-developed computer vision algorithms that completely automate this process using Structure from Motion procedures (SfM) [17]. This study investigates the potential of combining IHA and SfM technologies to enable inexpensive 3D vegetation measurements across landscapes.

SfM algorithms generate 3D geometry from large sets of overlapping but otherwise unordered 2D photographs through the automation of standard techniques from computer vision and photogrammetry [17]. SfM accomplishes this by automatically extracting “keypoints” from individual photos, identifying “feature correspondences” or “keypoint matches” among photos, and then optimizing the 3D location of feature correspondences as well as the location and orientation of cameras and camera internal parameters using bundle adjustment algorithms [18].

Bundler is a new open-source SfM software package [19] that combines the SIFT algorithm (Scale Invariant Feature Transform) [20] for keypoint extraction with bundle adjustment using the Sparse Bundle Adjustment package (SBA) [21]. SIFT and bundle adjustment have already been combined to automate the generation of 3D models of built structures and bare geological substrates using calibrated digital cameras deployed on the ground [22,23]. Unlike prior algorithms, Bundler was designed specifically to enable automated 3D reconstruction from photograph collections obtained using many different cameras with varying and even unknown internal camera parameters [17]. While Bundler and the related “Photosynth” application [24] were developed primarily for 3D visualization, these algorithms generate 3D “point clouds” with internally-consistent 3D geometry without requiring any prior information on camera position, orientation or internal parameters. Further, it has been demonstrated that point clouds produced by these methods can be geocorrected to create models of architectural and bare earth surfaces with acceptable levels of spatial accuracy [25,26].

Here we will demonstrate that vegetation canopy structure can be mapped and measured in 3D with a useful degree of precision by applying the Bundler SfM algorithm [19] to aerial photographs acquired using a consumer grade digital camera mounted on an IHA platform (kite) after a simple geocorrection involving a small number of easily obtained ground control points (GCPs). The general utility and precision of this new 3D vegetation scanning and measurement technique, named “Ecosynth” after the related “Photosynth” technology [24], will then be established by comparing its tree height estimates with those from conventional field-based and LiDAR methods across two partially forested test sites.

2. Methods

2.1. Test Sites and Field Measurements

Two 100 m × 225 m (2.25 ha) test sites were selected on the University of Maryland Baltimore County campus (UMBC) to assess 3D vegetation measurements by Ecosynth and LiDAR against tree measurements made in the field (Figure 1). The four corners of each test site were geolocated with submeter accuracy using a Trimble GeoXT and then divided into a grid of 25 m × 25 m subplots using a compass and meter tape, starting from the corner points. Heights of all trees greater than 12.7 cm diameter at breast height (DBH; measured using a DBH tape) were measured across each test site before leaf-off in fall 2009 (2009/11/1–2009/11/8) using a Haglöf Vertex laser hypsometer.

The Knoll test site (Figure 1a; 39°15′9″N 76°42′40″W) was composed primarily of a forested hill reaching 60 m above sea level (ASL; determined by LiDAR), then descending 5 to 10 m to a road and tennis courts, and then to 40 m ASL below the roadbed. The hill was covered by a mixed-age forest dominated by beech (Fagus grandifolia), oak (Quercus spp.), and hickory (Carya spp.) but also including several large mature white ash (Fraxinus americana) and tulip-poplar (Liriodendron tulipifera). One or two large oak or tulip-poplar crowns typically dominated the overstory within each subplot, with a beech and hickory understory. DBH of all trees with DBH > 12.7 cm was measured across this site.

The Herbert Run test site (Figure 1b; 39°15′28″N 76°42′24″W) straddles a steeply sloping forested riparian zone along a stream (Herbert Run) adjacent to a road. The maximum site elevation was along the road (58 m ASL) and an adjacent dam, sloping steeply by 20 m to the stream channel, with up to 50% grade in places. Forest canopy consisted mostly of an even-aged stand of black locust (Robinia pseudoacacia) overstory with black cherry (Prunus serotina) understory along the steep stream banks, with honey locust (Gleditsia triacanthos) and green ash (Fraxinus pennsylvanica) becoming dominant in closest proximity to the stream. The eastern corner of the site edges on a beech and oak forest similar to that of the Knoll site.

Figure 1. The Knoll (a) and Herbert Run (b) test sites on the campus of the University of Maryland Baltimore County. Sites and 25 m × 25 m subplots are outlined in red over 2008 leaf-off orthophotograph. Green lines delimit the approximate extent of kite aerial photograph acquisition at each site, blue crosses are GCPs used for geocorrection, and yellow circles are GCPs used in geocorrection accuracy assessment.

Figure 1. The Knoll (a) and Herbert Run (b) test sites on the campus of the University of Maryland Baltimore County. Sites and 25 m × 25 m subplots are outlined in red over 2008 leaf-off orthophotograph. Green lines delimit the approximate extent of kite aerial photograph acquisition at each site, blue crosses are GCPs used for geocorrection, and yellow circles are GCPs used in geocorrection accuracy assessment.

2.2. Image Acquisition and LiDAR

Aerial photographs for Ecosynth were acquired using an off-the-shelf Canon A470 digital camera and a kite aerial photography rig (KAP) composed of an ITW Alpine Delta Conyne kite and 1,000’ of 100-lb test Dacron kite line. The camera was mounted on a Picavet suspension frame [16] attached to the kite line approximately 30 m below the kite. Photographs with a resolution of 2,592 × 1,944 (5.0 megapixels) were taken at approximately 3–5 second intervals using the open-source CHDK camera firmware modification [27] installed on the camera. Photographs were acquired using the KAP rig in late summer 2009 (07/09, 09/06, 09/13) between 11:00 and 15:00 on sunny to partly cloudy days with light wind, with most photos acquired between 20 m and 110 m above surface. The camera and KAP frame were adjusted prior to flight to acquire photographs roughly at nadir, but this could not be controlled during flight and pictures were typically taken off-nadir. On flight completion, photographs were downloaded from the camera’s memory card for processing.

LiDAR data were acquired with the goal of mapping terrain at high spatial resolution across Baltimore County MD, USA by a local contractor under contract with the Baltimore County Office of Information Technology using an Optech ALTM 2050 LiDAR with Airborne GPS and IMU under leaf-off conditions in the spring of 2005 (2005/03/18–2005/04/15; ≈800–1,200 m above ground surface; ≈140 knots airspeed; 36 Hz scan frequency; 20° scan width half angle; 50,000 Hz pulse rate; ≈150 m swath overlap; mean point density 1.5 points·m⁻² ; see Table 1 for number of points per subplot).

Figure 2. Ecosynth procedure for vegetation measurements using computer vision.

Figure 2. Ecosynth procedure for vegetation measurements using computer vision.

2.3. Point Cloud Generation Using Bundler

Aerial photographs acquired across each test site were uploaded into Bundler software [19] for processing into 3D “point clouds” (Figure 2) using the default Bundler settings on an Intel Core i7-965 3.2 Ghz Quad-Core PC with 12 GB RAM. To reduce computational time, the pixel resolution of all photographs was reduced to 640 × 480 (0.3 megapixels) prior to uploading into Bundler, as Bundler processing time increased very substantially with image resolution [17]. The Bundler software first identified sets of matching potential features, or keypoints, across the set of photographs using a pattern recognition algorithm (automated feature recognition; Figure 2). Next, the software selected only those photographs and keypoints that contributed to an optimal 3D geometric solution, as determined by the bundle adjustment algorithm (Figure 2, Table 1); suboptimal keypoints and photographs were rejected. From these optimal keypoints and photographs, Bundler then created an optimized 3D point cloud dataset in an arbitrary Cartesian coordinate system and output the data as a text file containing a record of each keypoint with its X, Y, Z position and an RGB color value assigned from the first image used in the 3D reconstruction. Data points for each selected photograph were also output, indicating camera position (X, Y, Z) and rotation relative to the scene.

Table 1. Bundler points generated for Ecosynth compared with LiDAR across test sites (Figure 1).

Test site	Input images	Processing time (h)	Images selected	Keypoints	Trimmed	Outliers	Ecosynth Points		LiDAR points
							Total	Ground	First return	Bare earth
Knoll	237	2.1	145	36,524	2,658	517	33,349	1,897	19,074	15,657
Herbert Run	627	29.2	599	108,840	46,346	1,135	61,359	10,298	23,374	12,822

Table 1. Bundler points generated for Ecosynth compared with LiDAR across test sites (Figure 1).

Test site	Input images	Processing time (h)	Images selected	Keypoints	Trimmed	Outliers	Ecosynth Points		LiDAR points
							Total	Ground	First return	Bare earth
Knoll	237	2.1	145	36,524	2,658	517	33,349	1,897	19,074	15,657
Herbert Run	627	29.2	599	108,840	46,346	1,135	61,359	10,298	23,374	12,822

2.4. Geocorrection of Bundler Point Clouds

Bundler point clouds were geocorrected to a Universal Transverse Mercator projection (Zone 18N, WGS84 horizontal datum, NAVD 88 vertical datum) by transforming their scale, translation and rotation in all 3 Cartesian axes to fit five reference points in the Bundler point cloud to five matching GCPs for ground features. A 7-parameter Helmert transformation was used for geocorrection, with optimal parameter values for a single factor of scale, three factors of translation along each axis, and three angles of rotation along each axis [28] obtained by minimizing the transformed distance between five Bundler reference points and five GCPs in Cartesian space by minimizing the sum of squared residuals in X, Y, Z using the simplex optimization algorithm [29,30]. Geocorrected Bundler point clouds were then produced by applying the optimal Helmert transformation model to the full set of Bundler keypoints.

GCP features consisted of curbs, road markings and pavement corners visible in a 2008 leaf-off Baltimore County aerial orthophotograph (0.6 m horizontal accuracy, 0.3 m pixel resolution, collected 2008/03/01–2008/04/01). Bundler reference points were identified by manually interpreting 3D structures and RGB colors corresponding to GCP features identified in the orthophotograph. Sets of between 10 and 20 Bundler keypoints in the immediate vicinity of each GCP were averaged to obtain each Bundler reference point, as it proved impractical to identify individual keypoints within 3D point clouds that corresponded precisely to a single GCP in the orthophotograph. Horizontal coordinates for GCPs (X, Y) were determined directly from the orthophotograph. Vertical coordinates (elevations; Z) were obtained by two different methods: “Standard” and “Precision”. Standard GCP elevations were obtained by reference to a freely downloadable USGS 10 m Digital Terrain Model [DTM; 31]. Precision GCP elevations were obtained from a LiDAR DTM (Section 2.6, below). Standard and precision GCPs were then used to create “standard” and “precision” geocorrected Ecosynth point-clouds for each site, respectively. The accuracy of Helmert-transformed Bundler point clouds was assessed using an independent set of 8 pairs of Bundler reference points + precision GCPs by calculating root mean squared errors [RMSE; 32] between Helmert predicted reference point coordinates and the correct coordinates of precision GCPs. Geocorrection accuracy when using 3 instead of 5 pairs of GCPs + Bundler reference points in Helmert transformations was also evaluated, by calculating RMSEs across an independent set of 10 pairs of Bundler reference points + precision GCPs.

2.5. Outlier Filtering and Trimming of Geocorrected Point Clouds for Ecosynth

Geocorrected Bundler point clouds contained a small but significant number of points located far above or below the possible spatial limits of any real features in each test site, most likely as artifacts of incorrect matches between features [18]. Prior to removing these points using statistical outlier filtering, point clouds were trimmed to a 25 m buffer surrounding test sites (Table 1). Statistical outlier filtering was then performed on trimmed Bundler point clouds in two stages. First, a global filter was applied by computing standardized Z-scores across the elevation coordinates of all points in each trimmed point cloud [33] and all points with Z-scores > 3 or < −3 were removed. Next, a local filter was applied by overlaying a 25 m × 25 m grid across the trimmed point cloud of each site (not the same as the 25 m subplot grid), Z-scores were computed within each grid cell, and all points with Z-scores > 3 or < −3 were removed. Together, the global and local filters removed between 1% and 2% of keypoints (Table 1). While empirical filters did remove some verifiable canopy points, these were implemented instead of manual editing to facilitate automation.

2.6. Digital Terrain Models (DTM)

LiDAR DTMs for each test site were generated from “bare earth” points produced using the Terrascan software package (Terrasolid Ltd., Jyväskylä, Finland) by the LiDAR contractor. Ecosynth “ground points” were obtained from trimmed Ecosynth point clouds (site + 25 m buffer) by using a Progressive Morphological filter [34,35] to programmatically separate “ground” from “non-ground points” (Table 1). Ordinary Kriging [32,36] was used to interpolate 1 m resolution DTMs from both LiDAR bare earth points and Ecosynth “ground” points. DTM statistics for each site were then produced after trimming away the 25 m buffer area.

2.7. Canopy Height Models (CHM) and Tree Height Metrics

Canopy height models (CHMs) were produced from the elevation values of Ecosynth non-ground points and LiDAR first-return points by subtracting away DTM ground elevations. Summary height statistics (CHM height metrics) were then calculated across all CHM points within each subplot with height >2 m, including the median, mean, minimum, maximum, quantiles (25th, 75th, 90th, 95th and 99th = Q25, Q75, Q90, Q95 and Q99 respectively), and the mean of all points higher than the 99th percentile [8]. Heights < 2 m were excluded to avoid potential inclusion of non-tree objects such as shrubs and cars. The relative accuracy of tree height estimates from Ecosynth and LiDAR CHMs was then tested based on the predictive strength (R²) of the strongest models obtained for the prediction of field-measured mean tree heights (average of the 5 tallest trees in each subplot; representative of dominant canopy height) across the set of subplot CHM height metrics using stepwise multiple linear regression [37,38]. To facilitate direct comparisons between CHMs, 1 m resolution gridded CHMs were interpolated from Ecosynth and LiDAR CHM point clouds using Ordinary Kriging [32,36]. Tree-covered areas were identified and mapped across each site using gridded LiDAR CHMs by converting all 1 m grid cells with >2 m height into canopy cover polygons. The number of ground points under tree canopy produced by Ecosynth and LiDAR was determined by selecting all ground points located beneath canopy cover polygons at each site.

2.8. Aboveground Biomass Models (AGB)

Predictive models for aboveground biomass density (AGB density in kg dry weight m⁻²) across subplots at the Knoll site were generated from both Ecosynth and LiDAR CHM height metrics using standard LiDAR forestry methods developed to predict AGB and aboveground carbon across larger forested areas [39,40]. First, the AGB of all trees at the Knoll site with DBH > 12.7 cm was estimated using species-level allometric equations applied to field-measured DBH [41]. Next, total tree biomass in each 25 m × 25 m subplot was calculated as the sum of tree biomass within each subplot and divided by subplot area (625 m²) to produce AGB density estimates for each subplot in kg AGB·m⁻². Simple and stepwise multiple linear regressions were then used to predict AGB densities from both Ecosynth and LiDAR CHM tree height metrics, and the strongest predictive models for AGB density were identified as those with the highest R² values [39,40].

3. Results

3.1. General Characteristics and Geometric Precision of Ecosynth Point Clouds

Point clouds produced by Ecosynth and LiDAR are illustrated in Figure 3 and Figure 4 and described in Table 1 and Table 2. Side-by-side comparisons of Ecosynth and LiDAR highlight the high density of Ecosynth point clouds obtained across forested areas and also the spatial heterogeneity of Ecosynth point clouds relative to LiDAR (Figure 3 and Figure 4). This heterogeneity was at least partly caused by the use of automated feature recognition to acquire 3D points from imagery, producing denser point clouds in areas where larger numbers of features were identifiable in imagery [42]. This was clearly the case in areas covered by woody vegetation, as these areas tended to have the highest point densities in each site, confirming that SfM algorithms can produce high density 3D point captures across tree canopies and other coarsely vegetated surfaces (Figure 1, Figure 3 and Figure 4). A further source of heterogeneity was our use of kite aerial photography, as this method was difficult to control and resulted in very different numbers of images acquired across each site (237 vs. 627; Table 1). Automated filtering by Bundler ultimately selected 145 images for point cloud generation at the Knoll site and 599 images at Herbert Run, with the total number of points generated at each site roughly paralleling image numbers (Table 1), though point yields per image were substantially higher at the Knoll site (252 points/image) compared with Herbert Run (181 points/image). Ecosynth data processing and preparation (Section 2.4, Section 2.5 and Section 2.6 above) took approximately 1.5 days for both of the sample sites.

The geocorrected horizontal positional accuracy of Ecosynth point clouds ranged from 1.0 to 1.5 m RMSE and was little influenced by the precision of GCP elevation data used in geocorrection (data not shown) or by the use of 3 rather than 5 GCPs for geocorrection (Table 2). In contrast, vertical positional accuracy was strongly influenced by both elevation accuracy and number of GCPs used in geocorrection (Table 2), with vertical RMSEs ranging from 2.0 to 4.3 m when a USGS 10 m DTM provided elevations (standard geocorrection), improving to 0.6 to 0.9 m RMSE when LiDAR DEM elevations were used (precision geocorrection).

Table 2. Positional errors (RMSE) in Ecosynth and LiDAR point clouds.

Site	Horizontal		Vertical
			Standard		Precision
	3 GCPs	5 GCPs	3 GCPs	5 GCPs	3 GCPs	5 GCPs
Knoll	1.5 m	1.0 m	4.6 m	4.3 m	1.1 m	0.9 m
Herbert Run	1.1 m	1.3 m	6.1 m	2.0 m	0.6 m	0.6 m
LiDAR†	0.15 m		0.24 m
† Contractor reported.

Table 2. Positional errors (RMSE) in Ecosynth and LiDAR point clouds.

Site	Horizontal		Vertical
			Standard		Precision
	3 GCPs	5 GCPs	3 GCPs	5 GCPs	3 GCPs	5 GCPs
Knoll	1.5 m	1.0 m	4.6 m	4.3 m	1.1 m	0.9 m
Herbert Run	1.1 m	1.3 m	6.1 m	2.0 m	0.6 m	0.6 m
LiDAR†	0.15 m		0.24 m
† Contractor reported.

3.2. Terrain Models

Ground points and DTMs produced by Ecosynth and LiDAR are illustrated in Figure 5. Ecosynth produced far fewer ground points compared with LiDAR, especially under tree canopy (Figure 5a to Figure 5d; Table 1), an unsurprising result given that LiDAR data were acquired under leaf-off conditions, while images for Ecosynth were acquired under leaf-on conditions. Even with LiDAR, obtaining ground points by terrain filtering produces poor results when tree canopies are dense and closed, especially in highly varying terrain where neighboring canopy and ground points may occupy the same elevation [43]. Both of these conditions were present at both test sites during Ecosynth image acquisition, helping to explain the relatively poor quality of Ecosynth DTMs when compared with LiDAR (Figure 5).

The challenge of acquiring Ecosynth ground points under closed tree canopies is well described by the relatively sparse ground point densities obtained under tree covered areas using Ecosynth (0.02 points·m⁻² at Knoll, 0.03 points·m⁻² at Herbert Run) compared with LiDAR (0.6 points·m⁻² and 0.4 points·m⁻² respectively; Figure 4 and Figure 5). As a result of these sparse point densities, Ecosynth DTMs showed substantial errors when compared to LiDAR DTMs (Figure 5i & Figure 5j). Some Ecosynth ground points identified under canopy at the Knoll site appear substantially higher than those from LiDAR and therefore likely represent points obtained from shrubs or parts of trees rather than true ground surface points (Figure 5i). At Herbert Run, trees on steep stream banks paralleling a road had canopies close to the road’s elevation, causing substantial errors in the southwestern corner of the Ecosynth DTM (at left in Figure 5h); high elevations were produced where there should be steep banks and a stream channel (Figure 5g and Figure 5j). Even parameter adjustment of terrain filters and repeated iterations did not remove canopy points in this area.

Figure 3. Point clouds produced by LiDAR and Ecosynth across the Knoll (Figure 1a) and Herbert Run (Figure 1b) test sites, compared with 2008 leaf-off orthophotograph, with 25 m × 25 m subplot grid in red (a and d). Knoll image (a) LiDAR first return (b) and Ecosynth points (c). Herbert Run image (d) LiDAR first return (e) and Ecosynth points (f). Note relief displacement of tree canopy in (d). Height colors have the same scale within each site but not across sites. Black lines delimit tree canopy determined from LiDAR.

Figure 3. Point clouds produced by LiDAR and Ecosynth across the Knoll (Figure 1a) and Herbert Run (Figure 1b) test sites, compared with 2008 leaf-off orthophotograph, with 25 m × 25 m subplot grid in red (a and d). Knoll image (a) LiDAR first return (b) and Ecosynth points (c). Herbert Run image (d) LiDAR first return (e) and Ecosynth points (f). Note relief displacement of tree canopy in (d). Height colors have the same scale within each site but not across sites. Black lines delimit tree canopy determined from LiDAR.

3.3. Tree Heights

Ecosynth CHMs produced canopy height predictions with levels of precision considered acceptable in the LiDAR forestry literature [R² ≥ 0.8; 7, 8], at least when paired with a LiDAR DTM (Figure 6f). However, LiDAR CHMs (Figure 6b and Figure 6e) generally produced more precise predictions of field-measured canopy heights than Ecosynth (Figure 6a and Figure 6d), and results were not consistent across sites (Figure 6). Ecosynth CHMs showed vulnerability both to geocorrection errors caused by low precision elevation data in standard GCPs (Table 2) and to the low quality of Ecosynth DTMs generated under closed tree canopy (Figure 5f and Figure 5h). To quantify these errors, we geocorrected Ecosynth point clouds using precision GCPs (Table 2) and used LiDAR DTMs in place of Ecosynth DTMs to produce CHMs (Figure 6c and Figure 6f). With these improvements, Ecosynth predicted field-measured tree heights extremely well at the Herbert Run site, where the largest terrain errors were observed (Figure 6f, Adj. R² = 0.80), yet performed even more poorly at the Knoll site (Figure 6c, Adj. R² = 0.53), especially when compared with LiDAR estimates at each site (Figure 6e & Figure 6b, respectively). When standard Ecosynth point clouds were paired with LiDAR DTMs, results were similar to the previous at both the Herbert Run (Adj. R² = 0.80, RMSE = 2.9 m) and Knoll test sites (Adj. R² = 0.56, RMSE = 4.2 m), indicating that low quality Ecosynth DTMs were the main source of error in Ecosynth CHMs, at least at the Herbert Run test site.

Figure 4. Oblique views of Ecosynth and LiDAR point clouds at the Knoll (Figure 1a) and Herbert Run (Figure 1b) test sites. Knoll aerial photograph draped on LiDAR first return (a), LiDAR first return plus bare earth (b), and Ecosynth point cloud (c; RGB colors). Herbert Run aerial photograph draped on LiDAR first return (d), LiDAR first return plus bare earth (e), and Ecosynth point cloud (f; RGB colors). 25 m subplots are outlined in purple at constant 50 m elevation. Heights in (b) and (e) use same colors as Figure 3.

Figure 4. Oblique views of Ecosynth and LiDAR point clouds at the Knoll (Figure 1a) and Herbert Run (Figure 1b) test sites. Knoll aerial photograph draped on LiDAR first return (a), LiDAR first return plus bare earth (b), and Ecosynth point cloud (c; RGB colors). Herbert Run aerial photograph draped on LiDAR first return (d), LiDAR first return plus bare earth (e), and Ecosynth point cloud (f; RGB colors). 25 m subplots are outlined in purple at constant 50 m elevation. Heights in (b) and (e) use same colors as Figure 3.

Figure 5. LiDAR and Ecosynth ground points (a–d), DTMs (e–h) and DTM differences (i and j). Ground points for Knoll site from LiDAR (a) and Ecosynth (b) and Herbert Run LiDAR (c) and Ecosynth (d). DTMs from Knoll LiDAR (e) and Ecosynth (f) and from Herbert Run LiDAR (g) and Ecosynth (h). DTM differences, Ecosynth—LiDAR, for Knoll (i) and Herbert Run (j). Site orientation and height colors in (a) to (h) are same as Figure 3. Black lines delimit tree canopy as determined from LiDAR CHM.

Figure 5. LiDAR and Ecosynth ground points (a–d), DTMs (e–h) and DTM differences (i and j). Ground points for Knoll site from LiDAR (a) and Ecosynth (b) and Herbert Run LiDAR (c) and Ecosynth (d). DTMs from Knoll LiDAR (e) and Ecosynth (f) and from Herbert Run LiDAR (g) and Ecosynth (h). DTM differences, Ecosynth—LiDAR, for Knoll (i) and Herbert Run (j). Site orientation and height colors in (a) to (h) are same as Figure 3. Black lines delimit tree canopy as determined from LiDAR CHM.

Figure 6. Results of stepwise multiple linear regressions of subplot canopy height metrics from Ecosynth and LiDAR CHMs on field measured canopy heights. Knoll standard Ecosynth (a), LiDAR (b), and precision Ecosynth with LiDAR DTM (c). Herbert Run standard Ecosynth (d), LiDAR (e), and precision Ecosynth with LiDAR DTM (f). Dashed lines are regression models; solid line is observed = expected. Model parameters are described in Table 3.

Figure 6. Results of stepwise multiple linear regressions of subplot canopy height metrics from Ecosynth and LiDAR CHMs on field measured canopy heights. Knoll standard Ecosynth (a), LiDAR (b), and precision Ecosynth with LiDAR DTM (c). Herbert Run standard Ecosynth (d), LiDAR (e), and precision Ecosynth with LiDAR DTM (f). Dashed lines are regression models; solid line is observed = expected. Model parameters are described in Table 3.

Ecosynth tree height prediction errors are harder to explain at the Knoll test site, as use of a high quality DTM actually decreased the precision of tree height predictions from Ecosynth (Figure 6a vs. Figure 6c). One explanation for this may be errors caused by the relatively complex canopy structure at this site (Figure 4a). While trees at Herbert Run were fairly uniform in age and height, with field-measured heights ranging from 4.3 to 26.0 m and a subplot mean +/− SD of 15.0 +/− 6.1 m, trees at the Knoll site were highly non-uniform, ranging from 6.5 to 33.6 m in height with a subplot mean of 20.9 +/− 6.1 m. Further, at this site the canopies of one to two very large oaks or tulip-poplars tended to tower above other trees in each subplot. These taller trees may have obstructed other trees from view in the low-altitude aerial photographs we acquired for use in Ecosynth, and this may have resulted in a failure to capture points uniformly across the tree canopy of the Knoll site.

Figure 7. Maps and density plots of differences in Ecosynth CHMs after subtracting LiDAR CHMs, in m. Knoll Ecosynth CHM (a) and Ecosynth CHM with LiDAR DTM (b). Herbert Run Ecosynth CHM (c) and Ecosynth CHM with LiDAR DTM (d). Black lines in difference maps delimit tree canopy determined from LiDAR CHM. Colors are same as Figure 5i and Figure 5j. Dashed vertical lines in density plots are mean difference and 1 SD from mean, solid vertical lines at 0.

Figure 7. Maps and density plots of differences in Ecosynth CHMs after subtracting LiDAR CHMs, in m. Knoll Ecosynth CHM (a) and Ecosynth CHM with LiDAR DTM (b). Herbert Run Ecosynth CHM (c) and Ecosynth CHM with LiDAR DTM (d). Black lines in difference maps delimit tree canopy determined from LiDAR CHM. Colors are same as Figure 5i and Figure 5j. Dashed vertical lines in density plots are mean difference and 1 SD from mean, solid vertical lines at 0.

Sources of error in Ecosynth tree height measurements were further explored by subtracting LiDAR CHMs from Ecosynth CHMs to investigate differences between them (Figure 7). Negative canopy height differences at upper left in Figure 7c (labeled c.1) were completely removed when a LiDAR DTM was used instead of the Ecosynth DTM (Figure 7d), revealing errors in Ecosynth DTM generation (Figure 5h and Figure 5j). The negative patch labeled a.3 in Figure 7a resulted both from a tree removal and an error in terrain filtering when producing the Ecosynth DTM (Figure 5b), and again, substituting a LiDAR DTM removed the error, leaving just the difference in canopy height from tree removal.

Ecosynth CHMs also revealed real changes in canopy structure between 2005, when LiDAR data were acquired, and 2009 when Ecosynth images were acquired. Several large patches of negative difference between LiDAR and Ecosynth CHMs at the Knoll site (Figure 7a) were not removed when a LiDAR DTM was used to generate the Ecosynth CHM (Figure 7b), and are explained by tree gaps from the recent fall of a large beech (a.1), and a large tree removal identified by a large tree stump (a.2). Further, Ecosynth canopy heights were generally higher than LiDAR heights in the density plots and maps of Figure 7, especially in the relatively immature forests of the Herbert Run site, as would be expected after 4 years of tree growth.

3.4. Aboveground Biomass (AGB)

LiDAR CHM height metrics are commonly used to predict AGB across extensive areas of forest by calibrating these metrics to field-based estimates of AGB [39,40]. The relative strength of Ecosynth and LiDAR CHMs for this purpose was tested across the 25 m subplots of the Knoll site, where DBH of all trees with DBH > 12.7 cm was measured in the field. This was accomplished by simple linear regression and stepwise multiple linear regression of subplot CHM height metrics against subplot AGB estimates derived from field- measured DBH using allometric methods (Table 3).

Table 3. Regression models predicting aboveground biomass (AGB) from Ecosynth and LiDAR CHM height metrics across 25 m subplots at the Knoll site. Results of simple linear regression represent the single metric model with the highest R².

Simple Linear Regression
Method	Equation form and metrics†	R2	RMSE (kg AGB·m−2)
Standard Ecosynth	AGB = −2.0 + 1.5*Q25	0.41	12.6
LiDAR	AGB = −12.0 + 2.3*Q25	0.60	10.4
Precision Ecosynth + LIDAR DTM	AGB = 11.0 + 2.3*MIN	0.37	13.1
Multiple Linear Regression Models
Sensor	Equation form and metrics†	Adj. R2	RMSE (kg AGB·m−2)
Standard Ecosynth	AGB = −0.7 + 1.7*MIN + 2.1*Q25 − 17.2*Q90 + 15.9*Q95	0.52	11.3
LiDAR	AGB = −13.8 + 23.3*MEAN99 + 2.2*Q25 + 3.4*Q90 − 26.5*Q99	0.68	9.2
Precision Ecosynth + LIDAR DTM	AGB = −4.6 + 1.8*MIN + 0.7*Q99	0.46	11.9
† Subplot height metrics from CHMs: MEAN = mean height; MED = median height; MAX = maximum height; MIN = minimum height; MEAN99 = mean of all points > 99th quantile; Q25, Q75, Q90, Q95, Q99 = Height Quantiles.

Table 3. Regression models predicting aboveground biomass (AGB) from Ecosynth and LiDAR CHM height metrics across 25 m subplots at the Knoll site. Results of simple linear regression represent the single metric model with the highest R².

Simple Linear Regression
Method	Equation form and metrics†	R2	RMSE (kg AGB·m−2)
Standard Ecosynth	AGB = −2.0 + 1.5*Q25	0.41	12.6
LiDAR	AGB = −12.0 + 2.3*Q25	0.60	10.4
Precision Ecosynth + LIDAR DTM	AGB = 11.0 + 2.3*MIN	0.37	13.1
Multiple Linear Regression Models
Sensor	Equation form and metrics†	Adj. R2	RMSE (kg AGB·m−2)
Standard Ecosynth	AGB = −0.7 + 1.7*MIN + 2.1*Q25 − 17.2*Q90 + 15.9*Q95	0.52	11.3
LiDAR	AGB = −13.8 + 23.3*MEAN99 + 2.2*Q25 + 3.4*Q90 − 26.5*Q99	0.68	9.2
Precision Ecosynth + LIDAR DTM	AGB = −4.6 + 1.8*MIN + 0.7*Q99	0.46	11.9
† Subplot height metrics from CHMs: MEAN = mean height; MED = median height; MAX = maximum height; MIN = minimum height; MEAN99 = mean of all points > 99th quantile; Q25, Q75, Q90, Q95, Q99 = Height Quantiles.

As with tree heights, LiDAR CHM height metrics were stronger predictors of AGB than height metrics from Ecosynth, based on comparisons of R² values for subplot AGB models across the Knoll site (Table 3). However, both Ecosynth and LiDAR-based AGB models were within the range of error published in the LiDAR forestry literature [44,45], and all AGB models were fairly imprecise, with the best model from LiDAR producing an RMSE of 9.2 kg AGB·m⁻² and the worst model from Ecosynth an RMSE of 13.1 kg AGB·m⁻², representing errors in AGB prediction of 44% to 62% of site mean AGB (21 ± 16 kg AGB·m⁻²; mean ± SD; Table 3). Useful maps of vegetation carbon density could thus be produced from both Ecosynth and LiDAR AGB density maps using standard methods [40].

4. Conclusions

This study has demonstrated the potential of Ecosynth as a new technology for ultra-low-cost, user-deployed, rapid recovery remote sensing of vegetation structure that combines off-the shelf digital cameras, hobbyist aerial platforms and open source computer vision software to produce high-spatial-resolution multispectral 3D point clouds. While Ecosynth clearly offers tremendous potential as a remote sensing technology, it remains immature, with multiple challenges that must be overcome before it will be ready for general use in environmental mapping (Table 4). At this early stage of development, Ecosynth has yet to equal the precision and accuracy of LiDAR in producing CHMs for mapping tree heights, vegetation biomass, carbon and other parameters. Ecosynth was especially challenged in generating DTMs below closed canopy vegetation, an unsurprising result, but also seemed to perform less well in mapping tree heights across highly heterogeneous canopies. On the other hand, Ecosynth performed nearly as well as LiDAR in generating CHMs across relatively homogeneous canopies, especially when high-quality DTMs were used (Figure 6f). We expect that in general Ecosynth will perform better in fragmented landscapes with smaller patches of tree cover as compared with larger expanses of continuous forest canopy.

Table 4. Ecosynth: challenges and potential solutions.

Challenges	Solutions
Uneven point cloud coverage	Modify SIFT parameters [42] Apply additional computer vision analysis [46] Improve the geometry and density of image acquisition
Accurate georeferencing	Field placement of georeferenced markers
Optimal image acquisition	Improvements in hobbyist aircraft for remote sensing [15] Further testing to optimize image acquisitions: overlaps, numbers of photographs, pixel resolutions, angles, altitudes, geometries
Poor canopy penetration	Combine ground and aerial photography within Ecosynths Leaf-off image acquisitions
Limited spatial extent	Higher altitudes Longer flights Merging of multiple image acquisitions
Image processing time	Advances in computer vision algorithms

Table 4. Ecosynth: challenges and potential solutions.

Challenges	Solutions
Uneven point cloud coverage	Modify SIFT parameters [42] Apply additional computer vision analysis [46] Improve the geometry and density of image acquisition
Accurate georeferencing	Field placement of georeferenced markers
Optimal image acquisition	Improvements in hobbyist aircraft for remote sensing [15] Further testing to optimize image acquisitions: overlaps, numbers of photographs, pixel resolutions, angles, altitudes, geometries
Poor canopy penetration	Combine ground and aerial photography within Ecosynths Leaf-off image acquisitions
Limited spatial extent	Higher altitudes Longer flights Merging of multiple image acquisitions
Image processing time	Advances in computer vision algorithms

The quality of Ecosynth point clouds we obtained might have been substantially improved by fine-tuning and combining existing computer vision algorithms, and also by better controlling the configuration and density of images acquired for Ecosynth. Use of a kite aerial platform was far from ideal, producing relatively inconsistent image acquisitions; a radio-controlled hobbyist aircraft (fixed wing or helicopter) flying in a standard pattern might solve this problem. Moreover, our testing was limited to two partially tree-covered field sites in the Temperate zone of the United States; tests of Ecosynth under more varied conditions will certainly be needed to determine optimal methods for Ecosynth vegetation structure measurements more generally. Other challenges to optimal use of Ecosynth (Table 4) may be addressed by testing different configurations of computer vision algorithms and image acquisitions across field sites with varying conditions and comparing results with LiDAR point-clouds and direct measurements in the field.

Some of the most promising avenues by which Ecosynth technologies might advance remote sensing remain to be explored. In contrast with LiDAR, Ecosynth generates multispectral point clouds, offering the potential to map canopy density in more detail, including leaf carbon, nitrogen and even tree species, as demonstrated by LiDAR point cloud fusion with multi- and hyperspectral imagery [47,48,49]. Finally, the low expense and logistical simplicity of Ecosynth image acquisition enables its routine deployment by end users for 3D remote sensing of vegetation and other structures at field sites on demand, enabling observations at high temporal frequencies and the exploration of vegetation dynamics and phenology in 3D using off-the shelf or spectrally-modified consumer digital cameras [15,50]. The Ecosynth approach may also assist in characterizing vegetation fuels adjacent to structures at high spatial resolutions in 3D for fire research and rapid on-site fire risk assessments at the wildland-urban interface [6,7]. While Ecosynth remains a technology prototype in need of further development, there is no question that this development is merited, as it offers clear potential to transform 3D remote sensing from an expensive high technology enterprise to an inexpensive user-driven local activity.