Quality Assessment of DSMs Produced from UAV Flights Georeferenced with On-Board RTK Positioning

Abstract

High-resolution Digital Surface Models (DSMs) from unmanned aerial vehicles (UAVs) imagery with accuracy better than 10 cm open new possibilities in geosciences and engineering. The accuracy of such DSMs depends on the number and distribution of ground control points (GCPs). Placing and measuring GCPs are often the most time-consuming on-site tasks in a UAV project. Safety or accessibility concerns may impede their proper placement, so either costlier techniques must be used, or a less accurate DSM is obtained. Photogrammetric blocks flown by drones with on-board receivers capable of RTK (real-time kinematic) positioning do not need GCPs, as camera stations at exposure time can be determined with cm-level accuracy, and used to georeference the block and control its deformations. This paper presents an experimental investigation on the repeatability of DSM generation from several blocks acquired with a RTK-enabled drone, where differential corrections were sent from a local master station or a network of Continuously Operating Reference Stations (CORS). Four different flights for each RTK mode were executed over a test field, according to the same flight plan. DSM generation was performed with three block control configurations: GCP only, camera stations only, and with camera stations and one GCP. The results show that irrespective of the RTK mode, the first and third configurations provide the best DSM inner consistency. The average range of the elevation discrepancies among the DSMs in such cases is about 6 cm (2.5 GSD, ground sampling density) for a 10-cm resolution DSM. Using camera stations only, the average range is almost twice as large (4.7 GSD). The average DSM accuracy, which was verified on checkpoints, turned out to be about 2.1 GSD with the first and third configurations, and 3.7 GSD with camera stations only.

1. Introduction

1.1. High Resolution DSM from UAV Photogrammetry

In the past, specialised hardware and software, long processing times by trained personnel, high fixed costs, and limits in platform flexibility hampered or prevented the application of traditional airborne photogrammetry in many Earth science and engineering branches. Today, the combination of SfM (structure from motion) photogrammetry [1] and unmanned aerial vehicle (UAV) photogrammetry [2], i.e., of highly efficient software packages and great platform flexibility, supplies a very effective surface reconstruction tool, delivering DSM with unprecedented resolution at low cost. Photogrammetric surveys with UAVs are therefore increasingly carried out directly by research groups in many disciplines: in geology and geotechnics [3,4,5,6,7] to monitor landslides; in geomorphology [8,9,10,11] to study the time evolution of landforms; in glaciology [12,13,14] to estimate ice volume changes and motion; and in landscape monitoring and change detection [15,16] to support volume estimates in earthworks [17,18].

The ease of use of inexpensive micro UAVs and their ability to operate apart from regulatory restrictions, in practically any environment, make them very handy platforms. Ground operations can be managed in most cases by just a couple of people equipped with suitable targets and geodetic GNSS (Global Navigation Satellite Systems) receivers capable of delivering cm-level accuracy. Getting rid of the sometimes awkward search for appropriate instrument setting locations, UAVs are also taking a good deal of surveys that were thought to be the preserve of terrestrial laser scanning back to photogrammetry.

The smoothness of the ground operations and the high degree of automation of software packages in block orientation and DSM generation make it possible to produce high resolution Digital Surface Model (DSM) with a very limited budget and time effort. However, as James et al. [19] point out, ensuring end product quality and block design optimisation still require a deep understanding of the photogrammetric process, which the apparently straightforward SfM workflow might hide to some extent. SfM was developed in a close range environment, where the potential for block deformations inherent to the camera network geometry in aerial photogrammetry is not so apparent; as a result, the development of means for block control was not a priority. This is perhaps why software packages with a computer vision background often apply a rigid three-dimensional (3D) similarity transformation from an arbitrary reference system to the mapping system after SfM. It should be stressed that, although UAV blocks with high overlaps and large tie point multiplicity are more rigid compared to past aerial blocks with film cameras, control information should be included as a constraint in the bundle block adjustment (BBA), as non-linear model deformations are dealt with more effectively.

The accuracy and precision of photogrammetrically-generated DSMs depend on geometric and physical parameters, such as: the image scale, ground sampling density (GSD), stereo base-length to object distance ratio, camera network geometry (nadiral, cross, oblique strips), percentages of strip overlap (forward and sidelap), the accuracy and distribution of ground control points (GCPs), camera calibration, surface texture and albedo, lighting conditions, air refraction index, sensor quality (signal-to-noise ratio and dynamic range), image sharpness (no blur), as well as on processing: SfM, BBA, image matching, point cloud noise, and outlier removal algorithms. Measuring DSM quality is therefore a complex task, as the number of variables involved is very large, and no single experimental study can encompass all of the relevant aspects. Many studies addressed the accuracy of UAV-generated DSM in different environments; a valuable review of image-based surface reconstruction in geomorphometry can be found in Eltner et al. [20]. The theoretical elevation error in the normal and convergent cases of stereo restitution [21,22], though in principle a key element in block design, is seldom mentioned (Eltner et al. [20] being an exception). The influence of accuracy, precision, number and GCP distribution on DSM quality was extensively investigated by empirical studies [23,24,25] and Monte Carlo simulations [19] due to its relevance on survey costs and block deformation control. Attention was paid [26,27] to elevations errors due to residual errors in interior orientation parameters (the so-called dome effect), while limits of self-calibration effectiveness were discussed in Wu [28]. The differences between DSMs produced by different software packages [29,30] and the influence of processing parameter settings in SfM and dense matching on DSM accuracy [14] were systematically investigated. Methods for the co-registration of multi-temporal DSMs are presented in the context of morphology changes monitoring, erosion, and snow volume change estimates [4,31,32].

1.2. GNSS-Assisted Block Georeferencing and Control

The measurement of GCP coordinates is often today the most time-consuming on-site operation of an UAV photogrammetric survey. Artificial targets (even coded ones that are automatically recognised by the processing software) are necessary, as finding well-defined natural features of appropriate size to be used as GCP in pre-flight or post-flight surveys in outside urban areas might be difficult. GCPs are normally measured with GNSS surveys in real-time kinematic (RTK) mode, either with respect to a nearby base station or with differential corrections being sent by a Continuously Operating Reference Stations (CORS) Network (NRTK). Both techniques are accurate to within 1–2 cm in horizontal coordinates and 2–3 cm in elevation (i.e. to a level that often matches the GSD).

Dispensing with GCP altogether would be a big step forward for UAV photogrammetric surveys, not only for the time savings in the field and the office. The flexibility of the platform is not always matched by the ground operations. Sometimes, targets cannot be placed or measured easily (e.g., in areas with trees or dense bushes, or on mountain slopes) or safely (on a landslide’s body or on a glaciated area), or may interfere with ongoing activities (as in large open pit mines).

However, ground control is not the only option for block georeferencing and block deformation control. An alternative is available, if the drone on-board GNSS receiver is capable of delivering cm-level positioning accuracy. GNSS-assisted block orientation (AAT, Assisted Aerial Triangulation for short in the following) is a technique that has been used in aerial photogrammetry for many years [33,34]. In AAT, the coordinates of the camera stations at the exposure time are measured by an on-board GNSS receiver, and introduced as constraints in the BBA, or used to compute the transformation from the SfM arbitrary reference to the mapping system.

As RTK-enabled, dual-frequency receivers are being made available on an increasing number of commercial UAV platforms, it is important to investigate the error propagation of this alternative georeferencing technique. Recent simulations [35,36] suggest that block deformations are effectively kept in check by AAT. Previous empirical evidence [37,38,39,40] suggest that the traditional control through GCP is still the safest and most accurate way to georeference a UAV photogrammetric block. As far as DSM generation by AAT is concerned, an elevation root mean square error (RMSE) of about 9 cm was reported on 180 checkpoints [40], versus 3 cm by georeferencing with GCP.

In order to exploit the accuracy potential of AAT, system calibration is required. The receiver and the camera acquire data asynchronously, while the GNSS position refers to the antenna phase center, and rather than the camera center. Therefore, the exposure time of each image must be recorded to interpolate the trajectory, and the offset between the camera projection center and the antenna phase center [37] must be taken into account. Residual errors in camera interior orientation parameters are likely to affect the ground coordinates [41] rather than being partly absorbed by exterior orientation parameters, as happens in GCP-controlled nadiral blocks. An accurate camera calibration is therefore essential if no GCP are available. This problem is also shared by direct georeferencing [42,43], a technique coupling a GNSS receiver and an inertial measurement unit (IMU) to determine both camera position and attitude at shooting time. However, as far as DSM generation is concerned, direct georeferencing is unlikely to remove SfM from the UAV photogrammetry workflow, as tie points are needed in dense matching anyway.

Though cm-level accuracy can also be provided by single-frequency receivers, dual-frequency ones increase system reliability, as they solve the integer ambiguity faster [44]. RTK positioning allows image geotagging and on-site data quality checks. However, if GNSS processing skills are available, recording raw GNSS data is desirable in order to refine the camera station positions by post-processing.

1.3. Camera Calibration

Camera calibration in GCP-controlled UAV blocks is discussed by several papers [10,37,45]. While pre-calibration is recommended in principle, the stability of camera parameters is questionable: Cramer et al. [45] found less accurate results using laboratory calibration parameters, while Harwin et al. [10] found on-the-job self-calibration just as accurate. Self-calibration is likely to be the standard choice for practical reasons. An accurate and dense GCP network should be complemented by oblique imaging with rotary wings [10,27] and cross strips at different elevations [37] with fixed wings, in order to limit projective coupling between interior and exterior orientation parameters and DSM deformations.

On the other hand, as getting rid of GCP is the rationale for using AAT, optimal conditions for self-calibration will be hardly met in such cases, as only cross strips and oblique imaging would help. Pre-flight (on-site) calibration, if feasible, is probably the best option. However, in practice, measuring some GCPs is necessary, e.g. in order to get the same accuracy level of GCP-controlled blocks, or more importantly, to prevent biases from false ambiguity fixing or camera calibration errors. Notice that, even in blocks flown by UAV with direct georeferencing capability, tie point extraction and the use of at least one GCP are exploited [46]. Whenever high accuracy and a good confidence level on the outcome are sought, it is therefore just common sense to provide control and check information on the ground.

1.4. RTK versus NRTK

It is of practical interest to highlight whether RTK positioning from a local master station and NRTK positioning provide the same accuracy. NRTK offers savings and operational advantages, as it does not need a local GNSS master station; in practice, this is not advisable unless a good Global System for Mobile communication (GSM) network coverage is guaranteed. In fact, as the differential corrections from the CORS network are accessed through the internet via a GSM modem, poor GSM signal quality at the drone base station would impede the NRTK survey.

If the CORS network management allows for the generation of Virtual Rinex files, and if on-board GNSS raw data are recorded, the trajectory can also be recovered in case of a lack of GSM signal. However, as Virtual Rinex files are generated at a later time, there is no guarantee at the survey time that a high-quality navigation solution was achieved. Moreover, it should not be forgotten that CORS stations cannot predict accurate tropospheric corrections at altitudes that are very different from those of the network stations. Therefore, a bias can be expected for surveys in mountain areas. On the other hand, whenever access to the site by ground vehicles is difficult, if a NRTK service is available, only the lightweight drone needs to be carried to the survey site.

1.5. Paper Motivations and Objectives

Following a previous study by Benassi et al. [39], the objective of this paper is to investigate a further step of the photogrammetric workflow in UAV blocks oriented by AAT, i.e., the repeatability and the accuracy of DSM generation. In most cases, the main survey product of an UAV flight is the DSM. A DSM quality analysis cannot be restricted, as for block orientation, to a few checkpoints made by ideal-contrast targets manually collimated in multiple images. For a better spatial understanding of AAT error propagation on the DSM, a larger and well distributed point set is necessary.

As shown by James et al. [36], a valuable indication of the AAT error propagation spatial characteristics from image and camera station observations to tie point ground coordinates can be performed with Monte Carlo analysis, given the precision of both kind of observations. This is indeed the most appropriate technique for such an investigation. However, there are also reasons to collect experimental evidence from real data, as the complete and accurate modelling of the stochastic properties of GNSS observations is not so simple. It is well known to all geodesists that precision estimates of GNSS surveys output by data processing are optimistic; moreover, as projection center coordinates are output from a Kalman filter, they can hardly be considered truly independent observations in the BBA. On the other hand, only the diagonal entries of the variance-covariance matrix of the projection center coordinates are supplied by GNSS data processing, if any. As shown by Benassi et al. [39], the precision assigned to camera station coordinates in the Helmert transformation or in the BBA by three widely used software packages affects the elevation accuracy of checkpoints in different ways.

Tie points from SfM could be the natural candidates for studying the error spatial distribution, since besides block orientation errors, DSM points also inherit dense matching errors and DSM interpolation errors. Moreover, as noticed by James et al. [19], dense image matching does not contribute to the determination of camera exterior and interior parameters, so there is no advantage in using DSM points. However, for an empirical investigation on the repeatability of DSM generation with real blocks, as opposed to simulated ones, the tie points extracted at every repetition of the survey are different. Therefore, it is necessary to move further in the photogrammetric workflow and study the error propagation at the DSM level.

An analysis of the discrepancies among DSMs obtained by repeated surveys of the same, unchanged area helps to gauge the limits of a surveying technique in change detection. GNSS-Assisted Aerial Triangulation would be of great interest in monitoring changes in difficult areas. In periodic surveys with this technique, the stability of the reference system would depend on the GNSS kinematic survey of the camera network, rather than on GCPs positioned in stable areas (or surveyed at each campaign execution, if no suitable stable area can be found). Therefore, the experiment presented here was conceived through trying, as far as possible, to highlight the DSM elevation changes due to the different GNSS constellations that are experienced in the various flights. Hence, the same processing parameters were applied to all of the flights in the DSM generation workflow.

The paper is structured as follows: Section 2 describes the experiment organisation: test site, reference network, survey flights processing parameters, block control configurations, and DSM generation. Section 3 illustrates the main experiment results (with additional material provided in the Appendix). Section 4 discusses the results, and finally, Section 5 draws the conclusions.

2. Materials and Methods

2.1. Test Organisation

The experiment was organised as follows: eight flights were executed with a SenseFly eBee-RTK over the same test field and according to the same flight plan. Figure 1 shows, on the left-hand side, the data processing workflow for each flight. In four of the flights, the RTK corrections were sent from a local GNSS master station; in the remaining four, they were sent from a CORS Network. Block orientation was performed three times for each flight using a single set of tie points, but enforcing a different block control configuration: with GCPs only, with camera stations only, and with camera stations and one GCP. From each orientation parameter set, a point cloud was generated by dense matching and interpolated over a raster to produce a 10-cm resolution DSM. Therefore, the combination of the eight flights and three block control configurations generates 24 DSM.

Ground reference data were provided for an accuracy check (Section 2.3); the experiment evaluation methods are illustrated in Section 2.5.

2.2. Test Site Description

The test site is located in the southern part of the Parma University Campus (44°45′51′′N; 10°18′33′′E). It encloses an area of about (550 × 330) m² with buildings (from 6 m to 35 m high), roads, car parks, and meadows (see Figure 2). Trees are mostly located alongside the main road and in car parks.

2.3. Reference Network

Points to be used as ground control or as checkpoints for block orientation quality were established and measured, at least three times each, with dual-frequency geodetic receivers and different GNSS techniques: by Stop&Go and RTK from a local master station, and by NRTK with respect to the CORS Network to determine the local master station. On average, each point was occupied four times; in cases of less than four occupations, the mean position was taken. Otherwise, a trimmed mean position was computed. The average standard deviation turned out to be 7 mm for the horizontal coordinates, and 11 mm for the elevations. Finally, in order to assess DSM accuracy, more than 1600 points were measured with a GNSS NRTK survey over the area on hard surfaces (asphalt and concrete) and on meadows; the expected accuracy in elevation of these measurements, which was executed with the same GNSS receiver but under varying GNSS constellations and without repeated site occupation, can be assumed to be around 1–1.5 cm. The positions of the GCPs and of both types of checkpoints are shown in Figure 2.

2.4. Survey Flights

Flights were carried out with a fixed wing SenseFly eBee-RTK equipped with a compact camera Sony Cyber-Shot DSC-WX220 (18.2 Mpixel; focal length 4.5 mm). The camera was triggered by the drone’s autopilot and set exposure parameters automatically. The differential corrections were sent from the master station through the SenseFly Emotion2 flight control software and the ground radio modem. Geotags were stored in the image metadata as well as in the flight log.

Flight planning was designed with 85% forward overlap and 80% sidelap at ground level. Each block consisted of 12 strips (about 150 images), flown at about 90 m above ground level, with a GSD of about 2.3 cm. Flight lines were oriented along the main block side (roughly east–west), and images were acquired both ways along the passes.

The flight plan was repeated eight times in late summer 2016, between 12 a.m. and 6:15 p.m. In order to possibly enjoy similar satellite constellations and lighting conditions, RTK positioning through the local master and through the Network were alternated in consecutive flights. Image acquisition took on average 11 minutes per flight. The satellite constellation was always good, with PDOP (position dilution of precision) values below three in the first flight, and below two in all of the others. In the following, the flights are labelled as RTK1 to RTK4, and as NRTK1 to NRTK4.

2.5. Experiment Evaluation Methods

The experiment’s goals were to evaluate the repeatability and accuracy of DSM generated by UAV flights under different block control configurations. To this aim, the block orientation quality was first assessed on 14 signalised 3D checkpoints; then, the discrepancies among DSMs obtained under the same block control configuration were assessed, and then compared to those of different configurations. Finally, the DSM accuracy was evaluated comparing the elevations of a large set of checkpoints distributed over the area (see Figure 2) with the DSM elevation predicted at the same locations.

Block orientation accuracy was evaluated by computing the differences between the coordinates of the signalised checkpoints estimated by a BBA executed with a given control configuration, and those measured by GNSS. Mean values, standard deviations, and the root mean square error (RMSE) of the differences were computed. The degree of repeatability was inferred from the RMSE ranges.

DSM repeatability was measured by the discrepancy range among the DSM elevations produced from the four flights flown with the same RTK mode and adjusted with the same control configuration. DSM accuracy was evaluated by point-wise comparison of the DSM elevation with the elevation of a GNSS NRTK survey.

The software package PhotoScan (v 1.2.6 build 2834) by Agisoft LLC, St. Petersburg, Russia was used for block orientation and DSM generation (the reader is referred to PhotoScan manual v.1.4 [47] for details on the processing parameters mentioned in Figure 1 and in the following). The processing parameters were kept unchanged, as this paper’s focus is on the evaluation of block orientation accuracy and DSM uncertainty due to different GNSS datasets and imagery. A parameter optimisation on a case-by-case basis was therefore ruled out.

2.6. BBA Settings and Block Control Configurations

Three experiments were performed varying the block control configuration. In the first experiment, which is referred to as 12GCP in the following, only the coordinates of 12 well-distributed GCPs in the survey area were used (see Figure 2): no a priori information on camera stations was exploited. In the second experiment, which is referred to as RTK hereafter, only the camera positions from the navigation data encoded in the images were used: no a priori information on GCPs was exploited. In the last experiment (RTK+1GCP), besides the camera positions as in RTK, the coordinates of one GCP (in the middle of the block) were also used. This last configuration was tested, as it is well known [48] that unless GCPs are used, camera self-calibration might be inaccurate in nadiral blocks, with bias possibly affecting the ground coordinates. To distinguish between corrections being sent from a local master station and by a NRTK, the letter m (master) and n (network) was respectively added to RTK if necessary. Overall, 24 different BBA (three for each of the eight flights) were executed.

In all of the BBA, a standard deviation of one pixel was assigned to the tie points’ image coordinates, according to the mean reprojection error. The empirical precision found from repeated measurements of GCPs is in the order of 1 cm. However, a much lower standard deviation of 1 mm was assigned to the coordinates of the GCPs, as experience with PhotoScan processing shows that residuals on GCP coordinates would otherwise be much larger than expected. Finally, camera position standard deviations between 1–2 cm in horizontal coordinates, and between 3–5 cm in elevation were used, following the findings in Benassi et al. [39]. In all of the cases, a self-calibrating BBA was executed.

Table A1. Estimated camera calibration parameters in the bundle block adjustment (BBA).

Interior Orientation and Self-Calibration Parameters
Principal distance; principal point coordinates; radial distortion parameters K1, K2, and K3; tangential distortion parameters P1 and P2

in Appendix A reports the list of estimated parameters included in the camera model.

In all of the experiments, the remaining 14 known points were used as checkpoints to evaluate the RMSE of the ground coordinates.

2.7. DSM Generation

To measure the repeatability of DSM generation, the elevation range in each cell of the four DSMs based on the same block control configuration and RTK positioning mode was computed. The spatial distribution, the mean value, and the standard deviation of these ranges over the surveyed area were taken as variability indicators. Finally, though unrelated to block orientation configuration, cell-wise elevation ranges were also computed in areas of homogeneous ground surface characteristics, to find out whether different surfaces showed different variabilities.

To this aim, for each block and each block control configuration, a dense point cloud (average density 138 pts/m²) was generated with PhotoScan, setting the Quality parameter to Medium, and the Depth filtering parameter to Aggressive (Figure 1); no classification or filtering of the point cloud was performed with PhotoScan. This parameter setting was chosen to remove as much image-matching noise as possible from the DSM comparison, though at the expense of details, in order to focus on the effect of block orientation configuration only. No filtering or manual cleaning of the point cloud generated with PhotoScan was performed to remove vegetation, cars, and other disturbances at this stage. From each point cloud, a raster DSM with a cell size of 10 cm (about 4.3 GSD) was generated in PhotoScan, with the Interpolation processing parameter set to Enabled. The raster area with elevation data covers about 180.000 m². Though the flight plan was always the same, the actual ground coverage varied slightly from flight to flight, mainly due to wind and air turbulence, especially along strips.

For the comparisons, the 24 DSMs were grouped into six different sets, each coming from the processing of the four blocks flown with the same block control configuration and with the same RTK mode (from the local master or a NRTK).

Map algebra was used in QGIS to produce a raster with the elevation ranges among the four DSMs of each set. A minDSM raster was computed, where each pixel contained the minimum elevation value at that pixel among the four DSMs; likewise, a MAXDSM raster was generated that contained the maximum elevation at that pixel among the four-DSM set. Finally, the raster difference between MAXDSM and minDSM was computed, obtaining the pixel-wise elevation range for the four DSM.

The influence, if any, of the surface type on the DSM cell ranges was investigated. To this aim, three ground-level surface classes were selected: ploughed field, meadow, and asphalt (car parks and roads, concrete on pavements); in addition, another class of hard surfaces (roof) was considered. Areas belonging to each of the above-mentioned classes were manually contoured with polylines. Disturbances such as trees and bushes, cars, fences, and any other object with step edges or sharp discontinuities were left out; roof edges as well as heating and air conditioning equipment, pipes, or skylights were also excluded from the roof control areas. The rationale for selecting only smooth or gently varying surfaces is that it is well known [16] that image matching is not accurate along sharp discontinuities. Therefore, large discrepancies in elevation might originate along edges, due to perspective differences of vertical or sub-vertical surfaces in the images of different flights.

The size of the disturbance-free areas extracted for each surface class, shown in

Table 1. Size in square meters of the surface classes considered in the evaluation of DSM variability.

Surface Type	Asphalt	Roof	Meadow	Ploughed Field	Total
size (m2)	12883	8626	28504	17838	67851

, amount to about 37% of the total surveyed area. Figure 3 shows the areas contoured for the repeatability analysis, marked with different colours, according to surface class.

3. Results

3.1. Empirical Accuracy Assessment of Different Block Control Configurations

The graphs of Figure 4 depict the mean ground accuracy measured by the RMSE at the 14 checkpoints for each of the eight flights. The accuracy in the horizontal coordinates (RMSE XY) was provided by the RMSE of the horizontal component of the error vector, while RMSE Z refers to the vertical component of the same error vector. On the left-hand side, the bars refer to flights with GNSS positions computed from the local master station (from RTK1 to RTK4); on the right-hand side, the bars refer to positions computed with respect to a CORS Network (from NRTK1 to NRTK4).

Table 2. Best and worst RMSE computed on 14 checkpoints for the six sets of flights with the same block control configuration and real-time kinematic (RTK) mode.

	RMSE XY (m)
	12GCPm1	RTKm1	RTKm+1GCP1	12GCPn2	RTKn2	RTKn+1GCP2
Min	0.012	0.021	0.021	0.015	0.021	0.017
Max	0.020	0.025	0.029	0.016	0.042	0.032
	RMSE Z (m)
	12GCPm1	RTKm1	RTKm+1GCP1	12GCPn2	RTKn2	RTKn+1GCP2
Min	0.017	0.019	0.018	0.021	0.067	0.029
Max	0.024	0.095	0.039	0.033	0.126	0.047

¹ Flights RTK1 to RTK4; ² Flights NRTK1 to NRTK4

summarises the RMSE ranges in XY and Z on the 14 checkpoints for the different block control configurations.

The 12GCP configuration provided an average accuracy in the horizontal and vertical coordinates of 1.5 cm and 2.3 cm, respectively (worst case: 2.0 cm and 3.3 cm, respectively), while the corresponding figures for RTK were 2.4 cm and 4.6 cm from a local master RTK mode, and 3.0 cm and 9.5 cm for a NRTK mode. Finally, the RTK+1GCP configuration showed horizontal and vertical coordinates accuracy of 2.5 cm and 3.2 cm, respectively (worst case: 3.2 cm and 4.7 cm, respectively).

Though the results are obviously purely empirical, it is interesting to notice the behaviour of the different configurations when expressing the accuracy range (max RMSE—min RMSE) as a percentage of the worst case (max RMSE). In both the horizontal (XY) and vertical (Z) coordinates, block control with GCPs (12GCP) was the most stable, with 39% and 49%, respectively, in XY and Z; in contrast, when using AAT, only far higher variations are registered: 50% and 85%, respectively, in XY and Z. However, adding one GCP (RTK+1GCP) brings the relative variation back to 45% and 61%, respectively, in XY and Z.

As stated before, the statistics in Figure 4 and Table 2 refer to the 14 checkpoints available. However, the RMSE for the RTK and RTK+1GCP configurations were also computed using all of the available checkpoints: 26 and 25, respectively. The changes to the RMSE values shown in Figure 4 are in both cases negligible, as they amount to 1–2 mm in most cases, and to 5 mm in one case only.

3.2. DSM Repeatability and Accuracy

As mentioned in Section 2.7, the 24 DSM were grouped into six homogeneous sets for the repeatability analysis. Figure 5 shows the means and standard deviations in meters of the cell-wise DSM elevation ranges for each set, both overall and according to each surface class (see Table 1), which were computed in the check areas shown in Figure 3.

The mean value of the discrepancy ranges, irrespective of the surface type, was almost the same, about 6 cm, for the 12GCP and RTK+1GCP block configurations. In the RTK configuration, the mean was 12.5 cm and 9.5 cm for the local master and the Network RTK modes, respectively. Differences between surface type means were almost negligible, a few mm at most; the average range as a function of the block control configuration fairly reproduced the trend seen for the RMSE in Z for the checkpoints, though obviously with larger values.

The ranges standard deviations varied between 2–3 cm, irrespective of the block control configuration and the surface type, with the partial exception of the ploughed fields class, where values were larger than in the other classes and, surprisingly, the 12GCP cases performed worst.

The ranges spatial distribution for each configuration over the whole survey area are shown in Appendix B (see Figure A1, Figure A2, Figure A3, Figure A4, Figure A5 and Figure A6) as colour-coded maps of the pixel-wise ranges of elevations of DSM obtained from four blocks that were adjusted under the same control configurations and the same RTK mode. Areas with ranges exceeding 1 m, which are blanked in the figures, correspond to trees and bushes, (some) building roofs sections where matching failed, and car park plots that were empty or occupied in at least one of the flights.

Figure 6 shows two pairs of graphs, each with four terrain profiles drawn on the DSM. The former graph pair refers to a meadow close to the western border of the block, which originated from the four flights with the local RTK mode. The latter refers to a car park located in the middle of the test field, and originated from the four flights with the local NRTK mode. In each pair, the first graph depicts the profiles on the DSM derived from the RTK configuration (a1, b1), while the second one refers to the DSM derived from the RTK+1GCP configuration (a2, b2).

Figure 7 depicts the mean values (a) and standard deviations (b) of the DSM error for each block control configuration, which were computed on about 1600 GNSS points measured on roads, car parks, and meadows.

The mean error varied in a range smaller than 4 cm for all of the 12GCP cases, and in a range of about 9 cm with the RTK+1GCP configurations. With the RTK configuration, the mean error varied in a far larger range: from +1 cm to −14 cm; all of the large errors were negative (i.e. DSM elevations were lower than they should have been). The standard deviations were fairly homogeneous, and varied between 4.0–5.7 cm.

4. Discussion

4.1. Empirical Accuracy Assessment of AAT Compared to Traditional Ground Control

As far as the accuracy of horizontal coordinates is concerned, controlling block deformations with GCP provided the best results, which were on average 0.7 GSD. Dispensing entirely with GCPs (RTK) brings the average accuracy to 1.2 GSD, which is a marked decrease relative to 12GCP, though not much in absolute terms; adding at least one GCP does not bring on average any real improvement (1.1 GSD) with respect to AAT only. However, the worst result might be well above the average, if no GCP are used.

Likewise, as far as the vertical accuracy is concerned, using GCP provided the best results, which were on average 1.0 GSD. In the RTK configuration, to the contrary, the average accuracy was 3.1 GSD, which is a dramatic decrease in relative as well as in absolute terms. The reason is a bias that was most likely introduced by the estimation of interior orientation parameters in the self-calibrating BBA (see Section 1.3). Indeed, the mean error in elevation is correlated in excess of 95% with the RMSE Z. As soon as some information on ground elevation is enforced in the BBA, as in RTK+1GCP, the bias is almost entirely or at least greatly reduced, with an RMSE Z of 1.3 GSD; the residual error is still 30% worse than using GCP only, but is acceptable in most cases.

4.2. RTK versus NRTK

It is of practical interest to find out whether, ceteris paribus, the camera station and the ground coordinate accuracies depend on using a local master or a CORS Network to relay the differential corrections. It has already been clarified that a local master is certainly a safer and better choice whenever possible. From an empirical standpoint, both the AAT-only and the AAT with one GCP should be considered, though the discussion above shows that AAT-only is not reliable, especially in elevation. However, as the CORS Network closest station is located just 6 km from the test site, this is too small a distance to really test whether the accuracy levels of the Network and of the local master are critically different. Restricting the comparison to the AAT with one GCP, only a small difference can be noticed between the average RMSE XY of the RTK flights in RTK+1GCP configuration (2.4 cm) against the NRTK flights in the same configuration (2.7 cm); a slightly larger difference is found in elevation, with RMSE Z values of 2.7 cm and 3.7 cm, respectively.

Keeping the size of the dataset and the limited distance from the closest CORS station in mind, no statistically significant conclusions can yet be drawn. Therefore, more tests are needed in order to clarify this point.

4.3. DSM Repeatability and Accuracy

With traditional ground control (12GCP), the ranges of DSM elevation discrepancies within each DSM cell look rather stable over the area, with an average of about 6 cm, and a standard deviation of 3 cm. In gently varying and smooth surfaces, this is true almost independently of the surface characteristics, provided there is some texture. However, if texture is not ideal (though not necessarily uniform), there might be discrepancies in the order of several decimeters, which is all due to dense matching failures. In RTK configurations, the range of variability is higher, in the order of 10–12 cm, but still with a standard deviation of about 3 cm. Therefore, as found for checkpoints on targets, this larger range is unlikely to depend on a larger noise in the GNSS observations; it is rather due to an elevation bias introduced by the self-calibration. Using at least one GCP (RTK+1GCP) removes most of the bias, and though less precise than GCPs, GNSS observations of the camera stations spread the control over the block fairly homogeneously. This empirical result is in agreement with the simulations of Santise [35] and James et al. [36], where georeferencing with GCPs or AAT was found to yield comparable ground point vertical precisions.

The well-known rule of photogrammetry, stating that plotting is safe only inside the polygon enclosing the GCPs, somehow applies to the projection centers as well, and is reinforced to some extent by the larger overlaps adopted in UAV photogrammetry. Indeed, the only noticeable exception to the irrelevance of surface type in Figure 6 is the standard deviation in ploughed fields for 12GCP, which is slightly larger than in other cases. As can be seen from Figure 3, the check areas of the ploughed field class all fall outside the polygon enclosing the GCP. With AAT, to the contrary, the block control from the projection centers is still effective in the area.

From Figure A1, Figure A2, Figure A3, Figure A4, Figure A5 and Figure A6 in the Appendix, it can be seen that with 12GCP and RTK+1GCP, the discrepancies in elevations are fairly homogeneous inside the block, apart from along step edges or steep slopes, or where texture is not ideal for image matching. Inconsistencies up to 30 cm can be found along the block borders and at the strip ends in the 12GCP and RTK cases. In the former case, the likely reason is poor control outside the GCP polygon, in the latter, the likely reason is one of the flights with a bias in the principal distance.

Figure A2, which refers to the RTK configuration for the RTK1 to RTK4 flights, shows that in the right-hand side of the block, the discrepancies are higher than those in the left-hand side. Discrepancies in Figure A5, which refer also to the RTK configuration for the NRTK1 to NRTK4 flights, are on the other hand more homogenous over the area. This suggest that the bias in elevation due to estimation errors in the principal distance is not the only reason for the discrepancies in the elevation ranges.

In the first pair of terrain profiles in Figure 6, the RTK2 DSM is offset with respect to the other three in the RTK case (a1). The coherence of the four DSMs is improved to some extent in the RTK+1GCP case (a2). A better profile coherence is apparent with the four NRTK DSM of the second graph pair (b1, b2), where the profile is close to the location of the GCP constrained in the RTK+1GCP case. This might suggest that the location of the GCPs is relevant in order to enforce to remove the elevation bias. Previous investigations in Benassi [39] hinted that this should not be the case, though this is worth testing more. This is to some extent confirmed by a comparison of Figure A3 and Figure A6: at the block corners, the improvement in DSM coherence from the RTK to the RTK+1GCP configuration is different for the RTK and NRTK flights, despite the GCP enforced being the same.

The irrelevance of surface type on the DSM repeatability might be surprising. However, as the depth map was strongly filtered, some smoothness of the DSMs could be expected. In addition, the meadows at the end of a dry summer did not yet recover from a recent cut, so the grass was not well developed.

The assessment of the DSM accuracy on roads, car parks, and meadows (Figure 7) shows that the average RMSE is very similar when using a traditional ground control or AAT with at least one GCP: 5.2 cm in the former case, and 5.6 cm in the latter. In other words, the two control configurations have a very similar accuracy potential under the experiment settings (about two GSD). Overall, the DSM RMSE with the RTK configuration is 8.5 cm. These results compare well with those obtained by Hugenholtz et al. [40] with an eBee RTK on 180 checkpoints. From a flight in the RTK configuration, they found a 5 cm-resolution DSM in non-vegetated areas, with a RMSE in elevation of 8.9 cm, against 3.0 cm with traditional georeferencing with GCPs. They therefore concluded that AAT is not appropriate to projects requiring high vertical accuracy. Indeed, if self-calibration is employed with camera stations only, relatively large height errors can be expected with high probability, due to bias in the estimation of the camera principal distance. However, the results found with the RTK+1GCP configuration mitigate such conclusions to some extent.

5. Conclusions

An experimental investigation on the repeatability of DSM generation from UAVs was performed by repeating the same flight plan eight times from about 90 m aboveground, with a GSD of 2 cm. Using camera station positions measured at exposure time by a senseFly eBee RTK, the photogrammetric blocks were successfully oriented, without any GCPs, with an accuracy of the horizontal coordinates that was better than the GSD in most cases, and within 1 cm of the accuracy obtained using GCPs. However, dispensing entirely with GCPs makes block orientation unreliable, as significant biases, especially in elevations, can arise. Indeed, in three of the eight repetitions, an elevation bias of about 10 cm and more was registered. On the other hand, in all of the tests that were carried out, adding at least one GCP to the camera stations successfully removed most of the bias, bringing the RMSE in elevation to about 3 cm. As checkpoints must in any case be provided for an independent quality check, and using a local master is recommended, this does not appear to be a serious limitation of AAT.

An average repeatability of 6 cm in DSM generation with different GNSS datasets and changing lighting conditions (from noon to late afternoon) was found for the RTK+1GCP configuration, in areas suited to image matching (smooth, textured surfaces); the discrepancy ranges were also quite uniform over the block, with a standard deviation of only 3 cm. Exactly the same values were obtained using GCPs, almost irrespective of the surface type (hard surfaces or meadows); the average range was instead larger (by 5–7 cm) if only camera stations (RTK configuration) were used.

As far as the quality of the eight 10 cm-resolution DSMs generated by dense matching is concerned, using either GCPs or all of the camera stations and one GCP yielded similar results. Indeed, on roads, car parks, and meadows the mean RMSE in elevation turned out to be very close, in the order of 5.4 cm, with both types of block control configurations.

In all eight flights, using at least one GCP as well as AAT yielded good results. However, further investigations are needed to find out whether the location of the GCP in the survey area can be critical (in our test, the GCP was located in the middle of the area). It is also of special importance for high mountain surveys to find out whether an on-site small calibration block can be used when the survey area is not accessible. Moreover, though the benefits of a very dense control from the camera stations are clearly apparent from the simulations in Santise and James et al. [35,36], more experience should be gained about the effect of GNSS systematic errors and residual inner orientation errors.

It is not yet clear whether AAT will supersede GCPs to become the standard referencing technique for UAV blocks. Indeed, as the results shown in this test confirm, the incentives towards using it are strong not only for improving survey efficiency, but also because the accuracy loss compared with the traditional case seems negligible in horizontal coordinates and, with a small additional effort, also very limited in elevation. In any case, AAT will certainly contribute to further enlarging the domain of UAV photogrammetric surveys.