Evaluating SAR Radiometric Terrain Correction Products: Analysis-Ready Data for Users

Abstract

Operational applications for Synthetic Aperture Radar (SAR) are under development around the world, driven by the free-and-open access of SAR C-band observations that Sentinel-1 of Copernicus has provided since 2014. Radiometric Terrain Correction (RTC) data are key entry-level products for multiple applications ranging from ecosystem to hazard monitoring. Various open-source software packages exist to create RTC products from Single Look Complex (SLC) or Ground Range Detected (GRD) level SAR data, including the Interferometric SAR Computing Environment (ISCE), and the Sentinel-1 Toolbox from the European Space Agency (SNAP 8). Despite the growing availability of RTC software solutions, little work has been performed to identify differences between RTC products generated using different software packages. This work evaluates several Sentinel-1 RTC products and two other Sentinel-1 Analysis Ready Data (ARD) to address the following questions: (1) Which software provides the most accurate RTC product? and (2) how appropriate for analysis are other non-RTC products that are readily available? The RTCs are produced with GAMMA, ISCE-2, and SNAP 8. The other two ARD products evaluated consisted of an angular-based radiometric slope correction produced in Google Earth Engine (GEE) following Vollrath et al., and the Sentinel-1 GRD product. Products are evaluated across 10 sites in a single image approach for (1) radiometric calibration, (2) geometric corrections, and for (3) geolocation quality. In addition, time-series stacks over two sites representing varied terrain and ecosystems are evaluated. The GAMMA-derived RTC product implemented by the Alaska Satellite Facility (ASF) is used as a reference for some of the time-series metrics. The results provide direct guidance and recommendations about the quality of the RTC and ARD products obtained from open source methods. The results indicate that it is not recommended to use the GRD product with no radiometric or geometric corrections for any applications given low performance in multiple metrics. The radiometric calibration and geometric corrections have overall good performance for all open-source solutions, only the non-RTC products (Vollrath et al. and GRD) portray some significant variances in steep terrain. The geolocation assessment indicated that the GRD product has the most significant displacement errors, followed by SNAP 8 with Digital Elevation Model (DEM) matching, and ISCE-2. RTCs created without DEM-matching performed better for both GAMMA and SNAP 8. The time-series results indicate that SNAP 8 products align more closely to GAMMA products than other open-source software in terms of radiometric and geometric quality. This understanding of software performance for SAR image processing is key to designing the affordable and scalable solutions needed for the operational application of SAR Sentinel-1 data.

1. Background

Synthetic aperture radar (SAR) is an active microwave imaging method. The SAR system emits electromagnetic pulses and then measures the resulting backscatter received at the sensor [1]. The backscatter coefficient and scattering phase collected by the sensor provide information valuable for multiple applications [1,2,3]. SAR is largely unaffected by light and atmospheric conditions and is often used as a complement or an alternative to optical data for environmental monitoring, including land cover mapping [1,4,5,6].

SAR is a critical satellite data source within remote sensing applied science programs, such as SERVIR [7]. SERVIR, which means “to serve” in Spanish and French and stands for this meaning, is a joint initiative between the U.S. Agency for International Development (USAID) and the National Aeronautics and Space Administration (NASA) that seeks to enhance the capacity of stakeholders in Latin America, Asia, and Africa in using satellite remote sensing. SERVIR implements multiple applied science services globally, including applications in land cover monitoring, water mapping, agriculture, weather, and climate. SERVIR is at the forefront of remote sensing applications and has made significant contributions to lower the barrier to access, process, and apply SAR for ecosystem services and water mapping [5,8]. A takeaway from SERVIR’s experience with SAR is the need for a high-quality SAR polarimetric product to use in operational systems that provide satellite-derived information to end users.

An inherent challenge to working with SAR data is the geometric distortions caused by the flight path of the sensor over the target. The relationship between the sensor’s look angle and the slope and elevation of the terrain below particularly influence the resulting backscatter recorded in the SAR image [9,10]. In an effort to minimize the impacts of these distortions, multiple correction methods have been developed, including geometric terrain correction (GTC) and radiometric terrain correction (RTC) [10,11,12]. Radiometric terrain correction involves removing geometry-dependent radiometric distortions and normalizing the measured backscatter with respect to terrain slope [10,12].

Processing SAR data involves several challenges. First, it is computationally demanding. Second, in order to be useful as an input for decision-making tools, such as those built by SERVIR, products need to be provided as analysis-ready data (ARD) [3,5]. Currently for SAR data, the normalized radar backscatter ARD specifications provided by the Committee on Earth Observation Satellites (CEOS) require radiometric and terrain-corrected backscatter values in $\gamma$₀ for most land applications [13].

RTCs can be derived from a variety of software solutions, ranging from open-source software, like Sentinel Application Platform (SNAP) [14], to proprietary software such as GAMMA [15]. The Alaska Satellite Facility (ASF) Hybrid Pluggable Processing Pipeline (HyP3) [16] tool provides on-demand RTC products processed with GAMMA [15]. However, the provision of RTCs is not operational at the time of writing. This means that RTCs are not automatically generated when a new image is acquired and provided as a data product. It is up to the user to generate it. In order to take advantage of cloud processing and reduce computation impact on the user, SAR correction workflows have been developed to operate within Google Earth Engine (GEE) [17], such as Vollrath et al. (2020) [18]. In addition, other open-source options have been developed by the SAR expert user community, for example, the RTC module within the Interferometric SAR (InSAR) Scientific Computing Environment (ISCE-2).

Previous analyses [19] show that GAMMA produces optimal RTCs compared to the European Space Agency’s (ESA) open-source SNAP 6 platform, which demonstrated some displacement issues. In addition, SNAP 6 was computationally more demanding than GAMMA [19,20]. Despite the growing availability of RTC and other ARD software solutions, little work has been performed to identify differences between RTC products generated using different software.

This study expands on previous work by incorporating multiple metrics into one SAR ARD evaluation framework, evaluating a greater range of more recent products, and expanding the geographic area covered. Specifically, seven ARD products, some produced with open-source software and workflows, were compared by evaluating radiometric, geometric, and relative geolocation accuracy over the Amazon, Lower Mekong, and Hindu Kush Himalaya regions. Hence, this study informs users about the limitations of specific open-source software and workflows for generating quality SAR ARD products.

2. Materials and Methods

2.1. Areas of Interest

For the single scene analysis (Section 2.3), ten areas of interest (AOIs) were selected across the globe (see Figure 1). These locations represent a variety of terrain, including mountainous regions, tropical forests, and floodplains. Additionally, these AOIs overlap with current or projected SERVIR projects incorporating SAR ARD products. Two sites were selected as case studies for the time-series analysis (Section 2.4). Nepal (NP) was selected to provide information about the performance of ARD products over the complex topography of the Himalayan mountains. Madre de Dios (MDD) was selected to provide a baseline of performance over a mostly flat tropical forest.

2.2. Datasets

2.2.1. SAR

Sentinel-1 A and B scenes as level 1 Ground Range Detected (GRD) interferometric wide swath (IW) products from ESA were acquired through GEE and ASF [16] over each AOI for 2019. GRDs provide magnitude information at nominally 10-meter spatial resolution, are multilooked with a 5 × 1 window, and projected from slant to ground range using an Earth ellipsoid model [12,21]. For the time series, 29 ascending and 30 descending images were acquired over MDD. 28 ascending and 29 descending images were acquired over NP.

2.2.2. Land Cover

In total, four different land cover datasets were used to mask evergreen forests and other critical land cover classes. Local and regional datasets were selected over a global dataset. SERVIR-Mekong’s Regional Land Cover Management System (RLCMS [22]) dataset was used to identify evergreen forests for CT, CG, and NM. Since the 2019 classification for evergreen was not yet available, areas that showed a probability greater than 80% for evergreen were defined as evergreen forests [22]. The SERVIR-Hindu Kush Himalaya National Land Cover Database [23], with a mapping methodology based on the RLCMS, was used to identify evergreen forests for NP. As needle-leaved forest had been removed as a category for 2019, the overlap between needle-leaved forests in 2010 and forests in 2019 was defined as evergreen forest. To identify forests in BG, a 2017 land cover map with output land cover classes based on FAO’s Land Cover Classification System (LCCS), was used [24]. For RON, CR, MDD, and MJ, Collection 2 Version 2 annual land cover and land use maps from MapBiomas, a collaborative mapping initiative started by the Greenhouse Gas Emissions Estimation System (SEEG) of the Climate Observatory, were used to identify evergreen forest [25].

2.2.3. Elevation

The slope of the terrain was derived from the Shuttle Radar Topography Mission (SRTM) DEM. This version of the SRTM collected in 2000 at 30 m resolution [26], was gap-filled by ASF. The same DEM was used to create all RTC products evaluated.

2.2.4. High-Resolution Optical Imagery

High-resolution data from the Maxar Standard Satellite Imagery products were used to assess the geolocation accuracy of the multiple RTC products evaluated. These datasets were accessed using the NextView license from the Global Enhanced GEOINT Delivery (G-EGD) system. Scenes were selected based on overlap with the AOIs, having less than 10 percent cloud cover, and temporal proximity to the Sentinel-1 acquisition date. The Maxar imagery acquisition dates ranged from January 2019 to February 2022. See

Table 1. List of optical imagery and corresponding sensor resolution and product accuracy. The images were accessed through the ©Maxar Technologies ImageConnect ArGIS add-on. Sensors WorldView 1–3 and GeoEye-1 are ©Maxar formerly known as DigitalGlobe, Inc. (2022), provided by European Space Imaging.

Sensor	Product	Spatial Resolution	Approximate Horizontal Accuracy
Multiple	Mosaic	30 cm–15 m [28]	5 or 10 m
WorldView-1	Strip	50 cm [29]	<8.4 m
WorldView-2	Strip and Mosaic	46 cm [30]	<8.4 m
WorldView-3	Visible and Near-Infrared Strip	31 cm [31]	<8.4 m
GeoEye-1	Strip and Mosaic	41 cm [32]	<3 m

for a description of each Maxar image product used. More details on reference imagery can be found in the geolocation results dataset [27].

2.2.5. Generation of ARD Products

While each software and workflow employs slightly different algorithms that impact performance, the main steps to generate RTC products were kept consistent, and in the same sequence across all the RTC products following methods by Small (2011) [10] and shown in Figure 2. First, the orbit file provided the updated metadata from the original SAR file including accurate positioning of the SAR image [33]. Thermal noise, defined as instrument noise or the residual backscatter signal received by the sensor not originating from the target, was subsequently removed [5]. The image was then calibrated from digital numbers (DN) to $\beta$₀ units. GTC was applied to the image using a DEM. Next, a shadow mask was created and used to account for the areas affected by geometric distortions. The $\beta$₀ was converted to terrain corrected $\gamma$₀ units [10] using the projection and incident angles as modified by the DEM. Finally, the image was geocoded. Since ISCE v2.3.3 does not support geocoding, a standalone script relying on GDAL was used per [34]. The RTCs derived from SNAP and GAMMA include the variations with and without DEM-matching (see Table 1). All ARD products were generated at 10 mts spatial resolution. It should be noted that no speckle filtering was applied to the RTC products generated to preserve spatial resolution as described in [35]. It is also important to mention that speckle is inherent in SAR acquisitions [12,35].

Seven Sentinel-1 ARD products were evaluated (see

Table 2. List of ARD products evaluated in this analysis across all 10 AOIs.

ARD Product	Software Used	DEM-matching	Reference
RTC	GAMMA	N ’	[15,16]
RTC	GAMMA *	Y ”	[15,16]
RTC	ISCE-2	N ”	[3]
RTC	SNAP 8	N ’	[14]
RTC	SNAP 8 *	Y ”	[14]
Vollrath et al.	GEE	n/a ”	[18]
GRD	GEE	n/a ”	[21]

* Indicates DEM-matching in the table and figure captions, (’) Used only in single scene analysis (Section 2.3), (”) Used in both single scene and time-series analyses (Section 2.4), n/a: not applicable.

), including a GRD product available in GEE, processed using the Sentinel-1 Toolbox before ingestion [36]. Another ARD product was produced using a GEE workflow following methods by Vollrath et al. (2020) [18], which uses an angular-based radiometric slope correction, and applies additional border noise correction, and radiometric terrain normalization [18,37]. Five RTC datasets were produced for comparison. Two of the RTCs were produced using the commercial standard GAMMA software [15,38], as processed by ASF in their HyP3 platform [16], both with and without DEM-matching. Two RTCs were created with and without DEM-matching in SNAP 8, an open-source coding language developed by the European Space Agency (ESA) [14]. The DEM matching refers to co-registering the SAR data to the DEM, compared to using dead reckoning based on orbit files, the latter referred to as no DEM matching. An asterisk “*” indicates DEM-matching in table and figure captions. And one RTC product was created with the Interferometric SAR (InSAR) Scientific Computing Environment (ISCE-2) v2.3.3, an open-source, Python-based, modular software package originally developed to create interferograms from multiple sensors, using the rtcApp module [3].

Figure 3 lists and organizes all the metrics evaluated in this work, each of the metrics is described in detail in the following Section 2.3 and Section 2.4.

2.3. Single Scene Analysis

2.3.1. Radiometric Evaluation

Radiometric correction removes the influence of the terrain to appropriately represent the true surface characteristics such as structure and moisture [12,39]. To evaluate the radiometric calibration, the average values of backscatter over forested areas were compared with the expected average values. It is expected that for C-band, rainforests have −12.5 dB for VH polarization and −6.5 dB for VV polarization [20,40,41]. First, forested areas were selected using the appropriate land cover dataset (Section 2.2.2). Then, the average dB was calculated over forested areas in GEE for the first scene from 2019 for each AOI.

2.3.2. Topographic Normalization

To conduct the topographic normalization, the local incident angle values of the products were converted from radians to degrees and then added as a band to each product. In order to extract both the backscatter and incidence angle values from each scene, 3500 random points were generated exclusively over forested areas, over each AOI. All backscatter values were converted from power to decibel (dB) units (Equation (1)) for visualization according to best practices laid out by ASF and CEOS [42,43].

$${\gamma }^{o}dB=10\times {log}_{10}\left(DN\right)$$

(1)

where DN = Digital number representing power scale. To assess the topographic normalization of each product, the Coefficient of Variation (CV) was calculated for both VV and VH polarizations for forested areas. Backscatter over forested areas is generally more stable compared to other land cover classes [10], hence low variance should be expected in a properly corrected product. The CV (Equation (2)) quantifies the variation in backscatter observed compared to the mean backscatter and acts as an indicator of insufficiently corrected pixels. In addition, SAR backscatter is strongly influenced by the incidence angle and a RTC should account for this dependence and correct for it as much as possible [44]. Backscatter values over evergreen forest were compared to the local incidence angle at the pixel level by performing a linear regression using the ee.Reducer.linearFit() function in GEE. The optimal slope of the linear regression should be 0, to show no dependence on local incidence angle.

$$CV=(std/mean)\times 100$$

(2)

2.3.3. Geometric Evaluation

Following Small’s metrics [10], we assessed backscatter values across slopes of the terrain, perpendicular to the view angle and with similar vegetation on both sides of the slope. Evergreen forests were selected as the land cover class that complies with this requirement. The objective is to assess the correction of geometric distortions, and for this, the backscatter values of forested areas were compared by incidence angle and slope ranges. Incidence angle, slope, and backscatter values in the VV and VH polarizations were extracted from a random sample of 3500 evergreen forest points in each AOI.

2.3.4. Geolocation Evaluation

Accurate geolocation is a prerequisite to radar processing in order to allow for comparison through time [45]. According to the minimum requirements specified by CEOS for a SAR ARD product, subpixel accuracy is less than or equal to 0.2-pixel radial root mean square error (rRMSE) [35,45]. At each AOI, Maxar reference imagery was accessed through the ArcGIS web map viewer using the EnhancedView Web-Hosting Service more details on reference imagery can be found in our results dataset [27].

Corner reflectors, identified as landmarks were chosen by visually interpreting spectral differences and spatial patterns across VV and VH polarizations for each product and the corresponding Maxar image product. Consistent landmarks that were easily identifiable in the Maxar imagery as well as each RTC, Vollrath et al., and GRD images were selected for comparison. These rather permanent landmarks included human-made structures such as road intersections, the ends of bridges, buildings, and irrigation canals, as well as natural features such as small water bodies, bends in waterways, river islands, and patches of bare land. Around 10–12 broadly distributed points of these different landmarks were collected across each AOI. The points were then projected to the 1984 Web Mercator Auxiliary Sphere to match that of the Maxar-derived reference points. Finally, the geolocation distance offset was calculated in ArcGIS 7 [46] for each of the landmarks as seen in each ARD product and high-resolution reference image.

2.4. Time-Series Analysis

The time-series analysis included a radiometric and geometric evaluation over a subset of AOIs (NP and MDD) and a subset of ARD products (GAMMA and SNAP 8 without DEM-matching, ISCE-2, Vollrath et al., and GRD). The selection of ARD products to assess in this section is based on the lower performance obtained from DEM-matching products in the single scene analysis. Hence, selecting only products without DEM matching. The number of ascending and descending scenes was nearly equal over both AOIs for 2019, enabling both to be included in the same metric [20]. The same land cover maps described in Section 2.2.2) were used to define the classes of interest, forest, and grassland [20].

2.4.1. Radiometric Evaluation

For the radiometric analysis of the time series, the CoV was calculated on a monthly basis in GEE. Backscatter was extracted over 5000 randomly generated points stratified by land cover and CoV was calculated (Equation (2)). The slope from the gap-filled DEM was reclassified into two categories, steep and flat, defined as above 20% and below 12% respectively [20]. Then, the backscatter was extracted at 5000 random points for each of the slope classes and CoV was calculated (Equation (2)).

2.4.2. Geometric Evaluation

For the geometric analysis of the time series, the RMSE was calculated between the ARD products. For RMSE, GAMMA is treated as the “truth” dataset, since it is a known optimal RTC product and is currently implemented by the Alaska Satellite Facility (ASF) in their on-demand provision of RTCs [16,39,41]. The other four datasets are considered observation, or prediction, datasets. The 5000 random points generated in GEE for both flat and steep slopes over the evergreen land cover class were used to extract the backscatter value for each image within the year 2019. Then, the annual RMSE was calculated using Equation (3) and Pearson’s R using Equation (4) in Python.

$$RMSE=\sqrt{\frac{{\sum }_i^N{(Predicte{d}_i-Actua{l}_i)}^2}{N}}$$

(3)

$$r=\frac{\sum (x-m_x)(y-m_y)}{\sqrt{\sum {(x-m_x)}^2\sum {(y-m_y)}^2}}$$

(4)

where $m_x$ is the mean of the vector x and $m_y$ is the mean of the vector y.

3. Results

3.1. Single Scene

3.1.1. Radiometric Calibration

The average backscatter (dB) over forested areas for the seven ARD products is shown in Figure 4. The horizontal red line indicates the expected average backscatter value for rainforests. The primary rainforest study areas RON, MDD, and MJ, are outlined in black to highlight whether the backscatter values are close to what was expected in Section 2.3.1. The other AOIs have greater potential for differences in the performance of each product due to the more varied landscape and terrain.

The GRD product is the outlier, with greater than expected values for all three of the baseline AOIs, with a difference of about −1 dB for both polarizations. Looking at the VV polarization, the rest of the ARD products are within 0.2 dB of the expected average backscatter. The VH polarization values for the three rainforest AOIs are also close to the expected backscatter. With the exception of ISCE-2 values over CR, where both polarizations resulted in higher-than-expected values by an order of magnitude of 1.5 dB, all the RTC and Vollrath et al. products provided similar performance in this metric.

3.1.2. Topographic Normalization

The topographic normalization results are shown in Figure 5 and . The slope of the linear regression between the backscatter of the evergreen forest and the incidence angle (Figure 5) shows very low values in all RTC products and the GRD product, in all sites, for both polarizations VV and VH. An ideal value for this metric would be close to 0. This means that there is good topographic normalization since there is no dependence on incidence angle. The Vollrath et al. product shows slightly more negative values, yet the order of magnitude is not significant.

The CV results, shown in Figure 6, show significant variance, over 500% for the Vollrath et al. product over regions with sloped terrains, such as AND, BG, CR, and NP. This is to be expected, as the angular correction used in that approach does not perform well in rough terrain. Vollrath et al. [18] indicates that “those effects are linked to the assumption of homomorphism between map and radar geometry”. As explained by [10], this is due to the lack of bijectivity between radar and map geometry in regions with sloped terrain. ISCE-2 also portrays high variance over steep terrain. Over flatter areas, the performance of all terrain-corrected products is similar, such as in RN, MJ, MDD, and CG with a median CV of about 50%.

3.2. Geometric Evaluation

A comparison of two topographically different AOIs, Nepal, NP, (steep topography) and Madre de Dios, MDD, (generally flat), highlights the behavior of backscatter in relation to very different slopes and incidence angles (see Figure 7 and Figure 8). In a good quality geometric correction, it is expected that there is not much dispersion of the backscatter values, which in this case represent forest, across different slopes and incidence angles. For the NP site with more extreme slopes, we can see first that most of the data are located in slopes above 12.5 degrees, and second, there is a higher dispersion of values in these higher slopes compared to the MDD site, with a flatter topography.

The SNAP 8 and ISCE-2 products appeared to provide the least amount of correction for geometric distortions increased by slope. While the SNAP 8 product showed masked values in steeper slopes, ISCE-2 removed values in lower magnitude slopes. This is evident in the NP site; however, MDD shows consistent results across all ARD products evaluated. The GAMMA product with and without DEM-matching saw a greater amount of missing backscatter values with increasingly steep slopes, indicating that these values were considered to be unreliable due to geometric distortion and were filtered out during processing. The GRD and SNAP 8 products, on the other hand, had the largest amount of backscatter values at every slope, comparatively.

3.3. Geolocation Evaluation

Across all locations, GRD, SNAP 8 with DEM-matching, and ISCE-2 had the largest geolocation offset as seen in

Table 3. Summary of displacement metrics by SAR ARD product across all AOIs. n/a: not applicable.

ARD Product	Software Used	Avg (m)	Max (m)	Min (m)
RTC	GAMMA	18.45	49.23	1.09
RTC	GAMMA *	21.04	53.39	1.24
RTC	ISCE-2	40.38	111.74	5.92
RTC	SNAP 8	17.26	73.93	0.00
RTC	SNAP 8 *	47.84	117.57	9.78
Vollrath et al. [18]	GEE	16.85	53.84	2.38
GRD	GEE	710.66	49,181.76	0.53

(*) Indicates DEM-matching in the table and figure captions.

. More information about the quality of each point can be found in the dataset [27].

The GRD products had the greatest geolocation offset when compared to the displacement of other ARDs in every area of interest, except for sites BG and NM, where ISCE-2 and SNAP 8 depicted higher displacement offsets. See Figure A1. The mountainous AOI, NP, showed a clear relationship between greater geolocation offset with increasing slope in the multiple RTC products assessed. NP and CR were critically prone to geolocation errors across all ARD products and degrees of slope. This could be due to the lack of good elevation data over those regions.

These results also indicate that the RTCs created without DEM-matching perform better for GAMMA and SNAP 8. In addition, the Vollrath et al. product provides good results in this metric, particularly compared to the GRD product. In terms of best performance for this metric, both GAMMA products, Vollrath et al. and SNAP 8 without DEM-matching provided the smallest displacement offsets, while SNAP 8 with DEM-matching and ISCE-2 showed the highest displacement offsets. Overall, the GRD product had the largest error of all ARD products assessed, regardless of the degree of slope of the reference locations, as shown in Figure 9 and Figure 10.

3.4. Time Series

Figure 11 shows the CoV over MDD, comparing the land cover classes evergreen and grassland as well as flat and steep slopes, by product for VV and VH polarizations. As expected, for VV, evergreen has a consistently lower CV throughout the year for all products except Vollrath et al. Evergreen forest portrays a more stable signal than grassland, except for the Vollrath et al. product, with less variation when compared to the grassland class. In general, the evergreen forest is expected to maintain the same leaf structure throughout the year while grassland may have more variation due to dynamics on moisture and structure throughout the year than can be detected in C-band SAR. GAMMA and SNAP 8 have the lowest variation relative to the mean and follow each other closely month to month, within 0.4 percent. Overall, the ARD products have a more variable response over grassland, for both VV and VH polarization.

Figure 12 shows the CV over NP comparing the land cover classes evergreen and grassland as well as flat and steep slopes, by product for the VV and VH polarizations. ISCE-2 captured the same dip in September as GAMMA and SNAP 8. The definition of the evergreen class has a lower confidence in NP than that of MDD. In addition, there is significantly steep terrain in NP. These aspects made this AOI an important test case for the imperfect conditions of applications. Even the GAMMA products in this case appear to not perform much better than the other products, with high CV values and wide variation throughout the year. These variations appear to be aligned with seasonal variations. In fact, over steep samples, the VH polarization in SNAP 8, GRD, and ISCE-2 performed better, with a lower CV compared to the other products.

The Vollrath et al. product contains power values larger than what would be expected for backscatter; the maximum within our sample was 206,619 (pw) over flat areas at the VV polarization. This was also observed in the single-scene analysis. In

Table 4. RMSE (power) over forest class for MDD and NP compared by each product using GAMMA as the expected value, separated by polarization and slope classification.

	MDD				NP
	Flat		Steep		Flat		Steep
ARD	VV	VH	VV	VH	VV	VH	VV	VH
ISCE-2	0.37	0.08	0.29	0.06	0.20	0.04	0.45	0.08
SNAP 8	0.17	0.04	0.17	0.04	0.13	0.03	0.28	0.16
Vollrath et al. [18]	436.00	0.18	2.74	0.13	0.20	0.04	0.83	0.28
GRD	0.45	0.09	0.41	0.07	0.13	0.03	0.66	0.20

, the RMSE (power) closer to 0 indicates good performance. In this case, ‘good’ is defined by how similarly it performs to GAMMA products. Vollrath et al. performed the worst for this metric, specifically in the MDD site and VV polarization, with outliers RMSEs. However, VH polarization for the same site is within the range.

Table 5. Coefficient of Determination (Pearson R) over MDD and NP for the entire time series of data available for 2019, comparing each set of ARD products to products generated with GAMMA using 5000 samples. p = 0.0 unless otherwise specified.

	MDD				NP
	Flat		Steep		Flat		Steep
ARD	VV	VH	VV	VH	VV	VH	VV	VH
ISCE-2	0.026	0.040	0.27	0.20	0.085	0.074	0.043	0.054
SNAP 8	0.61	0.58	0.64	0.59	0.56	0.50	0.44	0.14
Vollrath et al. [18]	−0.0021	0.087	0.25	0.094	0.38	0.32	0.11	0.0020
GRD	0.25	0.22	0.25	0.23	0.52	0.47	0.13	0.013

shows that SNAP 8 has the highest coefficient of determination (at p = 0.0) for both slope classes over forested areas in MDD, indicating the greatest agreement of sample backscatter values between SNAP 8 and GAMMA. SNAP 8 performed about the same over time for both the flat and steep classes, as well as for both polarizations (with the greatest difference being between the VV and VH with 0.03 greater performance in VV steep as compared to VV flat). Vollrath et al. had the lowest agreement with GAMMA, with the greatest category of agreement surprisingly being steep slopes at R² = 0.25 (p = 0.0) for VV polarization. The next lowest was ISCE-2, with the greatest category of agreement also being the steep slopes at VV polarization R² = 0.27 (p = 0.0).

4. Discussion and Future Work

This study represents an exhaustive comparison including multiple metrics to assess the quality of SAR ARD products created through several software solutions. It also defines a framework to perform future inter-comparison analysis for SAR ARD products. Some of the metrics assessed have been evaluated separately in previous efforts [10,19,20], but never together in a comprehensive framework like the one presented in this paper.

The radiometric assessment indicated that the topographic normalization metric showed significant variance in areas of sloped terrain across all ARD products, and was significantly magnified in Vollrath et al. and ISCE-2 products in areas with uneven terrain. For example, all the ARD products showed variations above 100% in the Nepal (NP) site. Given these results, a consideration for the implementation of Vollrath et al.’s solution is to mask out areas with steep terrain and “0” gamma nought dB values where the angular correction is known to be insufficient.

The geometric results indicate similar performance across the RTC products, with more backscatter variability in areas with steep slopes. Geometric corrections are visible in the analysis of the GAMMA products. Flat areas show good geometric corrections across all RTC products and the Vollrath et al. product. As for the geolocation assessment, it indicates extremely high displacement for the GRD product. The time-series analysis indicates that SNAP 8 (without DEM-matching) is the most similar product to GAMMA regarding radiometric and geometric calibration.

These results indicate that it is not recommended to use the GRD product with no radiometric or geometric corrections for any applications. Regarding open-source methods, Vollrath et al. provide a good solution in flat areas; however, for appropriate topographic normalization in steep areas, SNAP 8 shows better performance. Our results also show better results without DEM-matching, for both GAMMA and SNAP 8. And while SNAP 8 performed better than ISCE-2 in multiple metrics, there are still concerns regarding geolocation displacement, even without DEM-matching, having a maximum displacement of 73.93 m. Regions with extreme topography are the most affected, which highlights the need to have multiple SAR acquisitions from different angles to address data gaps that result from geometric distortions.

At the time of writing, newer versions of SNAP (SNAP 9 [14]) and ISCE (ISCE-3 [47]) have been recently released. In addition, (a) there are updates to the Vollrath et al. angular approach [37], and (b) [48] describes a new python-based toolbox “wizard” that produces RTC gamma nought basckscatter. Future analysis should make use of the updated software versions and new solutions such as that listed by [48]. There is also availability of other GRD products, such as those from Sentinelhub [49], that could be evaluated. However, Sentinelhub-GRD is also produced using SNAP as the GRD from GEE. Hence, results could be very similar.

Furthermore, a new and enhanced DEM has been released by the European Space Agency, the Copernicus DEM [50]. Given the dependence of terrain flattening on good quality elevation data (e.g., DEM) a future analysis should use the new available DEM. It would be interesting to assess if better quality elevation data over the NP and CR sites, could address the displacement offsets seen in those sites across all ARD products. Such results indicate that the issue does not rely on the correction method, but rather that it is intrinsic to the site and associated input data used.

The use of the updated software, and new DEM will most likely provide different results in the metrics evaluated. Since there are many efforts underway to create RTC products, this study is timely and relevant to understand the performance of specific RTC quality metrics for multiple products, and potential focus areas to improve each.

Although not included in our analysis, the computing resource demands of each processing methodology are a critical deciding factor for remote sensing users. From all the ARD products evaluated, only the solutions implemented in GEE and/or otherwise provided by a space agency were fast, because either processing was running on the fly (e.g., Vollrath et al. implemented in GEE) and/or computing resources were covered by a third-party entity (e.g., ASF). The ARD products generated either through SNAP 8 or ISCE-2, were computationally expensive to create. Ref. [48] provides information on run times using SNAP 8. Both Vollrath et al. and Navacchi et al. [18,48] highlight the need to provide local contributing area and shadow mask as static data layers per relative orbit to facilitate and improve the generation of RTC products, in this case either using Vollrath et al. or wizard toolbox [48]. For Vollrath et al., this will address issues observed in steep terrain in that product [18].

Future endeavors should consider (a) a geolocation assessment in time-series analysis to investigate if the displacements per product are consistent across time, and (b) case studies for derived products, such as water masking or forest change detection. These follow-up studies would provide a better understanding of the implications of correction differences across SAR applications, which is ultimately what users want to understand. These results provide a good indication for such applications, but additional analysis would complement this work.