Reply to Mas et al.: Comment on Gebhardt et al. MAD-MEX: Automatic Wall-to-Wall Land Cover Monitoring for the Mexican REDD-MRV Program Using All Landsat Data. Remote Sens. 2014, 6, 3923–3943

Abstract

Mas, J.F. et al. have submitted a paper [1] for publication, which aims to respond to a paper published by Gebhardt et al. [2]. Mas, J.F. et al. had received a consultancy in 2013 to assess the quality of the early prototype products partly described in Gebhardt et al. in 2014. This consultancy, although a formal non-disclosure agreement had not been demanded, was awarded under the mutual understanding that the data handed over to Mas et al. constitute the early development phase of the program. Therefore, Mas et al. had been asked to give an assessment on the quality of the prototypes to obtain a proof of concept for the proposed workflow of MAD-Mex. It was clear that this assessment would suffer from limited availability of high quality training and validation data available in 2013. Mas et al. finally did not execute the consultancy due to the limited vector processing capacities in their lab. In October 2014, we sent the latest products, version 4.2 of the MAD-Mex products, including the more than 200,000 validation points gathered from independent expert interpreters of all Mexican ecosystems. Mas et al. did not respond to this transfer or to our request to collaborate in the quality control and assessment of MAD-Mex.

MAD-Mex is the working title of a Mexican federal government program executed by the National Forestry Commission (CONAFOR), the National Institute for Statistics and Geography (INEGI) and the National Comission for the Knowledge and Use of Biodiversity (CONABIO) aiming to develop a pre-processing system in response to the increased frequency of Mexican national and international reporting needs, amongst others INEGI statistics, REDD+ (Reducing Emissions from Deforestation and Forest Degradation in Developing Countries), or Aichi [3]. The objective of the program is to design and implement an optical satellite data pre-processing system which handles data management, data ingestion, data pre-processing (e.g., LEDAPS) and data classification to a point where it becomes feasible to produce annually high resolution cartography on land cover and land cover changes in order to report for instance REDD+ MRV AD (Mexico is going to use its National Forest and Soils Inventory (INFyS) and satellite-based remote sensing technologies as instruments for developing a robust national Measuring, Reporting, and Verification (MRV) system in which the state and change of land cover plays a key role in activity data (AD) monitoring).

MAD-Mex as a processing system is embedded in a national production system which consists of several interlinked procedures, involving the automated pre-processing of land cover and land cover change products to a number of INEGI defined classes (8, 10, 12 and 35 for Landsat data) and at a minimum quality (80% overall accuracy at level 1 comprising eight basic land cover classes adjusted to the land cover classification scheme provided by the IPCC). The result of the automated processing is then handed over to a team of independent expert interpreters, registered in the INEGI expert roster. This group edits class labels of objects and performs the accuracy assessment of the products using additional and independent data derived from very high resolution satellite imagery and/or aerial images and also (upon availability) national validation sample data. Additionally, there is a second QA/QC step by INEGI personal to finally achieve maturity of the products and to register them in the official national cartography system.

Regarding the institutions involved in the program, it remains unclear why Mas et al. decided to formulate their feedback in a scientific journal instead of responding directly to the funding agency of the consultancy (CONAFOR) and why they choose to ignore the latest products and validation data that has been recently published and made available to them [4] but instead worked on the prototype products described in a publication in 2014 [2].

Firstly, Mas et al. argue that the accuracy assessment of land cover maps is optimistically biased, because the reference dataset used is not randomly distributed within the different categories of the MAD-Mex maps. The data set, available to Mas et al. in 2013, did indeed feature this constraint, but as mentioned, this specific data set was intended only to prove the concept of the workflow proposed in MAD-Mex, not to assess accuracy of the products. In 2013, we used a set of official reference data for QA, the only one available at that time. Forestry samples were provided by the National Forestry Commission (CONAFOR), The Colegio de Posgraduados (COLPOS) and the Secretaría de la Reforma Agraria (SRA) provided samples for grassland and agricultural land use and INEGI for urban areas.

Aware of these constraints, early on in the program, the team (INEGI-CONAFOR-CONABIO) has started a strategy to generate completely independent validation data (VD) for several years of land cover classification. The generated new VD set complies with the demands for a random stratified national sampling. The recently finished VD set features more than 200,000 reference polygons for every distinct year. (For example, a set of 5125 points derived by random stratified sampling distributed over the Mexican territory has been generated. As strata the land cover classes from the previously published land cover dataset for the year 2000 haven been used. Based on these nodal points image chips of 1 km² have been subset from cloud free Landsat images and subsequently been segmented. This resulted in a total of 242,170 polygons, which have been labeled manually assigning the land use and vegetation class, including the state, i.e., secondary or primary, disturbed and undisturbed forest.) [4]. Those VD have been made available to Mas et al. in October 2014.

Secondly, the authors stress the lack of suitability of the products to monitor land cover change. This assumes that the program MAD-Mex will use land cover (LC) products, or rather the difference of two given LC products for change assessment.

Since the beginning of the program, MAD-Mex has emphasized that the team does not consider LC differencing a suitable approach. We discussed this fact on page 3939 of the original paper: “Main target of REDD+ is measuring and reporting deforestation and forest degradation which requires change analysis. The use of discrete land cover products as sole source is limited. It will propagate classification errors into the change products and derived change metrics. Our maps have a distinct error margin for each class. Even if we classified for some classes with very high accuracy, the error will always be greater than the rate of change for this specific class. Thus, a reporting of changes based on comparing discrete land cover maps is ill advised. We therefore propose the combination of data driven change detection algorithms in combination with the land cover products for final of the qualitative change, i.e., the actual change class.”

MAD-Mex implemented change detection algorithms that work directly on the input data and not on land cover maps. Based on those, several national change products have already been processed but not published so far because we are still in the process of reference data generation for validation. One can imagine, that reference data generation for thematic change detection for a country like Mexico, a time span of 4 decades, and map scales of 1:100,000 and 1:20,000 is quite demanding and costly.

Thirdly, the authors argue that the use of an entirely automatic classification approach is highly problematic in the case of a large and heterogeneous country such as Mexico.

We totally agree with that and, therefore, as already briefly described above, MAD-Mex as a program does not solely rely on automated processing, which we also stated in the introduction of the original paper (2014): “The paper presents the MAD-Mex system (Monitoring Activity Data for the Mexican REDD+ program) for automatic wall-to-wall baseline land cover mapping. It is the aim of the MAD-Mex system to deliver, standardized national land cover products in a timely and transparent fashion at a given reported classification accuracy. This will serve as the prime base product for further (visual) interpretation and enhanced land cover class assignment. We deem this the only feasible method to frequently update national land cover and land use products at a production scale of 1:100,000.”

As briefly described above, MAD-Mex will be used for data handling and pre-processing, rendering within 5 days pre-products of LC and LCC at 1:20,000 and 1:100,000 scale. These pre-products will be handed over to a team of experts for editing and QC. The experts will spend 6 months for this editing process. By 1 July every calendar year, the result of the expert revision is transferred to INEGI for a second QC and in parallel made available to independent national experts from universities and NGOs. The comments resulting from this step are expected to arrive by 30 September of each production year and in the remainder of the year, INEGI experts will do the final editing and publishing of the cartography.

Finally, Mas et al. argue that the MAD-Mex system is centralized in Mexico City and operated by a small team of experts without regional expertise. This represents a lack of consideration of information available to at least one of the co-authors, who was present during the whole development cycle of the program as a technical advisor, and who was given every 6 months an update of the status quo of MAD-Mex in the form of a 2-day workshop. MAD-Mex is not a research project of a group of universities but a federal government mapping program commanding considerable resources of all involved agencies.