Automating Drone Image Processing to Map Coral Reef Substrates Using Google Earth Engine
Round 1
Reviewer 1 Report
The paper presented a semi-automatic workflow for drone image processing with Google Earth Engine and free and open source software, and did an experiment to test the performance of the workflow. This can be of benefit to the ecologists in the field of coral reefs, and extended the application of drone images. However, I think that the authors should be answer the following questions before the paper can be published in drones journal.
- Why the authors selected to do the experiment in Heron Reef?
- What is the main contributions of the authors in the workflow, or which step of the workflow is originally proposed by the authors?
- In Accuracy Assessment section, why the authors did not utilize some in situ ground-truthing practices?
- Language and format questions, e.g. Line 171, 177, 204-205, 208-209, 193(rock/ dead or rock/dead?), etc.
- The title of Table 1: Why some parts of the words of the title is bold?
- Why only Heron Reef was used in the experiment? What is the performance if the workflow is applied to other coral reefs?
- As is well known, coral reefs are generally flooded under water column, so the RGB image captured by drones are undoubtly affected by the water column. What is the performance of the workflow under different water column and different water depths?
- Line 116-117: “Images were collected from 14:30 to 17:00 on February 20th, 116 2019…”; Line 177-178: “Predicted live coral cover percentages of Heron Reef range from less than 1 percent to 60 percent (Figure 3a).”; Line 213-216: “Coral cover substrate percentages produced from this classification workflow are within substrate cover ranges for Heron Reef published by Reef Check Australia in 2018. The Reef Check Australia 2018 Heron Island Reef Health Report reported an average of 42 percent hard coral cover observed across 15 monitoring sites, with coverage ranging from absent to 73 percent.” Why the authors did not make use of the data of the Reef Check Australia 2018 Heron Island Reef Health Report to assess the performance of the workflow? The Highest Predicted live coral cover percentages of Heron Reef was 60 percent, while the coverage ranging from absent to 73 percent according to the Reef Check Australia 2018 Heron Island Reef Health Report, what is the reasons for the difference (60 percent V.S. 73 percent) during in only a year?
- Line 221-222: “The presented methods provide steps for image preprocessing, so drone imagery is formatted for other classification techniques and algorithms in GEE that classify coral cover.” Is Random Forest Classifier is the optimum classifier except for Object Based Image Analysis?
- The authors said that “Using a different classification approach, such as Object Based Image Analysis (OBIA), may increase coral and rock/dead coral classification accuracy and reduce overpredictions of coral cover found in this study.” Why the authors did not integrate Object Based Image Analysis with Google Earth Engine to improve both the performance of the workflow and the time efficient of Object Based Image Analysis?
Author Response
Please see attachment.
Author Response File: 
Author Response.docx
Reviewer 2 Report
This paper describes the use of a drone for mapping corals. Quite a nice paper and it is showing that the processing can be done using free software available on the internet. This is of course of interest to many organisations, with less finances, who find the cost of some software packages prohibitive. Drones are now relatively cheap and can be bought easily. However software to process if often missing and therefore this paper will be useful to show such users how to process their own data, following the example presented in the paper.
The authors mention that they “present the steps” (line 95) to do processing , classification and assessment using Google Earth Engine and FOSS. The steps are diagrammatically shown in fig 2 but lack examples and detail in the text. For example for pre-processing the authors say they convert the JPG to GeoTiff using python. Is there supplementary materials such as the python program included? For image classification can an example of the actual use of GEE or FOSS be shown for Random Forest (was it done in R?). Accuracy assessment is part of the software usage and it would be useful to know which package was used and how.
This study could be used as a baseline assessment for temporal variation, but this is not mentioned. As the 230 images are well spaced, comparison with satellite imagery would be possible (if available and/or accessible financially). I therefore await the subsequent paper on the validation of satellite imagery or an addition to this paper.
Specifics
Figure 1 – the location map showing the whole of Australia is not useful. The reader can only determine that it is in the Barrier Reef area! Zoom in to just the coastline of NE Australia.
Line 113 – Is the approximate footprint on the water surface or on the seafloor?
Line 116&117 – what are the range of depths in the area at low tide?
Line 140&141 – remove the [m2] on both lines – does it mean anything?
Line 168 – describe the difference between User and producer accuracy metrics.
Line 176 – figures given in text do not match the figure 3 diagrams b or c and therefore are confusing. As they are for the whole area maybe percentages would be more useful, to report on the health of the reef that was surveyed, actual numbers are immaterial.
Figure 3 – Box colours are difficult to see – green on blue/green.
Table 1 – where did 11328 samples originate? – Text not specific.
Line 228 – is the increase in speed just a factor of computing power? Suggest “twenty times”. replaced by “much”.
Line 228 – Agree that OBIA can be difficult (and expensive e.g. e-Cognition) but there are other OBIA softwares available in QGIS, SAGA, and RSOBIA. These could be highlighted.
Line 281 – the straightforward and efficient workflow – needs to be available in supplementary materials. A “cookbook” would make this paper very useful and highly cited.
The supplementary materials is not well described or usable in its present form.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 3 Report
This paper presents a workflow for drone image processing by open software to classify substrates on a coral reef flat of Heron Reef. Monitoring of coral reefs, which are degrading world-wide, is essential for management, and the workflow provides an effective tool easy to use for other researchers and managers. Moreover, monitoring of substrates or vegetation in various shallow and land ecosystems by drones is now widely conducted, yet a standard data processing method has not been established, to which this study also would contribute. This paper is worth to be published in Remote Sensing. I raise two points the authors should take into consideration before publication for more general use of their workflow.
The substrate units are pre-determined by the authors as “We chose rock/dead coral, sand and live coral for substrate classification, as these substrates were most easily recognizable in the drone imaginary (lines 145-146)”. However, no ground-truthing was conducted not only for accuracy assessment but also for the choice of these units. I wonder how the authors “recognized” these units: based on previous studies by the authors themselves or references, which should be referred to. Heron Reef is the most studied reef in the world and it should be easy to identify these three units only by drone. However, in generalizing their work flow, researchers and managers will bring drones to the reefs with little or no information, of communities completely shifted. Thus, in a general workflow, the identification of basic units by a minimum field survey corresponding to the classified ones by Random Forest Algorithm should be added. This is particularly important for the reefs with seagrass and dense macroalgal cover, which are more difficult to differentiate from live corals. Those substrates do not seem to exist in Heron Reef with very simple (pristine) substrates, but more common in degraded reefs. Moreover, to extend this workflow to other ecosystems, identification process of the units should be included in the workflow.
The other concern is its spatial expansion. They applied drone survey for high-resolution monitoring along transect with 230 grids (20.1 x 16.7 m each) corresponding to coral colony scale identification.
The major concern of researchers and managers is reef scale mapping. I would like to know fly of the drone to higher altitude than 20 m would provide wide images while spatial resolution and resultant spatial resolution decrease, but whish altitude is the threshold. From where the authors bring the base map in Figures 1 and 3, which also show detailed substrates. By applying drone survey, we can draw a full mapping or Heron Reef.
Finally, more detailed image processing procedures in supplementary material will help general and entry-level readers to apply this workflow.
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Reviewer 4 Report
The study from Bennett et al. present a semi-automatic workflow to process and analyse drones images to enhance mapping capabilities of coral reefs. They collected images on Heron Island and use this example to present the workflow and interpret the results. I found the study interesting and can see the many applications possible for management purposes in accessing this type of information. The authors did well in presenting strengths and weaknesses of their approach.
My main comments regard three aspects of the study (1) reproducibility of the workflow (2) details of image processing and (3) accuracy assessment.
- (1) In complement to the paper, the supplementary materials (drones-887425-supplementary.zip\Drone_Image_Processing-master - ZIP archive, unpacked size 76,569 bytes) are composed of 6 files without a single instruction. I was expected a proper tutorial with all information needed to reproduce the workflow with a drone image from the study. Developing a new methodology comes with the responsibility to help people on how to use it. Also, the lack of information for reproducibility means that there is no way to test the workflow.
- (2) I couldn’t understand the first part of the workflow (how the training polygons per class and point per substrate were selected L150-154). Later in the discussion (L269), the authors explained that the training data was completed manually. I am not sure if there were some manual steps in the part b of the workflow? Maybe a figure that explain what you consider as a training polygons and the pixels per substrate class will clarify this part. It will also be important to show the readers some outputs from the Random Forest (RF) algorithm. How did you implement them? Which software? Can you share the codes as well? RF are tunes by a lot of parameters and this information are important to be shared if you wish that people use your method.
- (3) One important aspect of the method is the validation of model outputs. Accuracy assessments are important but a high accuracy doesn’t necessarily mean that the model is reliable. I suggest adding few performance metrics in your study by focusing on the live corals (the most interesting category). Metrics of sensitivity, specificity (i.e. recall) and precision will greatly help to validate the method.
Minor comments:
L29 – can you please add the reference/link to find the report
L29 - which models?
L31 - the frequency of tropical cyclones is not set to increase on the GBR but their intensity yes (see Marji Puotinen's work)
L39 - note that inconsistencies between in situ field surveys are not a stopping factor to use them together. The trend in statistical ecology is to develop methods that aim to analyse several datasets together while considering for their difference. The strength of your approach is about the high frequency of temporal surveys and broad spatial extent but at the price for ecological precision - which is usually ok for management purposes. Also, the fact that you can use the drone to survey shallow reef flats and crests can complement existing monitoring programs (LTMP, MMP) that focus on the reef slope only.
L50 - models of substrate cover - models for what? Estimating cover?
L52 - model = method?
L54 - yes - substrate classification models! This need to be tell before
L58 - could you tell the reader a rough meaningful cost to acquire these images?
L133 - is the script Python script available?
L137 - not sure what TFW means
L150-154 - I don't understand the workflow here. The training polygon per class and point per substrate class within polygon are pre-selected. How do you select them if you don't know where are classes? Are the polygons within image? More details and perhaps a figure will be useful here.
L155 - details of the tunning of the random forest algorithm are needed.
L162 - What is a pseudo accuracy assessment?
L168 - Producer and user: this is the first time that these terms are used. What does they mean?
L171 - User accuracy metric is not defined
Table 1- why you don't use % instead of number of pixels? You can keep to total columns (especially if you express the results in % in the text)
L 205 - 11 not eleven
L269 - not evident from the method section
L270 – what is interpreter error?
L273 – what is modeler?
L277 – what is modeler accuracy?
It would be great to see how your approach could fit in the global coral reef mapping Allen Coral Atlas project (Lyons et al. 2020). Do you see your work as competition (the workflow has the potential for scaling-up to more reefs) or complementing this project by bringing a potential new aspect into it? Additional words in the discussion will greatly leverage your paper – mapping coral reefs are very trendy, and your approach has the potential to bring something new in this field. I found that the open-access aspect of your workflow very exciting!
Author Response
Please see the attachment.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The authors have answered the questions carefully, and the paper have been greatly improved, so I suggest to accepted the updated manuscript if the authors can further checked the format and style as carefully as possible, for example, Line 346 (…The current spatial resolution of the GEE base map (where the scale bar equals 2 m) prevents…) was repeated by Line 347 and Line 348.
Author Response
We couldn't find a duplicate of line 346 in the text. However, we removed some repeating phrases in that paragraph that may have been redundant. That paragraph now reads:
"As the classification of substrates for training data in this study was completed manually, over and under-estimates of substrate cover percentages may also be the result of misidentification of classes due to human error when training polygons were created during the training stage. The current spatial resolution of the GEE base map (where the scale bar equals 2 m) prevents observing drone imagery at native resolution. Since mislabeling of classes by the interpreter during classifier training can decrease the accuracy of map products [54], efforts were taken to minimize substrate misinterpretation. Live coral, rock/dead coral, and sand substrates were chosen because they were the most easily identifiable substrates. As a result, the classifier classified substrates such as algae and rubble under the classes chosen, affecting classification accuracy. Since the main goal was to automate a workflow with FOSS to enable mapping of coral reef substrates in drone imagery, the number of substrate classes used was enough to achieve this goal."
Reviewer 2 Report
Happy with these revisions. All points adequately answered. Very happy to see proper supplementary info.
Author Response
Thank you very much!
