Remote Sensing of Lake Water Clarity: Performance and Transferability of Both Historical Algorithms and Machine Learning

Round 1

Reviewer 1 Report

The authors tested the performance and transferability of a random forest (RF) machine learning algorithm and a simple 4-band linear model with 13 previously published empirical non-machine learning algorithms. It is important of the remote sensing community related to lake water clarity. However, before the publication, the manuscript need to answer these general questions. (i) Have the authors consider the surface reflectance continuity of Landsat 4, 5, 7, 8? (ii) How do you use the Landsat 7 ETM+ SLC-off data? (iii) The author said “L137-138: We extracted pixel data within a buffer zone of 1.8 times the Landsat pixel size (30 meters)…”, why 1.8 times?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

This is an interesting manuscript investigating the transferability of empirical methods (e.g., spectral indexes and machine learning methods) in space and time for estimation of water clarity.  In general, the paper is technically sound and well written. However, there some major issues in terms of the choice of the methods and accuracy metrics. The results also need to be presented and discussed in more detail.  Here are my detailed comments:

 

Major comments

As the radiometric resolution of Landsat-8 OLI is higher than the previous Landsat missions, I would recommend performing a separate analysis with Landsat-8 data only. This can reflect on the impact of radiometric resolution in retrieving the water clarity (Secchi depth).

 

The empirical methods used in this study (Table 1) are selected from previous studies. I recommend considering also recent approaches like optimal band ratio analysis (OBRA) that explores the optimal band ratio among the others. There are also other novel indexes based on the transformation of RGB to HIS space, coordinate system transformation, etc. The optimal band combinations for all these indexes can be identified automatically. Here are some references that the mentioned empirical methods are applied to similar water quality and bathymetry retrieval tasks:

Novel spectra-derived features for empirical retrieval of water quality parameters: Demonstrations for OLI, MSI, and OLCI Sensors

Spectrally based remote sensing of river bathymetry

 

I recommend using more comprehensive metrics for accuracy assessment like bias and win rate. This will provide additional information on the performance of the models. I suggest also reporting R2 along with the pseudo-R2 as the first one is more known and easy to interoperate.

 

Figures 2,3, 4 are useful but I suggest providing also matchup scatterplots that can better show the performance of different models. Of course, the illustration of the scatter plots for all of the cases is not possible. You may select some examples. As such plots can better show the performance of methods and reveal possible biases.

 

Although the evaluation of the methods is demonstrated based on independent in-situ data, I expect to see some maps of the Secchi depth derived from some of these methods. This can provide insights into the effectiveness of methods when applied to the entire water body. For instance, how noisy are the retrievals within a lake? Are the retrievals are meaningful moving from the shoreline to the mid-lake? How do they compare with respect to some information about the turbidity in different lakes? Etc.

 Although the focus of the study is on the empirical methods, I recommend adding a description about physics-based methods. This is important when discussing about transferability of the methods in space and time. As physics-based methods can provide more generic solutions. There are several physics-based methods applied on different multispectral and hyperspectral data (e.g, the WASI model appeared in several publications in Remote Sensing).

 

Supplementary Fig 5 can be part of the main body of the paper with some discussions on the biases.

 

I recommend publishing also the data used in this study. This can enhance the visibility of the work and can be a good reference dataset for other researchers. For instance, the refined spectral data and associated Secchi depths can be provided as an excel file.

 

Minor comment

Lines 96-97: it should be clarified what the “4-band multivariate algorithm” refers to? 4 bands of Landsat-8? Which bands? Please clarify.

Author Response

Please see the attachment

Author Response File: Author Response.docx

Reviewer 3 Report

Dear authors,

this is a very interesting study in which the water clarity (represented as secchi disk depth) is investigated in numerous lakes of four US-states during the last 34 years. Existing remote sensing based methods are applied and compared to a new Random Forest Machine Learning approach.

I think the manuscript is already of good quality and can be published after a few small improvements.

In the introduction I am missing a general overview of the inherent optical properties of the water constituents and the resultung apparent optical properties (depending also on illumination and viewing geometry). There is talk of the fact that chlorophyll, suspended matter and CDOM influence the radiation, but that the apparent optical properties such as the attenuation result from this is not specifically named. A sentence or two to illustrate that would be helpful.

The term "Secchi depth" is not always used throughout the document. The unit is also missing e.g. in Table 1 (I assume in meters). In the discussion, on the other hand, there is talk of a "deep" or "shallow" Secchi depth, here the terms "large" and "small" would be more suitable. I would ask you to use the term "Secchi Depth" consistently.

The conclusion chapter is very long and I think some parts of it belong in the discussion chapter. Please check this accordingly.

A few more comments on passages in the text:

(58-60) mention the seasonal developments of these water constituents

(65) The Secchi depth is a proxy for the attenuation of the water body, this should be named

(84) I think it's not correct to differentiate between chlorophyll/suspended matter and the AOP attenuation which includes the effects of these water constituents. That should be worded differently.

(Figure 1) Please add a small overview map, since not a readers will know the location of these states within the USA. It would be also nice to display the water bodies to show large lakes like Lake Champlain or Oneida Lake.

(136) What do these 5 days depend on, and why are both those before and after taken into account. If, for example, a thunderstorm occurs under a cloud cover, which causes a lot of surface runoff or upheaval of sediment to increase turbidity, it only makes sense to look at the days afterwards. That should be discussed.

(149) what is the unit of 250?

(Table 1) Name the units of the Predicted Variable and replace "Secchi" with "Secchie depth". What does "mean(RED)" stand for? Is this the mean of the whole image, or the water surface?

(213) The sentence "remove the influence of specific lake properties" is phrased in an unfortunate way. That's exactly what you're trying to find out.

(Figure 2 - Figure 4) It is irritating that squares are chosen for the slopes when they do not appear in the figure.

(312) You should name some atmospheric conditions that are affecting the accuracy.

(322) deeper > greater

(325) deep > great/large, very deep > very large

(331) deeper > greater

(338) shallow > low, like in line 346 "lower in situ Secchi depths"

(409) Here, at the end of the document, the bio-optical properties are mentioned first, this should already take place in the introduction. 

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

This version is much better now

Author Response

Thank you for your feedback!

Reviewer 2 Report

The revised manuscript has been improved by considering the reviewers' comments. Most of my comments are addressed adequately. However, there are some replies that were not convincing to me. I would recommend considering these points before a possible publication:

 

I agree that each accuracy metric has its own limitations. That's why I suggested using a set of metrics because each metric gives a different perspective. There is no metric that can be considered perfect alone. All of the metrics are meaningful only when provided along with others.

 

The illustration of some maps is promised for future papers by authors. However, I think this is important to have a couple of example maps in this manuscript. Otherwise, there is no real indication of the utility of the developed regression models in capturing the spatial dynamics of water clarity within the lake. This is what a reader expects from analyzing satellite images. If the results are provided only for some points, the main advantage of remote sensing, i.e., providing spatially distributed information, is neglected.

Author Response

To address your first point: we absolutely agree that a set of metrics is very necessary to fully understand the error associated with predictions. Hence, we use MAE, RMSE, a pseudo-R², the slope of the line describing the relationship between predicted Secchi depth and observed Secchi depth., and, at your suggestion, bias.

To address your second point: we understand your wish for a figure and we agree that it would be nice if we could properly visualize spatial variability of within-lake water clarity predicted from the random forest algorithm; however, this paper is about the statistical analysis and our in-situ lake data were curated such that there is only a single in-situ measurement per lake per date. Thus, we do not think it is an appropriate application of the developed random forest algorithm given the out-of-bag performance and the inability to quantitatively assess predictions within a lake at the requested spatial scale.

While a map of the distribution of Secchi depth predictions might look interesting and seemingly reveal spatial patterns, it will not be statistically meaningful, and the error associated will render any patterns irrelevant. We could produce some sort of map, as requested, but we would need to include a significant caveat stating that the absolute values are only reliable to the extent described by the statistics. For example, the low end of Secchi measurements could be in the range of 0-4 m and the high end could be in the range of 2.5-6.5 m; there is such an overlap that no meaningful distinctions can be made between low and high transparency. If we do want to see a statistically meaningful application, for even a single small lake, we would need to comb through Landsat surface reflectance data and our original database to find a lake that has multiple diverse Secchi observations on a clear-sky day. As we stated, in-lake spatial heterogeneity will be the subject of another entire paper that some of the coauthors are working on.

Additionally, we have mentioned in the text that the random forest algorithm is good for a “general sense” of Secchi depth but does not perform sufficiently well to make accurate predictions, especially on a pixel-by-pixel basis within a lake as you requested. By applying the random forest algorithm to a whole lake on a specific image date we are actively doing what we state that we should not do with this (or any) algorithm given the uncertainty around the predictions.  

  We feel that putting a map in the text inherently suggests that we are constructing a product to be applied as-is and that is not our purpose. In this manuscript, we are describing the caveats and issues with the random forest algorithm outcomes and suggesting that it is a promising avenue for more research. Therefore, we regret to say that we do not think this request is within the scope of our work in this manuscript.