Monitoring Forest Health Using Hyperspectral Imagery: Does Feature Selection Improve the Performance of Machine-Learning Techniques?

Round 1

Reviewer 1 Report

Establishing a quantitative linkage between defoliation and hyperspectral reflectance is useful in the monitor of forest health. Due to the highly correlated, feature-rich characteristics of hyperpsectral data, feature selection has become a common procedure. However, given a real scenario, how to select a proper feature selection method is still an opening problem.   This study empirically studied the effectiveness of six typical filter-based feature selection methods in the context of tree defoliation estimation using airborne hyperspectral data. The paper is well written and easy to follow.  However, as a scientific publication, I have three comments as follows.   (1) the manuscript is a bit lengthy. I suggest the authors revise the paper to make it more focused. For example, between lines 521-526, the selected indices can be directly listed in table 9 rather than using the way in the paper.   (2）The authors concluded that no clear pattern is found in terms of the behavior of the selected feature selection techniques in the experiments. The conclusion makes the contribution of the work a bit weak. The experimental results may be further analyzed from a positive perspective for the benefit of practical use of the feature selection methods.   (3) The authors need to check the typo errors in the manuscript on the whole.  For example, in line 320, 28.09 p.p. can not be found in table 3 and RMSE in Table 3 may be wrongly placed.

Author Response

> Establishing a quantitative linkage between defoliation and hyperspectral reflectance is useful in the monitor of forest health.Due to the highly correlated, feature-rich characteristics of hyperspectral data, feature selection has become a common procedure. However, given a real scenario, how to select a proper feature selection method is still an opening problem. This study empirically studied the effectiveness of six typical filter-based feature selection methods in the context of tree defoliation estimation using airborne hyperspectral data. The paper is well written and easy to follow. However, as a scientific publication, I have three comments as follows.

Thanks for reviewing our manuscript!
We are glad you are having an overall positive opinion on our work and enjoyed reading it.
Please see our replies below.

> (1) the manuscript is a bit lengthy. I suggest the authors revise the paper to make it more focused. For example, between lines 521-526, the selected indices can be directly listed in table 9 rather than using the way in the paper.

Thank you for the feedback.
As mentioned in the "Methods" section, 89 vegetation indices were used in this study (with 86 remaining after a pairwise-correlation check).
Listing all of these in a table within the manuscript body would mean to include a multi-page table which would break the reading flow of the manuscript.
Hence we decided that placing the table in the appendix is probably the most concise way to include the information.

With respect to length, the manuscript essentially ends on page 20 with the remaining pages being appendices and references.
We believe that removing parts of the manuscript would introduce disturbance with respect to reading flow and content.
In addition, we have not seen similar requests from other reviewers and therefore would prefer to not remove parts of the manuscript just for the sake of shortening the manuscript.

> (2）The authors concluded that no clear pattern is found in terms of the behavior of the selected feature selection techniques in the experiments. The conclusion makes the contribution of the work a bit weak. The experimental results may be further analyzed from a positive perspective for the benefit of practical use of the feature selection methods.

Thank you for the feedback.
As often in science, results do not always present a clear and easy pattern which can be communicated in a definitive way.
More often instead, results show a fuzzy picture which should also be communicated this way.
Nevertheless, we are firmly convinced that the scientific community also needs to learn about research outcomes that are not exactly as expected, as this is necessary for scientific progress and avoids the well-known file drawer effect concerning unpublished research outcomes.
On the upside, the absence of strong and universal differences among feature selection techniques allows modelers to choose a methods based on criteria other than performance, such as, for example, computational efficiency, as pointed out in the Discussion in section 4.3:

"While filters can improve the performance of models, they might be more interesting in other aspects than performance: reducing variables can reduce computational efforts in high-dimensional scenarios and might enhance the interpretability of models.".

With respect to positive practical recommendations of this study, we would like to highlight some already existing points throughout the discussion (l.421 - l.426, l.443 - l.455) as well as in the Methods section, e.g. when explaining why and how filters can be used instead of wrapper methods for feature selection (section 2.3.1.), or why model-based optimization might be superior to simpler methods such as random search (section 2.4.3).
Also we mentioned the ease of integration filter methods into the optimization step and avoiding another dedicated feature selection layer on top, which comes with additional computational overhead (l.442).

> (3) The authors need to check the typo errors in the manuscript on the whole.  For example, in line 320, 28.09 p.p. can not be found in table 3 and RMSE in Table 3 may be wrongly placed.

Thank you checking Table 3 in detail.
The reference is in fact correct as it references the table with the best learner results.
The value itself (28.09) was in fact a typo as it should have been 28.12.
We have corrected this in the manuscript and apologize for the oversight.
As part of including feedback from other reviewers we also decided to use three digits instead of two for this table to better reflect differences between the model results.

In addition we have used a professional software to check for style and grammar issues.
We hope this improved the overall language quality of the manuscript.

Reviewer 2 Report

This study aimed to investigate if feature selection could affect tree defoliation estimation using hyperspectral data. The authors mainly focused on comparing the performance of different machine learners, feature sets and feature selection algorithms, which makes it just a comparison study. Also, due to the very small study sites, I think the results are not transferable in other contexts. My major concern is that the study did not well explain the relationship between the technical results and the response variable. The results and discussion were presented mainly focusing on the algorithms, which is not consistent with the title.

Introduction

• Tree defoliation is an important indicator of tree health and is the response variable of this study. However, I only saw two sentences mentioning it.
• Lines 55-57, there are many studies using ML and hyperspectral data to assess defoliation. It seems that the authors didn't do any literature review on this topic?
• Lines 58-62 should be moved to the end of the introduction where you briefly introduce the overall methodology of your study.

 

 

Author Response

> This study aimed to investigate if feature selection could affect tree defoliation estimation using hyperspectral data. The authors mainly focused on comparing the performance of different machine learners, feature sets and feature selection algorithms, which makes it just a comparison study. Also, due to the very small study sites, I think the results are not transferable in other contexts. My major concern is that the study did not well explain the relationship between the technical results and the response variable. The results and discussion were presented mainly focusing on the algorithms, which is not consistent with the title.

Thank you for the feedback and for reviewing this manuscript!
A sample of 1808 observations might appear small compared to other machine learning applications these days, however, sampling such an amount of observations in the field requires intense work over several days/weeks.
We also would have liked to work with even more observations; being able to even better showcase intra-plot differences and possible fitting better models.
Unfortunately in environmental work there are always practical limits, especially with respect to data collection.
We have discussed the limitations of the data used in section 4.5 and also mentioned the poor performance of the models.
Nevertheless, we still believe that the content of this study adds value for the community, event though more from a methodological than a practicable point of view.

Yet we agree that the relationship between the results and the response variable (and by that with tree health) were not addressed in greater detail so far.
Hence we have added the following section to improve this:

\subsection{Practical implications on defoliation and tree health mapping}
Even though this work has a strong methodological focus by comparing different benchmark settings on highly-correlated feature sets, implications on tree health should be briefly discussed in the following.
Due to the outlined dataset issues in \autoref{subsec:data-quality}, which are mainly responsible for the resulting poor model performances, the trained models are not suited for practical use, e.g. to predict defoliation in unknown area, due to the high mapping uncertainty.
Yet the general approach of utilizing hyperspectral data to classify the health status of trees partly lead to promising results:
For example, due to the narrow bandwidth of the hyperspectral sensor small parts of the spectra (mainly in the infrared region) were of higher importance to the models (e.g. see \autoref{fig:fi-permut-vi-hr}), meaning they helped the models to distinguish between low and high tree defoliation.
If spatial offset errors of the image data and possible background noise can be reduced (possibly by making use of image segmentation), we believe that model performances could be substantially enhanced.
Such improved models, starting around an RMSE of 20\% and less, should be able to provide added value to support the longterm monitoring of forest health and early detection of fungi affected tree plots.
Though overall the usage of defoliation as a proxy to forest health should be critically questioned as it comes with various practical issues, starting from potential offsets during data collection, varying leaf density due to tree age and differing effects between tree species, to just name a few.

> Tree defoliation is an important indicator of tree health and is the response variable of this study. However, I only saw two sentences mentioning it.

Thanks for the feedback.
Defoliation is an important part of this manuscript and we've reviewed our introduction of the terminology:
"Defoliation" was already six times mentioned in the introduction section.
Nevertheless, we have added more context to why defoliation can be a promising proxy for tree health in l.33 - l43.
We hope that overall the provided information will now suffice to introduce the topic of defoliation to the reader.
Please note that besides analyzing tree defoliation there is a strong focus on methodology in this work, which is why defoliation has been introduced only briefly compared to tree defoliation studies found in forest ecology journals.

> Lines 55-57, there are many studies using ML and hyperspectral data to assess defoliation. It seems that the authors didn't do any literature review on this topic?

Thanks for sharing your expertise with us.
We have done another literature research and it seems we have indeed missed related research on this topic.
We have now added five more references which relate to the overarching topic of hyperspectral data in relation to tree health and rephrased the paragraph as follows:

"In recent years, various studies which have utilized hyperspectral data to analyse pest/fungi infections on trees were published:
Pinto et al. \citep{pinto2020} successfully used hyperspectral imagery to characterize pest infections on peanut leafs using random forest while Yu et. al \cite{yu2021} aimed to detect Pine wilt disease in pine plots in China using vegetation indices derived from hyperspectral data.
Other studies which applied hyperspectral data for forest health monitoring are \cite{lin2014,kayet2019,dash2017}.
Building upon these successful applications of hyperspectral remote sensing usage in the field of leaf and tree health monitoring, this work analyses tree defoliation in northern Spain using airborne hyperspectral data."

> Lines 58-62 should be moved to the end of the introduction where you briefly introduce the overall methodology of your study.

Thanks for the suggestion.
We moved the mentioned paragraph to the end of the introduction.

---

Commenting on the remarks of grammar and style, we'd like to mention that we have used a professional software to check for style and grammar issues.
We hope this improved the language quality of the manuscript.

Reviewer 3 Report

This manuscript deals with analysis of hyper-spectral imagery with the aim to predict tree defoliation. It is not a typical research paper – it resembles more a technical/methodological document. As the authors state: “This study aims to show how high-dimensional datasets can be handled effectively with ML methods while still being able to interpret the fitted models”. This is a typical challenge for a researcher who applies ML, successfully resolved many times over. Therefore, the level of originality is low and inflated in places (NRI, line 49, is not a “less known index type”). Nevertheless the manuscript contains a significant amount of worthy material, despite suffering from being based on a relatively poor dataset. I would like to suggest that after a major revision the manuscript could be published in Remote Sensing.

Title

I find the title misleading, as the paper compares more variants to the ML application than just feature selection. Perhaps the sentence from Methods “The development of robust methods which enable an unbiased estimation of feature importance for highly correlated variables are subject to current research” is good clue about what the authors are after, and in such case it is placed in a wrong place.

Manuscript structure.

Introduction is short in comparison to Methods and Discussion. The latter section seems to contain some paragraphs which point out to important relevant research and which would be better placed in the Introduction. Some parts of the Methods contain general remarks that seem to be also suited better for Introduction (such as the Filter Methods section).

Moreover, Introduction does not narrow down, or explain why the scope includes filter methods and not wrapper methods. Only in the Methods (too late) the authors state: “Due to the focus on filter methods in this manuscript, only this sub-group of feature selection methods will be introduced in greater detail in the following sections.”

The objectives are not clearly stated in the Introduction. Some objectives are stated as late as line 398 (Discussion): “One objective of this study was whether expert-based or data-driven feature engineering has a positive influence on model performance.”

Design

Here I would like to flag three significant problems:

1. Sampling of the hyperspectral data is based on points with 1m buffer (to account for a positioning error), which in practice translates to 4 pixels per tree, in most cases. I would suggest that this means that each tree might be very poorly represented. I assume that these trees have crowns wider than 1 -2 m, and that ~4 m2 (possibly with some spectral information from outside of the tree) might represent only a small fraction of the crown. I recommend revising this approach.
2. The VI feature set is of completely different nature than HR and NRI, because the VIs only represent some portion of the analysed spectra, which is equivalent to elimination of a lot of features a priori, whereas in the case of the other sets this elimination occurs later. Moreover it is not clear how many VIs were used, as it is only mentioned that the 400-1000 nm allowed to calculate only some of them.
3. Computational requirements are not included in the comparison of variants, while constraints in this regards seem to have affected the results, as it was decided that 70 iterations for the XGBoost will be executed.

Other comments.

Line 231 – State by which % the number of features has been reduced through the analysis of pairwise correlation. Also, it would be worth discussing why do authors think that allowing most of the features to pass this test is better than eliminating them in subsequent, more computationally intensive, stage of the analysis.

Table 3. Please revise the results. RMSE and SE values are the same everywhere. Line 320 cites different RMSE value than what is in the table.

Figure 3. SE values not presented, but mentioned in the caption. Were RMSE the same for all SVM cases?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

In the manuscript, “Monitoring Forest health using hyperspectral imagery: Does feature selection improve the performance of machine-learning techniques?”, the authors described a study for the analysis of tree defoliation in the northern region of Spain, and this using hyperspectral data as input for machine learning which used hyperparameter tuning and filter-based feature selection. Subsequently, a comparison of the performance of the considered machine learning models. The main objective of this study is to demonstrate whether expert-based or data-driven feature engineering has a positive influence on model performance.

The manuscript is overall quite well written and well organized, although I find that the style of the latter needs to be improved to make it easier to read and understand. The study discussed is fairly well detailed and well presented. Therefore, I recommend for publication.

Author Response

> In the manuscript, “Monitoring Forest health using hyperspectral imagery: Does feature selection improve the performance of machine-learning techniques?”, the authors described a study for the analysis of tree defoliation in the northern region of Spain, and this using hyperspectral data as input for machine learning which used hyperparameter tuning and filter-based feature selection. Subsequently, a comparison of the performance of the considered machine learning models. The main objective of this study is to demonstrate whether expert-based or data-driven feature engineering has a positive influence on model performance.

> The manuscript is overall quite well written and well organized, although I find that the style of the latter needs to be improved to make it easier to read and understand. The study discussed is fairly well detailed and well presented. Therefore, I recommend for publication.

Thanks for the positive feedback and for reviewing the manuscript!

Round 2

Reviewer 2 Report

The current version can be accepted.