A Novel Outlier-Robust Accuracy Measure for Machine Learning Regression Using a Non-Convex Distance Metric

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Please see the PDF.

Comments for author File: Comments.pdf

Author Response

Review Report on title: “A Novel Outlier-Robust Accuracy Measure for Machine Learning Regression Using a Non-Convex Distance Metric”

This paper introduces a new accuracy metric for evaluating regression models. The author explores mean Hassanat distance (MHD) and mean Hassanat similarity percentage (MHSP) as alternatives to assess the strengths of a regression model. This new measures presents accuracy-like values, making it more intuitive as it scales from 0 to 1 or 0 to 100%. The paper validates this metric against traditional measures, demonstrating its effectiveness in various scenarios with different model types and data. The targets reveal that the Hassanat distance –based measure consistently provides reliable values.

Strength:

• The author not only explore new regression model, but also examines existing machine learning regression measures and studies their drawbacks.
• MHSP serves as a stable and well-defined regression measure.
• The proposed new regression accuracy measure is easy to interpret.

Response: Thank you for your encouraging and positive comments.

Weakness :

This paper includes some typo errors such as:

• Line 8 on Page No.2 (In Table 1), “\cite{allproblems}” should be replaced with appropriate reference.
• Line 10 on Page No.2 (In Table 1), “\cite{RMSEproblems}” should be replaced with appropriate reference.
• Line 14 on Page No.2 (In Table 1), \cite{MapeProblems}” should be replaced with appropriate reference.
• In line 65 “P Narloch” should be replaced with “Narloch”. · In line 89 “ Some” should be replaced with “ some”.
• In line 104 “[0, 100%],” should be replaced with “[0, 100%].”
• In line 138 “Tables 3 and 4” should be replaced with “Tables 3 and 4.”
• In line 174 “[ref]” should be replaced with suitable reference.
• In line 186 “.we randomly” should be replaced with “.We randomly”.
• In line 204 “R” should be replaced with “ R2 “.
• On page 5, in Table 4, “R 2” should be replaced with “ R2 ”. Decision:

Response: Thank you very much for pointing out the typos. We apologize for these oversights. We have carefully revised the manuscript, and to the best of our knowledge, the updated version is now free of typos.

The paper should be accepted after doing some minor revisions.

Response: Thank you for your positive feedback and for recommending acceptance with minor revisions. We appreciate your suggestions and we already made all the necessary changes to improve the manuscript.

Reviewer 2 Report

Comments and Suggestions for Authors

The authors propose the Mean Hassanat Similarity Percentage (MHSP) as a metric for evaluating the performance of machine learning regression models. They claim and demonstrate through examples and experiments, that this metric consistently reflects regression model performance, regardless of the problem domain, data scale, or presence of outliers. As someone who also works on regression models, I agree with the authors' points regarding the limitations of traditional metrics, including challenges like “a single value often fails to adequately reflect the effectiveness of a regression model” and the “debate about using them for specific applications.” I find the authors’ insights significant.

 

However, I have a few comments as follows:

 

1. In Table 1, some references are not displayed correctly, such as \cite{allproblems}, \cite{RMSEproblems}, and \cite{MapeProblems}.

2. In Line 60, the paper should avoid using "his"; using "the" or "their" for consistency is recommended.

3. In Line 101, the sentence “Here we revisit the mean Hassanat distance (MHD), which produces values between 0, high performance, and 1, low performance” is somewhat confusing.

4. In Line 133, the sentence “The formula for Hassanat similarity is detailed in Equation 6, where two cases are considered 6” has an extra “6” at the end. A similar issue appears in Line 203.

5. The sentence “Table 4. Results of different metrics on the data in Table 3” is missing a period at the end; this issue also appears in Line 138 and Line 156.

6. In Line 174, the reference “[ref]” is not cited correctly.

7. Acronyms should be defined upon their first use, such as “Random Forest” in Line 185 and Line 154.

8. The results presented in Tables 6, 7, 8, 9, 10, and 11, as well as Figures 4, 5, 6, 7, 8, and 9, do not clearly capture the advantages of the proposed MHSP metric. Please consider reorganizing the observations and results to make the benefits of MHSP more apparent. For example, dividing Section 6 (Results and Discussion) into two parts might help—first, highlighting the most compelling results and then discussing additional observations separately.

 

Author Response

The authors propose the Mean Hassanat Similarity Percentage (MHSP) as a metric for evaluating the performance of machine learning regression models. They claim and demonstrate through examples and experiments, that this metric consistently reflects regression model performance, regardless of the problem domain, data scale, or presence of outliers. As someone who also works on regression models, I agree with the authors' points regarding the limitations of traditional metrics, including challenges like “a single value often fails to adequately reflect the effectiveness of a regression model” and the “debate about using them for specific applications.” I find the authors’ insights significant.

Response: Thank you for your positive feedback and for recommending acceptance with minor revisions. We appreciate your suggestions and already made all the necessary changes to improve the manuscript based on your suggestions.

However, I have a few comments as follows:

1. In Table 1, some references are not displayed correctly, such as \cite{allproblems}, \cite{RMSEproblems}, and \cite{MapeProblems}.

Response: Thank you for pointing out the formatting issue in Table 1. These discrepancies were due to LaTeX errors, and we have corrected the references accordingly.

2. In Line 60, the paper should avoid using "his"; using "the" or "their" for consistency is recommended.

Response: Thank you for your suggestion. We have replaced 'his' with 'and colleagues' to enhance consistency in the manuscript.

3. In Line 101, the sentence “Here we revisit the mean Hassanat distance (MHD), which produces values between 0, high performance, and 1, low performance” is somewhat confusing.

Response: Thank you for your feedback. We have paraphrased the entire paragraph to clarify the statement and eliminate any confusion.

4. In Line 133, the sentence “The formula for Hassanat similarity is detailed in Equation 6, where two cases are considered 6” has an extra “6” at the end. A similar issue appears in Line 203.

Response: Thank you for bringing this to our attention. We have removed the extra '6' at the end of the sentence and corrected the related issue in Line 203.

5. The sentence “Table 4. Results of different metrics on the data in Table 3” is missing a period at the end; this issue also appears in Line 138 and Line 156.

Response: Thank you for your observation. We have corrected the punctuation in the mentioned locations, as well as in other instances throughout the manuscript.

6. In Line 174, the reference “[ref]” is not cited correctly.

Response: Thank you for pointing this out. We have added the missing reference and ensured it is cited correctly.

7. Acronyms should be defined upon their first use, such as “Random Forest” in Line 185 and Line 154.

Response: Thank you for your suggestion. We have defined the acronyms upon their first use. However, we have retained the full names of some regressors for readability, as there are many tables and results to help avoid confusion.

8. The results presented in Tables 6, 7, 8, 9, 10, and 11, as well as Figures 4, 5, 6, 7, 8, and 9, do not clearly capture the advantages of the proposed MHSP metric. Please consider reorganizing the observations and results to make the benefits of MHSP more apparent. For example, dividing Section 6 (Results and Discussion) into two parts might help— first, highlighting the most compelling results and then discussing additional observations separately.

Response: Thank you for your valuable feedback. We agree that breaking this section into three parts would enhance clarity. We have reorganized the section to present the most significant findings first, followed by a separate explanation of the remaining observations, and concluded with a discussion subsection.