A Novel Outlier-Robust Accuracy Measure for Machine Learning Regression Using a Non-Convex Distance Metric

Abstract

Regression, a supervised machine learning approach, establishes relationships between independent variables and a continuous dependent variable. It is widely applied in areas like price prediction and time series forecasting. The performance of regression models is typically assessed using error metrics such as the Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE). However, these metrics present challenges including sensitivity to outliers (notably MSE and RMSE) and scale dependency, which complicates comparisons across different models. Additionally, traditional metrics sometimes yield values that are difficult to interpret across various problems. Consequently, there is a need for a metric that consistently reflects regression model performance, independent of the problem domain, data scale, and outlier presence. To overcome these shortcomings, this paper introduces a new regression accuracy measure based on the Hassanat distance, a non-convex distance metric. This measure is not only invariant to outliers but also easy to interpret as it provides an accuracy-like value that ranges from 0 to 1 (or 0–100%). We validate the proposed metric against traditional measures across multiple benchmarks, demonstrating its robustness under various model scenarios and data types. Hence, we suggest it as a new standard for assessing regression models’ accuracy.

1. Introduction

Unlike classification models, which aim to find the relationship between independent variables and a categorical dependent variable, regression models predict a continuous dependent variable given one or more independent variables. Although regression analysis is widely used in machine learning research, there is no universal agreement on a single, unified standard metric for assessing regression outcomes [1,2,3].

Commonly used metrics include the mean squared error (MSE), its square root variant (RMSE), the mean absolute error (MAE), and its percentage variant (MAPE). While these metrics provide valuable insights, they share a common limitation: their values can range from zero to infinity, and a single value often fails to adequately reflect the effectiveness of a regression model relative to the distribution of the ground truth data. The limitations of these popular regression metrics, as outlined in

Table 1. The most popular regression metrics and some of their limitations.

Regression Metrics	Limitations
Mean Squared Error (MSE)	Sensitive to outliers, lack of interpretability, scale dependence.
Mean Absolute Error (MAE)	Never differentiable at zero; can be a restriction in some, derivative-based optimization strategies, scale dependence [1].
Root Mean Squared Error (RMSE)	Sensitive to outliers, scale dependence [2].
Mean Absolute Percentage Error (MAPE)	Sensitivity to zero values: when the actual values are zero or near zero, MAPE can produce undefined or endless percentage errors, scale dependence [3]
R-squared ($R^2$)	Sensitive to outliers, affected by the sample size: might give misleading results, cannot distinguish between linear and non-linear relationships, no universal definition for the strength of a correlation [4,5]

below, highlight the complexities involved in accurately evaluating regression models [1,4].

As one can see from Table 1, the most commonly used methods have limitations that affect their explainability and assessment of regression-based machine learning models.

The basic mathematical definition of regression is to find a function that maps the input features to the target variables. Let us call the input features X and the target variable Y. The purpose is to identify a function such that $f\left(X\right)\approx Y$. The link between X and Y is frequently described as follows:

$$Y=f\left(x\right)+e$$

(1)

where e is the error or noise in the relationship. In assessing regression model performance, it is ideal to seek a value $\lambda$ that encapsulates the model’s accuracy in a simple, bounded format (0–100%). A $\lambda$ would ideally indicate a lower error and higher performance, and vice versa. Unfortunately, commonly used metrics like $R^2$ or adjusted $R^2$ do provide bounded values between 0 and 1, which can be interpreted as percentages. However, the number of features still influences these metrics, the scale of the data, and the presence of outliers. Thus, while these metrics are useful, they do not universally and independently measure performance without being affected by these factors. Although insightful, metrics such as MSE, RMSE, and MAPE are not bounded between 0 and 1, vary with data scale, and are sensitive to outliers, limiting their universal applicability and comparability across different datasets or models. Therefore, this study proposes a new bounded performance metric for assessing regression models based on the Hassanat distance.

In summary, the proposed metric is:

• Outlier-robust: Outliers do not significantly affect the performance assessment.
• Bounded: The result is bounded between 0 and 1, which is highly explainable as a model performance.
• Interpretable: As being based on the well-known Hassanat distance metric [6,7,8].
• The performance of regression models can be evaluated and assessed without the need to compare them to other methods.

2. Related Work

The application of regression-based machine learning models is widely recognized across various fields due to their versatility. These models are frequently discussed in the literature, not only in terms of the different types of regression models available but also with respect to the diverse metrics used to assess their performance. For example, Putri and colleagues [9] used various metrics such as $R^2$, RMSE, and cross-validation to measure the accuracy of these models.

Also, Sreehari et al. [10] proposed a system for climate prediction. They claim that their model helps in selecting the appropriate decisions on crop yielding. Their model is assessed using the RMSE metric.

Narloch and others [11] used different RMSE values to evaluate their deep learning system, which aims to predict the compressive strength of cement samples based on microscopic images.

A similar approach was used in [12] to evaluate a deep learning system that tests cement-stabilized soils based on ultrasonic waves. RMSE and MHD were used to evaluate their model.

Even though utilizing these metrics to evaluate regression models is popular, the debate about using them for specific applications continues to be an ongoing discussion in the literature [13,14]. Some research papers suggest that $R^2$ is more informative than other metrics [15], while others argue that $R^2$ is not efficient in evaluating some models [16]. Also, in multi-regression problems, $R^2$ lacks interpretability [17].

Further, MAPE’s tendency towards low prediction makes it unsuitable for evaluating tasks with high predicted mistakes [18,19]. As a consequence, the symmetric mean absolute percentage error (SMAPE) [20] was proposed as an alternative to address some of the drawbacks associated with MAPE. Despite the lack of agreement on its best mathematical expression in publications, SMAPE is growing in popularity in the machine learning community due to its unique characteristics [21].

Moreno et al. showed that MAPE does not meet the validity criterion due to the distribution of the absolute percentage errors, which is usually skewed to the right, particularly in the presence of outliers. Therefore, they proposed an alternative measure based on the calculation of the Huber M-estimator to overcome such problems [22].

Plevris et al. [23] studied 14 regression-related metrics, including the most widely used, such as RMSE, MAE, and the $R^2$, among others, providing their mathematical foundation, and discussing their characteristics, pros, cons, and limitations. However, none of these metrics were shown to be a perfect choice, as some of the metrics showed poor performance, while others exhibited good performance.

Table 2. Regression metrics, their range, and perfect match (target) value [23].

Regression Metric	Range	Perfect Match
Mean Bias	$[-\infty ,+\infty$]	0
Mean Absolute Gross Error	$[0,+\infty$]	0
Root Mean Squared Error	$[0,+\infty$]	0
Centered Root Mean Squared Difference	$[0,+\infty$]	0
Mean Normalized Bias	$[-1,+\infty$]	0
Mean Normalized Gross Error	$[0,+\infty$]	0
Normalized Mean Bias	$[-1,+\infty$]	0
Normalized Mean Error	$[0,+\infty$]	0
Fractional Bias	$[-2,2$]	0
Fractional Gross Error	$[0,2$]	0
Theil’s UI	$[0,1$]	0
Index of Agreement	$[0,1$]	1
Pearson Correlation Coefficient $R^2$	$[-1,1$]	1
Variance Accounted For	$[-\infty ,1$]	1

shows the investigated regression metrics.

As can be seen from the literature, there is a plethora of regression metrics; however, each has its own strengths and weaknesses. Most of these metrics do not have an upper bound, which makes judgment on the performance of a regression method challenging. Furthermore, most of them are affected by outliers, i.e., one extreme false prediction can drift the regression method assessed to a lower rank. Therefore, evaluation of the quality of uncertainty estimates in machine learning regression leaves a lot of room for improvement [24].

As an improvement, we propose a regression metric based on the Hassanat distance, which is bounded between 0 and 1 and, therefore, does not allow any extremely false prediction to dominate the assessment of the regression model. Here, we revisit the mean Hassanat distance (MHD), which measures performance on a scale from 0 to 1, with 0 indicating the highest performance and 1 indicating the lowest performance [12,25]. To have an accuracy-like measure, we propose the use of the mean Hassanat similarity percentage, which is bounded in the range $[0,100\%]$.

3. Proposed Metric

The Hassanat distance metric is widely used for different machine learning applications [26,27], such as epilepsy diagnosis [28], alert systems [29], computing expectiles [30], content-based image retrieval [31,32], face detection/recognition [33], biometrics [34,35], cancer classification [36], fraudulent detection [37], validation of oversampling approaches [38,39], etc.

Moreover, many researchers have found that the Hassanat distance is very useful for various machine learning tasks, including but not limited to those mentioned in references [40,41,42,43,44,45,46,47,48,49,50,51].

The proposed regression measure is derived from the Hassanat distance, which is a non-convex distance metric given by Equation (2):

$$\mathrm{HasD}(X,Y)=\sum _{i=1}^{m}D(x_i,y_i)$$

(2)

where $D(x_i,y_i)$ is expressed in Equation (3):

$$D(X,Y)=\left\{\begin{array}{cc}1-{\displaystyle \frac{1+min(x_i,y_i)}{1+max(x_i,y_i)}}\hfill & \mathrm{if}min(x_i,y_i)\ge 0,\hfill \\ 1-{\displaystyle \frac{1+min(x_i,y_i)+|min(x_i,y_i)|}{1+max(x_i,y_i)+|min(x_i,y_i)|}}\hfill & \mathrm{if}min(x_i,y_i)<0.\hfill \end{array}\right.$$

(3)

As one can see from Equation (3), HasD is bounded by [0, 1]. It reaches 1 when the maximum value approaches ∞ assuming the minimum is $-\infty$, or when the minimum value approaches $-\infty$ assuming the maximum is ∞ [6,7,8]. HasD between 0 and values in the range of [−10, 10] is visualized in Figure 1.

Also, HasD can be simplified mathematically to the expression in Equation (4).

$$D(x,y)=\left\{\begin{array}{cc}{\displaystyle \frac{|x_i-y_i|}{1+max(x_i,y_i)}}\hfill & \mathrm{if}min(x_i,y_i)\ge 0,\hfill \\ {\displaystyle \frac{|x_i-y_i|}{1+max(x_i,y_i)+|min(x_i,y_i)|}}\hfill & \mathrm{if}min(x_i,y_i)<0.\hfill \end{array}\right.$$

(4)

Based on Equations (2)–(4), MHD between two vectors, i.e., actual and predicted, can be derived as presented in Equation (5):

$$\mathrm{MHD}={\displaystyle \frac{{\sum }_{i=0}^{n-1}1-\frac{min(x_i,y_i)+1}{max(x_i,y_i)+1}}{n}}$$

(5)

where n is the number of observations or the size of the test dataset, x is the vector of actual values, and y is the vector of predicted values provided by a regressor.

This distance was proved to be a metric, in which the distance will be in the range of 0 to 1, regardless of the difference between the two values. The distance is calculated in relation to the dimensionality of the tested vectors (actual and predicted). Therefore, increase in dimensions increases the distance linearly in the worst case.

The Hassanat similarity, denoted as HasS(x,y), quantifies the similarity between two vectors by comparing their respective components. This similarity measure is distinct in its consideration of both non-negative and negative values within the vectors, adapting the calculation based on the sign and magnitude of the components. The Hassanat similarity formula, as derived from HasD, is outlined in Equation (6), which encompasses two scenarios: Case 1 pertains to instances when the minimum of both measured values is zero or positive, while Case 2 addresses instances where the minimum of both measured values is negative.

$$\mathrm{HasS}(x,y)=\left\{\begin{array}{cc}{\displaystyle \frac{1+min(x_i,y_i)}{1+max(x_i,y_i)}}\hfill & \mathrm{if}min(x_i,y_i)\ge 0,\hfill \\ {\displaystyle \frac{1+min(x_i,y_i)+|min(x_i,y_i)|}{1+max(x_i,y_i)+|min(x_i,y_i)|}}\hfill & \mathrm{if}min(x_i,y_i)<0.\hfill \end{array}\right.$$

(6)

A classification accuracy-like value can be obtained by applying Equation (7), the Mean Hassanat Similarity Percentage (MHSP).

$$\mathrm{MHSP}={\displaystyle \frac{\mathrm{HasS}(\mathrm{x},\mathrm{y})}{n}}\ast 100\%$$

(7)

A toy example to illustrate the result (predicted values) of different hypothetical models on synthetic actual targets is illustrated in

Table 3. Hypothetical data actual and prediction results of two imaginary models.

Instances	MODEL1		MODEL2
	Actual Y	Predicted X	Actual Y	Predicted X
1	1	1.5	1	4
2	1.1	1	1.1	3
3	1	30	1	5
4	2	2	2	3
5	2.3	2	2.3	6
6	1.6	2.1	1.6	2.1
7	3	3.2	3	3.2
8	1.6	1.2	1.6	1.2
9	2.5	3	2.5	3
10	1	1.4	1	1.4

and

Table 4. Results of different metrics on the data in Table 3.

Measure	MODEL1	MODEL2
Mean actual Y	1.71	1.71
Mean predicted X	4.74	1.71
HasD	1.928434	3.1746603
MAE	3.19	1.56
MAPE	3.095051	1.2065135
SMAPE	3.225	2.45
RMSE	9.1772	2.1014281
MSE	84.221	4.416
${\mathit{R}}^{\mathbf{2}}$	−180.942	−8.5398574
MHD	0.192843	0.317466
MHSP	80.72%	68.25%

.

As illustrated in Table 3, Model 1 nearly achieved accurate prediction results except for instance number 3, where it failed to predict a reasonable value (actual = 1, predicted = 30). Such discrepancies are common in machine learning and may be attributed to outliers in the feature values of instance 3. This value can be considered an outlier in the prediction results.

In comparison, Model 2 performed poorly relative to Model 1. However, it is worth mentioning that most common measures showed Model 2 to be the best-performing regressor, indicating their susceptibility to outliers like the one in instance 30. On the other hand, the Hassanat distance and its derivatives, MHD and the proposed MHSP, favored Model 1. This suggests a more reasonable decision, as these measures are not influenced by extreme values like the outlier (30). HasD returns values between 0 and 1 regardless of the difference between predicted and actual values, providing a more insightful evaluation of both models’ true performance.

4. Experiments Setup

To conduct our experiments and comparisons, we employed several regressors, namely, Ridge, Huber, Quantile, Theil–Sen, XGBoost, Random Forest (RF), and KNN. The experiments were conducted using six datasets publicly available on the UCI Machine Learning Repository.

The proposed metric is compared to several well-known evaluation metrics.

4.1. Common Metrics

The following well-known metrics are used as benchmarks to evaluate the effectiveness of our proposed metric:

4.1.1. Mean Squared Error (MSE)

$$\mathrm{MSE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n{(x_i-{\widehat{y}}_i)}^2$$

(8)

4.1.2. Root Mean Squared Error (RMSE)

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(x_i-{\widehat{y}}_i)}^2}$$

(9)

4.1.3. Mean Absolute Error (MAE)

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n|x_i-{\widehat{y}}_i|$$

(10)

4.1.4. Mean Absolute Percentage Error (MAPE)

$$\mathrm{MAPE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|{\displaystyle \frac{x_i-{\widehat{y}}_i}{x_i}}\right|\times 100\%$$

(11)

4.1.5. Explained Variance (EVS)

$$\mathrm{EVS}={\displaystyle \frac{{\sigma }_e^2}{{\sigma }_t^2}}$$

(12)

where ${\sigma }_e^2$: Variance explained by the regression model, and ${\sigma }_t^2$: Total variance in the data.

4.1.6. Coefficient of Determination $R^2$

$$R^2=1-{\displaystyle \frac{\mathrm{RSS}}{\mathrm{TSS}}}$$

(13)

where

$$\mathrm{TSS}=\sum _{i=1}^n{(x_i-\overline{x})}^2$$

(14)

and

$$\mathrm{RSS}=\sum _{i=1}^n{(x_i-{\widehat{y}}_i)}^2$$

(15)

In the Formulas (8)–(15), x is a vector of the actual values, $\overline{x}$ is the mean of the actual values, and $\widehat{y}$ is the vector of predicted values.

5. Datasets

The proposed metric was tested using six publicly available datasets on the UCI Machine Learning Repository [52].

Table 5. Datasets names and characteristics.

Dataset	# of Observations	# of Features	Target
Real Estate Valuation	414	6	Price
Average Localization Error (ALE)	107	4	ALE
Concrete Compressive Strength	1030	8	Compr. Strength
Forest Fires	517	12	Area
Combined Cycle Power Plant	9568	4	EP
Abalone	4177	8	Rings

shows the used dataset characteristics.

Each dataset has its own characteristics and distribution of feature values. However, our concern is focused on the dependent variable, as it counts when evaluating the performance of a regression method. Figure 2 shows the distribution of the dependent variables of all datasets.

A box and whisker plot for the dependent variable in all datasets is presented in Figure 3.

6. Results and Discussion

This section would be easier to understand if it is broken into three parts. As a result, we rearranged the section so that the most significant findings appear first, followed by a separate explanation of the remaining observations, and concluded with a discussion subsection.

6.1. Key Results

In order to investigate the use of the proposed metric for regression accuracy, we used the dataset shown in Table 5 to evaluate a number of common machine learning regression methods, namely, Ridge Regression (RG), Huber Regression (HR), Theil–Sen Regression (TSR), Quantile Regression (QR), RF, XGBoost (XGB), and K-nearest neighbor (KNN). We randomly split each dataset into two sets, training with 80% of the data, and testing with 20% of the remaining examples.

Table 6. Regression performance on Real Estate Valuation data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.656	5.584	57.673	7.594	0.185	0.656	0.149	85.061
Huber Regression	0.638	5.577	60.663	7.789	0.185	0.639	0.150	85.039
Theil–Sen Regression	0.606	5.740	66.082	8.129	0.208	0.632	0.176	82.438
Quantile Regression	0.558	6.558	74.204	8.614	0.223	0.558	0.171	82.934
Random Forest	0.810	3.867	31.859	5.644	0.120	0.811	0.099	90.103
XGBoost	0.796	3.947	34.170	5.845	0.124	0.796	0.103	89.698
KNN	0.611	6.103	65.179	8.073	0.186	0.631	0.146	85.415

,

Table 7. Regression performance on ALE data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.568	0.106	0.023	0.153	2.033	0.568	0.071	92.895
Huber Regression	0.553	0.098	0.024	0.155	1.050	0.556	0.065	93.550
Theil–Sen Regression	0.559	0.104	0.024	0.154	0.892	0.559	0.069	93.149
Quantile Regression	0.337	0.115	0.036	0.189	1.113	0.355	0.075	92.549
Random Forest	0.480	0.123	0.028	0.167	1.460	0.493	0.081	91.877
XGBoost	0.418	0.101	0.031	0.177	1.249	0.437	0.063	93.697
KNN	0.520	0.102	0.026	0.161	1.304	0.540	0.067	93.277

,

Table 8. Regression performance on Concrete Compressive Strength data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.628	7.746	95.971	9.796	0.293	0.628	0.196	80.423
Huber Regression	0.559	7.808	113.670	10.662	0.279	0.562	0.188	81.228
Theil–Sen Regression	−0.120	9.561	288.580	16.988	0.301	−0.068	0.193	80.725
Quantile Regression	0.570	7.970	110.900	10.531	0.307	0.573	0.197	80.308
Random Forest	0.884	3.736	29.854	5.464	0.123	0.887	0.103	89.662
XGBoost	0.918	2.996	21.218	4.606	0.100	0.919	0.086	91.393
KNN	0.737	6.442	67.762	8.232	0.238	0.743	0.171	82.907

,

Table 9. Regression performance on Forest Fires data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.004	24.399	11,740.330	108.353	$2.39\times {10}^{16}$	0.011	0.721	27.884
Huber Regression	−0.026	19.659	12,089.042	109.950	$3.96\times {10}^{15}$	0.001	0.592	40.769
Theil–Sen Regression	−0.017	20.407	11,983.918	109.471	$9.19\times {10}^{15}$	0.003	0.657	34.316
Quantile Regression	−0.030	19.621	12,145.666	110.207	$\mathbf{1}.\mathbf{30}\times {\mathbf{10}}^{\mathbf{15}}$	0.000	0.509	49.084
Random Forest	−0.015	27.011	11,960.393	109.364	$3.00\times {10}^{16}$	−0.012	0.743	25.699
XGBoost	−0.065	26.846	12,554.664	112.048	$3.35\times {10}^{16}$	−0.060	0.702	29.763
KNN	−0.008	26.136	11,886.223	109.024	$2.56\times {10}^{16}$	−0.003	0.694	30.640

,

Table 10. Regression performance on Combined Cycle Power Plant data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.931	3.543	19.608	4.428	0.008	0.932	0.008	99.225
Huber Regression	0.915	3.924	24.190	4.918	0.009	0.916	0.009	99.145
Theil–Sen Regression	0.932	3.527	19.507	4.417	0.008	0.932	0.008	99.228
Quantile Regression	0.902	4.249	28.171	5.308	0.009	0.902	0.009	99.075
Random Forest	0.964	2.260	10.159	3.187	0.005	0.965	0.005	99.506
XGBoost	0.967	2.204	9.386	3.064	0.005	0.967	0.005	99.517
KNN	0.947	2.877	15.115	3.888	0.006	0.947	0.006	99.371

and

Table 11. Regression performance on Abalone data, Bold font indicates the best performer.

Regression	${\mathit{R}}^2$	MAE	MSE	RMSE	MAPE	EVS	MHD	MHSP %
Ridge Regression	0.539	1.611	4.990	2.234	0.163	0.539	0.129	87.120
Huber Regression	0.533	1.566	5.051	2.248	0.152	0.542	0.124	87.573
Theil–Sen Regression	0.536	1.605	5.026	2.242	0.161	0.538	0.129	87.132
Quantile Regression	−0.073	2.378	11.617	3.408	0.247	0.000	0.183	81.656
Random Forest	0.531	1.581	5.073	2.252	0.155	0.532	0.123	87.665
XGBoost	0.471	1.655	5.721	2.392	0.163	0.472	0.128	87.168
KNN	0.526	1.556	5.130	2.265	0.151	0.528	0.122	87.841

show the regression results for each dataset, namely, Real Estate Valuation, Average Localization Error (ALE), Concrete Compressive Strength, Forest Fires, Combined Cycle Power Plant, and Abalone, respectively. The visualizations of the regression performance are depicted in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9.

6.2. Additional Observations

After a careful look at Table 6, Table 7, Table 8, Table 9, Table 10 and Table 11, we can observe the following:

• There is no universal measure that can be used to find the best performer since different measures disagree on the one best performer among the regressors tested. This is evident from Table 7 when using ALE data, Table 9 when using ALE data, and Table 11 when using Abalone data.
• Some datasets were easy for regression, such as Real Estate, Concrete, and Combined Cycle Power Plant data, where Random Forest and XGBoost were the best performers, achieving 90%, 91%, and 99.5% MHSP, respectively. This is evident from Table 6, Table 8 and Table 10, where all measures used voted for one regressor. At the same time, there was no significant difference between the performance of different regressors. This indicates that these datasets are easy for regression tasks as their target variables are well distributed, lacking skewness, and outliers, as can be seen in Figure 2 and Figure 3.
• Some measures obtained unconvincing very low values, such as $R^2$ on Forest Fires data using all regressors, or very high values, such as MAPE on the same dataset, which makes it hard for such measures to distinguish the best regressor.
• Full agreement on the best regressor by all measures; this happened three times when using Random Forest on Real Estate Valuation data, XGBoost on Concrete data, and XGBoost on Combined Cycle Power Plant data. This includes the proposed MHSP, which not only voted for the same best classifier, but also provides an accuracy-like, stable, and solid measure.
• In contrast to widely recognized measures such as MAE, MAEP, RMSE, and MSE, the proposed MHSP is bound within the range $(0,1]$, offering a more structured evaluation approach.
• The proposed MHSP aligns with at least one common measure when determining the top regressor. Even in cases where it agrees with only a single measure, MHSP delivers a consistent accuracy of approximately 87% for all regressors supported by other measures. This demonstrates that MHSP is a reliable metric and can be utilized as a viable regression measure.
• When designating the best performer as the candidate supported by the most metrics, MHSP played a crucial role in selecting this top-ranked model in four out of the six datasets under examination.
• The concept of interpretability is illustrated through the outcomes displayed in all tables. It is often challenging to understand the significance of typical regression metrics, such as what specific value denotes superior, inferior, or average performance from a regressor. There is no unanimous consensus on defining these values because traditional measures are unable to provide clear definitions. However, the MHSP metric offers a classification accuracy-like assessment, with 100% indicating a flawless regressor devoid of any errors and values around 0% signifying an inaccurate one. The intermediate percentages attempt to depict the actual performance of a regressor. Consequently, it is no longer necessary to compare a regressor against another for interpreting MHSP results. Even when evaluating a single-regression system, MHSP can be employed without having to make comparisons with other regressors, particularly when the percentage achieved is high and satisfactory.
• Some measures provide unexpected values, such as EVS = 0, $R^2$ with very small values for all regressors, or MAPE with very high values; see Table 9. This makes such measures incapable of voting for the best regressor, while MHSP and MHD are more stable and provide a distinctive difference suggesting the best performer.
• Both MHD and MHSP vote for the same best performer in all cases, this is due to the fact that both measures share the same properties of the Hassanat distance metric, as both are derived from the same source.

6.3. Discussion

The common regression measures demonstrated inconsistencies in identifying the most effective regressor in three specific datasets—ALE, Forest Fires, and the Abalone dataset. Upon examining Figure 2, it is evident that these datasets possess a left-skewed dependent variable, implying smaller values, which could potentially impact the reliability of these metrics. Moreover, Figure 2 reveals the presence of numerous outliers in these datasets, another factor that might affect traditional regression measures.

In contrast, the proposed MHSP is unaffected by variations in value sizes and is also immune to the influence of outliers, owing to its foundation in the Hassanat distance. This characteristic of the Hassanat distance is manifested in the computation of MHSP and MHD, ensuring that these factors do not impact both measures.

Although it is not a common practice to consider the best regressor as the one that best fits the line with $slope=1$ in the actual vs. predicted plot, we used this approach just for visualization purposes, as demonstrated in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9.

Notably, the proposed MHSP voted for the top performer three times out of six, which can be observed in Figure 4, Figure 5 and Figure 8. It is also important to mention that even when it did not vote for the best regressor in Figure 6, Figure 7 and Figure 9, the MHSP still selected a reasonable regressor, as evidenced by the scatter points representing XGBoost, Quantile Regression, and Random Forest.

It is worth noting that all the previous measures are capable of estimating only a single-valued prediction; uncertainty measures, such as Negative Log-Likelihood (NLL) and the Continuous Ranked Probability Score (CRPS), are commonly used to evaluate the confidence of the prediction [53,54].

The average of NLL across all observations can be expressed as follows [55]:

$$\mathrm{NLL}=-{\displaystyle \frac{1}{n}}\sum _{i=1}^{n}log\left(p\left({\mathbf{x}}_i\right|\theta )\right)$$

where ${\mathbf{x}}_i$ represents the i-th data point, n is the total number of data points, and $\theta$ denotes the model parameters. The equation calculates the negative log-likelihood by summing over each data point, computing the natural logarithm (log) of the likelihood $p\left({\mathbf{x}}_i\right|\theta )$ for that specific data point. The likelihood measures how well a given model fits the observed data. A smaller value of the NLL indicates a better fit between the model and data.

Table 12. Regression uncertainty evaluation results using NLL for all regressors on all datasets, Bold font indicates the best performer.

Dataset	Abalone	Forest Fires	Real Estate	ALE	CCPP	Concrete
Ridge Regression	1.666	3.607	2.276	1.149	1.992	2.407
Huber Regression	1.728	3.610	2.297	1.335	2.050	2.439
Theil–Sen Regression	1.695	3.610	2.327	0.950	1.984	2.665
Quantile Regression	1.943	3.611	2.342	1.436	2.094	2.433
Random Forest	1.659	3.613	2.118	1.914	1.837	2.138
XGBoost	1.693	3.624	2.149	1.626	1.817	2.057
KNN	1.700	3.611	2.267	1.150	1.936	2.332
MSHP Voted for	KNN	QR	✓RF	XGB	✓XGB	✓XGB
Others Voted for	none	R, MSE, RMSE, EVS	all	MAPE	all	all
MHSP of Best %	88/88	28/49	90/90	93/94	99/99	91/ 91

shows the results of NLL for all regressors on all datasets tested [55,56].

A comparison of Table 12 reveals that the proposed MHSP method identifies three top-performing models based on the NLL criterion, while the other measures ($R^2$, MSE, RMSE, EVS, and MAPE) identify four. However, it is interesting to note that all of $R^2$, MSE, RMSE, and EVS voted for the same regressor on all datasets examined; this is due to the fact that they share almost the same characteristics, as they all depend on the variance between predicted and actual data. Nonetheless, upon examining the last row of the table, it is apparent that there is no substantial disparity between the MHSP accuracy for the best performer according to NLL and its highest accuracy for each dataset for each regressor. This suggests that the MHSP provides a reliable measure when assessed via the NLL uncertainty metric. However, the Forest Fires dataset presents an exception due to its complexity, presence of zero actual values, outliers, and left-skewed distribution indicating small and nearly identical numbers.

The Continuous Ranked Probability Score (CRPS) is a performance measure used for probabilistic forecasts. It can be expressed as follows:

$$\mathrm{CRPS}={\int }_{-\infty }^{\infty }{(F\left(\mathbf{x}\right)-H\left(\mathbf{x}\right))}^{2}d\mathbf{x}$$

where $F\left(\mathbf{x}\right)$ is the cumulative distribution function (CDF) of the forecast, and $H\left(\mathbf{x}\right)$ represents the Heaviside step function, which equals 1 for $\mathbf{x}\ge 0$ and 0 otherwise. This equation calculates the squared difference between the CDF of the forecast and the Heaviside step function over the entire range of possible values ($\mathbf{x}$). A smaller value of CRPS indicates a better forecast performance, as it measures how close the predicted distribution is to the actual observed data.

Table 13. Regression uncertainty evaluation results using CRPS for all regressors on all datasets, Bold font indicates the best performer.

Dataset	Abalone	Forest Fires	Real Estate	ALE	CCPP	Concrete
Ridge Regression	0.019	2.766	−0.050	−0.024	0.258	0.484
Huber Regression	0.101	5.390	−0.157	−0.010	0.181	0.441
Theil–Sen Regression	0.060	4.170	1.197	−0.016	0.295	0.537
Quantile Regression	0.455	7.841	0.393	−0.006	0.523	0.548
Random Forest	0.009	1.627	−0.125	−0.013	0.018	0.091
XGBoost	0.068	12.284	−0.090	0.007	0.006	0.087
KNN	0.087	2.474	0.166	−0.018	0.028	0.201
MSHP Voted for	KNN	QR	RF	XGB	✓XGB	✓XGB
Others Voted for	none	none	none	R, MSE, RMSE, EVS	all	all
MHSP of Best %	88/88	26/49	85/90	93/94	99/99	91/91

shows the results of CRPS for all regressors on all datasets tested [57].

Similar to the previous analysis, a comparison of Table 13 demonstrates that the proposed MHSP method identifies two top-performing models based on the CRPS criterion, while the other measures ($R^2$, MSE, RMSE, and EVS) identify three. However, these measures agree on the same regressor in all tested cases. Nonetheless, upon examining the last row of the table, it is evident that there is no significant divergence between the MHSP accuracy for the best performer according to CRPS and its highest accuracy for each dataset for each regressor. This implies that the MHSP offers a reliable measure when assessed via the CRPS uncertainty metric. Nonetheless, like before, the Forest Fires dataset serves as an exception due to its complexity, presence of zero actual values, outliers, and left-skewed distribution signifying small and nearly identical numbers, which negatively impact all regressors’ performance.

Consistent with the prior analysis and graphical representations, it is postulated that the proposed MHSP regression measure furnishes a dependable, stable, outlier-insensitive, bounded by $(0\%,100\%]$, well-defined measure that mimics the classification accuracy for diverse regression techniques in machine learning. Additionally, similar sentiments apply to the MHD as it stems from the same source; however, this metric becomes less interpretable since its lowest value denotes the best performance, unlike the clarity provided by the percentage-based accuracy offered by the MHSP. It is worth mentioning that the range interval of MHSP is left open due to the fact that its value approaches 0 only when a predicted value or an actual value is approaching infinity, as explained in Equations (4)–(6).

A final note on MHSP being similar to the classification accuracy measure: While common regression measures like MAE, MSE, RMSE, etc. are frequently referred to as accuracy measures for machine learning regression models, they actually quantify errors in these models and do not fully capture the true accuracy of a tested model. On the other hand, the accuracy of a classification model is precisely defined as the ratio of correctly classified instances to the total number of instances in the test dataset as follows:

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{n}}$$

where $TP$ represents true positives, $TN$ represents true negatives, i.e., both $TP$ and $TN$ can be considered as the total number of the correctly classified examples, and n is the number of observations or the size of the test dataset.

Recall Equation (7), which represents the HasS metric. This measure provides the number of correctly predicted examples, either completely, if there were no errors, or proportionally, based on the similarity between predicted values and actual ones. To obtain the MHSP, we divide this resulting number (HasS) by (the total number of instances in the test dataset). Hence, the MHSP serves as an analog to the classification accuracy definition, which is defined as the ratio of correctly classified instances to the total number of instances in the test dataset. Therefore, understanding and applying appropriate evaluation metrics like MHSP for regression problems helps ensure accurate model assessment and selection based on specific problem requirements.

7. Conclusions

This study involves examining existing machine learning regression measures, which are known to possess certain drawbacks, such as being influenced by outliers, presenting difficulties in terms of interpretation, depending on scale, exhibiting sensitivity to near-zero values, resulting in an infinite percentage of errors, lacking a specific range, and not having a universally accepted definition for evaluating the performance of a regressor. As a result, this research aims to explore MHD and MHSP as potential alternatives to assess the strengths or weaknesses of a regression model.

In order to assess various regression methods, we employed multiple public real-world datasets commonly utilized for regression analysis. These datasets were then evaluated using well-known regression techniques. To compare and contrast the results, we examined several common measures, including the MHD and the proposed MHSP. Additionally, we assessed MHSP against two other uncertainty metrics.

Our experimental findings reveal that the MHSP serves as a dependable, stable, interpretable, outlier-insensitive, and well-defined regression measure with values in the range of $(0\%,100\%]$. Importantly, this measure demonstrates similar characteristics to the well-known classification accuracy for a diverse array of machine learning regression techniques.

These findings highlight the potential of MHSP as a promising alternative evaluation metric for machine learning regression models. Building upon these encouraging results and relevant features, we propose that MHSP has the potential to serve effectively not only as a metric for evaluating regression problems but also for forecasting time series issues. However, it is essential to emphasize that while these findings are promising, further research and comprehensive studies are necessary to fully explore and validate this potential application of MHSP in time series forecasting. This will help establish its effectiveness as an evaluation tool and contribute to the ongoing advancements in machine learning methodologies for tackling various types of problems, including forecasting.