Further Study of the DEA-Based Framework for Performance Evaluation of Competing Crude Oil Prices’ Volatility Forecasting Models

Abstract

The super-efficiency data envelopment analysis model is innovative in evaluating the performance of crude oil prices’ volatility forecasting models. This multidimensional ranking, which takes account of multiple criteria, gives rise to a unified decision as to which model performs best. However, the rankings are unreliable because some efficiency scores are infeasible solutions in nature. What’s more, the desirability of indexes is worth discussing so as to avoid incorrect rankings. Hence, herein we introduce four models, which address the issue of undesirable characteristics of indexes and infeasibility of the super efficiency models. The empirical results reveal that the new rankings are more robust and quite different from the existing results.

1. Introduction

Among energy sources, crude oil remains the major part of energy consumption. Its price volatility is closely related to the stability of both the macro-economy and micro-economy [1]. The importance of crude oil prices in industry hence attracts great attention among academies to forecast the volatilities of crude oil prices.

Statistical forecasting models represent one type of the main crude oil price volatility forecasting models, refer to [2,3], etc. Based on the classic Generalized Autoregressive Conditional Heteroskedasticity (GARCH) process, Kang et al. (2014) found that Component GARCH (CGARCH) and Fractionally Integrated GARCH (FIGARCH) models are useful for modelling and forecasting persistence in the volatility of crude oil prices [4]. García-Martos et al. (2013) used multivariate models to improve prediction accuracy [5]. Tang et al. (2015) proposed a novel complementary ensemble empirical mode decomposition (CEEMD) based extended extreme learning machine (EELM) ensemble model that statistically outperforms all listed benchmarks in prediction accuracy [6]. Pany and Ghoshal (2015) introduced a local linear wavelet neural network model and showed its effectiveness for price forecasting [7]. Klein and Walther (2016) compared the recently introduced Mixture Memory GARCH (MMGARCH) with classical GARCH-type models [8]. Liu et al. (2018) investigated the impact of truncated jumps on improving the forecasting ability [9].

Although there is an enormous amount of literature available on predicting the volatility of crude oil prices, less attention is devoted to evaluating their forecasting performances. In addition, among the various emphases of forecasting models, there is no consistent conclusion as to which forecasting model outperforms all others. Regarding this, Xu and Ouenniche (2012) put forward a multidimensional performance evaluation framework for evaluating the crude oil prices’ volatility forecasting models [10]. To be more specific, a super-efficiency data envelopment analysis (DEA) model proposed by Andersen and Petersen (1993) is introduced [11]. The DEA method is widely applied in various areas (refer to [12,13,14,15,16], etc.). This multidimensional DEA-based framework contributes significantly to obtaining a unified ranking while taking all relevant criteria into account. However, several issues discredit the applicability of this super-efficiency DEA method. First, as indicated in [17,18], the super-efficiency model appears to be infeasible when the decision-making unit (DMU) under evaluation rests outside the production possibility set spanned by the other DMUs. In this sense, the evaluation results, which include these infeasibilities, are unreliable. Second, Xu and Ouenniche (2012) failed to reflect the essence of the evaluation criteria while taking them as inputs and outputs. The details are explained in the next section [10].

The contribution of this paper is three-fold. First, the desirability characteristics of the indexes for evaluation are discussed in detail. Second, two different evaluation processes are described, and the essence of criteria in these processes are discussed correspondingly. Third, the infeasibility problem arisen in applying a super-efficiency DEA model is tackled with an advanced model proposed by [19]. The structure of this contribution can be outlined as follows. In Section 2, a super-efficiency DEA-based framework is proposed for assessing the performance of the forecasting models for crude oil prices’ volatility. The essence of the evaluation criteria, namely, the desirability, is discussed in detail. The models for dealing with infeasibilities are also introduced. In Section 3, empirical results and a comparison with those in [10] are presented. Finally, concluding remarks are summarized in Section 4.

2. Materials and Methods

2.1. Super Efficiency DEA Models with Undesirable Inputs (Outputs)

While considering the forecasting process of a specific model as a production process, although all dimensions of criteria are calculated to evaluate the forecasting performance, decision makers who give preference to the ability of forecasting the correct changing direction will focus on the correct sign prediction, which means that the forecast is consistent with actual values. Consequently, it is reasonable that any measures of correct sign prediction could be regarded as desirable outputs, and any measures of goodness-of-fit and biasedness can be regarded as desirable inputs. Here, the goodness-of-fit conveys how close the forecasts are from the real values and the biasedness indicates whether over-estimation or under-estimation exists. The super-efficiency DEA model, which can discriminate among those efficient decision-making units is introduced in this subsection.

Suppose we have a set of $n$ DMUs. Let $\left(x_{ij},y_{rj}\right)$ denotes the $i$-th input and the $r$-th output of the $j$-th DMU. For the $DM{U}_k$ under evaluation. The input-oriented super-efficiency DEA model can be expressed as

$$\begin{array}{cc}\mathit{min}& {\theta }_k\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}\le {\theta }_k{x}_{ik};\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge y_{rk};\forall r\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & {\theta }_{k}unrestricted\end{array}$$

(1)

Similarly, the output-oriented super-efficiency DEA model can be expressed as

$$\begin{array}{cc}\mathit{max}& {\varphi }_k\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}\le x_{ik};\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge {\varphi }_k{y}_{rk};\forall r\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & {\varphi }_{k}unrestricted\end{array}$$

(2)

The efficiency of $DM{U}_k$ is obtained by comparing with a virtual benchmark. ${\lambda }_j$ indicates the proportional weight of $DM{U}_j$ that consists of this virtual benchmark. The input-oriented efficiency score is the optimal value of (1), that is ${\theta }_k^{Super}={\theta }_k^{*}$. The output-oriented efficiency score is the reciprocal of the optimal value of (2), that is ${\varphi }_k^{Super}=\frac{1}{{\varphi }_k^{*}}$.

In the following, we provide two approaches to evaluate the performance of competing crude oil prices’ volatility forecasting models. One is to regard the biasedness and goodness-of-fit level as inputs and the correct sign as outputs. The other is treat all of them as outputs.

2.1.1. Approach I

The inputs in this application of competing crude oil prices’ volatility forecasting models’ performance evaluation seem to be undesirable. The undesirability of inputs means that the increasing of the biasedness and goodness-of-fit level will lead to a corresponding drop of the correct sign. In contrast, the closer the forecasts are from the actual values, as well as less bias, the higher the correct sign should be. As for special instances, if the levels of biasedness and goodness-of-fit equal to zero, which means that the forecasting results are exactly the real prices, then the correct sign prediction reaches the highest score, that is the unity. If inefficiency exists in this process, the inputs, namely goodness-of-fit and biasedness, should be increased to improve the performance, which reveals the undesirability feature of inputs. Consequently, any measures of correct sign prediction should be treated as desirable outputs, while any measures of goodness-of-fit and biasedness should be regarded as undesirable inputs.

There are lots of discussions on undesirable inputs and undesirable outputs, please refer to [20,21] for more details. Here we treat undesirable inputs as desirable outputs, and undesirable outputs as desirable inputs by following [22]. The efficiency can be obtained by calculating the following models in

Table 1. Super efficiency DEA models with undesirable inputs.

Input-Oriented (Model 1)

Output-Oriented (Model 2)

$\begin{array}{cc}\mathit{max}& {\theta }_k\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}^U\ge {\theta }_k{x}_{ik}^U;\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge y_{rk};\forall r\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j=1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ \hfill {\theta }_k& unrestricted\hfill \end{array}$

$\begin{array}{cc}\mathit{max}& {\varphi }_k\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}^U\ge x_{ij}^U;\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge {\varphi }_k{y}_{rk};\forall r\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j=1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ \hfill {\varphi }_k& unrestricted\hfill \end{array}$

, where $x_{ij}^U$ indicates the undesirable parts of inputs. The input-oriented efficiency can be defined as ${\theta }_k^{Super}=\frac{1}{{\theta }_k^{*}}$ and the output-oriented efficiency is given by ${\varphi }_k^{Super}=\frac{1}{{\varphi }_k^{*}}$.

In Model 1, for an inefficient DMU, ${\theta }_k^{*}$ indicates the required proportional increase in undesirable inputs so as to become efficient. In Model 2, for an inefficient DMU, ${\varphi }_k^{*}$ indicates the possible required proportional increase in desirable outputs in order to become efficient.

2.1.2. Approach II

As aforementioned, if we treat the goodness-of-fit, the biasedness and the correct sign all as outputs, evaluating the performance corresponding to the practice. That is, considering the performance evaluation process as a process without explicit inputs, we treat measures of goodness-of-fit and biasedness as undesirable outputs, while those of correct sign desirable outputs. Specifically, a better performed crude oil prices’ volatility forecasting model possesses lower levels of goodness-of-fit and biasedness, while higher level of correct sign at the same time.

To formulate the situation with pure output data, we consider the DEA without explicit inputs (WEI) model developed by [23]. The super efficiency DEA WEI models with undesirable outputs are presented in

Table 2. Super efficiency DEA WEI models with undesirable outputs.

DEA-WEI (Model 3)

DEA-WEI (Model 4)

$\begin{array}{cc}\mathit{max}& {\varphi }_{k1}\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{pj}^g\ge {\varphi }_{k1}{y}_{pk}^g;\forall p\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{qj}^b\le y_{qk}^b;\forall q\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j\le 1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ \hfill {\varphi }_{k1}& unrestricted\hfill \end{array}$

$\begin{array}{cc}\mathit{min}& {\varphi }_{k2}\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{pj}^g\ge y_{pk}^g;\forall p\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{qj}^b\le {\varphi }_{k2}{y}_{qk}^b;\forall q\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j\le 1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ \hfill {\varphi }_{k2}& unrestricted\hfill \end{array}$

, where $y_p^g$ and $y_q^b$ represents the $p$th desirable (good) output and $q$th undesirable (bad) output, respectively. The WEI score of Model 3 which measures the desirable outputs can be defined as ${\varphi }_{k1}^{Super}=\frac{1}{{\varphi }_{k1}^{*}}$. The WEI score of Model 4 which measures the undesirable outputs is given by ${\varphi }_{k2}^{Super}={\varphi }_{k2}^{*}$ ${\varphi }_{k2}^{*}$.

In Model 3, for an inefficient DMU, the super efficiency score ${\varphi }_{k1}^{*}$ shows the required proportional increase in desirable outputs to be efficient. In Model 4, for an inefficient DMU, the efficiency score ${\varphi }_{k2}^{*}$ indicates the required proportional decrease in undesirable outputs to be efficient.

2.2. Dealing with Infeasibility and Zero Data

As indicated in [17,18], the input-oriented super-efficiency model may be infeasible when the outputs of the DMU under evaluation rest outside the production possibility set spanned by the other DMUs. And in the same spirit, the output-oriented super-efficiency model might be infeasible when the inputs of the DMU under evaluation lies outside the production possibility set spanned by the other DMUs. The current study finds that with undesirable inputs, infeasibility in the input-oriented model (Model 1) indicates super-efficiency can be regarded as output surplus while infeasibility in the output-oriented model (Model 2) indicates super-efficiency can be regarded as undesirable input surplus. As the approach in [19,24,25], when infeasibility occurs, corresponding models of Model 1, Model 2, Model 3, and Model 4 are presented as follows—see

Table 3. Modified version of super-efficiency DEA models with undesirable inputs.

Input-Oriented (Model 5)

Output-Oriented (Model 6)

$\begin{array}{cc}\mathit{max}& {\tau }_k-M{\displaystyle {\displaystyle \sum }_r}{\beta }_r\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}^I\ge \left(1-{\tau }_k\right)x_{ik}^I;\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge \left(1-{\beta }_r\right)y_{rk};\forall r\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j=1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & {\beta }_r\ge 0;\forall r\hfill \\ & {\tau }_{k}unrestricted\hfill \end{array}$

$\begin{array}{cc}\mathit{max}& {\gamma }_k-M{\displaystyle {\displaystyle \sum }_i}{t}_i\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{x}_{ij}^I\ge \left(1-t_i\right)x_{ik}^I;\forall i\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{rj}\ge \left(1-{\gamma }_k\right)y_{rk};\forall r\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j=1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & t_i\ge 0;\forall i\\ & {\gamma }_{k}unrestricted\end{array}$

and

Table 4. Modified version of super-efficiency DEA WEI model with undesirable outputs.

Output-Oriented (Model 7)

Output-Oriented (Model 8)

$\begin{array}{cc}\mathit{min}& {\gamma }_{k1}+M{\displaystyle {\displaystyle \sum }_q}{t}_q\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{pj}^g\ge \left(1-{\gamma }_{k1}\right)y_{pk}^g;\forall p\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{qj}^b-t_q{y}_q^{b_{\max }}\le y_{qk}^b;\forall q\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j\le 1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & t_i\ge 0;\forall i\\ \hfill {\gamma }_{k1}& \mathrm{unrestricted}\hfill \end{array}$

$\begin{array}{cc}\mathit{min}& {\gamma }_{k2}+M({\displaystyle {\displaystyle \sum }_p}{\beta }_p+{\displaystyle {\displaystyle \sum }_q}{\delta }_q)\\ s.t.& {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{pj}^g\ge \left(1-{\beta }_p\right)y_{pk}^g;\forall p\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j{y}_{qj}^b-{\delta }_q{y}_q^{b_{\max }}\le \left(1+{\gamma }_{k2}\right)y_{qk}^b;\forall q\\ & {\displaystyle {\displaystyle \sum }_{\begin{array}{c}j=1\\ j\ne k\end{array}}^n}{\lambda }_j\le 1\\ & {\lambda }_j\ge 0;\forall j\ne k\\ & t_i\ge 0;\forall i\\ \hfill {\gamma }_{k2}& \mathrm{unrestricted}\hfill \end{array}$

where $y_q^{b_{\max }}=ma{x}_{k=1}^n\left\{y_{qk}^b\right\}$. This is proposed to deal with the infeasibilities raised by zero values.

, where $M$ is a large positive number defined by a decision maker ${10}^5$.

Let $I=\left\{i|t_i^{*}>0\right\}$ and $R=\left\{r|{\beta }_r^{*}>0\right\}$, as denoted in [19]. We have the output surplus index calculated with ${\beta }_r^{*}$ obtained from Model 5:

$$o=\{\begin{array}{cc}0& ifR=\varnothing \\ \frac{1}{\left|R\right|}{\displaystyle {\displaystyle \sum }_{i\in R}}(\frac{1}{1-{\beta }_r^{*}})& fR\ne \varnothing \end{array}$$

and undesirable input surplus index calculated with $t_i^{*}$ obtained from Model 6:

$${\widehat{i}}_u=\{\begin{array}{cc}0& ifI=\varnothing \\ \frac{1}{\left|I\right|}{\displaystyle {\displaystyle \sum }_{i\in I}}(\frac{1}{1-t_i^{*}})& fI\ne \varnothing \end{array}$$

Then the modified input-oriented score from Model 5 can be defined as ${\breve{\theta }}_k^{Super}=\frac{1}{1-{\tau }^{*}}+o$. The modified output-oriented score ${\breve{\varphi }}_k^{Super}$ from Model 6 is given by ${\breve{\varphi }}_k^{Super}=\frac{1}{{\gamma }^{*}}+{\widehat{i}}_u$.

Denote $Q=\left\{i|t_q^{*}>0\right\}$ and $P=\left\{p|{\beta }_p^{*}>0\right\}$, we have the undesirable output saving index

$$\widehat{i}=\{\begin{array}{cc}0& ifQ=\varnothing \\ \frac{1}{\left|Q\right|}{\displaystyle {\displaystyle \sum }_{q\in Q}}\left(1+t_q^{*}\right)& fQ\ne \varnothing \end{array}$$

and the desirable output surplus index

$$\widehat{o}=\{\begin{array}{cc}0& ifP=\varnothing \\ \frac{1}{\left|P\right|}{\displaystyle {\displaystyle \sum }_{p\in P}}\left(\frac{1}{1-{\beta }_p^{*}}\right)& fP\ne \varnothing \end{array}$$

The output-oriented score of Model 7 can be defined as ${\breve{\varphi }}_{k1}^{Super}=\frac{1}{1-{\gamma }_{k1}^{*}}+\widehat{i}$. The modified undesirable output-oriented score ${\breve{\varphi }}_{k2}^{Super}$ of Model 8 is given by ${\breve{\varphi }}_{k2}^{Super}={\gamma }_{k2}^{*}+\widehat{o}+\widehat{i}$.

3. Results

This section is dedicated to the presentation of the empirical results. Fourteen competing crude oil prices’ volatility forecasting models under evaluation and corresponding performance measures used are consistent with those in [10]. The final rankings obtained from applying Model 5, Model 6, Model 3, and Model 7 are presented in the following tables. The models are solved and programmed with Matlab.

Table 5. Rankings of competing crude oil prices’ volatility forecasting models (Model 5).

Undesirable Inputs	Desirable Output	Rank from Best to Worst
ME, MSE	PCDCP	3→1→2→5→4→7→8→12→10→9→6→14→13→11
ME, MSVolScE	PCDCP	3→1→2→5→4→7→8→12→10→9→6→14→13→11
ME, MAE	PCDCP	3→1→2→4→12→14→5→10→9→13→7→8→6→11
ME, MAVolScE	PCDCP	3→1→2→4→14→13→(12/10)→5→9→8→7→6→11
ME, MMEU	PCDCP	3→1→2→5→8→7→4→12→10→9→6→14→13→11
ME, MMEO	PCDCP	3→1→13→4→7→8→5→14→10→12→9→6→11→2
MVolScE, MSE	PCDCP	3→1→2→5→4→7→8→12→10→9→6→14→13→11
MVolScE, MSVolScE	PCDCP	3→1→2→5→4→7→8→12→10→9→6→14→13→11
MVolScE, MAE	PCDCP	3→1→2→4→12→14→5→10→9→13→7→8→6→11
MVolScE, MAVolScE	PCDCP	3→1→2→4→14→13→(12/10)→5→9→8→7→6→11
MVolScE, MMEU	PCDCP	3→1→2→5→8→7→4→12→10→9→6→14→13→11
MVolScE, MMEO	PCDCP	3→1→13→4→7→5→8→14→10→9→12→6→11→2

summarizes rankings of competing forecasting models obtained by Model 5. The result show that forecasting Model 3 is always the best, no matter how we measure three criteria. Forecasting Model 1 systematically hits the second place whereas forecasting model 11 is always the worst. In addition, the rankings with MMEU and MMEO differ greatly, which suggests the performance of competing models, $HM$ and $PARCH\left(1,1\right)$ for example, are very sensitive to whether one penalizes positive errors more heavily than negative ones, or vice versa.

Table 6. Rankings of competing crude oil prices’ volatility forecasting models (Model 6).

Undesirable Inputs	Desirable Output	Rank from Best to Worst
ME, MSE	PCDCP	1→3→8→5→12→10→7→9→2→14→4→13→6→11
ME, MSVolScE	PCDCP	1→3→8→5→12→10→7→9→2→14→4→13→6→11
ME, MAE	PCDCP	1→2→3→4→8→5→12→14→10→13→9→7→6→11
ME, MAVolScE	PCDCP	1→2→3→4→8→5→(14/13)→12→10→9→7→6→11
ME, MMEU	PCDCP	1→3→8→5→12→10→7→4→9→14→2→6→13→11
ME, MMEO	PCDCP	1→3→8→5→12→10→7→9→14→4→13→6→11→2
MVolScE, MSE	PCDCP	1→3→5→8→7→(12/10)→9→2→4→14→13→6→11
MVolScE, MSVolScE	PCDCP	1→3→5→8→7→(12/10)→9→2→4→14→13→6→11
MVolScE, MAE	PCDCP	1→2→3→4→5→8→12→14→13→10→7→9→6→11
MVolScE, MAVolScE	PCDCP	1→2→3→4→5→8→13→14→(12/10)→9→7→6→11
MVolScE, MMEU	PCDCP	1→3→5→8→7→(12/10)→9→4→14→2→13→6→11
MVolScE, MMEO	PCDCP	1→3→5→8→7→10→12→9→4→14→13→6→11→2

summarizes the rankings of competing forecasting models obtained by Model 6. Notice that Model 1 is the best no matter how we measure goodness-of-fit, and is systematically followed by forecasting model 3. Whereas forecasting Model 6 and Model 11 are the worst. Further, the performance of Model 2 and Model 13 are relatively sensitive to whether we penalize positive errors more heavily than negative ones or vice versa.

With undesirable outputs, infeasibility only exists when measuring the undesirable outputs (Model 4). Consequently, we employ Model 8 rather than Model 4 to obtain a full ranking. While using the desirable outputs-oriented, the model used remains the same, that is Model 3.

Table 7. Rankings of competing crude oil prices’ volatility forecasting models (Model 3).

Undesirable Outputs	Desirable Outputs	Rank from Best to Worst
ME, MSE	PCDCP	3→14→5→10→12→9→13→4→6→11→8→7→1→2
ME, MSVolScE	PCDCP	3→14→5→10→12→9→13→4→6→11→8→7→1→2
ME, MAE	PCDCP	3→5→10→12→9→14→4→6→8→13→11→1→7→2
ME, MAVolScE	PCDCP	3→5→10→12→9→14→4→13→6→8→11→7→1→2
ME, MMEU	PCDCP	3→5→10→12→9→14→4→13→6→8→11→1→7→2
ME, MMEO	PCDCP	3→2→1→11→5→8→6→9→12→10→7→14→4→13
MVolScE, MSE	PCDCP	3→14→5→10→12→9→13→4→6→11→8→7→1→2
MVolScE, MSVolScE	PCDCP	3→14→5→10→12→9→13→4→6→11→8→7→1→2
MVolScE, MAE	PCDCP	3→5→10→12→9→14→4→6→8→13→11→1→7→2
MVolScE, MAVolScE	PCDCP	3→5→10→12→9→14→4→13→6→8→11→7→1→2
MVolScE, MMEU	PCDCP	3→5→10→12→9→14→4→13→6→8→11→1→7→2
MVolScE, MMEO	PCDCP	3→2→1→11→5→8→6→12→9→10→7→14→4→13

summarizes the rankings of forecasting models obtained by Model 3. Model 3 is the best and is followed by Model 5. Model 2 is systematically the worst. The rankings under MMEU and MMEO differ significantly. This indicates that the relative performance of models, $HM$ and $RW$ for example, are very sensitive to whether we penalize positive errors more than negative ones or vice versa.

Table 8. Rankings of competing crude oil prices’ volatility forecasting models (Model 8).

Undesirable Outputs	Desirable Outputs	Rank from Best to Worst
ME, MSE	PCDCP	3→14→5→10→12→9→13→11→6→4→8→7→1→2
ME, MSVolScE	PCDCP	3→14→5→10→12→9→13→11→6→4→8→7→1→2
ME, MAE	PCDCP	3→5→10→9→12→6→14→8→13→4→11→7→1→2
ME, MAVolScE	PCDCP	3→5→(12/10)→9→6→14→8→4→13→11→7→1→2
ME, MMEU	PCDCP	3→5→14→10→12→9→13→4→11→6→8→7→2→1
ME, MMEO	PCDCP	3→1→2→11→5→8→6→9→12→10→7→14→4→13
MVolScE, MSE	PCDCP	3→14→5→10→12→9→13→11→6→4→8→7→1→2
MVolScE, MSVolScE	PCDCP	3→14→5→10→12→9→13→11→6→4→8→7→1→2
MVolScE, MAE	PCDCP	3→5→10→9→12→6→14→8→13→4→11→7→1→2
MVolScE, MAVolScE	PCDCP	3→5→(12/10)→9→6→14→8→4→13→11→7→1→2
MVolScE, MMEU	PCDCP	3→5→14→10→12→9→13→4→11→6→8→7→2→1
MVolScE, MMEO	PCDCP	3→1→2→11→5→8→6→12→9→10→14→7→4→13

summarizes the rankings of competing forecasting models obtained by Model 8. Forecasting model 3 is always the best no matter what measure we use. Whereas forecasting model 2 is systematically the worst. Also, rankings under MMEU and MMEO differ significantly, which indicates the performance of models are very sensitive to whether one penalize positive errors more than negative ones or vice versa.

To test whether the results of our models are statistically different from that in [10], we run the Wilcoxon signed ranks tests. The results are shown in

Table 9. Wilcoxon signed ranks test statistics.

	Model 5	Model 6	Model 3	Model 8
ME, MSE	0.433	0.002	0.001	0.016
ME, MSVolScE	0.016	0.084	0.779	0.023
ME, MAE	0.002	0.006	0.084	0.016
ME, MAVolScE	0.001	0.001	0.002	0.016
ME, MMEU	0.221	0.002	0.004	0.016
ME, MMEO	0.002	0.013	0.144	0.500
MVolScE, MSE	0.433	0.002	0.001	0.001
MVolScE, MSVolScE	0.013	0.272	0.917	0.030
MVolScE, MAE	0.001	0.001	0.064	0.016
MVolScE, MAVolScE	0.001	0.001	0.004	0.001
MVolScE, MMEU	0.056	0.001	0.001	0.001
MVolScE, MMEO	0.001	0.013	0.465	0.001

. The statistical results reveal that at 10% significance level, the rankings obtained from this contribution are significantly different from those obtained by [10]. The previous results are not considered as reliable, mainly because there are infeasibilities in their results. In addition, the desirability of inputs and outputs are not discussed in the previous studies. We have clarified this point in our new models.

4. Conclusions

As a multi-criteria evaluation method, the super efficiency DEA method is an innovative way for achieving a consistent evaluation of crude oil prices’ volatility forecasting models. However, the infeasibility problem in the original model decreases the reliability of its ranking results. Further, the desirability of inputs and outputs are of great significance to its application. This contribution focuses on these two aspects and proposes a modified super efficiency DEA framework for evaluating forecasting models. Four models corresponding to undesirable inputs and undesirable outputs are developed, as well as the way to address the appearance of infeasibility. Several conclusions are obtained from the empirical analysis. First, the ranking of the best and the worst forecasting models appears to be robust with respect to different measures under the same model. Second, our empirical results seem to suggest that considering models with undesirable inputs, $RW$ and $SAM20$ outperform other forecasting models and $EGARCH\left(1,1\right)$ is systematically the worst one. With respect to models dealing with undesirable outputs, $RM$ turns into one of the worst models while $SAM20$ remains as the best one. Finally, the Wilcoxon signed ranks tests reveal that rankings obtained by applying our framework are significantly different from the previous studies.