Forecast Combinations for Structural Breaks in Volatility: Evidence from BRICS Countries

Abstract

The aim of this paper is to investigate the relevance of structural breaks for forecasting the volatility of daily returns on BRICS countries (Brazil, Russia, India, China and South Africa). The data set used in the analysis is the Morgan Stanley Capital International MSCI daily returns and covers the period from 19 July 1999 to 16 July 2015. To identify structural breaks in the unconditional variance, a binary segmentation algorithm with a test, which considers both the fourth order moment of the process and persistence in the variance, has been implemented. Some forecast combinations that account for the identified structural breaks have been introduced and their performance has been evaluated and compared by using the Model Confidence Set (MCS). The results give significant evidence of the relevance of the structural breaks. In particular, in the regimes identified by the structural breaks, a substantial change in the unconditional variance is quite evident. In forecasting volatility, the combination that averages forecasts obtained using different rolling estimation windows outperforms all the other combinations

1. Introduction

Measuring and forecasting volatility in financial markets has attracted growing interest by academics, policy makers and practitioners during the last few decades. Volatility in the stock, bond and foreign exchange market can be used as a measurement of risk and its impact on the economy and on the stability of financial markets is an important public policy issue; it also plays a central role in the pricing of derivative securities. As a consequence, there is a large amount of literature on forecasting volatility and, in the last few decades, a lot of volatility models based on low-frequency data, such as generalized autoregressive conditional heteroskedasticity (GARCH) models, have been proposed. However, the financial markets are buffeted by suddenly important events that can lead to sharp breaks in the markets and thus breaks in parameters governing the volatility models can occur. In such cases, the inference on the parameters may be misleading as well as any policy implications drawn from the model. The accuracy of post-sample forecasting is affected as well. The presence of structural breaks in volatility of financial time series has been confirmed in many empirical studies. Examples can be found in the case of stock returns (Hammoudeh and Li 2008, Wang and Moore 2009), in the exchange rate returns (Rapach and Strauss 2008), in sovereign bond markets (Tamakoshi and Hamori 2014), in the real Gross Domestic Product growth (Fang and Miller 2009) and in the Realized volatility (De Gaetano 2018a). Strong evidence of the existence of multiple break points has been highlighted for BRICS countries (Morales and Gassie-Falzone 2011) which have experienced severe crises in the last 20 years.

The BRICS countries (Brazil, Russia, India, China and South Africa) are recognized as the most developed economies among the emerging markets. They are all developing or newly industrialized countries and are all characterized by large, fast-growing economies and strong influence on regional and global business. All together, they account for $26.46\%$ of world land area, $42.58\%$ of world population, $13.24\%$ of World Bank voting power and $14.91\%$ of International Monetary Fund (IMF) quota shares. According to IFM’s estimates, BRICS countries generated $22.53\%$ of the world GDP in 2015 and has contributed more than $50\%$ of world economic growth during the last 10 years1. Moreover, the importance of these countries in the World economy and in particular in the global financial market has been highlighted in many papers in the last few decades. For example, many studies have recognized the strong linkage between the BRICS economies the US and Eurozone economy (see Bhuyan et al. 2016; Ahmad et al. 2013 inter alia) and their significant dependence with the global stock and commodity markets (Mensi et al. 2014).

The aim of this paper is twofold. Our first objective is to confirm the presence of structural breaks in the volatility of MSCI daily returns on BRICS countries. Secondly, we investigate the relevance of structural breaks in forecasting by considering opportune forecast combinations which take into account their presence. The data are the Morgan Stanley Capital International (MSCI) daily returns on the BRICS countries and cover the period from 19 July 1999 to 16 July 2015. The analysis begins by identifying structural breaks in the unconditional variance. More specifically, a test for a single break in the presence of conditional heteroskedasticity has been employed. It is based on a statistic proposed in Sansó et al. (2004) which takes into account both the fourth order moment of the process and persistence in the variance. In order to extend the single break point method to multiple ones, a binary segmentation algorithm has been implemented. Following (Rapach and Strauss 2008), the identified structural breaks have been used to identify different sub-samples in which different GARCH(1,1) models have been estimated. The GARCH(1,1) model has been considered since it is a very parsimonious model and usually it is adequate to obtain good performances in terms of fitting and forecasting. In order to account for possible structural breaks in the generation of forecasts, we focus on some particular forecast combinations generated by the same model but over different estimation windows. This strategy has been successfully used in many empirical works in different areas. For example, in a regression framework, some methods have been proposed to choose the individual forecasts entering the combination and the relative weights (Pesaran and Timmermann 2007; Pesaran and Pick 2011; Pesaran et al. 2013; Tian and Anderson 2014). Applications in forecasting the equity premium can be found in Tian and Zhou 2016 and in the context of realized volatility in De Gaetano (2018a). Also in a GARCH framework, combinations of forecasts have been shown to improve forecast accuracy with respect to a single model (see for example Rapach and Strauss 2008; Rapach et al. 2008; De Gaetano 2018b).

In this paper, we focus on combinations that average the forecasts obtained by the individual GARCH(1,1) model and those obtained by considering different estimation windows. This approach is in the spirit of Clark and McCracken (2004) who present analytical, Monte Carlo, and empirical evidence that combining recursive and rolling forecasts, when linear predictive models are subject to structural change, could be superior to the individual forecasts. The forecasting performance of the proposed forecast combinations has been evaluated and compared by using the Model Confidence Set (MCS) proposed by Hansen et al. (2011). This procedure allows for constructing a set of models from a specified collection that consists of the best models in terms of a loss function and a given level of confidence. In particular, for each country and for different horizons, the forecasting performance of the considered methods have been evaluated with respect to different loss functions, the MSE and the QLIKE.

The structure of the paper is as follows: Section 2 presents the data and the methodology employed in the analysis. In particular, in Section 2.1, the methodology used to identify the structural breaks is illustrated and the results of an in-sample analysis for BRICS countries is presented; in Section 2.2, the proposed forecast combinations are illustrated and examined. In Section 3, a comparison among the proposed forecast combinations and two alternative benchmark methods are discussed. Some remarks close the paper.

2. Data and Methodology

The data set is obtained from Datastream and consists of MSCI (Morgan Stanley Capital International) daily returns on the BRICS countries. BRICS is the acronym for an association of five major emerging national economies: Brazil, Russia, India, China and South Africa. The grouping was originally known as “BRIC” before the inclusion of South Africa in 2010.

These countries are all developing or newly industrialized countries and they are all characterized by large, fast-growing economies and a strong influence on regional and global business. The data covers the period from 19 July 1999 to 16 July 2015.

Table 1. Summary statistics. MSCI daily returns on BRICS countries from 19 July 1999 to 16 July 2015. p-values of the Jarque–Bera test are given in brackets.

	Brazil	Russia	India	China	South Africa
Min	−14.0700	−25.2800	−12.0500	−12.8300	−8.4480
1st Quant.	−0.8301	−0.9287	−0.6768	−0.8268	−0.6103
Median	0.0000	0.0683	0.0385	0.0111	0.0176
Mean	0.0364	0.0369	0.0438	0.0110	0.0456
3rd Quant.	0.9455	1.1310	0.8421	0.9111	0.7451
Max	13.4400	23.9500	16.4200	14.0400	5.9620
Standard Devation	1.6507	2.3788	1.5710	1.8429	1.2406
Skewness	−0.1283	−0.2913	−0.2244	0.0086	−0.1560
Kurtosis	5.8651	13.0661	7.4304	5.4583	3.1091
Jarque–Bera test	6002.84	29786.80	9650.02	5189.17	1701.21
	(0.000)	(0.000)	(0.000)	(0.000)	(0.000)

reports the descriptive statistics for MSCI daily returns on the BRICS countries. It shows the usual properties of daily financial returns data that are a small mean, a large standard deviation and evidence of non-normality as pointed out by the Jarque–Bera test. This feature is essentially due to an excess of kurtosis that highlights the presence of a large number of significant shocks.

2.1. In-Sample Analysis

Let $\left\{a_t\right\}$, $t=1,\cdots ,T$ denote the time series of the daily returns and assume, for simplicity, that the unconditional and conditional mean are zero.

In order to model the dynamic of $\left\{a_t\right\}$, a GARCH(1,1) model has been used; it has been established as an adequate model to obtain good performances in terms of fitting and forecasting. The canonical GARCH(1,1) model is:

$$\begin{array}{cc}\hfill a_t& ={\sigma }_t{ϵ}_t\hfill \\ \hfill {\sigma }_t^2& =\omega +\beta {\sigma }_{t-1}^2+\alpha a_{t-1}^2,\hfill \end{array}$$

(1)

where $\left\{{ϵ}_t\right\}$ is a sequence of i.i.d. random variables with mean zero and unit variance. Conditions on $\omega$, $\alpha$ and $\beta$ need to be imposed for the previous equation to be well defined. In particular, $\omega >0$ and $\alpha ,\beta \ge 0$ are imposed to ensure that the conditional variance ${\sigma }_t^2$ is positive. Moreover, $\alpha +\beta <1$ ensures that the process is stationary. For a GARCH(1,1) process, the unconditional variance is defined as $\omega /(1-\alpha -\beta )$.

The parameters of model (1) are estimated by using the Quasi Maximum Likelihood Estimation in which the likelihood corresponding to the assumed distribution of ${ϵ}_t$ is maximized under the previous assumptions.

We are interested in testing whether the unconditional variance is constant over the available sample since a constant unconditional variance implies a not stable GARCH process governing conditional volatility. In order to identify possible structural breaks, a test for a single break in the presence of conditional heteroskedasticity has been employed. It is based on a statistic proposed in Sansó et al. (2004) which takes into account both the fourth order moment of the process and persistence in the variance. The test is based on the following statistic:

$$K_2=su{p}_k\left|T^{-1/2}{G}_k\right|,$$

(2)

where $C_k={\sum }_{t=1}^k{a}_t^2$ for $k=1,\dots ,T$ is the cumulative sum of squares of $a_t$ and $G_k={\widehat{\omega }}_4^{-1/2}[C_k-(k/T)C_T]$ and ${\widehat{\omega }}_4$ is a consistent estimator of ${\omega }_4$, the long-run fourth order moment of $a_t$.

Under quite general conditions, in Sansó et al. (2004), it has shown that:

$$K_2\stackrel{A}{\to }su{p}_r\left|W^{*}\left(r\right)\right|,$$

(3)

where $W^{*}=W\left(r\right)-rW\left(1\right)$ is a Brownian bridge and $W\left(r\right)$ is a standard Brownian motion. Finite-sample critical values for the test can be determined by simulation.

The $K_2$ statistic is a generalization of the IT statistic proposed in Inclan and Tiao (1994), generally used to test the constancy of the unconditional variance of a time series. In particular, $K_2$ makes adjustments to the IT statistic to allow $a_t$ to obey a wide class of dependent processes, including GARCH processes, under the null.

In order to obtain a consistent estimator of the long-run fourth order moment of $a_t$, which is also the long-run variance of the zero mean random variable $\xi =a_t^2-{\sigma }^2$, a non-parametric approach based on the Bartlett kernel (see also Rapach and Strauss 2008) has been used used. In particular, it is:

$${\widehat{\omega }}_4={\widehat{\gamma }}_0+2\sum _{l=1}^m[1-l{(m+1)}^{-1}]{\widehat{\gamma }}_l,$$

(4)

where

$${\widehat{\gamma }}_l=T^{-1}\sum _{t=l+1}^T({ϵ}_t^2-{\widehat{\sigma }}^2)({ϵ}_{t-1}^2-{\widehat{\sigma }}^2)$$

(5)

and ${\widehat{\sigma }}^2=T^{-1}{C}_T$. This estimator depends on the bandwidth m which can be selected using the procedure in Newey and West (1994).

In order to extend the single break point method to multiple ones, a binary segmentation algorithm has been implemented. It is based on successive application of the test to sub-series obtained consecutively after a change-point is found. The procedure starts by applying the detection method to the whole series. If no change-point is found, the procedure is stopped; otherwise, the data are split into two segments and the detection method is applying to each of them. The procedure is repeated until no further change-points are detected. The choice of a binary segmentation algorithm is justified by its simplicity and efficiency; it is very fast and it could be implemented with a low computational cost. However, the procedure could produce spurious break points because of the presence of extreme observations which can be erroneously interpreted as being change points (see Ross 2013). To partially solve this problem and to better identify the break points location, a pruning procedure, in the spirit of the ICCS algorithm (Inclan and Tiao 1994), has been implemented. In the case that m breaks have been detected at times ${\tau }_1$, …, ${\tau }_m$ with ${\tau }_0=1$ and ${\tau }_{m+1}=T$, the pruning procedure can be implemented as follows:

• The detection method is applied to the segment $({\tau }_{i-1},{\tau }_{i+1})$ for $i=1,\dots ,m$.
• If no change-point is found in the segment $({\tau }_{i-1},{\tau }_{i+1}),$ the break at ${\tau }_i$ is not considered a change point. If a new change point is detected, it replaces the old one at ${\tau }_i$.
• The procedure is repeated until the number of change points does not change and the points found in each new step are ”close” to those on the previous step.

The main problem, when a detection method is applied with a searching algorithm, is that the use of the same critical value for any segments may distort the performance of the iterative procedure. To overcome this problem, the response surfaces’ methodology has been used (see MacKinnon 1994 for details).

Table 2. Volatility breaks dates identified by the binary segmentation with a $K_2$ test.

Brazil	Russia	India	China	South Africa
4 November 2002	6 March 2001	30 April 2001	15 November 2001	12 December 2007
23 July 2007	30 January 2002	14 January 2008	4 July 2003	15 July 2009
8 September 2008	1 August 2008	24 August 2009	24 June 2004
24 November 2008	25 November 2008	3 August 2011	17 April 2006
4 June 2009	9 November 2009	30 March 2012	26 July 2007
			17 August 2009
			21 June 2010
			4 August 2011
			17 Janurary 2012
			27 March 2015

reports the identified breaks dates.The proposed procedure identified five breaks for Brazil, Russia and India, ten breaks for China and only two breaks for South Africa. It is evident that the volatility may change in each country according not only to global, but also to specific financial, economic, social and political events.

In general, all of the BRICS countries, as the rest of the emerging markets, largely stood at the fringes of the global financial crisis that started in 2007 with the US Subprime market collapse and developed into a full-blown international banking crisis with the collapse of the investment bank Lehman Brothers on September 2008. This crisis affected global stock markets, where securities suffered large losses during 2008 and early 2009 with a reduction of volatility after this period. Therefore, in the period from 2007 to 2009, in all of the BRICS countries, there is evidence of the presence of structural break in volatility with different peaks.

Regarding individual BRICS countries, the identified breaks can be explained by looking at specific domestic events. According to Morales and Gassie-Falzone (2011), the first break identified for Brazil is in 2002 and it could be due to the Brazilian stock market crash and the pressures in the run up to the presidential election. The Indian and Russian markets share a common trend with a break point in April and March 2001, respectively, which is connecting with the dot-com bubble effects and to the energy crisis (Morales and Gassie-Falzone 2011). Moreover, for Russia, a high magnitude of unsystematic risk was observed in 2001–2002 when the Russian stock market was hardly on the radar of international portfolio managers (Nivorozhkin and Castagneto-Gissey 2016). Regarding India, the breaks in August 2011 is related to the ”August 2011 stock markets fall”, which is the sharp drop in stock prices due to fears of contagion of the European sovereign debt crisis. The effect of this crisis continued for the rest of the year; the break in March 2012 corresponds to the end of the effect of the crisis and a consequential reduction of volatility. The results for the Chinese market seem to be quite different; the large number of identified breaks and their locations could be also explained by regional volatility. As pointed out in Zhou et al. (2012), from 1996 to 2009, the Chinese stock market was not much influenced by other markets because it was not completely open to foreign investors. More precisely (see Li (2015) for a complete review of the market), in 1999, the government formally put forward the pilot plan of transferring state-owned stocks, but it did not work well, and the scheme was a shock to the stock market. However, due to the discrepancy between the market’s expectations and the implementation plan, the pilot project was led to some variations on the initial proposal from 2001 to 2003. In 2004, to solve the non-tradable shares issue some institutional reforms were made and, during 2006 and 2007, the Chinese market experienced the emergence of the stock market ‘bubble’. From 2007 to 2009, the market was affected by the global financial crisis which caused a peak in the volatility. In general, volatility spillovers among the Chinese and the other Asiatic markets, in particular Japanese and Indian markets, are more distinctive than those among the Chinese and Western markets are. This consideration could explain the presence of breaks in August 2011 and at the beginning of 2012 as previously pointed out for the Indian market. The last break in March 2015 could be explained by the Chinese stock market turbulence which began with the popping of the stock market bubble. For South Africa, the only identified structural breaks are those linked to the global financial crisis in the period 2007–2009.

Figure 1 shows the two-standard-deviation bands for each of the regimes defined by the structural breaks.

Table 3. Unconditional variance for the GARCH(1,1) full sample model and for the sub-sample defined by the identified structural breaks. The values 0* are obtained in correspondence of an estimated GARCH(1,1) model in which the persistence in near to one (IGARCH model) and $\omega$ is equal to 0.

	Brazil	Russia	India	China	South Africa
Full sample	2.4381	5.1029	3.0342	3.4390	1.6642
Subsample 1	2.7732	14.1344	5.2101	5.9551	1.6023
Subsample 2	1.9296	5.6764	1.7496	1.6869	3.7474
Subsample 3	3.1879	3.5846	7.7452	3.7053	0.9962
Subsample 4	34.2801	58.9268	1.2207	1.0579
Subsample 5	0*	9.8722	0*	2.0532
Subsample 6	1.5731	1.9922	0.7991	9.0545
Subsample 7				2.6656
Subsample 8				1.5155
Subsample 9				5.0387
Subsample 10				1.1182
Subsample 11				3.8351

presents the full-sample GARCH(1,1) unconditional variance, as well as its values for the sub-samples defined by the structural breaks identified by the binary segmentation algorithm with the $K_2$ test, for all the considered series. As expected, a substantial change in the unconditional variance is quite evident. Note that, for Brazil, the unconditional variance of subsample 5 collapses to zero indicating that, in this period, the model is an IGARCH(1,1) without trend ($\omega =0$), a model with the so-called ”persistent variance” property in which the current information remains important for the forecasts of the conditional variances for all horizons. For the other periods, the unconditional variance varies from 1.57, in the subsample 6 to 34.28 in subsample 4. For Russia, the unconditional variance of the GARCH(1,1) full sample model is equal to 5.10, whereas, in the subsamples, it varies from 1.99 to 58.93, a value more than 10 times larger than that observed in the full sample. In the case of India, as for Brazil, in the subsample 5, the estimated model is an IGARH(1,1) without a trend as suggested by the zero value of the unconditional variance. A significant variability among the identified subsamples is still evident. For China, the unconditional variance of the GARCH(1,1) full sample model is equal to 3.44 and, again, significant differences are observable in the eleven identified subsamples. The same feature is also noticeable for South Africa in which the unconditional variance varies from 0.99 to 3.75.

These features confirm the relevance of variance breaks in the analysis of MSCI daily returns for all five of the BRICS countries.

2.2. Out-of-Sample Analysis

Once the presence of structural breaks in the MSCI daily returns on the BRICS countries has been verified, the problem is how to take them into account when generating forecasts. In this paper, we focus on particular forecast combinations generated by the same model but over different estimation windows, a strategy that has been proved superior to forecasts generated by a single estimation window (see, for example, De Gaetano (2018b) and the references therein). In particular, we compare out-of-sample forecasts of volatility generated by two benchmark forecasting methods and four competing forecasting combinations. More specifically, we divide the sample for a given time series into in-sample and out-of-sample portions, where the in-sample portion spans the first R observations and the out-of-sample portion the last observations. The two benchmark methods are the following:

• Expanding win. This method forms out-of-sample forecasts using a recursive (expanding) estimation window. For 1-step-ahead forecasts, an initial sample using data from t = 1 to t = R is used to estimate the model and the 1-step ahead out-of-sample forecast is produced. The sample is increased by one, the model is re-estimated and 1-step ahead forecasts are produced. The procedure continues until the end of the available out-of-sample period. This method ignores possible past breaks for forecasting and it is generally used when a stable GARCH(1,1) is assumed.
• RiskMetrics. This model is defined as:

  $${\sigma }_t^2=(1-\lambda )\sum _{k=1}^{\infty }{\lambda }^{k-1}{a}_{t-k}^2,$$

  (6)

  where $\lambda$ is a parameter such that $0<\lambda <1$ This model, also known as exponential variance smoother for its structure, uses the exponentially weighted moving average method that is meant to represent the finite memory of the market. The weights on past squared returns decline to zero.
  The model has also an interesting formulation which makes it successful in financial applications. It is easy to show that it is equivalent to:

  $${\sigma }_t^2=(1-\lambda )a_t^2+\lambda {\sigma }_{t-1}^2.$$

  (7)
  Based on the assumption of Normally distributed returns, the RiskMetrics model completely ignores the presence of fat tails in the distribution function, which is an important feature of financial data. Nevertheless, despite the evident over simplification embedded in its formulation, it was commonly found that the model has satisfactory performances in forecasting financial data and it has become widely used in applications.
  The model depends on a single parameter $\lambda$ that has to be estimated. By evaluating a large number of assets, RiskMetrics Group proposed to fix $\lambda =0.94$. In this case, no estimation is needed.

The performance of these two benchmark methods have been compared with that of six competing forecasting combinations, which, in the spirit of Clark and McCracken (2004), combine recursive and rolling forecasts.

• Exp-Roll 0.25. This combination is the average of the forecasts obtained by a GARCH(1,1) expanding window and a GARCH that uses a rolling estimation window equal to 0.25 of the size of the in-sample period. In this second method, an initial sample using data from $t=(1-0.25)R$ to $t=R$ is used to estimate the model and the 1-step ahead out-of-sample forecast is produced. The window is moved ahead one time period, the model is re-estimated using data from $t=(1-0.25)R+1,\dots ,R+1$ and 1-step ahead out-of-sample forecast is produced. The procedure continues until the end of the available out-of-sample period. This model is generally used to take potential and unknown breaks in the series into account.
• Exp-Roll 0.50. The forecasts for this combination are generated in the same way as those for the GARCH(1,1) 0.25 rolling window, but with a rolling window equal to one-half of the size of the in-sample period is used. With respect to the previous model, this choice allows for having a trade-off between an accurate estimate of the parameters due to a relative long estimation window and the possibility that the data come from different regimes.
• Exp-Roll 0.75. The forecasts for this method are generated the same as those for the GARCH(1,1) 0.25 rolling window model, with the exception that we use a rolling window equal to one-quarter of the size of the in-sample period. In this case, even if the estimation procedure is based on less observations, the problem of data from different regimes is overcome.
• Exp-Break. This combination is the average of two forecasting methods. The first is the GARCH(1,1) expanding window. In the second, the forecasts are generated by using an estimation window determined by the last break. More precisely, the size of the estimation window is determined by applying the binary segmentation algorithm with the $K_2$ test to the data available at the time the forecast is made. For 1-step-ahead forecasts, an initial sample using data from $t=1$ to $t=R$ is used to detect the breaks’ points. The estimation window for the parameters of the GARCH(1,1) model is comprised of observations from the final break to R. If no breaks are detected over this period, the parameters are estimated using observations from 1 to R. The sample is increased by one and a new break point search is applied to observations from 1 to $R+1$. The estimation window is formed by observations to the new final break point to $R+1$. The procedure continues until the end of the viable out-of-sample period. The procedure uses only observations available during the period being analyzed for the detection of the more recent break point; therefore, it does nor suffer from the so-called look-ahead bias. However, if the break is detected near the end of the in-sample period, the parameters of the GARCH(1,1) model are estimated with a relatively short sample.
• Mean-win. This is the average of the five individual forecasting methods using different window sizes: GARCH(1,1) with breaks, GARCH which uses three rolling estimation windows equal to 0.25, 0.50 and 0.75 of the size of the in-sample period and a GARCH(1,1) expanding window. This method (see Pesaran and Timmermann 2007) incorporates the trade-off between the bias and the variance of forecasting errors because windows of earlier data are generally included in computing the combination forecasts.
• Trimmed-Mean-win. This is the average of the individual forecasts that result from excluding the highest and lowest ones from the considered mean-windows’ forecasts. This approach, in the spirit of Ahmad (1989), could be useful since it mitigates the influence of occasional outliers and, as a consequence, it is less sensitive to possible implausible forecasts.

3. Comparing Forecasting Models

The two benchmark methods and the proposed forecast combinations have been compared through a small forecasting exercise in which the out-of-sample sample spans the last p observations that cover the period from 1 March 2012 to the end of the sample, for a total of $p=873$ observations. In this context, the volatility forecast over the out-of-sample period has been calculated for various day horizons (s = 1, 5, 20). The comparison is made by using the Model Confidence set, proposed in Hansen (2005) and Hansen et al. (2011). The objective of this procedure is to determine which methods, from an initial set $M^0$ of methods indexed by $i=1,\dots ,M^0$, exhibit the same predictive ability in terms of a loss function, given a level of confidence. Let us consider $\widehat{M*}$ as the collection of the best methods, $M^0$ the initial collection of all the methods and $L_{i,t}$ the loss function associated with the method i in period t.

Define the relative performance variables as $d_{ij,t}=L_{i,t}-L_{j,t}\forall i,j\in M^0$ and assume that $E\left(d_{ij,t}\right)$ is finite and does not depend on t. The set of the best models is defined by:

$$\widehat{M*}=\left\{i\in M^0:E\left(d_{ij,t}\right)\le 0\forall j\in M^0\right\}.$$

(8)

In order to determine $\widehat{M*}$, a sequence of significance tests is made and the models that result to be significantly inferior to other elements of $M^0$ are eliminated. The MSC is a stepwise procedure which starts by setting $M=M_0$. The test $H_{0,M}$ is then implemented at level $\alpha$. If $H_{0,M}$ is not rejected, ${\widehat{M}}_{1-\alpha }^{*}=M$; if $H_{0,M}$ is rejected, an object from M is eliminated and the procedure is repeated until $H_{0,M}$ is not rejected. The set ${\widehat{M}}_{1-\alpha }^{*}$ is defined as the “superior set of models” (SSM) and it contains the surviving method.

Despite its sequential nature, the MCS procedure does not accumulate type I error. This is due to the fact that the test stops when the first hypothesis is not rejected.

In this procedure, a semi-quadratic test statistic $SQ$ has been employed; it is defined as:

$$SQ=\sum _{i<j}\frac{{\left({\overline{d}}_{ij}\right)}^2}{\sqrt{\widehat{var}\left({\overline{d}}_{ij}\right)}},$$

(9)

where: ${\overline{d}}_{ij}=\frac{1}{T}{\sum }_{t=1}^T(L_{i,t}-L_{j,t})$ $L_{i,t}$ being the loss function associated with the model i at time t. The critical values of the test as well as the estimation of the variance useful to construct the test statistic are obtained by using the block bootstrap (see Hansen et al. 2011).

In this application, $B=1000$ bootstrap resamples have been generated and $\alpha$ has been fixed at 0.05.

The choice of the loss function is arbitrary and depends on the nature of the competing models. In this paper, two different loss functions have been considered. They are defined as:

$$\begin{array}{cc}\hfill {\mathrm{MSE}}_t& ={({\tilde{\sigma }}_t^2-{\widehat{\sigma }}_t^2)}^2,\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill {\mathrm{QLIKE}}_t& =\frac{{\tilde{\sigma }}_t^2}{{\widehat{\sigma }}_t^2}-log\left(\frac{{\tilde{\sigma }}_t^2}{{\widehat{\sigma }}_t^2}\right)-1,\hfill \end{array}$$

(11)

where ${\tilde{\sigma }}_t$ is some volatility measure and ${\widehat{\sigma }}_t$ is the punctual volatility forecast. They are the most widely used loss functions and provide robust ranking of the models in the context of volatility forecasts (Patton 2011).

Table 4. MCS p-values for MSCI daily returns on BRICS countries. The forecasting horizon is equal to 1. The test statistic is $SQ$. The loss functions are MSE and QLIKE.

1 Step	Brazil		Russia		India		China		South Africa
	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE
Expanding win	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RiskMetrics	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.25	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.50	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.75	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Break	0.00	0.00	0.00	0.00	0.00	0.00	0.01	0.01	0.00	0.00
Mean-win.	0.29	1.00	1.00	1.00	1.00	0.99	1.00	1.00	1.00	0.65
Trimmed-Mean-win	1.00	0.83	0.00	0.00	0.46	1.00	0.01	0.01	0.81	1.00

reports the Composition of the Superior Set of Models for 1-day horizon for the five BRICS countries and for the two different loss functions. It is interesting to note that the forecasting methods based on the GARCH(1,1) specification and the RiskMetrics are the models always eliminated from the SSM. Among the combinations which make adjustments to accommodate potential structural breaks, the roll break is always present in the SSM, for all of the five BRICS countries and for both the loss functions MSE and QLIKE. However, while for Russia and China no other combination enters into the SSM, for the other three countries Brazil, India and South Africa, the set also contains the trimmed version of the roll break combination.

Results for horizon 5 and for horizon 20, reported in

Table 5. MCS p-values for MSCI daily returns on BRICS countries. The forecasting horizon is equal to 5. The test statistic is $SQ$. The loss functions are MSE and QLIKE.

5 Step	Brazil		Russia		India		China		South Africa
	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE
Expanding win	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RiskMetrics	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.25	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.50	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.75	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Break	0.00	0.00	0.00	0.00	0.00	0.00	0.03	0.08	0.00	0.00
Mean-win.	0.14	0.78	1.00	1.00	1.00	0.48	1.00	1.00	1.00	1.00
Trimmed-Mean-win	1.00	1.00	0.00	0.00	0.50	1.00	0.01	0.00	0.71	0.61

and

Table 6. MCS p-values for MSCI daily returns on BRICS countries. The forecasting horizon is equal to 20. The test statistic is $SQ$. The loss functions are MSE and QLIKE.

20 Step	Brazil		Russia		India		China		South Africa
	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE	QLIKE	MSE
Expanding win	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
RiskMetrics	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.25	0.00	0.01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.50	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Roll 0.75	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
Exp-Break	0.00	0.00	0.00	0.00	0.00	0.00	0.01	0.01	0.00	0.00
Mean-win.	1.00	1.00	1.00	1.00	0.95	0.28	1.00	1.00	0.86	0.92
Trimmed-Mean-win	0.40	0.03	0.00	0.00	1.00	1.00	0.01	0.00	1.00	1.00

, respectively, confirm the structure of the SSM. Again, for all five of the BRICS countries and for both of the loss functions considered, the forecasting methods based on the GARCH(1,1) specification and the RiskMetrics are always excluded from the SSM. Moreover, the roll break combination seems to have the best performance in terms of forecasting, being always in the SSM. For Brazil, India and South Africa, the trimmed version of the roll combination also enters in the SSM except for Brazil when the loss function is the MSE for horizon 20.

4. Conclusions

In this paper, the problem of structural breaks for forecasting the volatility of MSCI daily returns for BRICS countries has been investigated. The structural breaks in the unconditional variance of the five time series have been identified by using a binary segmentation algorithm with a test proposed by Sansó et al. (2004). It takes into account both the fourth order moment of the process and persistence in the variance and so it is suitable for financial time series. In particular, the attention has been focused on a GARCH(1,1) model.

The results of an in-sample analysis have highlighted that the procedure is able to identify structural breaks in all the BRICS countries and, as expected, significant changes in the unconditional variance are quite evident in the regimes identified by the structural breaks.

In the out-sample analysis, six different forecast combinations have been compared in terms of their predictive performance by using the Model Confidence Set. With this procedure, the set of “best models” in terms of two loss functions and a given level of confidence has been constructed. The results show that the forecasting methods which do not make adjustments to accommodate structural breaks have worse performance. Moreover, among the combinations that account for structural breaks, the one that averages forecasts obtained using different rolling estimation windows outperforms all the others, for all five of the BRICS countries.