The VIF and MSE in Raise Regression

Abstract

The raise regression has been proposed as an alternative to ordinary least squares estimation when a model presents collinearity. In order to analyze whether the problem has been mitigated, it is necessary to develop measures to detect collinearity after the application of the raise regression. This paper extends the concept of the variance inflation factor to be applied in a raise regression. The relevance of this extension is that it can be applied to determine the raising factor which allows an optimal application of this technique. The mean square error is also calculated since the raise regression provides a biased estimator. The results are illustrated by two empirical examples where the application of the raise estimator is compared to the application of the ridge and Lasso estimators that are commonly applied to estimate models with multicollinearity as an alternative to ordinary least squares.

1. Introduction

In the last fifty years, different methods have been developed to avoid the instability of estimates derived from collinearity (see, for example, Kiers and Smilde [1]). Some of these methods can be grouped within a general denomination known as penalized regression.

In general terms, the penalized regression parts from the linear model (with p variables and n observations), $\mathbf{Y}=\mathbf{X}\mathbf{\beta }+\mathbf{u}$, and obtains the regularization of the estimated parameters, minimizing the following objective function:

$${(\mathbf{Y}-\mathbf{X}\mathbf{\beta })}^t(\mathbf{Y}-\mathbf{X}\mathbf{\beta })+P\left(\mathbf{\beta }\right),$$

where $P\left(\mathbf{\beta }\right)$ is a penalty term that can take different forms. One of the most common penalty terms is the bridge penalty term ([2,3]) is given by

$$P\left(\mathbf{\beta }\right)=\lambda \sum _{j=1}^p{\left|{\beta }_j\right|}^{\alpha },\alpha >0,$$

where $\lambda$ is a tuning parameter. Note that the ridge ([4]) and the Lasso ([5]) regressions are obtained when $\alpha =2$ and $\alpha =1$, respectively. Penalties have also been called soft thresholding ([6,7]).

These methods are applied not only for the treatment of multicollinearity but also for the selection of variables (see, for example, Dupuis and Victoria-Feser [8], Li and Yang [9] Liu et al. [10], or Uematsu and Tanaka [11]), which is a crucial issue in many areas of science when the number of variables exceeds the sample size. Zou and Hastie [12] proposed elastic net regularization by using the penalty terms ${\lambda }_1$ and ${\lambda }_2$ that combine the Lasso and ridge regressions:

$$P\left(\mathbf{\beta }\right)={\lambda }_1\sum _{j=1}^p\left|{\beta }_j\right|+{\lambda }_2\sum _{j=1}^p{\beta }_j^2.$$

Thus, the Lasso regression usually selects one of the regressors from among all those that are highly correlated, while the elastic net regression selects several of them. In the words of Tutz and Ulbricht [13] “the elastic net catches all the big fish”, meaning that it selects the whole group.

From a different point of view, other authors have also presented different techniques and methods well suited for dealing with the collinearity problems: continuum regression ([14]), least angle regression ([15]), generalized maximum entropy ([16,17,18]), the principal component analysis (PCA) regression ([19,20]), the principal correlation components estimator ([21]), penalized splines ([22]), partial least squares (PLS) regression ([23,24]), or the surrogate estimator focused on the solution of the normal equations presented by Jensen and Ramirez [25].

Focusing on collinearity, the ridge regression is one of the more commonly applied methodologies and it is estimated by the following expression:

$$\widehat{\mathbf{\beta }}\left(K\right)={\left({\mathbf{X}}^t\mathbf{X}+K·\mathbf{I}\right)}^{-1}{\mathbf{X}}^t\mathbf{Y}$$

(1)

where $\mathbf{I}$ is the identity matrix with adequate dimensions and K is the ridge factor (ordinary least squares (OLS) estimators are obtained when $K=0$). Although ridge regression has been widely applied, it presents some problems with current practice in the presence of multicollinearity and the estimators derived from the penalty come into these same problems whenever $n>p$:

• In relation to the calculation of the variance inflation factors (VIF), measures that quantify the degree of multicollinearity existing in a model from the coefficient of determination of the regression between the independent variables (for more details, see Section 2), García et al. [26] showed that the application of the original data when working with the ridge estimate leads to non-monotone VIF values by considering the VIF as a function of the penalty term. Logically, the Lasso and the elastic net regression inherit this property.
• By following Marquardt [27]: “The least squares objective function is mathematically independent of the scaling of the predictor variables (while the objective function in ridge regression is mathematically dependent on the scaling of the predictor variables)”. That is to say, the penalized objective function will bring problems derived from the standardization of the variables. This fact has to be taken into account both for obtaining the estimators of the regressors and for the application of measures that detect if the collinearity has been mitigated. Other penalized regressions (such as Lasso and elastic net regressions) are not scale invariant and hence yield different results depending on the predictor scaling used.
• Some of the properties of the OLS estimator that are deduced from the normal equations are not verified by the ridge estimator and, among others, the estimated values for the endogenous variable are not orthogonal to the residuals. As a result, the following decomposition is verified

  $$\sum _{i=1}^n{(Y_i-\overline{Y})}^2=\sum _{i=1}^n{({\widehat{Y}}_i\left(K\right)-\overline{Y})}^2+\sum _{i=1}^n{e}_i{\left(K\right)}^2+2\sum _{i=1}^n({\widehat{Y}}_i\left(K\right)-\overline{Y})·e_i\left(K\right).$$

  When the OLS estimators are obtained ($K=0$), the third term is null. However, this term is not null when K is not zero. Consequently, the relationship $TSS\left(K\right)=ESS\left(K\right)+RSS\left(K\right)$ is not satisfied in ridge regression, and the definition of the coefficient of determination may not be suitable. This fact not only limits the analysis of the goodness of fit but also affects the global significance since the critical coefficient of determination is also questioned. Rodríguez et al. [28] showed that the estimators obtained from the penalties mentioned above inherit the problem of the ridge regression in relation to the goodness of fit.

In order to overcome these problems, this paper is focused on the raise regression (García et al. [29] and Salmerón et al. [30]) based on the treatment of collinearity from a geometrical point of view. It consists in separating the independent variables by using the residuals (weighted by the raising factor) of the auxiliary regression traditionally used to obtain the VIF. Salmerón et al. [30] showed that the raise regression presents better conditions than ridge regression and, more recently, García et al. [31] showed, among other questions, that the ridge regression is a particular case of the raise regression.

This paper presents the extension of the VIF to the raise regression showing that, although García et al. [31] showed that the application of the raise regression guarantees a diminishing of the VIF, it is not guaranteed that its value is lower the threshold traditionally established as troubling. Thus, it will be concluded that an unique application of the raise regression does not guarantee the mitigation of the multicollinearity. Consequently, this extension complements the results presented by García et al. [31] and determines, on the one hand, whether it is necessary to apply a successive raise regression (see García et al. [31] for more details) and, on the other hand, the most adequate variable for raising and the most optimal value for the raising factor in order to guarantee the mitigation of the multicollinearity.

On the other hand, the transformation of variables is common when strong collinearity exists in a linear model. The transformation to unit length (see Belsley et al. [32]) or standardization (see Marquardt [27]) is typical. Although the VIF is invariant to these transformations when it is calculated after estimation by OLS (see García et al. [26]), it is not guaranteed either in the case of the raise regression or in ridge regression as showed by García et al. [26]. The analysis of this fact is one of the goals of this paper.

Finally, since the raise estimator is biased, it is interesting to calculate its mean square error (MSE). It is studied whether the MSE of the raise regression is less than the one obtained by OLS. In this case, this study could be used to select an adequate raising factor similar to what is proposed by Hoerl et al. [33] in the case of the ridge regression. Note that estimators with MSE less than the one from OLS estimators are traditionally preferred (see, for example, Stein [34], James and Stein [35], Hoerl and Kennard [4], Ohtani [36], or Hubert et al. [37]). In addition, this measure allows us to conclude whether the raise regression is preferable, in terms of MSE, to other alternative techniques.

The structure of the paper is as follows: Section 2 briefly describes the VIF and the raise regression, and Section 3 extends the VIF to this methodology. Some desirable properties of the VIF are analyzed, and its asymptotic behavior is studied. It is also concluded that the VIF is invariant to data transformation. Section 4 calculates the MSE of the raise estimator, showing that there is a minimum value that is less than the MSE of the OLS estimator. Section 5 illustrates the contribution of this paper with two numerical examples. Finally, Section 6 summarizes the main conclusions of this paper.

2. Preliminaries

2.1. Variance Inflation Factor

The following model for p independent variables and n observations is considered:

$$\mathbf{Y}={\beta }_1+{\beta }_2{\mathbf{X}}_2+\cdots +{\beta }_i{\mathbf{X}}_i+\cdots +{\beta }_p{\mathbf{X}}_p+\mathbf{u}=\mathbf{X}\mathbf{\beta }+\mathbf{u},$$

(2)

where $\mathbf{Y}$ is a vector $n\times 1$ that contains the observations of the dependent variable, $\mathbf{X}=[\mathbf{1}{\mathbf{X}}_2\dots {\mathbf{X}}_i\dots {\mathbf{X}}_p]$ (with $\mathbf{1}$ being a vector of ones with dimension $n\times 1$) is a matrix with order $n\times p$ that contains (by columns) the observations of the independent variables, $\mathbf{\beta }$ is a vector $p\times 1$ that contains the coefficients of the independent variables, and $\mathbf{u}$ is a vector $n\times 1$ that represents the random disturbance that is supposed to be spherical ($E\left[\mathbf{u}\right]=\mathbf{0}$ and $Var\left(\mathbf{u}\right)={\sigma }^2\mathbf{I}$, where $\mathbf{0}$ is a vector with zeros with dimension $n\times 1$ and $\mathbf{I}$ the identity matrix with adequate dimensions, in this case $p\times p$).

Given the model in Equation (2), the variance inflation factor (VIF) is obtained as follows:

$$VIF\left(k\right)=\frac{1}{1-R_k^2},k=2,\dots ,p,$$

(3)

where $R_k^2$ is the coefficient of determination of the regression of the variable ${\mathbf{X}}_k$ as a function of the rest of the independent variables of the model in Equation (2):

$${\mathbf{X}}_k={\alpha }_1+{\alpha }_2{\mathbf{X}}_2+\cdots +{\alpha }_{k-1}{\mathbf{X}}_{k-1}+{\alpha }_{k+1}{\mathbf{X}}_{k+1}+\cdots +{\alpha }_p{\mathbf{X}}_p+\mathbf{v}={\mathbf{X}}_{-k}\mathbf{\alpha }+\mathbf{v},$$

(4)

where ${\mathbf{X}}_{-k}$ corresponds to the matrix $\mathbf{X}$ after the elimination of the column k (variable ${\mathbf{X}}_k$).

If the variable ${\mathbf{X}}_k$ has no linear relationship (i.e., is orthogonal) with the rest of the independent variables, the coefficient of determination will be zero ($R_k^2=0$) and the $VIF\left(k\right)=1$. As the linear relationship increases, the coefficient of determination ($R_k^2$) and consequently $VIF\left(k\right)$ will also increase. Thus, the higher the VIF associated with the variable ${\mathbf{X}}_k$, the greater the linear relationship between this variable and the rest of the independent variables in the model in Equation (2). It is considered that the collinearity is troubling for values of VIF higher than 10. Note that the VIF ignores the role of the constant term (see, for example, Salmerón et al. [38] or Salmerón et al. [39]), and consequently, this extension will be useful when the multicollinearity is essential; that is to say, when there is a linear relationship between at least two independent variables of the model of regression without considering the constant term (see, for example, Marquandt and Snee [40] for the definitions of essential and nonessential multicollinearity).

2.2. Raise Regression

Raise regression, presented by García et al. [29] and more developed further by Salmerón et al. [30], uses the residuals of the model in Equation (4), ${\mathbf{e}}_k$, to raise the variable k as ${\tilde{\mathbf{X}}}_k={\mathbf{X}}_k+\lambda {\mathbf{e}}_k$ with $\lambda \ge 0$ (called the raising factor) and to verify that ${\mathbf{e}}_k^t{\mathbf{X}}_{-k}=\mathbf{0}$, where $\mathbf{0}$ is a vector of zeros with adequate dimensions. In that case, the raise regression consists in the estimation by OLS of the following model:

$$\mathbf{Y}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right){\mathbf{X}}_2+\cdots +{\beta }_k\left(\lambda \right){\tilde{\mathbf{X}}}_k+\cdots +{\beta }_p\left(\lambda \right){\mathbf{X}}_p+\tilde{\mathbf{u}}=\tilde{\mathbf{X}}\mathbf{\beta }\left(\lambda \right)+\tilde{\mathbf{u}},$$

(5)

where $\tilde{\mathbf{X}}=[\mathbf{1}{\mathbf{X}}_2\dots {\tilde{\mathbf{X}}}_k\dots {\mathbf{X}}_p]=\left[{\mathbf{X}}_{-k}{\tilde{\mathbf{X}}}_k\right]$. García et al. [29] showed (Theorem 3.3) that this technique does not alter the global characteristics of the initial model. That is to say, the models in Equations (2) and (5) have the same coefficient of determination and experimental statistics for the global significance test.

Figure 1 illustrates the raise regression for two independent variables being geometrically separated by using the residuals weighted by the raising factor $\lambda$. Thus, the selection of an adequate value for $\lambda$ is essential, analogously to what occurs with the ridge factor K. A preliminary proposal about how to select the raising factor in a model with two independent standardized variables can be found in García et al. [41]. Other recently published papers introduce and highlight the various advantages of raise estimators for statistical analysis: Salmerón et al. [30] presented the raise regression for $p=3$ standardized variables and showed that it presents better properties than the ridge regression and that the individual inference of the raised variable is not altered, García et al. [31] showed that it is guaranteed that all the VIFs associated with the model in Equation (5) diminish but that it is not possible to quantify the decrease, García and Ramírez [42] presented the successive raise regression, and García et al. [31] showed, among other questions, that ridge regression is a particular case of raise regression.

The following section presents the extension of the VIF to be applied after the estimation by raise regression since it will be interesting whether, after the raising of one independent variable, the VIF falls below 10. It will be also analyzed when a successive raise regression can be recommendable (see García and Ramírez [42]).

3. VIF in Raise Regression

To calculate the VIF in the raise regression, two cases have to be differentiated depending on the dependent variable, ${\mathbf{X}}_k$, of the auxiliary regression:

• If it is the raised variable, ${\tilde{\mathbf{X}}}_i$ with $i=2,\dots ,p$, the coefficient of determination, $R_i^2\left(\lambda \right)$, of the following auxiliary regression has to be calculated:

  $$\begin{array}{ccc}\hfill {\tilde{\mathbf{X}}}_i& =& {\alpha }_1\left(\lambda \right)+{\alpha }_2\left(\lambda \right){\mathbf{X}}_2+\cdots +{\alpha }_{i-1}\left(\lambda \right){\mathbf{X}}_{i-1}+{\alpha }_{i+1}\left(\lambda \right){\mathbf{X}}_{i+1}+\cdots +{\alpha }_p\left(\lambda \right){\mathbf{X}}_p+\tilde{\mathbf{v}}\hfill \\ \hfill & =& {\mathbf{X}}_{-i}\mathbf{\alpha }\left(\lambda \right)+\tilde{\mathbf{v}}.\hfill \end{array}$$

  (6)
• If it is not the raised variable, ${\mathbf{X}}_j$ with $j=2,\dots ,p$ being $j\ne i$, the coefficient of determination, $R_j^2\left(\lambda \right)$, of the following auxiliary regression has to be calculated:

  $$\begin{array}{ccc}\hfill {\mathbf{X}}_j& =& {\alpha }_1\left(\lambda \right)+{\alpha }_2\left(\lambda \right){\mathbf{X}}_2+\cdots +{\alpha }_i\left(\lambda \right){\tilde{\mathbf{X}}}_i+\cdots +{\alpha }_{j-1}\left(\lambda \right){\mathbf{X}}_{j-1}+{\alpha }_{j+1}\left(\lambda \right){\mathbf{X}}_{j+1}\hfill \\ \hfill & & +\cdots +{\alpha }_p\left(\lambda \right){\mathbf{X}}_p+\tilde{\mathbf{v}}\hfill \\ \hfill & =& \left({\mathbf{X}}_{-i,-j}{\tilde{\mathbf{X}}}_i\right)\left(\begin{array}{c}{\mathbf{\alpha }}_{-i,-j}\left(\lambda \right)\\ {\alpha }_i\left(\lambda \right)\end{array}\right)+\tilde{\mathbf{v}},\hfill \end{array}$$

  (7)

  where ${\mathbf{X}}_{-i,-j}$ corresponding to the matrix $\mathbf{X}$ after the elimination of columns i and j (variables ${\mathbf{X}}_i$ and ${\mathbf{X}}_j$). The same notation is used for ${\mathbf{\alpha }}_{-i,-j}\left(\lambda \right)$.

Once these coefficients of determination are obtained (as indicated in the following subsections), the VIF of the raise regression will be given by the following:

$$VIF(k,\lambda )=\frac{1}{1-R_k^2\left(\lambda \right)},k=2,\dots ,p.$$

(8)

3.1. VIF Associated with Raise Variable

In this case, for $i=2,\dots ,p$, the coefficient of determination of the regression in Equation (6) is given by

$$\begin{array}{ccc}\hfill R_i^2\left(\lambda \right)& =& 1-\frac{(1+2\lambda +{\lambda }^2)RS{S}_i^{-i}}{TS{S}_i^{-i}+({\lambda }^2+2\lambda )RS{S}_i^{-i}}=\frac{ES{S}_i^{-i}}{TS{S}_i^{-i}+({\lambda }^2+2\lambda )RS{S}_i^{-i}}\hfill \\ \hfill & =& \frac{R_i^2}{1+({\lambda }^2+2\lambda )(1-R_i^2)},\hfill \end{array}$$

(9)

since:

$$\begin{array}{ccc}\hfill TS{S}_i^{-i}\left(\lambda \right)& =& {\tilde{\mathbf{X}}}_i^t{\tilde{\mathbf{X}}}_i-n·{\overline{\tilde{\mathbf{X}}}}_i^2={\mathbf{X}}_i^t{\mathbf{X}}_i+({\lambda }^2+2\lambda ){\mathbf{e}}_i^t{\mathbf{e}}_i-n·{\overline{\mathbf{X}}}_i^2\hfill \\ & =& TS{S}_i^{-i}+({\lambda }^2+2\lambda )RS{S}_i^{-i},\hfill \\ \hfill RS{S}_i^{-i}\left(\lambda \right)& =& {\tilde{\mathbf{X}}}_i^t{\tilde{\mathbf{X}}}_i-\widehat{\mathbf{\alpha }}{\left(\lambda \right)}^t{\mathbf{X}}_{-i}^t{\tilde{\mathbf{X}}}_i={\mathbf{X}}_i^t{\mathbf{X}}_i+({\lambda }^2+2\lambda ){\mathbf{e}}_i^t{\mathbf{e}}_i-{\widehat{\mathbf{\alpha }}}^t{\mathbf{X}}_{-i}^t{\mathbf{X}}_i\hfill \\ & =& ({\lambda }^2+2\lambda +1)RS{S}_i^{-i},\hfill \end{array}$$

where $TS{S}_i^{-i}$, $ES{S}_i^{-i}$ and $RS{S}_i^{-i}$ are the total sum of squares, explained sum of squares, and residual sum of squares of the model in Equation (4). Note that it has been taken into account that

$${\tilde{\mathbf{X}}}_i^t{\tilde{\mathbf{X}}}_i={\left({\mathbf{X}}_i+\lambda {\mathbf{e}}_i\right)}^t\left({\mathbf{X}}_i+\lambda {\mathbf{e}}_i\right)={\mathbf{X}}_i^t{\mathbf{X}}_i+({\lambda }^2+2\lambda ){\mathbf{e}}_i^t{\mathbf{e}}_i,$$

since ${\mathbf{e}}_i^t{\mathbf{X}}_i={\mathbf{e}}_i^t{\mathbf{e}}_i=RS{S}_i^{-i}$ and

$$\widehat{\mathbf{\alpha }}\left(\lambda \right)={\left({\mathbf{X}}_{-i}^t{\mathbf{X}}_{-i}\right)}^{-1}{\mathbf{X}}_{-i}^t{\tilde{\mathbf{X}}}_i=\widehat{\mathbf{\alpha }},$$

due to ${\mathbf{X}}_{-i}^t{\tilde{\mathbf{X}}}_i={\mathbf{X}}_{-i}^t{\mathbf{X}}_i$.

Indeed, from Equation (9), it is evident that

• $R_i^2\left(\lambda \right)$ decreases as $\lambda$ increases.
• $\underset{\lambda \to +\infty }{lim}{R}_i^2\left(\lambda \right)=0$.
• $R_i^2\left(\lambda \right)$ is continuous in zero; that is to say, $R_i^2\left(0\right)=R_i^2$.

Finally, from properties 1) and 3), it is deduced that $R_i^2\left(\lambda \right)\le R_i^2$ for all $\lambda$.

3.2. VIF Associated with Non-Raised Variables

In this case, for $j=2,\dots ,p,$ with $j\ne i$, the coefficient of determination of regression in Equation (7) is given by

$$\begin{array}{ccc}\hfill R_j^2\left(\lambda \right)& =& 1-\frac{RS{S}_j^{-j}\left(\lambda \right)}{TS{S}_j^{-j}\left(\lambda \right)}\hfill \\ \hfill & =& \frac{1}{TS{S}_j^{-j}}\left(TS{S}_j^{-j}-RS{S}_j^{-i,-j}+\frac{RS{S}_i^{-i,-j}·\left(RS{S}_j^{-i,-j}-RS{S}_j^{-j}\right)}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}\right),\hfill \end{array}$$

(10)

Taking into account that ${\tilde{\mathbf{X}}}_i^t{\mathbf{X}}_j={({\mathbf{X}}_i+\lambda {\mathbf{e}}_i)}^t{\mathbf{X}}_j={\mathbf{X}}_i^t{\mathbf{X}}_j$ since ${\mathbf{e}}_i^t{\mathbf{X}}_j=0$, it is verified that

$$TS{S}_j^{-j}\left(\lambda \right)={\mathbf{X}}_j^t{\mathbf{X}}_j-n·{\overline{\mathbf{X}}}_j^2=TS{S}_j^{-j},$$

and, from Appendix A and Appendix B,

$$\begin{array}{ccc}\hfill RS{S}_j^{-j}\left(\lambda \right)& =& {\mathbf{X}}_j^t{\mathbf{X}}_j-\widehat{\mathbf{\alpha }}{\left(\lambda \right)}^t\left(\begin{array}{c}{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j\\ {\tilde{\mathbf{X}}}_i^t{\mathbf{X}}_j\end{array}\right)\hfill \\ & =& {\mathbf{X}}_j^t{\mathbf{X}}_j-{\widehat{\mathbf{\alpha }}}_{-i,-j}{\left(\lambda \right)}^t{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j-{\widehat{\alpha }}_i{\left(\lambda \right)}^t{\mathbf{X}}_i^t{\mathbf{X}}_j\hfill \\ & \underset{\mathrm{Appendix}\mathrm{A}}{\underbrace{=}}& {\mathbf{X}}_j^t{\mathbf{X}}_j-{\mathbf{X}}_j^t{\mathbf{X}}_{-i,-j}{\left({\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_{-i,-j}\right)}^{-1}{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j\hfill \\ & & -\frac{RS{S}_i^{-i,-j}}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}·\hfill \\ & & ·\left(RS{S}_i^{-i,-j}{\mathbf{X}}_j^t{\mathbf{X}}_{-i,-j}·B·B^t·{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j\right.\hfill \\ & & \left.+{\mathbf{X}}_j^t{\mathbf{X}}_i·B^t·{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j\right)\hfill \\ & & -\frac{RS{S}_i^{-i,-j}}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}·{\widehat{\alpha }}_i^t{\mathbf{X}}_i^t{\mathbf{X}}_j\hfill \\ & =& {\mathbf{X}}_j^t\left(\mathbf{I}-{\mathbf{X}}_{-i,-j}{\left({\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_{-i,-j}\right)}^{-1}{\mathbf{X}}_{-i,-j}^t\right){\mathbf{X}}_j\hfill \\ & & -\frac{RS{S}_i^{-i,-j}}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}·\hfill \\ & & ·\left(RS{S}_i^{-i,-j}{\mathbf{X}}_j^t{\mathbf{X}}_{-i,-j}·B·B^t·{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j\right.\hfill \\ & & \left.+{\mathbf{X}}_j^t{\mathbf{X}}_i·B^t·{\mathbf{X}}_{-i,-j}^t{\mathbf{X}}_j+{\widehat{\alpha }}_i^t{\mathbf{X}}_i^t{\mathbf{X}}_j\right)\hfill \\ & \underset{\mathrm{Appendix}\mathrm{B}}{\underbrace{=}}& RS{S}_j^{-i,-j}\hfill \\ & & -\frac{RS{S}_i^{-i,-j}}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}·\left(RS{S}_j^{-i,-j}-RS{S}_j^{-j}\right),\hfill \end{array}$$

where $TS{S}_j^{-j}$ and $RS{S}_j^{-j}$ are the total sum of squares and residual sum of squares of the model in Equation (4) and where $RS{S}_i^{-i,-j}$ and $RS{S}_j^{-i,-j}$ are the residual sums of squares of models:

$$\begin{array}{ccc}\hfill {\mathbf{X}}_i& =& {\mathbf{X}}_{-i,-j}\mathbf{\gamma }+\mathbf{\eta },\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill {\mathbf{X}}_j& =& {\mathbf{X}}_{-i,-j}\mathbf{\delta }+\mathbf{\nu }.\hfill \end{array}$$

(12)

Indeed, from Equation (10), it is evident that

• $R_j^2\left(\lambda \right)$ decreases as $\lambda$ increases.
• $\underset{\lambda \to +\infty }{lim}{R}_j^2\left(\lambda \right)=\frac{TS{S}_j^{-j}-RS{S}_j^{-i,-j}}{TS{S}_j^{-j}}$.
• $R_j^2\left(\lambda \right)$ is continuous in zero. That is to say, $R_j^2\left(0\right)=\frac{TS{S}_j^{-j}-RS{S}_j^{-j}}{TS{S}_j^{-j}}=R_j^2$.

Finally, from properties 1) and 3), it is deduced that $R_j^2\left(\lambda \right)\le R_j^2$ for all $\lambda$.

3.3. Properties of $VIF(k,\lambda )$

From conditions verified by the coefficient of determination in Equations (9) and (10), it is concluded that $VIF(k,\lambda )$ (see expression Equation (8)), verifies that

• The VIF associated with the raise regression is continuous in zero because the coefficients of determination of the auxiliary regressions in Equations (6) and (7) are also continuous in zero. That is to say, for $\lambda =0$, it coincides with the VIF obtained for the model in Equation (2) when it is estimated by OLS:

  $$VIF(k,0)=\frac{1}{1-R_k^2\left(0\right)}=\frac{1}{1-R_k^2}=VIF\left(k\right),k=2,\dots ,p.$$
• The VIF associated with the raise regression decreases as $\lambda$ increases since this is the behavior of the coefficient of determination of the auxiliary regressions in Equations (6) and (7). Consequently,

  $$VIF(k,\lambda )=\frac{1}{1-R_k^2\left(\lambda \right)}\le \frac{1}{1-R_k^2}=VIF\left(k\right),k=2,\dots ,p,\forall \lambda \ge 0.$$
• The VIF associated with the raised variable is always higher than one since

  $$\underset{\lambda \to +\infty }{lim}VIF(i,\lambda )=\underset{\lambda \to +\infty }{lim}\frac{1}{1-R_i^2\left(\lambda \right)}=\frac{1}{1-0}=1,i=2,\dots ,p.$$
• The VIF associated with the non-raised variables has a horizontal asymptote since

  $$\begin{array}{ccc}\hfill \underset{\lambda \to +\infty }{lim}VIF(j,\lambda )& =& \underset{\lambda \to +\infty }{lim}\frac{1}{1-R_j^2\left(\lambda \right)}=\frac{1}{1-\frac{TS{S}_j^{-j}-RS{S}_j^{-i,-j}}{TS{S}_j^{-j}}}\hfill \\ & =& \frac{TS{S}_j^{-j}}{RS{S}_j^{-i,-j}}=\frac{TS{S}_j^{-i,-j}}{RS{S}_j^{-i,-j}}=\frac{1}{1-R_{ij}^2}=VI{F}_{-i}\left(j\right),\hfill \end{array}$$

  where $R_{ij}^2$ is the coefficient of determination of the regression in Equation (12) for $j=2,\dots ,p$ and $j\ne i$. Indeed, this asymptote corresponds to the VIF, $VI{F}_{-i}\left(j\right)$, of the regression $\mathbf{Y}={\mathbf{X}}_{-i}\mathbf{\xi }+\mathbf{w}$ and, consequently, will also always be equal to or higher than one.

Thus, from properties (1) to (4), $VIF(k,\lambda )$ has the very desirable properties of being continuous, monotone in the raise parameter, and higher than one, as presented in García et al. [26].

In addition, the property (4) can be applied to determine the variable to be raised only considering the one with a lower horizontal asymptote. If the asymptote is lower than 10 (the threshold established traditionally as worrying), the extension could be applied to determine the raising factor by selecting, for example, the first $\lambda$ that verifies $VIF(k,\lambda )<10$ for $k=2,\dots ,p$. If none of the $p-1$ asymptotes is lower than the established threshold, it will not be enough to raise one independent variable and a successive raise regression will be recommended (see García and Ramírez [42] and García et al. [31] for more details). Note that, if it were necessary to raise more than one variable, it is guaranteed that there will be values of the raising parameter that mitigate multicollinearity since, in the extreme case where all the variables of the model are raised, all the VIFs associated with the raised variables tend to one.

3.4. Transformation of Variables

The transformation of data is very common when working with models where strong collinearity exists. For this reason, this section analyzes whether the transformation of the data affects the VIF obtained in the previous section.

Since the expression given by Equation (9) can be expressed with $i=2,\dots ,p$ in the function of $R_i^2$:

$$R_i^2\left(\lambda \right)=\frac{R_i^2}{1+({\lambda }^2+2\lambda )·(1-R_i^2)},$$

it is concluded that it is invariant to origin and scale changes and, consequently, the VIF calculated from it will also be invariant.

On the other hand, the expression given by Equation (10) can be expressed for $j=2,\dots ,p,$ with $j\ne i$ as

$$\begin{array}{ccc}\hfill R_j^2\left(\lambda \right)& =& 1-\frac{RS{S}_j^{-i,-j}}{TS{S}_j^{-j}}+\frac{1}{TS{S}_j^{-j}}·\frac{RS{S}_i^{-i,-j}·(RS{S}_j^{-i,-j}-RS{S}_j^{-j})}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}\hfill \\ \hfill & =& R_{ij}^2+\frac{RS{S}_i^{-i,-j}}{RS{S}_i^{-i,-j}+({\lambda }^2+2\lambda )·RS{S}_i^{-i}}·\left(\frac{RS{S}_j^{-i,-j}}{TS{S}_j^{-i,-j}}-\frac{RS{S}_j^{-j}}{TS{S}_j^{-j}}\right)\hfill \\ \hfill & =& R_{ij}^2+\frac{R_j^2-R_{ij}^2}{1+({\lambda }^2+2\lambda )·\frac{RS{S}_i^{-i}}{RS{S}_i^{-i,-j}}},\hfill \end{array}$$

(13)

where it was applied that $TS{S}_j^{-j}=TS{S}_j^{-i,-j}$.

In this case, by following García et al. [26], transforming the variable ${\mathbf{X}}_i$ as

$${\mathbf{x}}_i=\frac{{\mathbf{X}}_i-a_i}{b_i},a_i\in \mathbb{R},b_i\in \mathbb{R}-\left\{0\right\},i=2,\dots ,p,$$

it is obtained that $RS{S}_i^{-i}\left(T\right)=\frac{1}{b_i^2}RS{S}_i^{-i}$ and $RS{S}_i^{-i,-j}\left(T\right)=\frac{1}{b_i^2}RS{S}_i^{-i,-j}$ where $RS{S}_i^{-i}\left(T\right)$ and $RS{S}_i^{-i,-j}\left(T\right)$ are the residual sum of squares of the transformed variables.

Taking into account that ${\mathbf{X}}_i$ is the dependent variables in the regressions of $RS{S}_i^{-i}$ and $RS{S}_i^{-i,-j}$, the following is obtained:

$$\frac{RS{S}_i^{-i}}{RS{S}_i^{-i,-j}}=\frac{RS{S}_i^{-i}\left(T\right)}{RS{S}_i^{-i,-j}\left(T\right)}.$$

Then, the expression given by Equation (13) is invariant to data transformations (As long as the dependent variables are transformed from the regressions of $RS{S}_i^{-i}$ and $RS{S}_i^{-i,-j}$ in the same form. For example, (a) for considering that $a_i$ is its mean and $b_i$ is its standard deviation (typification), (b) for considering that $a_i$ is its mean and $b_i$ is its standard deviation multiplied by the square root of the number of observations (standardization), or (c) for considering that $a_i$ is zero and $b_i$ is the square root of the squares sum of observations (unit length).) and, consequently, the VIF calculated from it will also be invariant.

4. MSE for Raise Regression

Since the estimator $\mathbf{\beta }$ obtained from Equation (5) is biased, it is interesting to study its Mean Square Error (MSE).

Taking into account that, for $k=2,\dots ,p$,

$$\begin{array}{ccc}\hfill {\tilde{\mathbf{X}}}_k& =& {\mathbf{X}}_k+\lambda {\mathbf{e}}_k\hfill \\ & =& (1+\lambda ){\mathbf{X}}_k-\lambda \left({\widehat{\alpha }}_0+{\widehat{\alpha }}_1{\mathbf{X}}_1+\cdots +{\widehat{\alpha }}_{k-1}{\mathbf{X}}_{k-1}+{\widehat{\alpha }}_{k+1}{\mathbf{X}}_{k+1}+\cdots +{\widehat{\alpha }}_p{\mathbf{X}}_p\right),\hfill \end{array}$$

it is obtained that matrix $\tilde{\mathbf{X}}$ of the expression in Equation (5) can be rewritten as $\tilde{\mathbf{X}}=\mathbf{X}·{\mathbf{M}}_{\lambda }$, where

$${\mathbf{M}}_{\lambda }=\left(\begin{array}{cccccccc}1& 0& \cdots & 0& -\lambda {\widehat{\alpha }}_0& 0& \cdots & 0\\ 0& 1& \cdots & 0& -\lambda {\widehat{\alpha }}_1& 0& \cdots & 0\\ ⋮& ⋮& & ⋮& ⋮& ⋮& & ⋮\\ 0& 0& \cdots & 1& -\lambda {\widehat{\alpha }}_{k-1}& 0& \cdots & 0\\ 0& 0& \cdots & 0& 1+\lambda & 0& \cdots & 0\\ 0& 0& \cdots & 0& -\lambda {\widehat{\alpha }}_{k+1}& 1& \cdots & 0\\ ⋮& ⋮& & ⋮& ⋮& ⋮& & ⋮\\ 0& 0& \cdots & 0& -\lambda {\widehat{\alpha }}_p& 0& \cdots & 1\end{array}\right).$$

(14)

Thus, we have $\widehat{\mathbf{\beta }}\left(\lambda \right)={({\tilde{\mathbf{X}}}^t·\tilde{\mathbf{X}})}^{-1}{\tilde{\mathbf{X}}}^t·\mathbf{Y}={\mathbf{M}}_{\lambda }^{-1}·\widehat{\mathbf{\beta }}$, and then, the estimator of $\mathbf{\beta }$ obtained from Equation (5) is biased unless ${\mathbf{M}}_{\lambda }=\mathbf{I}$, which only occurs when $\lambda =0$, that is to say, when the raise regression coincides with OLS. Moreover,

$$\begin{array}{ccc}\hfill tr\left(Var\left(\widehat{\mathbf{\beta }}\left(\lambda \right)\right)\right)& =& tr({\mathbf{M}}_{\lambda }^{-1}·Var\left(\widehat{\mathbf{\beta }}\right)·{\left({\mathbf{M}}_{\lambda }^{-1}\right)}^t)={\sigma }^{2}tr\left({\left({\tilde{\mathbf{X}}}^t\tilde{\mathbf{X}}\right)}^{-1}\right),\hfill \\ \hfill {(E\left[\widehat{\mathbf{\beta }}\left(\lambda \right)\right]-\mathbf{\beta })}^t(E\left[\widehat{\mathbf{\beta }}\left(\lambda \right)\right]-\mathbf{\beta })& =& {\mathbf{\beta }}^t{({\mathbf{M}}_{\lambda }^{-1}-\mathbf{I})}^t({\mathbf{M}}_{\lambda }^{-1}-\mathbf{I})\mathbf{\beta },\hfill \end{array}$$

where $tr$ denotes the trace of a matrix.

In that case, the MSE for raise regression is

$$\begin{array}{ccc}\hfill \mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left(\lambda \right)\right)& =& tr\left(Var\left(\widehat{\mathbf{\beta }}\left(\lambda \right)\right)\right)+{(E\left[\widehat{\mathbf{\beta }}\left(\lambda \right)\right]-\mathbf{\beta })}^t(E\left[\widehat{\mathbf{\beta }}\left(\lambda \right)\right]-\mathbf{\beta })\hfill \\ & =& {\sigma }^{2}tr\left({\left({\tilde{\mathbf{X}}}^t\tilde{\mathbf{X}}\right)}^{-1}\right)+{\mathbf{\beta }}^t{({\mathbf{M}}_{\lambda }^{-1}-\mathbf{I})}^t({\mathbf{M}}_{\lambda }^{-1}-\mathbf{I})\mathbf{\beta }\hfill \\ & \underset{\mathrm{Appendix}\mathrm{C}}{\underbrace{=}}& {\sigma }^{2}tr\left({\left({\mathbf{X}}_{-k}^t{\mathbf{X}}_{-k}\right)}^{-1}\right)+\left(1+\sum _{j=0,j\ne k}^p{\widehat{\alpha }}_j^2\right)·{\beta }_k^2·\frac{{\lambda }^2+h}{{(1+\lambda )}^2},\hfill \end{array}$$

where $h=\frac{{\sigma }^2}{{\beta }_k^2·RS{S}_k^{-k}}$.

We can obtain the MSE from the estimated values of ${\sigma }^2$ and ${\beta }_k$ from the model in Equation (2).

On the other hand, once the estimations are obtained and taking into account the Appendix C, ${\lambda }_{min}=\frac{{\widehat{\sigma }}^2}{{\widehat{\beta }}_k^2·RS{S}_k^{-k}}$ minimizes $\mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left(\lambda \right)\right)$. Indeed, it is verified that $\mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left({\lambda }_{min}\right)\right)<\mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left(0\right)\right)$; that is to say, if the goal is exclusively to minimize the MSE (as in the work presented by Hoerl et al. [33]), ${\lambda }_{min}$ should be selected as the raising factor.

Finally, note that, if ${\lambda }_{min}>1$, then $\mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left(\lambda \right)\right)<\mathrm{MSE}\left(\widehat{\mathbf{\beta }}\left(0\right)\right)$ for all $\lambda >0$.

5. Numerical Examples

To illustrate the results of previous sections, two different set of data will be used that collect the two situations shown in the graphs of Figure A1 and Figure A2. The second example also compares results obtained by the raise regression to results obtained by the application of ridge and Lasso regression.

5.1. Example 1: $h<1$

The data set includes different financial variables for 15 Spanish companies for the year 2016 (consolidated account and results between €800,000 and €9,000,000) obtained from the dabase Sistema de Análisis de Balances Ibéricos (SABI) database. The relationship is studied between the number of employees, $\mathit{E}$, and the fixed assets (€), $\mathit{FA}$; operating income (€), $\mathit{OI}$; and sales (€), $\mathit{S}$. The model is expressed as

$$\mathit{E}={\beta }_1+{\beta }_2\mathit{FA}+{\beta }_3\mathit{OI}+{\beta }_4\mathit{S}+\mathit{u}.$$

(15)

Table 1. Estimations of the models in Equations (15)–(18): Standard deviation is inside the parenthesis, $R^2$ is the coefficient of determination, $F_{3,11}$ is the experimental value of the joint significance contrast, and ${\widehat{\sigma }}^2$ is the variance estimate of the random perturbation.

	Model (15)	p-Value	Model (16) for ${\mathit{\lambda }}_{\mathit{vif}}^{\left(1\right)}=24.5$	p-Value	Model (17) for ${\mathit{\lambda }}_{\mathit{min}}^{\left(1\right)}=0.42$ and ${\mathit{\lambda }}_{\mathit{vif}}^{\left(2\right)}=17.5$	p-Value	Model (18) for ${\mathit{\lambda }}_{\mathit{mse}}^{\left(1\right)}=1.43$ and ${\mathit{\lambda }}_{\mathit{vif}}^{\left(2\right)}=10$	p-Value
Intercept	994.21 (17,940)	0.957	4588.68 (17,773.22)	0.801	5257.84 (1744.26)	0.772	5582.29 (17,740.18)	0.759
$\mathbf{FA}$	−1.28 (0.55)	0.039	−1.59 (0.50)	0.009	−1.59 (0.51)	0.009	−1.58 (0.51)	0.009
$\mathbf{OI}$	−81.79 (52.86)	0.150
${\widetilde{\mathbf{OI}}}_{{\lambda }_{vif}^{\left(1\right)}}$			−3.21 (2.07)	0.150
${\widetilde{\mathbf{OI}}}_{{\lambda }_{min}^{\left(1\right)}}$					1.67 (2.28)	0.478
${\widetilde{\mathbf{OI}}}_{{\lambda }_{mse}^{\left(1\right)}}$							1.51 (2.24)	0.517
$\mathbf{S}$	87.58 (53.29)	0.129	8.38 (2.35)	0.004
${\tilde{\mathbf{S}}}_{{\lambda }_{vif}^{\left(2\right)}}$					3.42 (2.03)	0.120	3.55 (1.99)	0.103
$R^2$	0.70		0.70		0.70		0.70
$F_{3,11}$	8.50		8.50		8.50		8.50
${\widehat{\sigma }}^2$	1,617,171,931		1,617,171,931		1,617,171,931		1,617,171,931
MSE	321,730,738		321,790,581		336,915,567		325,478,516

displays the results of the estimation by OLS of the model in Equation (15). The presence of essential collinearity in the model in Equation (15) is indicated by the determinant close to zero (0.0000919) of the correlation matrix of independent variables

$$R=\left(\begin{array}{ccc}1& 0.7264656& 0.7225473\\ 0.7264656& 1& 0.9998871\\ 0.7225473& 0.9998871& 1\end{array}\right),$$

and the VIFs (2.45664, 5200.315, and 5138.535) higher than 10. Note that the collinearity is provoked fundamentally by the relationship between OI and S.

In contrast, due to the fact that the coefficients of variation of the independent variables (1.015027, 0.7469496, and 0.7452014) are higher than 0.1002506, the threshold established as troubling by Salmerón et al. [39], it is possible to conclude that the nonessential multicollinearity is not troubling. Thus, the extension of the VIF seems appropriate to check if the application of the raise regression has mitigated the multicollinearity.

Remark 1.

${\lambda }^{\left(1\right)}$and${\lambda }^{\left(2\right)}$will be the raising factor of the first and second raising, respectively.

5.1.1. First Raising

A possible solution could be to apply the raise regression to try to mitigate the collinearity. To decide which variable is raised, the thresholds for the VIFs associated with the raise regression are calculated with the goal of raising the variable that the smaller horizontal asymptotes present. In addition to raising the variable that presents the lowest VIF, it would be interesting to obtain a lower mean squared error (MSE) after raising. For this, the ${\lambda }_{min}^{\left(1\right)}$ is calculated for each case. Results are shown in

Table 2. Horizontal asymptotes for variance inflation factors (VIF) after raising each variable and ${\lambda }_{min}^{\left(1\right)}$, ${\lambda }_{mse}^{\left(1\right)}$, and ${\lambda }_{vif}^{\left(1\right)}$.

Raised	$\underset{{\mathbf{\lambda }}^{\left(1\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(1\right)})$	$\underset{{\mathbf{\lambda }}^{\left(1\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{OI},{\mathbf{\lambda }}^{\left(1\right)})$	$\underset{{\mathbf{\lambda }}^{\left(1\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{S},{\mathbf{\lambda }}^{\left(1\right)})$
Variable 1	1	4429.22	4429.22
Variable 2	2.09	1	2.09
Variable 3	2.12	2.12	1
Raised	${\lambda }_{min}^{\left(1\right)}$	${\lambda }_{mse}^{\left(1\right)}$	${\lambda }_{vif}^{\left(1\right)}$
Variable 1	0.18	0.45	∄
Variable 2	0.42	1.43	24.5
Variable 3	0.37	1.18	24.7

. Note that the variable to be raised should be the second or third since their asymptotes are lower than 10, although in both cases ${\lambda }_{min}^{\left(1\right)}$ is lower than 1 and it is not guaranteed that the MSE of the raise regression will be less than the one obtained from the estimation by the OLS of the model in Equation (15). For this reason, this table also shows the values of ${\lambda }^{\left(1\right)}$ that make the MSE of the raise regression coincide with the MSE of the OLS regression, ${\lambda }_{mse}^{\left(1\right)}$, and the minimum value of ${\lambda }^{\left(1\right)}$ that leads to values of VIF less than 10, ${\lambda }_{vif}^{\left(1\right)}$.

Figure 2 displays the VIF associated with the raise regression for $0\le {\lambda }^{\left(1\right)}\le 900$ after raising the second variable. It is observed that VIFs are always higher than its corresponding horizontal asymptotes.

The model after raising the second variable will be given by

$$\mathit{E}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right)\mathbf{FA}+{\beta }_3\left(\lambda \right)\widetilde{\mathbf{OI}}+{\beta }_4\left(\lambda \right)\mathbf{S}+\tilde{\mathit{u}},$$

(16)

where $\widetilde{\mathbf{OI}}=\mathbf{OI}+{\lambda }^{\left(1\right)}·{\mathbf{e}}_{\mathbf{OI}}$ with ${\mathbf{e}}_{\mathbf{OI}}$ the residual of regression:

$$\mathbf{OI}={\alpha }_1+{\alpha }_2\mathbf{FA}+{\alpha }_3\mathbf{S}+\mathit{v}.$$

Remark 2.

The coefficient of variation of$\widetilde{\mathbf{OI}}$for${\lambda }^{\left(1\right)}=24.5$is equal to 0.7922063; that is to say, it was lightly increased.

As can be observed from

Table 3. VIF of regression Equation (16) for ${\lambda }^{\left(1\right)}$ equal to ${\lambda }_{min}^{\left(1\right)}$, ${\lambda }_{mse}^{\left(1\right)}$, and ${\lambda }_{vif}^{\left(1\right)}$.

	$\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(1\right)})$	$\mathit{VIF}(\widetilde{\mathbf{OI}},{\mathbf{\lambda }}^{\left(1\right)})$	$\mathit{VIF}(\mathbf{S},{\mathbf{\lambda }}^{\left(1\right)})$
${\lambda }_{min}^{\left(1\right)}$	2.27	2587.84	2557.66
${\lambda }_{mse}^{\left(1\right)}$	2.15	878.10	868.58
${\lambda }_{vif}^{\left(1\right)}$	2.09	9.00	9.99

, in Equation (16), the collinearity is not mitigated by considering ${\lambda }^{\left(1\right)}$ equal to ${\lambda }_{min}^{\left(1\right)}$ and ${\lambda }_{mse}^{\left(1\right)}$. For this reason, Table 1 only shows the values of the model in Equation (16) for the value of ${\lambda }^{\left(1\right)}$ that leads to VIF lower than 10.

5.1.2. Transformation of Variables

After the first raising, it is interesting to verify that the VIF associated with the raise regression is invariant to data transformation. With this goal, the second variable has been raised, obtaining the $VIF(\mathbf{FA},{\lambda }^{\left(1\right)})$, $VIF(\widetilde{\mathbf{OI}},{\lambda }^{\left(1\right)})$, and $VIF(\mathbf{S},{\lambda }^{\left(1\right)})$ for ${\lambda }^{\left(1\right)}\in \{0,0.5,1,1.5,2,\dots ,95,10\}$, supposing original, unit length, and standardized data. Next, the three possible differences and the average of the VIF associated with each variable are obtained.

Table 4. Effect of data transformations on VIF associated with raise regression.

	$\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(1\right)})$	$\mathit{VIF}(\widetilde{\mathbf{OE}},{\mathbf{\lambda }}^{\left(1\right)})$	$\mathit{VIF}(\mathbf{S},{\mathbf{\lambda }}^{\left(1\right)})$
Original–Unit length	$9.83·{10}^{-16}$	$1.55·{10}^{-11}$	$1.83·{10}^{-10}$
Original–Standardized	$-1.80·{10}^{-16}$	$-3.10·{10}^{-10}$	$2.98·{10}^{-10}$
Unit length–Standardized	$-1.16·{10}^{-15}$	$-3.26·{10}^{-10}$	$1.15·{10}^{-10}$

displays the results from which it is possible to conclude that differences are almost null and that, consequently, the VIF associated with the raise regression is invariant to the most common data transformation.

5.1.3. Second Raising

After the first raising, we can use the results obtained from the value of $\lambda$ that obtains all VIFs less than 10 or consider the results obtained for ${\lambda }_{min}$ or ${\lambda }_{mse}$ and continue the procedure with a second raising. By following the second option, we part from the value of ${\lambda }^{\left(1\right)}={\lambda }_{min}^{\left(1\right)}=0.42$ obtained after the first raising. From

Table 5. Horizontal asymptote for VIFs after raising each variable in the second raising for ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$ and ${\lambda }_{vif}^{\left(2\right)}$.

Raised	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(2\right)})$	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\widetilde{\mathbf{OI}},{\mathbf{\lambda }}^{\left(2\right)})$	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{S},{\mathbf{\lambda }}^{\left(2\right)})$
Variable 1	1	2381.56	2381.56
Variable 3	2.12	2.12	1
Raised	${\lambda }_{min}^{\left(2\right)}$	${\lambda }_{mse}^{\left(2\right)}$	${\lambda }_{vif}^{\left(2\right)}$
Variable 1	0.15	0.34	∄
Variable 3	0.35	1.09	17.5

, the third variable is selected to be raised.

Table 6. VIFs of regression Equation (16) for ${\lambda }^{\left(2\right)}$ equal to ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$, and ${\lambda }_{vif}^{\left(2\right)}$.

	$\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(2\right)})$	$\mathit{VIF}(\widetilde{\mathbf{OI}},{\mathbf{\lambda }}^{\left(2\right)})$	$\mathit{VIF}(\tilde{\mathbf{S}},{\mathbf{\lambda }}^{\left(2\right)})$
${\lambda }_{min}^{\left(2\right)}$	2.20	1415.06	1398.05
${\lambda }_{mse}^{\left(2\right)}$	2.15	593.98	586.20
${\lambda }_{vif}^{\left(2\right)}$	2.12	9.67	8.47

shows the VIF associated with the following model for ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$, and ${\lambda }_{vif}^{\left(2\right)}$:

$$\mathit{E}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right)\mathbf{FA}+{\beta }_3\left(\lambda \right)\widetilde{\mathbf{OI}}+{\beta }_4\left(\lambda \right)\tilde{\mathbf{S}}+\tilde{\mathit{u}},$$

(17)

where $\tilde{\mathbf{S}}=\mathbf{S}+{\lambda }^{\left(2\right)}·{\mathbf{e}}_{\mathbf{S}}$ with ${\mathbf{e}}_{\mathbf{S}}$ the residuals or regression:

$$\mathbf{S}={\alpha }_1\left(\lambda \right)+{\alpha }_2\left(\lambda \right)\mathbf{FA}+{\alpha }_3\left(\lambda \right)\widetilde{\mathbf{OI}}+\tilde{\mathbf{v}}.$$

Remark 3.

The coefficient of variation of$\widetilde{\mathbf{OI}}$for${\lambda }^{\left(1\right)}=0.42$is equal to 0.7470222, and the coefficient of variation of$\tilde{\mathbf{S}}$for${\lambda }^{\left(2\right)}=17.5$is equal to 0.7473472. In both cases, they were slightly increased.

Note than it is only possible to state that collinearity has been mitigated when ${\lambda }^{\left(2\right)}={\lambda }_{vif}^{\left(2\right)}=17.5$. Results of this estimation are displayed in Table 1.

Considering that, after the first raising, it is obtained that ${\lambda }^{\left(1\right)}={\lambda }_{mse}^{\left(1\right)}=1.43$, from

Table 7. Horizontal asymptote for VIFs after raising each variables in the second raising for ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$, and ${\lambda }_{vif}^{\left(2\right)}$.

Raised	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(2\right)})$	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\widetilde{\mathbf{OI}},{\mathbf{\lambda }}^{\left(2\right)})$	$\underset{{\mathbf{\lambda }}^{\left(2\right)}\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{S},{\mathbf{\lambda }}^{\left(2\right)})$
Variable 1	1	853.40	853.40
Variable 3	2.12	2.12	1
Raised	${\lambda }_{min}^{\left(2\right)}$	${\lambda }_{mse}^{\left(2\right)}$	${\lambda }_{vif}^{\left(2\right)}$
Variable 1	0.12	0.27	∄
Variable 3	0.32	0.92	10

, the third variable is selected to be raised.

Table 8. VIFs of regression Equation (16) for ${\lambda }^{\left(2\right)}$ equal to ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$, and ${\lambda }_{vif}^{\left(2\right)}$.

	$\mathit{VIF}(\mathbf{FA},{\mathbf{\lambda }}^{\left(2\right)})$	$\mathit{VIF}(\widetilde{\mathbf{OI}},{\mathbf{\lambda }}^{\left(2\right)})$	$\mathit{VIF}(\tilde{\mathbf{S}},{\mathbf{\lambda }}^{\left(2\right)})$
${\lambda }_{min}^{\left(2\right)}$	2.14	508.54	502.58
${\lambda }_{mse}^{\left(2\right)}$	2.13	239.42	236.03
${\lambda }_{vif}^{\left(2\right)}$	2.12	9.36	8.17

shows the VIF associated with the following model for ${\lambda }_{min}^{\left(2\right)}$, ${\lambda }_{mse}^{\left(2\right)}$, and ${\lambda }_{vif}^{\left(2\right)}$:

$$\mathit{E}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right)\mathbf{FA}+{\beta }_3\left(\lambda \right)\widetilde{\mathbf{OI}}+{\beta }_4\left(\lambda \right)\tilde{\mathbf{S}}+\tilde{\mathit{u}},$$

(18)

where $\tilde{\mathbf{S}}=\mathbf{S}+\lambda ·{\mathbf{e}}_{\mathbf{S}}$.

Remark 4.

The coefficient of variation of$\widetilde{\mathbf{OI}}$for${\lambda }^{\left(1\right)}=1.43$is equal to 0.7473033, and the coefficient of variation of$\tilde{\mathbf{S}}$for${\lambda }^{\left(2\right)}=10$is equal to 0.7651473. In both cases, they were lightly increased.

Remark 5.

Observing the coefficients of variation of$\widetilde{\mathbf{OI}}$for different raising factor. it is concluded that the coefficient of variation increases as the raising factor increases: 0.7470222 ($\lambda =0.42$), 0.7473033 ($\lambda =1.43$), and 0.7922063 ($\lambda =24.5$).

Note that it is only possible to state that collinearity has been mitigated when ${\lambda }^{\left(2\right)}={\lambda }_{vif}^{\left(2\right)}=10$. Results of the estimations of this model are shown in Table 1.

5.1.4. Interpretation of Results

Analyzing the results of Table 1, it is possible to conclude that

• In the model in Equation (16) (in which the second variable is raised considering the smallest $\lambda$ that makes all the VIFs less than 10, ${\lambda }^{\left(1\right)}=24.5$), the variable sales have a coefficient significantly different from zero, where in the original model this was not the case. In this case, the MSE is superior to the one obtained by OLS.
• In the model in Equation (17) (in which the second variable is raised considering the value of $\lambda$ that minimizes the MSE, ${\lambda }^{\left(1\right)}=0.42$, and after that, the third variable is raised considering the smallest $\lambda$ that makes all the VIFs less than 10, ${\lambda }^{\left(2\right)}=17.5$), there is no difference in the individual significance of the coefficient.
• In the model in Equation (18) (in which the second variable is raised considering the value of $\lambda$ that makes the MSE of the raise regression coincide with that of OLS, ${\lambda }^{\left(1\right)}=1.43$, and next, the third variable is raised considering the smallest $\lambda$ that makes all the VIFs less than 10, ${\lambda }^{\left(2\right)}=10$), there is no difference in the individual significance of the coefficient.
• Although the coefficient of variable $\mathbf{OI}$ is not significantly different from zero in any case, the not expected negative sign obtained in model in Equation (15) is corrected in models Equations (17) and (18).
• In the models with one or two raisings, all the global characteristics coincide with that of the model in Equation (15). Furthermore, there is a relevant decrease in the estimation of the standard deviation for the second and third variable.
• In models with one or two raisings, the MSE increases, with the model in Equation (16) being the one that presents the smallest MSE among the biased models.

Thus, in conclusion, the model in Equation (16) is selected as it presents the smallest MSE and there is an improvement in the individual significance of the variables.

5.2. Example 2: $h>1$

This example uses the following model previously applied by Klein and Goldberger [43] about consumption and salaries in the United States from 1936 to 1952 (1942 to 1944 were war years, and data are not available):

$$\mathbf{C}={\beta }_1+{\beta }_2\mathbf{WI}+{\beta }_3\mathbf{NWI}+{\beta }_4\mathbf{FI}+\mathbf{u},$$

(19)

where $\mathbf{C}$ is consumption, $\mathbf{WI}$ is wage income, $\mathbf{NWI}$ is non-wage, non-farm income, and $\mathbf{FI}$ is the farm income. Its estimation by OLS is shown in

Table 9. Estimation of the original and raised models: Standard deviation is inside the parentheses, $R^2$ is the coefficient of determination, $F_{3,10}$ is the experimental value of the joint significance contrast, and ${\widehat{\sigma }}^2$ is the variance estimate of the random perturbation.

	Model (19)	p-Value	Model (20) for ${\mathbf{\lambda }}_{\mathit{min}}=6.895$	p-Value	Model (21) for ${\mathbf{\lambda }}_{\mathit{min}}=0.673$	p-Value
Intercept	18.7021 (6.8454)	0.021	19.21507 (6.67216)	0.016	18.2948 (6.8129)	0.023
WI	0.3803 (0.3121)	0.251	0.43365 (0.26849)	0.137
$\widetilde{\mathbf{WI}}$					0.2273 (0.1866)	0.251
NWI	1.4186 (0.7204)	0.077	1.38479 (0.71329)	0.081	1.7269 (0.5143)	0.007
FI	0.5331 (1.3998)	0.711			0.8858 (1.2754)	0.503
$\widetilde{\mathbf{FI}}$			0.06752 (0.17730)	0.711
$R^2$	0.9187		0.9187		0.9187
$\widehat{\sigma }$	6.06		6.06		6.06
$F_{3,10}$	37.68		37.68		37.68
MSE	49.43469		45.61387		48.7497

.

However, this estimation is questionable since no estimated coefficient is significantly different to zero while the model is globally significant (with 5% significance level), and the VIFs associated with each variable (12.296, 9.23, and 2.97) indicate the presence of severe essential collinearity. In addition, the determinant of the matrix of correlation

$$\mathbf{R}=\left(\begin{array}{ccc}1& 0.9431118& 0.8106989\\ 0.9431118& 1& 0.7371272\\ 0.8106989& 0.7371272& 1\end{array}\right),$$

is equal to 0.03713592 and, consequently, lower than the threshold recommended by García et al. [44] ($1.013·0.1+0.00008626·n-0.01384·p=0.04714764$ being $n=14$ and $p=4$); it is maintained the conclusion that the near multicollinearity existing in this model is troubling.

Once again, the values of the coefficients of variation (0.2761369, 0.2597991, and 0.2976122) indicate that the nonessential multicollinearity is not troubling (see Salmerón et al. [39]). Thus, the extension of the VIF seems appropriate to check if the application of the raise regression has mitigated the near multicollinearity.

Next, it is presented the estimation of the model by raise regression and the results are compared to the estimation by ridge and Lasso regression.

5.2.1. Raise Regression

When calculating the thresholds that would be obtained for VIFs by raising each variable (see

Table 10. Horizontal asymptote for VIFs after raising each variable and ${\lambda }_{min}$.

Raised	$\underset{\mathbf{\lambda }\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{WI},\mathbf{\lambda })$	$\underset{\mathbf{\lambda }\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{NWI},\mathbf{\lambda })$	$\underset{\mathbf{\lambda }\to +\mathbf{\infty }}{lim}\mathit{VIF}(\mathbf{FI},\mathbf{\lambda })$	${\mathbf{\lambda }}_{\mathit{min}}$
Variable 1	1	2.19	2.19	0.673
Variable 2	2.92	1	2.92	0.257
Variable 3	9.05	9.05	1	6.895

), it is observed that, in all cases, they are less than 10. However, when calculating ${\lambda }_{min}$ in each case, a value higher than one is only obtained when raising the third variable. Figure 3 displays the MSE for $\lambda \in [0,37)$. Note that $MSE(\widehat{\mathbf{\beta }}(\lambda ))$ is always less than the one obtained by OLS, 49.434, and presents an asymptote in $\underset{\lambda \to +\infty }{lim}MSE(\widehat{\mathbf{\beta }}(\lambda ))=45.69422$.

The following model is obtained by raising the third variable:

$$\mathbf{C}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right)\mathbf{WI}+{\beta }_3\left(\lambda \right)\mathbf{NWI}+{\beta }_4\left(\lambda \right)\widetilde{\mathbf{FI}}+\tilde{\mathbf{u}},$$

(20)

where $\widetilde{\mathbf{FI}}=\mathbf{FI}+\lambda ·{\mathbf{e}}_{\mathbf{FI}}$ being ${\mathbf{e}}_{\mathbf{FI}}$ the residuals of regression:

$$\mathbf{FI}={\alpha }_1+{\alpha }_2\mathbf{WI}+{\alpha }_3\mathbf{NWI}+\mathbf{v}.$$

Remark 6.

The coefficient of variation$\widetilde{\mathbf{FI}}$for${\lambda }^{\left(1\right)}=6.895$is 1.383309. Thus, the application of the raise regression has mitigated the nonessential multicollinearity in this variable.

Table 9 shows the results for the model in Equation (20), being $\lambda =6.895$. In this case, the MSE is the lowest possible for every possible value of $\lambda$ and lower than the one obtained by OLS for the model in Equation (19). Furthermore, in this case, the collinearity is not strong once all the VIF are lower than 10 (9.098, 9.049, and 1.031, respectively). However, the individual significance in the variable was not improved.

With the purpose of improving this situation, another variable is raised. If the first variable is selected to be raised, the following model is obtained:

$$\mathbf{C}={\beta }_1\left(\lambda \right)+{\beta }_2\left(\lambda \right)\widetilde{\mathbf{WI}}+{\beta }_3\left(\lambda \right)\mathbf{NWI}+{\beta }_4\left(\lambda \right)\mathbf{FI}+\tilde{\mathbf{u}},$$

(21)

where $\widetilde{\mathbf{WI}}=\mathbf{WI}+\lambda ·{\mathbf{e}}_{\mathbf{WI}}$ being ${\mathbf{e}}_{\mathbf{WI}}$ the residuals of regression:

$$\mathbf{WI}={\alpha }_1+{\alpha }_2\mathbf{NWI}+{\alpha }_3\mathbf{FI}+\mathbf{v}.$$

Remark 7.

The coefficient of variation of$\widetilde{\mathbf{WI}}$for${\lambda }^{\left(1\right)}=0.673$is 0.2956465. Thus, it is noted that the raise regression has lightly mitigated the nonessential mutlicollinearity of this variable.

Table 9 shows the results for the model in Equation (21), being $\lambda =0.673$. In this case, the MSE is lower than the one obtained by OLS for the model in Equation (19). Furthermore, in this case, the collinearity is not strong once all the VIF are lower than 10 (5.036024, 4.705204, and 2.470980, respectively). Note that raising this variable, the values of VIFs are lower than raising the first variable but the MSE is higher. However, this model is selected as preferable due to the individual significance being better in this model and the MSE being lower than the one obtained by OLS.

5.2.2. Ridge Regression

This subsection presents the estimation of the model in Equation (19) by ridge regression (see Hoerl and Kennard [4] or Marquardt [45]). The first step is the selection of the appropriate value of K.

The following suggestions are addressed:

• Hoerl et al. [33] proposed the value of $K_{HKB}=p·\frac{{\widehat{\sigma }}^2}{{\widehat{\mathbf{\beta }}}^t\widehat{\mathbf{\beta }}}$ since probability higher than 50% leads to a MSE lower than the one from OLS.
• García et al. [26] proposed the value of K, denoted as $K_{VIF}$, that leads to values of VIF lower than 10 (threshold traditionally established as troubling).
• García et al. [44] proposed the following values:

  $$\begin{array}{ccc}\hfill K_{exp}& =& 0.006639·e^{1-det\left(\mathbf{R}\right)}-0.00001241·n+0.005745·p,\hfill \\ \hfill K_{linear}& =& 0.01837·(1-det(\mathbf{R}\left)\right)-0.00001262·n+0.005678·p,\hfill \\ \hfill K_{sq}& =& 0.7922·{(1-det\left(\mathbf{R}\right))}^2-0.6901·(1-det\left(\mathbf{R}\right))-0.000007567·n\hfill \\ & & -0.01081·p,\hfill \end{array}$$

  where $det\left(\mathbf{R}\right)$ denotes the determinant of the matrix of correlation, $\mathbf{R}$.

The following values are obtained $K_{HKB}=0.417083$, $K_{VIF}=0.013$, $K_{exp}=0.04020704$, $K_{linear}=0.04022313$, and $K_{sq}=0.02663591$.

Table 11. Estimation of the ridge models for $K_{HKB}=0.417083$ and $K_{VIF}=0.013$. Confidence interval, at 10% confidence, is obtained from bootstrap inside the parentheses, and $R^2$ is the coefficient of determination obtained from Rodríguez et al. [28].

	Model (19) for ${\mathit{K}}_{\mathit{HKB}}=0.417083$	Model (19) for ${\mathit{K}}_{\mathit{VIF}}=0.013$
Intercept	12.2395 (6.5394, 15.9444)	18.3981 (12.1725, 24.1816)
WI	0.3495 (−0.4376, 1.2481)	0.3787 (−0.4593, 1.216)
NWI	1.6474 (−0.1453, 3.4272)	1.4295 (−0.2405, 3.2544)
FI	0.8133 (−1.5584, 3.028)	0.5467 (−1.827, 2.9238)
$R^2$	0.8957	0.9353
MSE	64.20028	47.99713

and

Table 12. Estimation of the ridge models for $K_{exp}=0.04020704$ and $K_{sq}=0.02663591$. Confidence interval, at 10% confidence, is obtained from bootstrap inside the parentheses, and $R^2$ is the coefficient of determination obtained from Rodríguez et al. [28].

	Model (19) for ${\mathit{K}}_{\mathit{exp}}=0.04020704$	Model (19) for ${\mathit{K}}_{\mathit{sq}}=0.02663591$
Intercept	17.7932 (11.4986, 22.9815)	18.0898 (11.8745, 23.8594)
WI	0.3756 (−0.4752, 1.2254)	0.3771 (−0.4653, 1.2401)
NWI	1.4512 (−0.2249, 3.288)	1.4406 (−0.2551, 3.2519)
FI	0.5737 (−1.798, 2.9337)	0.5605 (−1.6999, 2.9505)
$R^2$	0.918034	0.9183955
MSE	45.76226	46.75402

show (The results for $K_{linear}$ are not considered as they are very similar to results obtained by $K_{exp}$.) the estimations obtained from ridge estimators (expression (1)) and the individual significance intervals obtained by bootstrap considering percentiles 5 and 95 for 5000 repeats. It is also calculated the goodness of the fit by following the results shown by Rodríguez et al. [28] and the MSE.

Note that only the constant term can be considered significatively different to zero and that, curiously, the value of K proposed by Hoerl et al. [33] leads to a value of MSE higher than the one from OLS while the values proposed by García et al. [26] and García et al. [44] lead to a value of MSE lower than the one obtained by OLS. All cases lead to values of VIF lower than 10; see García et al. [26] for its calculation:

$$\begin{array}{ccc}\hfill 2.0529,1.8933\mathrm{and}1.5678& \mathrm{for}& K_{HKB},\hfill \\ \hfill 9.8856,7.5541\mathrm{and}2.7991& \mathrm{for}& K_{VIF},\hfill \\ \hfill 7.1255,5.6191\mathrm{and}2.5473& \mathrm{for}& K_{exp},\hfill \\ \hfill 8.2528,6.4123\mathrm{and}2.65903& \mathrm{for}& K_{sq}.\hfill \end{array}$$

In any case, the lack of individual significance justifies the selection of the raise regression as preferable in comparison to the models obtained by ridge regression.

5.2.3. Lasso Regression

The Lasso regression (see Tibshirani [5]) is a method initially designed to select variables constraining the coefficient to zero, being specially useful in models with a high number of independent variables. However, this estimation methodology has been widely applied in situation where the model presents worrying near multicollinearity.

Table 13. Estimation of the Lasso model for $\lambda =0.1258925$: Confidence interval at 10% confidence (obtained from bootstrap inside the parentheses).

	Model (19) for $\mathbf{\lambda }=0.1258925$
Intercept	19.1444 (13.5814489, 24.586207)
WI	0.4198 (−0.2013491, 1.052905)
NWI	1.3253 (0.0000000, 2.752345)
FI	0.4675 (−1.1574169, 2.151648)

shows results obtained by the application of the Lasso regression to the model in Equation (19) by using the package glmnet of the programming environment R Core Team [46]. Note that these estimations are obtained for the optimal value of $\lambda =0.1258925$ obtained after a k-fold cross-validation.

The inference obtained by bootstrap methodology (with 5000 repeats) allows us to conclude that in, at least, the 5% of the cases, the coefficient of NWI is constrained to zero. Thus, this variable should be eliminated from the model.

However, we consider that this situation should be avoided, and as an alternative to the elimination of variable, that is, as an alternative from the following model, the estimation by raise or ridge regression is proposed.

$$\mathbf{C}={\pi }_1+{\pi }_2\mathbf{WI}+{\pi }_3\mathbf{FI}+ϵ,$$

(22)

It could be also appropriate to apply the residualization method (see, for example, York [47], Salmerón et al. [48], and García et al. [44]), which consists in the estimation of the following model:

$$\mathbf{C}={\tau }_1+{\tau }_2\mathbf{WI}+{\tau }_3\mathbf{FI}+{\tau }_4{\mathbf{res}}_{\mathbf{NWI}}+\epsilon ,$$

(23)

where, for example, ${\mathbf{res}}_{\mathbf{NWI}}$ represents the residuals of the regression of $\mathbf{NWI}$ as a function of $\mathbf{WI}$ that will be interpreted as the part of $\mathbf{NWI}$ not related to $\mathbf{WI}$. In this case (see García et al. [44]), it is verified that ${\widehat{\pi }}_i={\widehat{\tau }}_i$ for $i=1,2,3$. That is to say, the model in Equation (23) estimates the same relationship between $\mathbf{WI}$ and $\mathbf{FI}$ with $\mathbf{C}$ as in the model in Equation (22) with the benefit that the variable $\mathbf{NWI}$ is not eliminated due to a part of it being considered.

6. Conclusions

The Variance Inflation Factor (VIF) is one of the most applied measures to diagnose collinearity together with the Condition Number (CN). Once the collinearity is detected, different methodologies can be applied as, for example, the raise regression, but it will be required to check if the methodology has mitigated the collinearity effectively. This paper extends the concept of VIF to be applied after the raise regression and presents an expression of the VIF that verifies the following desirable properties (see García et al. [26]):

• continuous in zero. That is to say, when the raising factor ($\lambda$) is zero, the VIF obtained in the raise regression coincides with the one obtained by OLS;
• decreasing as a function of the raising factor ($\lambda$). That is to say, the degree of collinearity diminishes as $\lambda$ increases, and
• always equal or higher than 1.

The paper also shows that the VIF in the raise regression is scale invariant, which is a very common transformation when working with models with collinearity. Thus, it yields identical results regardless of whether predictions are based on unstandardized or standardized predictors. Contrarily, the VIFs obtained from other penalized regressions (ridge regression, Lasso, and Elastic Net) are not scale invariant and hence yield different results depending on the predictor scaling used.

Another contribution of this paper is the analysis of the asymptotic behavior of the VIF associated with the raised variable (verifying that its limit is equal to 1) and associated with the rest of the variables (presenting an horizontal asymptote). This analysis allows to conclude that

• It is possible to know a priori how far each of the VIFs can decrease simply by calculating their horizontal asymptote. This could be used as a criterion to select the variable to be raised, the one with the lowest horizontal asymptote being chosen.
• If there is asymptote under the threshold established as worrying, the extension of the VIF can be applied to select the raising factor considering the value of $\lambda$ that verifies $VIF(k,\lambda )<10$ for $k=2,\dots ,p$.
• It is possible that the collinearity is not mitigated with any value of $\lambda$. This can happen when at least one horizontal asymptote is greater than the threshold. In that case, a second variable has to be raised. García and Ramírez [42] and García et al. [31] show the successive raising procedure.

On the other hand, since the raise estimator is biased, the paper analyzes its Mean Square Error (MSE), showing that there is a value of $\lambda$ that minimizes the possibility of the MSE being lower than the one obtained by OLS. However, it is not guaranteed that the VIF for this value of $\lambda$ presents a value less than the established thresholds. The results are illustrated with two numerical examples, and in the second one, the results obtained by OLS are compared to the results obtained with the raise, ridge, and Lasso regressions that are widely applied to estimated models with worrying multicollinearity. It is showed that the raise regression can compete and even overcome these methodologies.

Finally, we propose as future lines of research the following questions:

• The examples showed that the coefficients of variation increase after raising the variables. This fact is associated with an increase in the variability of the variable and, consequently, with a decrease of the near nonessential multicollinearity. Although a deeper analysis is required, it seems that raise regression mitigates this kind of near multicollinearity.
• The value of the ridge factor traditionally applied, $K_{HKB}$, leads to estimators with smaller MSEs than the OLS estimators with probability greater than 0.5. In contrast, the value of the raising factor ${\lambda }_{min}$ always leads to estimators with smaller MSEs than OLS estimators. Thus, it is deduced that the ridge regression provides estimators with MSEs higher than the MSEs of OLS estimators with probability lower than 0.5. These questions seem to indicate that, in terms of MSE, the raise regression can present better behaviour than the ridge regression. However, the confirmation of this judgment will require a more complete analysis, including other aspects such as interpretability and inference.