The Raise Regression: Justification, Properties and Application
Román Salmerón-Gómez
Department of Quantitative Methods for Economic and Business, University of Granada, Granada, Spain
Search for more papers by this authorCorresponding Author
Catalina B. García-García
Department of Quantitative Methods for Economic and Business, University of Granada, Granada, Spain
Corresponding to: Catalina García, Department of Quantitative Methods for Economic and Business, Granada University, Granada, Spain.
Email: [email protected]
Search for more papers by this authorJosé García-Pérez
Department of Economy and Business, University of Almeria, Almería, Spain
Search for more papers by this authorRomán Salmerón-Gómez
Department of Quantitative Methods for Economic and Business, University of Granada, Granada, Spain
Search for more papers by this authorCorresponding Author
Catalina B. García-García
Department of Quantitative Methods for Economic and Business, University of Granada, Granada, Spain
Corresponding to: Catalina García, Department of Quantitative Methods for Economic and Business, Granada University, Granada, Spain.
Email: [email protected]
Search for more papers by this authorJosé García-Pérez
Department of Economy and Business, University of Almeria, Almería, Spain
Search for more papers by this authorSummary
Multicollinearity results in inflation in the variance of the ordinary least squares estimators due to the correlation between two or more independent variables (including the constant term). A widely applied solution is to estimate with penalised estimators such as the ridge estimator, which trade off some bias in the estimators to gain a reduction in the variance of these estimators. Although the variance diminishes with these procedures, all seem to indicate that the inference and goodness of fit are controversial. Alternatively, the raise regression allows mitigation of the problems associated with multicollinearity without the loss of inference or the coefficient of determination. This paper completely formalises the raise estimator. For the first time, the norm of the estimator, the behaviour of the individual and joint significance, the behaviour of the mean squared error and the coefficient of variation are analysed. We also present the generalisation of the estimation and the relation between the raise and the residualisation estimators. To have a better understanding of raise regression, previous contributions are also summarised: its mean squared error, the variance inflation factor, the condition number, adequate selection of the variable to be raised, the successive raising, and the relation between the raise and the ridge estimator. The usefulness of the raise regression as an alternative to mitigate multicollinearity is illustrated with two empirical applications.
References
- Belsley, D.A., Kuh, E. & Welsch, R.E. (2005). Regression diagnostics: Identifying influential data and sources of collinearity, Vol. 571. John Wiley & Sons.
- Björkström, A. (2001). Ridge regression and inverse problems. Stockholm University, Department of Mathematics.
- De Hierro, AFRL, García, C. & Salmerón, R. (2021). Analysis of the condition number in the raise regression. Commun. Stat.-Theory Methods, 50(24), 6195–6210.
- Dissanayake, A. & Wijekoon, P. 2016. lrmest: Different types of estimators to deal with multicollinearity. https://CRAN.R-project.org/package%3Dlrmest. R package version 3.0.
- Farebrother, R.W. (1976). Further results on the mean square error of ridge regression. J. R. Stat. Soc. Ser. B (Methodol.), 38(3), 248–250.
10.1111/j.2517-6161.1976.tb01588.x Google Scholar
- Gökpınar, E. & Ebegil, M. (2016). A study on tests of hypothesis based on ridge estimator. Gazi Univ. J. Sci., 29(4), 769–781.
- García, C.B., Salmerón, R., García, C. & García, J. (2020). Residualization: Justification, properties and application. J. Appl. Stat., 47(11), 1990–2010.
- García-Pérez, J., del Mar López-Martín, M., García-García, C. & Salmerón-Gómez, R. (2020). A geometrical interpretation of collinearity: A natural way to justify ridge regression and its anomalies. Int. Stat. Rev., 88(3), 776–792.
- García, C.G., Pérez, J.G. & Liria, J.S. (2011). The raise method. an alternative procedure to estimate the parameters in presence of collinearity. Qual. Quant., 45(2), 403–423.
- García, J. & Ramirez, D.E. (2017). The successive raising estimator and its relation with the ridge estimator. Commun. Stat.-Theory Methods, 46(22), 11123–11142.
- Goeman, J., Meijer, R. & Chaturvedi, N. 2018. L1 and L2 penalized regression models. Vignette R Package Penalized. URL http://cran.nedmirror.nl/web/packages/penalized/vignettes/penalized.pdf
- Halawa, A.M. & El Bassiouni, M.Y. (2000). Tests of regression coefficients under ridge regression models. J. Stat. Comput. Simul., 65(1-4), 341–356.
- Hoerl, A.E., Kannard, R.W. & Baldwin, K.F. (1975). Ridge regression: Some simulations. Commun. Stat.-Theory Methods, 4(2), 105–123.
- Hoerl, A.E. & Kennard, R.W. (1970a). Ridge regression: Applications to nonorthogonal problems. Technometrics, 12(1), 69–82.
- Hoerl, A.E. & Kennard, R.W. (1970b). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1), 55–67.
- Hoerl, A.E. & Kennard, W.R. (1990). Ridge regression: Degrees of freedom in the analysis of variance. Commun. Stat.-Simul. Comput., 19(4), 1485–1495.
- Jacob, J. & Varadharajan, R. (2023). Simultaneous raise regression: A novel approach to combating collinearity in linear regression models. Qual. Quant., 57(5), 4365–4386.
10.1007/s11135-022-01557-9 Google Scholar
- Kaçiranlar, S., Sakallioğlu, S., Akdeniz, F., Styan, G.P.H. & Werner, H.J. (1999). A new biased estimator in linear regression and a detailed analysis of the widely-analysed dataset on portland cement. Sankhyā: Ind. J. Stat. Ser. B, 61(3), 443–459.
- Kibria, B.M. & Lukman, A.F. (2020). A new ridge-type estimator for the linear regression model: Simulations and applications. Scientifica, 2020, 1–16.
- Li, Y. & Yang, H. (2012). A new liu-type estimator in linear regression model. Stat. Papers, 53(2), 427–437.
- Lukman, A.F., Ayinde, K., Binuomote, S. & Clement, O.A. (2019). Modified ridge-type estimator to combat multicollinearity: Application to chemical data. J. Chemometr., 33(5), e3125.
- Marquardt, D.W. (1970). Generalized inverses, ridge regression, biased linear estimation, and nonlinear estimation. Technometrics, 12(3), 591–612.
- Marquardt, D.W. (1980). Comment: You should standardize the predictor variables in your regression models. J. Am. Stat. Assoc., 75(369), 87–91.
- Marquardt, D.W. & Snee, R.D. (1975). Ridge regression in practice. Am. Stat., 29(1), 3–20.
- O'Brien, R.M. (2007). A caution regarding rules of thumb for variance inflation factors. Qual. Quant., 41(5), 673–690.
- Obenchain, R.L. (1977). Classical f-tests and confidence regions for ridge regression. Technometrics, 19(4), 429–439.
- Puntanen, S., Styan, G.P.H. & Isotalo, J. (2011). Matrix tricks for linear statistical models: Our personal top twenty. Springer Science & Business Media.
10.1007/978-3-642-10473-2 Google Scholar
- Roldán López De Hierro, A.F., Gómez Salmerón, R. & García García, C. (2018). About the limits of raise regression to reduce condition number when three explanatory variables are involved. Rect@, 19(1), 45–62.
10.24309/recta.2018.19.1.04 Google Scholar
- Salmerón Gómez, R., García, C.B.G. & Pérez, J.G. (2022). The multicoll package versus other existing packages in r to detect multicollinearity. Comput. Econ., 6, 439–450.
10.1007/s10614-021-10154-1 Google Scholar
- Salmerón, R., García, C.B. & García, J. (2018). Variance inflation factor and condition number in multiple linear regression. J. Stat. Comput. Simul., 88(12), 2365–2384.
- Salmerón, R., García, C.G. & García Pérez, J. (2020). Detection of near-multicollinearity through centered and noncentered regression. Mathematics, 8, 931.
- Salmerón, R., García, C., García, J. & López, M.M. (2017). The raise estimator estimation, inference, and properties. Commun. Stat.-Theory Methods, 46(13), 6446–6462.
- Salmerón, R., Rodríguez, A., García, C.G. & García Pérez, J. (2020). The vif and mse in raise regression. Mathematics, 8(4), 605.
- Salmerón, R., Rodríguez-Sánchez, A. & García-García, C. (2019). Diagnosis and quantification of the non-essential collinearity. Comput. Stat., 35, 647–666.
- Salmerón-Gómez, R., García-García, C. & García-Pérez, J. (2021). A guide to using the r package “multicoll” for detecting multicollinearity. Comput. Econ., 57, 529–536.
- Salmerón, R., García, C. & García, J. 2022. multicoll: Collinearity detection in a multiple linear regression model. https://CRAN.R-project.org/package%3DmultiColl. R package version 2.0.
- Sengupta, N. & Sowell, F. (2020). On the asymptotic distribution of ridge regression estimators using training and test samples. Econometrics, 8(4), 39.
- Singh, R. (2011). Two famous conflicting claims in econometrics. Asian-African J. Econ., 11(1), 185–186.
- Snee, R.D. & Marquardt, D.W. (1984). Comment: Collinearity diagnostics depend on the domain of prediction, the model, and the data. Am. Stat., 38(2), 83–87.
- Theobald, C.M. (1974). Generalizations of mean square error applied to ridge regression. J. R. Stat. Soc.: Ser. B (Methodol.), 36(1), 103–106.
10.1111/j.2517-6161.1974.tb00990.x Google Scholar
- Tikhonov, A.N. (1963). On the regularization of ill-posed problems. In Doklady akademii nauk, Vol. 153, pp. 49–52, Russian Academy of Sciences.
- Trenklar, G. (1980). Generalized mean squared error comparisons of biased regression estimators. Commun. Stat.-Theory Methods, 9(12), 1247–1259.
- Vanhove, J. (2021). Collinearity isn't a disease that needs curing. Meta-Psychol., 5, 1–11.
10.15626/MP.2021.2548 Google Scholar
- Willan, A.R. & Watts, D.G. (1978). Meaningful multicollinearity measures. Technometrics, 20(4), 407–412.
- Woods, H., Steinour, H.H. & Starke, H.R. (1932). Effect of composition of portland cement on heat evolved during hardening. Industr. Eng. Chem., 24(11), 1207–1214.