Comparisons of Penalized Regression Methods under High-Dimensional Sparse Data with Correlated Variables

Abstract

Regression models are frequently used to explain a response variable using independent variables in statistics. However, it is common to encounter situations in which the number of independent variables exceeds the number of observations and predictors are correlated. In this instance, a large number of predictors are statistically insignificant, also known as sparse data. Standard statistical methods do not always apply to such data. Problematic aspects include interpretation, estimation inefficiency, and computation. The penalized regression method, which consists of Ridge, least absolute shrinkage and selection operator (LASSO), elastic net (Enet), adaptive LASSO (ALASSO), and adaptive elastic net (AEnet), is frequently employed during the estimation and variable selection phases. The purpose of this paper was to assess the prediction and variable selection performances of Ridge, Enet, LASSO, ALASSO, and AEnet methods in multiple linear regression with normal or positively skewed error terms, sparse data, and correlated independent variables. In addition, Poisson and logistic regression models are studied. The adaptive weights are created using the remaining three estimators: Ridge, Enet, and LASSO. The results indicate that the Ridge estimator is a viable initial adaptive weight estimator for ALASSO and AEnet. In terms of prediction, AEnet and ALASSO typically outperform the competition. Given the objectives, different tactics are necessary to achieve the lowest false positive rate (FPR) and false negative rate (FNR). Enet or AEnet is essential to attain the lowest FPR, while LASSO or ALASSO will yield the lowest FNR.