Open Access
2022 LASSO risk and phase transition under dependence
Hanwen Huang
Author Affiliations +
Electron. J. Statist. 16(2): 6512-6552 (2022). DOI: 10.1214/22-EJS2092

Abstract

We consider the problem of recovering a k-sparse signal β0Rp from noisy observations y=Xβ0+wRn. One of the most popular approaches is the l1-regularized least squares, also known as LASSO. We analyze the mean square error of LASSO in the case of random designs in which each row of X is drawn from distribution N(0,Σ) with general Σ. We first derive the asymptotic risk of LASSO for w0 in the limit of n,p with npδ[0,). We then examine conditions on n, p, and k for LASSO to exactly reconstruct β0 in the noiseless case w=0. A phase boundary δc=δ(ϵ) is precisely established in the phase space defined by 0δ,ϵ1, where ϵ=kp. Above this boundary, LASSO perfectly recovers β0 with high probability. Below this boundary, LASSO fails to recover β0 with high probability. While the values of the non-zero elements of β0 do not have any effect on the phase transition curve, our analysis shows that δc does depend on the signed pattern of the nonzero values of β0 for general ΣIp×p. This is in sharp contrast to the previous phase transition results derived in i.i.d. case with Σ=Ip×p where δc is completely determined by ϵ regardless of the distribution of β0. Underlying our formalism is a recently developed efficient algorithm called approximate message passing (AMP) algorithm. We generalize the state evolution of AMP from i.i.d. case to general case with ΣIp×p. Extensive computational experiments confirm that our theoretical predictions are consistent with simulation results on moderate size system.

Funding Statement

This research is supported in part by Division of Mathematical Sciences (National Science Foundation) Grant DMS-1916411.

Acknowledgments

The author thanks the editor, associate editor, and two referees for many helpful comments and suggestions which led to a much improved presentation.

Citation

Download Citation

Hanwen Huang. "LASSO risk and phase transition under dependence." Electron. J. Statist. 16 (2) 6512 - 6552, 2022. https://doi.org/10.1214/22-EJS2092

Information

Received: 1 December 2021; Published: 2022
First available in Project Euclid: 7 December 2022

MathSciNet: MR4522375
zbMATH: 07633944
Digital Object Identifier: 10.1214/22-EJS2092

Subjects:
Primary: 62F12 , 62F12
Secondary: 62F12

Keywords: Asymptotic risk , Lasso , Mean square error , phase transition , state evolution

Vol.16 • No. 2 • 2022
Back to Top