On the Optimal Weighted ${\mathrm{\ell }}_2$ Regularization in Overparameterized Linear Regression

We consider the linear model $$\mathbf{y}$ = $\mathbf{X}$ $\mathbf{\beta }$\star + $\mathbf{ϵ}$$with$$\mathbf{X}$\in $\mathbb{R}$^{n\times p}$ in the overparameterized regime $p>n$. We estimate ${\mathbf{\beta }}_{\star }$ via generalized (weighted) ridge regression: $\hat{$\mathbf{\beta }$}\lambda = \left($\mathbf{X}$^T$\mathbf{X}$ + \lambda $\mathbf{\Sigma }$_w\right)^\dagger $\mathbf{X}$^T$\mathbf{y}$$,where$$\mathbf{\Sigma }$_w$ is the weighting matrix. Under a random design setting with general data covariance ${\mathbf{\Sigma }}_x$ and anisotropic prior on the true coefficients $\mathbb{E}{\mathbf{\beta }}_{\star }{\mathbf{\beta }}_{\star }^T={\mathbf{\Sigma }}_{\beta }$, we provide an exact characterization of the prediction risk $\mathbb{E}(y-{\mathbf{x}}^T{\widehat{\mathbf{\beta }}}_{\lambda }{)}^2$ in the proportional asymptotic limit $p\mathrm{/}n\to \gamma \in (1,\mathrm{\infty })$. Our general setup leads to a number of interesting findings. We outline precise conditions that decide the sign of the optimal setting ${\lambda }_{\mathrm{o}\mathrm{p}\mathrm{t}}$ for the ridge parameter $\lambda$ and confirm the implicit ${\mathrm{\ell }}_2$ regularization effect of overparameterization, which theoretically justifies the surprising empirical observation that ${\lambda }_{\mathrm{o}\mathrm{p}\mathrm{t}}$ can be negative in the overparameterized regime. We also characterize the double descent phenomenon for principal component regression (PCR) when both $\mathbf{X}$ and ${\mathbf{\beta }}_{\star }$ are anisotropic. Finally, we determine the optimal weighting matrix ${\mathbf{\Sigma }}_w$ for both the ridgeless ($\lambda \to 0$) and optimally regularized ($\lambda ={\lambda }_{\mathrm{o}\mathrm{p}\mathrm{t}}$) case, and demonstrate the advantage of the weighted objective over standard ridge regression and PCR.