Characterization of Gaussian Universality Breakdown in High-Dimensional Empirical Risk Minimization

We study high-dimensional convex empirical risk minimization (ERM) under general non-Gaussian data designs. By heuristically extending the Convex Gaussian Min-Max Theorem (CGMT) to non-Gaussian settings, we derive an asymptotic min-max characterization of key statistics, enabling approximation of the mean ${\mu }_{\widehat{\theta }}$ and covariance $C_{\widehat{\theta }}$ of the ERM estimator $\widehat{\theta }$. Specifically, under a concentration assumption on the data matrix and standard regularity conditions on the loss and regularizer, we show that for a test covariate $x$ independent of the training data, the projection ${\widehat{\theta }}^{\mathrm{\top }}x$ approximately follows the convolution of the generally non-Gaussian distribution of ${\mu }_{\widehat{\theta }}^{\mathrm{\top }}x$ with an independent centered Gaussian variable of variance $\mathrm{t}\mathrm{r}(C_{\widehat{\theta }}\mathbb{E}[x{x}^{\mathrm{\top }}])$. This result clarifies the scope and limits of Gaussian universality for ERMs. Additionally, we prove that any ${\mathcal{C}}^2$ regularizer is asymptotically equivalent to a quadratic form determined solely by its Hessian at zero and gradient at ${\mu }_{\widehat{\theta }}$. Numerical simulations across diverse losses and models are provided to validate our theoretical predictions and qualitative insights.