Power-Law Spectrum of the Random Feature Model

Scaling laws for neural networks, in which the loss decays as a power-law in the number of parameters, data, and compute, depend fundamentally on the spectral structure of the data covariance, with power-law eigenvalue decay appearing ubiquitously in vision and language tasks. A central question is whether this spectral structure is preserved or destroyed when data passes through the basic building block of a neural network: a random linear projection followed by a nonlinear activation. We study this question for the random feature model: given data $x\sim N(0,H)\in {\mathbb{R}}^v$ where $H$ has $\alpha$-power-law spectrum (${\lambda }_j(H)\asymp j^{-\alpha }$, $\alpha >1$), a Gaussian sketch matrix $W\in {\mathbb{R}}^{v\times d}$, and an entrywise monomial $f(y)=y^p$, we characterize the eigenvalues of the population random-feature covariance ${\mathbb{E}}_x[\frac{1}{d}f(W^{\mathrm{\top }}x{)}^{\otimes 2}]$. We prove matching upper and lower bounds: for all $1\le j\le c_{1}d{\log }^{-(p+1)}(d)$, the $j$-th eigenvalue is of order ${({\log }^{p-1}(j+1)\mathrm{/}j)}^{\alpha }$. For $c_{1}d{\log }^{-(p+1)}(d)\le j\le d$, the $j$-th eigenvalue is of order $j^{-\alpha }$ up to a polylog factor. That is, the power-law exponent $\alpha$ is inherited exactly from the input covariance, modified only by a logarithmic correction that depends on the monomial degree $p$. The proof combines a dyadic head-tail decomposition with Wick chaos expansions for higher-order monomials and random matrix concentration inequalities.