Phase Transition from Random Features to Fully Trainable Networks

Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent $\alpha >1$ and leverage compatibility, we establish a squared $L_2$ approximation rate $\mathcal{O}(({\kappa }_P\log (m\mathrm{/}\delta )\mathrm{/}m{)}^{\alpha })$ for two-layer networks of width $m$ and trainable proportion $P$, where ${\kappa }_P=(P\mathrm{/}{\kappa }_1^{\ast }+(1-P)\mathrm{/}{\kappa }_0^{\ast }{)}^{-1}$. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least $1-\delta$. The constants ${\kappa }_0^{\ast }$ and ${\kappa }_1^{\ast }$ quantify the fixed and trainable groups’ sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension $d$, we verify these assumptions with $\alpha =(d+2)\mathrm{/}(d-1)$ and ${\kappa }_0^{\ast }=4\sqrt{2}\tau (d+1)\mathrm{/}d$ for initialization scale $\tau \ge 1$, and ${\kappa }_1^{\ast }=1$ under ridge leverage sampling. For $d=5$ and $\tau =10$, a trainable proportion of approximately $4.1\mathrm{\%}$ captures $90\mathrm{\%}$ of the reduction in the approximation bound from $P=0$ to $P=1$. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.