Spectral Edge Dynamics: An Analytical-Empirical Study of Phase Transitions in Neural Network Training

We develop the spectral edge analysis: phase transitions in neural network training – grokking, capability gains, loss plateaus – are controlled by the spectral gap of the rolling-window Gram matrix of parameter updates. In the extreme aspect ratio regime (parameters $P\sim {10}^8$, window $W\sim 10$), the classical BBP detection threshold is vacuous; the operative structure is the intra-signal gap separating dominant from subdominant modes at position $k^{\ast }=\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{m}\mathrm{a}\mathrm{x}{\sigma }_j\mathrm{/}{\sigma }_{j+1}$. From three assumptions we derive: (i) gap dynamics governed by a Dyson-type ODE with curvature asymmetry, damping, and gradient driving; (ii) a spectral loss decomposition linking each mode’s learning contribution to its Davis–Kahan stability coefficient; (iii) the Gap Maximality Principle, showing that $k^{\ast }$ is the unique dynamically privileged position – its collapse is the only one that disrupts learning, and it sustains itself through an $\alpha$-feedback loop requiring no assumption on the optimizer. The adiabatic parameter $\mathcal{A}=\mathrm{\parallel }\mathrm{\Delta }G{\mathrm{\parallel }}_F\mathrm{/}(\eta g^2)$ controls circuit stability: $\mathcal{A}\ll 1$ (plateau), $\mathcal{A}\sim 1$ (phase transition), $\mathcal{A}\gg 1$ (forgetting). Tested across six model families (150K–124M parameters): gap dynamics precede every grokking event (24/24 with weight decay, 1/24 without), the gap position is optimizer-dependent (Muon: $k^{\ast }=1$, AdamW: $k^{\ast }=2$ on the same model), and 19/20 quantitative predictions are confirmed. The framework is consistent with the edge of stability, Tensor Programs, Dyson Brownian motion, the Lottery Ticket Hypothesis, and neural scaling laws.