Unified Description of Learning Dynamics in the Soft Committee Machine from Finite to Ultra-Wide Regimes

We study the learning dynamics of the soft committee machine (SCM) with Rectified Linear Unit (ReLU) activation using a statistical-mechanics approach within the annealed approximation. The SCM consists of a student network with $N$ input units and $K$ hidden units trained to reproduce the output of a teacher network with $M$ hidden units. We introduce a reduced set of macroscopic order parameters that yields a unified description valid from the conventional regime $K\ll N$ to the ultra-wide limit $K\ge N$. The control parameter $\alpha$, proportional to the ratio of training samples to adjustable weights, serves as an effective measure of dataset size. For small $\gamma =M\mathrm{/}N$, we recover a continuous phase transition at ${\alpha }_c\approx 2\pi$ from an unspecialized, permutation-symmetric state to a specialized state in which student units align with the teacher. For finite $\gamma$, the transition disappears and the generalization error decreases smoothly with dataset size, reaching a low plateau when $\gamma =1$. In the asymptotic limit $\alpha \to \mathrm{\infty }$, the error scales as ${\epsilon }_g\propto 1\mathrm{/}\alpha$, independent of $\gamma$ and $K$. The results highlight the central role of network dimensions in SCM learning and provide a framework extendable to other activations and quenched analyses.