Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance

Knowledge distillation trains a small student model to reproduce the outputs of a large teacher model, and its progress is typically monitored through the teacher–student discrepancy. The quantity of ultimate interest, however, is the student’s error with respect to the true task. We study the relation between these two objectives in a minimal three-party model, a true teacher (generative model), a teacher, and a student, all soft committee machines, in which the true teacher contains a shared latent factor that the teacher cannot represent, with mismatch strength controlled by a single scalar $\text{dmiss}$. Within an order-parameter description of online distillation, and exploiting closed-form (arcsine-type) expressions for all errors under error-function activations, we prove that the learning dynamics and the distillation error $\text{Ets}$ are exactly invariant to $\text{dmiss}$, whereas the true error $\text{Etzs}$ and the gap $\mathrm{\Delta }=\text{Etzs}-\text{Ets}$ are strictly increasing in $\text{dmiss}$, with a rate that is amplified linearly by the complexity $M_0$ of the true teacher. Numerical phase diagrams over the plane spanned by true-teacher complexity and student capacity confirm the predicted deformation: the contours of $\text{Ets}$ do not move while the landscape of $\text{Etzs}$ rises systematically, and a teacher-miss regime, where mimicry succeeds but the task fails, expands with $\text{dmiss}$. The results give a quantitative warning against evaluating distillation solely through teacher-mimicry metrics and identify the gap $\mathrm{\Delta }$ as a minimal diagnostic for distinguishing teacher-miss from capacity-limited failure.