Sharp Rates and a One-Line Correction for Spectral Representation Learning

A self-supervised encoder is trained once, frozen, and reused through lightweight probes on tasks nobody named at training time; the practitioner’s question is when the off-the-shelf features are good enough and when they need fixing. Canonical correlation analysis, HGR maximal correlation, and the population optimum of the spectral contrastive loss all return the top-$k$ singular subspace of a cross-view dependence operator, justified by isotropy: if the task prior has no directional preference, that subspace is universally optimal. We show isotropy is the wrong hypothesis. The prior enters the transfer risk only through the task covariance $\mathrm{\Lambda }=\mathbb{E}[\mathrm{\Delta }{\mathrm{\Delta }}^{\mathrm{\top }}]$, and only through its compression onto the operator’s leading singular directions; what matters is not whether $\mathrm{\Lambda }$ is isotropic but whether its preferred directions are ordered consistently with the operator’s spectrum. We prove matching two-sided rates—worst-case regret is exactly $1-1\mathrm{/}\kappa (\mathrm{\Lambda })$, refines to $1-A_k$ for an alignment coefficient $A_k$, localizes to the top-$2k$ subspace, becomes second order under a spectral gap, and is improvable by no task-agnostic representation—and show why alignment is generic: incoherent preferences cancel in high dimension, and $T$ diverse tasks force $\alpha =\tilde{O}(\sqrt{d_x\mathrm{/}T})$, a quantitative account of why task diversity, not symmetry, makes self-supervised features transfer. The governing statistics cost $O(k{d}_x^2)$, and when they signal misalignment a one-line reweighting of the positive-pair term provably restores exact optimality. The result is a diagnostic that answers the practitioner’s question from a small labelled budget and refuses when the task bank cannot support the width requested; on controlled data it takes a regret of $0.86$ down to $0.003$, and on a CIFAR-100 encoder it correctly predicts that no correction is needed.