LayerNorm as Implicit Gain Control in Looped Transformers

In pre-LayerNorm looped transformers, LayerNorm inside the recurrent block acts as an implicit gain controller: by coupling the block’s local Lipschitz constant inversely to the activation scale, it renders the recurrence Jacobian non-normal – asymptotically contractive at every verified fixed point even where its operator norm exceeds 1 – so the true stability budget is the spectral margin, not an operator-norm bound. That margin depletes as the carry $\rho \to 1$, and a minority of initializations never converge to a fixed point at all, so the diagonal carry constraint $\rho (\bar{A})<1$ is necessary but not sufficient for convergence of the full recurrence. Training experiments across six tasks, including a controlled ablation, reveal that the linear carry is not the depth-memory mechanism: gradient descent routes memory through the block’s more expressive nonlinear recurrence and leaves the stability-constrained carry at rest – the carry’s role is stabilization, not memory. We characterize the boundary of this claim: on tasks with axis-aligned per-channel structure, gradient descent does recruit the carry. All results are derived analytically and verified in a from-scratch, CPU-scale implementation; verification at larger scale is needed.