Language Model Circuits Are Sparse in the Neuron Basis

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques such as $\mathit{\text{sparse}}\mathit{\text{autoencoders}}$ (SAEs) to decompose the neuron basis into more interpretable units of model computation, for tasks such as $\mathit{\text{circuit}}\mathit{\text{tracing}}$. However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that $\mathbf{\text{MLP}}\mathbf{\text{neurons}}\mathbf{\text{are}}\mathbf{\text{as}}\mathbf{\text{sparse}}\mathbf{\text{a}}\mathbf{\text{feature}}\mathbf{\text{basis}}\mathbf{\text{as}}\mathbf{\text{SAEs}}$. We use this finding to develop an end-to-end pipeline for circuit tracing on the MLP neuron basis, which locates causal circuitry on a variety of tasks using gradient-based attribution. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx {10}^2$ MLP neurons is enough to control model behaviour. On the multi-hop city $\to$ state $\to$ capital task from Lindsey et al., 2025, we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g.~`map city to its state’), and can be steered to change the model’s output. This work thus advances automated interpretability of language models without additional training costs.