Learning to Forget Attention: Memory Consolidation for Adaptive Compute Reduction

Hybrid architectures combining state-space models with attention have achieved strong efficiency-quality tradeoffs, yet existing approaches either apply attention uniformly or learn static sparse patterns. This misses a key opportunity: \emph{attention demand should decrease over time as recurring patterns become familiar}. We present a surprising finding from analyzing GPT-2 models: $\mathbf{\text{88\%}}$ of attention operations retrieve information already predictable from the model’s hidden state, and this redundancy does \emph{not} decrease during training. Motivated by this observation, we introduce $\mathbf{\text{ours}}$ ($\mathbf{\text{C}}$onsolidation-based $\mathbf{\text{R}}$outing for $\mathbf{\text{A}}$daptive $\mathbf{\text{M}}$emory), a biologically inspired memory consolidation mechanism that gradually distills episodic retrievals into parametric semantic memory. Unlike prior sparse attention methods, \ours{} exhibits \emph{decreasing attention utilization} over training, achieving a $\mathbf{\text{37.8}}\times$ reduction through a sharp phase transition at approximately 3K steps. We prove that this capability is \emph{impossible} without consolidation: any static routing scheme requires $\mathrm{\Omega }(f\cdot n)$ attention for tasks with recurring patterns of frequency $f$. On our proposed SRCD benchmark, \ours{} achieves $\mathbf{\text{100\%}}\mathbf{\text{retrieval}}\mathbf{\text{accuracy}}$ at 1.6% attention compute (vs.\ 68% for baselines), and consolidated patterns transfer to unseen tasks with $\mathbf{\text{48–52\%}}$ attention reduction without retraining. Remarkably, the learned consolidation dynamics quantitatively match human episodic-to-semantic memory transition curves from cognitive psychology ($\gamma =0.43$ vs.\ ${\gamma }_{\text{human}}\approx 0.4$–$0.5$). Code and benchmarks are available at [anonymized].