SANE: State Anomaly Neutralization for Stable Extreme-Context Delta-Rule Models

Delta-Rule recurrent models maintain a fixed-size state, enabling $O(1)$ inference memory but potentially becoming unstable under extreme-context extrapolation. By tracking RWKV-7 over sequences of up to 100M tokens, we empirically identify a distinct failure pattern: $\mathbf{\text{localized}}\mathbf{\text{norm}}\mathbf{\text{explosion}}\mathbf{\text{atop}}\mathbf{\text{a}}\mathbf{\text{relatively}}\mathbf{\text{sparse}}\mathbf{\text{substrate}}$, rather than global state saturation. Analysis of the recurrent update suggests that persistent decay keeps weakly updated entries small, whereas uneven injections allow a few channels to accumulate extreme values. Motivated by this diagnosis, we propose $\mathbf{\text{State}}\mathbf{\text{Anomaly}}\mathbf{\text{Neutralization}}\mathbf{\text{(SANE)}}$, which applies adaptive $\tanh$ compression at chunk boundaries while preserving the intra-chunk parallel structure. Within a safe threshold range ($3\le \alpha \le 5$), SANE matches the baseline on 11 short-context reasoning benchmarks with no statistically significant degradation. After a 100M-token prefix, which exceeds the training length by over $\mathrm{24,000}\times$, SANE retains functional reasoning ($33.46$–$35.56$) while the baseline encounters numerical overflow. In contrast, overly permissive thresholds ($\alpha \ge 8$) remain numerically stable but lose reasoning capability entirely, showing that numerical stabilization alone does not guarantee functional reasoning and revealing a capacity–stability trade-off in state compression.