Spectral-Window Hybrid (SWH)

Scaling sequence modeling to extreme contexts requires balancing computational efficiency with representational expressivity. While Transformers provide precise retrieval via the attention mechanism, their quadratic $\mathcal{O}(T^2)$ complexity limits their application to long-horizon tasks. In this work, we propose the $\mathbf{\text{Spectral-Window}}\mathbf{\text{Hybrid}}\mathbf{\text{(SWH)}}$, an architecture that decouples sequence modeling into two $\mathit{\text{parallel}}$ streams: a global branch utilizing the Convolution Theorem to model long-range decay dynamics in $\mathcal{O}(T\log T)$ time, and a local branch employing sliding-window attention for token interactions within a bounded context. By aggregating these representations, SWH avoids the computational bottleneck of global attention while retaining local precision. We demonstrate that SWH matches the perplexity of standard Transformers on short contexts while enabling efficient linear scaling to extended sequences. The code is available at https://github.com/VladimerKhasia/SWH