FlashMemory-DeepSeek-V4: Lightning Index Ultra-Long Context via Lookahead Sparse Attention

Conventional LLMs keep the full KV cache loaded during decoding, causing a severe GPU memory bottleneck for ultra-long context serving. In this report, we propose $\mathbf{\text{Lookahead}}\mathbf{\text{Sparse}}\mathbf{\text{Attention}}\mathbf{\text{(LSA)}}$, a novel inference paradigm powered by a Neural Memory Indexer built upon the DeepSeek-V4 architecture. Rather than passively attending to all historical tokens, LSA proactively predicts future context demands and preserves only the query-critical KV chunks in the GPU memory. Crucially, we instantiate this architecture via a $\mathbf{\text{backbone-free}}\mathbf{\text{decoupled}}\mathbf{\text{training}}$ strategy. By formulating the indexer as a standard dual-encoder architecture, we train it independently using standard retrieval training frameworks without ever loading the massive backbone model into GPU memory. We demonstrate that this ``less is more’’ paradigm significantly maximizes serving efficiency while acting as an effective attention denoiser in tasks that rely on long-term global memory. Across primary long-context evaluation suites (e.g., LongBench-v2, LongMemEval, and RULER), $\mathtt{\text{FM-DS-V4}}$ compresses the average physical KV cache footprint down to merely 13.5% of the full-context baseline, while consistently preserving or slightly elevating downstream accuracy (+0.6% absolute margin on average). At 1M context, per-decode-token compute drops to 0.30$\times$ of the baseline and GPU KV cache shrinks by 90% (3.73$\to$0.37 GB), translating into $\mathbf{\text{2.8}}\times \mathbf{\text{aggregate}}\mathbf{\text{throughput}}\mathbf{\text{and}}\mathbf{\text{2.7}}\times \mathbf{\text{concurrency}}\mathbf{\text{gains}}$ in PD-disaggregated serving on 8$\times$H20 GPUs.