PiKV: KV Cache Management System for Mixture of Experts

As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce $\mathbf{\text{PiKV}}$, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages $\mathit{\text{expert-sharded}}\mathit{\text{KV}}\mathit{\text{storage}}$ to partition caches across GPUs, $\mathit{\text{PiKV}}\mathit{\text{routing}}$ to reduce token-to-KV access, and a $\mathit{\text{PiKV}}\mathit{\text{Scheduling}}$ to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates $\mathit{\text{PiKV}}\mathit{\text{Compression}}$ modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: \href{https://github.com/NoakLiu/PiKV}{https://github.com/NoakLiu/PiKV}. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.