Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving

Key-Value cache ($\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$) compression has emerged as a promising technique to optimize Large Language Model (LLM) serving. It primarily decreases the memory consumption of $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ to reduce the computation cost. Despite the development of many compression algorithms, their applications in production environments are still not prevalent. In this paper, we revisit mainstream $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ compression solutions from a practical perspective. Our contributions are three-fold. First, we comprehensively review existing algorithmic designs and benchmark studies for $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ compression and identify missing pieces in their performance measurement, which could hinder their adoption in practice. Second, we empirically evaluate representative $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ compression methods to uncover two key issues that affect the computational efficiency: (1) while compressing $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ can reduce memory consumption, current implementations (e.g., FlashAttention, PagedAttention) do not optimize for production-level LLM serving, resulting in suboptimal throughput performance; (2) compressing $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ may lead to longer outputs, resulting in increased end-to-end latency. We further investigate the accuracy performance of individual samples rather than the overall performance, revealing the intrinsic limitations in $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ compression when handling specific LLM tasks. Third, we provide tools to shed light on future $\mathtt{\text{KV}}$ $\mathtt{\text{cache}}$ compression studies and facilitate their practical deployment in production. They are open-sourced in \href{https://github.com/LLMkvsys/rethink-kv-compression}{https://github.com/LLMkvsys/rethink-kv-compression}.