Variation-aware Vision Token Dropping for Faster Large Vision-Language Models

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding tasks. However, the increasing demand for high-resolution image and long-video understanding results in substantial token counts, consequently leading to reduced inference efficiency. Token compression offers a direct solution by reducing the number of tokens to be processed, thereby improving computational efficiency without architectural changes. Through extensive analysis, we identify two critical limitations in existing inner-LLM token compression methods: positional bias and incompatibility with efficient operators, which critically hinder their practical deployment for LVLM acceleration. This paper presents the first approach from a dynamic token variation perspective, revealing that visual token variations within LLMs exhibit task-agnostic properties. We propose Variation-aware Vision Token Dropping ($\mathit{\text{i.e.}}$, $\mathbf{\text{V}}{}^2\mathbf{\text{Drop}}$), which progressively removes visual tokens with minimal variation during LVLM inference, thereby enhancing computational efficiency. Extensive experiments across multiple models and benchmarks consistently demonstrate that V${}^2$Drop maintains $\mathbf{\text{94.0\%}}$ and $\mathbf{\text{98.6\%}}$ of the original performance for image and video understanding tasks respectively, while reducing LLM generation latency by $\mathbf{\text{31.5\%}}$ and $\mathbf{\text{74.2\%}}$.