Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding

Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce $\mathit{\text{GammaTune}}$ and $\mathit{\text{GammaTune+}}$, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15% ($\pm$5%) with $\mathit{\text{GammaTune}}$ and 16% ($\pm$3%) with $\mathit{\text{GammaTune+}}$, while reducing performance variance. This makes $\mathit{\text{GammaTune}}$ a robust and efficient solution for real-world deployment.