SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many smaller tail models remain underutilized. We propose $\mathbf{\text{SPECTRE}}$ (Parallel $\mathbf{\text{SPEC}}$ulative Decoding with a Multi-$\mathbf{\text{T}}$enant $\mathbf{\text{RE}}$mote Drafter), a serving framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services through speculative decoding. SPECTRE enables draft generation and target-side verification to run in parallel, and makes such parallelism effective through three techniques: a hybrid ordinary-parallel speculative decoding strategy guided by a threshold derived from throughput analysis, speculative priority scheduling to preserve draft–target overlap under multi-tenant traffic, and draft-side prompt compression to reduce draft latency. We implement SPECTRE in $\mathtt{\text{SGLang}}$ and evaluate it across multiple draft–target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to $\mathbf{\text{2.28}}\times \mathbf{\text{speedup}}$ over autoregressive decoding and up to an additional $\mathbf{\text{66\%}}\mathbf{\text{relative}}\mathbf{\text{improvement}}$ over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.