Online speculative decoding

Analysis

Why This Paper Matters

Large language models (LLMs) are increasingly deployed in latency-sensitive applications like chatbots and code assistants. Speculative decoding is a promising technique to speed up inference by using a smaller draft model to generate candidate tokens that the target LLM verifies in parallel. However, traditional speculative decoding relies on a fixed draft model, which may not align well with the target model's distribution, limiting acceptance rates. This paper introduces online speculative decoding, which adapts the draft model on the fly based on the target LLM's output, significantly improving efficiency.

The work is significant because it addresses a key bottleneck in LLM serving: the mismatch between draft and target models. By dynamically adjusting the draft model, the method achieves higher acceptance rates and lower latency without requiring additional training data or model retraining. This makes speculative decoding more practical for real-world deployment where data distributions shift over time.

Technical Contributions

• Online adaptation mechanism: The draft model is updated during inference using the target LLM's output distribution, allowing it to better predict future tokens.
• Customization strategies: The paper explores different ways to customize draft models, including fine-tuning on recent outputs and using lightweight adaptation layers.
• Theoretical analysis: Provides insights into when online adaptation is most beneficial, showing that it reduces the divergence between draft and target distributions.
• Implementation details: Describes a practical algorithm that balances adaptation overhead with latency gains.

Results

The paper reports that online speculative decoding reduces serving latency by up to 30% compared to static draft models on standard benchmarks. The acceptance rate improves by 15-20%, leading to higher throughput. The method is evaluated on multiple LLM architectures (e.g., LLaMA, GPT) and shows consistent gains across different batch sizes and sequence lengths. The overhead of online adaptation is minimal (less than 5% of total inference time), making it practical for real-time use.

Significance

This work advances the state of the art in LLM inference optimization by introducing a dynamic approach to speculative decoding. It opens up new research directions in adaptive model serving, where models can self-improve during deployment. For practitioners, it offers a drop-in improvement to existing speculative decoding systems, potentially reducing cloud costs and improving user experience. The concept of online adaptation could also be applied to other areas of LLM serving, such as caching and prompt optimization.