Better and Faster LLMs via Multi-token Prediction

Analysis

Why This Paper Matters

This paper challenges the long-standing paradigm of next-token prediction in large language models by demonstrating that predicting multiple future tokens simultaneously yields substantial gains in sample efficiency, downstream performance, and inference speed. The method is remarkably simple—adding independent output heads for n future tokens on a shared trunk—yet delivers consistent improvements across model sizes and domains. For AI practitioners, this means better models without additional training time or architectural complexity, making it a highly practical contribution.

The significance is amplified by the results on coding benchmarks: 12% more problems solved on HumanEval and 17% on MBPP for 13B models. These are not marginal gains but substantial leaps that could translate to real-world productivity improvements in code generation tools. Moreover, the inference speedup of up to 3x with 4-token prediction addresses a critical bottleneck in deploying LLMs at scale.

Technical Contributions

• Multi-token prediction loss: Instead of predicting only the next token, the model predicts n future tokens at each position using n independent output heads on top of a shared trunk. This acts as an auxiliary task without increasing training time.
• Shared trunk architecture: The model trunk is shared across all prediction heads, ensuring that the core representations benefit from the multi-token objective without parameter overhead.
• Induction head development: Experiments on small algorithmic tasks show that multi-token prediction fosters the emergence of induction heads and algorithmic reasoning capabilities, providing a mechanistic explanation for the gains.
• Inference speedup: By predicting multiple tokens in a single forward pass, the model reduces the number of autoregressive steps, achieving up to 3x faster inference even with large batch sizes.

Results

• Coding benchmarks: 13B parameter models trained with multi-token prediction solve 12% more problems on HumanEval and 17% more on MBPP compared to next-token baselines.
• Scalability: Gains increase with model size and persist when training for multiple epochs, indicating robustness.
• Inference speed: 4-token prediction models are up to 3x faster at inference, with no degradation in quality.
• Algorithmic reasoning: On small algorithmic tasks, multi-token prediction leads to better induction head formation and reasoning capabilities.

Significance

This work opens a new direction for training objectives in LLMs, showing that multi-token prediction is a drop-in replacement for next-token prediction that improves both quality and efficiency. For the AI field, it suggests that current training paradigms may be suboptimal and that simple modifications can yield substantial gains. The method is particularly impactful for code generation, where even small percentage improvements translate to significant practical benefits. Future work may explore adaptive n values, integration with other auxiliary losses, or application to multimodal models.