Ultra-Fast Diffusion-based Language Models

Analysis

Why This Paper Matters

This paper introduces Mercury, the first commercial-scale diffusion-based large language models (LLMs) for coding, challenging the dominance of autoregressive generation. By leveraging diffusion to predict multiple tokens in parallel, Mercury achieves unprecedented inference speeds—up to 10x faster than speed-optimized frontier models—while maintaining competitive quality. This breakthrough addresses a critical bottleneck in deploying LLMs: latency and throughput for real-time applications like code completion and interactive coding assistants.

The significance is amplified by independent evaluations from Artificial Analysis and real-world validation on Copilot Arena, where Mercury ranks second in quality and is the fastest model overall. The release of a public API and playground lowers the barrier for practitioners to experiment with diffusion-based LLMs, potentially accelerating adoption in production environments.

Technical Contributions

• Diffusion-based LLMs: Mercury is parameterized via the Transformer architecture but trained using a diffusion process that predicts multiple tokens in parallel, departing from traditional autoregressive next-token prediction.
• Speed-quality frontier: The models set a new state-of-the-art on the speed-quality tradeoff, with Mercury Coder Mini achieving 1109 tokens/sec and Mercury Coder Small achieving 737 tokens/sec on NVIDIA H100 GPUs.
• Coding-specific design: The models are tailored for coding applications, with evaluations spanning multiple languages and use-cases, and real-world validation on Copilot Arena.
• Public release: A public API and free playground are provided, enabling community experimentation and integration.

Results

• Mercury Coder Mini: 1109 tokens/sec throughput on NVIDIA H100 GPUs.
• Mercury Coder Small: 737 tokens/sec throughput on NVIDIA H100 GPUs.
• Outperforms speed-optimized frontier models by up to 10x on average while maintaining comparable quality.
• Ranks second in quality and fastest overall on Copilot Arena based on developer feedback.
• Evaluated on a variety of code benchmarks spanning multiple languages and use-cases.

Significance

Mercury demonstrates that diffusion-based generation can be a practical alternative to autoregressive models for LLMs, especially in latency-sensitive applications like coding assistants. This could spur further research into diffusion for other modalities and tasks, potentially reshaping the inference landscape. The public release of the API and playground also provides a valuable resource for the AI community to explore and build upon this approach.