NVIDIA has introduced a new family of language models called Nemotron-Labs Diffusion that aim to break the speed bottleneck of traditional autoregressive text generation. The models generate multiple tokens in parallel and then refine them iteratively, a approach that differs from the token-by-token generation used by most large language models today.
Modern large language models power code generation, math problem solving, summarization, and document understanding. Yet most still rely on autoregressive generation, where each new token depends on all previous tokens. That approach has proven stable and effective, but it creates a hard limit: every token requires a full model pass and all weights must be loaded from memory before computation starts. For latency-sensitive applications, small batch sizes, or developers trying to maximize GPU utilization, this token-by-token process leaves performance on the table. Most GPU time is spent on memory operations rather than computation. Additionally, autoregressive models cannot revise previously generated tokens, so mistakes can propagate through the output.
Nemotron-Labs Diffusion offers an alternative through diffusion language models (DLM). These models generate multiple tokens in parallel, then refine them over multiple steps. This process not only leverages modern GPUs more efficiently but also allows the model to revise tokens, making it suitable for editing existing text and fill-in-the-middle objectives. By reducing the number of refinement steps, developers can control the inference budget and lower compute requirements at runtime.
Model Family and Availability
The Nemotron-Labs Diffusion family includes text models at 3B, 8B, and 14B parameters, as well as an 8B vision-language model (VLM). All text models are available under the commercially friendly NVIDIA Nemotron Open Model License. The VLM is released under the NVIDIA Source Code License, which grants broad research flexibility. NVIDIA is releasing both base models and instruction-tuned chat variants, along with the training code through the NVIDIA Megatron Bridge framework.
Three Generation Modes in One Model
Nemotron-Labs Diffusion supports three generation modes within a single model. Autoregressive mode runs like a standard left-to-right LLM, maintaining compatibility with existing workflows. Diffusion mode generates text block by block, gradually refining tokens over multiple steps. Self-speculation mode uses diffusion to draft multiple candidate tokens, then verifies them with autoregressive decoding, combining the speed of diffusion drafting with the reliability of AR verification.
Developers can switch between these modes at deployment time with minimal application-level changes. The flexible design is intended to handle workloads with unpredictable batch sizes or single queries.
Performance Highlights
Stay updated
Get the day's AI and automation news in your inbox. No spam, unsubscribe anytime.
The Nemotron-Labs Diffusion 8B model achieves an average accuracy improvement of 1.2% compared with Qwen3 8B. In terms of inference speed measured by tokens per forward pass (TPF), a hardware agnostic metric, diffusion mode reaches 2.6 times higher TPF than autoregressive models. Self speculation pushes that further to 6 times for linear self speculation and 6.4 times for quadratic self speculation, with comparable accuracy across evaluated tasks.
Training and Deployment
Diffusion language models have shown promise for years but faced barriers including lower accuracy than strong autoregressive models, difficult training, and limited compatibility with KV caching. Recent work from Efficient DLM showed that pretrained autoregressive models can be converted into diffusion language models through continued pretraining and altering the attention mechanism to a block wise approach.
Nemotron-Labs Diffusion builds on that insight by adding diffusion capabilities to an existing autoregressive model. The model was trained with a joint autoregressive and diffusion objective, retaining what it learned during initial autoregressive training while adding parallel drafting ability. Pre training was conducted on 1.3 trillion tokens from the NVIDIA Nemotron Pretraining datasets, followed by supervised fine tuning on 45 billion tokens from the NVIDIA Nemotron Post training datasets.
Deployment will soon be supported in the main branch of SGLang. Currently, inference support is available through a GitHub issue tracker request. The integration allows serving the same checkpoint in three ways selected by a single line in the algorithm configuration. In autoregressive mode, the model behaves like any causal LM. Diffusion mode fills in a 32 token block at a time by iterative denoising with a confidence threshold. Self speculation drafts a block bidirectionally and verifies it causally, achieving approximately 865 tokens per second on B200 hardware on the speedbench dataset, roughly four times the autoregressive baseline on the same hardware.
Getting Started
Nemotron Labs Diffusion brings diffusion style generation into a form developers can practically use with open models, familiar autoregressive compatibility, diffusion decoding, and self speculative acceleration in one family. Developers can explore the model family on Hugging Face, read the technical report, and try the available training recipe.
Related on Neura Market