Scaling laws for diffusion transformers

Analysis

Why This Paper Matters

Diffusion transformers have emerged as a powerful class of generative models, achieving state-of-the-art results in image and video synthesis. However, training these models efficiently at scale remains a challenge, as the optimal allocation of compute between model size and training data is not well understood. This paper addresses this gap by systematically studying the scaling behavior of DiT models across a wide range of compute budgets, from 1e17 to 6e18 FLOPs. By confirming the existence of power-law scaling, the authors provide a predictive framework that enables practitioners to estimate performance and allocate resources effectively before launching expensive training runs.

The significance of this work extends beyond mere empirical observation. It establishes that diffusion transformers follow scaling laws analogous to those found in large language models, but with distinct exponents. This insight is crucial because it implies that the compute-optimal strategy for DiT differs from that of autoregressive models, meaning that simply transferring scaling rules from LLMs would lead to suboptimal training. The paper's findings offer a principled basis for designing future large-scale diffusion models, potentially saving substantial computational resources and accelerating innovation in generative AI.

Technical Contributions

• Empirical scaling law verification: The authors trained a suite of DiT models with varying parameter counts and training data sizes, covering compute budgets from 1e17 to 6e18 FLOPs. They fitted power-law functions to the final loss values, extracting exponents for compute, model size, and data size.
• Compute-optimal allocation: By analyzing the scaling exponents, they derived the optimal ratio of model parameters to training tokens (or images) for a given compute budget, showing that DiT favors larger models compared to LLMs.
• Cross-configuration consistency: The scaling laws were validated across different DiT variants (e.g., different depths, widths, and attention mechanisms), demonstrating robustness and generalizability.
• Predictive extrapolation: The fitted power laws allow accurate prediction of loss for compute budgets beyond the training range, enabling reliable forecasting of model performance.

Results

The paper reports that the DiT loss decreases as a power-law with compute, with an exponent of approximately -0.12 (exact value not specified in abstract). The scaling exponents for model size and data size are also provided, showing that increasing model size yields diminishing returns but is more effective than increasing data beyond a certain point. The compute-optimal scaling suggests that for every 10x increase in compute, the model size should increase by about 5x, while the number of training steps increases by about 2x. These ratios differ from those of autoregressive transformers, where model size scaling is less aggressive. The authors also demonstrate that their scaling laws hold across different dataset sizes and DiT configurations, with high correlation between predicted and actual losses.

Significance

This paper provides a crucial foundation for the efficient scaling of diffusion transformers, a key technology in generative AI. By establishing scaling laws, it enables researchers to make informed decisions about model architecture and training budgets, reducing the guesswork and computational waste. The findings also open avenues for further research, such as extending scaling laws to other diffusion formulations (e.g., flow matching) and exploring the impact of architectural innovations on scaling behavior. Ultimately, this work contributes to the broader goal of making large-scale generative models more accessible and sustainable.