The Art of Scaling RL Compute for LLMs

Analysis

Why This Paper Matters

Reinforcement learning (RL) has become a cornerstone of LLM training, yet the field has lacked predictive scaling methodologies comparable to those for pre-training. As compute budgets for RL soar, researchers and practitioners are often left guessing how algorithmic changes will affect performance at scale. This paper addresses this critical gap by presenting the first large-scale systematic study of RL scaling, using over 400,000 GPU-hours of compute. The authors establish a principled framework for analyzing and predicting RL scaling, which is essential for making informed decisions about compute allocation and algorithm selection.

The significance of this work extends beyond mere empirical observation. By fitting sigmoidal compute-performance curves, the authors provide a quantitative tool that can extrapolate from small-scale runs to much larger ones. This capability is transformative: it allows researchers to predict the performance of a 100,000 GPU-hour run based on smaller experiments, saving time and resources. The paper also offers practical guidance by identifying which design choices affect asymptotic performance versus compute efficiency, enabling more efficient experimentation.

Technical Contributions

The paper makes several key technical contributions:

• Sigmoidal scaling curves: The authors fit sigmoidal functions to RL training runs, capturing the saturation behavior of performance as compute increases. This provides a more accurate model than simple power laws, which are common in pre-training scaling studies.
• Systematic ablation of design choices: They ablate a wide range of common RL design choices, including loss aggregation, normalization, curriculum, and off-policy algorithms. This analysis reveals which factors influence the asymptote (final performance) versus the compute efficiency (how quickly performance improves).
• ScaleRL recipe: Based on their findings, they propose a best-practice recipe, ScaleRL, which combines the design choices that lead to stable and predictable scaling. They demonstrate its effectiveness by scaling a single RL run to 100,000 GPU-hours and successfully predicting its validation performance.
• Predictability framework: The paper establishes that stable, scalable recipes follow predictable scaling trajectories, enabling extrapolation from smaller runs. This is a crucial step toward making RL training as predictable as pre-training.

Results

The study's results are both quantitative and qualitative. The authors observe that not all RL recipes yield similar asymptotic performance, indicating that some design choices have a lasting impact on final performance. In contrast, details such as loss aggregation, normalization, curriculum, and off-policy algorithms primarily modulate compute efficiency without materially shifting the asymptote. This distinction is crucial for practitioners: it means that some choices can be tuned to speed up training without sacrificing final performance, while others must be carefully selected to achieve the best possible outcome.

The most striking result is the successful scaling of a single RL run to 100,000 GPU-hours using the ScaleRL recipe. The validation performance predicted from smaller runs matched the actual performance, demonstrating the predictive power of their framework. This is a significant achievement, as it shows that RL scaling can be made predictable, much like pre-training scaling.

Significance

The broader impact of this work is substantial. For AI practitioners, it offers a practical recipe and a framework for predicting RL scaling, which can guide resource allocation and algorithm development. For the research community, it opens up new avenues for studying RL scaling laws and understanding the factors that influence them. By bringing RL training closer to the predictability achieved in pre-training, this paper could accelerate progress in RL-based LLM training, making it more efficient and reliable. The framework may also extend to other domains where RL is used, such as robotics or game playing, further amplifying its impact.