AceReason-Nemotron

Analysis

Why This Paper Matters

This paper challenges the prevailing assumption that only massive models or distillation from them can achieve top-tier reasoning performance. By applying large-scale reinforcement learning (RL) to small and mid-sized models (7B and 14B parameters), the authors show that RL can elicit and even extend foundational reasoning abilities beyond what supervised fine-tuning or distillation can achieve. This is particularly significant for practitioners who need efficient, deployable models without sacrificing accuracy on complex math and code tasks.

How It Works

The framework adopts the GRPO algorithm and separates training into math-only and code-only stages due to the large difference in verification time between the two domains.

Math-only RL uses a carefully curated dataset of about 49,000 high-quality math problems, filtered via 9-gram decontamination and rule-based verification. The training employs strict on-policy updates, stage-wise length extension from 8K to 32K tokens, and curriculum training with increasing difficulty. Rewards are binary (1 for correct, 0 for incorrect) based on symbolic equivalence using sympy.

Code-only RL uses 8,520 competitive programming problems, filtered to avoid multi-solution or interactive problems. Difficulty is scored using DeepSeek-R1-671B with 8 rollouts. The two-stage pipeline starts with easier problems (up to level 5 for 7B, level 7 for 14B) and then uses the full set with epoch-wise filtering, increasing temperature from 0.6 to 1.0 and rollouts from 8 to 16. Rewards are binary based on passing all test cases in a sandbox.

Results

Math-only RL significantly boosts code reasoning: LiveCodeBench v5 scores increase by 6.8% for the 7B model and 5.8% for the 14B model. The 14B model surpasses DeepCoder-14B (57.9% vs 58.9%). Notably, the improvement spans all coding topics, not just math-related ones.

Overall, AceReason-Nemotron models show substantial accuracy gains over SFT baselines. The 14B model achieves best-in-class math reasoning among open RL-based models of similar size and outperforms specialized distilled models, indicating RL can push performance beyond distillation limits.

Significance

This work provides strong evidence that RL is a powerful alternative to distillation for improving reasoning in smaller models. The separation of math and code RL stages is a practical insight that avoids verification bottlenecks. The results suggest that with careful data curation and training strategies, RL can unlock reasoning capabilities that distillation cannot, especially at the 14B scale and beyond. For AI practitioners, this opens the door to building high-performing reasoning models without relying on massive teacher models or proprietary distillation pipelines.