Pre-Training, Mid-Training, and RL Interplay

Analysis

Why This Paper Matters

This paper tackles a fundamental ambiguity in modern language model training: does reinforcement learning (RL) post-training genuinely extend reasoning capabilities beyond what pre-training already provides? With large-scale pre-training corpora being opaque and mid-training often overlooked, the field lacks causal clarity. By introducing a fully controlled experimental framework using synthetic reasoning tasks, the authors isolate the contributions of each training stage. This matters because it directly informs how practitioners allocate compute and design training pipelines for reasoning tasks.

Technical Contributions

The paper's key innovations include:

• A synthetic reasoning task suite with explicit atomic operations and parseable step-by-step traces, enabling precise measurement of extrapolative and contextual generalization.
• Systematic manipulation of training distributions to isolate causal effects of pre-training, mid-training, and RL.
• Introduction of process-level rewards that reduce reward hacking and improve reasoning fidelity.
• Clear identification of conditions under which RL yields true capability gains (pass@128): pre-training must leave headroom, and RL data must target the model's edge of competence.

Results

Key quantitative findings:

• RL produces true capability gains (pass@128) only when pre-training leaves sufficient headroom and RL data target the model's edge of competence.
• Contextual generalization requires minimal yet sufficient pre-training exposure, after which RL can reliably transfer.
• Mid-training significantly enhances performance under fixed compute compared with RL only, demonstrating its central but underexplored role.
• Process-level rewards reduce reward hacking and improve reasoning fidelity.

Significance

This work provides a rigorous causal framework for understanding the interplay between pre-training, mid-training, and RL in reasoning LMs. It challenges the assumption that RL alone drives capability gains and highlights the critical but often neglected role of mid-training. For AI practitioners, these insights offer actionable guidance on how to allocate training resources and design data strategies to maximize reasoning performance. The findings also open avenues for more principled training pipeline design, potentially reducing wasted compute and improving model reliability.