Demystifying Long Chain-of-Thought Reasoning in LLMs

Free

Uncovering the mechanics of long CoT reasoning in LLMs

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Demystifying Long Chain-of-Thought Reasoning in LLMs

This paper systematically investigates the mechanics of long chain-of-thought (CoT) reasoning in large language models (LLMs). Through extensive supervised fine-tuning (SFT) and reinforcement learning (RL) experiments, the authors identify four key findings: (1) SFT simplifies training but is not strictly necessary; (2) reasoning capabilities emerge with increased training compute but require reward shaping to stabilize CoT length growth; (3) scaling verifiable reward signals is critical for RL, with noisy web-extracted solutions showing strong potential for out-of-distribution STEM tasks; and (4) error correction abilities are inherently present in base models, but incentivizing them via RL demands significant compute and nuanced measurement. The paper offers practical guidance for optimizing training strategies to enhance long CoT reasoning in LLMs. Code is available on GitHub.

Key Features

SFT simplifies training but is not strictly necessary for long CoT emergence

Reward shaping is crucial for stabilizing CoT length growth during RL training

Scaling verifiable reward signals, including noisy web-extracted solutions, enhances RL performance

Error correction abilities are inherently present in base models but require significant compute to incentivize

Pros & Cons

Pros

Provides systematic and actionable insights into long CoT reasoning
Identifies specific conditions (reward shaping, scaling reward signals) for effective RL
Shows potential for using noisy web-extracted data to improve out-of-distribution reasoning
Code is publicly available for reproducibility

Cons

RL training requires substantial compute, limiting accessibility
Findings are based on specific experimental setups and may not generalize to all architectures or tasks
Paper is a preprint under review, not yet peer-reviewed

Best For

Optimizing RL training strategies for long chain-of-thought reasoningUnderstanding emergence of reasoning capabilities in LLMsImproving STEM reasoning through leverage of noisy web-extracted solutionsGuiding design choices for reward shaping and compute allocation in LLM training

FAQ

What is this paper about?

It investigates the mechanics of long chain-of-thought reasoning in LLMs, focusing on how supervised fine-tuning and reinforcement learning training strategies influence the emergence and stabilization of long CoT trajectories.

What are the main findings?

Four main findings: (1) SFT simplifies training but is not necessary; (2) effective reward shaping stabilizes CoT length growth; (3) scaling verifiable reward signals is critical for RL; (4) error correction abilities are inherent but hard to incentivize via RL.

Is the code available?

Yes, the code is available on GitHub, as mentioned in the paper.