OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

Free

Mid-training strategies that incentivize reinforcement learning scaling for base models.

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling

OctoThinker is a research paper and family of models that investigates how mid-training strategies can improve the reinforcement learning (RL) scalability of base language models, specifically focusing on the Llama and Qwen families. The study reveals that high-quality mathematical corpora (e.g., MegaMath-Web-Pro) and long chain-of-thought (CoT) reasoning data significantly enhance both base model and RL performance, while long CoT can also induce verbosity and training instability. The authors propose a two-stage 'Stable-then-Decay' mid-training strategy, yielding OctoThinker models that demonstrate strong RL compatibility. They release open-source models along with a curated math reasoning corpus of over 70 billion tokens (MegaMath-Web-Pro-Max).

Key Features

Stable-then-Decay mid-training strategy (200B tokens constant LR, then 20B tokens with decay across CoT branches)

Uses high-quality mathematical corpora (MegaMath-Web-Pro) that improve both base model and RL performance

Incorporates QA-style data with long chain-of-thought reasoning examples to enhance RL outcomes

Scaling mid-training consistently strengthens downstream RL performance

Open-source models and a curated 70B+ token math reasoning corpus (MegaMath-Web-Pro-Max)

Pros & Cons

Pros

Demonstrates improved RL compatibility for base models through systematic mid-training
Provides open-source models and a large, high-quality math corpus for further research
Reveals key insights about data quality and long CoT effects on RL training
Proposes a novel two-stage training strategy that yields consistent scaling benefits

Cons

Long chain-of-thought data can induce verbosity in model responses and instability during RL training
Findings are primarily based on math reasoning tasks; generalizability to other domains is not established
Requires significant computational resources for mid-training with hundreds of billions of tokens
Only tested on Llama and Qwen model families; may not transfer to other architectures

Best For

Research on reinforcement learning scaling for language modelsImproving base model compatibility with RL fine-tuning for reasoning-intensive tasksDeveloping mid-training strategies to enhance downstream RL performanceStudying the impact of data quality and formatting on RL training stability and verbosity

FAQ

What is OctoThinker?

OctoThinker is a family of language models and a research study that explores how mid-training strategies can incentivize reinforcement learning scalability, particularly for reasoning-intensive tasks. It was introduced in a paper on arXiv (2506.20512) by Zengzhi Wang, Fan Zhou, Xuefeng Li, and Pengfei Liu.

What is the Stable-then-Decay training strategy?

Stable-then-Decay is a two-stage mid-training approach where base models are first trained on 200B tokens with a constant learning rate, followed by 20B tokens across three chain-of-thought-focused branches with learning rate decay.

What data was used for OctoThinker training?

The models were trained on high-quality mathematical corpora such as MegaMath-Web-Pro, along with QA-style data and long chain-of-thought reasoning examples. A larger curated corpus called MegaMath-Web-Pro-Max (over 70B tokens) is also released.

Are the OctoThinker models open-source?

Yes, the OctoThinker models are released as open-source, along with the MegaMath-Web-Pro-Max corpus to support further research.