Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
FreeSTILL-2: A reproducible framework for training slow-thinking reasoning models
FreeFree tier
Inputs: textOutputs: text
About Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems
This paper presents a reproduction report on implementing o1-like slow-thinking reasoning systems. It introduces the STILL-2 framework, which consists of three stages: (1) imitative learning using distilled long-form thought data to fine-tune the reasoning model, (2) exploration by generating multiple rollouts for challenging problems, and (3) self-improvement through iterative refinement of the training dataset. Experiments on three challenging benchmarks show competitive performance compared to industry-level reasoning systems like o1.
Key Features
Imitative learning from distilled long-form thought data
Exploration via multiple rollouts on challenging problems
Iterative self-improvement through dataset refinement
Competitive performance on complex reasoning benchmarks
Open technical report with full experimental details
Pros & Cons
Pros
- Provides a clear, three-stage framework for training slow-thinking reasoning models
- Achieves competitive performance compared to industry-level systems like o1
- Openly published as a technical report with all details
- Builds on prior research to advance understanding of reasoning systems
Cons
- Core techniques of industry reasoning systems (e.g., o1) are not publicly disclosed, limiting direct comparison
- Only evaluated on three benchmarks, may not generalize to all reasoning tasks
- Framework is a reproduction report rather than a novel breakthrough
Best For
Training reasoning models for complex problem-solvingReproducing o1-like slow-thinking systemsAcademic research on reasoning in LLMsImproving model performance on math and logic benchmarks
FAQ
What is the STILL-2 framework?
STILL-2 is a three-stage approach for training slow-thinking reasoning models: imitate (fine-tune on distilled reasoning traces), explore (generate multiple solutions for hard problems), and self-improve (iteratively refine training data).
How does this paper relate to OpenAI's o1 model?
The paper is a reproduction report aiming to implement o1-like reasoning systems, as the core techniques of o1 are not publicly disclosed. It presents an open methodology to achieve competitive performance.
What kind of benchmarks were used?
The paper reports experiments on three challenging reasoning benchmarks, demonstrating competitive performance compared to industry-level systems.