Neura Market
Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving logo

Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

Free

Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

FreeFree tier
Inputs: textOutputs: text, code
Type
Open Source
Visit Website GitHub (40 stars)

About Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving

ZeroTIR (Zero-shot Tool-Integrated Reasoning) is a reinforcement learning framework that trains base large language models to autonomously generate and execute Python code for mathematical problem solving, without requiring any supervised tool-use examples. The core finding is that as RL training progresses, key metrics scale predictably: spontaneous code execution frequency, average response length, and final task accuracy all increase with training steps, revealing a quantifiable relationship between computational effort and emergent tool-augmented reasoning. The framework uses a decoupled code execution environment for safety and reproducibility. Experiments on challenging math benchmarks show ZeroTIR significantly outperforms non-tool ZeroRL baselines. The project is open source, with code released for reproducibility.

Key Features

Reinforcement learning from outcome-based rewards without supervised tool-use examples
Spontaneous generation and execution of Python code for mathematical computation
Demonstrated scaling law: training steps correlate with code frequency, response length, and accuracy
Decoupled code execution environment for safe and reproducible experiments
Surpasses non-tool ZeroRL baselines on challenging math benchmarks
Open source code release for reproducibility and further research

Pros & Cons

Pros
  • Significantly outperforms non-tool RL baselines on math benchmarks
  • Demonstrates quantifiable scaling laws for emergent tool use
  • No supervised tool-use examples required, enabling zero-shot generalization
  • Reproducible open source implementation
  • Free to use and modify
Cons
  • Requires substantial RL training compute resources
  • Currently evaluated primarily on mathematical reasoning tasks
  • Still a research method, not a production-ready tool
  • May struggle with tasks outside verifiable computation

Best For

Mathematical problem solving requiring precise computationResearch into emergent tool use in reinforcement learningScaling behavior studies in agent RLBenchmarking autonomous code generation in LLMs

FAQ

What is ZeroTIR?
ZeroTIR (Zero-shot Tool-Integrated Reasoning) is an RL framework that trains LLMs to spontaneously generate and execute Python code for mathematical problems, without needing supervised examples of tool use.
What are the main findings of the paper?
The paper demonstrates a scaling law: as RL training progresses, the frequency of code execution, response length, and accuracy all increase predictably, showing a clear relationship between training compute and emergent tool use.
Is the code available?
Yes, the code is released as open source and can be accessed via the paper's GitHub link on arXiv.
How does ZeroTIR compare to non-tool RL baselines?
ZeroTIR significantly surpasses non-tool ZeroRL baselines on challenging math benchmarks, showing the benefit of allowing spontaneous code execution.