Reinforcement Learning for Reasoning in Large Language Models with One Training Example

Free

One training example is enough to supercharge LLM math reasoning via RL.

FreeFree tier

Inputs: textOutputs: text

Type

Open Source

Visit Website GitHub (40 stars)

About Reinforcement Learning for Reasoning in Large Language Models with One Training Example

Reinforcement Learning for Reasoning in Large Language Models with One Training Example (1-shot RLVR) is a research paper and open-source project demonstrating that reinforcement learning with verifiable reward, using just a single training example, can significantly improve the math reasoning capabilities of large language models. The method boosts performance on MATH500 from 36.0% to 73.6% and improves average performance across six mathematical reasoning benchmarks from 17.6% to 35.7%. It is evaluated on multiple models (Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B) and RL algorithms (GRPO and PPO). The project identifies phenomena such as cross-category generalization, increased frequency of self-reflection, and post-saturation generalization. All resources are open source.

Key Features

Uses only one training example (1-shot RLVR) to improve LLM reasoning

Achieves 73.6% on MATH500 with Qwen2.5-Math-1.5B, up from 36.0%

Improves average performance across six math benchmarks from 17.6% to 35.7%

Works with multiple models and RL algorithms (GRPO, PPO)

Exhibits cross-category generalization and increased self-reflection

Sustained test performance improvement even after training accuracy saturates (post-saturation generalization)

Pros & Cons

Pros

Dramatic performance gains with as few as one or two training examples
Matches or exceeds results from much larger datasets (e.g., 1.2k DeepScaleR subset)
Generalization across different math categories and models
Encourages self-reflection and exploration behaviors in the model
Open source, allowing easy replication and extension

Cons

Requires verifiable reward signals, limiting applicability to tasks with clear ground truth
Performance may depend on the choice of the single training example (label robustness)
Effectiveness relies on appropriate hyperparameter tuning (e.g., entropy loss coefficient)
Evaluation currently focused on math reasoning – generalization to other domains not confirmed

Best For

Improving mathematical reasoning in large language models with minimal training dataInvestigating the efficiency of reinforcement learning with verifiable reward (RLVR)Research on grokking, label robustness, and exploration in RL for LLMsEnhancing reasoning capabilities on benchmarks like MATH500 and other math datasets

FAQ

What is 1-shot RLVR?

1-shot RLVR stands for reinforcement learning with verifiable reward using one training example. It is a method to enhance the reasoning abilities of large language models by training on just a single example.

How much improvement does 1-shot RLVR achieve on MATH500?

Using Qwen2.5-Math-1.5B, 1-shot RLVR improves MATH500 accuracy from 36.0% to 73.6%, an 8.6% gain beyond format correction.

Which models and RL algorithms are supported?

The method has been tested on Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, Llama3.2-3B-Instruct, and DeepSeek-R1-Distill-Qwen-1.5B, using both GRPO and PPO algorithms.

Is the project open source?

Yes, all resources are open source and available at the URL provided in the paper.