A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

Free

A minimalist RL approach for LLM reasoning: from rejection sampling to Reinforce

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce

This research paper presents a systematic analysis of reinforcement learning (RL) methods for fine-tuning large language models (LLMs) on complex reasoning tasks. The authors revisit Group Relative Policy Optimization (GRPO) from a reinforce-like perspective and find that a simple rejection sampling baseline (RAFT) – training only on positively rewarded samples – yields competitive performance compared to GRPO and PPO. Ablation studies reveal GRPO's main advantage comes from discarding prompts with entirely incorrect responses, not reward normalization. Building on this insight, the paper proposes Reinforce-Rej, a minimal extension of policy gradient that filters both entirely incorrect and entirely correct samples, improving KL efficiency and stability. The work advocates RAFT as a robust, interpretable baseline and cautions against indiscriminate use of negative samples, providing guidance for future reward-based LLM post-training.

Key Features

Revisit GRPO from a reinforce-like algorithm perspective

Find that simple rejection sampling (RAFT) achieves competitive performance to GRPO/PPO

Ablation studies show GRPO's advantage comes from discarding incorrect prompts

Propose Reinforce-Rej, a minimal policy gradient extension filtering correct/incorrect samples

Improves KL efficiency and stability over more complex RL algorithms

Advocate RAFT as a robust and interpretable baseline for LLM post-training

Pros & Cons

Pros

Simple rejection sampling baseline (RAFT) is competitive with complex RL methods
Proposed Reinforce-Rej improves KL efficiency and stability
Provides clear insights into what makes GRPO effective (discarding incorrect prompts)
Lightweight and effective alternative to GRPO/PPO
Interpretable baseline for future research

Cons

Research paper, not a ready-to-use software tool
Effectiveness may be limited to reward-based reasoning tasks
Requires access to a reward model for reinforcement learning
No guarantee of performance outside specific experimental settings

Best For

Fine-tuning large language models for complex reasoning tasksReward-based post-training of LLMsReinforcement learning research for LLM optimizationBaseline comparison for new RL algorithms in LLM reasoning

FAQ

What is the main contribution of this paper?

The paper finds that a simple rejection sampling baseline (RAFT) is competitive with more complex RL methods like GRPO and PPO for LLM reasoning, and proposes Reinforce-Rej, a minimal policy gradient extension that improves KL efficiency and stability by filtering both entirely correct and incorrect samples.

What is RAFT?

RAFT (Reinforcement learning from AI Feedback with Training) is a rejection sampling baseline that trains only on positively rewarded samples. The paper finds it yields competitive performance to GRPO and PPO.

What is Reinforce-Rej?

Reinforce-Rej is a minimal extension of policy gradient that filters out both entirely incorrect and entirely correct samples during training, leading to better KL efficiency and stability.

Is the paper's approach better than GRPO?

The paper shows that GRPO's main advantage stems from discarding prompts with entirely incorrect responses, not from reward normalization. Reinforce-Rej is proposed as a lightweight yet effective alternative that improves KL efficiency and stability.

Where can I access the paper?

The paper is available on arXiv at https://arxiv.org/abs/2504.11343 with the title 'A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce'.