ReFT: Reasoning with Reinforced Fine-Tuning

Free

Enhancing LLM reasoning via reinforced fine-tuning with multiple reasoning paths

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About ReFT: Reasoning with Reinforced Fine-Tuning

ReFT (Reasoning with Reinforced Fine-Tuning) is a method proposed in a 2024 ACL paper to enhance the reasoning capabilities of Large Language Models (LLMs), with a focus on math problem-solving. It first warms up the model with Supervised Fine-Tuning (SFT) using Chain-of-Thought (CoT) annotations, then applies online reinforcement learning (specifically the PPO algorithm) to further fine-tune the model. During this stage, multiple reasoning paths are automatically sampled for each question, and rewards are naturally derived from ground-truth answers. ReFT is evaluated on GSM8K, MathQA, and SVAMP datasets, demonstrating significant improvements over standard SFT without relying on extra or augmented training questions. The approach can also be combined with inference-time strategies like majority voting and re-ranking for additional gains.

Key Features

Warmup with supervised fine-tuning (SFT) using Chain-of-Thought annotations

Online reinforcement learning (PPO) for further fine-tuning

Automatic sampling of multiple reasoning paths for each question

Rewards derived from ground-truth answers

Improves generalization ability without extra training questions

Compatible with inference-time strategies like majority voting and re-ranking

Pros & Cons

Pros

Significantly outperforms standard supervised fine-tuning (SFT) on reasoning benchmarks
Learns from multiple reasoning paths, enhancing generalization
Does not require additional or augmented training data beyond the original questions
Performance can be further boosted by combining with inference-time strategies

Cons

Requires ground-truth answers for reward computation during RL training
Additional computational overhead compared to pure SFT due to reinforcement learning
Evaluation shown primarily on math reasoning; generalization to other domains not extensively tested

Best For

Math problem-solving (e.g., GSM8K, MathQA, SVAMP)General reasoning tasks for large language models

FAQ

What is ReFT?

ReFT (Reinforced Fine-Tuning) is a method that enhances LLM reasoning by first performing supervised fine-tuning with chain-of-thought annotations, then applying reinforcement learning (PPO) to fine-tune the model using multiple automatically sampled reasoning paths.

How does ReFT differ from standard supervised fine-tuning?

SFT trains on a single annotated reasoning path per question, while ReFT uses online RL to learn from many paths, improving generalization without extra training questions.

On which datasets was ReFT evaluated?

ReFT was evaluated on GSM8K, MathQA, and SVAMP datasets for math problem-solving.

Can ReFT be combined with other inference strategies?

Yes, the paper notes that performance can be further improved by combining ReFT with inference-time strategies such as majority voting and re-ranking.