Training Language Models to Self-Correct via Reinforcement Learning

Free

Training LLMs to self-correct via multi-turn reinforcement learning

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Training Language Models to Self-Correct via Reinforcement Learning

SCoRe is a multi-turn online reinforcement learning (RL) approach designed to significantly improve the self-correction ability of large language models (LLMs) using entirely self-generated data. Unlike prior methods that rely on multiple models, a more advanced model, or additional supervision, SCoRe addresses the shortcomings of supervised fine-tuning (SFT) which often leads to distribution mismatch or behavior collapse. The method involves an initial phase of multi-turn RL on a base model to generate a policy initialization less susceptible to collapse, followed by using a reward bonus to amplify self-correction. Evaluated on Gemini 1.0 Pro and 1.5 Flash models, SCoRe achieves state-of-the-art self-correction performance, improving base models' self-correction by 15.6% on MATH and 9.1% on HumanEval.

Key Features

Multi-turn online reinforcement learning for self-correction

Uses entirely self-generated data, no external supervision

Regularization to prevent distribution mismatch and behavior collapse

Initial multi-turn RL phase for robust policy initialization

Reward bonus to amplify self-correction behavior

State-of-the-art results on MATH and HumanEval benchmarks

Pros & Cons

Pros

Eliminates need for multiple models or external supervision
Achieves significant performance gains (15.6% on MATH, 9.1% on HumanEval)
Addresses failure modes of supervised fine-tuning (distribution mismatch, behavior collapse)
Uses entirely self-generated correction traces, reducing data dependency

Cons

Requires careful tuning of reinforcement learning components
Evaluated only on specific benchmarks (MATH, HumanEval); generalization to other tasks unknown

Best For

Improving self-correction capabilities of large language modelsEnhancing mathematical reasoning (MATH benchmark)Enhancing code generation (HumanEval benchmark)

FAQ

What is SCoRe?

SCoRe (Self-Correction via Reinforcement Learning) is a multi-turn online RL approach that trains LLMs to self-correct using only self-generated data, without needing multiple models or external supervision.

How does SCoRe differ from supervised fine-tuning (SFT) for self-correction?

SFT often suffers from distribution mismatch between data-collection policy and model's own responses, or behavior collapse. SCoRe addresses these by training under the model's own distribution and using regularization to steer learning toward effective self-correction.

What benchmarks were used to evaluate SCoRe?

SCoRe was evaluated on MATH (mathematical reasoning) and HumanEval (code generation) using Gemini 1.0 Pro and 1.5 Flash models, achieving improvements of 15.6% and 9.1% respectively.

Does SCoRe require a more advanced model or external supervision?

No, SCoRe uses only the model's own self-generated data and does not require a more advanced model or additional supervision.