RLHF Workflow

Analysis

Why This Paper Matters

Reinforcement Learning from Human Feedback (RLHF) is the backbone of aligning large language models (LLMs) with human preferences, powering systems like ChatGPT and Claude. However, most existing implementations are either offline (using a fixed preference dataset) or rely on complex deep RL methods like PPO that are hard to tune and memory-intensive. This paper bridges the gap by providing a complete, open-source recipe for online iterative RLHF—a process where the model continuously collects new preference data during training, adapts its reward model, and improves its policy. The result is a state-of-the-art 8B model that outperforms many larger models, proving that iterative alignment can be both practical and powerful.

How It Works

The workflow is divided into two main phases: reward modeling and iterative policy optimization.

Reward Modeling as Human Feedback Approximation

The authors start by assembling a large, diverse set of open-source preference datasets (HH-RLHF, SHP, HelpSteer, PKU-SafeRLHF, UltraFeedback, UltraInteract, Distilabel-Capybara, Distilabel-Orca3). They apply a filtering step that removes low-quality samples, empty conversations, and pairwise comparisons with small score margins (roughly 10% of data is discarded).

Two types of models are trained on this data:

• Bradley-Terry (BT) reward model: A scalar score predictor initialized from the SFT model.
• Preference model: A pairwise classifier that predicts which response is preferred, leveraging the LLM's next-token prediction capability.

Both are trained for one epoch. The authors experiment with two dataset mixes: Mix1 (HH-RLHF + SHP + UltraFeedback + Summarization) and Mix2 (all datasets).

Iterative Policy Optimization

The policy (LLaMA-3-8B) is first SFT on a mixture of instruction datasets. Then, iterative direct preference learning is applied:

1. Data Collection: Historical interactions (prompts, responses, feedback) are gathered.
2. Policy Training: A maximum-likelihood estimation (MLE) policy is trained on the data.
3. Exploration with Enhancer Agent: An enhancer agent—comprising multiple diverse policies derived from the MLE policy—explores response space regions the MLE policy might miss.
4. Online Interaction: Both the MLE policy and enhancer agent interact with users. The MLE policy is the default; the enhancer is used when uncertainty is high.
5. Feedback Incorporation: User preferences are collected and used to update both policies.
6. Iteration: Steps 1–5 repeat until a termination condition (e.g., performance plateau) is met.

Prompts for online collection are drawn from UltraFeedback, HelpSteer, OpenOrca, UltraInteract, Capybara, and DIBT-10K.

Results

The final model, SFR-Iterative-DPO-LLaMA-3-8B-R, was evaluated on AlpacaEval-2, MT-Bench, and Chat-Arena-Hard.

Key findings:

• The iterative model significantly outperforms its offline DPO counterpart and other sub-10B open-source models.
• It surpasses larger models like Tulu-2-DPO-70B and GPT-3.5-turbo-1106 on conversation quality.
• Ablation studies show that applying a length penalty during training produces shorter, higher-quality responses and improves win rates on AlpacaEval-2.
• The preference model outperforms the BT model on coding and math tasks, while the BT model excels in safety and reasoning.

Significance

This paper democratizes advanced RLHF by offering a fully open-source, reproducible pipeline. It demonstrates that iterative online data collection—combined with careful dataset filtering and a dual-agent exploration strategy—can yield models that rival proprietary systems. For practitioners, the provided code repositories and HuggingFace datasets lower the barrier to implementing state-of-the-art alignment. The work also underscores the importance of addressing reward model over-optimization through continuous data refresh, a lesson that applies broadly to AI alignment research.