Fine-Tuning LLMs with TRL: SFT, DPO, GRPO, and RLOO

TRL (Transformer Reinforcement Learning) is a library from Hugging Face that provides post-training methods for aligning language models with human preferences. You would reach for it when you have a base model and want to teach it to follow instructions, prefer one response over another, or optimize its outputs using a reward signal. This guide covers the four main training methods: supervised fine-tuning (SFT), direct preference optimization (DPO), group relative policy optimization (GRPO), and reinforcement learning from human feedback with RLOO.

What it does

TRL turns a pretrained language model into one that behaves the way you want. SFT teaches instruction following from prompt-completion pairs. DPO aligns the model with a dataset of chosen and rejected responses without needing a separate reward model. RLOO and GRPO are online reinforcement learning methods that use a reward function or reward model to guide the policy during training. The library ships with trainers for each method, a CLI for running them without writing Python, and configuration classes that mirror the Hugging Face Trainer API.

Before you start

You need a machine with an NVIDIA GPU and CUDA installed. The library and its dependencies are installed with pip:

pip install trl transformers datasets peft accelerate

VRAM requirements depend on the model size and method. For a 7B parameter model, expect roughly:

• SFT with LoRA: 16 GB
• DPO: 24 GB (stores a reference model)
• RLOO: 40 GB (policy plus reward model)
• GRPO: 24 GB (more memory efficient)

Multi-GPU training is supported via accelerate. Mixed precision with BF16 is recommended on A100 or H100 hardware. Memory can be reduced by using LoRA or QLoRA, enabling gradient checkpointing, and lowering batch sizes with gradient accumulation.

Supervised Fine-Tuning (SFT)

SFT is the first step in most alignment pipelines. You take a base model and train it on prompt-completion pairs so it learns to produce useful responses.

Quick start

from trl import SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset,  # Prompt-completion pairs
)
trainer.train()

Full pipeline step

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer
)
trainer.train()
trainer.save_model()

For deeper guidance on dataset formats, chat templates, packing strategies, and multi-GPU training, see the SFT training reference at references/sft-training.md.

Direct Preference Optimization (DPO)

DPO aligns a model with human preferences without training a separate reward model. You provide a dataset with chosen and rejected responses, and the trainer optimizes the policy directly.

Quick start

from trl import DPOTrainer, DPOConfig

config = DPOConfig(output_dir="model-dpo", beta=0.1)
trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=preference_dataset,  # chosen/rejected pairs
    processing_class=tokenizer
)
trainer.train()

Full workflow

Copy this checklist:

DPO Training:
- [ ] Step 1: Prepare preference dataset
- [ ] Step 2: Configure DPO
- [ ] Step 3: Train with DPOTrainer
- [ ] Step 4: Evaluate alignment

Step 1: Prepare preference dataset

Dataset format:

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "I don't know."
}

Load dataset:

from datasets import load_dataset

dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
# Or load your own
# dataset = load_dataset("json", data_files="preferences.json")

Step 2: Configure DPO

from trl import DPOConfig

config = DPOConfig(
    output_dir="Qwen2.5-0.5B-DPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=5e-7,
    beta=0.1,  # KL penalty strength
    max_prompt_length=512,
    max_length=1024,
    logging_steps=10
)

Step 3: Train with DPOTrainer

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOTrainer

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

trainer = DPOTrainer(
    model=model,
    args=config,
    train_dataset=dataset,
    processing_class=tokenizer
)

trainer.train()
trainer.save_model()

CLI alternative:

trl dpo \
    --model_name_or_path Qwen/Qwen2.5-0.5B-Instruct \
    --dataset_name argilla/Capybara-Preferences \
    --output_dir Qwen2.5-0.5B-DPO \
    --per_device_train_batch_size 4 \
    --learning_rate 5e-7 \
    --beta 0.1

For DPO variants like IPO, cDPO, and RPO, see the reference at references/dpo-variants.md.

Full RLHF Pipeline (SFT to Reward Model to RLOO)

This is the classic three-stage pipeline: supervised fine-tuning, reward model training, and online reinforcement learning. Note that PPO was removed in TRL 1.x. The closest drop-in is RLOO.

Copy this checklist:

RLHF Training:
- [ ] Step 1: Supervised fine-tuning (SFT)
- [ ] Step 2: Train reward model
- [ ] Step 3: RLOO reinforcement learning
- [ ] Step 4: Evaluate aligned model

Step 1: Supervised fine-tuning

Train base model on instruction-following data:

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset

# Load model
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")

# Load instruction dataset
dataset = load_dataset("trl-lib/Capybara", split="train")

# Configure training
training_args = SFTConfig(
    output_dir="Qwen2.5-0.5B-SFT",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=2e-5,
    logging_steps=10,
    save_strategy="epoch"
)

# Train
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer
)
trainer.train()
trainer.save_model()

Step 2: Train reward model

Train model to predict human preferences:

from transformers import AutoModelForSequenceClassification
from trl import RewardTrainer, RewardConfig

# Load SFT model as base
model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-SFT",
    num_labels=1  # Single reward score
)
tokenizer = AutoTokenizer.from_pretrained("Qwen2.5-0.5B-SFT")

# Load preference data (chosen/rejected pairs)
dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

# Configure training
training_args = RewardConfig(
    output_dir="Qwen2.5-0.5B-Reward",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=1e-5
)

# Train reward model
trainer = RewardTrainer(
    model=model,
    args=training_args,
    processing_class=tokenizer,
    train_dataset=dataset
)
trainer.train()
trainer.save_model()

Step 3: RLOO reinforcement learning

Optimize policy using the reward model. PPO was removed in TRL 1.x; use the RLOO CLI (trl rloo) with the trained reward model passed via --reward_model_name_or_path:

trl rloo \
    --model_name_or_path Qwen2.5-0.5B-SFT \
    --reward_model_name_or_path Qwen2.5-0.5B-Reward \
    --dataset_name trl-internal-testing/descriptiveness-sentiment-trl-style \
    --output_dir Qwen2.5-0.5B-RLOO \
    --learning_rate 3e-6 \
    --per_device_train_batch_size 64 \
    --num_generations 4

Equivalent Python (RLOOTrainer / RLOOConfig):

from trl import RLOOTrainer, RLOOConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer

reward_model = AutoModelForSequenceClassification.from_pretrained(
    "Qwen2.5-0.5B-Reward", num_labels=1
)

config = RLOOConfig(
    output_dir="Qwen2.5-0.5B-RLOO",
    per_device_train_batch_size=64,
    learning_rate=3e-6,
    num_generations=4,
)

trainer = RLOOTrainer(
    model="Qwen2.5-0.5B-SFT",
    reward_funcs=reward_model,   # a reward model (or a callable reward function)
    args=config,
    train_dataset=dataset,       # prompt-only dataset
    processing_class=tokenizer,
)
trainer.train()

Step 4: Evaluate

from transformers import pipeline

# Load aligned model
generator = pipeline("text-generation", model="Qwen2.5-0.5B-RLOO")

# Test
prompt = "Explain quantum computing to a 10-year-old"
output = generator(prompt, max_length=200)[0]["generated_text"]
print(output)

For more on reward modeling, see references/reward-modeling.md. For online RL methods including PPO, GRPO, RLOO, and OnlineDPO, see references/online-rl.md.

Memory-Efficient Online RL with GRPO

GRPO is an online RL method that uses less memory than RLOO because it does not require a separate value function. You define a reward function and the trainer generates multiple completions per prompt, then updates the policy based on the group-relative advantage.

Copy this checklist:

GRPO Training:
- [ ] Step 1: Define reward function
- [ ] Step 2: Configure GRPO
- [ ] Step 3: Train with GRPOTrainer

Step 1: Define reward function

def reward_function(completions, **kwargs):
    """
    Compute rewards for completions.

    Args:
        completions: List of generated texts

    Returns:
        List of reward scores (floats)
    """
    rewards = []
    for completion in completions:
        # Example: reward based on length and unique words
        score = len(completion.split())  # Favor longer responses
        score += len(set(completion.lower().split()))  # Reward unique words
        rewards.append(score)
    return rewards

Or use a reward model:

from transformers import pipeline

reward_model = pipeline("text-classification", model="reward-model-path")

def reward_from_model(completions, prompts, **kwargs):
    # Combine prompt + completion
    full_texts = [p + c for p, c in zip(prompts, completions)]
    # Get reward scores
    results = reward_model(full_texts)
    return [r["score"] for r in results]

Step 2: Configure GRPO

from trl import GRPOConfig

config = GRPOConfig(
    output_dir="Qwen2-GRPO",
    per_device_train_batch_size=4,
    num_train_epochs=1,
    learning_rate=1e-5,
    num_generations=4,  # Generate 4 completions per prompt
    max_new_tokens=128
)

Step 3: Train with GRPOTrainer

from datasets import load_dataset
from trl import GRPOTrainer

# Load prompt-only dataset
dataset = load_dataset("trl-lib/tldr", split="train")

trainer = GRPOTrainer(
    model="Qwen/Qwen2-0.5B-Instruct",
    reward_funcs=reward_function,  # Your reward function
    args=config,
    train_dataset=dataset
)

trainer.train()

CLI:

trl grpo \
    --model_name_or_path Qwen/Qwen2-0.5B-Instruct \
    --dataset_name trl-lib/tldr \
    --output_dir Qwen2-GRPO \
    --num_generations 4

For expert-level guidance on GRPO, including reward function design philosophy, training insights (why loss increases, mode collapse detection), hyperparameter tuning, multi-stage training, and troubleshooting, see references/grpo-training.md. A production-ready training script is available in templates/basic_grpo_training.py.

When to use vs alternatives

Use TRL when:

• Need to align model with human preferences
• Have preference data (chosen/rejected pairs)
• Want to use reinforcement learning (RLOO, GRPO)
• Need reward model training
• Doing RLHF (full pipeline)

Method selection:

• SFT: Have prompt-completion pairs, want basic instruction following
• DPO: Have preferences, want simple alignment (no reward model needed)
• RLOO: Have a reward model, want online RL (the reward-model-driven RLHF path; PPO was removed in TRL 1.x)
• GRPO: Memory-constrained, want online RL with reward functions
• Reward Model: Building RLHF pipeline, need to score generations

Use alternatives instead:

• HuggingFace Trainer: Basic fine-tuning without RL
• Axolotl: YAML-based training configuration
• LitGPT: Educational, minimal fine-tuning
• Unsloth: Fast LoRA training

Common issues

Issue: OOM during DPO training

Reduce batch size and sequence length:

config = DPOConfig(
    per_device_train_batch_size=1,  # Reduce from 4
    max_length=512,  # Reduce from 1024
    gradient_accumulation_steps=8  # Maintain effective batch
)

Or use gradient checkpointing:

model.gradient_checkpointing_enable()

Issue: Poor alignment quality

Tune beta parameter:

# Higher beta = more conservative (stays closer to reference)
config = DPOConfig(beta=0.5)  # Default 0.1

# Lower beta = more aggressive alignment
config = DPOConfig(beta=0.01)

Issue: Reward model not learning

Check loss type and learning rate:

config = RewardConfig(
    learning_rate=1e-5,  # Try different LR
    num_train_epochs=3  # Train longer
)

Ensure preference dataset has clear winners:

# Verify dataset
print(dataset[0])
# Should have clear chosen > rejected

Issue: Online RL (RLOO/GRPO) training unstable

Adjust the KL/beta regularization toward the reference policy:

from trl import RLOOConfig

config = RLOOConfig(
    beta=0.05,          # KL coefficient toward the reference model (increase for stability)
    num_generations=4,  # more samples per prompt = lower-variance advantage estimates
)

Limits and gotchas

• PPO was removed in TRL 1.x. PPOTrainer, PPOConfig, and python -m trl.scripts.ppo no longer exist. Use RLOO or GRPO instead.
• DPO stores a reference model, which roughly doubles the memory requirement compared to SFT.
• Online RL methods (RLOO, GRPO) require generating multiple completions per prompt, which increases training time.
• The reward model in the RLHF pipeline must output a single scalar score (num_labels=1).
• GRPO does not use a value function, which saves memory but may require careful reward function design to avoid mode collapse.

Related resources

• TRL documentation: https://huggingface.co/docs/trl/
• TRL GitHub: https://github.com/huggingface/trl
• Examples: https://github.com/huggingface/trl/tree/main/examples/scripts
• Papers: InstructGPT (2022), DPO (2023), GRPO (2024)