SimPO Reference Guide: Reference-Free Preference Optimization

SimPO is a reference-free preference optimization method that outperforms DPO without needing a reference model. It was introduced in 2024. This guide covers installation, configuration, training workflows, parameter tuning, and troubleshooting.

Overview

SimPO simplifies preference optimization by eliminating the reference model required by DPO. It supports sigmoid or hinge loss functions, optional SFT regularization, and works with both base and instruct models. It is especially suitable for reasoning-intensive tasks when used with a lower learning rate.

Prerequisites

Before using SimPO, ensure the following are in place:

• Python 3.10
• Conda environment
• PyTorch 2.2.2
• alignment-handbook installed from GitHub
• Flash Attention 2 installed
• Accelerate and DeepSpeed configured (specifically deepspeed_zero3.yaml)
• Preference dataset in chosen/rejected pairs format
• Sufficient GPU VRAM (see hardware requirements)
• Access to HuggingFace model hub or local model files

Installation

Create a conda environment, install PyTorch, clone and install the alignment-handbook, then install Flash Attention 2.

# Create environment
conda create -n simpo python=3.10 && conda activate simpo

# Install PyTorch 2.2.2
# Visit: https://pytorch.org/get-started/locally/

# Install alignment-handbook
git clone https://github.com/huggingface/alignment-handbook.git
cd alignment-handbook
python -m pip install .

# Install Flash Attention 2
python -m pip install flash-attn --no-build-isolation

Training (Mistral 7B)

Set the environment variable and launch training using the provided configuration.

ACCELERATE_LOG_LEVEL=info accelerate launch \
  --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py \
  training_configs/mistral-7b-base-simpo.yaml

Workflows

Workflow 1: Train from base model (Mistral 7B)

Create a configuration file named mistral-7b-base-simpo.yaml with the following content:

# Model
model_name_or_path: mistralai/Mistral-7B-v0.1
torch_dtype: bfloat16

# Dataset
dataset_mixer:
  HuggingFaceH4/ultrafeedback_binarized: 1.0
dataset_splits:
  - train_prefs
  - test_prefs

# SimPO hyperparameters
beta: 2.0                  # Reward scaling (2.0-10.0)
gamma_beta_ratio: 0.5       # Target margin (0-1)
loss_type: sigmoid          # sigmoid or hinge
sft_weight: 0.0             # Optional SFT regularization

# Training
learning_rate: 5e-7         # Critical: 3e-7 to 1e-6
num_train_epochs: 1
per_device_train_batch_size: 1
gradient_accumulation_steps: 8

# Output
output_dir: ./outputs/mistral-7b-simpo

Launch training:

accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml

Workflow 2: Fine-tune instruct model (Llama 3 8B)

Create a configuration file named llama3-8b-instruct-simpo.yaml with the following content:

model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct

dataset_mixer:
  argilla/ultrafeedback-binarized-preferences-cleaned: 1.0

beta: 2.5
gamma_beta_ratio: 0.5
learning_rate: 5e-7
sft_weight: 0.1             # Add SFT loss to preserve capabilities

num_train_epochs: 1
per_device_train_batch_size: 2
gradient_accumulation_steps: 4
output_dir: ./outputs/llama3-8b-simpo

Launch training:

accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py training_configs/llama3-8b-instruct-simpo.yaml

Workflow 3: Reasoning-intensive tasks (lower LR)

Create a configuration file with the following content:

model_name_or_path: deepseek-ai/deepseek-math-7b-base

dataset_mixer:
  argilla/distilabel-math-preference-dpo: 1.0

beta: 5.0                   # Higher for stronger signal
gamma_beta_ratio: 0.7       # Larger margin
learning_rate: 3e-7         # Lower LR for reasoning
sft_weight: 0.0

num_train_epochs: 1
per_device_train_batch_size: 1
gradient_accumulation_steps: 16

Launch training with:

accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml \
  scripts/run_simpo.py <config_file>

Parameters

All parameters below are required unless noted otherwise.

Parameter	Meaning	Required
beta	Reward scaling factor; typical range 2.0-10.0	yes
gamma_beta_ratio	Target margin ratio; range 0-1	yes
loss_type	Loss function type: sigmoid or hinge	yes
sft_weight	Weight for optional SFT regularization loss (0.0 to disable)	yes
learning_rate	Learning rate; critical range 3e-7 to 1e-6	yes
num_train_epochs	Number of training epochs	yes
per_device_train_batch_size	Batch size per device	yes
gradient_accumulation_steps	Number of gradient accumulation steps	yes
output_dir	Directory for model outputs	yes
model_name_or_path	HuggingFace model identifier or local path	yes
torch_dtype	Torch data type (e.g., bfloat16)	no
dataset_mixer	Dataset and mixing ratio	yes
dataset_splits	List of dataset splits to use	no
gradient_checkpointing	Enable gradient checkpointing to reduce memory	no

Hardware Requirements

• Single-node training is sufficient; DeepSpeed ZeRO-3 is recommended.
• GPU: NVIDIA A100/H100 recommended.
• VRAM estimates:
  • 7B model: 1x A100 40GB
  • 8B model: 2x A100 40GB
  • 70B model: 8x A100 80GB
• Mixed precision BF16 is recommended.

Memory Optimization

To reduce memory usage, use DeepSpeed ZeRO-3, gradient checkpointing, and Flash Attention 2. Enable gradient checkpointing by adding the following to your configuration:

gradient_checkpointing: true

Failure Modes and Fixes

Loss divergence

Reduce the learning rate or beta. For example:

learning_rate: 3e-7  # Reduce from 5e-7

beta: 1.0  # Reduce from 2.0

Model forgets capabilities

Add SFT regularization by setting sft_weight to a positive value:

sft_weight: 0.1  # Add SFT loss component

Poor preference separation

Increase beta and gamma_beta_ratio:

beta: 5.0            # Increase from 2.0
gamma_beta_ratio: 0.8  # Increase from 0.5

Out-of-memory (OOM) during training

Reduce per_device_train_batch_size, increase gradient_accumulation_steps, and enable gradient checkpointing:

per_device_train_batch_size: 1
gradient_accumulation_steps: 16  # Maintain effective batch

Constraints and Caveats

• Learning rate must be in range 3e-7 to 1e-6; this is critical for stability.
• beta range 2.0-10.0; gamma_beta_ratio range 0-1.
• For reasoning tasks, use a lower LR (3e-7), higher beta (5.0), and higher gamma_beta_ratio (0.7).
• Single-node training is sufficient; DeepSpeed ZeRO-3 is recommended.
• GPU: NVIDIA A100/H100 recommended; VRAM estimates: 7B model 1x A100 40GB, 8B model 2x A100 40GB, 70B model 8x A100 80GB.
• Mixed precision BF16 is recommended.
• Memory optimization: DeepSpeed ZeRO-3, gradient checkpointing, Flash Attention 2.
• Loss divergence: reduce learning rate or beta.
• Model forgets capabilities: add SFT regularization (sft_weight: 0.1).
• Poor preference separation: increase beta and gamma_beta_ratio.
• OOM: reduce per_device_train_batch_size, increase gradient_accumulation_steps, enable gradient checkpointing.