Diverse Preference Optimization

Analysis

Why This Paper Matters

Large language models (LLMs) fine-tuned with reinforcement learning from human feedback (RLHF) often suffer from a collapse in output diversity. Standard RLHF concentrates probability mass on the highest-reward responses, even when multiple high-quality options exist. This leads to repetitive, homogenous generations that fail to capture the richness of language. Diverse Preference Optimization (DivPO) directly tackles this problem by rethinking how preference pairs are selected during training. Instead of contrasting the best and worst responses, DivPO chooses the most diverse high-quality response as the chosen example and the least diverse low-quality response as the rejected example. This simple shift encourages the model to explore a broader set of high-reward outputs, preserving diversity without sacrificing quality.

How It Works

DivPO operates in an online optimization loop. For each prompt, the model generates a pool of responses. A reward model scores each response, and a diversity criterion measures how different each response is from the others. The key innovation is in pair selection: the chosen response is the most diverse among those above a reward threshold ρ, while the rejected response is the least diverse among those below ρ. This ensures the model learns to favor rare, high-quality outputs over common, low-quality ones.

Three diversity criteria are proposed:

• Model Probability: Responses with lower probability under the current policy are considered more diverse (D(y) = -log πθ(y|x)).
• Word Frequency: Responses containing rarer words (inverse frequency) are deemed more diverse.
• LLM-as-a-diversity-judge: A separate LLM is prompted to select the most and least diverse responses from the chosen and rejected sets.

The reward threshold ρ is a hyperparameter that controls the trade-off between diversity and quality. A low ρ allows more responses into the chosen set, potentially increasing diversity but risking quality drops. A high ρ restricts to only the highest-reward responses, reducing diversity. The authors use β=0.1 to regularize against the reference model.

DivPO can be used in both offline and online settings. Online training is more prone to collapse in standard methods because the model's generations become less diverse over time, but DivPO's diversity-aware pair selection mitigates this.

Results

In the Persona Generation Task, DivPO significantly improves diversity (measured by unique attributes and distribution uniformity) compared to SFT, DPO, and even strong baselines like Llama-3.1-8B-Instruct and GPT-4o. Online DivPO achieves up to 30% diversity improvement over Instruct and 45.6% over online DPO, while maintaining or improving quality.

In the Keyword Story Generation Task, DivPO again outperforms baselines across all diversity criteria. The ρ parameter allows fine-grained control over the diversity-quality trade-off.

In the Full Story Generation Task, similar trends hold. At ρ=10, DivPO matches SFT quality but with higher diversity. Larger ρ values further boost diversity with a slight quality drop.

Significance

DivPO addresses a fundamental limitation of RLHF: the tendency to collapse to a narrow set of high-reward outputs. By explicitly incorporating diversity into the preference pair selection, DivPO enables LLMs to generate a wider range of high-quality responses. This is particularly valuable for creative tasks like story generation, persona creation, and any application where variety is as important as quality. The method is simple to implement on top of existing RLHF pipelines and offers controllable trade-offs via the ρ hyperparameter. Future work could explore adaptive ρ schedules or more sophisticated diversity metrics. DivPO represents a practical step toward more versatile and less homogenized AI systems.