Tulu V3

Analysis

Why This Paper Matters

Post-training—the phase after pretraining where models are aligned to follow instructions and perform specific tasks—has become a critical bottleneck in developing capable language models. While many top-performing models like GPT-4 and Claude are closed-source, the open-source community has struggled to replicate their instruction-following abilities. TÜLU 3 changes this by providing a fully open recipe that not only matches but surpasses several closed models. This matters because it democratizes access to state-of-the-art alignment techniques, enabling researchers and practitioners to build, customize, and audit high-quality instruction-tuned models without relying on proprietary APIs.

How It Works

The TÜLU 3 pipeline is a multi-stage process that begins with careful data curation. The team surveyed public datasets like WildChat, Open Assistant, and FLAN v2, selecting those that promote diversity and target specific skills such as math reasoning (OpenMathInstruct), coding (Evol-CodeAlpaca), and multilingual understanding (Aya). To fill gaps, they synthesized new data using a persona-driven approach: ~250K personas from Persona Hub were used to generate prompts for precise instruction following, math, and coding. For example, they created 29,980 verifiable instruction-response pairs (IF-PERSONA-SFT) by prompting GPT-4o with a persona and a constraint type from the IFEval benchmark.

A critical step is prompt decontamination to prevent data leakage. The authors used 8-gram matching to identify training instances that overlap with evaluation benchmarks. A training set was considered contaminated if any instance overlapped with more than 2% of evaluation instances. This rigorous filtering ensures that performance gains are genuine.

Supervised fine-tuning (SFT) is performed on a carefully mixed dataset. The team identified lagging skills in their previous TÜLU 2 model and iteratively added or removed datasets to improve those skills. Key experiments showed that removing WildChat (diverse chat data) degraded AlpacaEval performance, while removing safety-specific SFT data primarily hurt safety metrics. They also found that model performance generally improves with more SFT data, though TruthfulQA decreased at larger sizes.

Preference tuning uses Direct Preference Optimization (DPO) on a mix of on-policy and off-policy data. The data creation pipeline involves prompt selection, response generation from a pool of models, and preference annotation using GPT-4o as a judge. Ablation studies revealed that scaling the number of unique prompts improves DPO performance, while duplicating prompts with different responses does not. Using new prompts (unused in SFT) combined with reused prompts yields the best results, and including on-policy data from the SFT model further boosts performance.

Finally, Reinforcement Learning with Verifiable Rewards (RLVR) is applied. Unlike traditional RLHF that uses a learned reward model, RLVR only provides rewards when the model's generation can be automatically verified as correct (e.g., math problem solutions). This eliminates the need for a separate reward model and aligns the model more directly with objective correctness.

Results

TÜLU 3 models, built on Llama 3.1 8B and 70B, achieve state-of-the-art results. They outperform the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models like GPT-4o-mini and Claude 3.5-Haiku on a range of benchmarks including GSM8K, MATH, HumanEval, and IFEval. The 70B model is particularly strong, demonstrating that the recipe scales effectively.

Significance

TÜLU 3 is a landmark contribution to open-source AI. By releasing the full recipe—data, code, and training configurations—it provides a reproducible path to building instruction-tuned models that rival proprietary systems. The introduction of RLVR is a promising alternative to traditional RLHF, potentially simplifying alignment and improving reliability. This work empowers the community to innovate on post-training techniques, leading to more capable and transparent AI systems.