Phi-4-Reasoning

Analysis

Why This Paper Matters

Large reasoning models like o1 and DeepSeek-R1 achieve state-of-the-art results but require massive compute and memory. Phi-4-Reasoning and its RL-enhanced sibling, Phi-4-Reasoning-Plus, challenge this trend by packing strong reasoning capabilities into a 14-billion parameter model. This matters because it demonstrates that careful data curation and targeted reinforcement learning can unlock reasoning skills in smaller models, making advanced AI more accessible and cost-effective for practitioners.

How It Works

The foundation is Phi-4, a 14B model pretrained on synthetic data emphasizing reasoning. The authors build a specialized dataset for supervised fine-tuning (SFT) by collecting diverse web-based questions and generating synthetic ones from filtered content. They focus on "teachable" prompts—those at the edge of Phi-4's current abilities—using agreement rates of weaker models (e.g., GPT-4o) with o3-mini-generated solutions as a difficulty proxy. Rubric-based LLM evaluators further filter for multi-step reasoning complexity.

SFT data includes over 1.4 million prompt-response pairs (8.3B tokens) across math, coding, and alignment. Two modifications to Phi-4's architecture are made: repurposing placeholder tokens as <think> and </think> markers, and doubling the RoPE base frequency to extend context length from 16K to 32K tokens. A fixed reasoning-focused system message is used to encourage structured chain-of-thought.

During exploration, the team finds that a learning rate of 1e-5, synthetic math data encouraging concise answers, and an additive property for data mixtures (optimal math and code mixtures can be combined independently) are key. The scaling stage reveals that o3-mini with high reasoning effort is a stronger teacher than medium effort, though it produces longer responses.

For Phi-4-Reasoning-Plus, outcome-based RL (GRPO) is applied for 90 steps on ~6K math examples. The reward function is a length-aware accuracy score:

Correct answers up to 25,600 tokens get full reward; incorrect answers below 3,702 tokens avoid penalty. Format violations incur penalties (-0.5 for incompleteness, -1.0 for invalid thinking tags). A repetition penalty based on 5-gram frequency is added:

The final reward combines accuracy and repetition components:

Results

Phi-4-Reasoning and Phi-4-Reasoning-Plus achieve competitive or better performance than larger models like DeepSeek-R1 and o1 on math reasoning, and outperform Claude 3.7 Sonnet and Gemini 2 Flash Thinking on most tasks. The RL variant improves AIME 2024 by over 10% compared to the SFT baseline. However, performance degrades on recent AIME problems and biology/chemistry tasks. The models also show strong generalization to diverse problem types.

Notably, Phi-4-Reasoning-Plus generates longer responses than Phi-4-Reasoning and o3-mini, indicating a trade-off between accuracy and token usage.

Significance

This work demonstrates that smaller models can rival much larger ones through targeted data curation and efficient RL. The modular additive property for data mixtures simplifies tuning, and the non-trivial transfer to general-purpose benchmarks suggests that reasoning improvements are broadly beneficial. For practitioners, this offers a practical path to deploying high-performance reasoning models with lower computational cost, though limitations remain in biology/chemistry and recent AIME problems. Future work could explore longer context windows (e.g., 64K tokens) to further improve performance.