SFT Memorizes, RL Generalizes

Analysis

Why This Paper Matters

This paper addresses a critical question in foundation model fine-tuning: does supervised fine-tuning (SFT) or reinforcement learning (RL) better promote generalization? As models are increasingly deployed in out-of-distribution (OOD) scenarios, understanding the trade-offs between memorization and generalization is essential. The finding that RL generalizes while SFT memorizes has direct implications for practitioners building robust AI systems, especially in safety-critical or rapidly changing environments.

The study spans both text and visual tasks (GeneralPoints and V-IRL), providing cross-modal evidence. The additional insight that SFT stabilizes output format—enabling subsequent RL gains—suggests a potential two-stage pipeline: SFT for format, RL for generalization. This is actionable for AI engineers designing training workflows.

Technical Contributions

• Comparative analysis of SFT vs. RL: Systematic evaluation on two distinct tasks (rule-based GeneralPoints and visual V-IRL) with in-distribution and OOD splits.
• Memorization vs. generalization characterization: Demonstrates that SFT overfits to training distribution, while RL learns transferable policies.
• Inference-time compute scaling: Shows that increasing verification steps (a form of test-time compute) further improves RL generalization, aligning with recent trends in chain-of-thought and self-consistency.
• Format stabilization role of SFT: Identifies that SFT's memorization of output structure is beneficial for downstream RL, suggesting a complementary relationship.

Results

• On GeneralPoints (text-based rule task), RL achieves higher OOD accuracy than SFT, with SFT showing near-perfect in-distribution accuracy but sharp OOD drop.
• On V-IRL (visual navigation), RL generalizes to unseen environments while SFT fails, indicating task-agnostic benefit.
• Scaling verification steps from 1 to 10 yields monotonic improvement in RL OOD performance, with diminishing returns beyond 5 steps.
• SFT alone shows no benefit from increased verification, confirming its reliance on memorized patterns.

Significance

This work provides strong empirical evidence that RL should be preferred over SFT when generalization is the goal, challenging the default use of SFT in many fine-tuning pipelines. For AI practitioners, it suggests investing in RL-based alignment (e.g., RLHF) and inference-time compute scaling rather than exhaustive SFT data collection. The finding that SFT stabilizes format could lead to hybrid approaches that combine the best of both methods. However, the lack of model and compute details limits direct applicability; future work should explore scaling laws and broader task coverage.