SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

Free

Comparative study of SFT and RL in foundation model post-training

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

This research paper, titled 'SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training,' provides a systematic comparison between supervised fine-tuning (SFT) and reinforcement learning (RL) in the post-training of foundation models. The study introduces GeneralPoints, an arithmetic reasoning card game, and utilizes V-IRL, a real-world navigation environment, to assess generalization across both textual rule-based variants and visual variants. Key findings demonstrate that RL, especially with outcome-based rewards, generalizes effectively to unseen variants, while SFT tends to memorize training data and struggles with out-of-distribution scenarios. Furthermore, the paper reveals that SFT is essential for stabilizing the model's output format, which enables subsequent RL to achieve performance gains. The work highlights the distinct roles of SFT and RL in acquiring generalizable knowledge for complex multi-modal tasks.

Key Features

Introduces GeneralPoints arithmetic reasoning card game for evaluating generalization

Adopts V-IRL real-world navigation environment for visual generalization assessment

Compares supervised fine-tuning (SFT) and reinforcement learning (RL) in post-training

Demonstrates RL generalizes across rule-based textual and visual variants with outcome-based rewards

Reveals SFT stabilizes model output format, enabling effective RL training

Analyzes underlying visual recognition improvements from RL

Pros & Cons

Pros

RL generalizes well to out-of-distribution rule-based and visual variants
SFT effectively stabilizes model output format, facilitating subsequent RL training
Provides clear empirical evidence distinguishing memorization from generalization in post-training

Cons

SFT tends to memorize training data and fails on out-of-distribution scenarios
RL may require initial SFT for format stabilization to achieve its full potential
Findings are demonstrated on specific tasks (GeneralPoints, V-IRL) and may not fully generalize to all domains

Best For

Understanding the generalization capabilities of post-training techniques in foundation modelsDeciding between SFT and RL for fine-tuning models on tasks with unseen variantsDesigning hybrid training pipelines that leverage SFT for format stabilization and RL for generalizationEvaluating model performance in arithmetic reasoning and visual navigation tasks

FAQ

What does the paper conclude about SFT and RL?

The paper concludes that reinforcement learning, especially with outcome-based rewards, generalizes across both rule-based textual and visual variants, while supervised fine-tuning tends to memorize training data and struggles with out-of-distribution scenarios. However, SFT is essential for stabilizing model output format, enabling RL to achieve performance gains.

What environments are used for evaluation?

The study uses GeneralPoints, an arithmetic reasoning card game, and V-IRL, a real-world navigation environment, to test generalization in textual and visual domains.

What are the implications for model training?

The findings suggest that RL is superior for acquiring generalizable knowledge in complex multi-modal tasks, but SFT should not be skipped as it provides necessary format stabilization.