Weak-to-Strong On-Policy Distillation

Analysis

Why This Paper Matters

On-policy distillation (OPD) has become a key technique for transferring capabilities from teacher to student LLMs, but it traditionally relies on a teacher that is at least as capable as the student. This assumption breaks at the frontier, where no larger model exists, and limits the applicability of distillation to scenarios where a stronger teacher is available. W2S-OPD flips this paradigm by showing that a collection of weaker models can be used to improve a stronger student, effectively turning the weakness of available models into an advantage.

The key insight is to construct a proxy teacher in logit space by combining the student's own logits with the difference between a positive and a negative model. This contrastive approach isolates the capability direction—whether it's the skill from RL, the benefit of scale, or the instance-level direction toward a solution—and adds it to the student's base distribution. This keeps the proxy teacher distributionally close to the student, making the distillation stable and effective. The paper demonstrates that even when every supervision source is weaker than the student, the student can still improve, which is a significant departure from conventional wisdom.

Technical Contributions

• Proxy teacher construction: The logit difference between a positive and negative model is added to the student's own logits, creating a teacher that is both capable and distributionally adjacent to the student.
• Three contrast instantiations: (1) post-RL vs pre-RL to isolate RL-instilled skills, (2) larger vs smaller base to isolate scale benefits, and (3) correct vs wrong hints to isolate instance-level solution direction.
• On-policy distillation with reverse KL: The student minimizes per-token reverse KL on its own rollouts, aligning with the proxy teacher's distribution.
• No need for larger teacher: All contrast models are smaller than the student, making the approach feasible at the frontier.
• Code release: The authors plan to release code, facilitating reproducibility and further research.

Results

W2S-OPD was evaluated on four math benchmarks and three code benchmarks. It consistently outperforms standard OPD, which typically uses a larger teacher. Notably, the student is able to surpass the domain teacher (the positive model) in several cases, indicating that the contrastive proxy teacher provides a stronger training signal than direct distillation from the teacher itself. The method also shows continued improvement even when all supervision sources are weaker than the student, which is a novel finding. The analysis reveals that different contrast pairs yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. This suggests that the choice of contrast pair can be tailored to the desired capability.

Significance

This work has broad implications for LLM training and deployment. It challenges the assumption that distillation requires a stronger teacher, opening up new possibilities for improving models at the frontier where no larger model exists. It also offers a cost-effective alternative to training multiple domain experts at the student's scale, as the weak models are cheap to obtain. The contrastive approach to constructing a proxy teacher could inspire further research into leveraging weak or even negative examples for distillation. Overall, W2S-OPD provides a practical and scalable method for continuous improvement of LLMs, which is crucial as models approach the limits of current architectures and data.