On-Policy Self-Distillation without Any Supervision

Analysis

Why This Paper Matters

This paper addresses a critical bottleneck in post-training large language models (LLMs): the heavy reliance on external supervision. Traditional methods like OPSD and GRPO require ground-truth labels, environmental feedback, or guidance from larger models, which are expensive and not always available. U-OPSD demonstrates that a model can improve itself using only its own generations, leveraging internal consistency. This is a significant step toward truly autonomous self-improvement, which is essential for scaling AI capabilities without human annotation.

The idea of using majority voting to create pseudo-labels is not new, but applying it in an on-policy distillation framework with a self-consistency threshold is novel. The paper shows that by focusing on disagreeing completions, the model can correct its own errors precisely, leading to substantial gains on mathematical reasoning benchmarks. This could have broad implications for other domains where supervision is scarce.

Technical Contributions

• Unsupervised on-policy self-distillation (U-OPSD): A new training paradigm that eliminates the need for any external supervision, using only the model's own rollouts.
• Self-consistency threshold: A mechanism to filter rollouts and construct reliable pseudo-solutions via majority vote, ensuring the pseudo-labels are of high quality.
• Distillation on disagreement: The model is trained to align its distribution with the pseudo-solution on the completions that disagree, enabling targeted correction of confidently wrong predictions.
• Compatibility with existing methods: U-OPSD can be integrated with base models and training settings, as demonstrated across multiple scales and modes.

Results

U-OPSD consistently improves over base models. On Qwen3 non-thinking mode, it achieves 8.5% and 10.7% average improvement over the base at 4B and 8B scales, respectively. It also outperforms supervised OPSD by 3.2% and 2.3% on average. In thinking mode, U-OPSD matches OPSD (ahead by 0.9% at 4B, level at 8B) and surpasses GRPO by 0.7% and 1.1%. These results are across five challenging mathematical reasoning benchmarks (AIME24, AIME25, HMMT25, MATH500, AMC23), indicating robustness.

Significance

The broader impact of U-OPSD is profound. It suggests that LLMs can self-improve without external feedback, which could reduce the cost and effort of post-training. This is particularly relevant for specialized domains where ground-truth labels are hard to obtain. The method's simplicity and effectiveness could make it a standard tool for autonomous model refinement. However, the reliance on majority voting may not generalize to tasks with high variance or open-ended generation, and the computational cost of multiple rollouts is a practical concern. Future work could explore adaptive thresholds and more efficient sampling strategies.