Self-Distillation Improves Code Generation (April 2026)
FreeApple: embarrassingly simple self-distillation (SSD) — sample from model, fine-tune on raw unverified samples via cross-entropy; no reward model, no verifier, no RL; Qwen3-30B 42.4% → 55.3% pass@1 on LiveCodeBench v6; gains concentrate on hard problems; open source
About Self-Distillation Improves Code Generation (April 2026)
Simple Self-Distillation (SSD) is a post-training method for improving code generation in large language models (LLMs). It works by sampling solutions from the model itself using specific temperature and truncation settings, then fine-tuning the base model on those raw, unverified samples with standard supervised fine-tuning. SSD requires no verifier, reward model, or reinforcement learning. When applied to Qwen3-30B-Instruct, it raises pass@1 on LiveCodeBench v6 from 42.4% to 55.3%, with gains concentrated on harder problems. The method generalizes across Qwen and Llama models at 4B, 8B, and 30B scales, including both instruct and thinking variants. The code is open source.
Key Features
Pros & Cons
- Extremely simple: only requires sampling and standard supervised fine-tuning
- No need for external verifiers, reward models, or reinforcement learning pipelines
- Significant improvement on hard problems where base model struggles
- Works across multiple model families and sizes (Qwen, Llama)
- Open source implementation available