Teaching small language models to reason

Analysis

Why This Paper Matters

Chain-of-thought prompting has emerged as a powerful technique to elicit step-by-step reasoning from large language models, achieving state-of-the-art results on complex reasoning tasks. However, the computational and memory demands of large models limit their deployment in many practical settings. This paper addresses a critical gap: can smaller, more efficient models be taught to reason in a similar manner? By demonstrating that knowledge distillation of chain-of-thought rationales is effective, the authors open the door to deploying reasoning-capable models on edge devices, in real-time applications, and in scenarios with limited compute budgets. The work is particularly timely as the AI community increasingly focuses on model efficiency and accessibility.

Technical Contributions

• Distillation Framework: The paper proposes a straightforward yet effective method: use a large teacher model (e.g., GPT-3) to generate chain-of-thought rationales for training examples, then fine-tune a smaller student model (e.g., T5-small) on these rationales and the corresponding answers.
• Generalization: The distilled models are evaluated not only on in-distribution tasks but also on unseen reasoning tasks, showing that the learned reasoning skill transfers beyond the training data.
• Ablation Studies: The authors compare against standard fine-tuning without rationales and against models trained only on answers, isolating the benefit of the chain-of-thought supervision.

Results

On arithmetic reasoning benchmarks (e.g., GSM8K, SVAMP), the distilled small models achieve accuracy within 5-10% of the teacher model, significantly outperforming small models fine-tuned without rationales (e.g., 40% vs. 15% on GSM8K). On commonsense reasoning datasets (e.g., CSQA, StrategyQA), similar gains are observed. The distilled models also show improved robustness to rephrased questions.

Significance

This research has immediate practical implications: it provides a recipe for creating compact, reasoning-capable models that can be deployed at scale. It also raises interesting questions about the nature of reasoning in neural networks—whether it can be effectively compressed and transferred. Future work may explore more efficient distillation techniques, multi-teacher ensembles, and applications to other modalities.