A survey on knowledge distillation of large language models

Analysis

Why This Paper Matters

As large language models (LLMs) grow in size and capability, their deployment becomes increasingly resource-intensive. Knowledge distillation (KD) offers a promising path to compress these models while retaining performance, making them accessible for real-world applications. This survey arrives at a critical juncture where proprietary LLMs dominate but are often inaccessible for fine-tuning or direct study. By systematically categorizing KD techniques, the paper provides a roadmap for researchers to navigate the rapidly expanding literature.

The significance lies in its comprehensive coverage of both black-box and white-box methods, addressing the practical constraints of working with closed-source models. It also highlights underexplored areas such as multi-teacher distillation and dynamic architectures, which could drive future innovations. For practitioners, this survey offers a clear taxonomy to select appropriate KD strategies based on available resources and model access.

Technical Contributions

• Taxonomy of KD Paradigms: The paper categorizes methods into black-box (using only teacher outputs), white-box (accessing intermediate representations), and hybrid approaches, providing a structured overview.
• Analysis of Knowledge Types: It distinguishes between logit-based, feature-based, and relation-based distillation, explaining their applicability to LLMs.
• Training Strategies: The survey reviews techniques like progressive distillation, multi-stage training, and auxiliary losses to mitigate catastrophic forgetting.
• Evaluation Challenges: It discusses the lack of standardized benchmarks and proposes evaluation dimensions such as task accuracy, inference speed, and model size.

Results

The survey does not present new experimental results but synthesizes findings from existing literature. Key observations include: black-box KD methods (e.g., using GPT-4 outputs) are most prevalent due to API accessibility, achieving 80-90% of teacher performance on benchmarks like GLUE and SuperGLUE. White-box methods (e.g., DistilBERT, TinyBERT) achieve higher compression ratios (up to 40% size reduction) with minimal accuracy loss. However, the paper notes that results vary significantly across tasks and architectures, and many studies lack reproducibility due to proprietary data.

Significance

This survey establishes a foundational reference for the emerging field of LLM distillation. By identifying open problems—such as handling distribution shift, multi-task distillation, and theoretical understanding of KD—it guides future research. For the AI community, it underscores the importance of efficient model deployment, aligning with trends toward democratizing AI. The paper's impact will likely grow as more organizations seek to deploy capable LLMs on resource-constrained devices.