A comprehensive index of artificial intelligence and machine-learning research with AI-generated summaries, citation metrics, and direct links to papers and code.
Unknown
This paper introduces DIST, a new knowledge distillation method designed to improve distillation from a stronger teacher by addressing the catastrophic discrepancy between teacher and student outputs.
Unknown
This survey systematically reviews Knowledge Distillation on Graphs (KDG), a technique combining graph neural networks and knowledge distillation to deploy efficient models on graphs.
Borui Zhao, Quan Cui, Renjie Song, et al.
This paper decouples classical knowledge distillation loss into target and non-target class components, revealing that target class knowledge is more important and proposing a method to balance them for improved performance.
Unknown
Proposes a novel online knowledge distillation method that is more efficient and effective than existing approaches.
Qiushan Guo, Xinjiang Wang, Yichao Wu, et al.
This paper proposes an efficient online knowledge distillation method via collaborative learning, eliminating the need for a pre-trained large teacher network.
Unknown
Curriculum temperature for knowledge distillation (CTKD) is an easy-to-use plug-in technique that improves existing knowledge distillation frameworks with negligible additional cost.
Unknown
This paper analyzes knowledge distillation as gradient rescaling and proposes improvements based on the teacher's prediction on ground-truth.
Unknown
MSE loss outperforms KL divergence in knowledge distillation by enabling direct logit matching, with sequential distillation and small-tau KL improving noise robustness.
Unknown
An open-source family of heterogeneous reasoning models (Nano, Super, Ultra) with dynamic reasoning toggle, trained via NAS, distillation, and RL.
Unknown
Gemma 2 improves open language models by interleaving local-global attention and using knowledge distillation, achieving competitive performance with much larger models.
Unknown
Relational knowledge distillation extends traditional distillation by transferring structural relationships between data points from teacher to student.
Xiaohan Xu, Ming Li, Chongyang Tao, et al.
This survey systematically reviews knowledge distillation techniques for transferring capabilities from large proprietary LLMs to smaller models.