Knowledge distillation on graphs: A survey

Analysis

Why This Paper Matters

Graph neural networks (GNNs) have become a cornerstone for learning on relational data, but their deployment is often hindered by high computational and memory costs. Knowledge Distillation on Graphs (KDG) emerges as a promising solution, enabling the compression of large GNNs into smaller, more efficient student models without significant loss in performance. This survey is timely as it consolidates the fragmented research in this area, providing a clear taxonomy and highlighting the unique challenges posed by graph data—such as preserving structural information and handling neighborhood aggregation.

For AI practitioners, understanding KDG is crucial for deploying graph-based models in resource-constrained environments like mobile devices or real-time systems. The survey not only maps the landscape but also identifies gaps, such as the need for more robust evaluation protocols and the handling of heterogeneous graphs. By offering a structured overview, it lowers the barrier to entry for those looking to apply distillation techniques to graph problems.

Technical Contributions

The survey's primary contribution is its systematic categorization of KDG methods. It distinguishes approaches based on:

• Distillation targets: Whether knowledge is distilled from intermediate layers, final outputs, or structural embeddings.
• Graph structure utilization: How methods incorporate graph topology, such as using neighborhood information or graph kernels.
• Teacher-student architectures: Variations in how the teacher and student models are designed, including homogeneous vs. heterogeneous architectures.

Additionally, the survey discusses the adaptation of classical distillation losses (e.g., KL divergence) to graph-specific objectives, and introduces novel concepts like structure-aware distillation that preserves local and global graph properties. It also compiles a list of benchmark datasets and evaluation metrics commonly used in KDG research, offering a practical guide for experimental design.

Results

Since this is a survey, it does not introduce new experimental results. Instead, it aggregates findings from numerous studies, reporting that KDG methods typically achieve compression rates of 10-50x while retaining 95-99% of the teacher model's accuracy on node classification and link prediction tasks. For instance, some methods show that distilling from a deep GCN into a shallow student can match or even exceed the teacher's performance on sparse graphs. The survey also notes that performance gains are more pronounced on large-scale graphs, where the student benefits from the teacher's distilled structural knowledge.

Significance

The broader impact of this survey lies in its potential to accelerate the adoption of GNNs in production systems. By providing a clear roadmap of KDG techniques, it enables practitioners to select appropriate methods for their specific constraints, whether it be latency, memory, or energy efficiency. Moreover, the survey highlights open challenges, such as developing distillation methods for dynamic graphs and ensuring robustness to distribution shifts, which could inspire future research. As graph-based AI continues to expand into fields like drug discovery, social network analysis, and recommendation systems, efficient model deployment becomes increasingly critical, making this survey a valuable resource for both academia and industry.