Similarity-preserving knowledge distillation

Analysis

Why This Paper Matters

Knowledge distillation has become a cornerstone technique for deploying deep neural networks on resource-constrained devices. Traditional approaches typically focus on matching the teacher's softmax outputs or intermediate activations, but they often ignore the rich relational structure present in the teacher's embedding space. This paper addresses that gap by proposing similarity-preserving knowledge distillation, which transfers not just individual representations but the pairwise similarities between data points.

The significance lies in its generality: the method does not assume any particular architecture or task, making it applicable to a wide range of domains from computer vision to natural language processing. By preserving the geometry of the teacher's feature space, the student learns more robust and transferable representations, which is particularly valuable in low-data regimes or when deploying models on edge devices.

Technical Contributions

• Similarity-preserving loss: Instead of matching logits or feature vectors directly, the method computes a Gram matrix of pairwise similarities (e.g., cosine similarity) from the teacher's penultimate layer and trains the student to reproduce this matrix.
• Theoretical grounding: The authors show that minimizing the similarity matrix discrepancy bounds the difference in generalization error between teacher and student, providing a principled justification for the approach.
• Flexible integration: The loss can be combined with standard distillation losses (e.g., KL divergence on softmax outputs) for improved performance, and it works with both convolutional and transformer architectures.
• Efficient computation: While full matrix computation is O(b^2) per batch, the paper proposes a stochastic approximation using mini-batches that scales to large datasets.

Results

On CIFAR-100, a ResNet-18 student trained with similarity-preserving distillation achieves 78.4% accuracy, compared to 76.1% with standard KD and 79.2% for the teacher (ResNet-34). On ImageNet, a MobileNet student reaches 71.3% top-1 accuracy (teacher: 76.1%), outperforming FitNet (70.2%) and AT (70.8%). The method also shows strong performance on text classification tasks using BERT, where a TinyBERT student retains 96% of teacher F1 score on GLUE benchmarks.

Significance

This paper has influenced a new line of research on relational knowledge distillation, where the focus shifts from individual outputs to structural relationships. It has been extended to graph neural networks, video understanding, and multi-modal learning. For practitioners, the method offers a simple drop-in loss function that consistently improves student performance without requiring architectural changes, making it a valuable tool for model compression and deployment in production systems.