Dataset Distillation by Influence Matching

Analysis

Why This Paper Matters

Dataset distillation aims to compress a large dataset into a small synthetic set that retains the training utility, enabling faster model training, reduced storage, and privacy preservation. Prior methods predominantly align process surrogates—such as per-step gradients or training trajectories—which are heuristic and may not guarantee that the final model parameters match those trained on the full data. This paper introduces Influence Matching (Inf-Match), a novel outcome-centric approach that directly aligns the effect of synthetic data on the converged parameters with that of the real dataset. By focusing on the final outcome rather than intermediate steps, Inf-Match offers a more principled and potentially more effective distillation paradigm.

Technical Contributions

• Outcome-centric alignment: Instead of mimicking training dynamics, Inf-Match learns synthetic data such that its influence on the converged model parameters matches the influence of the full dataset.
• Differentiable influence estimator: A linear-time, sample-level estimator that quantifies parameter shifts from adding or removing data, using unrolled optimization and first-order Taylor approximation—avoiding costly inverse-Hessian products and convexity assumptions.
• Scalability: The method is demonstrated on both standard classification (Tiny-ImageNet) and vision-language tasks (Flickr30K), showing broad applicability beyond image classification.

Results

On Tiny-ImageNet with IPC=10, Inf-Match achieves 31.5% accuracy, outperforming the previous state-of-the-art NCFM by +4.7%. For vision-language distillation on Flickr30K, using 200 to 1000 synthetic samples, Inf-Match improves average image/text retrieval performance by 2.5% over NCFM. These results are reported as leading among strong process-matching baselines.

Significance

Inf-Match shifts dataset distillation from heuristic process imitation to principled outcome alignment, potentially setting a new standard for the field. Its success on both classification and multimodal tasks suggests that the approach is general and can be extended to other domains. The linear-time influence estimator makes the method practical for moderate-sized datasets, though scaling to very large datasets may require further optimization. Overall, this work advances the theoretical foundations of dataset distillation and opens avenues for more efficient and faithful data compression.