Bayesian Influence Functions for Hessian-Free Data Attribution

Analysis

Why This Paper Matters

Classical influence functions are a cornerstone of data attribution, allowing practitioners to trace model predictions back to training examples. However, their application to deep neural networks has been severely limited by the need to invert the Hessian matrix—a computation that is both numerically unstable and infeasible for models with millions or billions of parameters. This paper tackles that bottleneck head-on by proposing a Bayesian reformulation that entirely avoids Hessian inversion.

The significance lies in making influence functions practical for modern-scale deep learning. By leveraging stochastic-gradient MCMC to estimate loss landscape statistics, the method captures higher-order interactions among parameters without the quadratic memory and cubic time costs of Hessian inversion. This opens the door to data attribution for large language models, vision transformers, and other billion-parameter architectures where classical methods fail.

Technical Contributions

• Hessian-free formulation: Replaces the inverse Hessian with statistics of the loss landscape estimated via stochastic-gradient MCMC, eliminating the need for explicit Hessian computation or inversion.
• Local Bayesian influence function (BIF): Extends classical influence functions by incorporating a Bayesian perspective, allowing for uncertainty quantification in attributions.
• Scalability: The method scales to neural networks with billions of parameters, as demonstrated by state-of-the-art results on retraining prediction tasks.
• Higher-order interactions: Captures parameter interactions beyond the local quadratic approximation used by classical influence functions.

Results

The paper reports state-of-the-art results on predicting retraining experiments, which is a standard benchmark for data attribution methods. While exact numerical metrics are not provided in the abstract, the claim of "state-of-the-art" suggests that BIF outperforms classical influence functions and other baselines on this task. The method's ability to scale to billions of parameters is a key empirical achievement.

Significance

This work has broad implications for model interpretability, debugging, and fairness auditing. By making influence functions practical for large-scale models, it enables practitioners to identify which training examples most influence a given prediction, aiding in data cleaning, bias detection, and model improvement. The Bayesian framing also provides uncertainty estimates, which are valuable for high-stakes applications. Future research may extend this approach to other Hessian-dependent methods, such as neural tangent kernels or second-order optimization.