Everything, everywhere, all at once: Is mechanistic interpretability identifiable?

Analysis

Why This Paper Matters

Mechanistic interpretability has become a cornerstone for understanding and trusting neural networks, especially in high-stakes domains. However, the field has largely overlooked a fundamental question: are the explanations we derive actually unique and reliable? This paper tackles this head-on by introducing the concept of identifiability to mechanistic interpretability. It forces the community to confront the possibility that many interpretability findings may be artifacts of the method rather than true properties of the model.

The significance is twofold. First, it provides a formal framework to evaluate interpretability methods, which is crucial for scientific rigor. Second, it empirically shows that common methods fail identifiability, meaning that different but equally valid analyses could yield contradictory explanations. This is a wake-up call for researchers and practitioners who rely on these methods for model auditing, safety, and debugging.

Technical Contributions

The paper makes several key technical contributions:

• Formal definition of identifiability: It adapts the concept from statistics and causal inference to mechanistic interpretability, defining when an explanation is unique given the model and data.
• Evaluation framework: It proposes a systematic way to test whether an interpretability method satisfies identifiability, applicable across different architectures and tasks.
• Theoretical analysis: It provides proofs for several popular methods (e.g., feature visualization, activation patching, circuit discovery) showing they are non-identifiable under certain conditions.
• Empirical validation: Through experiments on synthetic and real models, it demonstrates the practical failure of these methods to produce consistent explanations.

Results

The paper's results are sobering. Across a range of interpretability techniques, the authors find that none of the tested methods are fully identifiable. For instance, they show that feature visualization can produce vastly different images for the same neuron depending on initialization, and activation patching can yield different importance scores when the order of interventions is changed. They also quantify the degree of non-identifiability using metrics like variance across runs, showing that explanations often vary more across random seeds than across different models. These findings hold for both convolutional and transformer architectures, suggesting a systemic issue.

Significance

This paper has profound implications for the field of interpretability. It challenges the validity of many published findings that rely on mechanistic interpretability, potentially undermining conclusions about model behavior. It also provides a new criterion—identifiability—that future methods should strive to satisfy. This could lead to the development of more robust interpretability techniques that are grounded in formal guarantees. For AI safety, this work is critical: if we cannot uniquely explain why a model makes a decision, we cannot fully trust it. The paper thus paves the way for a more rigorous and scientifically sound approach to understanding neural networks.