Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Analysis

Why This Paper Matters

This paper addresses a critical blind spot in interpretability research: the assumption that high reconstruction scores from natural-language autoencoders certify faithful explanations. The authors demonstrate that reconstruction is structurally insensitive to individual false claims—only ~2% of specific claims affect the reconstruction score—meaning an explanation can be mostly wrong yet still score well. This undermines a widely used evaluation paradigm and has direct implications for AI safety, where deceptive models could produce plausible-sounding but false explanations.

The paper's key insight is that the standard recipe for training explanation autoencoders allows the development of co-adapted private codes: false wording that the reconstruction depends on, effectively hiding lies from the evaluation metric. By showing this occurs in 5/5 runs under synthetic ground truth, the authors provide strong evidence that the problem is systematic, not anecdotal.

Technical Contributions

The paper makes several concrete contributions:

• Audit protocols: The grounded-vs-true cross and evaluator swap detect co-adapted codes by comparing reconstruction behavior under different conditions.
• RECAP (Readable Encodings via Co-trained Auxiliary Predictors): Linear heads are trained alongside the target model to keep designated content decodable by independent probes, preventing the model from learning private codes.
• Verification framework: Instead of relying on the model's own verbalizer, RECAP enables external probes to score the truthfulness of explanations, providing a check that is robust to adversarial manipulation.

Results

Key quantitative results:

• Standard autoencoders: only ~2% of claims are reconstruction-dependent; co-adapted codes appear in 5/5 runs.
• RECAP-trained sandbox models: fresh verbalizers state designated content truly, codes vanish, at a cost of +0.001 nats.
• On Pythia-160M: designated content becomes reliably probe-decodable, but fresh verbalizers only achieve truth 0.44-0.46 (vs near-zero control).
• Probe AUC: 0.96 with RECAP vs 0.82 without; against adversarial edits that suppress ~87% of lie penalty, RECAP probe AUC 0.95 vs control probe 0.51 (chance).

Significance

This work has broad implications for AI interpretability and safety. It provides a practical method to make internal model content independently verifiable, reducing reliance on self-reported explanations that models can game. The RECAP approach could be integrated into training pipelines to ensure that designated safety-relevant features remain decodable by external auditors. The limitations on larger pretrained models (Pythia-160M) highlight the need for further research into scaling these techniques. Overall, the paper shifts the evaluation paradigm from reconstruction-based to probe-based verification, offering a more rigorous standard for explanation faithfulness.