Beyond Solver Verdicts: Generative Reward Models for Autoformalization

Analysis

Why This Paper Matters

Neurosymbolic systems often rely on mathematical solvers to guarantee correctness, but solvers only check whether a formal translation yields the expected verdict, not whether it is truly equivalent to the intended formalization. This paper identifies and formalizes a critical failure mode called Verdict-Preserving-Unfaithfulness (VPU), where an incorrect encoding executes successfully and matches the expected verdict. Such deceptively valid traces can silently corrupt downstream reasoning, making solver verdicts an unreliable signal for faithfulness.

The authors prove that structural, verdict-only verification heuristics are mathematically bounded to chance-level detection on these traces, meaning no amount of engineering on verdict-based checks can reliably catch VPU. This theoretical result reframes the problem: faithful verification requires a richer, semantic signal beyond solver outcomes. The proposed Generative Verification (GenV) addresses this by distilling an offline Z3-equivalence oracle into a continuous, reference-free score using the language model's own vocabulary space, turning the model into a generative verifier rather than a mere translator.

By showing that this generative readout natively extracts spatial error coordinates without explicit localization training, the paper connects reward modeling, mechanistic interpretability, and formal verification. It matters for anyone building agentic systems that allocate test-time compute based on verification signals, because unreliable verdicts can misguide search and self-correction.

Technical Contributions

• Verdict-Preserving-Unfaithfulness (VPU): A formal failure mode where an incorrect encoding passes solver execution and matches the expected verdict, exposing a blind spot in solver-based verification.
• Theoretical bound: Proof that structural, verdict-only heuristics are limited to chance-level detection on VPU traces, establishing the need for semantic verification.
• Generative Verification (GenV): A method that distills an offline Z3-equivalence oracle into a reference-free, continuous reference-equivalence score by repurposing the language model's native vocabulary space.
• Mechanistic analysis: Use of decision-projected logit lenses and sparse autoencoders to show that the generative readout extracts precise spatial error coordinates without explicit localization training.
• Oracle-mined verifier (GenV+HN): A practical verifier trained on oracle-mined data that generalizes zero-shot across unseen translators and divergent formal styles.

Results

The oracle-mined verifier GenV+HN achieves 0.961 AUROC in reference-equivalence verification, substantially outperforming verdict-only heuristics that are theoretically bounded to chance. It generalizes zero-shot across unseen translators and divergent formal styles, indicating that the learned score captures semantic equivalence rather than translator-specific artifacts. In agentic test-time compute allocation, GenV+HN yields an 11.3-point downstream accuracy gain, showing that better verification signals translate into concrete reasoning improvements. The mechanistic analysis further demonstrates that the model's generative readout contains precise spatial error coordinates, suggesting interpretable internal representations of formal errors.

Significance

This work challenges the assumption that solver verdicts are sufficient for verifying formal translations. By proving the limits of verdict-only heuristics and introducing a generative, reference-free equivalence score, it opens a new direction for reward modeling in neurosymbolic systems. The 0.961 AUROC and 11.3-point downstream gain indicate that GenV can serve as a reliable reward signal for autoformalization, test-time compute allocation, and self-improving agents. More broadly, the mechanistic finding that language models can natively represent spatial error coordinates without explicit localization training suggests that generative verification may generalize to other domains where correctness is defined by reference equivalence rather than executable outcomes. As AI systems increasingly combine language models with formal tools, methods like GenV could become essential for ensuring that formal translations remain faithful, not merely executable.