Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

Analysis

Why This Paper Matters

Benchmark contamination—where test data from public benchmarks leaks into pretraining corpora—has become a critical threat to the validity of large language model (LLM) evaluations. When models memorize test questions, their scores inflate, leading to misleading conclusions about capabilities and progress. This paper argues that current mitigation evaluation is insufficient because it relies on aggregate metrics that can hide contamination. By introducing stratified per-question probability evaluation, the authors provide a more sensitive diagnostic that can detect contamination even when overall accuracy appears unaffected. This matters because reliable evaluation is the foundation of trustworthy AI research and deployment.

The paper's central insight—that a zero gap between clean and contaminated evaluation does not imply restoration—challenges a common assumption in the field. Many practitioners use aggregate score differences to judge whether mitigation works, but this can be a false negative. The proposed step-wise mitigation framework offers a way to intervene during evaluation, potentially reducing the impact of memorized test data. As benchmarks continue to be reused and models grow larger, such fine-grained methods are essential for maintaining evaluation integrity.

Technical Contributions

• Stratified per-question probability evaluation: Instead of relying on aggregate accuracy, the method analyzes model output probabilities for each test question, stratified by difficulty or other criteria, to detect anomalous confidence patterns indicative of memorization.
• Zero gap is not restoration: The paper formally demonstrates that equal performance on clean and contaminated test sets does not guarantee that contamination effects have been eliminated, highlighting a flaw in current mitigation assessment.
• Step-wise mitigation framework: A procedure that intervenes at evaluation time, step by step, to adjust or filter predictions affected by contamination, aiming to recover uncontaminated performance.
• Diagnostic for benchmark leakage: The approach provides a practical tool for researchers to audit benchmarks and models for contamination at a granular level.

Results

The abstract does not provide specific quantitative metrics, but the key qualitative result is that stratified per-question probability evaluation reveals contamination signals that aggregate metrics miss. The authors show that in cases where the gap between clean and contaminated scores is zero, per-question probabilities still exhibit systematic differences, indicating residual memorization. The step-wise mitigation is reported to partially correct these anomalies, though the extent of improvement is not detailed in the abstract. This suggests that while mitigation can help, complete restoration may be challenging.

Significance

This work has broad implications for the AI community. It calls for a shift from coarse aggregate evaluation to fine-grained, probability-based diagnostics to ensure that benchmark scores reflect genuine generalization rather than memorization. The step-wise mitigation framework offers a practical path forward, but also underscores that contamination is a persistent issue that requires ongoing attention. As LLMs are increasingly used in high-stakes applications, robust evaluation methods like those proposed here are essential for building trust and guiding responsible development. The paper's emphasis on per-question analysis may also inspire new benchmark designs that are more resistant to leakage.