Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation

Analysis

Why This Paper Matters

Benchmark contamination has become a central concern in machine learning evaluation, yet the field often treats it as a single problem with a single fix: keep the test set private. This paper argues that this framing is dangerously incomplete. By showing that contamination is at least five distinct phenomena, the authors reveal that many widely adopted mitigation strategies—especially private held-out test sets—address only a fraction of the actual risk. For AI practitioners who rely on benchmark scores to compare models, this is a wake-up call: reported improvements may be inflated by contamination types that current evaluation protocols do not detect.

The paper's core insight is that contamination should be classified by which mitigation it defeats, not by superficial characteristics like whether test data appeared in training. This reframing shifts attention from a binary "contaminated or not" view to a more nuanced understanding of how contamination can arise at different stages of the model development lifecycle. The introduction of "acquired contamination" as a property of a run—rather than a static property of a dataset—is particularly important, as it suggests that contamination can emerge dynamically through repeated evaluation, hyperparameter tuning, or iterative model selection.

Technical Contributions

• A taxonomy of at least five contamination types, each defined by the mitigation strategy it circumvents. This provides a structured way to reason about contamination beyond simple train-test overlap.
• The concept of acquired contamination, which frames contamination as a property of a specific training or evaluation run, not just of the underlying data. This highlights how contamination can be introduced through the model development process itself.
• A critique of private held-out test sets, showing that they mitigate only one of the five contamination types. This challenges a common assumption in benchmark design and evaluation.
• A mitigation-oriented organization, which makes the taxonomy directly actionable: for each contamination type, the paper implies which defenses are ineffective and where new defenses are needed.

Results

The paper does not report quantitative metrics or empirical experiments; it is a conceptual and taxonomic contribution. Its main "result" is the identification and classification of at least five distinct contamination types, along with the demonstration that a private held-out test set addresses only one of them. This qualitative finding is significant because it exposes a gap between perceived and actual protection against contamination in current evaluation practices.

Significance

By decomposing benchmark contamination into multiple distinct failure modes, this paper has the potential to reshape how the AI community designs and reports evaluations. It calls for more rigorous, multi-layered contamination checks that go beyond private test sets, and it provides a vocabulary for discussing contamination in a more precise way. For practitioners, the taxonomy can serve as a checklist to audit evaluation pipelines for overlooked contamination risks. For researchers, it opens avenues for developing targeted mitigations for each contamination type. Ultimately, the paper argues that trustworthy evaluation requires acknowledging the full complexity of contamination—not settling for a single remedy that leaves most of the problem unaddressed.