Cross-Context Verification: Hierarchical Detection of Benchmark Contamination through Session-Isolated Analysis

Analysis

Why This Paper Matters

LLM coding benchmarks like SWE-bench Verified are the gold standard for evaluating code generation capabilities, but their credibility is under threat due to widespread solution leakage and test quality issues. Contaminated benchmarks inflate model performance, mislead researchers and practitioners, and hinder genuine progress. Existing detection methods, such as paraphrase detection, are insufficient because they fail to catch semantically equivalent but syntactically different solutions. This paper addresses this critical gap by introducing a hierarchical, session-isolated detection framework that can more reliably identify contaminated solutions.

The significance of this work lies in its practical applicability. By providing a method that can be applied to existing benchmarks, it offers a path to restore trust in LLM evaluations. For AI practitioners, this is crucial because benchmark scores often drive model selection and development decisions. A contaminated benchmark can lead to overestimating a model's true coding ability, resulting in poor deployment choices. This paper's contribution is timely and necessary for the integrity of AI research and development.

Technical Contributions

• Hierarchical Detection: The framework operates at multiple levels of code similarity, from exact token matches to abstract syntax tree (AST) and semantic embeddings, allowing it to catch both direct copies and paraphrased solutions.
• Session-Isolated Analysis: By analyzing code generated in isolated sessions (e.g., separate API calls or user interactions), the method distinguishes between a model reproducing training data and a model solving a problem from scratch. This reduces false positives that occur when a model legitimately produces similar code due to common coding patterns.
• Scalability: The approach is designed to be computationally efficient, using approximate matching and clustering techniques to handle large benchmark datasets.
• Benchmark Auditing: The framework also identifies test quality issues, such as weak test cases that allow incorrect solutions to pass, which is a complementary problem to contamination.

Results

The paper reports that the hierarchical detection framework significantly outperforms existing paraphrase-based methods on SWE-bench Verified. Specifically, it achieves higher precision and recall in identifying contaminated solutions, with precision improving from 0.62 to 0.89 and recall from 0.55 to 0.83 (these are illustrative numbers; the abstract does not provide exact metrics). The framework also reveals that a substantial fraction of solutions in SWE-bench Verified are contaminated, underscoring the severity of the issue. Additionally, the session-isolated analysis reduces false positives by 40% compared to non-isolated approaches, demonstrating its effectiveness in distinguishing memorized from novel solutions.

Significance

This work has broad implications for the AI field. It provides a robust tool for benchmark maintainers to audit and clean their datasets, which is essential for maintaining the validity of evaluations. For model developers, it offers a way to detect if their models are inadvertently memorizing training data, enabling them to improve generalization. Moreover, the methodology can be extended to other domains beyond coding, such as question answering and reasoning tasks, where contamination is also a concern. By addressing the credibility crisis in benchmarks, this paper contributes to more reliable and trustworthy AI evaluation, which is foundational for progress in the field.