ContextBench: A Benchmark for Context Retrieval in Coding Agents (2026)

ContextBench is a process-oriented benchmark designed to evaluate how LLM-based coding agents retrieve and use code context during automated issue resolution. It comprises 1,136 issue-resolution tasks sourced from 66 open-source repositories spanning eight programming languages, each augmented with human-annotated gold contexts. The benchmark includes an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout the resolution process. It evaluates four frontier LLMs and five coding agents, revealing that sophisticated agent scaffolding yields marginal gains in context retrieval, LLMs consistently favor recall over precision, and there are substantial gaps between explored and utilized context. ContextBench provides intermediate gold-context metrics that complement existing end-to-end benchmarks, offering valuable signals for understanding and improving LLM reasoning in software tasks.