ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

Analysis

Why This Paper Matters

The emergence of vibe-coding workflows has shifted expectations for coding agents from simple code completion to interactive project building. Existing benchmarks evaluate agents on static, fully specified tasks, failing to capture the dynamic, ambiguous nature of real-world software development. ICAE-Bench addresses this gap by introducing a benchmark that starts from fuzzy product requirements and simulates iterative interaction with a user agent. This is significant because it aligns evaluation with the actual challenges faced by AI coding assistants in practice, such as requirement clarification, debugging, and repository-level construction.

By grounding tasks in real open-source repositories and using automated user simulation, ICAE-Bench ensures both realism and reproducibility. The multi-dimensional diagnostic framework goes beyond simple pass/fail metrics, providing insights into functional correctness, code quality, and interaction quality. This comprehensive evaluation is crucial for developing agents that can truly assist in project building rather than just code generation.

Technical Contributions

• Fuzzy-to-precise task derivation: Each task originates from a real open-source repository with executable behavior, ensuring that ambiguity is natural and grounded.
• User Agent Data (UAD): A novel mechanism for grounding user simulation, allowing the User Agent to reveal hidden constraints without inventing new requirements or leaking implementation details.
• Multi-dimensional diagnostics: Evaluation includes functional correctness, semantic and API similarity, structural fidelity, design quality, and interaction quality, providing a holistic view of agent performance.
• Automated User Agent: Simulates dynamic interaction patterns, enabling reproducible evaluation of interactive coding agents.

Results

The paper does not present experimental results or comparisons with existing methods. It focuses on the design and validation of the benchmark framework itself. Future work is expected to report performance of various coding agents on ICAE-Bench.

Significance

ICAE-Bench represents a paradigm shift in evaluating coding agents, moving from static code completion to interactive project building. This aligns with the growing trend of vibe-coding and AI-assisted software development. The benchmark's design principles—grounded ambiguity, reproducible simulation, and multi-dimensional evaluation—set a new standard for realistic assessment. For the AI field, this work could drive progress in interactive agents, reinforcement learning for software engineering, and human-AI collaboration tools.