SWE-CI: Evaluating Agents on Codebase Maintenance via CI (2026)

Free

First CI-loop benchmark for long-term codebase maintainability — 100 tasks spanning 233 days and 71+ consecutive commits; shifts evaluation from static single-fix to dynamic long-horizon reasoning

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About SWE-CI: Evaluating Agents on Codebase Maintenance via CI (2026)

SWE-CI is a repository-level benchmark designed to evaluate the capabilities of LLM-powered agents in maintaining codebases through continuous integration. It addresses the limitation of static, one-shot bug-fixing benchmarks by focusing on long-term maintainability, tracking functional correctness over time. The benchmark comprises 100 tasks derived from real-world code repositories, each with an average development history of 233 days and 71 consecutive commits. Agents must resolve these tasks through multiple rounds of analysis and coding iterations, providing insights into their ability to sustain code quality during long-term evolution. SWE-CI shifts the evaluation paradigm from static functional correctness to dynamic, long-term maintainability.

Key Features

First repository-level benchmark built upon the Continuous Integration loop

Shift from static single-fix evaluation to dynamic long-term maintainability

100 tasks derived from real-world code repositories

Average development history of 233 days and 71 consecutive commits per task

Requires agents to resolve tasks through dozens of rounds of analysis and coding iterations

Evaluates ability to sustain code quality throughout long-term evolution

Pros & Cons

Pros

First benchmark to evaluate agents on continuous integration style codebase maintenance
Based on real-world repositories with long development histories
Measures maintainability by tracking functional correctness over time
Provides a more realistic evaluation of LLM performance in dynamic software engineering

Cons

Limited to 100 tasks, which may not cover all software engineering scenarios
Benchmark focuses on maintainability rather than other aspects like security or performance
May require significant computational resources to simulate CI loops

Best For

Benchmarking AI code generation models on long-term software maintenance tasksResearch on LLM agent capabilities for continuous integration workflowsEvaluating code quality sustainment in iterative development scenariosShifting focus from static bug fixing to dynamic maintainability in AI evaluation

FAQ

What is SWE-CI?

SWE-CI is the first repository-level benchmark built upon the Continuous Integration loop to evaluate LLM-powered agents on long-term codebase maintenance.

How many tasks does SWE-CI include?

The benchmark comprises 100 tasks, each derived from a real-world code repository with an average development history of 233 days and 71 consecutive commits.

What is the key insight behind SWE-CI?

The key insight is that maintainability can be revealed by tracking how functional correctness changes over time.