teamcity-ai-agent-testing-demo

![Image](https://blog.jetbrains.com/wp-content/uploads/2025/08/swe-bench-ai-agent-testing-teamcity-github-1.png)

<div align="center">
<p>
<a href="#overview">Overview</a> •
<a href="#project-structure">Project structure</a> •
<a href="#setup-instructions">Setup instructions</a> •
<a href="#monitoring-and-results">Monitoring and results</a> •
<a href="https://www.jetbrains.com/teamcity/use-cases/ai/" target="_blank">TeamCity for AI agent evaluation ↗️</a>
</p>


</div>

# SWE-Bench AI Agent Testing with TeamCity

[TeamCity](https://www.jetbrains.com/teamcity/use-cases/ai/) gives you a reproducible CI/CD backbone for AI agent evaluation: orchestrate parallel, isolated (Docker) runs across benchmark tasks (e.g., SWE-Bench), validate patches automatically, and capture metrics, logs, and artifacts to track performance and costs at scale via Kotlin-DSL pipelines.

This TeamCity configuration provides a complete framework for testing AI agents against the SWE-Bench Lite dataset, which contains 300+ software engineering tasks from popular Python repositories.

## Overview

The system evaluates AI agents by:
1. Preparing isolated Docker environments for each SWE-Bench task
2. Running AI agents against specific coding problems  
3. Evaluating solutions using the official SWE-Bench evaluation harness
4. Collecting performance metrics and success rates

## Architecture

### Projects Structure

- **JetBrains Junie AI Agent** (`JetBrain_Junie_AI_Agent.kt`)
  - Downloads Junie CLI from GitHub releases and IntelliJ IDEA
  - Creates task subsets for progressive testing
  - Individual task execution builds for all 300+ SWE-Bench tasks
  
- **Google Gemini CLI AI Agent** (`Google_Gemini_CLI_AI_Agent.kt`)
  - Builds from the official Google Gemini CLI repository (`https://github.com/google-gemini/gemini-cli.git`)
  - Uses Node.js execution environment with npm build process
  - Creates task subsets for progressive testing

- **SWE-Bench Lite** (`SWE_Bench_Lite.kt`): Core dataset and e