Neura Market
**Terminal-Bench** logo

**Terminal-Bench**

Free

—

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About **Terminal-Bench**

Terminal-Bench is an open-source benchmark designed to evaluate large language models (LLMs) on complex, real-world tasks within a terminal environment. It tests an agent's ability to autonomously perform end-to-end operations such as compiling code, training machine learning models, and setting up servers. The framework consists of two core components: a dataset of tasks (each with an instruction, a verification test script, and a reference oracle solution) and an execution harness that connects an LLM to a Docker-based sandboxed terminal. Currently in beta with approximately 100 tasks, Terminal-Bench aims to expand into a comprehensive testbed for AI agents in text-based environments. It is distributed as a pip package and features a CLI tool (tb) for easy use. The project welcomes community contributions, especially new and challenging tasks.

Key Features

Dataset of ~100 tasks with instructions, test scripts, and reference solutions
Execution harness connecting LLMs to a Docker-based sandboxed terminal environment
CLI tool (`tb`) for running benchmarks and managing tasks
Supports autonomous end-to-end tasks like compiling code, training models, and setting up servers
Open-source with community contributions encouraged for new tasks
Dashboard for visualizing benchmark results
Distributed as a pip package (`terminal-bench`)

Pros & Cons

Pros
  • Open-source and free to use (no licensing fees)
  • Focuses on practical, real-world tasks rather than synthetic benchmarks
  • Reproducible task suite with automated verification scripts
  • Actively evolving with community-driven task contributions
  • Comprehensive execution harness with Docker sandbox for safe isolation
  • Easy installation via pip and CLI interface
Cons
  • Currently limited to approximately 100 tasks (beta stage)
  • Requires Docker and uv/pip setup, which may increase initial configuration complexity
  • Benchmark coverage may not yet reflect all terminal use cases
  • No official hosted leaderboard or cloud execution option (must run locally)

Best For

Benchmarking LLM agents on real-world system administration and software development tasksStress-testing system-level reasoning capabilities of AI modelsEvaluating AI agent performance in autonomous terminal environmentsComparing different LLMs or agent frameworks on reproducible terminal-based tasksResearch into AI for DevOps, infrastructure automation, and end-to-end software lifecycle

FAQ

What is Terminal-Bench?
Terminal-Bench is an open-source benchmark for evaluating how well AI agents (LLMs) can handle complicated real-world tasks in a terminal environment, such as compiling code, training models, and setting up servers.
How do I install Terminal-Bench?
Terminal-Bench can be installed via pip: pip install terminal-bench, or using uv: uv tool install terminal-bench. Dependencies include uv and Docker.
How do I run a benchmark?
After installation, use the CLI tool 'tb'. Run 'tb run --help' for detailed usage instructions. The harness connects your language model to a sandboxed terminal environment.
Can I contribute new tasks?
Yes! Terminal-Bench welcomes contributions, especially new and challenging tasks. Tasks are located in the 'tasks' folder of the repository. See the Quickstart Guide for contributing instructions.
Is Terminal-Bench free?
Yes, Terminal-Bench is completely free and open-source. There are no paid tiers or usage limits.