SWE-Skills-Bench (2026)

Free

565 real-world SE tasks measuring whether agent skills actually improve outcomes — 39/49 public skills give zero gain; average improvement only +1.2%; reveals fundamental gap in skill design

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About SWE-Skills-Bench (2026)

SWE-Skills-Bench is the first requirement-driven benchmark that isolates the marginal utility of agent skills in real-world software engineering. It pairs 49 public SWE skills with authentic GitHub repositories pinned at fixed commits and requirement documents with explicit acceptance criteria, yielding approximately 565 task instances across six SWE subdomains. A deterministic verification framework maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill. Results show that skill injection benefits are far more limited than rapid adoption suggests: 39 of 49 skills yield zero pass-rate improvement, average gain is only +1.2%, token overhead varies from modest savings to a 451% increase while pass rates remain unchanged. Only seven specialized skills produce meaningful gains (up to +30%), while three degrade performance (up to -10%) due to version-mismatched guidance. The benchmark provides a testbed for evaluating the design, selection, and deployment of skills in software engineering agents.

Key Features

Pairs 49 public SWE skills with authentic GitHub repositories at fixed commits

565 task instances across six SWE subdomains

Deterministic verification framework mapping acceptance criteria to execution-based tests

Enables controlled paired evaluation with and without the skill

Evaluates token overhead and pass-rate impact

Identifies skills that degrade performance due to version mismatches

Pros & Cons

Pros

Rigorous controlled evaluation isolating skill contribution
Reveals that most public skills provide no pass-rate improvement, challenging hype
Identifies specific skills that yield up to +30% pass-rate gain
Quantifies token overhead up to 451% increase, informing cost-benefit analysis
Provides open-source testbed for skill design and deployment

Cons

39 of 49 skills yield zero pass-rate improvement, suggesting limited general utility
Three skills degrade performance by up to -10% due to version mismatches
Benchmark currently focused solely on pass-rate metric, not other aspects like code quality or maintainability
May not cover all real-world software engineering contexts beyond six subdomains

Best For

Evaluating the marginal utility of agent skills in real-world software engineering tasksBenchmarking LLM agents on requirement-driven software engineering problemsDesigning and selecting effective skills for software engineering agentsExposing version-mismatched guidance and contextual compatibility issues

FAQ

What is SWE-Skills-Bench?

SWE-Skills-Bench is a requirement-driven benchmark that measures whether agent skills actually help in real-world software engineering tasks. It pairs public SWE skills with GitHub repositories and acceptance criteria to evaluate pass-rate improvement.

How many task instances does SWE-Skills-Bench include?

The benchmark includes approximately 565 task instances across six software engineering subdomains.

What are the main findings from SWE-Skills-Bench?

39 of 49 public skills yield zero pass-rate improvement, the average gain is only +1.2%, token overhead can increase up to 451%, and only seven specialized skills produce meaningful gains (up to +30%). Three skills degrade performance by up to -10% due to version-mismatched guidance.

How is the evaluation performed?

The benchmark uses a deterministic verification framework that maps each task's acceptance criteria to execution-based tests, enabling controlled paired evaluation with and without the skill.