Agent Psychometrics: Task-Level Performance Prediction (April 2026)

Free

Shifts evaluation from simple QA to multi-turn agentic assessment; newer benchmarks like SWE-bench Verified and Terminal-Bench test iterative agent behavior with execution feedback

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Agent Psychometrics: Task-Level Performance Prediction (April 2026)

Agent Psychometrics presents a framework for predicting success or failure on individual tasks within multi-step agentic coding benchmarks. It augments Item Response Theory (IRT) with rich features extracted from tasks—including issue statements, repository contexts, solutions, and test cases—and introduces a novel decomposition of agent ability into LLM and scaffold components. This parameterization enables the aggregation of evaluation data across heterogeneous leaderboards and accurate prediction of task-level performance for unseen benchmarks and unseen LLM-scaffold combinations, offering practical utility for benchmark designers to calibrate task difficulty without expensive agent evaluations.

Key Features

Augments Item Response Theory (IRT) with rich task features (issue statements, repository contexts, solutions, test cases)

Decomposes agent ability into separate LLM and scaffold components

Aggregates evaluation data across heterogeneous leaderboards

Predicts task-level performance for unseen benchmarks and unseen LLM-scaffold combinations

Enables benchmark designers to calibrate task difficulty without expensive agent evaluations

Pros & Cons

Pros

Predicts performance at individual task level rather than aggregate pass rates
Separates LLM ability from scaffold ability for deeper insight
Works across heterogeneous leaderboards and unseen benchmarks
Reduces need for expensive multi-turn agent evaluations

Cons

Currently focused on coding benchmarks only
Requires feature extraction from tasks (issue statements, repos, tests)
Framework is a research paper; no ready-to-use software provided

Best For

Calibrating difficulty of new coding benchmark tasksAnalyzing which tasks challenge agentic LLMs and whyEvaluating agent performance across different LLM-scaffold combinationsReducing computational cost of benchmark evaluation through prediction

FAQ

What does Agent Psychometrics predict?

It predicts success or failure on individual tasks in multi-step agentic coding benchmarks, rather than aggregate pass rates.

How does the framework work?

It augments Item Response Theory with rich features from tasks (issue statements, repository context, solutions, test cases) and decomposes agent ability into LLM and scaffold components.

Who can benefit from this framework?

Benchmark designers can use it to calibrate task difficulty without running computationally expensive agent evaluations. Researchers can analyze which tasks challenge agents and why.

Does the framework support unseen benchmarks?

Yes, it can predict task-level performance for unseen benchmarks and unseen combinations of LLMs and scaffolds by aggregating evaluation data from heterogeneous leaderboards.