When Users Change Their Mind: Evaluating Interruptible Agents (April 2026)

Free

Tests agent ability to handle user interruptions during mid-task execution — critical requirement for realistic deployment in dynamic environments

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About When Users Change Their Mind: Evaluating Interruptible Agents (April 2026)

When Users Change Their Mind: Evaluating Interruptible Agents is a research paper and benchmark (InterruptBench) that systematically studies the ability of LLM-based agents to handle user interruptions during long-horizon web navigation tasks. The paper formalizes three realistic interruption types: addition (adding a new requirement), revision (changing an existing goal), and retraction (removing a requirement). The benchmark is derived from WebArena-Lite and includes a unified simulation framework for testing agents in single- and multi-turn interruption settings. The authors evaluate six strong LLM backbones and find that handling interruptions effectively remains challenging for even large-scale models. Code, dataset, and benchmark are freely available.

Key Features

InterruptBench benchmark derived from WebArena-Lite

Three interruption types: addition, revision, retraction

Unified interruption simulation framework

Evaluation of six LLM backbones across single- and multi-turn settings

Focus on long-horizon, environmentally grounded web navigation tasks

Open-source code and dataset

Pros & Cons

Pros

First systematic study of interruptible agents in long-horizon web navigation
Realistic interruption types grounded in real-world scenarios
Open-source benchmark and dataset for reproducibility
Evaluates multiple large-scale LLM backbones
Strict semantic constraints ensure high-quality interruption scenarios

Cons

Limited to web navigation domain; not tested in other environments
Interruption types (addition, revision, retraction) may not cover all possible user changes
Results show handling interruptions remains challenging even for large models
No focus on non-LLM or multi-modal agents

Best For

Evaluating robustness of LLM agents to mid-task changesTesting agent adaptability in dynamic web environmentsResearch on human-agent interaction in long-horizon tasksBenchmarking AI agents for real-world deployment with user interference

FAQ

What is InterruptBench?

InterruptBench is a benchmark derived from WebArena-Lite that synthesizes high-quality interruption scenarios for evaluating LLM agents in long-horizon web navigation tasks.

What interruption types does the study formalize?

The study formalizes three realistic interruption types: addition (adding a new requirement), revision (changing an existing goal), and retraction (removing a requirement).

Which LLM backbones were evaluated?

Six strong LLM backbones were evaluated across single- and multi-turn interruption settings, though the specific models are not listed in the abstract.

Is the code and dataset available?

Yes, the code and dataset are freely available at the URL provided in the paper.