Preprint
Computer Vision

Don't Drop the BATON: Long-Horizon Robot Manipulation via Agentic Subtask Exploration and Transition-aware Memory

Bingxin Xu, Yuzhang Shang, Emilio Ferrara
August 17, 2026
arXivPDF

0

Citations

0

Influential Citations

—

Venue

2026

Year

Abstract

Long-horizon robot manipulation chains many contact-rich skills into one multi-stage task. Vision-language-action (VLA) models increasingly master the individual skills, yet the chain still fails: errors compound beyond the policy's ability to correct, and one subtask silently constrains the next. A promising recipe freezes the VLA and puts an LLM agent in charge: it plans in language, moves in free space with analytic primitives, invokes the VLA only for contact-rich segments, and writes adaptation into language memory. Applied to long horizons, it breaks twice. (1) Competence comes from whole-task exploration at test time, whose cost is multiplicative in stages: if one stage needs T episodes, a K-stage task needs about T^K, and a failure does not reveal which stage caused it. (2) It has no representation of transitions: the VLA primitive carries an exit but no entry condition, so a subtask can succeed in a form its successor cannot use. We present BATON. Against (1), BATON makes the subtask the unit of exploration: each is explored in the cheap short-horizon regime and its solution stored in memory; a long-horizon trajectory is then composed from these solutions rather than discovered whole. Cost becomes additive (T*K) and every failure is attributed to a single stage. Against (2), BATON equips exploration with a transition-aware memory. Within a subtask, a verifier agent governs the invocation transition: the VLA is called only after the wrist view confirms the scene is ready. Across subtasks, a handoff transition restores an entry state disturbed by the predecessor's residue, and a lookahead transition selects the strategy whose outcome the successor can inherit. No parameters are updated. On the long-horizon benchmark RoboMemArena, BATON improves task success by 11.6% and cumulative success by 14.9% over the SoTA.

Analysis

Why This Paper Matters

Long-horizon robot manipulation remains a critical challenge in robotics, as errors compound across multiple contact-rich skills. While vision-language-action (VLA) models excel at individual skills, they fail when chained due to error accumulation and silent constraints between subtasks. This paper identifies two fundamental issues in existing LLM-agent-based approaches: the multiplicative cost of whole-task exploration and the lack of transition representation. BATON directly tackles these issues, making long-horizon manipulation more tractable and reliable.

The significance lies in shifting the unit of exploration from the whole task to subtasks, which dramatically reduces sample complexity and provides clear failure attribution. Additionally, the introduction of transition-aware memory addresses a subtle but crucial problem: ensuring that the output of one subtask is in a form that the next can use. This work is a step toward practical long-horizon robot control, potentially enabling robots to perform complex multi-stage tasks in real-world settings.

Technical Contributions

  • Subtask-level exploration: BATON explores each subtask in isolation, storing successful solutions in memory. This reduces exploration cost from exponential (T^K) to linear (T*K) and allows failures to be attributed to specific stages.
  • Transition-aware memory: Three types of transitions are introduced:
    • Verifier transition: Ensures the VLA is invoked only when the wrist view confirms the scene is ready, preventing premature actions.
    • Handoff transition: Restores the entry state of a subtask that may have been disturbed by the predecessor's residue.
    • Lookahead transition: Selects a strategy whose outcome can be inherited by the successor, ensuring compatibility.
  • No parameter updates: The VLA and LLM are frozen, making the approach lightweight and adaptable without retraining.
  • Composition of long-horizon trajectories: Instead of discovering a whole trajectory, BATON composes it from stored subtask solutions, improving reliability.

Results

On the RoboMemArena benchmark, BATON achieves a task success rate improvement of 11.6% and a cumulative success improvement of 14.9% over the state-of-the-art. These gains are obtained without any parameter updates, highlighting the effectiveness of the exploration and memory mechanisms. The results demonstrate that addressing transition awareness and subtask-level exploration can significantly boost performance in long-horizon manipulation.

Significance

BATON offers a new paradigm for long-horizon robot manipulation by decoupling exploration from the full task and focusing on subtask-level learning. This could inspire further research on hierarchical memory and transition modeling in robotics. The approach is also applicable to other domains where multi-stage tasks are common, such as autonomous driving or surgical robotics. By avoiding parameter updates, BATON is computationally efficient and can be deployed on existing VLA models, making it a practical solution for real-world applications. The emphasis on transition-aware memory may also influence the design of future agentic systems that require robust state management across subtasks.

robot-manipulationvision-language-actionllm-agentlong-horizonexplorationmemorysubtask
BenchmarksComputer Vision

Related Papers