OpenForgeRL: Train Harness-native Agents in Any Environment

Analysis

Why This Paper Matters

Modern AI agents increasingly rely on complex inference harnesses (e.g., Claude Code, Codex, OpenClaw) to orchestrate multi-turn reasoning, tool use, and external system access. However, these harnesses introduce a fundamental challenge: they are difficult to train end-to-end because existing open-source SFT/RL stacks cannot natively express the stateful, multi-process inference patterns they require. This gap means that most agent systems are either trained separately from their harness or not trained at all, limiting their ability to improve through reinforcement learning.

OpenForgeRL directly addresses this bottleneck by providing a lightweight, open-source framework that decouples training from inference. By doing so, it enables researchers to train agents directly within the exact harness and environment they will be deployed in, without needing to modify the harness itself. This is a practical and scalable solution that could democratize agent training, allowing smaller labs to experiment with and improve complex agent systems.

Technical Contributions

• Proxy-based data recording: A lightweight proxy sits between the harness and the model, recording all model calls as training data for a standard RL codebase (e.g., veRL). This allows the harness to run unchanged while generating training signals.
• Kubernetes orchestrator: Each rollout is executed in its own remote container, enabling parallel, scalable training across diverse environments without interference.
• Harness-agnostic design: The framework works with any inference harness (ZeroClaw, OpenClaw, Codex) and any environment, from tool-based to multimodal GUI tasks.
• End-to-end RL training: By feeding recorded interactions back into a standard RL pipeline, the agent learns to improve its behavior within the harness's native context.

Results

OpenForgeRL is validated on two challenging agent families: tool/claw-based agents and multimodal GUI agents. For claw-based agents, OpenForgeClaw achieves 31.7 pass^3 and 55.9 pass@3 on ClawEval, and 33.7 on QwenClawBench, outperforming open baselines of similar size. For GUI agents, OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager, matching or surpassing models several times larger. These results are achieved with only hundreds to a few thousand training tasks, demonstrating data efficiency.

Significance

OpenForgeRL fills a critical infrastructure gap in agentic AI research. By enabling end-to-end RL training on real harnesses, it allows researchers to study how harness design and RL interact—finding, for example, that some harnesses are substantially harder to learn than others, and that RL improves self-verification and tool coverage but not error recovery. This insight could guide future harness and training algorithm design. The open-source release lowers the barrier for entry, potentially accelerating progress in building more capable and reliable AI agents.