Agentprm: Process reward models for llm agents via step-wise promise and progress

Analysis

Why This Paper Matters

Current LLM agents typically rely on outcome-based rewards—only the final success or failure of a task is used to guide learning or search. This sparse signal makes it difficult for agents to recover from early mistakes or learn effective intermediate strategies. AgentPRM addresses this critical gap by introducing process reward models that evaluate each decision step, offering dense, informative feedback. This shift from outcome to process evaluation is analogous to the transition from sparse to dense rewards in reinforcement learning, which has been key to advances in game playing and robotics.

The paper is significant because it tackles a fundamental limitation of LLM agents: their inability to self-correct during multi-step tasks. By providing step-wise promise and progress scores, AgentPRM enables agents to identify and abandon unpromising paths early, leading to more robust and efficient behavior. This is especially important for real-world applications where tasks are long-horizon and errors compound.

Technical Contributions

• Step-wise Promise Metric: Quantifies how likely a given action is to lead to eventual task success, based on the current state and action history.
• Step-wise Progress Metric: Measures how much closer the agent moves toward the goal after each action, providing a continuous signal of advancement.
• Process Reward Model Architecture: A neural network that takes the agent's state-action history and outputs a scalar reward for the latest action, trained on step-level annotations.
• Integration with Agent Frameworks: The PRM can be used to guide beam search during inference or as a dense reward signal for fine-tuning via reinforcement learning.

Results

AgentPRM was evaluated on multiple agent benchmarks including WebShop, ALFWorld, and a custom multi-step reasoning dataset. Key results:

• Task completion rate: +12% absolute improvement over outcome-based reward models on WebShop.
• Action efficiency: 20% fewer steps on average to complete tasks on ALFWorld.
• Ablation: Removing the progress metric reduced performance by 8%, confirming its complementary value to the promise metric.
• Comparison to baselines: Outperforms both sparse reward and uniform step-reward baselines across all tasks.

Significance

AgentPRM offers a practical and scalable method for improving LLM agent decision-making without requiring full trajectory supervision. By decomposing agent evaluation into promise and progress, the approach provides interpretable signals that can be used for both inference-time search and training. This work has broad implications for deploying LLM agents in safety-critical domains where intermediate failures must be detected and corrected. Future work may extend the framework to multi-agent settings or integrate it with world models for even richer step evaluation.