Error typing for smarter rewards: Improving process reward models with error-aware hierarchical supervision

Analysis

Why This Paper Matters

Process reward models (PRMs) have become a cornerstone of test-time scaling for large language models, enabling fine-grained evaluation of intermediate reasoning steps. However, standard PRMs typically provide only a scalar correctness score, which lacks interpretability and often fails to capture the nuanced nature of reasoning errors. This paper addresses a critical gap by introducing error-aware hierarchical supervision, which not only judges step correctness but also categorizes the type of error. This shift from binary feedback to structured error information has the potential to significantly improve the quality of reward signals, leading to more robust and reliable reasoning in LLMs.

The significance of this work lies in its practical implications for AI systems that rely on step-by-step reasoning, such as mathematical problem solvers, code generation, and multi-hop QA. By providing richer supervision, the proposed method enables PRMs to better localize and diagnose errors, which is essential for effective self-correction and test-time search. This aligns with the broader trend toward more interpretable and controllable AI, where models not only produce answers but also explain and justify their reasoning processes.

Technical Contributions

• Error-Aware Hierarchical Supervision: The core innovation is a hierarchical training objective that jointly predicts step correctness and error type, using a shared representation but separate output heads. This allows the model to learn both coarse and fine-grained distinctions.
• Error Taxonomy: The authors define a taxonomy of common reasoning errors (e.g., arithmetic mistakes, logical fallacies, missing steps), which provides structured labels for supervision.
• Training Framework: They integrate the hierarchical loss into a standard PRM training pipeline, demonstrating that the auxiliary error classification task improves the primary correctness prediction through multi-task learning.
• Evaluation Protocol: The paper includes comprehensive experiments on multiple reasoning benchmarks, comparing against baseline PRMs and analyzing the impact of error-aware supervision on step-level accuracy and downstream task performance.

Results

The proposed error-aware PRM consistently outperforms baseline PRMs across several benchmarks. For instance, on a mathematical reasoning dataset, the model achieves higher step-level error detection accuracy (e.g., +5-10% absolute improvement) and improves the final answer accuracy when used for best-of-n sampling or step-wise beam search. The authors also report that the error type predictions are reasonably accurate, suggesting that the model learns meaningful error categories. These results highlight the practical benefits of incorporating error-aware supervision into PRM training.

Significance

This research contributes to the growing body of work on process supervision and reward modeling, offering a more nuanced alternative to scalar rewards. By making error types explicit, it opens the door to more targeted interventions, such as providing specific feedback to the model or guiding search algorithms toward more promising reasoning paths. The approach also has implications for interpretability, as the error type predictions can be used to explain why a particular step was deemed incorrect. Future work could extend this framework to other domains and explore automatic error annotation to reduce labeling costs. Overall, this paper represents a meaningful step toward more intelligent and self-aware reasoning systems.