New paper from DeepSeek w/ model coming soon: Inference-Time Scaling for Generalist Reward Modeling

Quote from the abstract: >A key challenge of reinforcement learning (RL) is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. \[...\] Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. **The models will be released and open-sourced.** Summary from Claude: >*Can you provide a two paragraph summary of this paper for an audience of people who are enthusiastic about running LLMs locally?* >This paper introduces DeepSeek-GRM, a novel approach to reward modeling that allows for effective "inference-time scaling" - getting better results by running multiple evaluations in parallel rather than requiring larger models. The researchers developed a method called Self-Principled Critique Tuning (SPCT) which trains reward models to generate tailored principles for each evaluation task, then produce detailed critiques based on those principles. Their experiments show that DeepSeek-GRM-27B with parallel sampling can match or exceed the performance of much larger reward models (up to 671B parameters), demonstrating that compute can be more effectively used at inference time rather than training time. >For enthusiasts running LLMs locally, this research offers a promising path to higher-quality evaluation without needing massive models. By using a moderately-sized reward model (27B parameters) and running it multiple times with different seeds, then combining the results through voting or their meta-RM approach, you can achieve evaluation quality comparable to much larger models. The authors also show that this generative reward modeling approach avoids the domain biases of scalar reward models, making it more versatile for different types of tasks. The models will be open-sourced, potentially giving local LLM users access to high-quality evaluation tools.