Inference-Time Scaling for Generalist Reward Modeling

Free

Inference-time scalable generalist reward modeling for LLMs via Self-Principled Critique Tuning

FreeFree tier

Inputs: text

Type

Open Source

Visit Website GitHub (40 stars)

About Inference-Time Scaling for Generalist Reward Modeling

This research paper introduces DeepSeek-GRM, a generalist reward model designed to improve reward signals for reinforcement learning (RL) post-training of large language models (LLMs) across diverse domains. The work proposes Self-Principled Critique Tuning (SPCT), a learning method that fosters scalable reward generation behaviors through online RL, enabling the model to adaptively generate principles and produce accurate critiques. For inference-time scalability, the approach uses pointwise generative reward modeling (GRM) and employs parallel sampling with a meta reward model to guide voting for better scaling performance. Empirically, DeepSeek-GRM outperforms existing reward models on various benchmarks while maintaining reduced biases. The models are publicly released on Hugging Face and ModelScope.

Key Features

Pointwise generative reward modeling (GRM) for flexible input types and inference-time scaling

Self-Principled Critique Tuning (SPCT) using online reinforcement learning to foster scalable reward generation behaviors

Adaptive principle generation and accurate critique generation

Parallel sampling to expand inference compute

Meta reward model to guide voting process for enhanced scaling performance

Open-source models released on Hugging Face and ModelScope

Pros & Cons

Pros

Demonstrates significant improvements in reward model quality and scalability compared to existing methods
Reduces biases commonly associated with reward models
Achieves better performance through inference-time compute scaling rather than scaling training
Open-source availability enables community use and further research

Cons

Still faces challenges on certain task types (acknowledged by authors)
Paper is a preprint under review, not yet peer-reviewed

Best For

Providing accurate reward signals for reinforcement learning post-training of large language modelsGeneralist reward modeling for queries beyond verifiable questions or artificial rulesScalable reward generation for diverse domains in LLM alignment and reasoning

FAQ

What is DeepSeek-GRM?

DeepSeek-GRM is a generalist reward model that uses Self-Principled Critique Tuning (SPCT) to improve inference-time scaling for reward modeling in LLM post-training.

How does Self-Principled Critique Tuning work?

SPCT uses online reinforcement learning to train the model to adaptively generate principles and produce accurate critiques, enabling scalable reward generation behaviors.

Where can I access the models?

The models are released open-source on Hugging Face and ModelScope.

What problem does this address?

It addresses the challenge of obtaining accurate reward signals for LLMs in domains beyond verifiable questions, especially when inference-time compute can be scaled.