Rubrics to Tokens: Token-Level Rewards for Instruction Following (April 2026)

Free

Rubric-based RL with Token-Level Relevance Discriminator — solves credit assignment for instruction following by predicting which tokens satisfy specific constraints; fine-grained optimization

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Rubrics to Tokens: Token-Level Rewards for Instruction Following (April 2026)

Rubrics to Tokens (RTT) is a novel rubric-based Reinforcement Learning (RL) framework designed to align Large Language Models (LLMs) with complex, open-domain instruction-following tasks. It addresses the severe reward sparsity and reward ambiguity problems of existing response-level reward methods by introducing a Token-Level Relevance Discriminator. This discriminator predicts which individual tokens in the response are responsible for satisfying a specific constraint. The policy is optimized via RTT-GRPO, which unifies response-level and token-level advantages within a single framework. Additionally, RTT proposes Intra-sample Token Group Normalization to handle the shift from one-dimensional outcome-level rewards to a three-dimensional reward space in token-level rubric-based RL. Extensive experiments show that RTT consistently outperforms baselines in both instruction-level and rubric-level accuracy across different models.

Key Features

Token-Level Relevance Discriminator predicts which tokens satisfy a given constraint, enabling fine-grained credit assignment.

RTT-GRPO integrates response-level and token-level advantages in a unified policy optimization.

Intra-sample Token Group Normalization adapts reward normalization to three-dimensional token-level reward space.

Solves reward sparsity and reward ambiguity problems inherent in response-level rubric-based RL.

Demonstrated state-of-the-art performance on instruction- and rubric-level accuracy benchmarks.

Pros & Cons

Pros

Provides fine-grained token-level credit assignment, overcoming reward sparsity and ambiguity.
Clearly outperforms response-level reward baselines in both instruction-level and rubric-level accuracy.
Intra-sample Token Group Normalization effectively handles multi-dimensional reward spaces.
Open-source framework with publicly available paper and likely code (no code link found on arXiv page but typical for such papers).

Cons

Requires training an additional Token-Level Relevance Discriminator, increasing computational overhead.
Complexity of implementing the three-dimensional reward space and group normalization may be a barrier for some practitioners.
Experiments only reported on specific benchmarks; generalization to other domains not fully explored in this paper.

Best For

Aligning large language models with complex, open-domain instruction following tasks.Fine-grained reward modeling for reinforcement learning in language generation.Improving constraint satisfaction in multi-constraint instruction following.Replacing or augmenting response-level rewards with token-level feedback for better policy learning.

FAQ

What does RTT stand for?

RTT stands for Rubrics to Tokens, a framework that bridges response-level rubrics and token-level rewards for instruction following.

How does RTT address reward sparsity?

RTT introduces a Token-Level Relevance Discriminator that predicts which tokens in the response satisfy each specific constraint, allowing fine-grained token-level rewards instead of a single scalar response-level reward.

What is Intra-sample Token Group Normalization?

It is a normalization method proposed for the three-dimensional reward space (constraint, token, and sample dimensions) in token-level rubric-based RL, enabling stable training.

What are the main components of RTT?

The main components are the Token-Level Relevance Discriminator, the RTT-GRPO training algorithm, and Intra-sample Token Group Normalization.