Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge (May 2026)

Free

ICML 2026: controlled comparisons show reasoning judges substantially improve accuracy on structured-verification tasks (math, coding) but yield limited or *negative* gains on simpler evaluations while costing significantly more compute; proposes RACER, a distributionally-robust routing policy that

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge (May 2026)

This ICML 2026 paper investigates the cost-benefit trade-off of using reasoning-capable LLMs as automated judges (LLM-as-a-Judge). Through controlled comparisons, the authors show that explicit reasoning substantially improves judgment accuracy on tasks requiring structured verification (e.g., math and coding), but yields limited or even negative gains on simpler evaluations while incurring significantly higher computational cost. To address this, they propose Robust Adaptive Cost-Efficient Routing (RACER), a distributionally-robust routing policy that dynamically selects between reasoning and non-reasoning judges under a fixed budget. RACER formulates routing as a constrained distributionally robust optimization problem with a KL-divergence uncertainty set, admits an efficient primal–dual algorithm, and enjoys theoretical guarantees including uniqueness of the optimal policy and linear convergence. Extensive experiments demonstrate superior accuracy–cost trade-offs under distribution shift.

Key Features

Controlled comparisons between reasoning and non-reasoning LLM judges

Explicit reasoning substantially improves accuracy on structured verification tasks (math, coding)

Reasoning yields limited or negative gains on simpler evaluations

RACER routing policy dynamically selects judges under fixed budget

Constrained distributionally robust optimization with KL-divergence uncertainty set

Efficient primal–dual algorithm with theoretical guarantees (unique optimal policy, linear convergence)

Superior accuracy–cost trade-offs demonstrated under distribution shift

Pros & Cons

Pros

Improves judgment accuracy on complex structured tasks (math, coding)
Selective use of reasoning reduces computational cost compared to universal reasoning
Robust to distribution shift via distributionally robust optimization
Theoretical guarantees for optimality and convergence
Practical experimental validation of cost-accuracy trade-offs

Cons

Reasoning can degrade accuracy on simple evaluation tasks
Requires careful tuning of budget and uncertainty set parameters
Assumes availability of both reasoning and non-reasoning judge variants
Routing policy may add overhead in deployment

Best For

Automated evaluation of LLM outputs in math and coding tasksCost-sensitive LLM-as-a-Judge deployments where reasoning should be used selectivelyScenarios with distribution shift where robust routing is neededResearch on cost-efficient LLM evaluation pipelines

FAQ

What is RACER?

RACER (Robust Adaptive Cost-Efficient Routing) is a routing policy that dynamically selects between reasoning and non-reasoning LLM judges under a fixed budget. It uses constrained distributionally robust optimization to account for distribution shift.

Why is reasoning not free?

Reasoning-capable LLMs improve accuracy on tasks like math and coding but cost significantly more compute and can even hurt accuracy on simpler tasks, so they should be used selectively.

What theoretical guarantees does RACER provide?

RACER admits a unique optimal policy and its primal–dual algorithm achieves linear convergence.

What tasks benefit most from reasoning judges?

Tasks requiring structured verification, such as math and coding, benefit most from explicit reasoning.