Concise Reasoning via RL

Analysis

Why This Paper Matters

This paper addresses a critical practical problem in the deployment of large reasoning models: excessive token usage. As reasoning models become more prevalent, their computational cost and latency are major bottlenecks. The authors challenge the common assumption that verbosity is a sign of deeper reasoning, instead showing it is an artifact of reinforcement learning (RL) loss minimization when models produce incorrect answers. This reframing is significant because it suggests that token inefficiency is not an inherent trade-off of reasoning capability but a fixable optimization issue.

The theoretical proof that PPO and GRPO inherently drive policies toward verbosity, even with γ=1, is a novel contribution. It provides a formal basis for why reasoning models tend to be verbose, which has been observed empirically but not well understood. The finding that conciseness correlates with correctness across models is also important, as it suggests that optimizing for conciseness does not harm accuracy and may even improve it. This has immediate implications for model design and training.

Technical Contributions

• Theoretical analysis: The paper provides proofs that PPO and GRPO, when trained on incorrect answers, create a gradient that favors longer outputs. This is shown to occur even when the discount factor γ=1, which is a strong result because it eliminates the usual explanation of temporal discounting.
• Two-phase RL procedure: The authors propose a simple yet effective method: after standard RL training, a brief secondary RL stage is run on a small set of solvable problems. This stage is designed to reduce verbosity without sacrificing accuracy.
• Empirical correlation: The paper demonstrates a consistent correlation between conciseness and correctness across both reasoning and non-reasoning models, suggesting a general principle.
• GRPO collapse modes: The paper identifies that GRPO, a variant of PPO, can exhibit collapse modes that make it unreliable for concise reasoning, providing a cautionary note for practitioners.

Results

The paper reports that the two-phase RL procedure significantly reduces response length while preserving or improving accuracy. Although the abstract does not provide specific numbers, it claims extensive experiments support these findings. The correlation between conciseness and correctness is also empirically validated across multiple model types. The theoretical results are backed by proofs, and the GRPO collapse modes are demonstrated experimentally.

Significance

This work has broad implications for the AI field, particularly for the efficiency and cost-effectiveness of reasoning models. By showing that verbosity is an optimization artifact, it opens the door to more token-efficient training methods that can reduce computational costs and latency. The two-phase RL procedure is a practical solution that can be easily integrated into existing training pipelines. Furthermore, the theoretical insights into PPO and GRPO could inform future RL algorithms designed to avoid such artifacts. The correlation between conciseness and correctness also suggests that future models could be trained to be both more accurate and more efficient, which is a win-win for the field.