Language Models that Think, Chat Better

Analysis

Why This Paper Matters

This paper addresses a critical gap in reinforcement learning for language models: while RLVR has shown remarkable success in verifiable domains like math and code, its application to open-ended chat tasks has been limited. The authors demonstrate that by requiring models to generate long chain-of-thought reasoning before responding, RLVR can be effectively extended to general-purpose chat, creative writing, and knowledge tasks. This is significant because it suggests that the benefits of RL-based reasoning training are not confined to narrow, verifiable domains but can improve a wide range of language model capabilities.

The work also challenges the conventional post-training pipeline. The finding that RLMT applied to a base model with only 7K prompts can outperform a complex multi-stage pipeline trained on 25M+ examples is striking. It implies that the quality and structure of training data (i.e., requiring thinking) may matter more than sheer quantity, which has practical implications for reducing the cost and complexity of model alignment.

Technical Contributions

• RLMT Framework: Combines RLVR-style training with preference-based reward models from RLHF, enabling the application of reasoning-focused RL to open-ended tasks.
• Long CoT Requirement: Forces models to generate extended chain-of-thought reasoning before producing the final response, which is shown to improve planning and output quality.
• Algorithm-Agnostic: Validated across multiple RL algorithms (DPO, PPO, GRPO) and model families (Llama-3.1-8B, Qwen-2.5-7B), demonstrating robustness.
• SFT-Free Training: Shows that RLMT can be applied directly to base models, bypassing the standard supervised fine-tuning stage, similar to R1-Zero training.

Results

• On chat benchmarks, RLMT achieves 3-7 point improvements over standard RLHF: AlpacaEval2, WildBench, and ArenaHardV2.
• On creative writing and general knowledge tasks, gains of 1-3 points are reported.
• The best 8B model surpasses GPT-4o in chat and creative writing and rivals Claude-3.7-Sonnet (Thinking).
• Llama-3.1-8B base trained with RLMT on 7K prompts outperforms Llama-3.1-8B-Instruct, which was post-trained with a multi-stage pipeline using 25M+ examples.
• Results are consistent across 40 training runs, indicating reliability.

Significance

This work has the potential to reshape how language models are aligned for general-purpose use. By demonstrating that thinking before responding can be trained via RL for open-ended tasks, it opens up new avenues for improving model quality without relying on massive supervised datasets. The finding that base models can be directly trained with RLMT to surpass instruct-tuned models suggests that the current multi-stage post-training pipeline may be unnecessarily complex. This could lead to more efficient and effective training recipes for future models, reducing both computational cost and data requirements. The qualitative analysis of planning behavior also provides a foundation for future research into understanding and improving model reasoning in conversational contexts.