Thinking Preference Optimization

Analysis

Why This Paper Matters

Large language models (LLMs) have shown remarkable reasoning capabilities when prompted to produce long chain-of-thought (CoT) reasoning. However, generating high-quality long CoT data is expensive and time-consuming, often requiring human annotation or distillation from powerful models. Thinking Preference Optimization (ThinkPO) addresses this bottleneck by showing that existing supervised fine-tuning (SFT) data and readily available short CoT responses can be repurposed to encourage longer, more reasoned outputs. This is significant because it lowers the barrier for improving reasoning in LLMs, making it accessible to more practitioners without the need for costly new data generation.

How It Works

ThinkPO operates in two stages, as illustrated in the pipeline below.

Stage 1: Reasoning SFT. The authors first collect long-reasoning responses for each question using DeepSeek-R1 as a teacher model. These responses are filtered for correctness using GPT-4o-mini, forming the dataset Dsft. The base model (e.g., Qwen2.5-Math-7B) is then fine-tuned on Dsft to acquire advanced reasoning capabilities. This step prepares the model to generate longer, more detailed reasoning chains.

Stage 2: Reasoning DPO. In this stage, the model is further encouraged to produce extended reasoning via Direct Preference Optimization (DPO). The long-reasoning responses from Stage 1 serve as the chosen (preferred) outputs. For each question, a smaller model (Qwen2.5-Math-7B-Instruct) generates a short reasoning response, which is used as the rejected (dispreferred) output. Only samples where the short response's answer matches the long response's answer are retained, ensuring quality. This yields the dataset Ddpo. The model from Stage 1 is then fine-tuned on Ddpo using DPO, which explicitly teaches the model to favor longer, more reasoned outputs over shorter ones.

The key insight is that DPO can leverage existing data without requiring new, expensive long CoT data. By pairing long and short responses for the same question, the model learns to prefer the longer reasoning path, effectively internalizing the benefits of extended deliberation.

Results

The evaluation shows that ThinkPO consistently improves reasoning performance across multiple benchmarks.

• The fine-tuned model achieves scores comparable to Bespoke-Stratos-7B, with improvements on almost all datasets, validating the effectiveness of ThinkPO.

• ThinkPO training improved accuracy for both Bespoke-Stratos-7B and DeepSeek-R1-Distill-Qwen-7B across most datasets. Notable gains include ~5% on Olympiad Bench Math and GPQA-Diamond for Bespoke-Stratos-7B, and an increase from 87.4% to 91.2% on MATH500 for DeepSeek-R1-Distill-Qwen-7B.
• Average response length increased for both models, suggesting enhanced reasoning capacity. DeepSeek-R1-Distill-Qwen-7B's response length grew by ~500 tokens on MATH500, while Bespoke-Stratos-7B's increased by ~1000 tokens.

• ThinkPO works across model sizes (3B, 7B, 14B), consistently improving accuracy by 1–2% on MATH500 and showing gains on four out of five datasets for the larger models.

Significance

ThinkPO presents a practical, data-efficient method for enhancing LLM reasoning. By reusing existing SFT data and short CoT responses, it avoids the high cost of generating new long CoT data. The approach is robust across model sizes and can further improve already-distilled models, suggesting it could be a valuable tool for practitioners seeking to boost reasoning without extensive resources. The alignment with test-time scaling—where longer reasoning leads to better accuracy—underscores the method's theoretical grounding. However, the reliance on a strong teacher model (DeepSeek-R1) and careful filtering may limit applicability in domains where such teachers are unavailable. Overall, ThinkPO offers a promising direction for making reasoning improvements more accessible and scalable.