Weak-to-Strong Reasoning

Analysis

Why This Paper Matters

As large language models (LLMs) surpass human-level capabilities, traditional supervision methods become increasingly inadequate. The weak-to-strong learning paradigm offers a promising alternative by using a less capable model to unlock the latent abilities of a stronger model. However, its application to complex reasoning tasks has been largely untested, and existing approaches often suffer from blind imitation of the weak supervisor, including its errors. This paper directly addresses these gaps by introducing a progressive learning framework that enables the strong model to autonomously refine its training data without requiring input from a more advanced model or human-annotated data. This is a significant step toward scalable and autonomous AI improvement, particularly relevant for reasoning domains where high-quality supervision is scarce.

Technical Contributions

• Progressive Learning Framework: Combines supervised fine-tuning on a selective small but high-quality dataset with preference optimization on contrastive samples identified by the strong model itself. This two-stage process avoids blind imitation and encourages the strong model to surpass its weak supervisor.
• Autonomous Data Refinement: The strong model autonomously identifies contrastive samples (e.g., correct vs. incorrect reasoning paths) for preference optimization, eliminating the need for external human or advanced model input.
• Forward-Looking Validation: The method is validated not only on standard benchmarks (GSM8K, MATH) but also on the highly challenging OlympicArena dataset, demonstrating its robustness and scalability.

Results

• On GSM8K and MATH, the proposed method significantly enhances the reasoning capabilities of Llama2-70b when supervised by three separate weak models (e.g., Llama2-7b, Llama2-13b). The improvements are consistent across different weak supervisors, indicating the framework's generalizability.
• In a forward-looking experimental setup, Llama3-8b-instruct effectively supervises Llama3-70b on the OlympicArena dataset, achieving notable performance gains. This demonstrates the method's potential for real-world scenarios where the supervisor is much weaker than the student.
• The paper reports concrete metrics (e.g., accuracy improvements) but does not provide exact numbers in the abstract; full details are in the paper.

Significance

This work paves the way for a more scalable and sophisticated strategy to enhance AI reasoning powers. By enabling strong models to autonomously refine their training data, it reduces reliance on human annotation and advanced model supervision, which are both expensive and limited. The progressive learning framework is particularly valuable for reasoning tasks, where high-quality supervision is critical but often unavailable. The open-source release of code and resources further accelerates research in weak-to-strong learning and autonomous AI improvement.