Neura Market
Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling logo

Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

Free
FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling

This research paper, titled 'Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling,' challenges the common practice of using stronger, more expensive language models (SE) to generate synthetic training data for improving LLM reasoning. The authors propose and evaluate a compute-optimal approach that instead leverages weaker but cheaper (WC) models for synthetic data generation. Through systematic analysis of coverage, diversity, and false positive rates in generated data, combined with fine-tuning experiments in knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup, the paper demonstrates that models trained on WC-generated data consistently outperform those trained on SE-generated data across multiple benchmarks. This work suggests a more efficient and effective strategy for training advanced LLM reasoners under fixed inference budgets.

Key Features

Investigates trade-offs between stronger expensive (SE) and weaker cheaper (WC) models for synthetic data generation
Analyzes synthetic data across coverage, diversity, and false positive rate metrics
Compares fine-tuning in knowledge distillation, self-improvement, and weak-to-strong improvement setups
Demonstrates that WC-generated data yields better reasoning performance under fixed compute budgets
Challenges prevailing practice of relying on strong models for synthetic data
Provides a compute-optimal framework for training advanced LLM reasoners

Pros & Cons

Pros
  • Proposes a more compute-efficient alternative to using very large teacher models
  • Backed by empirical results across multiple benchmarks
  • Introduces a novel weak-to-strong improvement setup not previously explored
  • Provides clear metrics (coverage, diversity, false positive rate) to evaluate synthetic data
  • Open access on arXiv with full paper and code availability implied
Cons
  • Research paper only, not a ready-to-use tool or library
  • Findings may not generalize to all reasoning tasks or model architectures
  • Higher false positive rates in WC-generated data require careful filtering
  • Limited to reasoning benchmarks; other domains not explored

Best For

Training LLM reasoners with synthetic dataOptimizing compute budgets for synthetic data generationImproving reasoning abilities in language models through weak-to-strong teachingDesigning efficient knowledge distillation pipelinesBenchmarking synthetic data quality across different teacher models

FAQ

What is the main finding of this paper?
Training LLM reasoners on synthetic data generated by weaker but cheaper (WC) models consistently outperforms training on data from stronger but more expensive (SE) models under a fixed inference budget.
What metrics are used to evaluate synthetic data?
The paper analyzes coverage, diversity, and false positive rate of the generated synthetic data.
What fine-tuning setups are examined?
Knowledge distillation, self-improvement, and a novel weak-to-strong improvement setup where a weaker LM teaches reasoning to a stronger LM.
Is the paper open access?
Yes, the paper is freely available on arXiv under arXiv:2408.16737.