Neura Market
Not All LLM Reasoners Are Created Equal logo

Not All LLM Reasoners Are Created Equal

Free

Analyzing the depth of math reasoning in LLMs.

FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About Not All LLM Reasoners Are Created Equal

This paper from arXiv (2410.01748) investigates the depth of grade-school math reasoning capabilities in large language models (LLMs). The authors evaluate LLMs on compositional pairs of math word problems, where the answer to the second problem depends on correctly answering the first. They discover significant reasoning gaps in most LLMs—performance differences between solving compositional pairs versus independent questions. The gap is larger in smaller, more cost-efficient, and math-specialized models. The study also examines effects of instruction-tuning, code generation, and finetuning, revealing that large reasoning gaps stem from distraction by additional context and poor second-hop reasoning, not test-set leakage.

Key Features

Evaluation of LLMs on compositional math problem pairs
Identification of reasoning gaps beyond standard benchmarks
Analysis of instruction-tuning and finetuning effects on reasoning
Findings on distraction and second-hop reasoning limitations

Pros & Cons

Pros
  • Provides nuanced understanding of LLM reasoning depth
  • Identifies specific failure modes (distraction, second-hop reasoning)
  • Reveals that standard benchmarks may mask reasoning gaps
Cons
  • Limited to grade-school math word problems
  • Findings may not generalize to other reasoning domains
  • No accompanying code or dataset provided in the paper itself

Best For

Benchmarking LLM reasoning capabilitiesUnderstanding overfitting in math reasoning tasksImproving multi-step reasoning in LLMs

FAQ

What is the reasoning gap measured?
The reasoning gap is the performance difference between solving compositional pairs of problems and solving each problem independently.
Which models were tested?
The paper studies various LLMs including smaller and math-specialized models, but specific model names are not mentioned in the abstract.
Does the paper release code or data?
The paper page includes links to potential code repositories via CatalyzeX, DagsHub, Hugging Face, etc., but no explicit code is provided in the abstract.