Not All LLM Reasoners Are Created Equal

Analysis

Why This Paper Matters

This paper challenges the assumption that high performance on standard math benchmarks like GSM8K indicates robust reasoning in LLMs. By introducing a compositional evaluation method, it uncovers a 'reasoning gap' that is often masked by aggregate scores. This is crucial for AI practitioners who rely on benchmarks to select models for real-world tasks that require multi-step reasoning.

The findings are particularly relevant for cost-sensitive deployments, as smaller and math-specialized models—often chosen for efficiency—show disproportionately larger reasoning gaps. This suggests that efficiency gains may come at the cost of reasoning robustness, a trade-off that practitioners must consider.

Technical Contributions

The paper's key innovation is the compositional pair evaluation: taking existing math word problems and pairing them so that the second problem's answer depends on the first. This simple yet effective method reveals reasoning depth without needing new datasets.

• Introduces a metric for reasoning gap: the difference in accuracy between solving paired problems and solving each independently.
• Systematically analyzes the effect of model size, instruction-tuning, code generation, and finetuning on the reasoning gap.
• Provides evidence against test-set leakage as an explanation for the gap, instead pointing to distraction and poor second-hop reasoning.

Results

The paper reports that most LLMs show a significant reasoning gap, with smaller models (e.g., those under 10B parameters) exhibiting gaps up to 20-30% lower accuracy on compositional pairs. Math-specialized models, despite high standalone performance, also show larger gaps. Instruction-tuning and code generation have mixed effects: they sometimes reduce the gap for larger models but can worsen it for smaller ones. Finetuning on GSM leads to overfitting, increasing the gap on compositional tasks.

Significance

This work has broad implications for LLM evaluation and development. It underscores that standard benchmarks are insufficient for assessing reasoning abilities, and that compositional evaluations should be adopted more widely. For AI practitioners, it highlights the need to test models on multi-hop reasoning tasks before deployment, especially when using smaller or specialized models. The findings also encourage research into improving second-hop reasoning and reducing distraction from context, which could lead to more robust reasoning in LLMs.