GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

Free

A benchmark exposing LLM mathematical reasoning fragility

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

GSM-Symbolic is a research paper and improved benchmark for evaluating mathematical reasoning in large language models (LLMs). It extends the GSM8K benchmark by using symbolic templates to generate diverse question sets with controlled variations, enabling more reliable metrics. The study reveals that LLMs exhibit significant performance variance when numerical values are altered, and their reasoning deteriorates sharply as the number of clauses increases—adding an irrelevant clause can cause up to a 65% performance drop. The authors conclude that current LLMs rely on replicating reasoning steps from training data rather than performing genuine logical reasoning. The paper is available on arXiv and was presented at ICLR 2025.

Key Features

Based on symbolic templates for controllable question generation

Enables systematic variation of numerical values and clause complexity

Evaluates multiple state-of-the-art open and closed LLMs

Measures performance variance across different instantiations of the same question

Identifies significant performance drops when irrelevant clauses are added

Pros & Cons

Pros

Provides more reliable metrics than static GSM8K evaluations
Reveals critical limitations in LLM reasoning through controlled experiments
Demonstrates performance drops of up to 65% with irrelevant clauses, highlighting over-reliance on pattern matching
Open access paper with detailed methodology and findings

Cons

Does not provide a readily usable code repository or dataset download link on the page
Focuses only on grade-school-level math questions, not broader reasoning tasks
Findings may not generalize to all types of reasoning or model architectures

Best For

Assessing genuine logical reasoning capabilities of LLMsBenchmarking mathematical reasoning beyond static datasetsResearch on fragility and reliance on training data patterns in LLMs

FAQ

What does GSM-Symbolic benchmark evaluate?

It evaluates the mathematical reasoning capabilities of large language models using symbolic templates that allow controlled variations of numerical values and clause structure, based on the GSM8K dataset.

What were the key findings of the GSM-Symbolic study?

The study found that LLMs show significant performance variance when numerical values are changed, and their reasoning deteriorates sharply as the number of clauses increases. Adding an irrelevant clause can cause up to a 65% performance drop, suggesting LLMs replicate reasoning steps from training data rather than performing genuine logical reasoning.

Is the GSM-Symbolic benchmark code available?

The arXiv page does not provide a direct link to code or data. The paper describes the methodology; associated resources may be available from the authors upon request or through other channels.