To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
FreeChain-of-thought helps mainly on math and symbolic reasoning
FreeFree tier
About To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
This paper presents a quantitative meta-analysis of over 100 papers using Chain-of-Thought (CoT) prompting and evaluations on 20 datasets across 14 models. It finds that CoT provides significant performance benefits primarily on math and logic tasks, with minimal gains on other types. CoT's advantage comes from improving symbolic execution, but it underperforms compared to using a symbolic solver. The paper suggests using CoT selectively to maintain performance while saving inference costs and calls for new paradigms beyond prompt-based CoT.
Key Features
Quantitative meta-analysis over 100 papers using CoT
Evaluations on 20 datasets across 14 models
Identifies that CoT benefits primarily math and logic tasks
Shows CoT's gain comes from improving symbolic execution
Suggests selective CoT use to save inference costs
Published at ICLR 2025
Compares CoT against tool-augmented LLMs and symbolic solvers
Pros & Cons
Pros
- Provides extensive quantitative evidence from over 100 papers
- Clear findings on when CoT is beneficial (math/logic vs. other tasks)
- Actionable suggestions for selective CoT use to save inference costs
- Published in top-tier venue (ICLR 2025)
- Includes comparisons with symbolic solvers and tool-augmented LLMs
Cons
- Does not cover all possible task types or domains
- Results may change as LLMs evolve
- Focuses on existing prompt-based CoT; does not propose new methods
Best For
Research on LLM reasoning capabilitiesDeciding whether to apply CoT for specific tasksUnderstanding limitations of prompt-based reasoningGuidance for selective CoT deployment to reduce inference costs
FAQ
For what tasks is Chain-of-Thought most effective?
CoT is most effective for math and symbolic reasoning tasks. On other tasks, gains are much smaller.
Does CoT help on MMLU?
CoT provides minimal gains on MMLU unless the question or the model's response contains an equals sign, indicating symbolic operations.
Should CoT always be used?
No. The paper suggests CoT can be applied selectively, maintaining performance while saving inference costs. For many tasks, direct answer generation or using a symbolic solver may be better.
What is the main finding of the paper?
Chain-of-thought prompting primarily benefits math and logic tasks; its gains stem from improving symbolic execution, but it underperforms relative to dedicated symbolic solvers.