Self-Consistency (2022)
FreeMulti-path sampling + majority vote: GSM8K 57% → 74%
FreeFree tier
About Self-Consistency (2022)
Self-consistency is a decoding strategy proposed for chain-of-thought prompting in large language models. Instead of using greedy decoding, it samples multiple diverse reasoning paths and then selects the most consistent answer by marginalizing over the sampled paths. This approach is based on the intuition that complex reasoning problems often have multiple valid reasoning paths leading to the same correct answer. Empirical evaluations show significant improvements across arithmetic and commonsense reasoning benchmarks, including GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%), and ARC-challenge (+3.9%). The paper was published at ICLR 2023.
Key Features
Samples multiple diverse reasoning paths instead of greedy decoding
Selects the most consistent answer via majority vote (marginalizing over paths)
Compatible with chain-of-thought prompting
Boosts performance on arithmetic and commonsense reasoning benchmarks
Pros & Cons
Pros
- Significantly improves chain-of-thought prompting accuracy with a striking margin
- Leverages the intuition that multiple reasoning paths lead to unique correct answers
- Simple to implement as a post-processing step over sampled paths
Cons
- Increased computational cost due to sampling multiple reasoning paths instead of greedy decoding
Best For
Complex reasoning tasks requiring multi-step thought processesArithmetic reasoning (e.g., GSM8K, SVAMP, AQuA)Commonsense reasoning (e.g., StrategyQA, ARC-challenge)
FAQ
What is self-consistency in language model reasoning?
Self-consistency is a decoding strategy that samples multiple diverse reasoning paths from a language model and then selects the most consistent answer by marginalizing over the sampled paths, rather than relying on a single greedy path.
How does self-consistency improve chain-of-thought prompting?
It replaces greedy decoding with sampling multiple reasoning paths and uses majority vote to choose the final answer. This exploits the idea that correct reasoning often admits multiple valid paths.
On which benchmarks was self-consistency evaluated?
It was tested on GSM8K (+17.9%), SVAMP (+11.0%), AQuA (+12.2%), StrategyQA (+6.4%), and ARC-challenge (+3.9%) among others.