Conference Paper
Large Language Models
Featured

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Jason Wei(Google Research, Brain Team), Xuezhi Wang(Google Research, Brain Team), Dale Schuurmans(Google Research, Brain Team), Maarten Bosma(Google Research, Brain Team), Brian Ichter(Google Research, Brain Team), Fei Xia(Google Research, Brain Team), Ed Chi(Google Research, Brain Team), Quoc V. Le(Google Research, Brain Team), Denny Zhou(Google Research, Brain Team)
January 28, 2022NeurIPS 20226,000 citations
arXivPDFDOI

6.0k

Citations

900

Influential Citations

NeurIPS 2022

Venue

2022

Year

Abstract

We explore how generating a chain of thought — a series of intermediate reasoning steps — significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain-of-thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain-of-thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a PaLM 540B with just eight chain-of-thought exemplars achieves state-of-the-art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

Analysis

Why This Paper Matters

Chain-of-Thought (CoT) prompting is one of the most influential prompting techniques ever discovered. By simply including step-by-step reasoning examples in the prompt, Wei et al. showed that large language models can dramatically improve their performance on tasks requiring multi-step reasoning — without any changes to the model itself.

Technical Contributions

  • Chain-of-thought prompting: A simple technique where few-shot exemplars include intermediate reasoning steps, not just input-output pairs.
  • Emergent reasoning: Demonstrated that CoT reasoning is an emergent ability — it only appears in sufficiently large models (roughly 100B+ parameters). Smaller models show flat or even degraded performance with CoT.
  • No model modification needed: The technique works purely through prompting, requiring no fine-tuning, no architectural changes, and no additional training data.

Results

Across arithmetic, commonsense, and symbolic reasoning benchmarks:

  • GSM8K (math word problems): PaLM 540B with CoT achieved 56.9% → 74.4% accuracy, surpassing fine-tuned GPT-3 with a verifier
  • MultiArith: 72.2% → 99.3% accuracy
  • SVAMP: 79.0% → 86.7% accuracy
  • Performance gains were consistent across PaLM 540B, LaMDA 137B, and GPT-3 175B

Legacy

CoT prompting became a standard technique used in virtually every LLM application involving reasoning. It inspired a family of follow-up work including self-consistency, tree-of-thought, and step-by-step verification methods. The insight that prompting strategy matters as much as model scale changed how practitioners interact with LLMs.

chain-of-thoughtpromptingreasoningfew-shotemergent-abilitiesmathgoogle
Chain Of ThoughtPromptingReasoningLarge Language Models

Related Papers