PAPERS-01-012 · Paper — Think Step by Step

The paper

"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models," by Jason Wei and colleagues at Google, 2022, published at NeurIPS (arXiv 2201.11903). It made a small, almost trivial change to how you write a prompt — and showed that it unlocked multi-step reasoning in large language models that plain prompting left buried. The result was influential precisely because it needed no new model, no fine-tuning, and no extra data: just a better way of asking.

Before this paper

By 2022, scaling had made language models remarkably fluent, but fluency is not reasoning. Ask a big model a grade-school math word problem, or any task that needs several steps chained together, and it would leap straight to a final answer — often the wrong one. Making the model bigger helped with knowledge and language, yet multi-step arithmetic, commonsense, and symbolic reasoning stayed stubbornly unreliable. The standard remedy was to fine-tune on large collections of worked-out solutions, which is expensive and demands a fresh labeled dataset for every new task.

The key idea

The insight is almost embarrassingly simple: in your few-shot exemplars, do not just show the question and the final answer — show the reasoning in between. Each example walks through the intermediate steps out loud, and that worked-out path is called a chain of thought. Prompted with a handful of examples that reason step by step, the model imitates the pattern and produces its own reasoning trace on your new question before committing to an answer.

How it works

You hand the model two or three exemplars, each of which states a problem, walks through the reasoning line by line, and only then gives the answer. At inference the model sees a new question and continues that pattern — it generates its own intermediate steps first. Because it writes the steps out, it can carry the arithmetic and the logic one piece at a time, rather than guessing the whole answer in a single shot. The reasoning was already latent in the weights; the prompt simply draws it out into the open where it can be executed.

See it — chain-of-thought unrolls the reasoning before the answer

flowchart LR
    A["Few-shot exemplars<br/>with reasoning"] --> B["Model imitates<br/>the pattern"]
    B --> C["Writes steps<br/>one at a time"]
    C --> D["Final answer"]

What they showed

The gains were large and, crucially, emergent — chain-of-thought barely helps small models and can even hurt them, but it kicks in sharply for very large models, around 100 billion parameters and up. On GSM8K, a benchmark of grade-school math word problems, PaLM 540B with chain-of-thought reached about 58% — a new state of the art at the time. The lift was not confined to arithmetic: chain-of-thought also improved commonsense and symbolic reasoning across a family of benchmarks, all with no fine-tuning and no additional training data.

What it changed

Chain-of-thought turned the prompt itself into a lever for reasoning and launched a wave of follow-ups — zero-shot "let's think step by step," self-consistency (sampling many reasoning paths and voting), and tree-of-thought (searching over branches of reasoning). It reframed how the field thinks about eliciting capability: sometimes the ability is already in the model and the job is to ask for it well. But reasoning has a hard ceiling — a model can only reason over what it already learned in training. Ask it about last week, or about your private documents, and step-by-step thinking cannot conjure facts it never saw. That gap points straight to the next paper: Retrieval-Augmented Generation (RAG) fetches the right documents first, then lets the model reason over them.

The one-line takeaway

Ask a large model to show its work, and its reasoning appears — at scale.

Read it / meet it

Read it on arXiv: https://arxiv.org/abs/2201.11903. You meet chain-of-thought every time you tell a model to "think step by step," in the reasoning traces of modern assistants, and in the self-consistency and tree-of-thought methods built on top of it — it is now a default tool for any task that needs more than one step.

Named concepts: chain-of-thought prompting, emergent abilities, in-context reasoning