PAPERS-01-008 · Paper — Few-Shot Learners

The paper

"Language Models are Few-Shot Learners," by Brown et al. at OpenAI, 2020, published at NeurIPS (arXiv 2005.14165). It introduced GPT-3, an autoregressive language model with 175 billion parameters — more than ten times larger than any dense model before it. The paper's claim was not a new architecture but a new behavior: at that scale, a plain next-token model starts to learn tasks from the prompt itself.

Before this paper

The dominant recipe after BERT and GPT-2 was pretrain then fine-tune: take a pretrained model and update its weights on a labeled dataset for each downstream task. This worked, but every new task needed its own curated dataset and its own training run, and the fine-tuned model was narrow — good at that one task, and prone to overfitting the quirks of its training set. Humans, by contrast, pick up a new task from a handful of examples or a short instruction. The open question was whether scale alone could close that gap without any task-specific gradient updates.

The key idea

The headline is in-context learning: you specify the task inside the prompt — a natural-language description and/or a few worked examples — and the frozen model completes the pattern, with no gradient step and no weight update. The paper studies three settings on a spectrum: zero-shot (instruction only), one-shot (one example), and few-shot (a handful, up to the context window). The central empirical finding is that the benefit of showing examples grows with model scale — the ability emerges as the model gets bigger.

How it works

GPT-3 is a decoder-only Transformer trained on one objective: predict the next token over a large web-scale corpus. At inference you build a prompt that concatenates the task description and K example pairs, then append the real query; the model conditions on all of it and generates the answer as a continuation. Nothing about the weights changes — the "learning" is the model recognizing and extending the pattern within its context window. Because it is all forward passes, a single trained model handles hundreds of tasks by prompt alone.

See it — in-context learning: examples in the prompt, no gradient step

flowchart LR
  A["Task description<br/>+ K examples"] --> B["Append the<br/>real query"]
  B --> C["Frozen GPT-3<br/>175B params"]
  C --> D["Generate the<br/>answer as text"]
  C -. "no weight update" .-> C

What they showed

Few-shot GPT-3 was strong across many benchmarks — translation, question answering, cloze and completion tasks, and more — often approaching or matching fine-tuned models without a single gradient update. The paper also documents a clear trend: zero-, one-, and few-shot performance all improve as the model gets larger, and the gap that few-shot examples buy widens with scale. Crucially, the authors are honest about the limits: GPT-3 struggles on some reasoning and on multi-digit arithmetic, can be sensitive to prompt wording, and inherits biases from its training data. The result is a capability curve, not a solved problem.

What it changed

GPT-3 reframed the interface to a model: instead of training for a task, you prompt for it. This launched prompting as a discipline, made a single general model the default product surface, and set up the alignment and reasoning work that followed — InstructGPT to make the model follow instructions, chain-of-thought to coax out the reasoning it lacked here. But it also raised the obvious next question: scale clearly helps — but how much, and how predictably? Next: Scaling Laws.

The one-line takeaway

At 175B parameters, a next-token model learns tasks from examples in its prompt — no fine-tuning required.

Read it / meet it

Read it at https://arxiv.org/abs/2005.14165. The idea lives today in every prompt you write: zero- and few-shot prompting is how people use ChatGPT, Claude, and every hosted LLM, and "just put examples in the context" is now a first reflex before anyone reaches for fine-tuning.

Named concepts: in-context learning, few-shot prompting, GPT-3