PAPERS-01-007 · Paper — Generative Pretraining

The paper

Two OpenAI technical reports, one year apart, from Alec Radford and colleagues. The first — Improving Language Understanding by Generative Pre-Training (2018) — introduced the model now called GPT. The second — Language Models are Unsupervised Multitask Learners (2019) — introduced GPT-2, a 1.5-billion-parameter model trained on a web corpus the authors called WebText. Neither was published at a conference or posted to arXiv; both circulated as OpenAI reports, which is why you cite the reports themselves.

Before this paper

By 2018 the recipe for using a neural network on a language task was task-specific: gather a labeled dataset for sentiment, or entailment, or question answering, and train a bespoke model with a custom architecture on top. Pretraining existed, but mostly as word vectors or shallow features that a supervised model then consumed. BERT, arriving the same year, pushed pretraining further with a masked, bidirectional objective — but it was still fundamentally a fine-tune-per-task system. The open gap: could one general model, trained once on raw text, simply do many tasks without a labeled dataset and a gradient step for each?

The key idea

Train a Transformer decoder on the single most boring objective imaginable — predict the next token — over a very large pile of ordinary text, and let capability fall out of scale. There is no task label, no supervision beyond "what word comes next." The bet is that to predict the next word well across the whole internet, a model is forced to learn grammar, facts, translation, and reasoning as a side effect. Generative pretraining says the task is not something you bolt on; it is latent in the text itself.

How it works

GPT is a stack of Transformer decoder blocks with masked self-attention, so each position can attend only to tokens before it — the model never peeks at the future it is trying to predict. You feed it text, it outputs a probability distribution over the next token, and you train it to raise the probability of the token that actually came next, across billions of tokens. In the 2018 paper you then fine-tune this pretrained model on each downstream task with a small task-specific head. The 2019 leap was to drop fine-tuning: GPT-2 was prompted in plain language — a passage followed by "TL;DR:" for summarization, or an English-French pattern for translation — and it performed the task zero-shot, with no gradient update at all.

See it — next-token pretraining, then zero-shot prompting

flowchart LR
  A["Raw web text<br/>(WebText)"] --> B["Transformer decoder<br/>masked self-attention"]
  B --> C["Predict next token<br/>(the only objective)"]
  C --> D["Pretrained GPT / GPT-2"]
  D --> E["Prompt in plain language<br/>no fine-tuning"]
  E --> F["Zero-shot task<br/>summarize / translate"]

What they showed

The 2018 GPT paper showed that one pretrained decoder, fine-tuned, set a new state of the art across a spread of natural-language-understanding benchmarks — a single architecture beating an assortment of task-specific models. The 2019 GPT-2 result was the more startling one: at 1.5 billion parameters, trained only to predict the next token on WebText, the model performed reading comprehension, summarization, translation, and question answering when merely prompted, with no fine-tuning. Scale alone had turned a next-token predictor into a general-purpose task performer.

What it changed

Generative pretraining reframed the field. The unit of progress stopped being "a clever architecture for task X" and became "a bigger next-token model prompted for task X." Fine-tuning went from mandatory to optional, and the prompt became the interface. GPT-2 also seeded the whole practice of steering a model in natural language rather than with labeled data. And it raised the obvious next question: if zero-shot prompting works at all, does giving the model a few examples inside the prompt work even better? Next: Few-Shot Learners (GPT-3).

The one-line takeaway

Predict the next token at scale, and the tasks come for free.

Read it / meet it

There is no arXiv id — cite the OpenAI reports: Improving Language Understanding by Generative Pre-Training (2018) and Language Models are Unsupervised Multitask Learners (2019). The idea lives today in every decoder-only LLM you use: the next-token objective is still the entire pretraining recipe, and prompting — zero-shot and few-shot — is still how you ask the model to do a task.

Named concepts: generative pretraining, next-token prediction, zero-shot learning, GPT