PAPERS-14-005 · Paper — DALL·E — Zero-Shot Text-to-Image Generation

The paper

The 2021 paper "DALL·E — Zero-Shot Text-to-Image Generation" by researchers at OpenAI, presented at the arXiv pre‑print server, is a landmark because it showed that a single model could turn any text description into a plausible image, opening a new era of text‑to‑image generation.

Before this paper

Before DALL·E, generating images from arbitrary text was largely out of reach. Existing systems were narrow, domain‑specific, and struggled when asked to combine novel concepts. They could not handle the wide variety of everyday language or absurd combinations. The field faced a wall of limited generalization and creativity.

The key idea

The key insight was to treat image creation as a language problem. DALL·E represents an image as a sequence of discrete visual tokens, just like words in a sentence. A large Transformer is then trained to read text tokens and continue the sequence with image tokens, enabling it to generate pictures from captions, even for absurd combinations it never saw.

How it works

First, a discrete autoencoder compresses each image into a grid of visual tokens drawn from a fixed codebook. Next, a massive autoregressive Transformer is trained on a combined stream of text tokens followed by the corresponding image tokens, learning to predict the next token in the sequence. During generation, the model receives a new textual prompt, samples image tokens one by one conditioned on the text, and finally the decoder of the autoencoder converts those tokens back into pixel values. This step‑by‑step token prediction lets the system produce a complete image that matches the caption.

See it — text tokens to image tokens

flowchart LR
  A["Text prompt"] --> B["Text tokens"]
  B --> C["Predict image<br/>tokens one by one"]
  C --> D["Decode tokens<br/>to pixels"]
  D --> E["Novel image"]

What they showed

The authors demonstrated that DALL·E could produce coherent, creative images for a wide range of prompts, including novel concept combinations, without any task‑specific training. It showed surprising zero‑shot ability to combine concepts such as "an armchair in the shape of an avocado."

What it changed

DALL·E launched the modern text‑to‑image era and proved that generative modeling of images could be framed as sequence modeling. By showing that a single Transformer could handle both language and vision tokens, it shifted research toward unified, large‑scale models. This breakthrough inspired a wave of subsequent text‑to‑image systems that build on the same principle. Next: Flamingo.

The one-line takeaway

Treating images as sequences of discrete tokens lets a language model generate pictures from any caption.

Read it / meet it

Reading the original DALL·E paper gives insight into how the token‑based approach works and why it matters. Today, every text‑to‑image product you encounter traces its lineage back to the path DALL·E opened.