PAPERS-14-002 · Paper — An Image is Worth 16x16 Words (Vision Transformer)

The paper

The paper “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” by Alexey Dosovitskiy et al., published in 2020 at ICLR, is a landmark because it showed that a pure Vision Transformer can rival or beat convolutional networks on image classification.

Before this paper

Before this work, computer vision was dominated by convolutional neural networks, which relied on built‑in assumptions about locality and translation invariance. Those inductive biases were thought essential for good performance. Meanwhile, transformers had transformed natural‑language processing but had not displaced CNNs for images. The field lacked a simple, unified architecture that could handle vision without heavy vision‑specific engineering.

The key idea

The key insight is that an image can be treated like a sentence: split it into small patches, embed each patch as a token, add positional information, and feed the resulting sequence to a standard transformer encoder. By doing so, the model needs almost no vision‑specific machinery yet can match or exceed CNNs when trained on enough data.

How it works

First, the input image is divided into fixed‑size patches, for example 16 × 16 pixels. Each patch is flattened and linearly projected into a token embedding. A learned position embedding is added to each token so the model knows the spatial order. A special classification token is prepended to the sequence. The whole token sequence passes through a standard transformer encoder consisting of multi‑head self‑attention and feed‑forward layers. Finally, the output corresponding to the classification token is fed to a small classification head that produces the image label.

See it — an image as a sentence of patches

flowchart LR
  A["Image"] --> B["Cut into<br/>16x16 patches"]
  B --> C["Embed each<br/>patch as a token"]
  C --> D["Standard<br/>Transformer"]
  D --> E["Classify"]

What they showed

The authors demonstrated that Vision Transformers matched or exceeded top convolutional networks on image classification when pretrained on very large datasets, while using far less vision‑specific inductive bias.

What it changed

This work unified vision and language under a single architecture, making the transformer the common backbone for many later multimodal models. It showed that the transformer could serve as a general‑purpose image encoder, reducing the need for handcrafted vision components. As a result, nearly every modern vision‑language model now uses a ViT‑style image encoder. Next: CLIP.

The one-line takeaway

A pure transformer, with minimal vision‑specific tricks, can rival CNNs if given enough data.

Read it / meet it

Reading the original ViT paper gives a clear view of how simple patch embeddings and transformers can replace convolutions. Its legacy lives on in today’s vision‑language systems, which almost universally adopt a ViT‑style image encoder.