PAPERS-02-007 · Paper — Attention Comes to Vision (ViT)

The paper

Attention Comes to Vision (ViT) by Dosovitskiy et al. (2020, ICLR 2021) introduced the first pure Transformer architecture for image classification, proving that language‑style self‑attention can replace convolutional backbones and become a new landmark in vision research.

Before this paper

Before ViT, image models relied almost exclusively on convolutional neural networks, which embed strong locality and translation bias but struggle to benefit from massive data without careful architectural tweaks. Scaling CNNs required ever deeper stacks and hand‑crafted tricks to improve receptive fields. As a result, the community lacked a unified, data‑efficient architecture that could seamlessly leverage the same self‑attention mechanisms that had transformed natural‑language processing.

The key idea

ViT treats an image as a sequence of fixed‑size patches, typically 16x16 pixels, and linearly embeds each patch into a vector analogous to a word token. Positional embeddings are added to retain spatial order, and the resulting token stream is fed directly into a standard Transformer encoder. This design removes almost all vision‑specific components, relying on self‑attention to model global relationships.

How it works

First, the input image is divided into non‑overlapping 16x16 patches, each of which is flattened and projected with a learned linear layer to form patch embeddings. A special class token is prepended to the sequence, and learnable positional embeddings are added to every token. The full sequence passes through multiple Transformer encoder layers, each consisting of multi‑head self‑attention followed by a feed‑forward network with residual connections and layer normalization. After the final encoder block, the representation of the class token is taken through a classification head to produce the image‑level logits.

See it — an image as a sequence of patches

A diagram of ViT shows the patch extraction, embedding, positional encoding, and the stacked Transformer blocks leading to the classification head.

flowchart LR
  A["image"] --> B["split into<br/>16x16 patches"]
  B --> C["linear embed<br/>+ position"]
  C --> D["Transformer encoder<br/>self-attention"]
  D --> E["class prediction"]

What they showed

When pretrained on massive datasets such as ImageNet‑21k or the 300M‑image JFT dataset, the largest ViT-H/14 model achieved about 88.55% top‑1 accuracy on ImageNet, matching or surpassing the best CNNs at that scale. In contrast, with only modest data ViT underperforms CNNs because it lacks convolution's built‑in locality bias.

What it changed

ViT demonstrated that a pure Transformer can dominate vision tasks at scale, unifying the architectures for language and image domains and showing cheaper training costs compared to state‑of‑the‑art CNNs. Its success sparked a wave of transformer‑based vision models, from hybrid designs to fully vision‑only variants, and reshaped research priorities toward data‑centric scaling. The approach also encouraged the integration of vision Transformers into production pipelines and open‑source libraries.

Next: Vision Meets Language (CLIP).

The one-line takeaway

Vision finally learned to speak the language of self‑attention, turning every patch into a word and rewriting the story of image recognition.

Read it / meet it

Read the original ViT paper for the full experimental details, then explore the many follow‑up surveys and tutorials that break down the architecture. Today you can find pretrained ViT models in libraries like Hugging Face Transformers and timm, and they are used in everything from image classification APIs to multimodal research.