PAPERS-14-004 · Paper — ALIGN — Scaling Vision-Language with Noisy Data

The paper

ALIGN — Scaling Vision-Language with Noisy Data, by Cheng et al., 2021, presented at ICML, is a landmark because it showed that massive, loosely‑filtered web data can rival carefully curated datasets for vision‑language learning.

Before this paper

Before ALIGN, building image‑text datasets required painstaking curation and cleaning, which limited their size. Researchers believed high‑quality, manually filtered data were essential for strong performance. It was unclear whether raw, noisy web data could replace these clean collections. This uncertainty created a wall around scaling multimodal models.

The key idea

The core insight of ALIGN is that sheer scale can outweigh the need for meticulous data cleaning. By training on more than a billion image‑alt‑text pairs with only light filtering, the model learns robust image‑text representations. The noise in the data is compensated by its massive volume, allowing the system to match or exceed the quality of models trained on smaller, cleaner sets.

How it works

First, the team scraped over a billion image‑alt‑text pairs from the public web, applying only minimal filtering to remove obvious junk. Next, they built a dual‑encoder architecture: a convolutional image encoder and a transformer text encoder, each producing a vector embedding. During training, a contrastive loss pulled the embeddings of matching image‑text pairs together while pushing apart mismatched pairs. This objective was applied across the entire billion‑pair dataset, iterating many epochs. The large‑scale training produced a shared embedding space that works well for image‑text retrieval and zero‑shot classification.

See it — scale beats cleaning

flowchart LR
  A["1B+ noisy<br/>alt-text pairs"] --> B["Minimal<br/>filtering"]
  B --> C["Contrastive<br/>dual encoders"]
  C --> D["Strong image-text<br/>representations"]

What they showed

ALIGN achieved state‑of‑the‑art image‑text retrieval results and matched or beat more carefully curated systems, demonstrating that noisy, web‑scale data can produce representations as strong as those from high‑quality datasets.

What it changed

The paper validated the "scale over cleaning" recipe for multimodal pretraining, shifting the community’s focus toward gathering massive, lightly‑filtered data rather than exhaustive curation. This insight reshaped how later foundation models source and process their training data, leading to the web‑scale pipelines used today. By proving that noise can be tolerated at scale, ALIGN opened the door to ever larger multimodal models. Next: DALL-E.

The one-line takeaway

Scale can replace careful cleaning in vision‑language pretraining.

Read it / meet it

Reading ALIGN reveals how a simple data‑scaling strategy can transform multimodal learning, and its legacy lives on in today’s large‑scale image‑text models that rely on web‑derived data pipelines.