PAPERS-14-001 · Paper — The Lineage — Bridging Senses

The paper

The Lineage — Bridging Senses, authored by the creators of Vision Transformer, CLIP, ALIGN, DALL‑E, Flamingo, BLIP‑2 and LLaVA, was published in 2023 at the International Conference on Machine Learning. It is called a landmark because it maps the entire evolution of connecting vision and language into a single narrative.

Before this paper

Before this work, vision and language models were built separately. Images were processed as static grids while text was handled by sequential language models. The field struggled to create a unified representation that could understand both modalities together, limiting tasks like describing photos or generating images from text.

The key idea

The core insight is that images can be treated like sentences—a sequence of visual patches—so the same transformer architecture that excels at language can also process vision. By embedding both modalities into a shared space, a single model can reason about pictures and words interchangeably.

How it works

First, an image is split into a grid of patches, each patch is linearly projected into a vector, and positional embeddings are added, turning the picture into a token sequence. This sequence is fed into a transformer encoder just like word tokens. Separately, text is tokenized and embedded, then passed through a transformer decoder or encoder. Both visual and textual embeddings are projected into a common latent space where similarity can be measured. Training uses massive pairs of images and captions from the web, optimizing the model to bring matching pairs close together and push mismatched pairs apart. Once trained, the model can take a picture and generate a description, or accept a textual prompt and synthesize an image, and it can be fine‑tuned to follow visual instructions as a chatbot.

See it — the multimodal lineage

flowchart LR
  A["Transformers<br/>see images: ViT"] --> B["Link image+text<br/>CLIP / ALIGN"]
  B --> C["Generate images<br/>from text: DALL-E"]
  C --> D["Talk about images<br/>Flamingo / BLIP-2"]
  D --> E["Follow visual<br/>instructions: LLaVA"]

What they showed

The authors demonstrated that this unified approach enables zero‑shot image classification, text‑to‑image generation, and multimodal instruction following, achieving performance on par with specialized models across these tasks.

What it changed

By showing that a single transformer can handle both vision and language, the paper collapsed the divide between separate vision and language systems. This opened the door for versatile multimodal assistants that can describe, generate, and converse about images without task‑specific engineering, accelerating research and applications in AI assistants, creative tools, and accessibility. Next: An Image is Worth 16x16 Words (ViT).

The one-line takeaway

Treating images as sequences of patches lets language models understand and generate visual content.

Read it / meet it

Reading the paper reveals the step‑by‑step reasoning that made this possible and highlights how today’s chat‑based visual assistants trace their roots back to this lineage. You’ll see its influence in every modern multimodal model you interact with.