PAPERS-14-007 · Paper — BLIP-2 — Bootstrapping Vision-Language with Frozen Models

The paper

BLIP-2—Bootstrapping Vision-Language with Frozen Models (2023) by Li et al., presented at CVPR, is a landmark because it showed how to build powerful vision‑language systems without retraining huge vision and language backbones.

Before this paper

Before BLIP-2, training multimodal models end‑to‑end was prohibitively expensive because it required updating massive vision and language backbones together. Researchers tried reusing frozen backbones, but they struggled to connect two systems that speak different representations, creating a costly wall for scaling.

The key idea

The core insight of BLIP-2 is to insert a tiny, trainable bridge—the Q‑Former—between a frozen image encoder and a frozen large language model. This bridge translates visual features into a form the language model can understand, letting the big models stay frozen while only a small set of parameters are learned.

How it works

First, a frozen image encoder extracts visual features from an input image. Next, a lightweight Transformer called the Q‑Former, equipped with learnable query tokens, attends to those visual features and compresses them into a few query outputs. These query outputs are then fed directly into the frozen language model as its input tokens. Training occurs in two stages: the Q‑Former first learns to align vision and language representations, then it learns to produce outputs that the frozen LLM can use for generation. Because both the encoder and the LLM remain frozen, only the Q‑Former’s parameters are updated, drastically reducing training cost.

See it — a lightweight query bridge

flowchart LR
  A["Frozen image<br/>encoder"] --> B["Q-Former<br/>query tokens"]
  B --> C["Frozen LLM"]
  C --> D["Captions + VQA<br/>cheaply trained"]

What they showed

BLIP-2 achieved competitive or state‑of‑the‑art results on image captioning and visual question answering while training far fewer parameters than end‑to‑end approaches.

What it changed

The paper introduced the query‑bridge design, which quickly became a standard, efficient blueprint for turning any strong LLM into a vision‑language model. This shift allowed many subsequent open‑source multimodal projects to adopt a Q‑Former‑style connector, lowering the barrier to building high‑performing multimodal systems. Next: LLaVA.

The one-line takeaway

A tiny trainable bridge can unlock the power of frozen vision and language models.

Read it / meet it

Reading BLIP‑2 is rewarding because it explains a simple yet powerful recipe that many modern multimodal models now follow. You can see its legacy in numerous open‑source projects that use Q‑Former‑style connectors to combine vision encoders with large language models.