PAPERS-14-006 · Paper — Flamingo — a Visual Language Model for Few-Shot Learning

The paper

Flamingo — a Visual Language Model for Few-Shot Learning is a 2021 paper by DeepMind researchers, presented at NeurIPS. It is considered a landmark because it showed how to get multimodal models to learn new tasks from just a few examples, much like large language models do with text.

Before this paper

Before Flamingo, vision‑language models typically required heavy fine‑tuning for each new task. Researchers could not simply prompt an image‑and‑text model with a few examples the way they could with pure language models. This made it hard to adapt models quickly to new visual tasks. The field lacked a way to do in‑context learning for multimodal data.

The key idea

The core insight of Flamingo is to connect a frozen vision encoder with a frozen language model using a small set of trainable adapter layers. These adapters let the language model attend to visual features while keeping the large pretrained components unchanged. By doing so, the system can ingest interleaved images, video, and text and perform new tasks from just a few prompt examples. This bridges the gap between powerful static models and flexible few‑shot learning.

How it works

First, a pretrained vision encoder processes each image or video frame and produces visual embeddings. A resampler then compresses those embeddings into a fixed number of visual tokens. Next, new gated cross‑attention layers are inserted into the frozen language model; these layers allow the language model to attend to the visual tokens while its original weights stay frozen. The whole bridge (cross‑attention and resampler) is trained on large collections of interleaved image‑text data. At inference time, a user provides a prompt that mixes a few image‑answer pairs with a new query, and the model generates answers without any weight updates.

See it — bolting vision onto a frozen LLM

flowchart TD
  A["Frozen vision<br/>encoder"] --> C["Gated cross-<br/>attention layers"]
  B["Frozen language<br/>model"] --> C
  C --> D["Few-shot answers<br/>about images+video"]

What they showed

Flamingo set new few‑shot state‑of‑the‑art performance across many vision‑language benchmarks, sometimes beating models that had been fine‑tuned on far more task data.

What it changed

The paper popularized the recipe of freezing large pretrained models and training only a small bridging component. It introduced in‑context few‑shot learning to multimodal AI, influencing the design of today’s multimodal assistants. Researchers now routinely build adapters on top of frozen vision and language backbones, a direct legacy of Flamingo. Next: BLIP-2.

The one-line takeaway

Flamingo proved that a tiny trainable bridge can give frozen vision‑language models powerful few‑shot abilities.

Read it / meet it

Reading the Flamingo paper reveals how the adapter‑bridge design works and why it matters for modern AI. Its ideas live on in today’s multimodal assistants, so exploring the paper will help you understand the foundations of current systems.