PAPERS-04-008 · Paper — Latent Diffusion (Stable Diffusion)

The paper

Latent Diffusion Models (often called Stable Diffusion) was introduced by Rombach, Blattmann, Lorenz, Esser, and Ommer in 2022 at the CVPR conference. It is considered a landmark because it showed how to make high‑quality text‑to‑image generation affordable for everyday computers.

Before this paper

Before this work, diffusion models operated directly on pixel grids. Running the diffusion process on full‑resolution images was extremely slow and required huge amounts of compute, making training and sampling impractical for most researchers and users. This computational bottleneck limited the resolution and accessibility of generative image AI. As a result, only a few large labs could afford to experiment with diffusion at scale.

The key idea

The core insight is to move the diffusion process from pixel space into a compact latent space created by an autoencoder. By compressing images first, the diffusion steps work on a much smaller representation, cutting the computational cost dramatically while still preserving the essential visual information needed for generation.

How it works

First, an autoencoder is trained to encode images into a low‑dimensional latent representation and to decode latents back into full‑resolution pictures. Next, a diffusion model is trained to denoise and generate data directly in this latent space, learning how to transform random noise into a meaningful latent code. During inference, a random latent is iteratively refined by the diffusion model. Finally, the decoder of the autoencoder translates the refined latent back into a high‑quality image. Cross‑attention layers are added to the diffusion network so that a text prompt can be injected, guiding the generation toward the desired description.

See it — diffusion in latent space

flowchart LR
  T["Text prompt"] --> X["Diffusion in<br/>latent space"]
  N["Noise"] --> X
  X --> D["Decoder<br/>(VAE)"]
  D --> I["Image"]

What they showed

The authors demonstrated that latent diffusion matched the image quality of the best pixel‑based diffusion models while using only a fraction of the compute. This efficiency enabled the open release of Stable Diffusion, which could run on consumer‑grade hardware and still produce impressive results.

What it changed

The paper democratized generative image AI by making powerful text‑to‑image generation affordable and widely available. It sparked a massive ecosystem of open‑source tools, extensions, and creative applications built on Stable Diffusion and its descendants. Because the method is both fast and open, it quickly became the backbone of many modern AI art platforms. Next: the frontier — multimodal generation.

The one-line takeaway

Latent diffusion proved that moving generation into a compressed latent space can deliver high‑quality images with far less compute.

Read it / meet it

Reading the original Latent Diffusion paper gives insight into the clever combination of autoencoders and diffusion that powers today’s AI art tools. You can see its legacy everywhere, from the Stable Diffusion model itself to the countless apps and plugins that now let anyone create images from text.