PAPERS-04-003 · Paper — Variational Autoencoders (VAE)

The paper

The landmark paper that introduced Variational Autoencoders presented a new way to turn autoencoders into probabilistic generative models, establishing a cornerstone for modern deep generative research.

Before this paper

Before this work, autoencoders could compress and reconstruct images but their latent space was a jumble you could not sample from meaningfully. Researchers struggled to turn the compressed codes into a smooth, interpretable space. Without a principled way to generate new points, the models were limited to reconstruction only. This wall prevented using autoencoders as true generative tools.

The key idea

The key insight was to make the encoder output a probability distribution—a mean and a variance—over the latent space. By treating each code as a distribution, the latent space becomes smooth and continuous. This allows us to sample new points that correspond to realistic images. In short, the model learns a structured, generative latent space instead of a chaotic codebook.

How it works

First, the encoder processes an input image and predicts a mean vector and a variance vector for each latent dimension. Then, using the reparameterization trick, a random sample is drawn by adding the mean to the product of a standard normal noise and the standard deviation (the square‑root of the variance). This operation is differentiable, so gradients can flow through the sampling step during back‑propagation. The decoder then takes this sampled latent vector and tries to reconstruct the original image. The training loss combines a reconstruction term with a regularization term (the KL‑divergence) that keeps the latent distribution close to a standard normal, ensuring a well‑behaved space.

See it — a smooth latent space

flowchart LR
  X["Input image"] --> E["Encoder<br/>mean + variance"]
  E --> Z["Sample latent z"]
  Z --> D["Decoder<br/>reconstruct"]
  D --> Y["Output image"]

What they showed

The authors demonstrated that VAEs provide a principled, probabilistic generative model with a continuous latent space you could interpolate through and sample from. Their results showed smooth interpolations and realistic samples, confirming that the latent space was no longer a jumble but a usable generative manifold.

What it changed

This paper became a foundational generative model and its autoencoder‑into‑latent‑space idea reappears at the heart of latent diffusion. It shifted the field from deterministic reconstruction toward probabilistic generation, enabling downstream applications like representation learning and image synthesis. The approach opened the door to many later advances in deep generative modeling. Next: DCGAN.

The one-line takeaway

VAEs turned autoencoders into smooth, sampleable generative models by learning a distribution over latent codes.

Read it / meet it

Reading the original VAE paper reveals the elegant reasoning behind the reparameterization trick and the balance of reconstruction and regularization. Its legacy lives on in modern tools such as Stable Diffusion, where VAEs serve as the compressor inside the pipeline.