PAPERS-09-004 · Paper — Dropout

The paper

The 2014 paper "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" by Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov, presented at NIPS, is a landmark because it introduced a surprisingly simple yet powerful regularization technique that reshaped how deep networks are trained.

Before this paper

Before dropout, large neural networks tended to overfit dramatically. Their units would co‑adapt, memorising noise in the training data instead of learning robust, reusable features. This made it hard to scale models to bigger datasets and more complex tasks. Researchers were searching for a practical way to curb this over‑reliance on specific neurons.

The key idea

The core insight was to randomly drop, i.e., set to zero, a fraction of the hidden units on each training step. By doing so, the network can’t depend on any single unit and must spread useful information across many neurons. This simple stochastic trick effectively trains an enormous ensemble of thinned networks without the computational cost of training each one separately.

How it works

During each forward pass, every hidden unit is kept with a fixed probability (e.g., 0.5) and otherwise set to zero. The same random mask is applied to the backward pass, so only the kept units receive gradient updates. Over many steps, each unit learns to be useful on its own because it cannot rely on a fixed set of partners. At test time, all units are active, but their outputs are scaled (usually multiplied by the keep probability) to match the expected activation during training. This scaling ensures the network’s predictions remain consistent with the averaged ensemble of thinned models.

See it — randomly drop units

flowchart LR
  A["Each training step"] --> B["Randomly zero<br/>half the units"]
  B --> C["No co-adaptation"]
  C --> D["Like averaging<br/>many networks"]

What they showed

The authors showed that dropout substantially reduced overfitting and improved generalization across vision, speech, and other tasks, demonstrating clear performance gains over baseline models without dropout.

What it changed

Dropout quickly became one of the most widely used regularizers in deep learning, influencing the design of modern architectures and training pipelines. Its success showed that stochastic regularization could be both simple to implement and highly effective, encouraging further research into dropout variants and related techniques. The community adopted dropout layers in dense and attention blocks of most large networks, making it a standard component in today’s models. Next: He Initialization.

The one-line takeaway

Dropout teaches networks to be robust by forcing every unit to stand on its own.

Read it / meet it

Reading the original paper gives insight into the elegant simplicity behind a technique that still powers today’s models. You’ll see how the authors framed the problem and validated their solution, and you’ll recognize dropout’s legacy in the regularization layers that appear in virtually every modern deep‑learning library.