PAPERS-13-006 · Paper — Understanding Deep Learning Requires Rethinking Generalization

The paper

Understanding Deep Learning Requires Rethinking Generalization, by Chiyuan Zhang, Samy Bengio, Moritz Hardt, Michael C. Mozer and Yoram Singer (2017, ICLR), is a landmark because it exposed a fundamental gap in classical learning theory by showing that modern deep nets can memorize random data yet still generalize on real tasks.

Before this paper

Before this work, classical theory held that a model capable of fitting arbitrary noise must have too much capacity and therefore would generalize poorly. Practitioners observed deep networks generalizing well despite being highly over‑parameterized, but no precise explanation existed. The field lacked a concrete experiment that could challenge the capacity‑based intuition. This created a wall: theory could not account for the empirical success of deep learning.

The key idea

The key insight was a simple yet striking experiment: state‑of‑the‑art deep networks can be trained to achieve zero training error on data with completely random labels—pure noise—yet the same architectures still generalize well when trained on real labels. This shows that raw capacity and traditional uniform‑convergence bounds are insufficient to explain generalization. The result forces us to look beyond hypothesis‑class size toward the role of data and optimization.

How it works

The authors took standard image classification models and created several corrupted versions of the training set. In one version they shuffled the labels randomly; in another they even randomized the pixel values. They then trained the networks using the usual optimization procedures (SGD, with or without weight decay or dropout). Despite the noise, the training loss dropped to zero, indicating the networks could perfectly fit the random data. They also compared runs with explicit regularizers, finding that such tricks helped a little but were neither required nor enough for good test performance on real data. This step‑by‑step protocol demonstrated the networks’ effective capacity far exceeds the dataset size.

See it — nets can memorize noise

flowchart TD
  A["Big neural net"] --> B["Train on random<br/>labels"]
  B --> C["Fits them<br/>perfectly"]
  C --> D["So capacity alone<br/>cant explain why it<br/>generalizes on real data"]

What they showed

The experiments proved that deep networks can drive training error to zero on datasets with random labels or random pixels, confirming their effective capacity exceeds the size of the data. Explicit regularization like weight decay and dropout improved performance only modestly and were not necessary for the networks to fit the noise. The result directly contradicts the intuition that limited capacity alone prevents overfitting.

What it changed

The paper reshaped the central open question of deep learning theory, shifting focus from capacity‑based arguments to the interplay of data structure, optimization dynamics, and implicit bias. It sparked a wave of research into why over‑parameterized models still generalize, including studies of margin‑based bounds, neural tangent kernels, and the role of stochastic gradient descent. By exposing the inadequacy of classical uniform‑convergence bounds, it opened new directions for developing more realistic theoretical frameworks. Next: Double Descent.

The one-line takeaway

Over‑parameterized deep nets can memorize random data, so generalization must arise from factors beyond mere model capacity.

Read it / meet it

Reading the original ICLR 2017 paper gives a clear view of the experiments that started this line of inquiry. Today, its legacy lives in a vibrant literature on implicit regularization, double‑descent phenomena, and modern generalization theory, making it essential reading for anyone curious about why deep learning works.