PAPERS-09-005 · Paper — He Initialization (Delving Deep into Rectifiers)

The paper

He et al. (2015) introduced the landmark paper “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification” at CVPR, showing why it matters as a turning point for deep learning with ReLUs.

Before this paper

Before this work, deep networks were hard to train because signals shrank or exploded across layers when weights were initialized poorly. The earlier Xavier scheme did not account for the ReLU nonlinearity, leading to unstable activations and gradients. As a result, very deep rectifier networks often failed to converge. Researchers struggled to scale depth without careful tricks.

The key idea

The key insight was to tune the weight initialization variance to the ReLU activation itself, so that the scale of the signal stays stable through many layers. By matching the variance to the rectifier’s behavior, both forward activations and backward gradients keep roughly the same magnitude. This simple change removes the exploding/vanishing problem for deep ReLU nets.

How it works

He initialization sets the variance of each layer’s weights to 2 divided by the number of inputs to that layer. Concretely, for a layer with n input units, weights are drawn from a zero‑mean distribution with variance 2/n (often a Gaussian or uniform scaled accordingly). This scaling ensures that the expected variance of the outputs after the ReLU remains close to the input variance. The paper also introduced the parametric ReLU (PReLU) as a flexible activation, but the core mechanism is the variance‑scaled weight draw. Implementations simply replace the Xavier 1/n rule with 2/n for ReLU layers.

See it — start with the right scale

flowchart LR
  A["Deep ReLU net"] --> B["Init variance = 2/n"]
  B --> C["Signal variance<br/>stays stable"]
  C --> D["Very deep nets<br/>train"]

What they showed

The authors demonstrated that with this proper initialization, very deep rectifier networks trained reliably and achieved performance that surpassed human‑level accuracy on the ImageNet classification benchmark.

What it changed

After the paper, variance‑scaled initialization became the default for deep networks, replacing older schemes. It spread quickly into every major deep‑learning framework, making training of deep ReLU models routine. This shift enabled researchers to build much deeper architectures without custom tricks, accelerating progress across vision and beyond. Next: Batch Normalization.

The one-line takeaway

Match the weight variance to ReLU (2 / fan‑in) and deep nets train smoothly.

Read it / meet it

Reading the original CVPR paper gives insight into why such a simple formula transformed deep learning. Today you’ll see He/Xavier init as the default in all major libraries, a living legacy of this work.