Interactive Lab · Training Deep Networks

The activations that explode, and the trick that tames them

Push a batch of numbers through a deep network and something quietly goes wrong: each layer rescales the activations a little, and over many layers those little changes compound. Soon the numbers are either exploding into the thousands or collapsing to zero, and a network whose activations look like that can't learn. Batch normalization is a dead-simple fix: after each layer, re-center and re-scale the batch to mean 0, std 1. Flip it on and watch every layer snap back to health.

▶ The playground, watch the activations flow through depth

depth 12 last-layer std status exploding

healthy (std ≈ 0.1–3) exploded / vanished each bar = the spread (std) of that layer's activations, log scale

Batch normalizationOFF

After each layer, subtract the batch mean and divide by the batch std, forcing that layer's activations back to a clean mean-0, std-1 distribution before they flow on. Click to toggle.

Depth (layers) 12

3 · shallowdeep · 24

Weight scale (init) 2.0

0.6 · vanishexplode · 3.0

Activation

ReLU tanh

🧭 Where this fits

Normalization (BatchNorm/LayerNorm) is what keeps deep networks trainable, it stops activations from exploding or vanishing across layers. LayerNorm sits in every transformer block. When training is unstable or the loss goes NaN, normalization is one of the first things to check.

The code that's actually running

🔮 Predict, then test

With BatchNorm off, slide the weight scale from 3.0 down to 0.6. At the top, the bars rocket off the chart (activations in the hundreds); at the bottom, they shrink to nothing (dead units). There's only a razor-thin "just right" setting in between, which is exactly why deep nets were so fiddly to initialize. Now turn BatchNorm on and sweep the same slider: the bars barely move. BatchNorm makes the network robust to the init, you can be sloppy about weights and use a bigger learning rate, and it still trains. That robustness, more than any single trick, is why it made very deep networks practical.

The takeaway

Your new mental model of BatchNorm

Depth compounds small errors. Each layer multiplies the scale of its inputs a little; over many layers that factor becomes huge or tiny, so activations explode or vanish.
BatchNorm resets the scale every layer. Re-center to 0, re-scale to 1 across the batch, so no matter what the layer did, the next layer receives a clean, standard distribution.
It buys robustness. Networks tolerate bad initialization and larger learning rates, train faster, and go deeper, the practical unlock behind modern deep learning.
Cousins everywhere. LayerNorm (used in every transformer), GroupNorm, RMSNorm, all the same idea: keep activations in a sane range so gradients stay healthy.

The gradient version: ResNet skips →Neural Net Playground 📖 Lecture notes