Interactive Lab · Deep Networks & ResNets

The gradient that vanishes, and the one line that saves it

To train a network, the error signal has to travel backward through every layer. In a deep plain network it fades a little at each step, and after 20 layers it's a rounding error, so the early layers never learn. That "vanishing gradient" is why deep nets were untrainable for years. Then came one absurdly simple fix: add the input back to the output of each block (out = x + F(x)). Flip the switch and watch the dead signal come back to life.

▶ The playground, watch the gradient flow backward

depth 20 first-layer gradient vs last-layer status vanished

healthy gradient vanished (< 10⁻⁴) each bar = the gradient's strength as it reaches that layer (log scale) · left = first layer

Skip connections (residual)OFF

Add the block's input back to its output: out = x + F(x). The "+x" gives the gradient a shortcut straight to the early layers, an uninterrupted highway that never fades. Click to toggle.

Depth (layers) 20

4 · shallowvery deep · 40

Width 16

832

Activation

sigmoid tanh ReLU

🧭 Where this fits

Residual (skip) connections are why we can train networks hundreds of layers deep without gradients vanishing. They unlocked modern computer vision and are baked into every transformer and LLM block. When a deep net won't train, the first question is: does it have residual connections?

The code that's actually running

🔮 Predict, then test

With skips OFF, switch the activation to ReLU and then sigmoid, watching the first-layer bar. Sigmoid's slope is at most 0.25, so the gradient shrinks by ≥4× every layer, it's dead within a dozen layers. ReLU's slope is 1 for positive inputs, so it fades far more slowly (that's why ReLU alone bought a few more layers of depth). But crank the depth up and even ReLU eventually starves, until you turn skips on, and the highway keeps every layer fed no matter how deep you go.

The takeaway

Your new mental model of ResNets

Backprop multiplies. The gradient at layer 1 is the product of the slopes of every layer above it. Many numbers below 1 multiplied together race to zero, the early layers get no signal.
Depth used to hurt. Past a point, adding layers made networks worse, because the new layers couldn't be trained. The problem wasn't capacity, it was gradient flow.
The skip is an identity highway. out = x + F(x) means the gradient has a path with slope exactly 1 all the way down. It can't vanish, so even 100+ layers train.
One idea, everywhere. Residual connections are now in almost every deep model, including the transformer blocks inside every LLM.

How backprop multiplies →Neural Net Playground 📖 Lecture notes