PAPERS-02-006 · Paper — Making Depth Free (ResNet)

The paper

"Deep Residual Learning for Image Recognition" — He, Zhang, Ren, and Sun at Microsoft Research, 2015 (arXiv:1512.03385) — introduced the residual connection and with it trained networks of 152 layers, an order of magnitude deeper than anything before, winning ImageNet 2015 (ILSVRC) and COCO detection the same year. Residual connections are now in almost every deep network built since, from Transformers to diffusion models. If AlexNet proved depth helps, ResNet made depth usable.

Before this paper

After AlexNet and VGG, the field's rule of thumb was "deeper is better" — until it stopped being true. Stacking more layers past ~20 made accuracy worse, and not from overfitting: training error went up too. This was the degradation problem. It was counter-intuitive, because a deeper network can in principle represent everything a shallower one can (just set the extra layers to identity) — yet plain optimization could not find that solution. Something about very deep plain networks made them hard to train, and the field was stuck around 20–30 layers.

The key idea

Instead of asking a stack of layers to learn a target mapping $H(x)$ directly, ask it to learn the residual $F(x) = H(x) - x$, and add the input back: the block computes $F(x) + x$. The + x is a skip connection — an identity shortcut that jumps over the layers. The insight is that if the best thing a block can do is nothing (pass its input through), learning $F(x) = 0$ is trivial, whereas making a plain stack learn the identity is hard. So the network can always fall back to identity and only spends capacity where a layer actually helps. Depth stops hurting.

How it works

A residual block is two or three convolutions whose output is added to the block's input before the next nonlinearity. That addition does two things. First, it gives optimization an easy default — identity — so extra depth never degrades the signal. Second, and just as important, it creates a gradient highway: during backprop the + x passes the gradient straight through, so even very deep networks get a clean gradient to their early layers, defusing the vanishing-gradient problem. When input and output dimensions differ (at downsampling), a 1×1 convolution on the shortcut matches shapes. Stacking these blocks yields ResNet-34, -50, -101, -152; the deeper ones use a "bottleneck" block (1×1, 3×3, 1×1) to stay cheap.

See it — plain block vs residual block

flowchart LR
  X["input x"] --> W1["conv -> ReLU -> conv"]
  W1 --> ADD["( + )"]
  X -->|"identity skip"| ADD
  ADD --> OUT["ReLU -> output F(x) + x"]

What they showed

On ImageNet, a 34-layer plain network had higher training error than an 18-layer one — the degradation problem, reproduced. Adding residual connections reversed it: the 34-layer ResNet beat the 18-layer, and accuracy kept improving out to 152 layers. ResNet-152 reached 3.57% top-5 error on the ImageNet test set, winning ILSVRC-2015 and surpassing human-level performance on that benchmark. The same backbone swept COCO object detection and segmentation. A striking control experiment trained a 1,000-layer network on CIFAR-10 and it still optimized — depth was, for the first time, essentially free.

What it changed

Residual connections became a permanent fixture of deep learning, far beyond vision. The Transformer wraps every sublayer in a residual connection for exactly the same reason — clean gradients through great depth — which is why modern LLMs can stack dozens to hundreds of layers. Batch-norm-plus-residual became the default recipe for deep convolutional nets for years. More broadly, ResNet reframed the goal: don't force a block to compute the answer, let it compute a small correction to what it already has. That idea echoes in gradient boosting, diffusion models, and iterative refinement everywhere. Next: Attention Comes to Vision (ViT).

The one-line takeaway

Learn the residual, not the whole mapping: an identity skip connection makes "do nothing" the easy default, so networks can go arbitrarily deep without degrading — and the gradient flows straight home.

Read it / meet it

Read "Deep Residual Learning for Image Recognition" (arXiv:1512.03385) — it is remarkably clear, and the degradation-problem figure alone is worth the visit. You meet residual connections in essentially every large model you use: the skip connection that made 152-layer vision networks trainable in 2015 is the same one carrying gradients through today's Transformers.