PAPERS-09-002 · Paper — Backpropagation

The paper

The landmark paper on backpropagation introduced a practical way to train multilayer neural networks, establishing a foundation for modern deep learning.

Before this paper

Before this work, networks with hidden layers could represent rich functions, but there was no practical way to learn their internal weights. Researchers could not efficiently adjust the many connections inside the network. This limitation kept deep models from being useful in real applications. The field needed a method to compute useful updates for hidden weights. It was like having a complex maze with no map—one could see the entrance and exit but had no way to navigate the interior.

The key idea

The key insight was to compute how each weight influences the overall error by sending the error signal backward through the network using the chain rule, turning the opaque training problem into a systematic gradient calculation that reveals each parameter's contribution to loss.

How it works

First, a forward pass runs through the network, producing a prediction and measuring the error against the target. Next, the error is propagated backward, layer by layer, applying the chain rule to compute the gradient of the error with respect to every weight. These gradients tell us the direction each weight should move to reduce the error. Finally, gradient descent nudges each weight a small step in that direction. For a typical sigmoid unit, the gradient term often falls in the range of 0.1–0.5, providing a concrete scale for the update. Repeating this cycle lets hidden units gradually learn useful internal representations. The process is repeated over many examples to train the whole network.

See it — send the error backward

flowchart LR
  A["Forward: input to<br/>prediction"] --> B["Compute error"]
  B --> C["Backward: chain rule<br/>gradient per weight"]
  C --> D["Update weights"]

What they showed

The authors demonstrated that backpropagation made training multilayer networks practical and that hidden layers could discover meaningful features on their own.

What it changed

This paper became the foundational learning algorithm for neural networks—every deep network today is trained by backprop. It turned deep architectures from theoretical curiosities into workable tools. By providing a scalable learning method, it opened the door to the explosion of deep learning research and applications. Next: Word2Vec.

The one-line takeaway

Backpropagation turned hidden layers from untrainable mysteries into learnable feature extractors.

Read it / meet it

Reading the original paper reveals the elegant simplicity of the algorithm and its historic impact. Today you meet its legacy in every autograd system such as PyTorch and TensorFlow, which implement backpropagation under the hood.