PAPERS-09-001 · Paper — The Lineage — The Ideas That Made Deep Learning Work

The paper

Before this paper

The key idea

How it works

See it — the toolkit that trains deep nets

flowchart LR
  A["Backprop<br/>compute gradients"] --> B["Init + Dropout<br/>start + regularize"]
  B --> C["BatchNorm / LayerNorm<br/>stabilize"]
  C --> D["Adam<br/>optimize"]

What they showed

What it changed

Next: Backpropagation.

The one-line takeaway

Read it / meet it