PAPERS-09-007 · Paper — Adam Optimizer

The paper

The 2015 paper "Adam: A Method for Stochastic Optimization" by Diederik P. Kingma and Jimmy Ba, presented at ICLR, is a landmark because it introduced a widely adopted optimizer that dramatically simplified training deep networks.

Before this paper

Before Adam, researchers relied on plain gradient descent, which required careful hand‑tuned learning rates. A single global rate often worked for some parameters but not for others, making training fragile and time‑consuming. This limitation slowed progress and demanded extensive hyperparameter searches. Consequently, many experiments stalled on unstable convergence.

The key idea

The core insight of Adam is to give each parameter its own adaptive learning rate, computed from running estimates of both the gradient and its variance. By tracking these moments, the optimizer can automatically adjust step sizes for each weight. This lets noisy, large‑gradient parameters take smaller steps while smoother parameters move faster, all without manual tuning.

How it works

Adam maintains an exponentially decaying average of past gradients (the first moment) and an exponentially decaying average of past squared gradients (the second moment). At each iteration it updates these averages, then applies bias‑correction to account for their initialization at zero. The corrected first‑moment estimate is divided by the square root of the corrected second‑moment estimate plus a small constant for stability. This ratio is multiplied by a base learning rate to produce the parameter‑specific update. The process repeats for every training step, automatically adapting to the geometry of the loss surface.

See it — a personal step size per weight

flowchart LR
  A["Gradient"] --> B["Track 1st moment<br/>(momentum)"]
  A --> C["Track 2nd moment<br/>(variance)"]
  B --> D["Per-parameter<br/>adaptive step"]
  C --> D

What they showed

The authors showed that Adam converged fast and robustly across a wide range of problems with little tuning, becoming the go‑to optimizer for many deep‑learning tasks.

What it changed

Adam quickly became the default optimizer for most deep‑learning training today, replacing many hand‑crafted learning‑rate schedules. Its ease of use and strong performance lowered the barrier to entry for new researchers and accelerated experimentation. By standardizing a reliable optimizer, it helped unify training practices across the community. Next: Layer Normalization.

The one-line takeaway

Adam lets every weight learn at its own pace, removing the need for painstaking learning‑rate tuning.

Read it / meet it

Reading the original Adam paper gives insight into the simple yet powerful moment‑based adaptation that underpins modern training pipelines. You’ll see its legacy every time you launch a deep‑learning experiment, as Adam (or its variant AdamW) is the optimizer most practitioners reach for.