PAPERS-13-008 · Paper — Neural Tangent Kernel — Convergence & Generalization

The paper

The paper “Neural Tangent Kernel — Convergence & Generalization” by Arthur Jacot, François Gabriel, and Clément Hongler (2018, presented at NeurIPS) is a landmark because it gave the first rigorous way to understand how gradient descent trains very wide neural networks, linking deep learning to classical kernel theory.

Before this paper

Before this work, training deep neural networks was treated as a black box. The optimization problem was highly non‑convex and lived in a huge parameter space, making it seem impossible to analyze mathematically. Researchers had no clear picture of what gradient descent would actually converge to, and there was no tractable theory of generalization for deep nets. This lack of theory limited our ability to predict or explain the success of deep learning.

The key idea

The authors realized that when a network’s width goes to infinity, its behavior under gradient descent simplifies dramatically. In that limit the network acts like a linear model defined by a fixed kernel – the Neural Tangent Kernel (NTK). Because the NTK stays essentially constant during training, the whole learning process becomes a well‑understood kernel regression problem. This insight turns an intractable non‑convex problem into a tractable linear one.

How it works

First they wrote down how the network’s output changes as its parameters move under gradient descent. Then they examined the limit where each hidden layer has infinitely many neurons. In that limit the Jacobian of the output with respect to the parameters converges to a deterministic kernel that depends only on the architecture and the random initialization. They proved that this kernel – the NTK – changes negligibly during training, so the dynamics of the parameters reduce to a linear differential equation driven by the NTK. Solving that equation shows that the network’s predictions evolve exactly like kernel ridge regression with the NTK, giving explicit convergence guarantees and generalization bounds.

See it — wide nets become kernels

flowchart TD
  A["Neural net"] --> B["Let width to<br/>infinity"]
  B --> C["Training acts like<br/>a fixed kernel"]
  C --> D["Dynamics become<br/>analyzable"]

What they showed

They showed that in the infinite‑width limit a neural network trained by gradient descent behaves like a linear model in a fixed kernel – the Neural Tangent Kernel – which stays essentially constant during training. This gave the first rigorous handle on the training dynamics and generalization of very wide networks, connecting deep learning to classical kernel theory.

What it changed

The paper opened a whole new branch of deep‑learning theory by providing a tractable model where many empirical phenomena can be studied precisely. It sparked extensive research on "lazy training" and on how the NTK can explain or predict the behavior of real, finite‑width networks. As a result, much of modern deep‑learning theory now builds on the NTK framework. Next: the Multimodal track.

The one-line takeaway

In the infinite‑width limit, deep networks train like kernel regression with a fixed Neural Tangent Kernel.

Read it / meet it

Reading the original NTK paper gives you a clear view of how deep learning can be linked to classical mathematics, and it’s a gentle entry point to the theory of wide networks. Today you’ll see its legacy in many recent works on kernel methods, lazy training, and the theoretical foundations of deep learning.