PAPERS-09-008 · Paper — Layer Normalization

The paper

The landmark paper “Layer Normalization” was introduced by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton in 2016 at the Conference on Neural Information Processing Systems (NeurIPS). It is celebrated because it offered a simple, batch‑independent way to normalize neural activations, solving a key limitation of earlier methods.

Before this paper

Before this work, most deep networks relied on batch normalization, which computes statistics across the mini‑batch. This approach works well for feed‑forward models but is awkward for recurrent networks and for small or variable‑size batches, creating a wall for training sequence models effectively. Think of batch norm as a choir that can only sing in perfect harmony when everyone is present, whereas many real‑world tasks need a soloist to sound good even when the choir is missing.

The key idea

The core insight is to normalize across the features of a single example rather than across the batch, turning the normalization into a per‑instance operation. By anchoring the statistics to each example’s own feature distribution, the method becomes immune to batch‑size fluctuations and works uniformly for any sequence length.

How it works

Layer norm first collects all feature values for one training example. It then computes the mean of those values and their variance across the feature dimension. For a typical hidden size of 768, this means averaging over 768 numbers. Each feature is centered by subtracting the mean and scaled by dividing by the standard deviation (plus a small epsilon for stability). Finally, learnable scale (γ) and shift (β) parameters are applied to allow the network to restore useful representations. Because this computation is per‑example, the same procedure is used at both training and test time, and it works regardless of batch size.

See it — normalize across features, not the batch

flowchart LR
  A["One example"] --> B["Normalize across<br/>its features"]
  B --> C["Independent of<br/>batch size"]
  C --> D["Great for RNNs<br/>+ transformers"]

What they showed

The authors demonstrated that layer normalization stabilized and sped up training for recurrent networks and other sequence models where batch normalization struggled, showing clear improvements in convergence speed and training stability.

What it changed

This technique quickly became the standard normalization method inside transformer architectures, and therefore inside modern large language models. Its batch‑independent nature made it possible to train deep sequence models reliably, reshaping the design of state‑of‑the‑art NLP systems. Next: the frontier — modern training recipes.

The one-line takeaway

Layer normalization makes neural networks work reliably across any batch size by normalizing each example individually.

Read it / meet it

Reading the original paper gives you a deeper appreciation of the simple yet powerful idea that changed modern NLP. You’ll see its legacy every time you encounter a transformer block, which always includes layer normalization.