AIM-02-003-P · Primer — Activation Functions in Depth

The idea in one sentence

Leaky ReLU is ReLU with a small negative slope instead of a hard zero, so neurons that go negative can still receive a gradient and recover instead of dying forever.

Why it exists

Plain ReLU (\(f(z) = \max(0, z)\)) is the default activation for a reason — cheap, non-saturating for positive inputs, and it made deep networks trainable. But it has an exact-zero flat region for every negative pre-activation: \(f'(z) = 0\) when \(z < 0\). If a large gradient step (or a big negative bias) ever pushes a neuron's pre-activation permanently negative for the whole training set, its output is 0, its local gradient is 0, and no update can ever pull it back — that neuron is "dead" and stays dead for the rest of training. Think of it like a one-way valve: water (gradient signal) flows through fine when pressure is positive, but the instant pressure goes negative the valve slams shut completely, and nothing — not even a tiny nudge — can reopen it. Leaky ReLU replaces the slammed valve with one that stays cracked open a sliver: negative inputs still produce a (small, scaled) output and a (small, nonzero) gradient, so the neuron always has a path back to relevance.

How it works

The function is piecewise linear: \(f(z) = z\) if \(z > 0\), else \(\alpha \cdot z\), where \(\alpha\) is a small constant (commonly \(0.01\)). Its derivative is \(f'(z) = 1\) for \(z > 0\) and \(f'(z) = \alpha\) for \(z < 0\) (at \(z = 0\) it's a non-differentiable kink, usually just assigned \(\alpha\) or \(1\) by convention in code — it doesn't matter in practice since it's a measure-zero event). The key property versus ReLU: the negative-side derivative is \(\alpha\) instead of exactly \(0\).

Concrete worked example: suppose a neuron's pre-activation, from \(z = w \cdot x + b\), comes out to \(z = 2 \cdot 0.5 + (-3) \cdot 1.5 + 1 = 1 - 4.5 + 1 = -2.5\). With plain ReLU, \(a(z) = \max(0, -2.5) = 0\), and during backprop the local gradient \(\frac{da}{dz} = 0\), so \(\frac{dL}{dz} = \frac{dL}{da} \cdot 0 = 0\) no matter how large the upstream error \(\frac{dL}{da}\) is — the weight \(w\) gets zero update this step, and if \(z\) stays negative on every future batch too, \(w\) and \(b\) never move again (a dead unit). With Leaky ReLU and \(\alpha = 0.01\): \(a(z) = 0.01 \times (-2.5) = -0.025\) (a small but nonzero output), and \(\frac{da}{dz} = \alpha = 0.01\). So \(\frac{dL}{dz} = \frac{dL}{da} \times 0.01\) — a hundredfold-attenuated but nonzero gradient still flows back to \(w\) and \(b\), letting them adjust and potentially push \(z\) positive again on a later step.

The tradeoff is explicit: \(\alpha\) too small (e.g. \(0.001\)) barely differs from ReLU and recovery is glacial; \(\alpha\) too large (e.g. \(0.5\)) makes the function close to linear/symmetric and reduces the nonlinearity's expressive asymmetry. \(\alpha \approx 0.01\)–\(0.2\) is the practical sweet spot used in most implementations.

See it — Gradient flow through the positive vs. negative branch of Leaky ReLU.

flowchart TD
  Z["pre-activation z"] --> Pos{"z > 0?"}
  Pos -->|"yes"| A["a = z<br/>(local grad = 1)"]
  Pos -->|"no"| B["a = alpha * z<br/>(local grad = alpha)"]
  A --> Out["forward output a"]
  B --> Out
  Out --> Grad["backprop dL/dz<br/>= dL/da * local grad"]
  Grad --> Update["weight update<br/>(never exactly zero)"]

The one gotcha

The most common misconception: people think "Leaky ReLU has zero gradient for negative inputs" (that's plain ReLU) or, conversely, that a leaky slope "fully solves" vanishing/dying gradients. It doesn't — it only guarantees the gradient in the negative regime is \(\alpha\) (e.g. \(0.01\)) instead of exactly \(0\), so a stuck neuron can recover, but learning there is still ~\(100\times\) slower than the positive branch. Also, \(\alpha\) is a fixed hyperparameter you choose (typically \(0.01\)–\(0.2\)) — it is only learned per-channel if you switch to the PReLU variant, not in plain Leaky ReLU.

The one-line takeaway

A small nonzero slope keeps gradients — and neurons — alive.

Where you'll meet it

PyTorch nn.LeakyReLU / F.leaky_relu(x, negative_slope=0.01); TensorFlow/Keras tf.keras.layers.LeakyReLU(alpha=0.01); the learned-slope cousin nn.PReLU.

Named concepts: activation depth