Interactive Lab · LLMs & Transformers

How do raw scores become clean probabilities?

A neural net's final layer spits out raw scores that could be anything-3, 12, 0.5. How do those become clean probabilities that sum to 100%? One function: softmax. It exponentiates the scores, then normalizes, so the biggest score gets an exaggerated share. Drag the logits and watch the distribution form.

🎬 Watch the 45-second explainer, then turn the knobs yourself below ↓

▶ The playground, drag each logit, then bend the temperature

z₁ · 🐱 cat 2.0

-6+10

z₂ · 🐶 dog 1.0

-6+10

z₃ · 🐦 bird 0.0

-6+10

z₄ · 🐟 fish -1.0

-6+10

z₅ · 🐸 frog -2.0

-6+10

Temperature T 1.00

❄️ 0.2 · sharpflat · 4.0 🔥

Σ probabilities100.0%softmax always sums to exactly 1.

winner cat its prob 42.6% entropy 0.00 / 1.00 bits certainty 0%

🧭 Where this fits

Softmax is the last layer of every classifier and the heart of attention weights and LLM token sampling, logits become the probabilities behind every 'next word'. The shift-invariance you just proved is a real numerical-stability trick (subtract the max) present in every production implementation, and temperature scaling (see that lab) divides these exact logits.

The code that's actually running

🔮 Predict, then test

Set two logits nearly equal (say cat = 3, dog = 2.9) and drag the rest way down. At T = 0.2 vs T = 3, how does the split between the top two change? Now hit “+5 to every logit”predict what happens to the bars before you look (hint: softmax is shift-invariant, only the differences between logits matter).

The takeaway

Your new mental model of softmax

Softmax = exp(logits / T), then normalize → a valid probability distribution that sums to 1. The exp exaggerates the biggest logit; dividing by the sum makes the pieces add up.
It's the bridge from raw scores to probabilities, used in classification output layers, attention weights, and next-token sampling.
Temperature controls confidence: low T → peaky, winner-take-all (approaches argmax); high T → flat, near-uniform (maximum uncertainty).
It's shift-invariant, adding the same constant to every logit changes nothing. Only the differences between logits carry information.

📖 Read the lecture notes Try: Temperature Continue the journey →