Interactive Lab · Attention & Transformers

How every word decides which words to look at

A transformer's superpower is attention: to understand a word, the model lets it look at every other word and decide how much each one matters. It works by a simple recipe, compare each word to every other (a dot product), then softmax the scores into a set of weights that sum to 1. Type a sentence and watch the attention matrix light up. Hover a row to see what that word is paying attention to.

▶ The playground, type a sentence, read the attention

Your sentence

Attention matrix row = a word (query) · brighter cell = it attends more to that column (key) · hover a row

Temperature 1.0

0.2 · laser focusblurry · 4

Position influence 1.0

0 · pure word-similarityattend nearby · 3

🧭 Where this fits

Attention is the core mechanism of every transformer and LLM: each token decides how much to “look at” every other token. It replaced RNNs for language, vision, and audio. Understanding this one operation is understanding how GPT-class models read and relate information.

The code that's actually running

🔮 Predict, then test

Drop the temperature toward 0.2 and watch the matrix collapse, each word attends almost entirely to a single other word (its best match). Push it to 4 and every row goes flat grey: the word spreads its attention evenly, gathering a little from everyone. Real models learn a temperature-like scaling (the √d in the formula) so attention is sharp enough to be useful but soft enough to blend context. Now slide position influence up: attention stops caring about which word and starts hugging the diagonal, each token just looks at its neighbours. Meaning vs. position, traded with one knob.

The takeaway

Your new mental model of attention

Attention is a soft lookup. Each word builds a query, compares it (dot product) against every word's key, and softmaxes the scores into weights that sum to 1, a recipe for "how much of each other word should I mix in?"
The matrix is the whole story. Row i, column j = how much word i attends to word j. Bright diagonal = self-focus; bright off-diagonal = a relationship the model found.
√d keeps it sane. Dividing scores by the square root of the dimension stops the softmax from saturating, the difference between a model that learns and one that doesn't.
Stack & repeat. Real transformers do this with learned query/key/value projections, many "heads" in parallel, and dozens of layers, but every one is doing exactly what you're watching here.

Q, K, V in detail →Multi-head attention 📖 Lecture notes