Interactive Lab · Rotary Positional Embeddings

Give a transformer a sense of position, by spinning its vectors

Attention on its own is order-blind: shuffle the tokens and the math is unchanged. RoPE fixes this with a beautifully simple trick, rotate each query and key vector by an angle proportional to its position. No learned lookup table, no added parameters. And it buys something deep: when a query at position m dots with a key at position n, the rotations combine so the score depends only on the relative distance n − m. Move the query anywhere and the whole attention pattern slides along with it. Watch below.

▶ The playground, position becomes rotation

query position 10 head dim 16 rotary base θ 10000 attends most near relative-only? yes ✓

query @ m keys @ nearby positions a 2nd query, same pattern, shifted

Query position m 10

0 · start of sequence30 · later

Rotary base θ 10000

100 · fast · short range100k · slow · long range

🧭 Where this fits

RoPE is how modern LLMs (Llama, Mistral, Qwen) encode word order so attention knows what came before what. It's also the dial behind long-context models, raising the rotary base is how a model trained on 4K tokens is stretched to 128K. This is that mechanism.

The code that's actually running

🔮 Predict, then test

Drag query position m and watch the lower chart: the attention peak follows the query, and the faint second query (9 positions later) traces an identical curve, just shifted by 9. Same shape, different centre, that's the whole point: RoPE encodes relative position, so a pattern learned at position 5 works unchanged at position 5,000. Absolute positional embeddings can't promise that. Then sweep the rotary base θ: a small base spins the vectors fast (fine detail, short reach), a big base spins them slowly (coarser, but distinguishes far-apart tokens), the reason long-context models crank the base way up.

The takeaway

Your new mental model of RoPE

Position = rotation. Split each vector into 2-D pairs and rotate pair i by position × θ_i. Different pairs spin at different speeds, encoding position across scales.
Relative by construction. Rotations compose, so <R_mq, R_nk> = <q, R_n−mk> the score sees only the gap n − m, never the absolute positions. (Verified here to 1e-15.)
Free and parameter-less. No position embedding table to learn or to run out of, it extends to sequence lengths never seen in training.
The base is the range dial. Bigger rotary base = slower rotation = the model can tell apart tokens that are far apart, which is why long-context LLMs raise it.

Attention →Quantization All playgrounds