AIM-03-003-P · Primer — Positional Encoding

The idea in one sentence

Positional encoding injects each token's sequence position into its embedding using fixed sine and cosine waves of different frequencies, since self-attention itself is otherwise blind to word order.

Why it exists

Self-attention computes a weighted sum over all tokens using only content-based similarity (query dot key) — if you shuffled the input tokens, attention would produce the exact same set of weights, just permuted, because there is no built-in notion of 'first' or 'third.' It's like reading a room full of people shouting their opinions simultaneously: you can tell what each person said and how loud they were, but nothing tells you who spoke first unless someone also announces their position in line. Positional encoding is that announcement — a fixed signal added to each token's embedding that says 'I am at position 5,' so attention can learn to care about relative order (e.g., 'the word right before me' or 'ten tokens back') even though its core math is order-agnostic.

How it works

The original Transformer (Vaswani et al., 2017) builds a vector of length \(d\) (the model's embedding dimension) for every position \(pos = 0, 1, 2, \dots\) and adds it directly to the token embedding before the first attention layer. Each dimension index \(i\) in that vector oscillates at its own fixed frequency: even dimensions use sine and odd dimensions use the paired cosine at the same frequency, so dimensions 0/1 share one frequency, dimensions 2/3 share a slower one, and so on down to the slowest pair at the top of the vector. Formally, \[ PE(pos, i) = \begin{cases} \sin\left(\dfrac{pos}{10000^{\,i/d}}\right) & i \text{ even} \\[6pt] \cos\left(\dfrac{pos}{10000^{\,(i-1)/d}}\right) & i \text{ odd} \end{cases} \] Low-index dimensions (small \(i\)) have exponent near 0, so \(10000^{i/d} \approx 1\) and the angle \(pos/1 = pos\) grows fast — these dimensions oscillate rapidly and distinguish nearby positions. High-index dimensions have exponent near 1, so \(10000^{i/d} \approx 10000\) and the angle grows extremely slowly — these dimensions barely change across a sentence but distinguish very distant positions, like the hour, minute, and second hands of a clock each ticking at a different rate to jointly encode the exact time.

Concrete worked example with \(d = 4\) at \(pos = 1\): dimension 0 (even, exponent \(0/4=0\)) gives \(\sin(1/10000^0) = \sin(1) \approx 0.8415\); dimension 1 (odd, paired with dim 0) gives \(\cos(1) \approx 0.5403\); dimension 2 (even, exponent \(2/4=0.5\)) gives \(\sin(1/10000^{0.5}) = \sin(0.01) \approx 0.0100\); dimension 3 (odd, paired with dim 2) gives \(\cos(0.01) \approx 0.9999\). Notice dims 0–1 moved a lot going from \(pos=0\) (where they'd be \(\sin(0)=0, \cos(0)=1\)) to \(pos=1\), while dims 2–3 barely moved — exactly the fast-vs-slow clock-hand behavior. This specific sinusoidal choice also gives a useful algebraic property: \(PE_{pos+k}\) can be written as a fixed linear (rotation) transform of \(PE_{pos}\) for any offset \(k\), which is why the original paper argued this scheme should let the model learn to attend by relative position, not just absolute position — a property later formalized more directly as rotary position embeddings (RoPE) in models like LLaMA.

See it — Each dimension of the positional vector is a wave at its own frequency; stacking them lets every position get a unique fingerprint.

flowchart TD
    A["Token embedding<br/>(content only)"] --> C["Add elementwise"]
    B["Positional encoding<br/>(sin/cos by dim)"] --> C
    C --> D["Position-aware<br/>input to attention"]
    E["Dim 0-1: fast wave<br/>(nearby positions)"] --> B
    F["Dim d-2,d-1: slow wave<br/>(distant positions)"] --> B

The one gotcha

Rotating or reflecting dimension pairs mid-training (or mismatching even/odd indexing) silently scrambles the frequency pairing, degrading position sensitivity without any error.

The one-line takeaway

Fixed sine/cosine waves per dimension give attention order sense.

Where you'll meet it

Hugging Face Transformers modeling_utils sinusoidal defaults, PyTorch tutorials' PositionalEncoding module, and rotary variants (RoPE) in LLaMA/Mistral served via Hugging Face and vLLM.

Named concepts: positional encoding