Interactive Lab · Tokenization

Watch subwords build themselves from characters

Before an LLM sees your prompt, a tokenizer chops it into pieces. But which pieces? Byte-Pair Encoding figures it out from data with one dead-simple rule, repeated: find the most common adjacent pair and glue it into a new token. Do that a few thousand times and useful chunks like ·the and cat emerge on their own. Slide the merges below and watch raw characters fuse into subwords.

▶ The playground, type text, then merge

Your text edit it; BPE re-learns from what you type

merges 0 tokens characters compression vocab size

Merges applied 0 / 0

0 · pure characterssubwords emerge →

Tokenization now · marks a leading space; orange = a learned multi-character subword

Merge rules learned so far rank: pair → new token (times it occurred)

🧭 Where this fits

Before an LLM sees text, a tokenizer chops it into subword pieces (BPE). It's why models have token limits, why rare words cost more, and why the same prompt is pricier in some languages. Every API call's cost and context window is counted in these tokens.

The code that's actually running

🔮 Predict, then test

Type a word many times (e.g. paste banana banana banana banana) and slide the merges up. Watch it collapse from b·a·n·a·n·a down to a single token, because a token that appears constantly is worth its own vocabulary slot. Now add a word you only typed once: it stays broken into small pieces. That's the whole trade-off BPE balances, common strings get short tokens, rare strings get spelled out, which is exactly why an LLM handles a typo or a made-up word gracefully instead of choking.

The takeaway

Your new mental model of tokenization

Tokens aren't words. They're learned chunks, sometimes a whole word, often a piece (token + ization), sometimes a single character for anything rare.
The vocabulary is data-driven. Nobody wrote the token list by hand. BPE grew it by greedily merging the most frequent pair, over and over, from a giant corpus.
Frequency buys brevity. Common strings earn short (single-token) encodings; rare ones get spelled out from smaller pieces, so any text is representable, and common text is cheap.
It's why token counts matter. Your API bill and the model's context window are measured in these tokens, and now you know exactly where they come from.

Tokens become vectors: Embeddings →Sampling: Temperature 📖 Lecture notes