Interactive Lab · Speculative Decoding

Make a big model faster with a small one, for free

Generating tokens one-at-a-time from a huge model is slow: every token needs a full, expensive forward pass. Speculative decoding cheats: a small, fast draft model guesses the next few tokens, and the big target model checks them all in a single pass. Guesses the target agrees with are kept; the first disagreement is corrected. The astonishing part, proven, not approximate, is that the tokens you emit are distributed exactly as if you'd sampled from the target alone. Same output, 2–4× fewer expensive passes. Watch it below.

▶ The playground, draft proposes, target verifies

draft accept rate tokens / target pass speedup tokens emitted 0 output = target? exact ✓

accepted (draft agreed) rejected → resampled discarded (after reject) bonus token (all accepted)

Draft quality 0.85

0 · random guesser1 · matches target

Speculation length γ 4

1 · cautious8 · greedy guesses

🧭 Where this fits

Speculative decoding is how production LLM serving (vLLM, TensorRT-LLM) gets 2–4× faster tokens for free: a small draft model guesses, the big model verifies. It's a pure latency win with identical output, reach for it any time inference speed matters and you have a smaller model handy.

The code that's actually running

🔮 Predict, then test

Crank draft quality up and hit Run: the accept rate climbs, one target pass emits several tokens, and the speedup shoots up, yet the two distributions in the lower chart stay locked together. Now drag draft quality down to a near-random guesser without resetting: watch the accept rate collapse and the speedup fall back toward 1×… but the output distribution still matches the target exactly. That's the whole point, a worse draft only costs you speed, never correctness. Finally, push γ to 8 with a weak draft: you propose 8 tokens but almost all get thrown away, so the extra draft work is wasted. Match γ to how good your draft is.

The takeaway

Your new mental model of speculative decoding

Guess, then verify. A cheap draft model proposes γ tokens; the expensive target checks them all in one parallel pass instead of γ sequential ones.
Accept / reject / resample. Keep each guess with probability min(1, p/q); on the first miss, resample from the corrected residual distribution. Rigorously distribution-preserving.
Lossless, not lossy. The emitted tokens follow the target's distribution exactly, a weaker draft lowers speed, never quality.
Speedup = acceptance. Throughput is (1 − α^γ+1)/(1 − α). Better draft (higher α) and a γ matched to it give 2–4× fewer expensive passes. It's how fast LLM serving actually works.

Sampling knobs →Beam search All playgrounds