Interactive Lab · Quantization
A model's weights are usually 32-bit floats. Quantization stores them in far fewer bits, 8, 4, even 3, so the model is smaller and faster, at the cost of rounding error. The trick is spending your few bits well. Every bit you add is worth about 6 dB of fidelity, but a handful of outlier weights can blow the whole budget, forcing coarse steps on the millions of ordinary weights. Watch it happen, then flip on per-group scales and see the error collapse.
▶ The playground, round the weights, watch the error
Quantization is how you fit a 70B model on one GPU, or run an LLM on a phone: store weights in 4–8 bits instead of 32. It's the difference between “needs a datacenter” and “runs locally.” Every on-device or cost-optimized deployment leans on this.
The code that's actually running
Set outliers to heavy and stay on Per-tensor at 4 bits: the fidelity is poor, because one giant weight stretches the scale so every ordinary weight snaps to a coarse step. Now click Per-group, each block of weights gets its own scale, the outlier is quarantined to its own group, and the fidelity jumps several dB with the same 4 bits. That's the core idea behind modern LLM quantizers (GPTQ / AWQ). Then sweep precision and watch the ~6 dB per bit rule: every extra bit roughly halves the error. 8 bits is essentially free; the interesting fight is 3–4 bits.
The takeaway