PAPERS-08-004 · Paper — Quantization

The paper

The paper titled "Quantization" introduced a landmark approach to model efficiency, but specific author, year, and venue details are not provided in the source material.

Before this paper

Before this work, deep learning models were stored and computed using 32-bit floating point numbers. This representation was memory‑heavy, slower, and consumed a lot of power, making it unsuitable for phones and other edge devices. The community needed a way to shrink models and speed up inference without sacrificing too much accuracy. Consequently, deploying high‑quality models on resource‑constrained hardware was a major challenge.

The key idea

The core insight was to represent both weights and activations with far fewer bits, typically 8‑bit integers, instead of 32‑bit floats. By doing so, the model becomes much smaller and can run using cheap integer arithmetic. This reduction dramatically cuts memory usage and power consumption while keeping accuracy largely intact.

How it works

First, each tensor’s floating‑point range is analyzed to compute a scale factor and a zero‑point that map the real values to integer values. Then the floating‑point numbers are quantized to 8‑bit integers using this linear mapping. During inference, all operations are performed with integer arithmetic, applying the scale and zero‑point to interpret results correctly. The authors described two paths: post‑training quantization, which converts a fully trained model directly, and quantization‑aware training, which simulates the rounding effects during training to recover any lost accuracy. These steps enable integer‑only inference on standard hardware.

See it — fewer bits per number

flowchart LR
  A["32-bit floats"] --> B["Map to 8-bit<br/>integers"]
  B --> C["Integer math"]
  C --> D["~4x smaller,<br/>faster, less power"]

What they showed

The authors demonstrated that eight‑bit quantization yields models roughly four times smaller and enables faster, lower‑power inference with little accuracy loss. Integer‑only inference runs efficiently on commodity hardware, confirming the practical benefits of the approach.

What it changed

This work made quantization the default method for shipping efficient models on phones and edge accelerators. It transformed how practitioners think about model deployment, shifting the focus toward integer‑only pipelines. As a result, nearly every on‑device model—and many served models—are now quantized, broadening the reach of deep learning to constrained environments. Next: LoRA.

The one-line takeaway

Quantization turns heavyweight 32‑bit models into lightweight, fast, and power‑efficient integer models without sacrificing accuracy.

Read it / meet it

Reading the original paper offers deeper insight into the quantization process and its impact on modern AI deployment. Its legacy lives on in today’s on‑device models and edge accelerators, where quantized inference is now the norm.