PAPERS-01-014 · Paper — Low-Rank Adaptation

The paper

LoRA: Low-Rank Adaptation of Large Language Models, by Edward Hu and colleagues at Microsoft. It appeared on arXiv in 2021 (id 2106.09685) and was published at ICLR 2022. It is the paper that made specializing a giant pretrained model something you could do on a single GPU, and it is the origin point of the modern parameter-efficient fine-tuning ecosystem.

Before this paper

By 2021 the recipe was settled: take an enormous pretrained model and full fine-tune it — continue training and update every weight — to adapt it to your task. That worked, and it was the accuracy gold standard, but it did not scale. Full fine-tuning of a model like GPT-3 means training and storing all 175 billion parameters, and every new task produces another full-size copy of the model. Ten tasks meant ten giants to train, store, and serve. Cheaper tricks existed — freezing most layers, adding adapter modules — but they either underperformed full fine-tuning or added inference latency by inserting extra layers into the forward pass.

The key idea

LoRA's idea is deceptively simple: freeze the pretrained weights entirely, and represent the adaptation as a small, low-rank update added alongside them. The motivating hypothesis is that the change a fine-tune actually needs to make to a weight matrix has low intrinsic rank — it lives in a tiny subspace. If that is true, you never need to move the full weight matrix; you only need to learn a compact factorization of the delta.

How it works

For a pretrained weight matrix W, LoRA leaves W frozen and injects two small trainable matrices next to it — A (shape d×r) and B (shape r×d) — where the rank r is tiny (often 1 to 8). The effective weight becomes W + B·A: the original output plus a low-rank correction. During training only A and B receive gradients; the billions of frozen parameters never move, which is where the memory savings come from. At inference you can fold the product B·A back into W to get a single merged matrix — so unlike adapter layers, LoRA adds no extra modules and no inference latency. Each task is then just its own small (A, B) pair you can swap in.

See it — freeze W, learn a low-rank patch B·A

flowchart LR
  X["input x"] --> W["frozen W<br/>(pretrained)"]
  X --> A["A trainable<br/>(d×r)"]
  A --> B["B trainable<br/>(r×d)"]
  W --> S(("+"))
  B --> S
  S --> Y["output h"]

What they showed

On GPT-3 175B, LoRA cut the number of trainable parameters by roughly 10,000× and reduced the GPU memory required to train by about 3×, compared with full fine-tuning. Crucially it did so with no added inference latency (the update folds into the frozen weights) and delivered task quality on par with full fine-tuning across their benchmarks. So the headline is not "a cheaper approximation" — it is "essentially the same quality, for four orders of magnitude fewer trained parameters and a fraction of the memory."

What it changed

LoRA turned per-task adaptation of huge models from an infrastructure project into a routine operation: a task's adapter is now megabytes, not hundreds of gigabytes, and you can hold thousands of them and hot-swap per request. It became the root of today's PEFT (parameter-efficient fine-tuning) ecosystem — the default way people specialize open models, and the substrate later methods like quantized LoRA build on. And this closes the arc of the whole track: from a single word turned into a vector, through models that read with attention, ground their answers, and align to human feedback, we arrive at a giant you can also cheaply specialize — which loops us right back to where the lineage began, one idea compounding into the next.

The one-line takeaway

Freeze the weights and train a tiny low-rank patch — adapt a giant model for almost nothing, with no quality or latency cost.

Read it / meet it

Read the paper on arXiv: arxiv.org/abs/2106.09685. You meet LoRA every time someone fine-tunes an open model on a single consumer GPU, ships a swappable per-customer adapter, or uses a PEFT library — it is the workhorse behind almost all practical fine-tuning today.

Named concepts: LoRA, low-rank adaptation, parameter-efficient fine-tuning, PEFT