Interactive Lab · LoRA (Low-Rank Adaptation)
Fine-tuning a giant model the naive way means learning a full weight update ΔW, millions of numbers per layer. LoRA makes the bet that this update is low-rank: it can be written as two skinny matrices B·A with a small rank r, a few thousand numbers instead of millions. Freeze the original weights, train only B and A. Slide the rank below and watch the LoRA patch sharpen toward the true update, then watch the singular-value spectrum tell you exactly why so few numbers are enough.
▶ The playground, approximate the update with rank r
LoRA is how you fine-tune a giant model on a single GPU: train two tiny matrices instead of billions of weights. It's why there are thousands of custom LLM and Stable-Diffusion adapters. Reach for it to specialize a frozen base model cheaply, and keep many adapters for one base.
The code that's actually running
Keep update structure on low-rank (the real-fine-tuning case) and drag rank down to 1: the LoRA patch goes blocky and the error jumps, one component can't capture everything. Now nudge the rank up a couple of steps and watch the reconstruction snap almost perfectly onto ΔW while the parameter count stays tiny. That's the whole trick. Then drag update structure all the way to full-rank / random and try again: now even a big rank barely helps, because a random update has no low-rank structure to exploit. Real fine-tuning updates do, which is exactly why LoRA works in practice and not on noise.
The takeaway