THE TRAINING GROUND · CHAPTER 06
You do not have to move all 8.03B weights to teach a clerk a new form. Freeze the base. Learn a thin residual. If even the frozen base is too big, store it in 4-bit and keep the residual in bf16.
Full fine-tuning rewrites every page of the textbook. LoRA leaves the book on the shelf and writes a sticky note. The note is small — a thin pair of matrices — but when you read a page you add the note’s correction.
At serve time you can press the note into the page: merge \(\Delta W\) into \(W\) and throw the adapter away. Or you keep several notes and swap them (one per task) without storing five copies of the book.
The hypothesis is that the change you need, \(\Delta W\), has low intrinsic rank. You never store a full \(d_{\text{out}} \times d_{\text{in}}\) update. You store two skinny factors whose product can be that update.
PyTorch stores a linear weight as \(W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}\). LoRA adds a residual of the same shape, built from two factors:
NF4 · adapters · memory
sticky · BA · freeze · QLoRA
B has r columns. A has r rows. Their product matches W. Scale by alpha over rank.
That order is easy to get backwards. \(A\) is not \(d_{\text{out}} \times r\). If you swap the letters, \(BA\) is no longer defined. The paper and this chapter both use \(h = Wx + (\alpha/r) BAx\).
Initialize \(B\) to zero and \(A\) to a small random matrix. Then \(\Delta W = 0\) at step 0, so training starts at the pretrained forward. If both factors start random, the first step is a shock.
\(\alpha / r\) keeps the update’s size stable when you change rank. Double \(r\) and, without the \(1/r\), the product grows. With it, you can try rank 8 and rank 16 without retuning the learning rate from scratch.
"""LoRA on a single linear layer, with the paper's shapes.
W is (d_out, d_in), the usual PyTorch layout.
B is (d_out, r). A is (r, d_in). Then ΔW = B @ A has the same shape as W.
B starts at zero so the adapter is a no-op at step 0. A is random. The
forward scales the update by alpha / rank so you can change rank without
retuning the step size as hard.
"""
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
class LoRALinear(nn.Module):
"""Frozen dense layer plus a trainable low-rank residual."""
def __init__(self, d_in, d_out, rank=8, alpha=16):
super().__init__()
self.weight = nn.Parameter(torch.empty(d_out, d_in), requires_grad=False)
nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
# B @ A is (d_out, d_in). Zero-B keeps the product at 0 at init.
self.A = nn.Parameter(torch.randn(rank, d_in) * (1 / math.sqrt(d_in)))
self.B = nn.Parameter(torch.zeros(d_out, rank))
self.scale = alpha / rank
def forward(self, x):
base = F.linear(x, self.weight)
update = self.scale * (x @ self.A.T @ self.B.T)
return base + update
def merged_weight(self):
"""W + (alpha/r) B A. Same numbers as forward, one matrix at serve time."""
return self.weight + self.scale * (self.B @ self.A)
def lora_params(d_in, d_out, rank):
"""Trainable scalars in A and B. The frozen W is not counted."""
return rank * d_in + d_out * rank
def llama3_8b_lora_params(rank=8):
"""LoRA on every linear in every layer of Llama 3 8B (no embed)."""
d, d_h, h, h_kv, inter, layers = 4096, 128, 32, 8, 14_336, 32
q = lora_params(d, h * d_h, rank)
k = lora_params(d, h_kv * d_h, rank)
v = lora_params(d, h_kv * d_h, rank)
o = lora_params(h * d_h, d, rank)
gate = lora_params(d, inter, rank)
up = lora_params(d, inter, rank)
down = lora_params(inter, d, rank)
return layers * (q + k + v + o + gate + up + down)
One Llama 3 8B q_proj is 4096 × 4096 = 16,777,216 weights. Rank 8 LoRA stores
A has 8 × 4096 entries, B has 4096 × 8. That is 256× fewer trainable scalars than the full matrix.
Put a rank-8 adapter on every linear in every layer (Q, K, V, O, gate, up, down; no embedding): 20,971,520 trainable parameters. Against 8.03B frozen weights that is about a quarter of a percent.
AdamW state follows the trainable set. Full fine-tuning paid ~16 bytes per parameter on 8.03B. LoRA pays that tax on 21M. That is the point.
Early recipes targeted only \(W_Q\) and \(W_V\). Current practice, QLoRA included, usually targets all linears. Rank is still a knob: 8 is a common start, 16 or 64 if the task is far from the base.
Swap notes per task. Serving many tasks means many small files, one frozen base.
Zero extra latency. You give up easy task-switching unless you keep a copy of the original W.
LoRA shrinks what you train. It does not shrink the frozen \(W\) you still have to load. QLoRA stores that base in 4-bit (normally NF4) and keeps the LoRA factors in a higher precision, usually bf16.
half a byte per parameter for a 4-bit base. Plus about 42 MiB of rank-8 adapters in bf16.
That is why a 7B–8B SFT run fits on a single 24 GiB card: the book is thin, the note is small, paged optimizers can spill Adam state if needed. The original QLoRA extras — double quantization of the 4-bit scales, paged optimizers — are memory tricks around that picture. The formats themselves are Beyond the Forward Pass 03.
At serve time you can dequantize and merge, or keep the 4-bit base and apply the adapter on the fly. Quality is close to 16-bit LoRA on many instruction tasks, because the residual still has room to move.
Swapping A and B. If A is \(d_{\text{out}} \times r\) and B is \(r \times d_{\text{in}}\), then \(BA\) is not \(d_{\text{out}} \times d_{\text{in}}\). B is tall, A is wide, \(\Delta W = BA\).
Random B at init. Then the first forward is not the pretrained model. Zero B, random A.
Forgetting \(\alpha / r\). Changing rank then secretly changes the update scale, which looks like “rank 16 is unstable.”
“QLoRA trains in 4-bit.” The base is stored in 4-bit. The adapters, and usually the activations they see, stay in a wider format.
B is (4096, 8). A is (8, 4096). ΔW = B @ A is (4096, 4096).
65,536. 256× smaller.
So BA is the zero matrix and the first forward matches the pretrained layer.
It drops from 2 to 1. The \(1/r\) is doing its job. If the run looks weaker, raise α with r, or keep α/r fixed.
No. The base is 4-bit. The adapters stay wider, usually bf16.