THE TRAINING GROUND · CHAPTER 06

LoRA & QLoRA

You do not have to move all 8.03B weights to teach a clerk a new form. Freeze the base. Learn a thin residual. If even the frozen base is too big, store it in 4-bit and keep the residual in bf16.

~16 min readCorrect BA shapesTested PyTorch

What you'll learn

  • The LoRA residual \(\Delta W = (\alpha / r)\, BA\) and which matrix is which
  • Why \(B\) starts at zero, and why \(\alpha / r\) is there
  • How many trainable scalars a rank-8 adapter adds on Llama 3 8B
  • What QLoRA actually mixes: a 4-bit frozen base plus a LoRA in higher precision

Before you start

A sticky note on a textbook

Full fine-tuning rewrites every page of the textbook. LoRA leaves the book on the shelf and writes a sticky note. The note is small — a thin pair of matrices — but when you read a page you add the note’s correction.

At serve time you can press the note into the page: merge \(\Delta W\) into \(W\) and throw the adapter away. Or you keep several notes and swap them (one per task) without storing five copies of the book.

The hypothesis is that the change you need, \(\Delta W\), has low intrinsic rank. You never store a full \(d_{\text{out}} \times d_{\text{in}}\) update. You store two skinny factors whose product can be that update.

B is tall. A is wide. ΔW = B @ A.

PyTorch stores a linear weight as \(W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}\). LoRA adds a residual of the same shape, built from two factors:

NF4 · adapters · memory

MANIM QLoRA: NF4 base, higher-precision adapters.

sticky · BA · freeze · QLoRA

MANIM LoRA = low-rank sticky note; QLoRA freezes a 4-bit base.

B has r columns. A has r rows. Their product matches W. Scale by alpha over rank.

That order is easy to get backwards. \(A\) is not \(d_{\text{out}} \times r\). If you swap the letters, \(BA\) is no longer defined. The paper and this chapter both use \(h = Wx + (\alpha/r) BAx\).

Initialize \(B\) to zero and \(A\) to a small random matrix. Then \(\Delta W = 0\) at step 0, so training starts at the pretrained forward. If both factors start random, the first step is a shock.

\(\alpha / r\) keeps the update’s size stable when you change rank. Double \(r\) and, without the \(1/r\), the product grows. With it, you can try rank 8 and rank 16 without retuning the learning rate from scratch.

lora.pyshapes, zero-B init, merged weight equals unmerged forward
"""LoRA on a single linear layer, with the paper's shapes.

W is (d_out, d_in), the usual PyTorch layout.
B is (d_out, r). A is (r, d_in). Then ΔW = B @ A has the same shape as W.

B starts at zero so the adapter is a no-op at step 0. A is random. The
forward scales the update by alpha / rank so you can change rank without
retuning the step size as hard.
"""
import math

import torch
import torch.nn as nn
import torch.nn.functional as F


class LoRALinear(nn.Module):
    """Frozen dense layer plus a trainable low-rank residual."""

    def __init__(self, d_in, d_out, rank=8, alpha=16):
        super().__init__()
        self.weight = nn.Parameter(torch.empty(d_out, d_in), requires_grad=False)
        nn.init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        # B @ A is (d_out, d_in). Zero-B keeps the product at 0 at init.
        self.A = nn.Parameter(torch.randn(rank, d_in) * (1 / math.sqrt(d_in)))
        self.B = nn.Parameter(torch.zeros(d_out, rank))
        self.scale = alpha / rank

    def forward(self, x):
        base = F.linear(x, self.weight)
        update = self.scale * (x @ self.A.T @ self.B.T)
        return base + update

    def merged_weight(self):
        """W + (alpha/r) B A. Same numbers as forward, one matrix at serve time."""
        return self.weight + self.scale * (self.B @ self.A)


def lora_params(d_in, d_out, rank):
    """Trainable scalars in A and B. The frozen W is not counted."""
    return rank * d_in + d_out * rank


def llama3_8b_lora_params(rank=8):
    """LoRA on every linear in every layer of Llama 3 8B (no embed)."""
    d, d_h, h, h_kv, inter, layers = 4096, 128, 32, 8, 14_336, 32
    q = lora_params(d, h * d_h, rank)
    k = lora_params(d, h_kv * d_h, rank)
    v = lora_params(d, h_kv * d_h, rank)
    o = lora_params(h * d_h, d, rank)
    gate = lora_params(d, inter, rank)
    up = lora_params(d, inter, rank)
    down = lora_params(inter, d, rank)
    return layers * (q + k + v + o + gate + up + down)

How small is the sticky note?

One Llama 3 8B q_proj is 4096 × 4096 = 16,777,216 weights. Rank 8 LoRA stores

A has 8 × 4096 entries, B has 4096 × 8. That is 256× fewer trainable scalars than the full matrix.

Put a rank-8 adapter on every linear in every layer (Q, K, V, O, gate, up, down; no embedding): 20,971,520 trainable parameters. Against 8.03B frozen weights that is about a quarter of a percent.

AdamW state follows the trainable set. Full fine-tuning paid ~16 bytes per parameter on 8.03B. LoRA pays that tax on 21M. That is the point.

Early recipes targeted only \(W_Q\) and \(W_V\). Current practice, QLoRA included, usually targets all linears. Rank is still a knob: 8 is a common start, 16 or 64 if the task is far from the base.

Keep adapters separate

Swap notes per task. Serving many tasks means many small files, one frozen base.

Merge into W

Zero extra latency. You give up easy task-switching unless you keep a copy of the original W.

4-bit book, bf16 sticky note

LoRA shrinks what you train. It does not shrink the frozen \(W\) you still have to load. QLoRA stores that base in 4-bit (normally NF4) and keeps the LoRA factors in a higher precision, usually bf16.

half a byte per parameter for a 4-bit base. Plus about 42 MiB of rank-8 adapters in bf16.

That is why a 7B–8B SFT run fits on a single 24 GiB card: the book is thin, the note is small, paged optimizers can spill Adam state if needed. The original QLoRA extras — double quantization of the 4-bit scales, paged optimizers — are memory tricks around that picture. The formats themselves are Beyond the Forward Pass 03.

At serve time you can dequantize and merge, or keep the 4-bit base and apply the adapter on the fly. Quality is close to 16-bit LoRA on many instruction tasks, because the residual still has room to move.

Mistakes learners make

Swapping A and B. If A is \(d_{\text{out}} \times r\) and B is \(r \times d_{\text{in}}\), then \(BA\) is not \(d_{\text{out}} \times d_{\text{in}}\). B is tall, A is wide, \(\Delta W = BA\).

Random B at init. Then the first forward is not the pretrained model. Zero B, random A.

Forgetting \(\alpha / r\). Changing rank then secretly changes the update scale, which looks like “rank 16 is unstable.”

“QLoRA trains in 4-bit.” The base is stored in 4-bit. The adapters, and usually the activations they see, stay in a wider format.

What to remember

  • \(W\) stays frozen. \(\Delta W = (\alpha/r)\,BA\) with \(B \in \mathbb{R}^{d_{\text{out}} \times r}\) and \(A \in \mathbb{R}^{r \times d_{\text{in}}}\).
  • Zero \(B\) at init. Merge \(W + \Delta W\) when you want one matrix at serve time.
  • Rank 8 on a 4096 × 4096 map is 65,536 trainable scalars — 256× fewer.
  • QLoRA: 4-bit frozen base (~3.74 GiB for 8.03B) plus a bf16 LoRA.

Check yourself

1. W is (4096, 4096), r = 8. Give the shapes of B and A.

B is (4096, 8). A is (8, 4096). ΔW = B @ A is (4096, 4096).

2. How many trainable scalars is that, and how many times smaller than W?

65,536. 256× smaller.

3. Why start B at zero?

So BA is the zero matrix and the first forward matches the pretrained layer.

4. You switch from rank 8, α = 16 to rank 16, α = 16. What happens to the scale?

It drops from 2 to 1. The \(1/r\) is doing its job. If the run looks weaker, raise α with r, or keep α/r fixed.

5. Does QLoRA store the LoRA factors in 4-bit?

No. The base is 4-bit. The adapters stay wider, usually bf16.

Go to the source

Keep going