NEURONS ALL THE WAY DOWN · CHAPTER 03

Neural Language Models

Replace the count table with a function: embed each token, compose the prefix, project to logits, softmax, cross-entropy. The loss is the same NLL you already know.

~18 min read1 interactiveTested PyTorch

What you'll learn

  • Why a count table cannot share strength across similar contexts
  • How an embedding matrix turns a token id into a vector — and why that is one-hot times \(C\)
  • How concatenating a fixed window keeps order, and why summing it does not
  • Softmax, cross-entropy, and the log-sum-exp trick that keeps them stable

Before you start

One vector, many contexts

The harbor-log bigram stored a cell for every pair it had seen. “Fog on” and “rain on” shared nothing. If you later saw “mist on”, the table had no idea that mist behaves like fog.

A neural language model gives every token one vector, an embedding, and reuses that vector wherever the token appears. The rest of the net is a function that reads a few of those vectors and writes a next-token distribution. Train the vectors and the function together, with the same NLL as chapter 01.

We will do it on five 3-letter trail words — bee, bee, bet, tee, ten — over a 5-symbol alphabet . b e n t. The dot is the start and the end. Context is the last two characters. Twenty supervised pairs. Small enough to count, big enough to need a net.

Counting stores a cell per n-gram. A neural model stores a vector per token plus a small MLP. The table grows as \(V^n\). The net grows as \(V \cdot d\) plus a hidden layer that does not care how large \(V\) is until the last projection.

A row lookup is a matrix multiply

Let \(C \in \mathbb{R}^{V \times d}\) be a matrix with one row per symbol. Ours is \(5 \times 2\). The embedding of token \(i\) is row \(i\). In code that is C[x]. It is also a one-hot vector times \(C\):

a one-hot that is 1 in position x picks out row x. There is no extra magic in Embedding.

The 2003 neural probabilistic language model did this with a 17,000-word vocabulary and 30-dimensional rows — 510,000 embedding parameters — plus a 100-unit hidden layer on a 3-word window. Same picture, bigger \(C\).

Because the row for b is shared, a gradient from “bee” and a gradient from “bet” both edit the same two numbers. That is the whole point.

separate cells  →  C ∈ ℝ⁵ˣ²  →  e_bᵀ C  →  bee+bet share row b

MANIM One row of C is shared wherever the token appears. Use the beat buttons to jump.

Keep the order, then add a nonlinearity

You have two context vectors, each of width 2. You need one prefix vector to hand to the classifier. Three historical choices:

Sum

Add the two vectors. Cheap, and “be” becomes identical to “eb”. Order dies. A bag of characters is not a word.

Concatenate

Stick them end to end. Shape \((B, 4)\). This is a neural bigram with memory of position. It is what we train below.

Recurrent / attention

A running state, or a weighted look-up over the whole prefix. The next series. Needed once the window has to grow.

be  ·  eb  →  sum kills order  ·  concat keeps it

MANIM Sum kills order; concat keeps it. Use the beat buttons to jump.

After the concat, a tanh hidden layer of width 8, then a linear map to \(V = 5\) logits. Shapes, for a batch of 20:

TensorShapeWhat it is
X(20, 2)two character ids of context
C[X](20, 2, 2)an embedding per slot
view(20, 4)concat, no extra memory
h(20, 8)\(\tanh(x W_1 + b_1)\)
logits(20, 5)\(h W_2 + b_2\), one score per next symbol

95 parameters in all. A raw character-trigram table on the same alphabet would have \(5^3 = 125\) cells, and would not share the vector for b.

Two stacked linear maps with no tanh in the middle collapse to one map. The squash is what makes the hidden layer worth having. That is the same remark as chapter 02.

Logits in, a distribution and a loss out

Softmax turns a vector of scores into a distribution:

exponentiate to make everything positive, then divide by the sum so it adds to 1.

Cross-entropy against a one-hot target \(y\) is exactly the NLL of chapter 01:

take the negative log of the probability the model put on the true next token. The second form is log-sum-exp.

Worked row: \(z = (2.0,\, 0.5,\, -1.0)\), target \(= 0\).

  1. Subtract the max (2.0) so the exponentials are \(e^{0}, e^{-1.5}, e^{-3} \approx (1,\, 0.223,\, 0.050)\).
  2. Sum \(\approx 1.273\). Softmax \(\approx (\mathbf{0.786},\, 0.175,\, 0.039)\).
  3. Loss \(= -\ln 0.786 \approx \mathbf{0.241}\).

Subtracting the row max does not change the softmax. It stops exp from overflowing when a logit is 100. F.cross_entropy does this, and fuses the backward. Writing it by hand is how you check you understand it. The gradient \(\partial\mathcal{L}/\partial z = P - e_y\) is derived in chapter 05.

Interactive · Softmax and the loss

How does each logit push the probability of the target, and the loss?

Twenty rows, ninety-five numbers, one loss

Each word is three letters plus a closing dot, and we start from two dots of context. For bee the rows are ..→b, .b→e, be→e, ee→.. Five words give 20 rows.

A freshly drawn randn head is not uniform. On this seed the mean NLL starts at 2.92, well above the uniform baseline \(\ln 5 \approx \mathbf{1.609}\). The model is confidently wrong. That is a theme of the next chapter. Two hundred full-batch steps at learning rate 0.3 bring the loss to 0.351.

It cannot go to zero. The context be is followed by e (twice, from bee) and by t (once, from bet). The best possible distribution on that row is \((0,0,2/3,0,1/3)\) over . b e n t, with NLL \(\approx 0.637\) on those three rows alone. Mixed labels are an ambiguity floor, not a bug.

neural_lm.pymanual CE = F.cross_entropy; one-hot @ C = C[X]
"""A tiny neural language model: embeddings → concat → MLP → softmax.

The corpus is five 3-letter words over a 5-symbol alphabet. Context is
the last two characters (a neural bigram-plus). Every shape on the
chapter page is printed from this file.

Why a neural net, not a count table: the same two embedding vectors
are reused in every context. “bee” and “bet” share the vector for b.
A count table cannot do that.
"""
import math
import torch
import torch.nn.functional as F

# Five trail-blaze words. Alphabet: '.', 'b', 'e', 'n', 't'
# The dot is both the start and the end marker, like the harbor-log <s>/<s>.
CORPUS = ["bee", "bee", "bet", "tee", "ten"]
CHARS = [".", "b", "e", "n", "t"]
STOI = {ch: i for i, ch in enumerate(CHARS)}
ITOS = {i: ch for ch, i in STOI.items()}
V = len(CHARS)          # 5
BLOCK = 2               # characters of context
N_EMB = 2               # embedding width
N_HIDDEN = 8            # tanh hidden width


def build_dataset(words=CORPUS, block=BLOCK):
    """Rolling windows: '..'→b, '.b'→e, 'be'→e, 'ee'→.  for the word 'bee'."""
    xs, ys = [], []
    for w in words:
        ctx = [STOI["."]] * block
        for ch in w + ".":
            xs.append(ctx[:])
            ys.append(STOI[ch])
            ctx = ctx[1:] + [STOI[ch]]
    return torch.tensor(xs), torch.tensor(ys)


X, Y = build_dataset()
N = X.shape[0]  # 5 words × 4 predictions = 20


class NeuralLM(torch.nn.Module):
    """Bengio-style MLP. One-hot lookup is just C[X]."""

    def __init__(self):
        super().__init__()
        # C has one row per character. C[X] gathers a row for each context slot.
        self.C = torch.nn.Embedding(V, N_EMB)
        # Two context slots, concatenated: 2 * 2 = 4 inputs to the hidden layer.
        self.W1 = torch.nn.Parameter(torch.randn(BLOCK * N_EMB, N_HIDDEN))
        self.b1 = torch.nn.Parameter(torch.zeros(N_HIDDEN))
        self.W2 = torch.nn.Parameter(torch.randn(N_HIDDEN, V))
        self.b2 = torch.nn.Parameter(torch.zeros(V))

    def logits(self, x):
        # x: (B, BLOCK) integer ids  →  emb: (B, BLOCK, N_EMB)  →  (B, BLOCK*N_EMB)
        emb = self.C(x)
        h = torch.tanh(emb.view(x.shape[0], -1) @ self.W1 + self.b1)
        return h @ self.W2 + self.b2

    def forward(self, x, y):
        return F.cross_entropy(self.logits(x), y)


def manual_cross_entropy(logits, y):
    """Softmax + NLL, written out, with the max-subtraction that keeps it stable."""
    # Subtract the row max so exp never overflows. Softmax is shift-invariant.
    shifted = logits - logits.max(dim=1, keepdim=True).values
    log_z = shifted.exp().sum(dim=1, keepdim=True).log()
    log_probs = shifted - log_z
    return -log_probs[torch.arange(y.shape[0]), y].mean()


# --- Worked 3-class softmax used on the page (separate from the LM). ---
SOFTMAX_LOGITS = torch.tensor([2.0, 0.5, -1.0])
SOFTMAX_TARGET = 0  # the first class is correct


def softmax_ce(logits, target):
    """Single-row cross-entropy. Returns (probs, loss)."""
    shifted = logits - logits.max()
    probs = shifted.exp() / shifted.exp().sum()
    return probs, -math.log(probs[target].item())


SOFTMAX_PROBS, SOFTMAX_CE = softmax_ce(SOFTMAX_LOGITS, SOFTMAX_TARGET)

# Expected loss at a uniform (or freshly-init, roughly uniform) head.
UNIFORM_NLL = math.log(V)  # ln 5 ≈ 1.609

# Ambiguity floor: context "be" is followed by both "e" (bee) and "t" (bet).
BE_CONTEXT = [STOI["b"], STOI["e"]]
BE_TARGETS = [ITOS[Y[i].item()] for i in range(N) if X[i].tolist() == BE_CONTEXT]


def train(steps=200, lr=0.3, seed=0):
    """A short full-batch run so the page can quote a start and end loss."""
    torch.manual_seed(seed)
    model = NeuralLM()
    # Start-of-training loss. With this seed it sits near the uniform 1.609.
    start = model(X, Y).item()
    for _ in range(steps):
        loss = model(X, Y)
        model.zero_grad()
        loss.backward()
        with torch.no_grad():
            for p in model.parameters():
                p -= lr * p.grad
    end = model(X, Y).item()
    return model, start, end


MODEL, LOSS_START, LOSS_END = train()

# Manual CE must match the fused kernel on the trained logits.
with torch.no_grad():
    MANUAL_CE = manual_cross_entropy(MODEL.logits(X), Y).item()
    FUSED_CE = F.cross_entropy(MODEL.logits(X), Y).item()


def compose_sum(ids):
    """Sum the two context embeddings. “be” and “eb” become the same vector."""
    return MODEL.C.weight[ids[0]] + MODEL.C.weight[ids[1]]


def compose_cat(ids):
    """Concatenate. Order survives."""
    return torch.cat([MODEL.C.weight[ids[0]], MODEL.C.weight[ids[1]]])
  • emb.view(B, -1) — concat by reshape. torch.cat(unbind(...)) does the same and copies memory.
  • F.cross_entropy — fused softmax + NLL. manual_cross_entropy is the same number, written out.
  • The start loss 2.92 is a clue that random W2 is too large. Chapter 04 shrinks it.

Mistakes learners make

Softmax without subtracting the max. exp(100) overflows to inf, and the row becomes NaN. The probabilities you wanted are unchanged by a shift. Always shift, or call the fused op.

Summing embeddings and expecting order to survive. Addition is commutative. “be” and “eb” become the same prefix vector. Concatenate a short window, or use attention later.

Reading a start loss above \(\ln V\) as a wiring bug. It usually means the logits are too sharp in a random direction. The next chapter treats that as an initialization problem, not a broken loss.

Chasing loss 0 on a dataset with mixed labels. The same context can have two true next tokens. The floor is the entropy of that empirical conditional.

What to remember

  • An embedding is a learned row. C[x] equals one-hot(\(x\)) times \(C\).
  • A fixed window is concatenated, then an MLP, then a \(V\)-way linear head.
  • Softmax + NLL is cross-entropy. Log-sum-exp with a max-shift is the stable form.
  • The loss is the same quantity as chapter 01. The parameters now generalize across contexts.

Check yourself

1. Why is C[x] equal to a one-hot times \(C\)?

A one-hot with a 1 at index \(x\) dotted into each column of \(C\) (or left-multiplied) selects row \(x\). Embedding is that multiply, implemented as a gather.

2. Logits \((2.0, 0.5, -1.0)\), target 0. What is the loss?

Softmax \(\approx (0.786, 0.175, 0.039)\). Loss \(= -\ln 0.786 \approx 0.241\).

3. Why can this trail-word model never reach loss 0?

The context be is followed by both e and t. The best row still puts mass on two symbols, so NLL stays positive.

4. A uniform 5-way head has what NLL?

\(\ln 5 \approx 1.609\). A start loss of 2.92 is worse than uniform: the random head is peaked on the wrong symbols.

5. What does summing two context embeddings throw away?

Order. The sum of “b then e” equals the sum of “e then b”. Concatenation (or attention) keeps the sequence.

Go to the source

Keep going