THE EVOLVING TRANSFORMER · STOP 01

The original Transformer (2017)

An encoder reads the source. A decoder writes the target, one token at a time, looking back at the source. This is the 2017 model, built so later stops can diff against it.

~12 min readEncoder–decoderTested PyTorch code

What you'll learn

  • The 2017 stack: embeddings × √d, sinusoids, MHA, ReLU FFN
  • Why this code is post-norm, and how that differs from modern pre-norm
  • How to build a padding mask and a causal decoder mask
  • How greedy decode encodes once, then argmaxes the last position

Before you start

This is the baseline

There is no previous stop. You are building the 2017 encoder–decoder: source and target each get their own embedding and sinusoids, the encoder is bidirectional, the decoder is causal and also reads the encoder through cross-attention. Residuals are post-norm — LayerNorm(x + sublayer(x)) — as in Figure 1 of the paper. Later stops switch to pre-norm because deep stacks train more stably that way. This file stays honest about 2017.

Think of a clerk copying a harbour log from one language to another. The encoder is the finished reading of the source page. The decoder writes the new page left to right, never peeking at ink it has not written yet, but it may reread any word on the source page.

The “why” of attention, heads, and residual blocks lives in Deep Cuts. This stop is the runnable skeleton those chapters point at. Read the block chapter →

A tiny batch you can check by hand

Teaching config: d_model = 32, H = 4 so d_k = 8, N = 2, vocab 50. Batch of 2. Source length 6, target length 5.

enc–dec · bi / causal / cross · logits (2,5,50)

MANIM Two towers, three attentions, logits (2, 5, 50).
TensorShapeNote
src tokens(2, 6)integer ids
src embeddings(2, 6, 32)lookup × √32
src mask(2, 1, 1, 6)1 = keep, 0 = pad
encoder out(2, 6, 32)memory for cross-attention
tgt tokens(2, 5)teacher-forced prefix
tgt mask(2, 1, 5, 5)causal ∩ pad
logits(2, 5, 50)one vocab row per target position

The 2017 paper used 512 / 2048 / 8 heads / 6 layers. Same graph, bigger numbers.

One file, three entry points

encode, decode, and project stay separate so greedy decode can encode the source once. Masks are built from token ids: padding for the encoder, padding and a lower triangle for the decoder.

transformer.pytiny batch + greedy decode
"""2017 encoder-decoder Transformer (Vaswani et al.).

This is post-norm, as in the paper: LayerNorm(x + dropout(sublayer(x))).
Modern decoders (GPT-2 onward) flip that to pre-norm.

Teaching default is tiny so a laptop can forward a batch. The published
2017 config lives on the page: d_model 512, d_ff 2048, 8 heads, 6 layers.
"""

from dataclasses import dataclass
import math

import torch
import torch.nn as nn


@dataclass
class TransformerConfig:
    dmodel: int = 32
    vocabsize: int = 50
    max_seq_len: int = 16
    dff: int = 64
    H: int = 4
    N: int = 2
    dropout: float = 0.0
    norm_eps: float = 1e-6
    pad_id: int = 0
    bos_id: int = 1
    eos_id: int = 2

    @property
    def dk(self):
        return self.dmodel // self.H


# ---------------------------------------------------------------------------
# Masks — padding for the encoder, padding ∩ causal for the decoder
# ---------------------------------------------------------------------------
def padding_mask(token_ids, pad_id=0):
    """1 where the token is real, 0 on pad. Shape (B, 1, 1, S)."""
    return (token_ids != pad_id).unsqueeze(1).unsqueeze(2)


def causal_mask(seq_len, device=None):
    """Lower-triangular ones. Shape (1, 1, S, S)."""
    return torch.tril(torch.ones(seq_len, seq_len, device=device)).view(1, 1, seq_len, seq_len)


def decoder_mask(tgt_ids, pad_id=0):
    """Causal mask AND target padding. Shape (B, 1, S, S)."""
    pad = padding_mask(tgt_ids, pad_id)  # (B, 1, 1, S)
    causal = causal_mask(tgt_ids.size(1), tgt_ids.device)
    return pad & causal.bool()


# ---------------------------------------------------------------------------
# Building blocks
# ---------------------------------------------------------------------------
class InputEmbedding(nn.Module):
    """Token ids → vectors, scaled by sqrt(d_model) as in §3.4."""

    def __init__(self, config):
        super().__init__()
        self.embedding = nn.Embedding(config.vocabsize, config.dmodel)
        self.scale = math.sqrt(config.dmodel)

    def forward(self, x):
        return self.embedding(x) * self.scale


class PositionalEncoding(nn.Module):
    """Fixed sinusoids. PE(pos, 2i) = sin(pos / 10000^(2i/d))."""

    def __init__(self, config):
        super().__init__()
        pe = torch.zeros(config.max_seq_len, config.dmodel)
        pos = torch.arange(config.max_seq_len, dtype=torch.float).unsqueeze(1)
        div = 1.0 / (10000 ** (torch.arange(0, config.dmodel, 2).float() / config.dmodel))
        pe[:, 0::2] = torch.sin(pos * div)
        pe[:, 1::2] = torch.cos(pos * div)
        self.register_buffer("pe", pe.unsqueeze(0))

    def forward(self, x):
        return x + self.pe[:, : x.size(1)]


class FeedForward(nn.Module):
    """Position-wise ReLU MLP: d_model → d_ff → d_model."""

    def __init__(self, config):
        super().__init__()
        self.layer1 = nn.Linear(config.dmodel, config.dff)
        self.layer2 = nn.Linear(config.dff, config.dmodel)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, x):
        return self.layer2(self.dropout(torch.relu(self.layer1(x))))


class MultiHeadAttention(nn.Module):
    """Scaled dot-product attention over H heads."""

    def __init__(self, config):
        super().__init__()
        self.h = config.H
        self.dk = config.dk
        self.dmodel = config.dmodel
        self.wq = nn.Linear(config.dmodel, config.dmodel)
        self.wk = nn.Linear(config.dmodel, config.dmodel)
        self.wv = nn.Linear(config.dmodel, config.dmodel)
        self.wo = nn.Linear(config.dmodel, config.dmodel)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, q, k, v, mask):
        B, S, _ = q.shape
        Sk, Sv = k.size(1), v.size(1)
        q = self.wq(q).view(B, S, self.h, self.dk).transpose(1, 2)
        k = self.wk(k).view(B, Sk, self.h, self.dk).transpose(1, 2)
        v = self.wv(v).view(B, Sv, self.h, self.dk).transpose(1, 2)
        scores = q @ k.transpose(-1, -2) / math.sqrt(self.dk)
        if mask is not None:
            # mask == 0 is blocked (pad or future)
            scores = scores.masked_fill(mask == 0, float("-inf"))
        weights = self.dropout(scores.softmax(dim=-1))
        out = (weights @ v).transpose(1, 2).contiguous().view(B, S, self.dmodel)
        return self.wo(out)


class ResidualConnection(nn.Module):
    """POST-norm residual: LayerNorm(x + dropout(sublayer(x)))."""

    def __init__(self, config):
        super().__init__()
        self.norm = nn.LayerNorm(config.dmodel, eps=config.norm_eps)
        self.dropout = nn.Dropout(config.dropout)

    def forward(self, x, sublayer):
        return self.norm(x + self.dropout(sublayer(x)))


class EncoderBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.attn = MultiHeadAttention(config)
        self.ff = FeedForward(config)
        self.res = nn.ModuleList([ResidualConnection(config) for _ in range(2)])

    def forward(self, x, mask):
        x = self.res[0](x, lambda t: self.attn(t, t, t, mask))
        return self.res[1](x, self.ff)


class Encoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.layers = nn.ModuleList([EncoderBlock(config) for _ in range(config.N)])

    def forward(self, x, mask):
        for layer in self.layers:
            x = layer(x, mask)
        return x


class DecoderBlock(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.self_attn = MultiHeadAttention(config)
        self.cross_attn = MultiHeadAttention(config)
        self.ff = FeedForward(config)
        self.res = nn.ModuleList([ResidualConnection(config) for _ in range(3)])

    def forward(self, x, memory, src_mask, tgt_mask):
        x = self.res[0](x, lambda t: self.self_attn(t, t, t, tgt_mask))
        x = self.res[1](x, lambda t: self.cross_attn(t, memory, memory, src_mask))
        return self.res[2](x, self.ff)


class Decoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.layers = nn.ModuleList([DecoderBlock(config) for _ in range(config.N)])

    def forward(self, x, memory, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, memory, src_mask, tgt_mask)
        return x


class Transformer(nn.Module):
    """Full 2017 encoder-decoder. encode / decode / project stay separate
    so greedy decoding can encode the source once."""

    def __init__(self, config):
        super().__init__()
        self.config = config
        self.src_embedding = InputEmbedding(config)
        self.tgt_embedding = InputEmbedding(config)
        self.src_pe = PositionalEncoding(config)
        self.tgt_pe = PositionalEncoding(config)
        self.drop = nn.Dropout(config.dropout)
        self.encoder = Encoder(config)
        self.decoder = Decoder(config)
        # raw logits — the paper's pre-softmax projection
        self.proj = nn.Linear(config.dmodel, config.vocabsize)

    def encode(self, src, src_mask):
        x = self.drop(self.src_pe(self.src_embedding(src)))
        return self.encoder(x, src_mask)

    def decode(self, tgt, memory, src_mask, tgt_mask):
        x = self.drop(self.tgt_pe(self.tgt_embedding(tgt)))
        return self.decoder(x, memory, src_mask, tgt_mask)

    def project(self, x):
        return self.proj(x)

    def forward(self, src, tgt):
        src_mask = padding_mask(src, self.config.pad_id)
        tgt_mask = decoder_mask(tgt, self.config.pad_id)
        memory = self.encode(src, src_mask)
        hidden = self.decode(tgt, memory, src_mask, tgt_mask)
        return self.project(hidden)


@torch.no_grad()
def greedy_decode(model, src, max_len=None):
    """Encode once, then argmax one target token at a time."""
    cfg = model.config
    max_len = max_len or cfg.max_seq_len
    src_mask = padding_mask(src, cfg.pad_id)
    memory = model.encode(src, src_mask)
    ys = torch.full((src.size(0), 1), cfg.bos_id, dtype=torch.long, device=src.device)
    for _ in range(max_len - 1):
        tgt_mask = decoder_mask(ys, cfg.pad_id)
        logits = model.project(model.decode(ys, memory, src_mask, tgt_mask))
        next_id = logits[:, -1].argmax(dim=-1, keepdim=True)
        ys = torch.cat([ys, next_id], dim=1)
        if (next_id == cfg.eos_id).all():
            break
    return ys

Test note. The check script forwards (2, 6) source and (2, 5) target, asserts every row of the shape table, and runs greedy decode from a BOS token.

Mistakes this stop used to make

Calling post-norm “pre-LN”. The residual here is LN(x + f(x)). GPT-2 flips it to x + f(LN(x)). The names are not interchangeable.

Forgetting the causal mask on the target. Teacher forcing feeds the whole target at once. Without the triangle, position 2 can see token 3.

Re-encoding every greedy step. Source tokens do not change. Encode once, then grow the target.

What to remember

  • 2017 is encoder–decoder, sinusoidal positions, ReLU FFN, post-norm residuals.
  • Encoder mask hides padding. Decoder mask hides padding and the future.
  • Greedy decode: encode once, argmax the last decoder position, append, repeat.

Check yourself

1. Is this implementation pre-norm or post-norm?

Post-norm: LayerNorm(x + dropout(sublayer(x))). Modern models switch to pre-norm.

2. What is the teaching-config logit shape for batch 2 and target length 5?

(2, 5, 50) — vocab 50.

3. Why keep encode / decode / project as three methods?

So inference can encode the source once and loop only on the decoder.

Go to the source

Keep going