THE EVOLVING TRANSFORMER · STOP 01
An encoder reads the source. A decoder writes the target, one token at a time, looking back at the source. This is the 2017 model, built so later stops can diff against it.
There is no previous stop. You are building the 2017 encoder–decoder: source and target each get their own embedding and sinusoids, the encoder is bidirectional, the decoder is causal and also reads the encoder through cross-attention. Residuals are post-norm — LayerNorm(x + sublayer(x)) — as in Figure 1 of the paper. Later stops switch to pre-norm because deep stacks train more stably that way. This file stays honest about 2017.
Think of a clerk copying a harbour log from one language to another. The encoder is the finished reading of the source page. The decoder writes the new page left to right, never peeking at ink it has not written yet, but it may reread any word on the source page.
The “why” of attention, heads, and residual blocks lives in Deep Cuts. This stop is the runnable skeleton those chapters point at. Read the block chapter →
Teaching config: d_model = 32, H = 4 so d_k = 8, N = 2, vocab 50. Batch of 2. Source length 6, target length 5.
enc–dec · bi / causal / cross · logits (2,5,50)
| Tensor | Shape | Note |
|---|---|---|
| src tokens | (2, 6) | integer ids |
| src embeddings | (2, 6, 32) | lookup × √32 |
| src mask | (2, 1, 1, 6) | 1 = keep, 0 = pad |
| encoder out | (2, 6, 32) | memory for cross-attention |
| tgt tokens | (2, 5) | teacher-forced prefix |
| tgt mask | (2, 1, 5, 5) | causal ∩ pad |
| logits | (2, 5, 50) | one vocab row per target position |
The 2017 paper used 512 / 2048 / 8 heads / 6 layers. Same graph, bigger numbers.
encode, decode, and project stay separate so greedy decode can encode the source once. Masks are built from token ids: padding for the encoder, padding and a lower triangle for the decoder.
"""2017 encoder-decoder Transformer (Vaswani et al.).
This is post-norm, as in the paper: LayerNorm(x + dropout(sublayer(x))).
Modern decoders (GPT-2 onward) flip that to pre-norm.
Teaching default is tiny so a laptop can forward a batch. The published
2017 config lives on the page: d_model 512, d_ff 2048, 8 heads, 6 layers.
"""
from dataclasses import dataclass
import math
import torch
import torch.nn as nn
@dataclass
class TransformerConfig:
dmodel: int = 32
vocabsize: int = 50
max_seq_len: int = 16
dff: int = 64
H: int = 4
N: int = 2
dropout: float = 0.0
norm_eps: float = 1e-6
pad_id: int = 0
bos_id: int = 1
eos_id: int = 2
@property
def dk(self):
return self.dmodel // self.H
# ---------------------------------------------------------------------------
# Masks — padding for the encoder, padding ∩ causal for the decoder
# ---------------------------------------------------------------------------
def padding_mask(token_ids, pad_id=0):
"""1 where the token is real, 0 on pad. Shape (B, 1, 1, S)."""
return (token_ids != pad_id).unsqueeze(1).unsqueeze(2)
def causal_mask(seq_len, device=None):
"""Lower-triangular ones. Shape (1, 1, S, S)."""
return torch.tril(torch.ones(seq_len, seq_len, device=device)).view(1, 1, seq_len, seq_len)
def decoder_mask(tgt_ids, pad_id=0):
"""Causal mask AND target padding. Shape (B, 1, S, S)."""
pad = padding_mask(tgt_ids, pad_id) # (B, 1, 1, S)
causal = causal_mask(tgt_ids.size(1), tgt_ids.device)
return pad & causal.bool()
# ---------------------------------------------------------------------------
# Building blocks
# ---------------------------------------------------------------------------
class InputEmbedding(nn.Module):
"""Token ids → vectors, scaled by sqrt(d_model) as in §3.4."""
def __init__(self, config):
super().__init__()
self.embedding = nn.Embedding(config.vocabsize, config.dmodel)
self.scale = math.sqrt(config.dmodel)
def forward(self, x):
return self.embedding(x) * self.scale
class PositionalEncoding(nn.Module):
"""Fixed sinusoids. PE(pos, 2i) = sin(pos / 10000^(2i/d))."""
def __init__(self, config):
super().__init__()
pe = torch.zeros(config.max_seq_len, config.dmodel)
pos = torch.arange(config.max_seq_len, dtype=torch.float).unsqueeze(1)
div = 1.0 / (10000 ** (torch.arange(0, config.dmodel, 2).float() / config.dmodel))
pe[:, 0::2] = torch.sin(pos * div)
pe[:, 1::2] = torch.cos(pos * div)
self.register_buffer("pe", pe.unsqueeze(0))
def forward(self, x):
return x + self.pe[:, : x.size(1)]
class FeedForward(nn.Module):
"""Position-wise ReLU MLP: d_model → d_ff → d_model."""
def __init__(self, config):
super().__init__()
self.layer1 = nn.Linear(config.dmodel, config.dff)
self.layer2 = nn.Linear(config.dff, config.dmodel)
self.dropout = nn.Dropout(config.dropout)
def forward(self, x):
return self.layer2(self.dropout(torch.relu(self.layer1(x))))
class MultiHeadAttention(nn.Module):
"""Scaled dot-product attention over H heads."""
def __init__(self, config):
super().__init__()
self.h = config.H
self.dk = config.dk
self.dmodel = config.dmodel
self.wq = nn.Linear(config.dmodel, config.dmodel)
self.wk = nn.Linear(config.dmodel, config.dmodel)
self.wv = nn.Linear(config.dmodel, config.dmodel)
self.wo = nn.Linear(config.dmodel, config.dmodel)
self.dropout = nn.Dropout(config.dropout)
def forward(self, q, k, v, mask):
B, S, _ = q.shape
Sk, Sv = k.size(1), v.size(1)
q = self.wq(q).view(B, S, self.h, self.dk).transpose(1, 2)
k = self.wk(k).view(B, Sk, self.h, self.dk).transpose(1, 2)
v = self.wv(v).view(B, Sv, self.h, self.dk).transpose(1, 2)
scores = q @ k.transpose(-1, -2) / math.sqrt(self.dk)
if mask is not None:
# mask == 0 is blocked (pad or future)
scores = scores.masked_fill(mask == 0, float("-inf"))
weights = self.dropout(scores.softmax(dim=-1))
out = (weights @ v).transpose(1, 2).contiguous().view(B, S, self.dmodel)
return self.wo(out)
class ResidualConnection(nn.Module):
"""POST-norm residual: LayerNorm(x + dropout(sublayer(x)))."""
def __init__(self, config):
super().__init__()
self.norm = nn.LayerNorm(config.dmodel, eps=config.norm_eps)
self.dropout = nn.Dropout(config.dropout)
def forward(self, x, sublayer):
return self.norm(x + self.dropout(sublayer(x)))
class EncoderBlock(nn.Module):
def __init__(self, config):
super().__init__()
self.attn = MultiHeadAttention(config)
self.ff = FeedForward(config)
self.res = nn.ModuleList([ResidualConnection(config) for _ in range(2)])
def forward(self, x, mask):
x = self.res[0](x, lambda t: self.attn(t, t, t, mask))
return self.res[1](x, self.ff)
class Encoder(nn.Module):
def __init__(self, config):
super().__init__()
self.layers = nn.ModuleList([EncoderBlock(config) for _ in range(config.N)])
def forward(self, x, mask):
for layer in self.layers:
x = layer(x, mask)
return x
class DecoderBlock(nn.Module):
def __init__(self, config):
super().__init__()
self.self_attn = MultiHeadAttention(config)
self.cross_attn = MultiHeadAttention(config)
self.ff = FeedForward(config)
self.res = nn.ModuleList([ResidualConnection(config) for _ in range(3)])
def forward(self, x, memory, src_mask, tgt_mask):
x = self.res[0](x, lambda t: self.self_attn(t, t, t, tgt_mask))
x = self.res[1](x, lambda t: self.cross_attn(t, memory, memory, src_mask))
return self.res[2](x, self.ff)
class Decoder(nn.Module):
def __init__(self, config):
super().__init__()
self.layers = nn.ModuleList([DecoderBlock(config) for _ in range(config.N)])
def forward(self, x, memory, src_mask, tgt_mask):
for layer in self.layers:
x = layer(x, memory, src_mask, tgt_mask)
return x
class Transformer(nn.Module):
"""Full 2017 encoder-decoder. encode / decode / project stay separate
so greedy decoding can encode the source once."""
def __init__(self, config):
super().__init__()
self.config = config
self.src_embedding = InputEmbedding(config)
self.tgt_embedding = InputEmbedding(config)
self.src_pe = PositionalEncoding(config)
self.tgt_pe = PositionalEncoding(config)
self.drop = nn.Dropout(config.dropout)
self.encoder = Encoder(config)
self.decoder = Decoder(config)
# raw logits — the paper's pre-softmax projection
self.proj = nn.Linear(config.dmodel, config.vocabsize)
def encode(self, src, src_mask):
x = self.drop(self.src_pe(self.src_embedding(src)))
return self.encoder(x, src_mask)
def decode(self, tgt, memory, src_mask, tgt_mask):
x = self.drop(self.tgt_pe(self.tgt_embedding(tgt)))
return self.decoder(x, memory, src_mask, tgt_mask)
def project(self, x):
return self.proj(x)
def forward(self, src, tgt):
src_mask = padding_mask(src, self.config.pad_id)
tgt_mask = decoder_mask(tgt, self.config.pad_id)
memory = self.encode(src, src_mask)
hidden = self.decode(tgt, memory, src_mask, tgt_mask)
return self.project(hidden)
@torch.no_grad()
def greedy_decode(model, src, max_len=None):
"""Encode once, then argmax one target token at a time."""
cfg = model.config
max_len = max_len or cfg.max_seq_len
src_mask = padding_mask(src, cfg.pad_id)
memory = model.encode(src, src_mask)
ys = torch.full((src.size(0), 1), cfg.bos_id, dtype=torch.long, device=src.device)
for _ in range(max_len - 1):
tgt_mask = decoder_mask(ys, cfg.pad_id)
logits = model.project(model.decode(ys, memory, src_mask, tgt_mask))
next_id = logits[:, -1].argmax(dim=-1, keepdim=True)
ys = torch.cat([ys, next_id], dim=1)
if (next_id == cfg.eos_id).all():
break
return ys
Test note. The check script forwards (2, 6) source and (2, 5) target, asserts every row of the shape table, and runs greedy decode from a BOS token.
Calling post-norm “pre-LN”. The residual here is LN(x + f(x)). GPT-2 flips it to x + f(LN(x)). The names are not interchangeable.
Forgetting the causal mask on the target. Teacher forcing feeds the whole target at once. Without the triangle, position 2 can see token 3.
Re-encoding every greedy step. Source tokens do not change. Encode once, then grow the target.
Post-norm: LayerNorm(x + dropout(sublayer(x))). Modern models switch to pre-norm.
(2, 5, 50) — vocab 50.
So inference can encode the source once and loop only on the decoder.