Each question is one small piece of the Transformer — coded from scratch in pure Python/NumPy. Every answer also shows the PyTorch equivalent so you can use both fluently.
# Building a vocabulary from scratch text = "the cat sat on the mat" words = text.split() vocab = {word: idx for idx, word in enumerate(sorted(set(words)))} print("vocab:", vocab) token_ids = [vocab[w] for w in words] print("token_ids:", token_ids)
token_ids_tensor = torch.tensor(token_ids, dtype=torch.long) — PyTorch needs a LongTensor for embedding lookups. For real tokenizers, use tiktoken (GPT) or AutoTokenizer from HuggingFace.
import numpy as np # Character-level tokenization (like Karpathy's nanoGPT) text = "hello" chars = sorted(set(text)) stoi = {ch: i for i, ch in enumerate(chars)} itos = {i: ch for ch, i in stoi.items()} encoded = [stoi[ch] for ch in text] decoded = "".join([itos[i] for i in encoded]) print("chars:", chars) print("stoi:", stoi) print("encoded:", encoded) print("decoded:", decoded)
stoi (string-to-int) and itos (int-to-string) are inverse mappings. "hello" → [1,0,2,2,3]. Note 'l' appears twice → same ID (2) each time. Decoding reverses it perfectly. This is the approach used in Karpathy's "Let's build GPT from scratch" — simpler than BPE but same principle.
torch.tensor(encoded). For BPE: import tiktoken; enc = tiktoken.get_encoding("cl100k_base"); ids = enc.encode("hello") — this is what GPT-4 uses.
import numpy as np # Embedding lookup — what nn.Embedding actually does np.random.seed(0) vocab_size, d_model = 5, 4 W_emb = np.random.randn(vocab_size, d_model).round(2) token_ids = [2, 0, 4] embeddings = W_emb[token_ids] # fancy indexing = embedding lookup print("W_emb shape:", W_emb.shape) print("Embeddings shape:", embeddings.shape) print("Token 2 embedding:", embeddings[0])
W_emb[token_id] selects row token_id. Three token IDs → three rows selected → shape (3, 4). No matrix multiplication needed — it's a lookup. The weights in W_emb are learned during training. This is how every Transformer converts token IDs into dense vectors.
emb = nn.Embedding(vocab_size, d_model) then emb(torch.tensor([2, 0, 4])). Under the hood, it's the same row-select operation. The weight matrix is emb.weight with shape (vocab_size, d_model).
import numpy as np # Embedding scaling — why multiply by sqrt(d_model)? d_model = 512 np.random.seed(42) emb = np.random.randn(3, d_model) # 3 tokens print("Before scaling — std:", emb.std().round(4)) emb_scaled = emb * np.sqrt(d_model) print("After scaling — std:", emb_scaled.std().round(4)) print("sqrt(512):", np.sqrt(512).round(4))
self.embedding(x) * math.sqrt(self.d_model) — this line appears in most Transformer implementations right after the embedding lookup. GPT-style models (decoder-only) often skip this scaling.
import numpy as np # Sinusoidal PE from scratch — the full implementation def positional_encoding(max_len, d_model): pe = np.zeros((max_len, d_model)) pos = np.arange(max_len)[:, None] div = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model)) pe[:, 0::2] = np.sin(pos * div) pe[:, 1::2] = np.cos(pos * div) return pe pe = positional_encoding(4, 6) print("shape:", pe.shape) print("pos 0:", pe[0].round(3)) print("pos 1:", pe[1].round(3))
torch.zeros, torch.arange, torch.exp. Store with self.register_buffer('pe', pe) so it moves to GPU with the model but isn't treated as a learnable parameter. Usage: x = x + self.pe[:x.size(1)].
Why does the positional encoding formula use exp(arange * -log(10000) / d_model) instead of directly computing 1 / 10000^(2i/d_model)?
10000^(2i/d_model) can produce astronomically large numbers (e.g. 10000^1 = 10000). Computing in log-space — exp(2i * -log(10000)/d_model) — is mathematically identical but avoids overflow in intermediate calculations. This log-space trick appears throughout ML: whenever you see exp(log(x)) instead of x directly, it's for numerical stability. Same result, safer computation.
import numpy as np # Adding positional encoding to embeddings np.random.seed(0) batch, seq_len, d_model = 2, 3, 4 # Token embeddings (batch, seq, d_model) tok_emb = np.random.randn(batch, seq_len, d_model).round(2) # Positional encoding (seq_len, d_model) — no batch dim! pe = np.zeros((seq_len, d_model)) pe[0] = [0, 1, 0, 1] pe[1] = [0.84, 0.54, 0.01, 1.0] pe[2] = [0.91, -0.42, 0.02, 1.0] result = tok_emb + pe # broadcasting over batch dim print("tok_emb[0,0]:", tok_emb[0,0]) print("pe[0]:", pe[0]) print("result[0,0]:", result[0,0]) print("result shape:", result.shape)
import numpy as np # Computing Q, K, V from input — the linear projections np.random.seed(1) seq_len, d_model, d_k = 3, 4, 2 x = np.random.randn(seq_len, d_model) W_q = np.random.randn(d_model, d_k) W_k = np.random.randn(d_model, d_k) W_v = np.random.randn(d_model, d_k) Q = x @ W_q # (seq, d_model) @ (d_model, d_k) = (seq, d_k) K = x @ W_k V = x @ W_v print("x shape:", x.shape) print("Q shape:", Q.shape) print("K shape:", K.shape) print("V shape:", V.shape)
self.W_q = nn.Linear(d_model, d_k, bias=False). Then Q = self.W_q(x). Under the hood it's the same x @ W.T — PyTorch stores the weight transposed.
import numpy as np # Complete scaled dot-product attention — the 6-line version def attention(Q, K, V, mask=None): d_k = Q.shape[-1] scores = Q @ K.T / np.sqrt(d_k) if mask is not None: scores = np.where(mask == 0, -1e9, scores) weights = np.exp(scores - scores.max(axis=-1, keepdims=True)) weights /= weights.sum(axis=-1, keepdims=True) return weights @ V, weights # Test with 3 tokens, d_k=4, causal mask np.random.seed(42) Q = np.random.randn(3, 4) K = np.random.randn(3, 4) V = np.random.randn(3, 4) mask = np.tril(np.ones((3, 3))) out, w = attention(Q, K, V, mask) print("output shape:", out.shape) print("weight row sums:", w.sum(axis=-1).round(4))
F.scaled_dot_product_attention(Q, K, V, attn_mask=mask, is_causal=True) — a single fused call that includes Flash Attention optimizations. Or manually: scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k).
In self-attention, Q, K, and V are all derived from the same input x. In cross-attention (used in the decoder), which is different?
import numpy as np # Split into heads: (batch, seq, d_model) → (batch, n_heads, seq, d_k) def split_heads(x, n_heads): batch, seq_len, d_model = x.shape d_k = d_model // n_heads return x.reshape(batch, seq_len, n_heads, d_k).transpose(0, 2, 1, 3) x = np.arange(48).reshape(2, 3, 8) # batch=2, seq=3, d_model=8 out = split_heads(x, n_heads=2) print("out shape:", out.shape) print("head 0, token 0:", out[0, 0, 0]) print("head 1, token 0:", out[0, 1, 0])
x.view(batch, seq_len, n_heads, d_k).transpose(1, 2) — identical logic. view is PyTorch's reshape (requires contiguous memory).
import numpy as np # Merge heads back: (batch, n_heads, seq, d_k) → (batch, seq, d_model) def merge_heads(x): batch, n_heads, seq_len, d_k = x.shape # transpose back, then reshape to merge heads return x.transpose(0, 2, 1, 3).reshape(batch, seq_len, n_heads * d_k) x = np.arange(48).reshape(2, 2, 3, 4) # batch=2, heads=2, seq=3, d_k=4 out = merge_heads(x) print("out shape:", out.shape) print("token 0:", out[0, 0])
W_o projects back to d_model.
x.transpose(1, 2).contiguous().view(batch, seq_len, -1). The .contiguous() call is needed because transpose creates a non-contiguous view, and view requires contiguous memory.
import numpy as np # Layer normalization from scratch def layer_norm(x, gamma, beta, eps=1e-5): mean = x.mean(axis=-1, keepdims=True) std = x.std(axis=-1, keepdims=True) x_norm = (x - mean) / (std + eps) return gamma * x_norm + beta x = np.array([[1.0, 2.0, 3.0, 4.0]]) gamma = np.ones(4) # learnable scale (init to 1) beta = np.zeros(4) # learnable shift (init to 0) out = layer_norm(x, gamma, beta) print("mean:", out.mean(axis=-1).round(4)) print("std:", out.std(axis=-1).round(4)) print("output:", out.round(4))
nn.LayerNorm(d_model). Internally it computes the same thing. Parameters: weight (gamma) and bias (beta), both shape (d_model,).
What's the difference between LayerNorm and RMSNorm (used in LLaMA)?
x / sqrt(mean(x²) + eps) * gamma. It skips the mean subtraction step of LayerNorm, making it ~10-15% faster. LLaMA, Mistral, and most modern LLMs use RMSNorm. The insight: the re-centering (subtracting mean) doesn't help much, but the re-scaling (dividing by RMS) is the important part. The formula: rms = sqrt(mean(x**2)); return x / (rms + eps) * gamma.
import numpy as np # RMSNorm from scratch (used in LLaMA, Mistral) def rms_norm(x, gamma, eps=1e-5): rms = np.sqrt(np.mean(x**2, axis=-1, keepdims=True) + eps) return (x / rms) * gamma x = np.array([[3.0, 4.0]]) gamma = np.ones(2) rms = np.sqrt(np.mean(x**2)) print("x²:", (x**2).flatten()) print("mean(x²):", np.mean(x**2)) print("rms:", rms.round(4)) print("output:", rms_norm(x, gamma).round(4))
import numpy as np # Feed-Forward Network: expand then contract def ffn(x, W1, b1, W2, b2): # x: (seq, d_model) → (seq, d_ff) → (seq, d_model) hidden = np.maximum(0, x @ W1 + b1) # ReLU activation return hidden @ W2 + b2 d_model, d_ff = 4, 8 # d_ff is typically 4x d_model np.random.seed(42) W1 = np.random.randn(d_model, d_ff) * 0.1 b1 = np.zeros(d_ff) W2 = np.random.randn(d_ff, d_model) * 0.1 b2 = np.zeros(d_model) x = np.ones((2, d_model)) out = ffn(x, W1, b1, W2, b2) print("input shape:", x.shape) print("output shape:", out.shape) print("d_ff (hidden):", d_ff)
nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model)). For SwiGLU (LLaMA): nn.SiLU() with a gating mechanism — F.silu(self.w1(x)) * self.w3(x) then self.w2(...).
import numpy as np # GELU activation (used in GPT, BERT) vs ReLU def relu(x): return np.maximum(0, x) def gelu_approx(x): # Approximate GELU (the fast version used in practice) return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3))) x = np.array([-1.0, -0.5, 0.0, 0.5, 1.0]) print("ReLU:", relu(x)) print("GELU:", gelu_approx(x).round(4))
F.gelu(x) or nn.GELU(). For SiLU/Swish: F.silu(x) or nn.SiLU().
In a "pre-norm" Transformer block (used by GPT-2, LLaMA), what's the correct order of operations?
x = x + Attention(LayerNorm(x)) then x = x + FFN(LayerNorm(x)). The original Transformer paper uses "post-norm" (option A), but pre-norm is more stable for training deep networks and is used by GPT-2, GPT-3, LLaMA, and most modern LLMs.
import numpy as np # Dropout from scratch — used during training only def dropout(x, p=0.5, training=True): if not training: return x mask = (np.random.rand(*x.shape) > p).astype(float) return x * mask / (1 - p) # scale up to preserve expected value np.random.seed(0) x = np.ones(10) dropped = dropout(x, p=0.3) print("original sum:", x.sum()) print("dropped:", dropped.round(2)) print("zeros:", (dropped == 0).sum())
nn.Dropout(p=0.1) — automatically handles training/eval mode via model.train() / model.eval(). Typical Transformer dropout rates: 0.1 for large models, 0.0 for very large models.
import numpy as np # Complete Transformer decoder block (simplified, no batch dim) def layer_norm(x, eps=1e-5): return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + eps) def attention(Q, K, V, mask=None): d_k = Q.shape[-1] s = Q @ K.T / np.sqrt(d_k) if mask is not None: s = np.where(mask == 0, -1e9, s) w = np.exp(s - s.max(-1, keepdims=True)) w /= w.sum(-1, keepdims=True) return w @ V def ffn(x): return np.maximum(0, x @ np.eye(x.shape[-1])) # identity for demo # Pre-norm decoder block x = np.array([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]) # (seq=2, d=3) mask = np.tril(np.ones((2, 2))) # Sub-layer 1: masked self-attention with residual x_norm = layer_norm(x) attn_out = attention(x_norm, x_norm, x_norm, mask) x = x + attn_out # residual connection # Sub-layer 2: FFN with residual x_norm = layer_norm(x) ffn_out = ffn(x_norm) x = x + ffn_out # residual connection print("output shape:", x.shape) print("This is one complete decoder block!")
import numpy as np # LM Head: project from d_model to vocab_size → get logits d_model, vocab_size = 4, 6 np.random.seed(10) W_out = np.random.randn(d_model, vocab_size) * 0.5 # Last hidden state of last token h = np.array([1.0, 0.5, -0.3, 0.8]) logits = h @ W_out # (d_model,) @ (d_model, vocab) = (vocab,) print("logits shape:", logits.shape) print("logits:", logits.round(3)) print("predicted token:", np.argmax(logits))
argmax picks the highest-scoring token (greedy decoding). Token 5 has the highest logit (0.52). In many Transformers, the LM head weight is tied to the embedding matrix (W_out = W_emb.T) — this saves parameters and works well.
self.lm_head = nn.Linear(d_model, vocab_size, bias=False). Weight tying: self.lm_head.weight = self.embedding.weight.
import numpy as np # Temperature scaling — controlling randomness in generation def softmax(x): e = np.exp(x - x.max()) return e / e.sum() logits = np.array([2.0, 1.0, 0.5, 0.1]) print("T=1.0:", softmax(logits / 1.0).round(3)) print("T=0.5:", softmax(logits / 0.5).round(3)) print("T=2.0:", softmax(logits / 2.0).round(3))
import numpy as np # Top-k sampling: only consider the k most likely tokens def top_k(logits, k): top_k_idx = np.argsort(logits)[-k:] # indices of top-k top_k_logits = np.full_like(logits, -np.inf) top_k_logits[top_k_idx] = logits[top_k_idx] return top_k_logits logits = np.array([0.1, 3.5, 0.5, 2.8, 0.2]) filtered = top_k(logits, k=2) print("original:", logits) print("top-2:", filtered) print("kept indices:", np.where(filtered > -np.inf)[0])
sample(softmax(top_k(logits/T, k))).
import numpy as np # Greedy autoregressive decoding loop # (Simplified — in reality each step runs the full Transformer) vocab = ["<s>", "the", "cat", "sat", "</s>"] # Fake "model" that returns predetermined next-token logits responses = [ [0, 5, 1, 0, 0], # after <s> → "the" (idx 1) [0, 0, 4, 1, 0], # after "the" → "cat" (idx 2) [0, 0, 0, 3, 1], # after "cat" → "sat" (idx 3) [0, 0, 0, 0, 9], # after "sat" → </s> (idx 4) ] tokens = [0] # start with <s> for step in range(4): logits = np.array(responses[step]) next_token = np.argmax(logits) tokens.append(next_token) if next_token == 4: # </s> — stop break print("token ids:", tokens) print("decoded:", " ".join(vocab[t] for t in tokens))
During training, the Transformer processes the entire sequence in parallel (not token-by-token). How does the causal mask enable this?