Chapter 15

Coding Transformer Building Blocks

Each question is one small piece of the Transformer — coded from scratch in pure Python/NumPy. Every answer also shows the PyTorch equivalent so you can use both fluently.

Questions: 25
Format: Predict Output + Code + Multiple Choice
Difficulty: ★★★★☆
How this chapter works: Each question builds one Transformer component from scratch. The answer shows the pure Python/NumPy implementation, then a pink callout shows the PyTorch equivalent. By the end, you'll have coded every piece needed to build a Transformer end-to-end.
Input Pipeline — Text to Tensors
Q 01 Predict the Output
# Building a vocabulary from scratch
text = "the cat sat on the mat"
words = text.split()
vocab = {word: idx for idx, word in enumerate(sorted(set(words)))}
print("vocab:", vocab)

token_ids = [vocab[w] for w in words]
print("token_ids:", token_ids)
Output
vocab: {'cat': 0, 'mat': 1, 'on': 2, 'sat': 3, 'the': 4} token_ids: [4, 0, 3, 2, 4, 1]
Why? This is the simplest tokenizer: (1) split on spaces, (2) build a vocab dict from sorted unique words, (3) map each word to its integer ID. "the" appears twice but gets the same ID (4) both times. This is word-level tokenization — real Transformers use subword (BPE/WordPiece) but the concept is identical: text → list of integers.
PyTorch equivalent: token_ids_tensor = torch.tensor(token_ids, dtype=torch.long) — PyTorch needs a LongTensor for embedding lookups. For real tokenizers, use tiktoken (GPT) or AutoTokenizer from HuggingFace.
Q 02 Predict the Output
import numpy as np

# Character-level tokenization (like Karpathy's nanoGPT)
text = "hello"
chars = sorted(set(text))
stoi = {ch: i for i, ch in enumerate(chars)}
itos = {i: ch for ch, i in stoi.items()}

encoded = [stoi[ch] for ch in text]
decoded = "".join([itos[i] for i in encoded])

print("chars:", chars)
print("stoi:", stoi)
print("encoded:", encoded)
print("decoded:", decoded)
Output
chars: ['e', 'h', 'l', 'o'] stoi: {'e': 0, 'h': 1, 'l': 2, 'o': 3} encoded: [1, 0, 2, 2, 3] decoded: hello
Why? Character-level tokenization: vocab = unique sorted characters. stoi (string-to-int) and itos (int-to-string) are inverse mappings. "hello" → [1,0,2,2,3]. Note 'l' appears twice → same ID (2) each time. Decoding reverses it perfectly. This is the approach used in Karpathy's "Let's build GPT from scratch" — simpler than BPE but same principle.
PyTorch: Same logic, then torch.tensor(encoded). For BPE: import tiktoken; enc = tiktoken.get_encoding("cl100k_base"); ids = enc.encode("hello") — this is what GPT-4 uses.
Q 03 Predict the Output
import numpy as np

# Embedding lookup — what nn.Embedding actually does
np.random.seed(0)
vocab_size, d_model = 5, 4
W_emb = np.random.randn(vocab_size, d_model).round(2)

token_ids = [2, 0, 4]
embeddings = W_emb[token_ids]   # fancy indexing = embedding lookup

print("W_emb shape:", W_emb.shape)
print("Embeddings shape:", embeddings.shape)
print("Token 2 embedding:", embeddings[0])
Output
W_emb shape: (5, 4) Embeddings shape: (3, 4) Token 2 embedding: [ 0.98 -0.15 -0.1 0.44]
Why? An embedding layer is just a table: W_emb[token_id] selects row token_id. Three token IDs → three rows selected → shape (3, 4). No matrix multiplication needed — it's a lookup. The weights in W_emb are learned during training. This is how every Transformer converts token IDs into dense vectors.
PyTorch: emb = nn.Embedding(vocab_size, d_model) then emb(torch.tensor([2, 0, 4])). Under the hood, it's the same row-select operation. The weight matrix is emb.weight with shape (vocab_size, d_model).
Q 04 Predict the Output
import numpy as np

# Embedding scaling — why multiply by sqrt(d_model)?
d_model = 512
np.random.seed(42)
emb = np.random.randn(3, d_model)   # 3 tokens

print("Before scaling — std:", emb.std().round(4))
emb_scaled = emb * np.sqrt(d_model)
print("After scaling — std:", emb_scaled.std().round(4))
print("sqrt(512):", np.sqrt(512).round(4))
Output
Before scaling — std: 0.9965 After scaling — std: 22.5485 sqrt(512): 22.6274
Why? In the original Transformer paper, embeddings are multiplied by √d_model before adding positional encodings. Random embeddings have std≈1. Positional encodings (sin/cos) also have values in [-1,1]. Without scaling, the embedding signal would be drowned out by positional encodings when d_model is large. Scaling by √512≈22.6 makes the embedding magnitudes proportionally larger.
PyTorch: self.embedding(x) * math.sqrt(self.d_model) — this line appears in most Transformer implementations right after the embedding lookup. GPT-style models (decoder-only) often skip this scaling.
Positional Encoding
Q 05 Predict the Output
import numpy as np

# Sinusoidal PE from scratch — the full implementation
def positional_encoding(max_len, d_model):
    pe = np.zeros((max_len, d_model))
    pos = np.arange(max_len)[:, None]
    div = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
    pe[:, 0::2] = np.sin(pos * div)
    pe[:, 1::2] = np.cos(pos * div)
    return pe

pe = positional_encoding(4, 6)
print("shape:", pe.shape)
print("pos 0:", pe[0].round(3))
print("pos 1:", pe[1].round(3))
Output
shape: (4, 6) pos 0: [0. 1. 0. 1. 0. 1.] pos 1: [ 0.841 0.54 0.046 0.999 0.002 1. ]
Why? Position 0: all sines are sin(0)=0, all cosines are cos(0)=1 → [0,1,0,1,0,1]. Position 1: different frequencies give different values. High-frequency dimensions (columns 0,1) change rapidly; low-frequency dimensions (columns 4,5) change slowly. This gives each position a unique "fingerprint" that the model can learn to use for ordering information.
PyTorch: Same math but use torch.zeros, torch.arange, torch.exp. Store with self.register_buffer('pe', pe) so it moves to GPU with the model but isn't treated as a learnable parameter. Usage: x = x + self.pe[:x.size(1)].
Q 06 Multiple Choice

Why does the positional encoding formula use exp(arange * -log(10000) / d_model) instead of directly computing 1 / 10000^(2i/d_model)?

Why? 10000^(2i/d_model) can produce astronomically large numbers (e.g. 10000^1 = 10000). Computing in log-space — exp(2i * -log(10000)/d_model) — is mathematically identical but avoids overflow in intermediate calculations. This log-space trick appears throughout ML: whenever you see exp(log(x)) instead of x directly, it's for numerical stability. Same result, safer computation.
Q 07 Predict the Output
import numpy as np

# Adding positional encoding to embeddings
np.random.seed(0)
batch, seq_len, d_model = 2, 3, 4

# Token embeddings (batch, seq, d_model)
tok_emb = np.random.randn(batch, seq_len, d_model).round(2)

# Positional encoding (seq_len, d_model) — no batch dim!
pe = np.zeros((seq_len, d_model))
pe[0] = [0, 1, 0, 1]
pe[1] = [0.84, 0.54, 0.01, 1.0]
pe[2] = [0.91, -0.42, 0.02, 1.0]

result = tok_emb + pe   # broadcasting over batch dim
print("tok_emb[0,0]:", tok_emb[0,0])
print("pe[0]:", pe[0])
print("result[0,0]:", result[0,0])
print("result shape:", result.shape)
Output
tok_emb[0,0]: [ 1.76 0.4 0.98 2.24] pe[0]: [0. 1. 0. 1.] result[0,0]: [1.76 1.4 0.98 3.24] result shape: (2, 3, 4)
Why? The PE (3,4) broadcasts over the batch dimension of tok_emb (2,3,4) → every batch gets the same positional encoding added. This is element-wise addition: [1.76+0, 0.4+1, 0.98+0, 2.24+1] = [1.76, 1.4, 0.98, 3.24]. After this addition, the model knows both what the token is (from embedding) and where it is (from PE).
Self-Attention from Scratch
Q 08 Predict the Output
import numpy as np

# Computing Q, K, V from input — the linear projections
np.random.seed(1)
seq_len, d_model, d_k = 3, 4, 2

x = np.random.randn(seq_len, d_model)
W_q = np.random.randn(d_model, d_k)
W_k = np.random.randn(d_model, d_k)
W_v = np.random.randn(d_model, d_k)

Q = x @ W_q     # (seq, d_model) @ (d_model, d_k) = (seq, d_k)
K = x @ W_k
V = x @ W_v

print("x shape:", x.shape)
print("Q shape:", Q.shape)
print("K shape:", K.shape)
print("V shape:", V.shape)
Output
x shape: (3, 4) Q shape: (3, 2) K shape: (3, 2) V shape: (3, 2)
Why? Three separate linear projections turn the input x into Q, K, V. The weight matrices W_q, W_k, W_v project from d_model down to d_k (the per-head dimension). In practice, d_k = d_model / n_heads. Each projection is just a matrix multiply — no bias in many implementations, though adding bias is optional.
PyTorch: self.W_q = nn.Linear(d_model, d_k, bias=False). Then Q = self.W_q(x). Under the hood it's the same x @ W.T — PyTorch stores the weight transposed.
Q 09 Predict the Output
import numpy as np

# Complete scaled dot-product attention — the 6-line version
def attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)
    if mask is not None:
        scores = np.where(mask == 0, -1e9, scores)
    weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
    weights /= weights.sum(axis=-1, keepdims=True)
    return weights @ V, weights

# Test with 3 tokens, d_k=4, causal mask
np.random.seed(42)
Q = np.random.randn(3, 4)
K = np.random.randn(3, 4)
V = np.random.randn(3, 4)
mask = np.tril(np.ones((3, 3)))

out, w = attention(Q, K, V, mask)
print("output shape:", out.shape)
print("weight row sums:", w.sum(axis=-1).round(4))
Output
output shape: (3, 4) weight row sums: [1. 1. 1.]
Why? This is the complete attention mechanism in 6 lines: (1) compute similarity via Q@K^T, (2) scale by 1/√d_k, (3) apply causal mask, (4) stable softmax, (5) multiply by V. Output shape = same as V: (seq, d_k). Each row of weights sums to 1.0 (softmax guarantee). This function is the heart of every Transformer.
PyTorch: F.scaled_dot_product_attention(Q, K, V, attn_mask=mask, is_causal=True) — a single fused call that includes Flash Attention optimizations. Or manually: scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k).
Q 10 Multiple Choice

In self-attention, Q, K, and V are all derived from the same input x. In cross-attention (used in the decoder), which is different?

Why? In cross-attention, the decoder "queries" the encoder's output. Q = decoder hidden state (what am I looking for?), K = encoder output (what's available?), V = encoder output (what values to retrieve). This lets the decoder attend to any part of the input sequence. Think of Q as the question, K as the index, V as the content.
Multi-Head Attention
Q 11 Predict the Output
import numpy as np

# Split into heads: (batch, seq, d_model) → (batch, n_heads, seq, d_k)
def split_heads(x, n_heads):
    batch, seq_len, d_model = x.shape
    d_k = d_model // n_heads
    return x.reshape(batch, seq_len, n_heads, d_k).transpose(0, 2, 1, 3)

x = np.arange(48).reshape(2, 3, 8)  # batch=2, seq=3, d_model=8
out = split_heads(x, n_heads=2)
print("out shape:", out.shape)
print("head 0, token 0:", out[0, 0, 0])
print("head 1, token 0:", out[0, 1, 0])
Output
out shape: (2, 2, 3, 4) head 0, token 0: [0 1 2 3] head 1, token 0: [4 5 6 7]
Why? We reshape d_model=8 into n_heads=2, d_k=4. Then transpose to put heads before seq_len. Head 0 gets the first 4 elements [0,1,2,3], head 1 gets the next 4 [4,5,6,7]. Each head sees a different "subspace" of the embedding and can learn different attention patterns (e.g., head 0 learns syntax, head 1 learns semantics).
PyTorch: x.view(batch, seq_len, n_heads, d_k).transpose(1, 2) — identical logic. view is PyTorch's reshape (requires contiguous memory).
Q 12 Predict the Output
import numpy as np

# Merge heads back: (batch, n_heads, seq, d_k) → (batch, seq, d_model)
def merge_heads(x):
    batch, n_heads, seq_len, d_k = x.shape
    # transpose back, then reshape to merge heads
    return x.transpose(0, 2, 1, 3).reshape(batch, seq_len, n_heads * d_k)

x = np.arange(48).reshape(2, 2, 3, 4)  # batch=2, heads=2, seq=3, d_k=4
out = merge_heads(x)
print("out shape:", out.shape)
print("token 0:", out[0, 0])
Output
out shape: (2, 3, 8) token 0: [ 0 1 2 3 12 13 14 15]
Why? Merge is the reverse of split: transpose (batch, heads, seq, d_k) → (batch, seq, heads, d_k), then reshape to (batch, seq, heads*d_k) = (batch, seq, d_model). Token 0 combines head 0's output [0,1,2,3] with head 1's output [12,13,14,15] into a single vector [0,1,2,3,12,13,14,15]. After merging, a final linear layer W_o projects back to d_model.
PyTorch: x.transpose(1, 2).contiguous().view(batch, seq_len, -1). The .contiguous() call is needed because transpose creates a non-contiguous view, and view requires contiguous memory.
Layer Norm & Feed-Forward Network
Q 13 Predict the Output
import numpy as np

# Layer normalization from scratch
def layer_norm(x, gamma, beta, eps=1e-5):
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    x_norm = (x - mean) / (std + eps)
    return gamma * x_norm + beta

x = np.array([[1.0, 2.0, 3.0, 4.0]])
gamma = np.ones(4)    # learnable scale (init to 1)
beta = np.zeros(4)     # learnable shift (init to 0)

out = layer_norm(x, gamma, beta)
print("mean:", out.mean(axis=-1).round(4))
print("std:", out.std(axis=-1).round(4))
print("output:", out.round(4))
Output
mean: [-0.] std: [1.] output: [[-1.3416 -0.4472 0.4472 1.3416]]
Why? With gamma=1 and beta=0, layer norm just standardizes each sample to mean≈0, std≈1. The output values are (x-mean)/std: (1-2.5)/1.118 ≈ -1.34, etc. In Transformers, gamma and beta are learnable — they let the model "undo" the normalization if needed. LayerNorm normalizes across the feature dimension (d_model), NOT across the batch — each token is normalized independently.
PyTorch: nn.LayerNorm(d_model). Internally it computes the same thing. Parameters: weight (gamma) and bias (beta), both shape (d_model,).
Q 14 Multiple Choice

What's the difference between LayerNorm and RMSNorm (used in LLaMA)?

Why? RMSNorm = x / sqrt(mean(x²) + eps) * gamma. It skips the mean subtraction step of LayerNorm, making it ~10-15% faster. LLaMA, Mistral, and most modern LLMs use RMSNorm. The insight: the re-centering (subtracting mean) doesn't help much, but the re-scaling (dividing by RMS) is the important part. The formula: rms = sqrt(mean(x**2)); return x / (rms + eps) * gamma.
Q 15 Predict the Output
import numpy as np

# RMSNorm from scratch (used in LLaMA, Mistral)
def rms_norm(x, gamma, eps=1e-5):
    rms = np.sqrt(np.mean(x**2, axis=-1, keepdims=True) + eps)
    return (x / rms) * gamma

x = np.array([[3.0, 4.0]])
gamma = np.ones(2)

rms = np.sqrt(np.mean(x**2))
print("x²:", (x**2).flatten())
print("mean(x²):", np.mean(x**2))
print("rms:", rms.round(4))
print("output:", rms_norm(x, gamma).round(4))
Output
x²: [ 9. 16.] mean(x²): 12.5 rms: 3.5355 output: [[0.8485 1.1314]]
Why? RMS = √(mean(x²)) = √((9+16)/2) = √12.5 ≈ 3.5355. Then x/rms = [3/3.5355, 4/3.5355] = [0.8485, 1.1314]. Note: unlike LayerNorm, the output doesn't have mean=0. It preserves the "direction" of the vector but normalizes its magnitude. LLaMA uses "pre-norm" style: normalize first, then apply attention/FFN.
Q 16 Predict the Output
import numpy as np

# Feed-Forward Network: expand then contract
def ffn(x, W1, b1, W2, b2):
    # x: (seq, d_model) → (seq, d_ff) → (seq, d_model)
    hidden = np.maximum(0, x @ W1 + b1)   # ReLU activation
    return hidden @ W2 + b2

d_model, d_ff = 4, 8    # d_ff is typically 4x d_model
np.random.seed(42)
W1 = np.random.randn(d_model, d_ff) * 0.1
b1 = np.zeros(d_ff)
W2 = np.random.randn(d_ff, d_model) * 0.1
b2 = np.zeros(d_model)

x = np.ones((2, d_model))
out = ffn(x, W1, b1, W2, b2)
print("input shape:", x.shape)
print("output shape:", out.shape)
print("d_ff (hidden):", d_ff)
Output
input shape: (2, 4) output shape: (2, 4) d_ff (hidden): 8
Why? The FFN is: Linear(d_model→d_ff) → ReLU → Linear(d_ff→d_model). It expands to a larger dimension (4→8), applies nonlinearity, then contracts back (8→4). This "bottleneck" pattern lets the model learn complex transformations. In the original Transformer, d_ff = 4 * d_model = 2048 when d_model = 512. Modern LLMs (LLaMA) use SwiGLU instead of ReLU.
PyTorch: nn.Sequential(nn.Linear(d_model, d_ff), nn.ReLU(), nn.Linear(d_ff, d_model)). For SwiGLU (LLaMA): nn.SiLU() with a gating mechanism — F.silu(self.w1(x)) * self.w3(x) then self.w2(...).
Q 17 Predict the Output
import numpy as np

# GELU activation (used in GPT, BERT) vs ReLU
def relu(x):
    return np.maximum(0, x)

def gelu_approx(x):
    # Approximate GELU (the fast version used in practice)
    return 0.5 * x * (1 + np.tanh(np.sqrt(2/np.pi) * (x + 0.044715 * x**3)))

x = np.array([-1.0, -0.5, 0.0, 0.5, 1.0])
print("ReLU:", relu(x))
print("GELU:", gelu_approx(x).round(4))
Output
ReLU: [0. 0. 0. 0.5 1. ] GELU: [-0.1588 -0.1543 0. 0.3457 0.8412]
Why? Key difference: ReLU kills all negative values to exactly 0. GELU is "smooth" — negative values get small but nonzero outputs (-1 → -0.1588). This means gradients still flow through negative values in GELU, which helps training. GELU(x) ≈ x * Φ(x) where Φ is the CDF of a standard normal. GPT-2, BERT, and many modern models use GELU. LLaMA uses SiLU (swish) which is similar: x * sigmoid(x).
PyTorch: F.gelu(x) or nn.GELU(). For SiLU/Swish: F.silu(x) or nn.SiLU().
Assembling a Transformer Block
Q 18 Multiple Choice

In a "pre-norm" Transformer block (used by GPT-2, LLaMA), what's the correct order of operations?

Why? Pre-norm: normalize first, then apply the sub-layer, then add the residual. x = x + Attention(LayerNorm(x)) then x = x + FFN(LayerNorm(x)). The original Transformer paper uses "post-norm" (option A), but pre-norm is more stable for training deep networks and is used by GPT-2, GPT-3, LLaMA, and most modern LLMs.
Q 19 Predict the Output
import numpy as np

# Dropout from scratch — used during training only
def dropout(x, p=0.5, training=True):
    if not training:
        return x
    mask = (np.random.rand(*x.shape) > p).astype(float)
    return x * mask / (1 - p)   # scale up to preserve expected value

np.random.seed(0)
x = np.ones(10)
dropped = dropout(x, p=0.3)
print("original sum:", x.sum())
print("dropped:", dropped.round(2))
print("zeros:", (dropped == 0).sum())
Output
original sum: 10.0 dropped: [1.43 1.43 1.43 1.43 1.43 0. 1.43 1.43 0. 1.43] zeros: 2
Why? Dropout randomly zeroes elements with probability p=0.3. Remaining values are scaled by 1/(1-p) = 1/0.7 ≈ 1.43 so the expected value stays the same. 2 out of 10 elements were dropped (close to the expected 3). In Transformers, dropout is applied after attention weights and after the FFN. At inference, dropout is disabled (returns x unchanged).
PyTorch: nn.Dropout(p=0.1) — automatically handles training/eval mode via model.train() / model.eval(). Typical Transformer dropout rates: 0.1 for large models, 0.0 for very large models.
Q 20 Predict the Output
import numpy as np

# Complete Transformer decoder block (simplified, no batch dim)
def layer_norm(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + eps)

def attention(Q, K, V, mask=None):
    d_k = Q.shape[-1]
    s = Q @ K.T / np.sqrt(d_k)
    if mask is not None:
        s = np.where(mask == 0, -1e9, s)
    w = np.exp(s - s.max(-1, keepdims=True))
    w /= w.sum(-1, keepdims=True)
    return w @ V

def ffn(x):
    return np.maximum(0, x @ np.eye(x.shape[-1]))  # identity for demo

# Pre-norm decoder block
x = np.array([[1.0, 2.0, 3.0],
               [4.0, 5.0, 6.0]])  # (seq=2, d=3)
mask = np.tril(np.ones((2, 2)))

# Sub-layer 1: masked self-attention with residual
x_norm = layer_norm(x)
attn_out = attention(x_norm, x_norm, x_norm, mask)
x = x + attn_out            # residual connection

# Sub-layer 2: FFN with residual
x_norm = layer_norm(x)
ffn_out = ffn(x_norm)
x = x + ffn_out             # residual connection

print("output shape:", x.shape)
print("This is one complete decoder block!")
Output
output shape: (2, 3) This is one complete decoder block!
Why? This is a full pre-norm decoder Transformer block in ~15 lines of NumPy: (1) LayerNorm → self-attention → add residual, (2) LayerNorm → FFN → add residual. Output shape is same as input (2, 3) — Transformer blocks preserve dimensions. Stack N of these blocks and you have GPT. The real version adds learned W_q/W_k/W_v projections, multi-head splitting, dropout, and proper FFN weights.
Output Head & Decoding
Q 21 Predict the Output
import numpy as np

# LM Head: project from d_model to vocab_size → get logits
d_model, vocab_size = 4, 6
np.random.seed(10)
W_out = np.random.randn(d_model, vocab_size) * 0.5

# Last hidden state of last token
h = np.array([1.0, 0.5, -0.3, 0.8])
logits = h @ W_out     # (d_model,) @ (d_model, vocab) = (vocab,)
print("logits shape:", logits.shape)
print("logits:", logits.round(3))
print("predicted token:", np.argmax(logits))
Output
logits shape: (6,) logits: [ 0.506 0.371 0.091 0.329 -0.379 0.52 ] predicted token: 5
Why? The LM head is a linear projection from d_model to vocab_size — it produces a score (logit) for every word in the vocabulary. argmax picks the highest-scoring token (greedy decoding). Token 5 has the highest logit (0.52). In many Transformers, the LM head weight is tied to the embedding matrix (W_out = W_emb.T) — this saves parameters and works well.
PyTorch: self.lm_head = nn.Linear(d_model, vocab_size, bias=False). Weight tying: self.lm_head.weight = self.embedding.weight.
Q 22 Predict the Output
import numpy as np

# Temperature scaling — controlling randomness in generation
def softmax(x):
    e = np.exp(x - x.max())
    return e / e.sum()

logits = np.array([2.0, 1.0, 0.5, 0.1])

print("T=1.0:", softmax(logits / 1.0).round(3))
print("T=0.5:", softmax(logits / 0.5).round(3))
print("T=2.0:", softmax(logits / 2.0).round(3))
Output
T=1.0: [0.467 0.172 0.104 0.07 ] T=0.5: [0.724 0.098 0.036 0.016] T=2.0: [0.345 0.206 0.16 0.135]
Why? Temperature divides logits before softmax. T=0.5 (low): distribution is "peaky" — model is very confident, almost always picks the top token. T=2.0 (high): distribution is "flat" — more random, creative outputs. T→0: becomes argmax (greedy). T→∞: becomes uniform random. This is the "temperature" slider in ChatGPT/Claude.
Q 23 Predict the Output
import numpy as np

# Top-k sampling: only consider the k most likely tokens
def top_k(logits, k):
    top_k_idx = np.argsort(logits)[-k:]        # indices of top-k
    top_k_logits = np.full_like(logits, -np.inf)
    top_k_logits[top_k_idx] = logits[top_k_idx]
    return top_k_logits

logits = np.array([0.1, 3.5, 0.5, 2.8, 0.2])
filtered = top_k(logits, k=2)
print("original:", logits)
print("top-2:", filtered)
print("kept indices:", np.where(filtered > -np.inf)[0])
Output
original: [0.1 3.5 0.5 2.8 0.2] top-2: [-inf 3.5 -inf 2.8 -inf] kept indices: [1 3]
Why? Top-k keeps only the k highest logits, setting the rest to -inf. After softmax, -inf → 0 probability. Here k=2 keeps indices 1 (logit 3.5) and 3 (logit 2.8). This prevents the model from sampling very unlikely tokens. Combined with temperature, top-k sampling gives good quality generation: sample(softmax(top_k(logits/T, k))).
Q 24 Predict the Output
import numpy as np

# Greedy autoregressive decoding loop
# (Simplified — in reality each step runs the full Transformer)
vocab = ["<s>", "the", "cat", "sat", "</s>"]

# Fake "model" that returns predetermined next-token logits
responses = [
    [0, 5, 1, 0, 0],   # after <s> → "the" (idx 1)
    [0, 0, 4, 1, 0],   # after "the" → "cat" (idx 2)
    [0, 0, 0, 3, 1],   # after "cat" → "sat" (idx 3)
    [0, 0, 0, 0, 9],   # after "sat" → </s> (idx 4)
]

tokens = [0]  # start with <s>
for step in range(4):
    logits = np.array(responses[step])
    next_token = np.argmax(logits)
    tokens.append(next_token)
    if next_token == 4:  # </s> — stop
        break

print("token ids:", tokens)
print("decoded:", " ".join(vocab[t] for t in tokens))
Output
token ids: [0, 1, 2, 3, 4] decoded: <s> the cat sat </s>
Why? This is the autoregressive decoding loop: (1) start with <s>, (2) feed tokens to model, get logits, (3) pick the highest-scoring next token (argmax = greedy), (4) append it, (5) repeat until </s> or max length. Each step, the model sees ALL previous tokens (via causal attention). This is how every GPT-style model generates text — one token at a time, left-to-right.
Q 25 Multiple Choice

During training, the Transformer processes the entire sequence in parallel (not token-by-token). How does the causal mask enable this?

Why? The causal (lower-triangular) mask ensures position i can only attend to positions 0..i, even though all positions are processed in one forward pass. This means we can compute the loss for ALL positions simultaneously during training (teacher forcing): predict token 1 from token 0, predict token 2 from tokens 0-1, etc. — all in a single matrix multiply. This massive parallelism is why Transformers train so much faster than RNNs.
← Chapter 14: Tensor Fundamentals Chapter 16: PyTorch for Transformers →