Before you code a Transformer, you need to be fluent in dimension manipulation. Reshape, transpose, broadcast, matmul, softmax — these are the real building blocks.
import numpy as np # Simulating a batch of 2 sequences, each 4 tokens, embedding dim 6 x = np.arange(48).reshape(2, 4, 6) print(x.shape) print(x[0, 2, 3])
arange(48) creates [0..47]. reshape(2,4,6) gives shape (batch=2, seq_len=4, d_model=6). Element at [0,2,3] = 0*24 + 2*6 + 3 = 15. In Transformers, you'll constantly index into (batch, seq_len, d_model) tensors. Know your strides!
import numpy as np x = np.arange(24).reshape(2, 3, 4) # Reshape to merge last two dims (like flattening seq_len * d_model) y = x.reshape(2, -1) print(y.shape) print(y[1, 0])
reshape(2, -1) keeps the batch dimension and flattens the rest: 3*4=12. The element y[1,0] is x[1,0,0] = 12. This pattern appears when you flatten multi-head outputs back into a single vector — .reshape(batch, seq_len, d_model) after attention.
import numpy as np # Splitting d_model=8 into n_heads=2, d_k=4 x = np.arange(24).reshape(3, 8) # (seq_len=3, d_model=8) y = x.reshape(3, 2, 4) # (seq_len, n_heads, d_k) print(y.shape) print(y[0, 1, 0])
(batch, seq, d_model) → (batch, seq, n_heads, d_k) — is the core of multi-head attention splitting.
In multi-head attention, after splitting (batch, seq_len, d_model) into (batch, seq_len, n_heads, d_k), we transpose to get (batch, n_heads, seq_len, d_k). Which NumPy call does this transpose?
transpose(0, 2, 1, 3). This swaps seq_len and n_heads so each head can independently attend over the full sequence. You'll see this in every multi-head attention implementation.
import numpy as np x = np.arange(12).reshape(3, 4) print(x.T.shape) print(x.T[0]) # first row of the transpose
.T transposes (3,4) → (4,3). The first row of the transpose is the first column of the original: [0, 4, 8]. Transposing the Key matrix (so you can do Q @ K^T) uses this exact operation. In 2D, .T is clean. For higher dims, use .transpose() with explicit axis order.
import numpy as np # Adding positional encoding to token embeddings embeddings = np.ones((2, 4, 3)) # (batch=2, seq=4, d=3) pos_enc = np.array([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9], [1.0, 1.1, 1.2]]) # (seq=4, d=3) result = embeddings + pos_enc print(result.shape) print(result[0, 0]) print(result[1, 0])
import numpy as np # Applying a causal mask via broadcasting scores = np.ones((3, 3)) # (seq_len, seq_len) mask = np.tril(np.ones((3, 3))) # lower-triangle = 1 print(mask) print() result = np.where(mask == 0, -1e9, scores) print(result)
np.tril creates a lower-triangular matrix (1s below diagonal, 0s above). np.where replaces 0-positions with -1e9 (effectively -infinity after softmax). This is the causal mask in decoder self-attention — token i can only attend to tokens 0..i. After softmax, -1e9 becomes ~0, preventing future-token leakage.
You have attention scores of shape (2, 4, 5, 5) — (batch, heads, seq, seq). You want to apply a causal mask of shape (5, 5). Does broadcasting work?
[None, None, :, :] makes the intent clearer).
import numpy as np # Scaling attention scores by sqrt(d_k) d_k = 64 scores = np.array([[2.0, 4.0], [1.0, 3.0]]) scaled = scores / np.sqrt(d_k) print(np.sqrt(d_k)) print(scaled)
sqrt(64) = 8.0. Dividing by a scalar broadcasts to every element. This is the "scaled" in "scaled dot-product attention." Without this scaling, when d_k is large, dot products grow large, pushing softmax into saturation regions where gradients vanish. The 1/√d_k factor keeps the variance of the dot products around 1.
import numpy as np Q = np.ones((3, 4)) # (seq_len=3, d_k=4) K = np.ones((3, 4)) # (seq_len=3, d_k=4) # Q @ K^T → (seq, d_k) @ (d_k, seq) = (seq, seq) scores = Q @ K.T print(scores.shape) print(scores)
Q @ K.T is the dot-product attention score matrix. (3,4) @ (4,3) = (3,3). Each entry is the dot product of a query with a key vector: sum of 1*1 four times = 4. This (seq_len, seq_len) matrix tells you how much each token "attends to" every other token. The @ operator is Python's matrix multiply (same as np.matmul).
import numpy as np # Batched matmul: attention for 2 batches simultaneously Q = np.ones((2, 3, 4)) # (batch, seq, d_k) K = np.ones((2, 3, 4)) # Q @ K^T — NumPy broadcasts matmul over batch dims scores = Q @ K.transpose(0, 2, 1) print(scores.shape) print(scores[0, 0])
@ treats the last two dimensions as the matrix and broadcasts over all leading dimensions. K.transpose(0,2,1) swaps axes 1 and 2: (2,3,4) → (2,4,3). Then (2,3,4) @ (2,4,3) = (2,3,3) — one attention score matrix per batch. This is the pattern for batched multi-head attention.
import numpy as np # einsum: another way to write Q @ K^T with batch + heads Q = np.ones((2, 4, 3, 8)) # (batch, heads, seq, d_k) K = np.ones((2, 4, 3, 8)) scores = np.einsum('bhqd,bhkd->bhqk', Q, K) print(scores.shape) print(scores[0, 0, 0, 0])
einsum('bhqd,bhkd->bhqk', Q, K) reads as: "for each batch b, head h, query position q, key position k — sum over d." This produces the attention score matrix (batch, heads, query_seq, key_seq). Each entry is 8.0 because d_k=8 and all values are 1. einsum is extremely powerful for Transformer math — it makes the dimension intent explicit.
import numpy as np def softmax(x): e_x = np.exp(x) return e_x / e_x.sum(axis=-1, keepdims=True) scores = np.array([[1.0, 2.0, 3.0]]) print(softmax(scores).round(4)) print(softmax(scores).sum()) # should sum to 1
axis=-1 means we softmax along the last dimension (each row independently). keepdims=True preserves shape for broadcasting. But this version has a bug — it's not numerically stable! See the next question.
The naive softmax exp(x) / sum(exp(x)) can overflow for large values. What's the standard fix used in every production Transformer?
max(x) from each element is the standard "stable softmax" trick. It doesn't change the output (the max cancels in numerator/denominator) but ensures the largest exponent is exp(0)=1, preventing overflow. The stable version: x_shifted = x - max(x); exp(x_shifted) / sum(exp(x_shifted)). Every deep learning framework (PyTorch, JAX) does this internally.
import numpy as np def stable_softmax(x): x_max = x.max(axis=-1, keepdims=True) e_x = np.exp(x - x_max) return e_x / e_x.sum(axis=-1, keepdims=True) # Apply softmax row-wise to a (2, 3) attention score matrix scores = np.array([[1.0, -1e9, -1e9], [1.0, 1.0, -1e9]]) print(stable_softmax(scores).round(4))
import numpy as np # Simulating an embedding lookup table vocab_size = 5 d_model = 3 embedding_table = np.arange(15).reshape(vocab_size, d_model).astype(float) print("Table:") print(embedding_table) token_ids = np.array([2, 0, 4]) embeddings = embedding_table[token_ids] print("\nLookup result:") print(embeddings)
embedding_table[token_ids] is fancy indexing — it grabs rows 2, 0, and 4. This is exactly how nn.Embedding works under the hood: it's just a lookup table indexed by token IDs. No matrix multiply needed for the forward pass — it's a simple row selection. The embedding table shape is always (vocab_size, d_model).
import numpy as np # Boolean masking — used for padding masks x = np.array([[1.5, 2.3, 0.0, 0.0], [3.1, 0.7, 1.2, 0.0]]) pad_mask = (x != 0) print(pad_mask.astype(int)) print(pad_mask.sum(axis=1)) # actual lengths per sequence
x != 0 creates a boolean mask where True = real token, False = padding. Sequence 0 has 2 real tokens, sequence 1 has 3. In Transformers, you use this padding mask to prevent attention to pad tokens by setting their scores to -inf before softmax (similar to the causal mask, but data-dependent).
import numpy as np # Concatenating multi-head outputs head_0 = np.ones((2, 3, 4)) # (batch, seq, d_k) head_1 = np.ones((2, 3, 4)) * 2 # Concat along last dim → (batch, seq, d_model) merged = np.concatenate([head_0, head_1], axis=-1) print(merged.shape) print(merged[0, 0])
concatenate(axis=-1) joins along the last dimension: d_k=4 + d_k=4 = d_model=8. This is how multi-head attention merges outputs: each head produces (batch, seq, d_k), then we concat all heads to get back (batch, seq, d_model). After this, a linear projection (W_o) maps it back to d_model.
What's the difference between np.stack([a, b]) and np.concatenate([a, b]) when a and b are both shape (3, 4)?
np.stack creates a new dimension and stacks arrays along it: two (3,4) arrays → (2,3,4). np.concatenate joins along an existing dimension (default axis=0): two (3,4) arrays → (6,4). When building Transformers: use stack to create a batch dimension from individual sequences, and concatenate to merge head outputs along the feature dimension.
import numpy as np # Layer normalization from scratch x = np.array([[1.0, 2.0, 3.0], [4.0, 4.0, 4.0]]) mean = x.mean(axis=-1, keepdims=True) std = x.std(axis=-1, keepdims=True) normalized = (x - mean) / (std + 1e-5) print("mean:", mean.flatten()) print("std:", std.flatten().round(4)) print("norm:", normalized.round(4))
axis=-1 means normalize across the d_model dimension.
import numpy as np # Residual connection pattern x = np.array([1.0, 2.0, 3.0]) def sublayer(inp): return inp * 0.5 # pretend attention/FFN output output = x + sublayer(x) # residual connection print(output)
x + sublayer(x) is a residual (skip) connection. The output is [1+0.5, 2+1, 3+1.5] = [1.5, 3, 4.5]. In Transformers, every sub-layer has this pattern: output = LayerNorm(x + SubLayer(x)). Residuals solve the vanishing gradient problem — gradients can flow directly through the addition, even in very deep networks.
import numpy as np # Sinusoidal positional encoding — the core formula d_model = 4 pos = np.array([0, 1, 2])[:, None] # (3, 1) i = np.arange(0, d_model, 2)[None, :] # (1, 2) print("pos shape:", pos.shape) print("i shape:", i.shape) print("i values:", i)
pos is (3,1) — positions 0,1,2. i is (1,2) — dimension indices 0,2 (even indices). When you compute pos / 10000^(i/d_model), broadcasting creates a (3,2) matrix — one row per position, one column per frequency. Then you apply sin to even columns and cos to odd columns.
import numpy as np # Full sinusoidal positional encoding for 3 positions, d_model=4 d_model = 4 max_len = 3 pe = np.zeros((max_len, d_model)) pos = np.arange(max_len)[:, None] div_term = np.exp(np.arange(0, d_model, 2) * -np.log(10000.0) / d_model) pe[:, 0::2] = np.sin(pos * div_term) # even indices pe[:, 1::2] = np.cos(pos * div_term) # odd indices print("PE shape:", pe.shape) print("Position 0:", pe[0].round(4)) print("Position 1:", pe[1].round(4))
You have a tensor of shape (batch=2, seq=5, d_model=512). You want to split it into 8 heads for multi-head attention. What's the correct sequence of operations?
import numpy as np # Full single-head attention in 5 lines np.random.seed(42) seq_len, d_k = 3, 4 Q = np.random.randn(seq_len, d_k) K = np.random.randn(seq_len, d_k) V = np.random.randn(seq_len, d_k) scores = Q @ K.T / np.sqrt(d_k) # (3,3) mask = np.tril(np.ones((seq_len, seq_len))) scores = np.where(mask == 0, -1e9, scores) weights = np.exp(scores - scores.max(axis=-1, keepdims=True)) weights = weights / weights.sum(axis=-1, keepdims=True) output = weights @ V # (3,4) print("output shape:", output.shape) print("weights row 0 sum:", weights[0].sum().round(4)) print("weights row 2 nonzero:", (weights[2] > 0.001).sum())
weights @ V produces a weighted combination of value vectors. If you understand these 5 lines, you understand the core of every Transformer.