Chapter 14

NumPy & Tensor Fundamentals for Transformers

Before you code a Transformer, you need to be fluent in dimension manipulation. Reshape, transpose, broadcast, matmul, softmax — these are the real building blocks.

Questions: 25
Format: Predict Output + Multiple Choice
Difficulty: ★★★☆☆
Why this chapter? Every Transformer is really just a sequence of tensor operations — reshaping for multi-head attention, broadcasting for masks, matrix multiplications for Q·K^T. Master these small patterns and the full Transformer code becomes obvious. Each question maps to a real operation you'll use when building Transformers.
Shape & Reshape
Q 01 Predict the Output
import numpy as np

# Simulating a batch of 2 sequences, each 4 tokens, embedding dim 6
x = np.arange(48).reshape(2, 4, 6)
print(x.shape)
print(x[0, 2, 3])
Output
(2, 4, 6) 15
Why? arange(48) creates [0..47]. reshape(2,4,6) gives shape (batch=2, seq_len=4, d_model=6). Element at [0,2,3] = 0*24 + 2*6 + 3 = 15. In Transformers, you'll constantly index into (batch, seq_len, d_model) tensors. Know your strides!
Q 02 Predict the Output
import numpy as np

x = np.arange(24).reshape(2, 3, 4)
# Reshape to merge last two dims (like flattening seq_len * d_model)
y = x.reshape(2, -1)
print(y.shape)
print(y[1, 0])
Output
(2, 12) 12
Why? reshape(2, -1) keeps the batch dimension and flattens the rest: 3*4=12. The element y[1,0] is x[1,0,0] = 12. This pattern appears when you flatten multi-head outputs back into a single vector — .reshape(batch, seq_len, d_model) after attention.
Q 03 Predict the Output
import numpy as np

# Splitting d_model=8 into n_heads=2, d_k=4
x = np.arange(24).reshape(3, 8)   # (seq_len=3, d_model=8)
y = x.reshape(3, 2, 4)              # (seq_len, n_heads, d_k)
print(y.shape)
print(y[0, 1, 0])
Output
(3, 2, 4) 4
Why? We split d_model=8 into n_heads=2, d_k=4. Element y[0,1,0] = the first element of the second head for token 0, which is x[0,4] = 4. This exact reshape — (batch, seq, d_model) → (batch, seq, n_heads, d_k) — is the core of multi-head attention splitting.
Q 04 Multiple Choice

In multi-head attention, after splitting (batch, seq_len, d_model) into (batch, seq_len, n_heads, d_k), we transpose to get (batch, n_heads, seq_len, d_k). Which NumPy call does this transpose?

Why? We want (batch, seq_len, n_heads, d_k) → (batch, n_heads, seq_len, d_k). That means axes 0→0, 1→2, 2→1, 3→3. So transpose(0, 2, 1, 3). This swaps seq_len and n_heads so each head can independently attend over the full sequence. You'll see this in every multi-head attention implementation.
Q 05 Predict the Output
import numpy as np

x = np.arange(12).reshape(3, 4)
print(x.T.shape)
print(x.T[0])          # first row of the transpose
Output
(4, 3) [0 4 8]
Why? .T transposes (3,4) → (4,3). The first row of the transpose is the first column of the original: [0, 4, 8]. Transposing the Key matrix (so you can do Q @ K^T) uses this exact operation. In 2D, .T is clean. For higher dims, use .transpose() with explicit axis order.
Broadcasting
Q 06 Predict the Output
import numpy as np

# Adding positional encoding to token embeddings
embeddings = np.ones((2, 4, 3))   # (batch=2, seq=4, d=3)
pos_enc = np.array([[0.1, 0.2, 0.3],
                     [0.4, 0.5, 0.6],
                     [0.7, 0.8, 0.9],
                     [1.0, 1.1, 1.2]])  # (seq=4, d=3)

result = embeddings + pos_enc
print(result.shape)
print(result[0, 0])
print(result[1, 0])
Output
(2, 4, 3) [1.1 1.2 1.3] [1.1 1.2 1.3]
Why? Broadcasting! pos_enc has shape (4,3) and embeddings has shape (2,4,3). NumPy automatically broadcasts pos_enc to (1,4,3) → (2,4,3). Both batches get the same positional encoding added — exactly how positional encoding works in Transformers. The position info is independent of which batch item it is.
Q 07 Predict the Output
import numpy as np

# Applying a causal mask via broadcasting
scores = np.ones((3, 3))          # (seq_len, seq_len)
mask = np.tril(np.ones((3, 3)))   # lower-triangle = 1
print(mask)
print()
result = np.where(mask == 0, -1e9, scores)
print(result)
Output
[[1. 0. 0.] [1. 1. 0.] [1. 1. 1.]] [[ 1.e+00 -1.e+09 -1.e+09] [ 1.e+00 1.e+00 -1.e+09] [ 1.e+00 1.e+00 1.e+00]]
Why? np.tril creates a lower-triangular matrix (1s below diagonal, 0s above). np.where replaces 0-positions with -1e9 (effectively -infinity after softmax). This is the causal mask in decoder self-attention — token i can only attend to tokens 0..i. After softmax, -1e9 becomes ~0, preventing future-token leakage.
Q 08 Multiple Choice

You have attention scores of shape (2, 4, 5, 5) — (batch, heads, seq, seq). You want to apply a causal mask of shape (5, 5). Does broadcasting work?

Why? NumPy/PyTorch broadcasting rules: dimensions are compared right-to-left. (5,5) vs (2,4,5,5) — the last two dims match, and missing dims are treated as 1. So (5,5) → (1,1,5,5) → (2,4,5,5) automatically. The same mask is applied to every batch and every head. No manual reshape needed (though adding [None, None, :, :] makes the intent clearer).
Q 09 Predict the Output
import numpy as np

# Scaling attention scores by sqrt(d_k)
d_k = 64
scores = np.array([[2.0, 4.0], [1.0, 3.0]])
scaled = scores / np.sqrt(d_k)
print(np.sqrt(d_k))
print(scaled)
Output
8.0 [[0.25 0.5 ] [0.125 0.375]]
Why? sqrt(64) = 8.0. Dividing by a scalar broadcasts to every element. This is the "scaled" in "scaled dot-product attention." Without this scaling, when d_k is large, dot products grow large, pushing softmax into saturation regions where gradients vanish. The 1/√d_k factor keeps the variance of the dot products around 1.
Matrix Multiplication
Q 10 Predict the Output
import numpy as np

Q = np.ones((3, 4))   # (seq_len=3, d_k=4)
K = np.ones((3, 4))   # (seq_len=3, d_k=4)

# Q @ K^T  →  (seq, d_k) @ (d_k, seq) = (seq, seq)
scores = Q @ K.T
print(scores.shape)
print(scores)
Output
(3, 3) [[4. 4. 4.] [4. 4. 4.] [4. 4. 4.]]
Why? Q @ K.T is the dot-product attention score matrix. (3,4) @ (4,3) = (3,3). Each entry is the dot product of a query with a key vector: sum of 1*1 four times = 4. This (seq_len, seq_len) matrix tells you how much each token "attends to" every other token. The @ operator is Python's matrix multiply (same as np.matmul).
Q 11 Predict the Output
import numpy as np

# Batched matmul: attention for 2 batches simultaneously
Q = np.ones((2, 3, 4))   # (batch, seq, d_k)
K = np.ones((2, 3, 4))

# Q @ K^T — NumPy broadcasts matmul over batch dims
scores = Q @ K.transpose(0, 2, 1)
print(scores.shape)
print(scores[0, 0])
Output
(2, 3, 3) [4. 4. 4.]
Why? For 3D+ tensors, @ treats the last two dimensions as the matrix and broadcasts over all leading dimensions. K.transpose(0,2,1) swaps axes 1 and 2: (2,3,4) → (2,4,3). Then (2,3,4) @ (2,4,3) = (2,3,3) — one attention score matrix per batch. This is the pattern for batched multi-head attention.
Q 12 Predict the Output
import numpy as np

# einsum: another way to write Q @ K^T with batch + heads
Q = np.ones((2, 4, 3, 8))  # (batch, heads, seq, d_k)
K = np.ones((2, 4, 3, 8))

scores = np.einsum('bhqd,bhkd->bhqk', Q, K)
print(scores.shape)
print(scores[0, 0, 0, 0])
Output
(2, 4, 3, 3) 8.0
Why? einsum('bhqd,bhkd->bhqk', Q, K) reads as: "for each batch b, head h, query position q, key position k — sum over d." This produces the attention score matrix (batch, heads, query_seq, key_seq). Each entry is 8.0 because d_k=8 and all values are 1. einsum is extremely powerful for Transformer math — it makes the dimension intent explicit.
Softmax
Q 13 Predict the Output
import numpy as np

def softmax(x):
    e_x = np.exp(x)
    return e_x / e_x.sum(axis=-1, keepdims=True)

scores = np.array([[1.0, 2.0, 3.0]])
print(softmax(scores).round(4))
print(softmax(scores).sum())         # should sum to 1
Output
[[0.0900 0.2447 0.6652]] 0.9999999999999998
Why? Softmax converts raw scores into a probability distribution. exp([1,2,3]) = [2.718, 7.389, 20.086], sum = 30.193. Dividing each by the sum gives the probabilities. axis=-1 means we softmax along the last dimension (each row independently). keepdims=True preserves shape for broadcasting. But this version has a bug — it's not numerically stable! See the next question.
Q 14 Multiple Choice

The naive softmax exp(x) / sum(exp(x)) can overflow for large values. What's the standard fix used in every production Transformer?

Why? Subtracting max(x) from each element is the standard "stable softmax" trick. It doesn't change the output (the max cancels in numerator/denominator) but ensures the largest exponent is exp(0)=1, preventing overflow. The stable version: x_shifted = x - max(x); exp(x_shifted) / sum(exp(x_shifted)). Every deep learning framework (PyTorch, JAX) does this internally.
Q 15 Predict the Output
import numpy as np

def stable_softmax(x):
    x_max = x.max(axis=-1, keepdims=True)
    e_x = np.exp(x - x_max)
    return e_x / e_x.sum(axis=-1, keepdims=True)

# Apply softmax row-wise to a (2, 3) attention score matrix
scores = np.array([[1.0, -1e9, -1e9],
                    [1.0,  1.0, -1e9]])
print(stable_softmax(scores).round(4))
Output
[[1. 0. 0. ] [0.5 0.5 0. ]]
Why? This is exactly what happens after masking in causal attention. Row 0: token 0 can only see itself → softmax([1, -1e9, -1e9]) ≈ [1, 0, 0]. Row 1: token 1 can see tokens 0 and 1 → softmax([1, 1, -1e9]) ≈ [0.5, 0.5, 0]. The -1e9 values become effectively 0 after softmax, preventing attention to future tokens.
Fancy Indexing & Gathering
Q 16 Predict the Output
import numpy as np

# Simulating an embedding lookup table
vocab_size = 5
d_model = 3
embedding_table = np.arange(15).reshape(vocab_size, d_model).astype(float)
print("Table:")
print(embedding_table)

token_ids = np.array([2, 0, 4])
embeddings = embedding_table[token_ids]
print("\nLookup result:")
print(embeddings)
Output
Table: [[ 0. 1. 2.] [ 3. 4. 5.] [ 6. 7. 8.] [ 9. 10. 11.] [12. 13. 14.]] Lookup result: [[ 6. 7. 8.] [ 0. 1. 2.] [12. 13. 14.]]
Why? embedding_table[token_ids] is fancy indexing — it grabs rows 2, 0, and 4. This is exactly how nn.Embedding works under the hood: it's just a lookup table indexed by token IDs. No matrix multiply needed for the forward pass — it's a simple row selection. The embedding table shape is always (vocab_size, d_model).
Q 17 Predict the Output
import numpy as np

# Boolean masking — used for padding masks
x = np.array([[1.5, 2.3, 0.0, 0.0],
               [3.1, 0.7, 1.2, 0.0]])
pad_mask = (x != 0)
print(pad_mask.astype(int))
print(pad_mask.sum(axis=1))   # actual lengths per sequence
Output
[[1 1 0 0] [1 1 1 0]] [2 3]
Why? x != 0 creates a boolean mask where True = real token, False = padding. Sequence 0 has 2 real tokens, sequence 1 has 3. In Transformers, you use this padding mask to prevent attention to pad tokens by setting their scores to -inf before softmax (similar to the causal mask, but data-dependent).
Concatenation & Stacking
Q 18 Predict the Output
import numpy as np

# Concatenating multi-head outputs
head_0 = np.ones((2, 3, 4))   # (batch, seq, d_k)
head_1 = np.ones((2, 3, 4)) * 2

# Concat along last dim → (batch, seq, d_model)
merged = np.concatenate([head_0, head_1], axis=-1)
print(merged.shape)
print(merged[0, 0])
Output
(2, 3, 8) [1. 1. 1. 1. 2. 2. 2. 2.]
Why? concatenate(axis=-1) joins along the last dimension: d_k=4 + d_k=4 = d_model=8. This is how multi-head attention merges outputs: each head produces (batch, seq, d_k), then we concat all heads to get back (batch, seq, d_model). After this, a linear projection (W_o) maps it back to d_model.
Q 19 Multiple Choice

What's the difference between np.stack([a, b]) and np.concatenate([a, b]) when a and b are both shape (3, 4)?

Why? np.stack creates a new dimension and stacks arrays along it: two (3,4) arrays → (2,3,4). np.concatenate joins along an existing dimension (default axis=0): two (3,4) arrays → (6,4). When building Transformers: use stack to create a batch dimension from individual sequences, and concatenate to merge head outputs along the feature dimension.
Practical Transformer Patterns
Q 20 Predict the Output
import numpy as np

# Layer normalization from scratch
x = np.array([[1.0, 2.0, 3.0],
               [4.0, 4.0, 4.0]])

mean = x.mean(axis=-1, keepdims=True)
std = x.std(axis=-1, keepdims=True)
normalized = (x - mean) / (std + 1e-5)

print("mean:", mean.flatten())
print("std:", std.flatten().round(4))
print("norm:", normalized.round(4))
Output
mean: [2. 4.] std: [0.8165 0. ] norm: [[-1.2247 0. 1.2247] [ 0. 0. 0. ]]
Why? Layer norm normalizes each sample independently along the feature dimension. Row 0: mean=2, std≈0.8165 → centered and scaled. Row 1: all values are 4, so mean=4, std=0 → all zeros (the epsilon prevents division by zero). In Transformers, LayerNorm is applied after every sub-layer (attention + FFN) to stabilize training. Note: axis=-1 means normalize across the d_model dimension.
Q 21 Predict the Output
import numpy as np

# Residual connection pattern
x = np.array([1.0, 2.0, 3.0])

def sublayer(inp):
    return inp * 0.5   # pretend attention/FFN output

output = x + sublayer(x)     # residual connection
print(output)
Output
[1.5 3. 4.5]
Why? x + sublayer(x) is a residual (skip) connection. The output is [1+0.5, 2+1, 3+1.5] = [1.5, 3, 4.5]. In Transformers, every sub-layer has this pattern: output = LayerNorm(x + SubLayer(x)). Residuals solve the vanishing gradient problem — gradients can flow directly through the addition, even in very deep networks.
Q 22 Predict the Output
import numpy as np

# Sinusoidal positional encoding — the core formula
d_model = 4
pos = np.array([0, 1, 2])[:, None]      # (3, 1)
i = np.arange(0, d_model, 2)[None, :]  # (1, 2)

print("pos shape:", pos.shape)
print("i shape:", i.shape)
print("i values:", i)
Output
pos shape: (3, 1) i shape: (1, 2) i values: [[0 2]]
Why? This sets up broadcasting for the positional encoding formula. pos is (3,1) — positions 0,1,2. i is (1,2) — dimension indices 0,2 (even indices). When you compute pos / 10000^(i/d_model), broadcasting creates a (3,2) matrix — one row per position, one column per frequency. Then you apply sin to even columns and cos to odd columns.
Q 23 Predict the Output
import numpy as np

# Full sinusoidal positional encoding for 3 positions, d_model=4
d_model = 4
max_len = 3

pe = np.zeros((max_len, d_model))
pos = np.arange(max_len)[:, None]
div_term = np.exp(np.arange(0, d_model, 2) * -np.log(10000.0) / d_model)

pe[:, 0::2] = np.sin(pos * div_term)  # even indices
pe[:, 1::2] = np.cos(pos * div_term)  # odd indices

print("PE shape:", pe.shape)
print("Position 0:", pe[0].round(4))
print("Position 1:", pe[1].round(4))
Output
PE shape: (3, 4) Position 0: [0. 1. 0. 1.] Position 1: [ 0.8415 0.5403 0.01 0.9999]
Why? Position 0: sin(0)=0 and cos(0)=1 for all frequencies → [0, 1, 0, 1]. Position 1: sin(1)≈0.8415, cos(1)≈0.5403 for the high-frequency pair; sin(0.01)≈0.01, cos(0.01)≈0.9999 for the low-frequency pair. The key insight: different dimensions oscillate at different frequencies (like a binary clock), giving each position a unique encoding. This is the original "Attention Is All You Need" formula.
Q 24 Multiple Choice

You have a tensor of shape (batch=2, seq=5, d_model=512). You want to split it into 8 heads for multi-head attention. What's the correct sequence of operations?

Why? The correct sequence: (1) reshape (2,5,512) → (2,5,8,64) to split d_model into n_heads * d_k, then (2) transpose to move heads before seq_len: (2,5,8,64) → (2,8,5,64). Now each head has its own (seq_len, d_k) matrix and can compute attention independently. You can't reshape directly to (2,8,5,64) because the data layout would be wrong — reshape doesn't reorder elements.
Q 25 Predict the Output
import numpy as np

# Full single-head attention in 5 lines
np.random.seed(42)
seq_len, d_k = 3, 4
Q = np.random.randn(seq_len, d_k)
K = np.random.randn(seq_len, d_k)
V = np.random.randn(seq_len, d_k)

scores = Q @ K.T / np.sqrt(d_k)                      # (3,3)
mask = np.tril(np.ones((seq_len, seq_len)))
scores = np.where(mask == 0, -1e9, scores)
weights = np.exp(scores - scores.max(axis=-1, keepdims=True))
weights = weights / weights.sum(axis=-1, keepdims=True)
output = weights @ V                                   # (3,4)

print("output shape:", output.shape)
print("weights row 0 sum:", weights[0].sum().round(4))
print("weights row 2 nonzero:", (weights[2] > 0.001).sum())
Output
output shape: (3, 4) weights row 0 sum: 1.0 weights row 2 nonzero: 3
Why? This is complete single-head causal attention in 5 lines of NumPy. Row 0 sums to 1 (softmax property). Row 2 has 3 nonzero weights because with a causal mask, token 2 can attend to tokens 0, 1, and 2 (all positions). The final weights @ V produces a weighted combination of value vectors. If you understand these 5 lines, you understand the core of every Transformer.
← Chapter 13: Python Gotchas Chapter 15: Transformer Blocks →