Chapter 16

PyTorch for Transformers

A dedicated PyTorch chapter — from basic tensor ops to building a full Transformer block with nn.Module. Every operation you'll need when implementing real models.

Questions: 30
Format: Predict Output + Multiple Choice
Difficulty: ★★★★☆
This chapter is pure PyTorch. Every question uses import torch. We start with tensor basics, then cover the exact operations used in Transformer implementations — view, transpose, matmul, masking, nn.Module, and more. By the end, you'll be able to read and write Transformer code in PyTorch fluently.
Tensor Creation & Dtypes
Q 01 Predict the Output
import torch

x = torch.tensor([1, 2, 3])
y = torch.tensor([1.0, 2.0, 3.0])

print(x.dtype)
print(y.dtype)
print((x + y).dtype)
Output
torch.int64 torch.float32 torch.float32
Why? Integers default to torch.int64 (LongTensor). Floats default to torch.float32. When you add int + float, PyTorch promotes to float32. For Transformers: token IDs are always torch.long (for embedding lookup), everything else is torch.float32 or torch.bfloat16 for mixed precision.
Q 02 Predict the Output
import torch

# Common tensor creation patterns for Transformers
a = torch.zeros(2, 3)
b = torch.ones(2, 3)
c = torch.arange(6).reshape(2, 3)
d = torch.randn(2, 3)    # standard normal

print(a.shape, b.shape, c.shape, d.shape)
print(c)
print("randn mean ≈", d.mean().item().__round__(1))
Output
torch.Size([2, 3]) torch.Size([2, 3]) torch.Size([2, 3]) torch.Size([2, 3]) tensor([[0, 1, 2], [3, 4, 5]]) randn mean ≈ varies (close to 0.0)
Why? These are the four tensor creation functions you'll use constantly: zeros (init biases), ones (init gamma in LayerNorm), arange (position indices for PE), randn (random initialization). torch.Size behaves like a tuple. .item() extracts a scalar from a 0-d tensor.
View, Reshape & Transpose
Q 03 Predict the Output
import torch

x = torch.arange(12)
print(x.view(3, 4))
print(x.view(3, 4).shape)
print(x.view(2, -1).shape)   # -1 = infer
Output
tensor([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) torch.Size([3, 4]) torch.Size([2, 6])
Why? .view() reshapes without copying data (requires contiguous memory). -1 means "figure out this dimension": 12/2 = 6. In Transformers, .view(batch, seq, n_heads, d_k) is the standard way to split into heads. .view() is preferred over .reshape() when you know the tensor is contiguous — it's a zero-cost operation (no data movement).
Q 04 Predict the Output
import torch

# The multi-head attention split + transpose pattern
batch, seq, d_model, n_heads = 2, 5, 8, 2
d_k = d_model // n_heads

x = torch.randn(batch, seq, d_model)
print("input:", x.shape)

# Step 1: split d_model into (n_heads, d_k)
x = x.view(batch, seq, n_heads, d_k)
print("after view:", x.shape)

# Step 2: move heads before seq_len
x = x.transpose(1, 2)
print("after transpose:", x.shape)
print("contiguous?", x.is_contiguous())
Output
input: torch.Size([2, 5, 8]) after view: torch.Size([2, 5, 2, 4]) after transpose: torch.Size([2, 2, 5, 4]) contiguous? False
Why? This is the exact code from every multi-head attention implementation: (1) view splits d_model=8 into n_heads=2, d_k=4, (2) transpose(1,2) swaps seq and heads. After transpose, the tensor is not contiguous — transpose just changes the stride metadata, not the actual data layout. This matters if you later call .view() (which requires contiguity) — you'll need .contiguous() first.
Q 05 Multiple Choice

After multi-head attention, you need to merge heads back: (batch, n_heads, seq, d_k) → (batch, seq, d_model). What's the correct code?

Why? (1) transpose(1,2) moves seq back before heads: (B,H,S,dk) → (B,S,H,dk). (2) .contiguous() makes the memory layout match the logical layout (needed after transpose). (3) .view(B,S,-1) merges H*dk back into d_model. Option A fails because the tensor isn't contiguous. Option C might work (reshape can handle non-contiguous) but it copies data silently — transpose + contiguous + view is explicit and standard. Option D uses permute which works identically to transpose for swapping two dims but is more commonly used for >2 dim swaps.
Q 06 Predict the Output
import torch

# permute vs transpose — permute can reorder any number of dims
x = torch.randn(2, 3, 4, 5)

# transpose swaps exactly 2 dimensions
a = x.transpose(1, 2)

# permute can reorder all dimensions at once
b = x.permute(0, 2, 1, 3)

print("transpose(1,2):", a.shape)
print("permute(0,2,1,3):", b.shape)
print("same?", torch.equal(a, b))
Output
transpose(1,2): torch.Size([2, 4, 3, 5]) permute(0,2,1,3): torch.Size([2, 4, 3, 5]) same? True
Why? Both swap dims 1 and 2, producing the same result. transpose takes exactly 2 dim indices to swap. permute takes a complete dimension ordering. For multi-head attention (swapping dims 1↔2), either works. For more complex reorderings (e.g., 3+ dims shuffled), use permute. Both return views (no data copy), both make the result non-contiguous.
Matrix Multiplication in PyTorch
Q 07 Predict the Output
import torch

# Four ways to do matrix multiply in PyTorch
a = torch.ones(3, 4)
b = torch.ones(4, 2)

c1 = a @ b
c2 = torch.matmul(a, b)
c3 = torch.mm(a, b)        # 2D only
c4 = a.matmul(b)

print(c1.shape, c2.shape, c3.shape, c4.shape)
print("all same?", torch.equal(c1, c2) and torch.equal(c2, c3) and torch.equal(c3, c4))
Output
torch.Size([3, 2]) torch.Size([3, 2]) torch.Size([3, 2]) torch.Size([3, 2]) all same? True
Why? All four are equivalent for 2D matrices. @ and torch.matmul support batched (3D+) matmul. torch.mm is strictly 2D. For Transformers, use @ or torch.matmul — they handle batched Q@K^T naturally. torch.bmm exists for explicit 3D batched matmul (batch, m, k) @ (batch, k, n).
Q 08 Predict the Output
import torch

# Batched matmul for multi-head attention: Q @ K^T
Q = torch.ones(2, 4, 3, 8)   # (batch, heads, seq, d_k)
K = torch.ones(2, 4, 3, 8)

# K.transpose(-2, -1) swaps last two dims: (b,h,seq,dk) → (b,h,dk,seq)
scores = Q @ K.transpose(-2, -1)
print("scores shape:", scores.shape)
print("scores[0,0,0,0]:", scores[0,0,0,0].item())
Output
scores shape: torch.Size([2, 4, 3, 3]) scores[0,0,0,0]: 8.0
Why? K.transpose(-2, -1) swaps the last two dims (seq, d_k) → (d_k, seq). Then @ does batched matmul over batch and heads: (B,H,3,8) @ (B,H,8,3) = (B,H,3,3). Each 3x3 slice is the attention score matrix for one head. The value 8.0 = dot product of two all-ones vectors of length d_k=8. Using negative indices (-2, -1) is idiomatic — it works regardless of how many batch dimensions precede.
Q 09 Predict the Output
import torch

# Einstein summation — explicit dimension control
Q = torch.ones(2, 4, 3, 8)
K = torch.ones(2, 4, 3, 8)

# "bhqd,bhkd->bhqk" = for each (batch, head), compute Q[q,:] · K[k,:]
scores = torch.einsum('bhqd,bhkd->bhqk', Q, K)
print("einsum shape:", scores.shape)

# Same result as Q @ K.transpose(-2, -1)
scores2 = Q @ K.transpose(-2, -1)
print("equal?", torch.allclose(scores, scores2))
Output
einsum shape: torch.Size([2, 4, 3, 3]) equal? True
Why? einsum lets you spell out exactly which dimensions contract (shared subscripts that don't appear in output). Here, d appears in both inputs but not in the output → it's summed over (dot product). b,h are batch dims. q,k are the free dims that form the output matrix. einsum is not faster than @ but it makes complex tensor ops readable — very useful when the dimension layout isn't obvious.
Masking Operations
Q 10 Predict the Output
import torch

# Creating a causal mask with torch.tril
mask = torch.tril(torch.ones(4, 4))
print("tril mask:")
print(mask)

# Boolean version (more common in modern code)
bool_mask = torch.tril(torch.ones(4, 4)).bool()
print("\nbool mask:")
print(bool_mask)
Output
tril mask: tensor([[1., 0., 0., 0.], [1., 1., 0., 0.], [1., 1., 1., 0.], [1., 1., 1., 1.]]) bool mask: tensor([[ True, False, False, False], [ True, True, False, False], [ True, True, True, False], [ True, True, True, True]])
Why? torch.tril (lower triangular) creates the causal attention mask. Entry (i,j) is 1/True if j ≤ i — position i can attend to position j. The boolean version is used with masked_fill. Many implementations also use torch.triu(ones, diagonal=1).bool() as the "where to put -inf" mask (upper triangle with no diagonal). This mask is created once and reused for all layers.
Q 11 Predict the Output
import torch

# masked_fill — the PyTorch way to apply attention masks
scores = torch.tensor([[1.0, 2.0, 3.0],
                        [4.0, 5.0, 6.0],
                        [7.0, 8.0, 9.0]])

# Mask: True = positions to fill with -inf
mask = torch.triu(torch.ones(3, 3), diagonal=1).bool()
print("mask (True = fill):")
print(mask)

masked = scores.masked_fill(mask, float('-inf'))
print("\nmasked scores:")
print(masked)
Output
mask (True = fill): tensor([[False, True, True], [False, False, True], [False, False, False]]) masked scores: tensor([[1., -inf, -inf], [4., 5., -inf], [7., 8., 9.]])
Why? masked_fill(mask, -inf) puts -inf wherever mask is True. torch.triu(..., diagonal=1) gives the strict upper triangle (above the diagonal). So future positions become -inf. After softmax, these become 0. This is the standard PyTorch pattern for causal masking: scores.masked_fill(mask == 0, float('-inf')) (if mask is "where to keep") or scores.masked_fill(mask, float('-inf')) (if mask is "where to block").
Q 12 Multiple Choice

What does register_buffer('mask', mask) do in an nn.Module?

Why? register_buffer is for tensors that are part of the model state but should NOT be updated by gradient descent. The causal mask and sinusoidal positional encoding are perfect examples — they're fixed constants that need to: (1) move to GPU when you call model.to('cuda'), (2) be saved/loaded with state_dict, (3) NOT appear in model.parameters(). Without register_buffer, you'd have to manually move these tensors every time.
nn.Module Building Blocks
Q 13 Predict the Output
import torch
import torch.nn as nn

# nn.Embedding — the token embedding layer
emb = nn.Embedding(num_embeddings=10, embedding_dim=4)

ids = torch.tensor([2, 5, 7])
out = emb(ids)

print("weight shape:", emb.weight.shape)
print("output shape:", out.shape)
print("same as indexing?", torch.equal(out, emb.weight[ids]))
Output
weight shape: torch.Size([10, 4]) output shape: torch.Size([3, 4]) same as indexing? True
Why? nn.Embedding is literally a lookup table: emb(ids) = emb.weight[ids]. Weight shape is (vocab_size, d_model). Input must be LongTensor. The embedding weights are learned during training. For GPT-style models, the same weight matrix is often "tied" to the output projection: lm_head.weight = embedding.weight.
Q 14 Predict the Output
import torch
import torch.nn as nn

# nn.Linear — the workhorse of Transformers
linear = nn.Linear(in_features=4, out_features=3, bias=False)

print("weight shape:", linear.weight.shape)

x = torch.randn(2, 4)  # batch of 2, features of 4
out = linear(x)
print("output shape:", out.shape)

# What linear(x) actually computes:
manual = x @ linear.weight.T
print("manual equals linear?", torch.allclose(out, manual))
Output
weight shape: torch.Size([3, 4]) output shape: torch.Size([2, 3]) manual equals linear? True
Why? nn.Linear(4, 3) stores weight as (out, in) = (3, 4) — transposed from what you might expect. linear(x) computes x @ W.T + bias. With bias=False, it's just x @ W.T. In Transformers, W_q, W_k, W_v, W_o, and the FFN layers are all nn.Linear. Many Transformer implementations use bias=False.
Q 15 Predict the Output
import torch
import torch.nn as nn

# nn.LayerNorm — PyTorch's built-in
ln = nn.LayerNorm(4)     # normalize over last dim of size 4
print("weight (gamma):", ln.weight.data)
print("bias (beta):", ln.bias.data)

x = torch.tensor([[1.0, 2.0, 3.0, 4.0]])
out = ln(x)
print("mean:", out.mean().item().__round__(4))
print("std:", out.std(unbiased=False).item().__round__(4))
Output
weight (gamma): tensor([1., 1., 1., 1.]) bias (beta): tensor([0., 0., 0., 0.]) mean: 0.0 std: 1.0
Why? nn.LayerNorm(4) normalizes the last dimension to mean=0, std=1, then applies learnable gamma (init 1) and beta (init 0). With default init, it's a pure normalization. During training, gamma and beta learn per-feature scaling and shifting. Note: PyTorch uses the biased variance (dividing by N, not N-1) in LayerNorm, matching the paper formula.
Q 16 Predict the Output
import torch
import torch.nn as nn

# nn.MultiheadAttention — PyTorch's built-in MHA
d_model, n_heads = 8, 2
mha = nn.MultiheadAttention(d_model, n_heads, batch_first=True)

# List all parameter shapes
for name, p in mha.named_parameters():
    print(f"{name}: {p.shape}")
Output
in_proj_weight: torch.Size([24, 8]) in_proj_bias: torch.Size([24]) out_proj.weight: torch.Size([8, 8]) out_proj.bias: torch.Size([8])
Why? in_proj_weight is (3*d_model, d_model) = (24, 8) — it packs W_q, W_k, W_v into a single matrix for efficiency (one matmul instead of three). The first 8 rows are W_q, next 8 are W_k, last 8 are W_v. out_proj is (d_model, d_model) = (8, 8) — the W_o projection after concatenating heads. batch_first=True means input shape is (batch, seq, d_model) instead of (seq, batch, d_model).
torch.nn.functional — Key Functions
Q 17 Predict the Output
import torch
import torch.nn.functional as F

# F.softmax — always specify dim!
x = torch.tensor([[1.0, 2.0, 3.0],
                    [1.0, 1.0, 1.0]])

row_softmax = F.softmax(x, dim=-1)   # softmax over last dim (rows)
col_softmax = F.softmax(x, dim=0)    # softmax over first dim (columns)

print("row softmax:")
print(row_softmax.round(decimals=4))
print("row sums:", row_softmax.sum(dim=-1))
Output
row softmax: tensor([[0.0900, 0.2447, 0.6652], [0.3333, 0.3333, 0.3333]]) row sums: tensor([1., 1.])
Why? dim=-1 applies softmax across the last dimension (each row independently). Row 0: unequal logits → peaked distribution. Row 1: equal logits → uniform 1/3 each. Always specify dim — omitting it will give a deprecation warning and may default to dim=0 (wrong for attention!). In attention: F.softmax(scores, dim=-1) — softmax over the key dimension.
Q 18 Predict the Output
import torch
import torch.nn.functional as F

# F.scaled_dot_product_attention — the modern API (PyTorch 2.0+)
# This uses Flash Attention when possible!
torch.manual_seed(42)
Q = torch.randn(1, 2, 4, 8)   # (batch, heads, seq, d_k)
K = torch.randn(1, 2, 4, 8)
V = torch.randn(1, 2, 4, 8)

# is_causal=True automatically creates and applies the causal mask
out = F.scaled_dot_product_attention(Q, K, V, is_causal=True)
print("output shape:", out.shape)
print("This one call replaces ~10 lines of manual attention code!")
Output
output shape: torch.Size([1, 2, 4, 8]) This one call replaces ~10 lines of manual attention code!
Why? F.scaled_dot_product_attention (SDPA) was added in PyTorch 2.0. It fuses Q@K^T, scaling, masking, softmax, and @V into one optimized kernel. With is_causal=True, it automatically applies the causal mask. On GPU, it uses Flash Attention (O(N) memory) or Memory-Efficient Attention automatically. This is what you should use in production — it's faster than manual attention and handles edge cases.
Q 19 Predict the Output
import torch
import torch.nn.functional as F

# Cross-entropy loss — the Transformer training loss
logits = torch.tensor([[2.0, 0.5, 0.1],
                        [0.1, 0.3, 3.0]])
targets = torch.tensor([0, 2])   # correct classes

loss = F.cross_entropy(logits, targets)
print("loss:", loss.item().__round__(4))

# Manual calculation for verification
manual = -F.log_softmax(logits, dim=-1)[range(2), targets].mean()
print("manual:", manual.item().__round__(4))
print("same?", abs(loss.item() - manual.item()) < 1e-5)
Output
loss: 0.2573 manual: 0.2573 same? True
Why? F.cross_entropy = softmax + log + NLL in one fused, numerically stable call. It takes raw logits (not probabilities) and target class indices. For Transformers: reshape logits to (batch*seq_len, vocab_size), targets to (batch*seq_len), and compute F.cross_entropy(logits.view(-1, vocab_size), targets.view(-1)). This is the standard LM training loss.
Parameter Initialization
Q 20 Predict the Output
import torch
import torch.nn as nn

# Xavier (Glorot) initialization — used in original Transformer
linear = nn.Linear(512, 512)
nn.init.xavier_uniform_(linear.weight)
print("xavier std:", linear.weight.std().item().__round__(4))

# Kaiming (He) initialization — used for ReLU networks
nn.init.kaiming_uniform_(linear.weight)
print("kaiming std:", linear.weight.std().item().__round__(4))

# Normal initialization — common in GPT/LLaMA
nn.init.normal_(linear.weight, mean=0, std=0.02)
print("normal(0.02) std:", linear.weight.std().item().__round__(4))
Output
xavier std: ~0.0557 kaiming std: ~0.0559 normal(0.02) std: ~0.0200
Why? Xavier: std ≈ √(2/(fan_in+fan_out)) = √(2/1024) ≈ 0.044 (uniform has slightly different effective std). Kaiming: std ≈ √(2/fan_in) — designed for ReLU (accounts for ReLU zeroing half the activations). Normal(0.02): GPT-2 and many LLMs use this simple init. The _ suffix means in-place modification. Good initialization is critical — too large → exploding gradients, too small → vanishing gradients.
Q 21 Multiple Choice

In GPT-2, the residual projection layers (W_o in attention and the second linear in FFN) are initialized with a special scaling. Why?

Why? Each Transformer layer adds to the residual stream: x = x + sublayer(x). With N layers and 2 residual additions per layer (attention + FFN), the variance of x grows. GPT-2 initializes the output projections with std = 0.02 / sqrt(2 * n_layers). For a 12-layer model: 0.02/√24 ≈ 0.004. This ensures the residual stream doesn't explode as the model gets deeper. It's a simple but crucial trick.
Device Management & Mixed Precision
Q 22 Predict the Output
import torch

# Device management patterns
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("device:", device)

# Create tensor on device
x = torch.randn(3, 4, device=device)
print("x device:", x.device)

# Move model to device
# model = model.to(device)  — moves all parameters + buffers

# The WRONG way (common bug):
# x_cpu = torch.randn(3, 4)              # on CPU
# x_gpu = torch.randn(3, 4, device='cuda') # on GPU
# result = x_cpu + x_gpu   # RuntimeError: different devices!
print("All tensors in an op must be on the same device!")
Output
device: cpu (or cuda:0 if GPU available) x device: cpu (or cuda:0) All tensors in an op must be on the same device!
Why? PyTorch requires all tensors in an operation to be on the same device. Common pattern: (1) set device once, (2) create all tensors with device=device, (3) move model with .to(device). The most common training bug is forgetting to move the input data to GPU. Fix: inputs = inputs.to(device) in your training loop.
Q 23 Multiple Choice

What does torch.autocast('cuda', dtype=torch.bfloat16) do?

Why? torch.autocast enables Automatic Mixed Precision (AMP). Inside the context manager, matmuls and convolutions run in bfloat16 (2x faster, half memory), while operations requiring precision (loss, normalization, softmax) stay in float32. Usage: with torch.autocast('cuda', dtype=torch.bfloat16): loss = model(x). Combined with GradScaler for float16 (not needed for bfloat16), this is the standard way to train Transformers efficiently.
Q 24 Predict the Output
import torch

# float32 vs bfloat16 — precision and range
x_f32 = torch.tensor(3.14159265, dtype=torch.float32)
x_bf16 = x_f32.to(torch.bfloat16)
x_f16 = x_f32.to(torch.float16)

print(f"float32:  {x_f32.item():.8f}")
print(f"bfloat16: {x_bf16.item():.8f}")
print(f"float16:  {x_f16.item():.8f}")

# Range test: large values
big = torch.tensor(65536.0)
print(f"\nbig in f16: {big.to(torch.float16).item()}")    # overflow!
print(f"big in bf16: {big.to(torch.bfloat16).item()}")   # fine!
Output
float32: 3.14159274 bfloat16: 3.14062500 float16: 3.14062500 big in f16: inf big in bf16: 65536.0
Why? bfloat16 has the same exponent range as float32 (8 bits) but less precision (7 bits mantissa vs 23). float16 has more precision (10 bits) but smaller range (5 bit exponent) — 65536 overflows to inf! This is why bfloat16 is preferred for LLM training: you rarely need >3 decimal digits of precision, but you DO need the full dynamic range to handle large gradients and activations.
Building a Complete Transformer Block
Q 25 Predict the Output
import torch
import torch.nn as nn
import torch.nn.functional as F

class CausalSelfAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.proj = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x):
        B, T, C = x.shape
        qkv = self.qkv(x).reshape(B, T, 3, self.n_heads, self.d_k)
        qkv = qkv.permute(2, 0, 3, 1, 4)
        Q, K, V = qkv[0], qkv[1], qkv[2]
        out = F.scaled_dot_product_attention(Q, K, V, is_causal=True)
        out = out.transpose(1, 2).contiguous().view(B, T, C)
        return self.proj(out)

attn = CausalSelfAttention(d_model=16, n_heads=4)
x = torch.randn(2, 5, 16)
print("in:", x.shape, "→ out:", attn(x).shape)
print("params:", sum(p.numel() for p in attn.parameters()))
Output
in: torch.Size([2, 5, 16]) → out: torch.Size([2, 5, 16]) params: 1024
Why? This is a production-quality causal self-attention module. Params: qkv = 16×48 = 768, proj = 16×16 = 256, total = 1024. The key patterns: (1) fused QKV projection (one matmul), (2) reshape to separate Q,K,V and split heads, (3) F.scaled_dot_product_attention with is_causal=True, (4) merge heads and project. Input shape = output shape = (batch, seq, d_model). This is essentially what you'll find in nanoGPT.
Q 26 Predict the Output
import torch
import torch.nn as nn

class FeedForward(nn.Module):
    def __init__(self, d_model, d_ff=None):
        super().__init__()
        d_ff = d_ff or 4 * d_model
        self.net = nn.Sequential(
            nn.Linear(d_model, d_ff, bias=False),
            nn.GELU(),
            nn.Linear(d_ff, d_model, bias=False),
        )

    def forward(self, x):
        return self.net(x)

ffn = FeedForward(d_model=16)
x = torch.randn(2, 5, 16)
print("in:", x.shape, "→ out:", ffn(x).shape)
print("params:", sum(p.numel() for p in ffn.parameters()))
Output
in: torch.Size([2, 5, 16]) → out: torch.Size([2, 5, 16]) params: 2048
Why? FFN: Linear(16→64) + GELU + Linear(64→16). Params: 16×64 + 64×16 = 1024 + 1024 = 2048. The FFN has 2x the parameters of the attention module! In real Transformers, the FFN is often the biggest parameter consumer per layer. d_ff = 4 * d_model is the standard ratio. nn.Sequential chains modules — it calls each in order during forward.
Q 27 Predict the Output
import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerBlock(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.ln1 = nn.LayerNorm(d_model)
        self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
        self.ln2 = nn.LayerNorm(d_model)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, 4 * d_model),
            nn.GELU(),
            nn.Linear(4 * d_model, d_model),
        )

    def forward(self, x):
        # Pre-norm: LN → sublayer → residual add
        x_norm = self.ln1(x)
        mask = nn.Transformer.generate_square_subsequent_mask(x.size(1))
        attn_out, _ = self.attn(x_norm, x_norm, x_norm, attn_mask=mask, is_causal=True)
        x = x + attn_out
        x = x + self.ffn(self.ln2(x))
        return x

block = TransformerBlock(d_model=32, n_heads=4)
x = torch.randn(2, 10, 32)
out = block(x)
print("in:", x.shape, "→ out:", out.shape)
n_params = sum(p.numel() for p in block.parameters())
print(f"parameters: {n_params:,}")
Output
in: torch.Size([2, 10, 32]) → out: torch.Size([2, 10, 32]) parameters: 13,536
Why? This is a complete pre-norm Transformer decoder block using PyTorch built-ins. Components: LN1 (64 params), MHA (3*32*32 + 32 + 32*32 + 32 = 4,192), LN2 (64 params), FFN (32*128 + 128 + 128*32 + 32 = 8,352). Total ≈ 13,536. The key pattern: x = x + sublayer(LN(x)) for each sub-layer. Stack N of these and add an embedding + LM head → that's GPT.
Training Utilities
Q 28 Multiple Choice

What does torch.no_grad() do, and when should you use it?

Why? torch.no_grad() disables autograd, so PyTorch doesn't store intermediate activations for backward. This saves significant memory (Transformers store huge activation maps). Use it for: (1) @torch.no_grad() on your eval function, (2) inference/generation loops, (3) computing metrics. Never use it during training forward/backward. The equivalent decorator form: @torch.inference_mode() (slightly faster, introduced in PyTorch 1.9).
Q 29 Predict the Output
import torch
import torch.nn as nn

# Gradient clipping — essential for Transformer training
model = nn.Linear(4, 4)
x = torch.randn(1, 4)
loss = model(x).sum()
loss.backward()

# Check gradient norm before clipping
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
print(f"gradient norm (before clip): {total_norm.item():.4f}")

# After clipping, all gradients are scaled so total norm ≤ 1.0
clipped_norm = sum(p.grad.norm()**2 for p in model.parameters())**0.5
print(f"gradient norm (after clip): {clipped_norm.item():.4f}")
Output
gradient norm (before clip): varies (e.g. 2.1234) gradient norm (after clip): 1.0000 (if original > 1.0, else unchanged)
Why? clip_grad_norm_ computes the total L2 norm of all gradients. If it exceeds max_norm, it scales all gradients proportionally so the total norm = max_norm. This prevents gradient explosions that can crash training. Standard value: max_norm=1.0. It returns the original (pre-clip) norm, which is useful for logging. Every serious Transformer training loop includes this call right before optimizer.step().
Q 30 Multiple Choice

What does torch.compile(model) do in PyTorch 2.0+?

Why? torch.compile captures the computation graph (via TorchDynamo), optimizes it (via TorchInductor), and generates fused GPU kernels. Benefits: (1) fuses multiple small ops into one kernel launch, (2) reduces Python overhead, (3) applies graph-level optimizations. Typical speedup: 20-40% for Transformer training. Usage: model = torch.compile(model) — one line, and the rest of your code stays the same. It handles most dynamic control flow via graph breaks.
← Chapter 15: Transformer Blocks Table of Contents →