Mistral 7B — The Evolving Transformer

Mistral 7B · decoder stack (32× blocks, SWA W=4096, GQA 32/8)

This chapter evolves chapter 03 LLaMA into Mistral 7B — same decoder-only stack (wte-only embeddings, RoPE, RMSNorm, SwiGLU, GQA, no biases, weight tying), with one key innovation: sliding window attention. Each token attends only to the last W=4096 positions instead of the full causal history. Hyperparameters follow Mistral 7B from Mistral AI. Same teaching pattern: one config object up front, then each layer in order until the full model runs.

LLaMA 3 vs Mistral 7B at a glance: the block diagrams look almost the same — both are dense decoder-only Transformers, not MoE. LLaMA 3 optimizes for scale (128K vocab, full global attention, larger model family). Mistral 7B optimizes for efficiency: same GQA idea, but adds sliding-window attention so cost scales like O(N×W) instead of O(N²) over the local window. This chapter is the code path for that efficiency trade-off.

Each section below is one building block. You get a short explanation, the code for that module, a delta from chapter 03 where it helps, and how it connects to what came before. By the end you have a runnable Mistral class, not just a diagram.

Delta from LLaMA 3 (03)

Full causal attention → sliding window attention (W=4096)
GQABlock → SlidingWindowGQA (same projections, different mask)
At inference: rolling buffer KV cache bounds memory to W (not shown in this training-style code)
Everything else unchanged: wte-only embeddings, RoPE, RMSNorm, SwiGLU, GQA ratio, no bias, weight tying

Full side-by-side with chapter 03: §08 Comparison.

00 · CONFIG

One config object for the whole model

Before any module, we define a single MistralConfig dataclass for Mistral 7B. Every class below takes config instead of a long list of constructor arguments — same pattern as chapter 03, plus one new field: sliding_window.

Mistral 7B uses d_model=4096, N=32 blocks, H=32 query heads, n_kv_heads=8 (4:1 GQA ratio), vocab=32,000, context length 8,192, d_ff=14,336, and sliding window W=4096 — verified against Mistral’s Mistral-7B-v0.1 config.json (~7.3B parameters). The window is the main architectural delta from LLaMA 3; everything else in the stack matches chapter 03.

dk property: dmodel // H → 4096 // 32 = 128. Multi-head attention splits d_model across H query heads; d_model must be divisible by H.

sliding_window: W=4096 — each token attends to at most W past tokens (causal ∩ band). Built by make_sliding_window_mask (section 05):

 34def make_sliding_window_mask(max_seq_len, sliding_window):
 35    """Causal mask ∩ sliding window band: token i attends to [max(0, i-W+1) .. i]."""
 36    causal = torch.tril(torch.ones(max_seq_len, max_seq_len))
 37    window = torch.triu(torch.ones(max_seq_len, max_seq_len), diagonal=-(sliding_window - 1))
 38    return causal * window  # (max_seq_len, max_seq_len)

vocabsize: 32,000 for Mistral 7B (vs 128,256 in LLaMA 3 8B) — the only other config default that differs from chapter 03 besides the window.

All values below are Mistral 7B — verified against official config.json. Change MistralConfig fields to explore other Mistral sizes.

dmodel4096 — hidden_size

N32 — num_hidden_layers

H32 — num_attention_heads

n_kv_heads8 — num_key_value_heads (4:1 GQA)

vocabsize32,000 — vocab_size

max_seq_len8,192 in mistral.py (8K paper context; RoPE + mask buffer). HF max_position_embeddings: 32,768

dff14,336 — intermediate_size

sliding_window4,096 — sliding_window (local attention band)

norm_eps1e-5 — rms_norm_eps

B2 — batch size for demos only

dk128 — dmodel // H (head dim)

mistral.pyimports + MistralConfig

  1from dataclasses import dataclass
  2import torch
  3import torch.nn as nn
  4import torch.nn.functional as F
  5import math
  6
  7
  8# ============================================================
  9# Config — Mistral 7B from Jiang et al., 2023 (Mistral AI)
 10# d_model=4096, N=32, H=32, n_kv_heads=8, vocab=32000, W=4096 sliding window
 11# ============================================================
 12@dataclass
 13class MistralConfig:
 14    dmodel: int = 4096           # Mistral 7B
 15    N: int = 32                  # transformer blocks
 16    H: int = 32                  # query heads
 17    n_kv_heads: int = 8          # GQA — 4:1 ratio (32 Q / 8 KV)
 18    vocabsize: int = 32000       # BPE vocabulary size (Mistral 7B)
 19    max_seq_len: int = 8192      # max context length
 20    dff: int = 14336             # SwiGLU inner dim
 21    norm_eps: float = 1e-5       # RMSNorm epsilon
 22    sliding_window: int = 4096   # attention window W — each token sees W past tokens
 23    B: int = 2                   # batch size (for demo runs)
 24
 25    @property
 26    def dk(self):
 27        # per-head dimension — dmodel must be divisible by H
 28        return self.dmodel // self.H
 29
 30
 31config = MistralConfig()  # Mistral 7B — shared by every module below

01 · EMBEDDINGS

Token embedding only — unchanged from LLaMA

Mistral keeps the same wte-only input as chapter 03: a token embedding table mapping IDs to d_model vectors, no learned position table, no dropout. Position is handled by RoPE inside attention (section 02).

The forward pass is simply wte(x) — one lookup, straight into the block stack. No delta from LLaMA here.

mistral.pywte only

  1# inside Mistral.__init__ — token embedding only (no wpe):
  2self.wte = nn.Embedding(config.vocabsize, config.dmodel)  # wte(x): (B, S, dmodel)
  3
  4# inside Mistral.forward — no positional sum, no dropout:
  5_, S = x.shape  # x: (B, S)
  6x = self.wte(x)  # (B, S, dmodel) — no wpe; RoPE handles position in attention

▼ Show original version

LLaMA in chapter 03 — identical wte pattern (excerpt from llama.py).

llama.pywte only

  1# inside LLAMA.__init__ — token embedding only (no wpe):
  2self.wte = nn.Embedding(config.vocabsize, config.dmodel)  # wte(x): (B, S, dmodel)
  3
  4# inside LLAMA.forward — no positional sum, no dropout:
  5_, S = x.shape  # x: (B, S)
  6x = self.wte(x)  # (B, S, dmodel) — no wpe; RoPE handles position in attention

02 · RoPE

Rotary positional embeddings — unchanged from LLaMA

RoPE encodes position by rotating Q and K vectors in each head's 2D subspaces. Cos/sin tables are precomputed for all positions up to max_seq_len and registered as buffers — same implementation as chapter 03.

No delta from LLaMA. Sliding window attention (section 05) limits which keys each query sees; RoPE still rotates Q and K the same way.

mistral.pyRotaryPositionalEmbedding

 65class RotaryPositionalEmbedding(nn.Module):
 66    def __init__(self, config, base: float = 10000.0) -> None:
 67        super().__init__()
 68        self.dk = config.dk
 69        freqs = 1.0 / (
 70            base ** (torch.arange(0, self.dk, 2).float() / self.dk)
 71        )  # (dk//2,)
 72        pos = torch.arange(0, config.max_seq_len).float()  # (max_seq_len,)
 73        angles = torch.outer(pos, freqs)  # (max_seq_len, dk//2)
 74        self.register_buffer("cos_cached", torch.cos(angles))  # (max_seq_len, dk//2)
 75        self.register_buffer("sin_cached", torch.sin(angles))  # (max_seq_len, dk//2)
 76
 77    def forward(self, x):
 78        B, H, S, dk = x.shape  # x: (B, H, S, dk)
 79        x_even = x[..., 0::2]  # (B, H, S, dk//2)
 80        x_odd = x[..., 1::2]  # (B, H, S, dk//2)
 81        cos = self.cos_cached[:S].unsqueeze(0).unsqueeze(0)  # (1, 1, S, dk//2)
 82        sin = self.sin_cached[:S].unsqueeze(0).unsqueeze(0)  # (1, 1, S, dk//2)
 83        rotated_even = x_even * cos - x_odd * sin  # (B, H, S, dk//2)
 84        rotated_odd = x_even * sin + x_odd * cos  # (B, H, S, dk//2)
 85        rotated = torch.stack([rotated_even, rotated_odd], dim=-1)  # (B, H, S, dk//2, 2)
 86        return rotated.view(B, H, S, dk)  # (B, H, S, dk)

▼ Show original version

LLaMA in chapter 03 — same RotaryPositionalEmbedding (excerpt from llama.py).

llama.pyRotaryPositionalEmbedding

 57class RotaryPositionalEmbedding(nn.Module):
 58    def __init__(self, config, base: float = 10000.0) -> None:
 59        super().__init__()
 60        self.dk = config.dk
 61        freqs = 1.0 / (
 62            base ** (torch.arange(0, self.dk, 2).float() / self.dk)
 63        )  # (dk//2,)
 64        pos = torch.arange(0, config.max_seq_len).float()  # (max_seq_len,)
 65        angles = torch.outer(pos, freqs)  # (max_seq_len, dk//2)
 66        self.register_buffer("cos_cached", torch.cos(angles))  # (max_seq_len, dk//2)
 67        self.register_buffer("sin_cached", torch.sin(angles))  # (max_seq_len, dk//2)
 68
 69    def forward(self, x):
 70        B, H, S, dk = x.shape  # x: (B, H, S, dk)
 71        x_even = x[..., 0::2]  # (B, H, S, dk//2)
 72        x_odd = x[..., 1::2]  # (B, H, S, dk//2)
 73        cos = self.cos_cached[:S].unsqueeze(0).unsqueeze(0)  # (1, 1, S, dk//2)
 74        sin = self.sin_cached[:S].unsqueeze(0).unsqueeze(0)  # (1, 1, S, dk//2)
 75        rotated_even = x_even * cos - x_odd * sin  # (B, H, S, dk//2)
 76        rotated_odd = x_even * sin + x_odd * cos  # (B, H, S, dk//2)
 77        rotated = torch.stack([rotated_even, rotated_odd], dim=-1)  # (B, H, S, dk//2, 2)
 78        return rotated.view(B, H, S, dk)  # (B, H, S, dk)

03 · RMSNORM

Root-mean-square normalization — unchanged from LLaMA

RMSNorm divides by the root-mean-square of the features plus norm_eps, then multiplies by a learnable gamma vector — no mean centering. Pre-norm placement in each block is identical to chapter 03.

No delta from LLaMA.

mistral.pyRMSNorm

 41class RMSNorm(nn.Module):
 42    def __init__(self, config) -> None:
 43        super().__init__()
 44        self.norm_eps = config.norm_eps
 45        self.gamma = nn.Parameter(torch.ones(config.dmodel))  # (dmodel,)
 46
 47    def forward(self, x):
 48        # x: (B, S, dmodel)
 49        rms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.norm_eps)  # (B, S, 1)
 50        return x * rms * self.gamma  # (B, S, dmodel)

▼ Show original version

LLaMA in chapter 03 — same RMSNorm (excerpt from llama.py).

llama.pyRMSNorm

 33class RMSNorm(nn.Module):
 34    def __init__(self, config) -> None:
 35        super().__init__()
 36        self.norm_eps = config.norm_eps
 37        self.gamma = nn.Parameter(torch.ones(config.dmodel))  # (dmodel,)
 38
 39    def forward(self, x):
 40        # x: (B, S, dmodel)
 41        rms = torch.rsqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.norm_eps)  # (B, S, 1)
 42        return x * rms * self.gamma  # (B, S, dmodel)

04 · SWIGLU

SwiGLU feed-forward — unchanged from LLaMA

Mistral uses the same SwiGLU FFN as LLaMA: three bias-free matrices (w1 gate, w2 up, w3 down) with activation silu(w1(x)) * w2(x). Inner dimension dff=14336 comes from config.

No delta from LLaMA.

mistral.pySwiGLUFeedForward

 53class SwiGLUFeedForward(nn.Module):
 54    def __init__(self, config) -> None:
 55        super().__init__()
 56        self.w1 = nn.Linear(config.dmodel, config.dff, bias=False)  # gate
 57        self.w2 = nn.Linear(config.dmodel, config.dff, bias=False)  # up
 58        self.w3 = nn.Linear(config.dff, config.dmodel, bias=False)  # down
 59
 60    def forward(self, x):
 61        # x: (B, S, dmodel) → silu(w1) * w2 → w3 → (B, S, dmodel)
 62        return self.w3(F.silu(self.w1(x)) * self.w2(x))

▼ Show original version

LLaMA in chapter 03 — same SwiGLUFeedForward (excerpt from llama.py).

llama.pySwiGLUFeedForward

 45class SwiGLUFeedForward(nn.Module):
 46    def __init__(self, config) -> None:
 47        super().__init__()
 48        self.w1 = nn.Linear(config.dmodel, config.dff, bias=False)  # (B,S,dmodel) → (B,S,dff) gate
 49        self.w2 = nn.Linear(config.dmodel, config.dff, bias=False)  # (B,S,dmodel) → (B,S,dff) up
 50        self.w3 = nn.Linear(config.dff, config.dmodel, bias=False)  # (B,S,dff) → (B,S,dmodel) down
 51
 52    def forward(self, x):
 53        # x: (B, S, dmodel) → silu(w1) * w2 → w3 → (B, S, dmodel)
 54        return self.w3(F.silu(self.w1(x)) * self.w2(x))

05 · SLIDING WINDOW GQA

Grouped-query attention with a sliding window mask

This is the main delta from chapter 03. SlidingWindowGQA keeps the same GQA projections and RoPE application as GQABlock, but the attention mask is no longer full causal — it is causal ∩ sliding window band.

vs LLaMA 3 8B: LLaMA uses GQA with full causal attention — every token sees all past tokens (O(N²) attention cost). Mistral adds a local window of W=4096, so each token only attends to the last W positions (O(N×W) over the band). GQA is the same in both; the mask is what changes.

make_sliding_window_mask builds the mask: lower-triangular (causal) multiplied by a band matrix where token i can attend to positions in [max(0, i−W+1) .. i]. At inference, production systems use a rolling KV cache so memory stays bounded by W rather than growing with sequence length — that optimization is not in this training-style forward pass, but it is why sliding window matters in practice.

mask = tril(1) ⊙ triu(1, diag=−(W−1)) · Attention(Q, K, V) = softmax(QK^T / √d_k + mask) V

mistral.pymake_sliding_window_mask + SlidingWindowGQA

 34def make_sliding_window_mask(max_seq_len, sliding_window):
 35    """Causal mask ∩ sliding window band: token i attends to [max(0, i-W+1) .. i]."""
 36    causal = torch.tril(torch.ones(max_seq_len, max_seq_len))
 37    window = torch.triu(torch.ones(max_seq_len, max_seq_len), diagonal=-(sliding_window - 1))
 38    return causal * window  # (max_seq_len, max_seq_len)

 89class SlidingWindowGQA(nn.Module):
 90    def __init__(self, config) -> None:
 91        super().__init__()
 92        self.dk = config.dk
 93        self.H = config.H
 94        self.n_kv_heads = config.n_kv_heads
 95        self.n_rep = self.H // config.n_kv_heads
 96        self.wq = nn.Linear(config.dmodel, config.H * self.dk, bias=False)
 97        self.wk = nn.Linear(config.dmodel, config.n_kv_heads * self.dk, bias=False)
 98        self.wv = nn.Linear(config.dmodel, config.n_kv_heads * self.dk, bias=False)
 99        self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)
100
101    @staticmethod
102    def repeat_kv(x, n_rep):
103        if n_rep == 1:
104            return x
105        B, n_kv_heads, S, dk = x.shape
106        x = x.unsqueeze(2).expand(B, n_kv_heads, n_rep, S, dk)
107        return x.contiguous().view(B, n_kv_heads * n_rep, S, dk)  # (B, H, S, dk)
108
109    def forward(self, x, rope, mask):
110        B, S, _ = x.shape  # x: (B, S, dmodel)
111        query = self.wq(x).view(B, S, self.H, self.dk).transpose(1, 2)  # (B, H, S, dk)
112        key = self.wk(x).view(B, S, self.n_kv_heads, self.dk).transpose(1, 2)  # (B, n_kv, S, dk)
113        value = self.wv(x).view(B, S, self.n_kv_heads, self.dk).transpose(1, 2)  # (B, n_kv, S, dk)
114        query = rope(query)  # (B, H, S, dk)
115        key = rope(key)  # (B, n_kv, S, dk)
116        key = SlidingWindowGQA.repeat_kv(key, self.n_rep)  # (B, H, S, dk)
117        value = SlidingWindowGQA.repeat_kv(value, self.n_rep)  # (B, H, S, dk)
118        attn_score = query @ key.transpose(-1, -2) / math.sqrt(self.dk)  # (B, H, S, S)
119        attn_score = attn_score.masked_fill(mask[:S, :S] == 0, float("-inf"))
120        attn_score = F.softmax(attn_score, dim=-1)  # (B, H, S, S)
121        x = attn_score @ value  # (B, H, S, dk)
122        x = x.transpose(1, 2).contiguous().view(B, S, self.H * self.dk)  # (B, S, dmodel)
123        return self.wo(x)  # (B, S, dmodel)

▼ Show original version

LLaMA in chapter 03: GQABlock with a full causal mask torch.tril(...) — every past token is visible (excerpt from llama.py).

llama.pyGQABlock + full causal mask

 81class GQABlock(nn.Module):
 82    def __init__(self, config) -> None:
 83        super().__init__()
 84        self.dk = config.dk
 85        self.H = config.H
 86        self.n_kv_heads = config.n_kv_heads
 87        self.n_rep = self.H // config.n_kv_heads
 88        self.wq = nn.Linear(config.dmodel, config.H * self.dk, bias=False)  # dmodel → H*dk
 89        self.wk = nn.Linear(config.dmodel, config.n_kv_heads * self.dk, bias=False)
 90        self.wv = nn.Linear(config.dmodel, config.n_kv_heads * self.dk, bias=False)
 91        self.wo = nn.Linear(config.dmodel, config.dmodel, bias=False)  # dmodel → dmodel
 92
 93    @staticmethod
 94    def repeat_kv(x, n_rep):
 95        if n_rep == 1:
 96            return x
 97        B, n_kv_heads, S, dk = x.shape
 98        x = x.unsqueeze(2).expand(B, n_kv_heads, n_rep, S, dk)
 99        return x.contiguous().view(B, n_kv_heads * n_rep, S, dk)  # (B, H, S, dk)
100
101    def forward(self, x, rope, mask):
102        B, S, _ = x.shape  # x: (B, S, dmodel)
103        query = self.wq(x).view(B, S, self.H, self.dk).transpose(1, 2)  # (B, H, S, dk)
104        key = self.wk(x).view(B, S, self.n_kv_heads, self.dk).transpose(1, 2)  # (B, n_kv, S, dk)
105        value = self.wv(x).view(B, S, self.n_kv_heads, self.dk).transpose(1, 2)  # (B, n_kv, S, dk)
106        query = rope(query)  # (B, H, S, dk)
107        key = rope(key)  # (B, n_kv, S, dk)
108        key = GQABlock.repeat_kv(key, self.n_rep)  # (B, H, S, dk)
109        value = GQABlock.repeat_kv(value, self.n_rep)  # (B, H, S, dk)
110        attn_score = query @ key.transpose(-1, -2) / math.sqrt(self.dk)  # (B, H, S, S)
111        attn_score = attn_score.masked_fill(mask[:S, :S] == 0, float("-inf"))
112        attn_score = F.softmax(attn_score, dim=-1)  # (B, H, S, S)
113        x = attn_score @ value  # (B, H, S, dk)
114        x = x.transpose(1, 2).contiguous().view(B, S, self.H * self.dk)  # (B, S, dmodel)
115        return self.wo(x)  # (B, S, dmodel)

142        self.register_buffer(
143            "mask", torch.tril(torch.ones(config.max_seq_len, config.max_seq_len))
144        )  # (max_seq_len, max_seq_len)

06 · BLOCK

RMSNorm + SlidingWindowGQA + RMSNorm + SwiGLU

Block structure is unchanged from LLaMA: two RMSNorms, one attention module, one SwiGLU FFN, pre-norm residuals. The only swap is GQABlock → SlidingWindowGQA; the block still passes (x, rope, mask) through.

mistral.pyBlock

126class Block(nn.Module):
127    def __init__(self, config) -> None:
128        super().__init__()
129        self.gqa = SlidingWindowGQA(config)
130        self.ff = SwiGLUFeedForward(config)
131        self.rms_1 = RMSNorm(config)  # pre-norm before attention
132        self.rms_2 = RMSNorm(config)  # pre-norm before FFN
133
134    def forward(self, x, rope, mask):
135        # x: (B, S, dmodel)
136        x = x + self.gqa(self.rms_1(x), rope, mask)  # (B, S, dmodel)
137        x = x + self.ff(self.rms_2(x))  # (B, S, dmodel)
138        return x  # (B, S, dmodel)

▼ Show original version

LLaMA in chapter 03: same block layout with GQABlock (excerpt from llama.py).

llama.pyBlock

118class Block(nn.Module):
119    def __init__(self, config) -> None:
120        super().__init__()
121        self.gqa = GQABlock(config)
122        self.ff = SwiGLUFeedForward(config)
123        self.rms_1 = RMSNorm(config)  # pre-norm before attention
124        self.rms_2 = RMSNorm(config)  # pre-norm before FFN
125
126    def forward(self, x, rope, mask):
127        # x: (B, S, dmodel)
128        x = x + self.gqa(self.rms_1(x), rope, mask)  # (B, S, dmodel)
129        x = x + self.ff(self.rms_2(x))  # (B, S, dmodel)
130        return x  # (B, S, dmodel)

07 · ASSEMBLE MISTRAL

Token embed, RoPE, N blocks, final norm, lm_head

Mistral owns everything. wte maps token IDs to vectors. A single RotaryPositionalEmbedding is shared across all blocks. N Blocks via nn.ModuleList, each taking (x, rope, mask). Final RMSNorm then lm_head projects to vocab with weight tying.

The delta from LLaMA is the registered mask: make_sliding_window_mask(max_seq_len, sliding_window) instead of a plain causal torch.tril. With the §00 defaults, that is the full Mistral 7B stack.

mistral.pyMistral

141class Mistral(nn.Module):
142    def __init__(self, config) -> None:
143        super().__init__()
144        self.wte = nn.Embedding(config.vocabsize, config.dmodel)  # wte(x): (B, S, dmodel)
145        self.rope = RotaryPositionalEmbedding(config)
146        self.blocks = nn.ModuleList([Block(config) for _ in range(config.N)])
147        self.norm = RMSNorm(config)  # (B, S, dmodel) → (B, S, dmodel)
148        self.lm_head = nn.Linear(config.dmodel, config.vocabsize, bias=False)
149        self.lm_head.weight = self.wte.weight  # weight tying
150        # sliding window mask — causal ∩ band of width W
151        self.register_buffer(
152            "mask",
153            make_sliding_window_mask(config.max_seq_len, config.sliding_window),
154        )  # (max_seq_len, max_seq_len)
155
156    def forward(self, x):
157        _, S = x.shape  # x: (B, S)
158        x = self.wte(x)  # (B, S, dmodel)
159        for block in self.blocks:
160            x = block(x, self.rope, self.mask)  # (B, S, dmodel)
161        x = self.norm(x)  # (B, S, dmodel)
162        return self.lm_head(x)  # (B, S, vocabsize)

08 · COMPARISON

LLaMA 3 (03) vs Mistral 7B (04)

Canonical comparison for this chapter — same pair as the delta card above, with code-level detail after you have read the stack.

      
        AxisLLaMA 3 8B (03)Mistral 7B (04)

        Design goalScale + capability (family to 405B)Efficiency at ~7B (dense, fast inference)
Attention maskFull causal — every past token visibleSliding window — causal ∩ band W=4096
Attention moduleGQABlockSlidingWindowGQA (same Q/K/V projections)
GQA32 Q · 8 KV (4:1)32 Q · 8 KV (same)
KV cache (inference)Grows with sequence lengthRolling buffer bounded by W
RoPEHigher rope_theta for long contextStandard θ=10,000 (v0.1)
Embeddings / FFNwte · RMSNorm · SwiGLU · bias=FalseSame stack
Context / vocab8,192 base; 3.1+ extends with RoPE scaling · 128,256 vocab32k max_position_embeddings (HF v0.1); local SWA band 4,096 · 32,000 vocab
Scale8B · 70B · 405B family~7.3B dense (efficiency-focused)

      
    

Axis	LLaMA 3 8B (03)	Mistral 7B (04)
Design goal	Scale + capability (family to 405B)	Efficiency at ~7B (dense, fast inference)
Attention mask	Full causal — every past token visible	Sliding window — causal ∩ band W=4096
Attention module	`GQABlock`	`SlidingWindowGQA` (same Q/K/V projections)
GQA	32 Q · 8 KV (4:1)	32 Q · 8 KV (same)
KV cache (inference)	Grows with sequence length	Rolling buffer bounded by W
RoPE	Higher `rope_theta` for long context	Standard θ=10,000 (v0.1)
Embeddings / FFN	`wte` · RMSNorm · SwiGLU · `bias=False`	Same stack
Context / vocab	8,192 base; 3.1+ extends with RoPE scaling · 128,256 vocab	32k `max_position_embeddings` (HF v0.1); local SWA band 4,096 · 32,000 vocab
Scale	8B · 70B · 405B family	~7.3B dense (efficiency-focused)

Mistral 7B is a LLaMA-style transformer tuned for fast inference via sliding-window attention; LLaMA 3 keeps full global attention and scales tokenizer, context, and model family size for stronger general capability.

09 · REFERENCES

Papers & technical sources

Primary reports and references for this chapter. Read these for full equations, training details, and official hyperparameters.

Mistral 7B (Jiang et al., 2023)Sliding-window attention, GQA, SwiGLU — efficiency-focused 7B dense model.
Announcing Mistral 7B (Mistral AI)Product blog with sliding-window and rolling-buffer KV cache context.

Delta from LLaMA 3 (03)

One config object for the whole model

Token embedding only — unchanged from LLaMA

Rotary positional embeddings — unchanged from LLaMA

Root-mean-square normalization — unchanged from LLaMA

SwiGLU feed-forward — unchanged from LLaMA

Grouped-query attention with a sliding window mask

RMSNorm + SlidingWindowGQA + RMSNorm + SwiGLU

Token embed, RoPE, N blocks, final norm, lm_head

LLaMA 3 (03) vs Mistral 7B (04)

Papers & technical sources

Comments