75 LLM interview questions — from foundational concepts to senior-level traps drawn from Google DeepMind, OpenAI, Meta, and Anthropic. Test yourself, reveal answers, and track your progress.
Core building blocks — what LLMs are, how text becomes numbers, and how models manage vocabulary.
LLMs are AI systems trained on vast text corpora to understand and generate human-like language. With billions of parameters, they excel in tasks like translation, summarization, and question answering, leveraging contextual learning for broad applicability.
What makes them "large" is the combination of massive model size (billions of weights), enormous training data (terabytes of text), and the emergent abilities that appear only at scale — like in-context learning and chain-of-thought reasoning.
Tokenization involves breaking down text into smaller units, or tokens, such as words, subwords, or characters. For example, "artificial" might be split into "art," "ific," and "ial."
This process is vital because LLMs process numerical representations of tokens, not raw text. Tokenization enables models to handle diverse languages, manage rare or unknown words, and optimize vocabulary size — enhancing computational efficiency and model performance.
Key insight: The choice of tokenizer (BPE, WordPiece, SentencePiece) directly affects model performance. A well-designed tokenizer balances vocabulary size against sequence length.
Embeddings are dense vectors that represent tokens in a continuous space, capturing semantic and syntactic properties. They are often initialized randomly or with pretrained models like GloVe, then fine-tuned during training.
For example, the embedding for "dog" might evolve to reflect its context in pet-related tasks, enhancing model accuracy. The embedding dimension (e.g., 768 for BERT, 12288 for GPT-3) is a critical architectural choice that determines the model's representational capacity.
LLMs use subword tokenization, like Byte-Pair Encoding (BPE), to break OOV words into known subword units. For instance, "cryptocurrency" might split into "crypto" and "currency."
This approach allows LLMs to process rare or new words, ensuring robust language understanding and generation. Other methods include WordPiece (used by BERT) and Unigram (used by SentencePiece), each with different merge strategies for building the subword vocabulary.
The context window refers to the number of tokens an LLM can process at once, defining its "memory" for understanding or generating text. A larger window, like 32,000 tokens, allows the model to consider more context, improving coherence in tasks like summarization.
However, it increases computational costs quadratically with standard attention (O(n²)). Balancing window size with efficiency is crucial for practical LLM deployment. Modern approaches like sliding window attention, sparse attention, and RoPE scaling help extend context without full quadratic cost.
LLMs use transformer architectures, massive datasets, and unsupervised pretraining, unlike statistical models (e.g., N-grams) that rely on simpler, supervised methods.
LLMs handle long-range dependencies, contextual embeddings, and diverse tasks, but require significant computational resources. Traditional N-gram models struggle beyond short contexts (typically 3-5 tokens), while transformers can attend to thousands of tokens and capture complex semantic relationships.
Architecture fundamentals — attention, encoders, decoders, and what makes transformers superior to RNNs.
Sequence-to-sequence (Seq2Seq) models transform an input sequence into an output sequence, often of different lengths. They consist of an encoder to process the input and a decoder to generate the output.
Applications include machine translation (e.g., English to Spanish), text summarization, and chatbots, where variable-length inputs and outputs are common. The original Seq2Seq used RNNs/LSTMs, but modern implementations use transformers for better parallelism and long-range dependency handling.
Transformers overcome Seq2Seq limitations through three key innovations:
These features enhance scalability and performance in tasks like translation, and enable training on much larger datasets efficiently.
The attention mechanism allows LLMs to weigh the importance of different tokens in a sequence when generating or interpreting text. It computes similarity scores between query, key, and value vectors, using operations like dot products, to focus on relevant tokens.
For instance, in "The cat chased the mouse," attention helps the model link "mouse" to "chased." This mechanism improves context understanding, making transformers highly effective for NLP tasks. Each token creates a Q, K, V vector, and attention scores determine how much each token "attends to" every other token.
Positional encodings add sequence order information to transformer inputs, as self-attention lacks inherent order awareness. Using sinusoidal functions or learned vectors, they ensure tokens like "king" and "crown" are interpreted correctly based on position.
The original transformer used fixed sinusoidal encodings. Modern models often use Rotary Position Embeddings (RoPE) which encode relative positions through rotation matrices, or ALiBi which adds linear biases to attention scores. These are critical for tasks like translation where word order changes meaning.
Multi-head attention splits queries, keys, and values into multiple subspaces, allowing the model to focus on different aspects of the input simultaneously.
For example, in a sentence, one head might focus on syntax, another on semantics, and another on coreference. This improves the model's ability to capture complex patterns. Modern variants include Grouped Query Attention (GQA) — used in LLaMA 2+ — which shares key-value heads across query heads, and Multi-Query Attention (MQA) which uses a single KV head for all queries, trading some quality for significantly faster inference.
Encoders process input sequences into abstract representations, capturing context through bidirectional self-attention. Decoders generate outputs token-by-token, using encoder outputs and prior tokens through causal (masked) self-attention.
In translation, the encoder understands the source language, and the decoder produces the target language. Encoder-only models (BERT) excel at understanding tasks, decoder-only models (GPT) excel at generation, and encoder-decoder models (T5) handle sequence-to-sequence tasks.
The math under the hood — softmax, scaled dot products, and the full attention formula.
The softmax function normalizes attention scores into a probability distribution:
In attention, it converts raw similarity scores (from query-key dot products) into weights, emphasizing relevant tokens. This ensures the model focuses on contextually important parts of the input. The exponential function amplifies differences — high-scoring tokens get disproportionately more weight, creating a "soft" selection mechanism.
In self-attention, the dot product between query (Q) and key (K) vectors computes similarity scores:
High scores indicate relevant tokens. The scaling factor √d_k (square root of key dimension) prevents dot products from growing too large, which would push softmax into regions with extremely small gradients.
While efficient, its quadratic complexity O(n²) for long sequences has spurred research into sparse attention alternatives like Flash Attention, which computes exact attention with reduced memory through tiling.
The complete attention formula brings Q, K, V together:
The scaled dot product measures token relevance, and softmax normalizes scores to focus on key tokens. The result is multiplied by the value matrix V to produce the weighted combination of value vectors — the actual output of the attention layer.
This single formula is the core of every transformer. Everything else (multi-head, causal masking, KV cache) is built around it.
How models learn — masked vs. autoregressive training, next-sentence prediction, and the generative/discriminative divide.
Masked language modeling (MLM) involves hiding random tokens in a sequence and training the model to predict them based on context. Used in models like BERT, MLM fosters bidirectional understanding of language.
Typically 15% of tokens are masked. Of those, 80% are replaced with [MASK], 10% with a random token, and 10% kept unchanged. This strategy prevents the model from simply learning to detect [MASK] tokens and encourages robust contextual representations useful for downstream tasks like sentiment analysis or question answering.
Autoregressive models (like GPT) predict tokens sequentially based on prior tokens, excelling in generative tasks such as text completion. They see only left context (causal attention).
Masked models (like BERT) predict masked tokens using bidirectional context, making them ideal for understanding tasks like classification. They see the full sequence minus the masked positions.
Their training objectives shape their strengths: autoregressive models are natural generators, while masked models produce better representations for downstream classification and extraction tasks.
Next sentence prediction (NSP) trains models to determine if two sentences are consecutive or unrelated. During pretraining, models like BERT learn to classify 50% positive (sequential) and 50% negative (random) sentence pairs.
NSP improves coherence in tasks like dialogue systems or document summarization by understanding sentence relationships. However, later research (RoBERTa) showed that NSP provides minimal benefit and can even hurt performance — leading most modern models to drop it in favor of longer contiguous sequences.
Generative models (like GPT) model joint probabilities P(x, y) to create new data, such as text or images. Discriminative models (like BERT for classification) model conditional probabilities P(y|x) to distinguish classes.
Generative models excel in creation tasks (writing, code generation), while discriminative models focus on accurate classification (sentiment analysis, NER). In practice, the line has blurred — modern LLMs can do both by framing classification as generation.
Discriminative AI, like sentiment classifiers, predicts labels based on input features, modeling conditional probabilities. Generative AI, like GPT, creates new data by modeling joint probabilities, suitable for tasks like text or image generation.
Discriminative models tend to be more sample-efficient for specific classification tasks, while generative models offer creative flexibility and can be adapted to many tasks through prompting. The generative approach has become dominant because one large pretrained model can handle diverse tasks.
Turning probabilities into words — beam search, temperature, and sampling strategies.
Beam search explores multiple word sequences during text generation, keeping the top k candidates (beams) at each step, unlike greedy decoding, which selects only the most probable word.
This approach, with k = 5 for instance, ensures more coherent outputs by balancing probability and diversity. It's especially effective in tasks like machine translation or dialogue generation where the locally best token isn't always globally optimal.
Trade-off: Higher beam width improves quality but increases computation linearly. In practice, beam widths of 4-10 are common, and some tasks benefit more from sampling-based methods instead.
Temperature is a hyperparameter that adjusts the randomness of token selection in text generation by scaling the logits before softmax.
Setting temperature to ~0.7-0.8 often balances creativity and coherence for tasks like storytelling, while ~0.2 works well for factual Q&A.
Top-k sampling selects the k most probable tokens (e.g., k = 20) for random sampling, ensuring controlled diversity. The number of candidates is fixed regardless of the probability distribution.
Top-p (nucleus) sampling chooses tokens whose cumulative probability exceeds a threshold p (e.g., 0.95), adapting to context. When the model is confident, fewer tokens are considered; when uncertain, more are included.
Top-p offers more flexibility, producing varied yet coherent outputs. In practice, many systems combine both: apply top-k first to cap the maximum candidates, then top-p to further narrow based on probability mass.
Adapting pretrained models — LoRA, PEFT, distillation, and preventing catastrophic forgetting.
LoRA (Low-Rank Adaptation) adds low-rank matrices (A and B, where ΔW = BA) to a model's layers, enabling efficient adaptation with minimal memory overhead. Only A and B are trained — the original weights stay frozen.
QLoRA extends this by applying quantization (e.g., 4-bit NormalFloat precision) to the frozen base weights, further reducing memory usage while maintaining accuracy. For example, QLoRA can fine-tune a 70B-parameter model on a single GPU, making it ideal for resource-constrained environments.
Key difference: LoRA reduces trainable parameters; QLoRA additionally reduces memory for stored parameters through quantization. Both keep the base model frozen.
Catastrophic forgetting occurs when fine-tuning erases prior knowledge. Mitigation strategies include:
These methods ensure LLMs retain versatility across tasks while still learning new capabilities.
Parameter-Efficient Fine-Tuning (PEFT) updates only a small subset of parameters, freezing the rest to preserve pretrained knowledge. Techniques include:
By modifying less than 1% of total parameters, PEFT ensures LLMs adapt to new tasks without losing core capabilities, maintaining performance across domains.
Model distillation trains a smaller "student" model to mimic a larger "teacher" model's outputs, using soft probabilities (logits) rather than hard labels. The soft targets carry richer information about the teacher's learned distribution.
This reduces memory and computational requirements, enabling deployment on devices like smartphones while retaining near-teacher performance. Examples include DistilBERT (40% smaller, 97% of BERT's performance) and various distilled versions of LLaMA. It's ideal for real-time applications where latency matters.
Adaptive Softmax groups words by frequency, reducing computations for rare words. Common words get full-dimension representations while rare words use smaller clusters with lower-dimensional projections.
This lowers the cost of handling large vocabularies (100k+ tokens), speeding up training and inference while maintaining accuracy. It's especially valuable in resource-limited settings where computing softmax over the full vocabulary at every step is prohibitively expensive.
Getting the best out of LLMs without retraining — prompt engineering, zero-shot, few-shot, and chain-of-thought.
Prompt engineering involves designing inputs to elicit desired LLM responses. A clear prompt, like "Summarize this article in 100 words," improves output relevance compared to vague instructions.
It's especially effective in zero-shot or few-shot settings, enabling LLMs to tackle tasks like translation or classification without extensive fine-tuning. Key techniques include specifying output format, providing role context ("You are an expert in..."), using delimiters for structure, and iteratively refining based on model responses.
CoT prompting guides LLMs to solve problems step-by-step, mimicking human reasoning. For example, in math problems, it breaks down calculations into logical steps rather than jumping to the answer.
This improves accuracy and interpretability in complex tasks like logical inference, multi-step queries, and arithmetic. Variants include zero-shot CoT (just adding "Let's think step by step"), few-shot CoT (providing examples with reasoning traces), and self-consistency (sampling multiple reasoning paths and taking the majority vote).
Zero-shot learning allows LLMs to perform untrained tasks using general knowledge from pretraining. For example, prompted with "Classify this review as positive or negative," an LLM can infer sentiment without task-specific data.
This showcases the model's versatility and is possible because of the broad knowledge encoded during pretraining on diverse text. Larger models tend to have stronger zero-shot capabilities — a key finding from the GPT-3 paper that demonstrated scaling enables emergent abilities.
Few-shot learning enables LLMs to perform tasks with minimal examples (typically 1-10), leveraging pretrained knowledge. The examples are provided in the prompt as demonstrations.
Benefits include reduced data needs, faster adaptation, and cost efficiency, making it ideal for niche tasks like specialized text classification. Unlike fine-tuning, few-shot learning requires no gradient updates — the model uses in-context learning to generalize from the provided examples at inference time.
Cutting-edge approaches — GPT-4, Gemini, MoE, RAG, knowledge graphs, and foundation model taxonomy.
GPT-4 surpasses GPT-3 with several key advances:
These improvements expand its use in visual question answering, complex dialogues, code generation, and professional applications like legal analysis and medical reasoning.
Gemini enhances efficiency through several innovations:
These features make Gemini more stable and scalable than models that retrofit multimodality onto text-only architectures.
Foundation models span multiple modalities:
These models leverage broad pretraining for diverse applications and can be fine-tuned for specific downstream tasks.
MoE uses a gating function (router) to activate specific expert sub-networks per input, reducing computational load. For example, only 10% of a model's parameters might be used per query.
This enables billion-parameter models to operate efficiently while maintaining high performance. Mixtral 8x7B, for instance, has 47B total parameters but only activates ~13B per token. DeepSeek V3 pushed this further with 671B total parameters and fine-grained expert selection. The key challenge is load balancing — ensuring all experts get trained evenly.
RAG combines retrieval with generation in three steps:
RAG enhances factual accuracy, reduces hallucinations, and allows models to access up-to-date information without retraining. It's the dominant pattern for enterprise LLM applications.
Knowledge graphs provide structured, factual data that enhances LLMs by:
This is particularly valuable for question answering, entity recognition, and domain-specific applications (medical, legal) where factual accuracy is non-negotiable.
The math that makes it all work — loss functions, gradients, eigenvalues, KL divergence, and backpropagation through transformers.
Cross-entropy loss measures the divergence between predicted and true token probabilities:
It penalizes incorrect predictions, encouraging accurate token selection. In language modeling, it ensures the model assigns high probabilities to correct next tokens. The logarithm means that assigning very low probability to the correct token is heavily penalized, while getting close to 1.0 gives diminishing returns — exactly the behavior we want for training.
Gradients for embeddings are computed using the chain rule during backpropagation:
These gradients adjust embedding vectors to minimize loss, refining their semantic representations for better task performance. Only the embedding rows corresponding to tokens in the current batch receive non-zero gradients — making embedding updates sparse and efficient.
The Jacobian matrix captures partial derivatives of outputs with respect to inputs. For a function f: R^n → R^m, the Jacobian is an m × n matrix where entry (i,j) is ∂f_i/∂x_j.
In transformers, it helps compute gradients for multidimensional outputs (like the attention scores matrix), ensuring accurate updates to weights and embeddings during backpropagation. This is critical for optimizing complex models where outputs are vector-valued rather than scalar.
Eigenvectors define principal directions in data, and eigenvalues indicate their variance (importance). In techniques like PCA, selecting eigenvectors with high eigenvalues reduces dimensionality while retaining most variance.
This enables efficient data representation for LLM input processing, visualization of embedding spaces, and compression of hidden states. PCA on embedding matrices is commonly used to understand what semantic dimensions the model has learned.
KL divergence quantifies the difference between two probability distributions:
In LLMs, it evaluates how closely model predictions match true distributions, guiding fine-tuning to improve output quality. It's central to RLHF — the KL penalty term prevents the policy model from diverging too far from the reference model, maintaining generation quality while aligning with human preferences.
Important: KL divergence is asymmetric — D_KL(P||Q) ≠ D_KL(Q||P). The choice of direction matters: forward KL (mode-covering) vs reverse KL (mode-seeking) produce different behaviors in training.
The ReLU function, f(x) = max(0, x), has a simple derivative:
Its sparsity (zeroing out negative values) and non-linearity prevent vanishing gradients, making ReLU computationally efficient and widely used in neural networks. However, the "dying ReLU" problem (neurons stuck at 0) led to variants like Leaky ReLU, GELU (used in GPT), and SwiGLU (used in LLaMA/modern LLMs).
The chain rule computes derivatives of composite functions:
In gradient descent, it enables backpropagation to calculate gradients layer by layer, updating parameters to minimize loss efficiently across deep LLM architectures. Without the chain rule, computing gradients through dozens of transformer layers (each a composition of attention, layer norm, and FFN) would be intractable.
Transformers mitigate vanishing gradients through three mechanisms:
Together, these ensure effective training of very deep models (100+ layers), unlike RNNs which struggle with sequences longer than a few hundred tokens.
Real-world challenges — overfitting, bias, hyperparameters, and the hurdles of putting LLMs into production.
Overfitting occurs when a model memorizes training data, failing to generalize to unseen inputs. Mitigation includes:
For LLMs specifically, using diverse training data and appropriate model size relative to dataset size are the most important factors.
A systematic approach to addressing biased or incorrect outputs:
These steps improve fairness and accuracy while maintaining model utility.
Hyperparameters are preset values that control model training — they are not learned from data but set before training begins. Examples include learning rate, batch size, number of layers, and attention heads.
They critically influence convergence and performance. A high learning rate may cause instability and divergence, while too low a rate leads to slow training. Modern LLM training uses sophisticated schedules: warmup + cosine decay for learning rate, gradient accumulation for effective batch size, and careful attention to the interplay between all hyperparameters.
Key deployment challenges include:
Addressing these through careful system design, guardrails, and monitoring ensures ethical and effective LLM use in production.
Senior-level traps from top AI labs — learning rate warmup, gradient clipping, optimizer memory, and curriculum ordering.
Most candidates trust Adam's adaptive rates to handle initialization automatically. At Step 0, however, Transformer layers are completely misaligned and gradient variance is effectively infinite.
Applying a full-magnitude update immediately pushes parameters into catastrophically steep curvature — the loss diverges within 50 steps. The fix is a linear warm-up schedule: start the learning rate at 0, increase linearly over the first ~4,000 steps until gradient variance stabilizes, then switch to an inverse-square-root decay.
The answer that gets you hired: "Transformers don't tolerate high learning rates at initialization due to high gradient variance. I would implement a linear warm-up from 0 to d_model^(-0.5) to let the curvature stabilize, preventing early divergence before switching to the decay schedule."
Most candidates drop the learning rate drastically. This slows convergence to a crawl — the gradient direction is correct; only the magnitude is too large for the current loss curvature.
The targeted fix is Global L2 Gradient Clipping: if ||g||₂ > 1.0, rescale g ← g / ||g||₂. This caps the maximum step size while strictly preserving the descent direction — unlike a blanket learning-rate cut, which wastes GPU-hours across the entire run.
The answer that gets you hired: "I don't touch the learning rate. I apply Global Gradient Clipping (norm = 1.0), which decouples step size from gradient magnitude and prevents divergence without stalling convergence."
Most candidates optimize for VRAM and set a comfortable per-GPU batch. In supervised learning, batch size only affects gradient estimation. In contrastive learning, batch size IS the training signal: N samples yield N² − N negatives per step.
At N = 512, the model distinguishes each positive from only 511 distractors — trivially easy, so it learns shortcut features. The original CLIP paper required massive batches for quality, not speed. Target a global batch of 4,096+, using all_gather across GPUs and gradient caching to simulate larger matrices if memory is limited.
The answer that gets you hired: "In contrastive learning, batch size is a quality hyperparameter, not just a hardware choice. If we can't reach 4k+ samples per step, we shouldn't start training."
Most candidates guess computational cost or batch size. The real cause is the 3x memory rule: unlike stateless SGD, Adam stores two additional scalar states per parameter — first moment (momentum) and second moment (variance).
A 7B model in FP16 occupies ~14 GB for weights; Adam's optimizer states add ~28 GB more, tripling the total VRAM footprint without changing a single parameter count. The fix is PagedAdam, which offloads optimizer states to CPU memory and fetches them only at each update step.
The answer that gets you hired: "Adam imposes a 3x memory penalty over model weights alone due to stored momentum and variance states. I would use PagedAdam to offload those states to CPU, or plan the cluster allocation to account for the 3x overhead upfront."
Most candidates say "Adam uses momentum for faster convergence." The real mechanism is frequency bias: SGD updates are proportional to gradient magnitude, so tokens appearing once every 10,000 documents receive tiny, infrequent updates and stay near random initialization.
Adam tracks the running variance v_t of each parameter's gradient; for a rare feature, v_t stays near zero. Adam's update divides by √v_t, automatically amplifying the effective learning rate for underrepresented parameters — like an equalizer boosting quiet frequencies.
The answer that gets you hired: "SGD fails because updates are proportional to feature frequency. Adam's variance normalization — dividing by √v_t — amplifies updates for rare features, ensuring they are learned despite their sparsity."
Most candidates shuffle all three together to avoid catastrophic forgetting. DeepSeek's experiments show this caps the ceiling for complex reasoning.
The optimal curriculum is General Text → Code → Math. Code is not trained for programming proficiency — it teaches the model the grammar of logic: strict sequential dependency tracking that is a prerequisite for mathematical deduction. Skipping Code and jumping straight to Math produces a model that has memorized formulas but lacks the internal execution stack to apply them to novel multi-step problems.
The answer that gets you hired: "Code is a reasoning primitive, not a domain task. General → Code → Math is optimal because code training aligns the model's internal representations for step-by-step chain-of-thought before we demand it on mathematics."
The pitfalls that trip up 90% of candidates — backbone unfreezing, LoRA initialization, QLoRA compute tax, and quantization-aware training.
Most candidates unfreeze everything immediately, arguing it converges faster. The danger is the Gradient Shockwave: the randomly initialized head produces mathematically incoherent gradients that shatter pre-trained backbone features before the head even knows which direction is "up."
The correct two-phase protocol: (1) Freeze the backbone and train only the head until it is sensible (effectively a convex problem, fast convergence), then (2) unfreeze the backbone and fine-tune the full model with a much lower learning rate (e.g., 1e-5).
The answer that gets you hired: "Never fine-tune a backbone against a random head. We must align the head to the existing feature space before we allow the feature space to shift."
Most candidates apply Xavier or Kaiming initialization to both matrices. LoRA inserts a residual update W_new = W_frozen + B × A: if both are random, their product is random noise injected directly into the pre-trained weights at Step 0, destroying existing knowledge before training begins.
The correct asymmetric initialization: A sampled from a Gaussian (breaks symmetry, enables gradient flow), B = 0 (all zeros). This makes B × A = 0, preserving the pre-trained model as a mathematical identity until B slowly moves away from zero during training.
The answer that gets you hired: "We initialize the up-projection to zero and the down-projection to Gaussian random. This makes the adapter an identity function at Step 0, preserving pre-trained weights while keeping gradient flow active."
Most candidates equate smaller storage with faster computation. QLoRA is a memory optimization, not a compute optimization. Weights are stored in VRAM as INT4, but NVIDIA Tensor Cores do not execute backpropagation in INT4.
Before every matrix multiplication, the kernel must dequantize those 4-bit weights back to BF16 — an "unzipping tax" added to every forward and backward pass. If the training run was not previously memory-bandwidth-bound, this extra step is pure overhead. QLoRA trades wall-clock time for memory capacity: the ability to fit a model that would otherwise not fit at all.
The answer that gets you hired: "QLoRA adds on-the-fly dequantization overhead before every matrix multiply. We accept slightly slower wall-clock training in exchange for fitting a much larger model — it is a capacity trade-off, not a speed optimization."
Most candidates stop at "rounding is non-differentiable, so gradients vanish." The production fix is the Straight-Through Estimator (STE): decouple the two passes.
Forward pass: Apply true quantization (rounding) so the model experiences real discrete-precision errors. Backward pass: Approximate the rounding derivative as the identity function (∂round/∂x ≈ 1), passing gradients unchanged through the rounding node to the full-precision weights. A gradient mismatch is accepted because a noisy gradient is infinitely better than a zero gradient.
The answer that gets you hired: "We decouple the passes. The forward pass uses quantized weights to simulate inference precision; the backward pass applies the Straight-Through Estimator, treating rounding as an identity function to keep gradients flowing."
Where things break in production — speculative decoding, dropout at inference, and model cascade bottlenecks.
Most candidates argue that speculative decoding adds overhead when the GPU is already compute-saturated at large batches. That logic holds for short contexts but breaks for long-context inference (100k+ tokens).
Each request carries a unique KV cache; moving 128 massive per-request caches from VRAM to compute units every decode step makes memory bandwidth the bottleneck, not FLOPs. Self-speculative KV compression exploits this: speculate the next token using only the top 1% of the KV cache (cheap to load), then verify with the full cache — near-zero extra cost because the hardware was already I/O bound.
The answer that gets you hired: "For long-context inference, high batch sizes remain memory-bound due to unique per-request KV caches. Self-speculation bypasses the I/O bottleneck by trading cheap compute for expensive memory bandwidth."
Most candidates say "just turn off the random masking." This causes magnitude mismatch: with p = 0.5 dropout during training, the next layer learned to expect signals from only half its inputs. Suddenly activating all neurons doubles the incoming signal, saturating activations.
The production fix is inverted dropout: scale surviving activations by 1/(1 − p) during training so that the expected magnitude matches the full network. At inference, simply remove the mask — no post-hoc weight scaling needed because the training-time scaling already calibrated the magnitudes.
The answer that gets you hired: "I use inverted dropout: activations are scaled by 1/(1 − p) during training so the expected magnitude is already calibrated. At inference I simply remove the mask — no weight adjustment needed."
Most candidates cite routing latency or query difficulty classifiers. The fundamental bottleneck is confidence calibration: LLMs are notoriously overconfident — a log-probability of 0.99 often says nothing about factual accuracy.
Without a calibrated confidence signal or a lightweight verifier, the cheap model cannot reliably identify its own failures. The cascade then faces two failure modes: (1) confident hallucinations pass through unchecked, or (2) the system routes everything to the expensive model anyway, eliminating all cost savings.
The answer that gets you hired: "The bottleneck is failure detection. Without a calibrated confidence score or lightweight verifier, we can't reliably distinguish easy from hard — we either pass hallucinations through or route everything to the expensive model."
Traps in measuring model quality — attention entropy illusions, perplexity comparison, PPO vs. DPO, bias detection, and annotator agreement.
Most candidates reason that peaked attention signals confidence. This is incorrect: inference-time attention is fully deterministic. There is no sampling, so entropy of the weights measures information dispersal, not probabilistic confidence — a model can attend sharply to one token while hallucinating.
Real uncertainty requires stochasticity: Monte Carlo Dropout (keep dropout active at inference, run N forward passes, measure output variance) or a Variational Autoencoder with an explicit sampled latent variable.
The answer that gets you hired: "Attention weights are deterministic mixing coefficients, not probabilities. To measure uncertainty I would use MC Dropout — running multiple stochastic forward passes and measuring the variance."
Most candidates check King − Man + Woman = Queen, which only verifies semantic correctness. The WEAT (Word Embedding Association Test) exposes stereotypical projections: it measures the cosine-similarity gap between sets of neutral concepts (e.g., {Math, Physics}) and gendered attribute words ({he, him} vs. {she, her}).
A low p-value on the effect-size test reveals that the embedding space is projecting gender onto neutral domains, regardless of whether surface analogies pass.
The answer that gets you hired: "Analogies only test semantic correctness, not allocational harm. To prove safety I run WEAT to quantify the cosine distance between protected groups and neutral concepts; we don't ship until the differential effect size drops below our threshold (e.g., < 0.05)."
Most candidates immediately declare the lower-perplexity model superior. Perplexity is measured per token, and the denominator N depends entirely on the tokenizer's segmentation efficiency.
A verbose tokenizer splitting "Unbelievable" into 4 tokens divides the sequence log-probability by 4, artificially deflating per-token loss. A compact tokenizer treating it as 1 token divides by 1. Comparing raw perplexity across different tokenizer families is like comparing running speeds in miles versus kilometres without unit conversion. The correct measure is Bits Per Byte (BPB), which normalizes by byte count and is tokenizer-agnostic.
The answer that gets you hired: "Raw perplexity is not comparable across tokenizer families. I would normalize by Bits Per Byte — until we standardize the denominator, the 'lower' score might be measuring tokenizer verbosity, not model quality."
Most candidates approve the switch: DPO removes the separate reward model and value head, simplifying the training stack. The trap is the gradient saturation problem: DPO optimizes a probability ratio passed through a sigmoid.
When the model already strongly prefers the winning response, the sigmoid saturates and the gradient vanishes — DPO stops learning from pairs it has mastered. PPO, driven by a scalar reward from an external model, keeps squeezing improvements regardless of the preference gap. DPO requires hard negatives (ambiguous preference pairs close in quality) to maintain a meaningful learning signal.
The answer that gets you hired: "Approve the switch only if the dataset contains hard negatives. When the contrastive margin is too large, DPO gradients vanish where PPO would keep climbing. We trade architectural complexity for data complexity."
Most candidates celebrate the number. With a 99% majority "safe" class, two annotators can achieve 99% agreement by marking every comment as "safe" — they identify 0% of the actual toxicity while appearing perfect on paper. This is chance agreement masquerading as quality.
Cohen's Kappa corrects for this: κ = (P_o − P_e) / (1 − P_e). Two annotators who only mark "safe" produce κ = 0, instantly exposing the failure. Production datasets require κ ≥ 0.6 for borderline reliability; κ ≥ 0.8 for strong agreement.
The answer that gets you hired: "99% agreement on a 1% minority class is meaningless due to chance agreement. I would compute Cohen's Kappa — a Kappa below 0.6 reveals a guideline problem, not a high-quality dataset."
Data-layer traps — Semitic tokenization, prompt formatting sensitivity, distillation dark knowledge, and back-translation direction.
Most candidates retrain BPE on the new multilingual corpus. BPE assumes concatenative morphology (English: "play" + "-ing"), but Semitic languages use infix morphology — the root consonants mutate internally across conjugations.
BPE shatters the same semantic root (e.g., k-t-b in Arabic) into disjoint token IDs for different surface forms, forcing the model to relearn every conjugation as an unrelated concept. The solution is decompositional tokenization: a morphological analyzer separates the root from the template, and a character-level encoder builds embeddings dynamically so the shared root is visible across all forms.
The answer that gets you hired: "Standard BPE treats non-concatenative morphology like random noise. I would implement morphological decomposition to ensure the embedding space shares parameters for the semantic root across all conjugations."
Most candidates treat the top-1 prediction as the ground truth and apply standard cross-entropy. This discards dark knowledge: the ensemble's full probability distribution encodes the geometry of the decision boundary.
A distribution of "Cat (0.6), Dog (0.3), Truck (0.0001)" tells the student that "Dog" is a plausible near-miss while "Truck" is a category error — information erased by the hard label "Cat." The correct approach minimizes KL divergence between full teacher and student distributions using temperature-scaled logits, transferring not just the decision but the teacher's uncertainty structure.
The answer that gets you hired: "Hard targets discard the ensemble's uncertainty. I match full probability distributions via KL Divergence on temperature-scaled logits — we aren't copying the output, we are cloning the ensemble's intuition about which errors are plausible."
Most candidates attribute it to grammar confusion. In reality, LLMs process integer sequences, not text. Deleting a whitespace changes the tokenization boundary: in many tokenizers, " word" (with a leading space) and "word" (without) map to completely different token IDs, as distinct to the model as "apple" and "orange."
Furthermore, instruction-tuned models latch onto specific formatting tokens (colons, XML tags, newlines) as strong predictive features during SFT. Removing them pushes the input out-of-distribution relative to the fine-tuning corpus, breaking those learned shortcuts.
The answer that gets you hired: "It is a distribution shift caused by tokenization artifacts — the model learned specific formatting tokens as strong features during SFT. In production, I use Automated Prompt Optimization or enforce the exact chat template the model was fine-tuned on."
Most candidates generate synthetic English targets to give the model more good outputs to learn from. Training on synthetic targets teaches the decoder to reproduce machine-translation artifacts instead of real human language.
The correct direction: take the large monolingual English corpus, back-translate it into synthetic source-language text, then train on (Synthetic Source, Real English Target) pairs. The encoder learns to handle noisy, translationese inputs; the decoder always learns from gold-standard human text, preserving its fluency.
The answer that gets you hired: "We pair synthetic sources with natural targets. This ensures the decoder learns the true distribution of real human language rather than overfitting to artifacts from another model."
Architecture-level traps — MoE routing deadlocks, RoPE vs. learned positions, and loss function choices that kill production models.
Most candidates endorse Top-1 routing for maximum sparsity and throughput. The trap: selecting only the winning expert performs a hard argmax, which has a gradient of zero almost everywhere.
Non-selected experts receive zero signal, stagnate, and the gating network freezes because it never learns the cost of not routing to the runners-up. Training must use Top-2 routing to maintain gradient flow to both winner and runner-up. Top-1 can optionally be re-enabled at inference after the router has fully converged.
The answer that gets you hired: "Sparsity requires redundancy during learning. I would use Top-2 routing during training to maintain gradient flow to the gating network, even at the cost of 2x active parameters per forward pass."
Most candidates describe efficiency. The critical architectural property is zero-shot length extrapolation. Standard learned embeddings assign unique vectors to absolute positions up to the training maximum — position 4,097 is unseen if the model trained on 4,096 tokens (a hard wall).
RoPE encodes relative distances via rotation angles: the attention score between tokens m and n depends only on offset (m − n), not absolute indices. This enables post-training techniques like YaRN and NTK-aware scaling to extend the effective context to 32k+ without full retraining.
The answer that gets you hired: "Standard embeddings treat positions like unique zip codes — the model is lost beyond its training range. RoPE encodes relative distances so post-training context extension (YaRN, NTK scaling) can run a 4k-trained model on 32k documents."
Most candidates argue Hinge Loss is better because it stops updating on correct samples, reducing overfitting. The production failure is the vanishing gradient on correctness: once a prediction clears the margin, Hinge Loss assigns zero gradient regardless of whether the model is 51% or 99% confident.
Sigmoid + Cross-Entropy never settles; even at 0.9 probability the loss continues pushing toward 1.0, providing a calibrated probability output. Production classifiers require those calibrated probabilities to set operational precision/recall thresholds without retraining.
The answer that gets you hired: "Hinge Loss kills the gradient on correct examples, preventing the model from learning certainty. Production systems need calibrated probabilities — not raw margin scores — to tune thresholds, which only Sigmoid + Cross-Entropy provides."
Test yourself with random questions. Try answering before revealing — it's the fastest way to find gaps.