The Gauntlet: LLM

75 LLM interview questions — from foundational concepts to senior-level traps drawn from Google DeepMind, OpenAI, Meta, and Anthropic. Test yourself, reveal answers, and track your progress.

75Questions
16Topics
0Mastered
Progress0 / 75
00

LLM Foundations

Core building blocks — what LLMs are, how text becomes numbers, and how models manage vocabulary.

Q01 What defines a Large Language Model (LLM)? show ▾

LLMs are AI systems trained on vast text corpora to understand and generate human-like language. With billions of parameters, they excel in tasks like translation, summarization, and question answering, leveraging contextual learning for broad applicability.

What makes them "large" is the combination of massive model size (billions of weights), enormous training data (terabytes of text), and the emergent abilities that appear only at scale — like in-context learning and chain-of-thought reasoning.

Q02 What does tokenization entail, and why is it critical for LLMs? show ▾

Tokenization involves breaking down text into smaller units, or tokens, such as words, subwords, or characters. For example, "artificial" might be split into "art," "ific," and "ial."

This process is vital because LLMs process numerical representations of tokens, not raw text. Tokenization enables models to handle diverse languages, manage rare or unknown words, and optimize vocabulary size — enhancing computational efficiency and model performance.

Key insight: The choice of tokenizer (BPE, WordPiece, SentencePiece) directly affects model performance. A well-designed tokenizer balances vocabulary size against sequence length.

Q03 What are embeddings, and how are they initialized in LLMs? show ▾

Embeddings are dense vectors that represent tokens in a continuous space, capturing semantic and syntactic properties. They are often initialized randomly or with pretrained models like GloVe, then fine-tuned during training.

For example, the embedding for "dog" might evolve to reflect its context in pet-related tasks, enhancing model accuracy. The embedding dimension (e.g., 768 for BERT, 12288 for GPT-3) is a critical architectural choice that determines the model's representational capacity.

Q04 How do LLMs manage out-of-vocabulary (OOV) words? show ▾

LLMs use subword tokenization, like Byte-Pair Encoding (BPE), to break OOV words into known subword units. For instance, "cryptocurrency" might split into "crypto" and "currency."

This approach allows LLMs to process rare or new words, ensuring robust language understanding and generation. Other methods include WordPiece (used by BERT) and Unigram (used by SentencePiece), each with different merge strategies for building the subword vocabulary.

Q05 What is the context window in LLMs, and why does it matter? show ▾

The context window refers to the number of tokens an LLM can process at once, defining its "memory" for understanding or generating text. A larger window, like 32,000 tokens, allows the model to consider more context, improving coherence in tasks like summarization.

However, it increases computational costs quadratically with standard attention (O(n²)). Balancing window size with efficiency is crucial for practical LLM deployment. Modern approaches like sliding window attention, sparse attention, and RoPE scaling help extend context without full quadratic cost.

Q06 How do LLMs differ from traditional statistical language models? show ▾

LLMs use transformer architectures, massive datasets, and unsupervised pretraining, unlike statistical models (e.g., N-grams) that rely on simpler, supervised methods.

LLMs handle long-range dependencies, contextual embeddings, and diverse tasks, but require significant computational resources. Traditional N-gram models struggle beyond short contexts (typically 3-5 tokens), while transformers can attend to thousands of tokens and capture complex semantic relationships.

01

The Transformer Engine

Architecture fundamentals — attention, encoders, decoders, and what makes transformers superior to RNNs.

Q07 What are sequence-to-sequence models, and where are they applied? show ▾

Sequence-to-sequence (Seq2Seq) models transform an input sequence into an output sequence, often of different lengths. They consist of an encoder to process the input and a decoder to generate the output.

Applications include machine translation (e.g., English to Spanish), text summarization, and chatbots, where variable-length inputs and outputs are common. The original Seq2Seq used RNNs/LSTMs, but modern implementations use transformers for better parallelism and long-range dependency handling.

Q08 How do transformers improve on traditional Seq2Seq models? show ▾

Transformers overcome Seq2Seq limitations through three key innovations:

  • Parallel Processing: Self-attention enables simultaneous token processing, unlike sequential RNNs.
  • Long-Range Dependencies: Attention captures distant token relationships regardless of position.
  • Positional Encodings: These preserve sequence order without recurrence.

These features enhance scalability and performance in tasks like translation, and enable training on much larger datasets efficiently.

Q09 How does the attention mechanism function in transformer models? show ▾

The attention mechanism allows LLMs to weigh the importance of different tokens in a sequence when generating or interpreting text. It computes similarity scores between query, key, and value vectors, using operations like dot products, to focus on relevant tokens.

For instance, in "The cat chased the mouse," attention helps the model link "mouse" to "chased." This mechanism improves context understanding, making transformers highly effective for NLP tasks. Each token creates a Q, K, V vector, and attention scores determine how much each token "attends to" every other token.

Q10 What are positional encodings, and why are they used? show ▾

Positional encodings add sequence order information to transformer inputs, as self-attention lacks inherent order awareness. Using sinusoidal functions or learned vectors, they ensure tokens like "king" and "crown" are interpreted correctly based on position.

The original transformer used fixed sinusoidal encodings. Modern models often use Rotary Position Embeddings (RoPE) which encode relative positions through rotation matrices, or ALiBi which adds linear biases to attention scores. These are critical for tasks like translation where word order changes meaning.

Q11 What is multi-head attention, and how does it enhance LLMs? show ▾

Multi-head attention splits queries, keys, and values into multiple subspaces, allowing the model to focus on different aspects of the input simultaneously.

For example, in a sentence, one head might focus on syntax, another on semantics, and another on coreference. This improves the model's ability to capture complex patterns. Modern variants include Grouped Query Attention (GQA) — used in LLaMA 2+ — which shares key-value heads across query heads, and Multi-Query Attention (MQA) which uses a single KV head for all queries, trading some quality for significantly faster inference.

Q12 How do encoders and decoders differ in transformers? show ▾

Encoders process input sequences into abstract representations, capturing context through bidirectional self-attention. Decoders generate outputs token-by-token, using encoder outputs and prior tokens through causal (masked) self-attention.

In translation, the encoder understands the source language, and the decoder produces the target language. Encoder-only models (BERT) excel at understanding tasks, decoder-only models (GPT) excel at generation, and encoder-decoder models (T5) handle sequence-to-sequence tasks.

02

Attention Mathematics

The math under the hood — softmax, scaled dot products, and the full attention formula.

Q13 How is the softmax function applied in attention mechanisms? show ▾

The softmax function normalizes attention scores into a probability distribution:

softmax(x_i) = e^(x_i) / Σ_j e^(x_j)

In attention, it converts raw similarity scores (from query-key dot products) into weights, emphasizing relevant tokens. This ensures the model focuses on contextually important parts of the input. The exponential function amplifies differences — high-scoring tokens get disproportionately more weight, creating a "soft" selection mechanism.

Q14 How does the dot product contribute to self-attention? show ▾

In self-attention, the dot product between query (Q) and key (K) vectors computes similarity scores:

Score = Q · K / √d_k

High scores indicate relevant tokens. The scaling factor √d_k (square root of key dimension) prevents dot products from growing too large, which would push softmax into regions with extremely small gradients.

While efficient, its quadratic complexity O(n²) for long sequences has spurred research into sparse attention alternatives like Flash Attention, which computes exact attention with reduced memory through tiling.

Q15 How are attention scores calculated in transformers? show ▾

The complete attention formula brings Q, K, V together:

Attention(Q, K, V) = softmax(Q·K^T / √d_k) · V

The scaled dot product measures token relevance, and softmax normalizes scores to focus on key tokens. The result is multiplied by the value matrix V to produce the weighted combination of value vectors — the actual output of the attention layer.

This single formula is the core of every transformer. Everything else (multi-head, causal masking, KV cache) is built around it.

03

Training & Pretraining

How models learn — masked vs. autoregressive training, next-sentence prediction, and the generative/discriminative divide.

Q16 What is masked language modeling, and how does it aid pretraining? show ▾

Masked language modeling (MLM) involves hiding random tokens in a sequence and training the model to predict them based on context. Used in models like BERT, MLM fosters bidirectional understanding of language.

Typically 15% of tokens are masked. Of those, 80% are replaced with [MASK], 10% with a random token, and 10% kept unchanged. This strategy prevents the model from simply learning to detect [MASK] tokens and encourages robust contextual representations useful for downstream tasks like sentiment analysis or question answering.

Q17 How do autoregressive and masked models differ in LLM training? show ▾

Autoregressive models (like GPT) predict tokens sequentially based on prior tokens, excelling in generative tasks such as text completion. They see only left context (causal attention).

Masked models (like BERT) predict masked tokens using bidirectional context, making them ideal for understanding tasks like classification. They see the full sequence minus the masked positions.

Their training objectives shape their strengths: autoregressive models are natural generators, while masked models produce better representations for downstream classification and extraction tasks.

Q18 What is next sentence prediction, and how does it enhance LLMs? show ▾

Next sentence prediction (NSP) trains models to determine if two sentences are consecutive or unrelated. During pretraining, models like BERT learn to classify 50% positive (sequential) and 50% negative (random) sentence pairs.

NSP improves coherence in tasks like dialogue systems or document summarization by understanding sentence relationships. However, later research (RoBERTa) showed that NSP provides minimal benefit and can even hurt performance — leading most modern models to drop it in favor of longer contiguous sequences.

Q19 What are generative versus discriminative models in NLP? show ▾

Generative models (like GPT) model joint probabilities P(x, y) to create new data, such as text or images. Discriminative models (like BERT for classification) model conditional probabilities P(y|x) to distinguish classes.

Generative models excel in creation tasks (writing, code generation), while discriminative models focus on accurate classification (sentiment analysis, NER). In practice, the line has blurred — modern LLMs can do both by framing classification as generation.

Q20 How do discriminative and generative AI differ in practice? show ▾

Discriminative AI, like sentiment classifiers, predicts labels based on input features, modeling conditional probabilities. Generative AI, like GPT, creates new data by modeling joint probabilities, suitable for tasks like text or image generation.

Discriminative models tend to be more sample-efficient for specific classification tasks, while generative models offer creative flexibility and can be adapted to many tasks through prompting. The generative approach has become dominant because one large pretrained model can handle diverse tasks.

04

Text Generation & Decoding

Turning probabilities into words — beam search, temperature, and sampling strategies.

Q21 How does beam search improve text generation compared to greedy decoding? show ▾

Beam search explores multiple word sequences during text generation, keeping the top k candidates (beams) at each step, unlike greedy decoding, which selects only the most probable word.

This approach, with k = 5 for instance, ensures more coherent outputs by balancing probability and diversity. It's especially effective in tasks like machine translation or dialogue generation where the locally best token isn't always globally optimal.

Trade-off: Higher beam width improves quality but increases computation linearly. In practice, beam widths of 4-10 are common, and some tasks benefit more from sampling-based methods instead.

Q22 What role does temperature play in controlling LLM output? show ▾

Temperature is a hyperparameter that adjusts the randomness of token selection in text generation by scaling the logits before softmax.

  • Low temperature (e.g., 0.3): Favors high-probability tokens, producing predictable, deterministic outputs.
  • High temperature (e.g., 1.5): Increases diversity by flattening the probability distribution, allowing more unlikely tokens.
  • Temperature = 1.0: Uses the raw model probabilities unchanged.
  • Temperature → 0: Approaches greedy decoding (always picks the top token).

Setting temperature to ~0.7-0.8 often balances creativity and coherence for tasks like storytelling, while ~0.2 works well for factual Q&A.

Q23 How do top-k and top-p sampling differ in text generation? show ▾

Top-k sampling selects the k most probable tokens (e.g., k = 20) for random sampling, ensuring controlled diversity. The number of candidates is fixed regardless of the probability distribution.

Top-p (nucleus) sampling chooses tokens whose cumulative probability exceeds a threshold p (e.g., 0.95), adapting to context. When the model is confident, fewer tokens are considered; when uncertain, more are included.

Top-p offers more flexibility, producing varied yet coherent outputs. In practice, many systems combine both: apply top-k first to cap the maximum candidates, then top-p to further narrow based on probability mass.

05

Fine-Tuning & Efficiency

Adapting pretrained models — LoRA, PEFT, distillation, and preventing catastrophic forgetting.

Q24 What distinguishes LoRA from QLoRA in fine-tuning LLMs? show ▾

LoRA (Low-Rank Adaptation) adds low-rank matrices (A and B, where ΔW = BA) to a model's layers, enabling efficient adaptation with minimal memory overhead. Only A and B are trained — the original weights stay frozen.

QLoRA extends this by applying quantization (e.g., 4-bit NormalFloat precision) to the frozen base weights, further reducing memory usage while maintaining accuracy. For example, QLoRA can fine-tune a 70B-parameter model on a single GPU, making it ideal for resource-constrained environments.

Key difference: LoRA reduces trainable parameters; QLoRA additionally reduces memory for stored parameters through quantization. Both keep the base model frozen.

Q25 How can LLMs avoid catastrophic forgetting during fine-tuning? show ▾

Catastrophic forgetting occurs when fine-tuning erases prior knowledge. Mitigation strategies include:

  • Rehearsal: Mixing old and new data during training so the model revisits previous knowledge.
  • Elastic Weight Consolidation (EWC): Prioritizing critical weights by adding a penalty for changing parameters that were important for previous tasks.
  • Modular Architectures: Adding task-specific modules (adapters, LoRA) to avoid overwriting the base model.

These methods ensure LLMs retain versatility across tasks while still learning new capabilities.

Q26 How does PEFT mitigate catastrophic forgetting? show ▾

Parameter-Efficient Fine-Tuning (PEFT) updates only a small subset of parameters, freezing the rest to preserve pretrained knowledge. Techniques include:

  • LoRA: Low-rank matrix decomposition of weight updates.
  • Prefix Tuning: Learning a small set of "virtual tokens" prepended to the input.
  • Adapters: Small bottleneck layers inserted between transformer blocks.

By modifying less than 1% of total parameters, PEFT ensures LLMs adapt to new tasks without losing core capabilities, maintaining performance across domains.

Q27 What is model distillation, and how does it benefit LLMs? show ▾

Model distillation trains a smaller "student" model to mimic a larger "teacher" model's outputs, using soft probabilities (logits) rather than hard labels. The soft targets carry richer information about the teacher's learned distribution.

This reduces memory and computational requirements, enabling deployment on devices like smartphones while retaining near-teacher performance. Examples include DistilBERT (40% smaller, 97% of BERT's performance) and various distilled versions of LLaMA. It's ideal for real-time applications where latency matters.

Q28 How does Adaptive Softmax optimize LLMs? show ▾

Adaptive Softmax groups words by frequency, reducing computations for rare words. Common words get full-dimension representations while rare words use smaller clusters with lower-dimensional projections.

This lowers the cost of handling large vocabularies (100k+ tokens), speeding up training and inference while maintaining accuracy. It's especially valuable in resource-limited settings where computing softmax over the full vocabulary at every step is prohibitively expensive.

06

Prompting & Learning Paradigms

Getting the best out of LLMs without retraining — prompt engineering, zero-shot, few-shot, and chain-of-thought.

Q29 Why is prompt engineering crucial for LLM performance? show ▾

Prompt engineering involves designing inputs to elicit desired LLM responses. A clear prompt, like "Summarize this article in 100 words," improves output relevance compared to vague instructions.

It's especially effective in zero-shot or few-shot settings, enabling LLMs to tackle tasks like translation or classification without extensive fine-tuning. Key techniques include specifying output format, providing role context ("You are an expert in..."), using delimiters for structure, and iteratively refining based on model responses.

Q30 What is Chain-of-Thought (CoT) prompting, and how does it aid reasoning? show ▾

CoT prompting guides LLMs to solve problems step-by-step, mimicking human reasoning. For example, in math problems, it breaks down calculations into logical steps rather than jumping to the answer.

This improves accuracy and interpretability in complex tasks like logical inference, multi-step queries, and arithmetic. Variants include zero-shot CoT (just adding "Let's think step by step"), few-shot CoT (providing examples with reasoning traces), and self-consistency (sampling multiple reasoning paths and taking the majority vote).

Q31 What is zero-shot learning, and how do LLMs implement it? show ▾

Zero-shot learning allows LLMs to perform untrained tasks using general knowledge from pretraining. For example, prompted with "Classify this review as positive or negative," an LLM can infer sentiment without task-specific data.

This showcases the model's versatility and is possible because of the broad knowledge encoded during pretraining on diverse text. Larger models tend to have stronger zero-shot capabilities — a key finding from the GPT-3 paper that demonstrated scaling enables emergent abilities.

Q32 What is few-shot learning, and what are its benefits? show ▾

Few-shot learning enables LLMs to perform tasks with minimal examples (typically 1-10), leveraging pretrained knowledge. The examples are provided in the prompt as demonstrations.

Benefits include reduced data needs, faster adaptation, and cost efficiency, making it ideal for niche tasks like specialized text classification. Unlike fine-tuning, few-shot learning requires no gradient updates — the model uses in-context learning to generalize from the provided examples at inference time.

07

Advanced Architectures & Techniques

Cutting-edge approaches — GPT-4, Gemini, MoE, RAG, knowledge graphs, and foundation model taxonomy.

Q33 How does GPT-4 differ from GPT-3 in features and applications? show ▾

GPT-4 surpasses GPT-3 with several key advances:

  • Multimodal Input: Processes both text and images (GPT-3 was text-only).
  • Larger Context: Handles up to 128K tokens versus GPT-3's 4,096.
  • Enhanced Accuracy: Reduces factual errors through better fine-tuning and RLHF alignment.
  • Improved Reasoning: Significantly better at complex multi-step tasks.

These improvements expand its use in visual question answering, complex dialogues, code generation, and professional applications like legal analysis and medical reasoning.

Q34 How does Gemini optimize multimodal LLM training? show ▾

Gemini enhances efficiency through several innovations:

  • Unified Architecture: Combines text and image processing natively for parameter efficiency (not a bolted-on vision encoder).
  • Advanced Attention: Improves cross-modal learning stability across modalities.
  • Data Efficiency: Uses self-supervised techniques to reduce labeled data needs.

These features make Gemini more stable and scalable than models that retrofit multimodality onto text-only architectures.

Q35 What types of foundation models exist? show ▾

Foundation models span multiple modalities:

  • Language Models: BERT, GPT-4 for text understanding and generation.
  • Vision Models: ResNet, ViT for image classification and understanding.
  • Generative Models: DALL-E, Stable Diffusion for content creation from text prompts.
  • Multimodal Models: CLIP for text-image alignment, LLaVA for visual question answering.
  • Code Models: Codex, StarCoder for programming tasks.

These models leverage broad pretraining for diverse applications and can be fine-tuned for specific downstream tasks.

Q36 How does Mixture of Experts (MoE) enhance LLM scalability? show ▾

MoE uses a gating function (router) to activate specific expert sub-networks per input, reducing computational load. For example, only 10% of a model's parameters might be used per query.

This enables billion-parameter models to operate efficiently while maintaining high performance. Mixtral 8x7B, for instance, has 47B total parameters but only activates ~13B per token. DeepSeek V3 pushed this further with 671B total parameters and fine-grained expert selection. The key challenge is load balancing — ensuring all experts get trained evenly.

Q37 What are the steps in Retrieval-Augmented Generation (RAG)? show ▾

RAG combines retrieval with generation in three steps:

  1. Retrieval: Fetching relevant documents using query embeddings and vector similarity search (e.g., from a vector database like Pinecone, Chroma, or FAISS).
  2. Ranking: Sorting retrieved documents by relevance, often with a cross-encoder reranker for higher precision.
  3. Generation: Using retrieved context as additional input to generate accurate, grounded responses.

RAG enhances factual accuracy, reduces hallucinations, and allows models to access up-to-date information without retraining. It's the dominant pattern for enterprise LLM applications.

Q38 How does knowledge graph integration improve LLMs? show ▾

Knowledge graphs provide structured, factual data that enhances LLMs by:

  • Reducing Hallucinations: Verifying generated facts against the graph's ground truth.
  • Improving Reasoning: Leveraging entity relationships for multi-hop inference.
  • Enhancing Context: Offering structured context (subject-predicate-object triples) for better, more precise responses.

This is particularly valuable for question answering, entity recognition, and domain-specific applications (medical, legal) where factual accuracy is non-negotiable.

08

Mathematical Foundations

The math that makes it all work — loss functions, gradients, eigenvalues, KL divergence, and backpropagation through transformers.

Q39 Why is cross-entropy loss used in language modeling? show ▾

Cross-entropy loss measures the divergence between predicted and true token probabilities:

L = -Σ y_i · log(ŷ_i)

It penalizes incorrect predictions, encouraging accurate token selection. In language modeling, it ensures the model assigns high probabilities to correct next tokens. The logarithm means that assigning very low probability to the correct token is heavily penalized, while getting close to 1.0 gives diminishing returns — exactly the behavior we want for training.

Q40 How are gradients computed for embeddings in LLMs? show ▾

Gradients for embeddings are computed using the chain rule during backpropagation:

∂L/∂E = (∂L/∂logits) · (∂logits/∂E)

These gradients adjust embedding vectors to minimize loss, refining their semantic representations for better task performance. Only the embedding rows corresponding to tokens in the current batch receive non-zero gradients — making embedding updates sparse and efficient.

Q41 What is the Jacobian matrix's role in transformer backpropagation? show ▾

The Jacobian matrix captures partial derivatives of outputs with respect to inputs. For a function f: R^n → R^m, the Jacobian is an m × n matrix where entry (i,j) is ∂f_i/∂x_j.

In transformers, it helps compute gradients for multidimensional outputs (like the attention scores matrix), ensuring accurate updates to weights and embeddings during backpropagation. This is critical for optimizing complex models where outputs are vector-valued rather than scalar.

Q42 How do eigenvalues and eigenvectors relate to dimensionality reduction? show ▾

Eigenvectors define principal directions in data, and eigenvalues indicate their variance (importance). In techniques like PCA, selecting eigenvectors with high eigenvalues reduces dimensionality while retaining most variance.

This enables efficient data representation for LLM input processing, visualization of embedding spaces, and compression of hidden states. PCA on embedding matrices is commonly used to understand what semantic dimensions the model has learned.

Q43 What is KL divergence, and how is it used in LLMs? show ▾

KL divergence quantifies the difference between two probability distributions:

D_KL(P || Q) = Σ P(x) · log(P(x) / Q(x))

In LLMs, it evaluates how closely model predictions match true distributions, guiding fine-tuning to improve output quality. It's central to RLHF — the KL penalty term prevents the policy model from diverging too far from the reference model, maintaining generation quality while aligning with human preferences.

Important: KL divergence is asymmetric — D_KL(P||Q) ≠ D_KL(Q||P). The choice of direction matters: forward KL (mode-covering) vs reverse KL (mode-seeking) produce different behaviors in training.

Q44 What is the derivative of the ReLU function, and why is it significant? show ▾

The ReLU function, f(x) = max(0, x), has a simple derivative:

f'(x) = 1 if x > 0, else 0

Its sparsity (zeroing out negative values) and non-linearity prevent vanishing gradients, making ReLU computationally efficient and widely used in neural networks. However, the "dying ReLU" problem (neurons stuck at 0) led to variants like Leaky ReLU, GELU (used in GPT), and SwiGLU (used in LLaMA/modern LLMs).

Q45 How does the chain rule apply to gradient descent in LLMs? show ▾

The chain rule computes derivatives of composite functions:

d/dx f(g(x)) = f'(g(x)) · g'(x)

In gradient descent, it enables backpropagation to calculate gradients layer by layer, updating parameters to minimize loss efficiently across deep LLM architectures. Without the chain rule, computing gradients through dozens of transformer layers (each a composition of attention, layer norm, and FFN) would be intractable.

Q46 How do transformers address the vanishing gradient problem? show ▾

Transformers mitigate vanishing gradients through three mechanisms:

  • Self-Attention: Direct connections between any two positions avoid the sequential degradation of RNNs.
  • Residual Connections: x + f(x) skip connections allow direct gradient flow through the network, providing a "gradient highway."
  • Layer Normalization: Stabilizes activations and gradients at each layer, preventing them from exploding or vanishing.

Together, these ensure effective training of very deep models (100+ layers), unlike RNNs which struggle with sequences longer than a few hundred tokens.

09

Practical & Deployment

Real-world challenges — overfitting, bias, hyperparameters, and the hurdles of putting LLMs into production.

Q47 What is overfitting, and how can it be mitigated in LLMs? show ▾

Overfitting occurs when a model memorizes training data, failing to generalize to unseen inputs. Mitigation includes:

  • Regularization: L1/L2 penalties (weight decay) constrain model complexity.
  • Dropout: Randomly disables neurons during training, forcing redundant representations.
  • Early Stopping: Halts training when validation performance plateaus.
  • Data Augmentation: Increases effective training set size through transformations.

For LLMs specifically, using diverse training data and appropriate model size relative to dataset size are the most important factors.

Q48 How would you fix an LLM generating biased or incorrect outputs? show ▾

A systematic approach to addressing biased or incorrect outputs:

  1. Analyze Patterns: Identify bias sources in training data, prompts, or model architecture. Use red-teaming and automated evaluation.
  2. Enhance Data: Use balanced, diverse datasets and apply debiasing techniques like counterfactual data augmentation.
  3. Fine-Tune: Retrain with curated data, apply RLHF with diverse annotators, or use adversarial training methods.
  4. Post-Processing: Add safety filters, constitutional AI constraints, or output validation layers.

These steps improve fairness and accuracy while maintaining model utility.

Q49 What is a hyperparameter, and why is it important? show ▾

Hyperparameters are preset values that control model training — they are not learned from data but set before training begins. Examples include learning rate, batch size, number of layers, and attention heads.

They critically influence convergence and performance. A high learning rate may cause instability and divergence, while too low a rate leads to slow training. Modern LLM training uses sophisticated schedules: warmup + cosine decay for learning rate, gradient accumulation for effective batch size, and careful attention to the interplay between all hyperparameters.

Q50 What challenges do LLMs face in deployment? show ▾

Key deployment challenges include:

  • Resource Intensity: High computational demands for inference (GPU memory, latency). Solutions include quantization, KV cache optimization, and speculative decoding.
  • Bias: Risk of perpetuating training data biases in outputs. Requires ongoing monitoring and mitigation.
  • Interpretability: Complex models are hard to explain to stakeholders and regulators.
  • Privacy: Potential data security concerns, including memorization of training data and prompt injection attacks.
  • Hallucination: Models can generate plausible-sounding but factually incorrect content.

Addressing these through careful system design, guardrails, and monitoring ensures ethical and effective LLM use in production.

10

Training Traps

Senior-level traps from top AI labs — learning rate warmup, gradient clipping, optimizer memory, and curriculum ordering.

Q51 Why does a constant learning rate of 1e-3 cause Transformer training to diverge? show ▾

Most candidates trust Adam's adaptive rates to handle initialization automatically. At Step 0, however, Transformer layers are completely misaligned and gradient variance is effectively infinite.

Applying a full-magnitude update immediately pushes parameters into catastrophically steep curvature — the loss diverges within 50 steps. The fix is a linear warm-up schedule: start the learning rate at 0, increase linearly over the first ~4,000 steps until gradient variance stabilizes, then switch to an inverse-square-root decay.

The answer that gets you hired: "Transformers don't tolerate high learning rates at initialization due to high gradient variance. I would implement a linear warm-up from 0 to d_model^(-0.5) to let the curvature stabilize, preventing early divergence before switching to the decay schedule."

Q52 How do you fix exploding gradients in large Transformer training? show ▾

Most candidates drop the learning rate drastically. This slows convergence to a crawl — the gradient direction is correct; only the magnitude is too large for the current loss curvature.

The targeted fix is Global L2 Gradient Clipping: if ||g||₂ > 1.0, rescale g ← g / ||g||₂. This caps the maximum step size while strictly preserving the descent direction — unlike a blanket learning-rate cut, which wastes GPU-hours across the entire run.

The answer that gets you hired: "I don't touch the learning rate. I apply Global Gradient Clipping (norm = 1.0), which decouples step size from gradient magnitude and prevents divergence without stalling convergence."

Q53 What batch size should you target when training a CLIP-style contrastive model? show ▾

Most candidates optimize for VRAM and set a comfortable per-GPU batch. In supervised learning, batch size only affects gradient estimation. In contrastive learning, batch size IS the training signal: N samples yield N² − N negatives per step.

At N = 512, the model distinguishes each positive from only 511 distractors — trivially easy, so it learns shortcut features. The original CLIP paper required massive batches for quality, not speed. Target a global batch of 4,096+, using all_gather across GPUs and gradient caching to simulate larger matrices if memory is limited.

The answer that gets you hired: "In contrastive learning, batch size is a quality hyperparameter, not just a hardware choice. If we can't reach 4k+ samples per step, we shouldn't start training."

Q54 Why does switching from SGD to Adam crash a 7B training run with an OOM error? show ▾

Most candidates guess computational cost or batch size. The real cause is the 3x memory rule: unlike stateless SGD, Adam stores two additional scalar states per parameter — first moment (momentum) and second moment (variance).

A 7B model in FP16 occupies ~14 GB for weights; Adam's optimizer states add ~28 GB more, tripling the total VRAM footprint without changing a single parameter count. The fix is PagedAdam, which offloads optimizer states to CPU memory and fetches them only at each update step.

The answer that gets you hired: "Adam imposes a 3x memory penalty over model weights alone due to stored momentum and variance states. I would use PagedAdam to offload those states to CPU, or plan the cluster allocation to account for the 3x overhead upfront."

Q55 Why does SGD fail on rare tokens, and how does Adam fix it? show ▾

Most candidates say "Adam uses momentum for faster convergence." The real mechanism is frequency bias: SGD updates are proportional to gradient magnitude, so tokens appearing once every 10,000 documents receive tiny, infrequent updates and stay near random initialization.

Adam tracks the running variance v_t of each parameter's gradient; for a rare feature, v_t stays near zero. Adam's update divides by √v_t, automatically amplifying the effective learning rate for underrepresented parameters — like an equalizer boosting quiet frequencies.

The answer that gets you hired: "SGD fails because updates are proportional to feature frequency. Adam's variance normalization — dividing by √v_t — amplifies updates for rare features, ensuring they are learned despite their sparsity."

Q56 In what order should you train on General Text, Code, and Math data? show ▾

Most candidates shuffle all three together to avoid catastrophic forgetting. DeepSeek's experiments show this caps the ceiling for complex reasoning.

The optimal curriculum is General Text → Code → Math. Code is not trained for programming proficiency — it teaches the model the grammar of logic: strict sequential dependency tracking that is a prerequisite for mathematical deduction. Skipping Code and jumping straight to Math produces a model that has memorized formulas but lacks the internal execution stack to apply them to novel multi-step problems.

The answer that gets you hired: "Code is a reasoning primitive, not a domain task. General → Code → Math is optimal because code training aligns the model's internal representations for step-by-step chain-of-thought before we demand it on mathematics."

11

Fine-Tuning Traps

The pitfalls that trip up 90% of candidates — backbone unfreezing, LoRA initialization, QLoRA compute tax, and quantization-aware training.

Q57 Should you immediately unfreeze all backbone layers when fine-tuning with a new head? show ▾

Most candidates unfreeze everything immediately, arguing it converges faster. The danger is the Gradient Shockwave: the randomly initialized head produces mathematically incoherent gradients that shatter pre-trained backbone features before the head even knows which direction is "up."

The correct two-phase protocol: (1) Freeze the backbone and train only the head until it is sensible (effectively a convex problem, fast convergence), then (2) unfreeze the backbone and fine-tune the full model with a much lower learning rate (e.g., 1e-5).

The answer that gets you hired: "Never fine-tune a backbone against a random head. We must align the head to the existing feature space before we allow the feature space to shift."

Q58 How should you initialize the A and B matrices in LoRA? show ▾

Most candidates apply Xavier or Kaiming initialization to both matrices. LoRA inserts a residual update W_new = W_frozen + B × A: if both are random, their product is random noise injected directly into the pre-trained weights at Step 0, destroying existing knowledge before training begins.

The correct asymmetric initialization: A sampled from a Gaussian (breaks symmetry, enables gradient flow), B = 0 (all zeros). This makes B × A = 0, preserving the pre-trained model as a mathematical identity until B slowly moves away from zero during training.

The answer that gets you hired: "We initialize the up-projection to zero and the down-projection to Gaussian random. This makes the adapter an identity function at Step 0, preserving pre-trained weights while keeping gradient flow active."

Q59 Why doesn't switching to QLoRA (4-bit) give a 4x training speedup? show ▾

Most candidates equate smaller storage with faster computation. QLoRA is a memory optimization, not a compute optimization. Weights are stored in VRAM as INT4, but NVIDIA Tensor Cores do not execute backpropagation in INT4.

Before every matrix multiplication, the kernel must dequantize those 4-bit weights back to BF16 — an "unzipping tax" added to every forward and backward pass. If the training run was not previously memory-bandwidth-bound, this extra step is pure overhead. QLoRA trades wall-clock time for memory capacity: the ability to fit a model that would otherwise not fit at all.

The answer that gets you hired: "QLoRA adds on-the-fly dequantization overhead before every matrix multiply. We accept slightly slower wall-clock training in exchange for fitting a much larger model — it is a capacity trade-off, not a speed optimization."

Q60 How do you backpropagate through the rounding operation in Quantization-Aware Training? show ▾

Most candidates stop at "rounding is non-differentiable, so gradients vanish." The production fix is the Straight-Through Estimator (STE): decouple the two passes.

Forward pass: Apply true quantization (rounding) so the model experiences real discrete-precision errors. Backward pass: Approximate the rounding derivative as the identity function (∂round/∂x ≈ 1), passing gradients unchanged through the rounding node to the full-precision weights. A gradient mismatch is accepted because a noisy gradient is infinitely better than a zero gradient.

The answer that gets you hired: "We decouple the passes. The forward pass uses quantized weights to simulate inference precision; the backward pass applies the Straight-Through Estimator, treating rounding as an identity function to keep gradients flowing."

12

Inference & Production Traps

Where things break in production — speculative decoding, dropout at inference, and model cascade bottlenecks.

Q61 When does speculative decoding actually help at large batch sizes? show ▾

Most candidates argue that speculative decoding adds overhead when the GPU is already compute-saturated at large batches. That logic holds for short contexts but breaks for long-context inference (100k+ tokens).

Each request carries a unique KV cache; moving 128 massive per-request caches from VRAM to compute units every decode step makes memory bandwidth the bottleneck, not FLOPs. Self-speculative KV compression exploits this: speculate the next token using only the top 1% of the KV cache (cheap to load), then verify with the full cache — near-zero extra cost because the hardware was already I/O bound.

The answer that gets you hired: "For long-context inference, high batch sizes remain memory-bound due to unique per-request KV caches. Self-speculation bypasses the I/O bottleneck by trading cheap compute for expensive memory bandwidth."

Q62 How do you correctly handle Dropout during inference? show ▾

Most candidates say "just turn off the random masking." This causes magnitude mismatch: with p = 0.5 dropout during training, the next layer learned to expect signals from only half its inputs. Suddenly activating all neurons doubles the incoming signal, saturating activations.

The production fix is inverted dropout: scale surviving activations by 1/(1 − p) during training so that the expected magnitude matches the full network. At inference, simply remove the mask — no post-hoc weight scaling needed because the training-time scaling already calibrated the magnitudes.

The answer that gets you hired: "I use inverted dropout: activations are scaled by 1/(1 − p) during training so the expected magnitude is already calibrated. At inference I simply remove the mask — no weight adjustment needed."

Q63 What is the core bottleneck in building a reliable model cascade? show ▾

Most candidates cite routing latency or query difficulty classifiers. The fundamental bottleneck is confidence calibration: LLMs are notoriously overconfident — a log-probability of 0.99 often says nothing about factual accuracy.

Without a calibrated confidence signal or a lightweight verifier, the cheap model cannot reliably identify its own failures. The cascade then faces two failure modes: (1) confident hallucinations pass through unchecked, or (2) the system routes everything to the expensive model anyway, eliminating all cost savings.

The answer that gets you hired: "The bottleneck is failure detection. Without a calibrated confidence score or lightweight verifier, we can't reliably distinguish easy from hard — we either pass hallucinations through or route everything to the expensive model."

13

Alignment & Evaluation Traps

Traps in measuring model quality — attention entropy illusions, perplexity comparison, PPO vs. DPO, bias detection, and annotator agreement.

Q64 Can attention weight entropy reliably measure model uncertainty? show ▾

Most candidates reason that peaked attention signals confidence. This is incorrect: inference-time attention is fully deterministic. There is no sampling, so entropy of the weights measures information dispersal, not probabilistic confidence — a model can attend sharply to one token while hallucinating.

Real uncertainty requires stochasticity: Monte Carlo Dropout (keep dropout active at inference, run N forward passes, measure output variance) or a Variational Autoencoder with an explicit sampled latent variable.

The answer that gets you hired: "Attention weights are deterministic mixing coefficients, not probabilities. To measure uncertainty I would use MC Dropout — running multiple stochastic forward passes and measuring the variance."

Q65 How do you prove word embeddings are free of gender bias? show ▾

Most candidates check King − Man + Woman = Queen, which only verifies semantic correctness. The WEAT (Word Embedding Association Test) exposes stereotypical projections: it measures the cosine-similarity gap between sets of neutral concepts (e.g., {Math, Physics}) and gendered attribute words ({he, him} vs. {she, her}).

A low p-value on the effect-size test reveals that the embedding space is projecting gender onto neutral domains, regardless of whether surface analogies pass.

The answer that gets you hired: "Analogies only test semantic correctness, not allocational harm. To prove safety I run WEAT to quantify the cosine distance between protected groups and neutral concepts; we don't ship until the differential effect size drops below our threshold (e.g., < 0.05)."

Q66 Can you directly compare perplexity scores across different model families? show ▾

Most candidates immediately declare the lower-perplexity model superior. Perplexity is measured per token, and the denominator N depends entirely on the tokenizer's segmentation efficiency.

A verbose tokenizer splitting "Unbelievable" into 4 tokens divides the sequence log-probability by 4, artificially deflating per-token loss. A compact tokenizer treating it as 1 token divides by 1. Comparing raw perplexity across different tokenizer families is like comparing running speeds in miles versus kilometres without unit conversion. The correct measure is Bits Per Byte (BPB), which normalizes by byte count and is tokenizer-agnostic.

The answer that gets you hired: "Raw perplexity is not comparable across tokenizer families. I would normalize by Bits Per Byte — until we standardize the denominator, the 'lower' score might be measuring tokenizer verbosity, not model quality."

Q67 Is replacing PPO with DPO a free engineering win in RLHF? show ▾

Most candidates approve the switch: DPO removes the separate reward model and value head, simplifying the training stack. The trap is the gradient saturation problem: DPO optimizes a probability ratio passed through a sigmoid.

When the model already strongly prefers the winning response, the sigmoid saturates and the gradient vanishes — DPO stops learning from pairs it has mastered. PPO, driven by a scalar reward from an external model, keeps squeezing improvements regardless of the preference gap. DPO requires hard negatives (ambiguous preference pairs close in quality) to maintain a meaningful learning signal.

The answer that gets you hired: "Approve the switch only if the dataset contains hard negatives. When the contrastive margin is too large, DPO gradients vanish where PPO would keep climbing. We trade architectural complexity for data complexity."

Q68 Is 99% inter-annotator agreement on a 1% toxic dataset a sign of quality? show ▾

Most candidates celebrate the number. With a 99% majority "safe" class, two annotators can achieve 99% agreement by marking every comment as "safe" — they identify 0% of the actual toxicity while appearing perfect on paper. This is chance agreement masquerading as quality.

Cohen's Kappa corrects for this: κ = (P_o − P_e) / (1 − P_e). Two annotators who only mark "safe" produce κ = 0, instantly exposing the failure. Production datasets require κ ≥ 0.6 for borderline reliability; κ ≥ 0.8 for strong agreement.

The answer that gets you hired: "99% agreement on a 1% minority class is meaningless due to chance agreement. I would compute Cohen's Kappa — a Kappa below 0.6 reveals a guideline problem, not a high-quality dataset."

14

Tokenization & Data Traps

Data-layer traps — Semitic tokenization, prompt formatting sensitivity, distillation dark knowledge, and back-translation direction.

Q69 How do you adapt a BPE tokenizer to Semitic languages like Arabic? show ▾

Most candidates retrain BPE on the new multilingual corpus. BPE assumes concatenative morphology (English: "play" + "-ing"), but Semitic languages use infix morphology — the root consonants mutate internally across conjugations.

BPE shatters the same semantic root (e.g., k-t-b in Arabic) into disjoint token IDs for different surface forms, forcing the model to relearn every conjugation as an unrelated concept. The solution is decompositional tokenization: a morphological analyzer separates the root from the template, and a character-level encoder builds embeddings dynamically so the shared root is visible across all forms.

The answer that gets you hired: "Standard BPE treats non-concatenative morphology like random noise. I would implement morphological decomposition to ensure the embedding space shares parameters for the semantic root across all conjugations."

Q70 Why is training a distillation student on the ensemble's hard labels wasteful? show ▾

Most candidates treat the top-1 prediction as the ground truth and apply standard cross-entropy. This discards dark knowledge: the ensemble's full probability distribution encodes the geometry of the decision boundary.

A distribution of "Cat (0.6), Dog (0.3), Truck (0.0001)" tells the student that "Dog" is a plausible near-miss while "Truck" is a category error — information erased by the hard label "Cat." The correct approach minimizes KL divergence between full teacher and student distributions using temperature-scaled logits, transferring not just the decision but the teacher's uncertainty structure.

The answer that gets you hired: "Hard targets discard the ensemble's uncertainty. I match full probability distributions via KL Divergence on temperature-scaled logits — we aren't copying the output, we are cloning the ensemble's intuition about which errors are plausible."

Q71 Why did removing one whitespace from a prompt cause a 12% accuracy drop? show ▾

Most candidates attribute it to grammar confusion. In reality, LLMs process integer sequences, not text. Deleting a whitespace changes the tokenization boundary: in many tokenizers, " word" (with a leading space) and "word" (without) map to completely different token IDs, as distinct to the model as "apple" and "orange."

Furthermore, instruction-tuned models latch onto specific formatting tokens (colons, XML tags, newlines) as strong predictive features during SFT. Removing them pushes the input out-of-distribution relative to the fine-tuning corpus, breaking those learned shortcuts.

The answer that gets you hired: "It is a distribution shift caused by tokenization artifacts — the model learned specific formatting tokens as strong features during SFT. In production, I use Automated Prompt Optimization or enforce the exact chat template the model was fine-tuned on."

Q72 In Back-Translation, which direction should you generate synthetic data? show ▾

Most candidates generate synthetic English targets to give the model more good outputs to learn from. Training on synthetic targets teaches the decoder to reproduce machine-translation artifacts instead of real human language.

The correct direction: take the large monolingual English corpus, back-translate it into synthetic source-language text, then train on (Synthetic Source, Real English Target) pairs. The encoder learns to handle noisy, translationese inputs; the decoder always learns from gold-standard human text, preserving its fluency.

The answer that gets you hired: "We pair synthetic sources with natural targets. This ensures the decoder learns the true distribution of real human language rather than overfitting to artifacts from another model."

15

Architecture Deep-Dive Traps

Architecture-level traps — MoE routing deadlocks, RoPE vs. learned positions, and loss function choices that kill production models.

Q73 Should you use Top-1 routing in Mixture-of-Experts training? show ▾

Most candidates endorse Top-1 routing for maximum sparsity and throughput. The trap: selecting only the winning expert performs a hard argmax, which has a gradient of zero almost everywhere.

Non-selected experts receive zero signal, stagnate, and the gating network freezes because it never learns the cost of not routing to the runners-up. Training must use Top-2 routing to maintain gradient flow to both winner and runner-up. Top-1 can optionally be re-enabled at inference after the router has fully converged.

The answer that gets you hired: "Sparsity requires redundancy during learning. I would use Top-2 routing during training to maintain gradient flow to the gating network, even at the cost of 2x active parameters per forward pass."

Q74 What is the critical advantage of RoPE over learned absolute position embeddings? show ▾

Most candidates describe efficiency. The critical architectural property is zero-shot length extrapolation. Standard learned embeddings assign unique vectors to absolute positions up to the training maximum — position 4,097 is unseen if the model trained on 4,096 tokens (a hard wall).

RoPE encodes relative distances via rotation angles: the attention score between tokens m and n depends only on offset (m − n), not absolute indices. This enables post-training techniques like YaRN and NTK-aware scaling to extend the effective context to 32k+ without full retraining.

The answer that gets you hired: "Standard embeddings treat positions like unique zip codes — the model is lost beyond its training range. RoPE encodes relative distances so post-training context extension (YaRN, NTK scaling) can run a 4k-trained model on 32k documents."

Q75 Why prefer Sigmoid + Cross-Entropy over Hinge Loss for binary text classification? show ▾

Most candidates argue Hinge Loss is better because it stops updating on correct samples, reducing overfitting. The production failure is the vanishing gradient on correctness: once a prediction clears the margin, Hinge Loss assigns zero gradient regardless of whether the model is 51% or 99% confident.

Sigmoid + Cross-Entropy never settles; even at 0.9 probability the loss continues pushing toward 1.0, providing a calibrated probability output. Production classifiers require those calibrated probabilities to set operational precision/recall thresholds without retraining.

The answer that gets you hired: "Hinge Loss kills the gradient on correct examples, preventing the model from learning certainty. Production systems need calibrated probabilities — not raw margin scores — to tune thresholds, which only Sigmoid + Cross-Entropy provides."

Quiz

Random Question Mode

Test yourself with random questions. Try answering before revealing — it's the fastest way to find gaps.

Click "Next Question" to begin.