AI Engineering Interview Questions: LLM Foundations
Reviewed by Mark Dickie · Last updated
LLM foundations are the architectural and mathematical concepts that underpin large language models, from tokenization and embedding spaces to self-attention, transformer layers, and the training objectives that shape model behavior. For an AI engineering interview, you need a working grasp of how tokens map to vectors, how attention scales with sequence length, what fine-tuning actually changes in the weights, and where inference-time tradeoffs (batching, KV caching, quantization) matter. Interviewers also probe your understanding of model limitations — context window boundaries, hallucination causes, and the gap between pretraining knowledge and instruction-following.
The quiz below covers the core areas you are likely to face:
| Topic Area | What Interviewers Ask |
|---|---|
| Tokenization | BPE vs. WordPiece, special tokens, how vocab size affects latency |
| Embeddings & vector spaces | Cosine similarity, dimensionality tradeoffs, positional encoding |
| Self-attention | Q/K/V computation, multi-head vs. single-head, attention complexity |
| Training objectives | Causal LM vs. masked LM, RLHF vs. DPO, loss functions |
| Fine-tuning methods | Full fine-tuning, LoRA, QLoRA, parameter-efficient approaches |
| Inference optimization | KV cache, speculative decoding, quantization, batching strategies |
What does an AI engineering interview test on LLM foundations?
Most interviews split into three layers. First, conceptual recall: can you explain how a transformer processes a sequence end to end? Second, applied reasoning: given a specific architecture change or hyperparameter, can you predict the effect on output quality or cost? Third, practical engineering: how do you deploy, monitor, and constrain a model in production?
- Transformer architecture walkthrough — trace an input token from embedding through attention, feed-forward layers, and output projection. Know what each component contributes.
- Attention mechanics — compute attention complexity for a sequence of length n, explain why FlashAttention exists, and discuss sparse or sliding-window attention as alternatives.
- Training vs. inference — distinguish pretraining, supervised fine-tuning, and alignment stages. Understand what data each stage uses and what behavior it shapes.
- Tokenization edge cases — how OOV tokens are handled, why BPE can produce surprising splits, and how token boundaries affect prompt engineering.
- Context windows and memory — explain KV cache growth, the cost of long-context inference, and techniques like ring attention or context compression.
- Model evaluation — know when to use perplexity vs. task-specific benchmarks, and how to detect overfitting in a fine-tuned checkpoint.
How should I prepare if I am short on time?
Focus on the mechanics of attention and the training pipeline. If you can explain self-attention from scratch, describe the difference between LoRA and full fine-tuning, and reason about inference latency under batching, you will handle the majority of LLM-foundations questions. The quiz below lets you check which of those areas still need work.
Key facts
- Tarmac has 84 AI Engineering interview questions on this topic, 10 of them on this page, at difficulty 1–5 of 5.
- Tarmac last reviewed these AI Engineering interview questions on 17 August 2026.
At a glance
| Questions | 10 shown · 84 in the bank |
|---|---|
| Difficulty | 1–5 of 5 |
| Formats | Multiple choice, Fill in the blank, Code output, Multiple answer |
What you'll review
- llm foundations
Practice questions
AI Engineering/llm-foundations
During autoregressive text generation, what does a decoder-only LLM (e.g., GPT-style) explicitly output at each step that is then fed back as input for the next step?#
Options
Show answer
An autoregressive (decoder-only) LLM outputs a probability distribution over its entire vocabulary at each step. One token is sampled from that distribution, appended to the running context, and fed back in so the model can produce the next-step distribution — repeating until a stop condition is met.
A decoder-only LLM applies a softmax over its vocabulary head at the final position, producing a probability distribution. The sampler picks one token from that distribution, appends it to the context, and the model re-runs (or caches forward through) to produce the next-step distribution. It does not output a single embedding, generate tokens in parallel, or emit a loss value during inference.
AI Engineering/llm-foundations
Modern decoder-only LLMs such as GPT and Llama are built on the Transformer architecture. The core mechanism that lets every token attend to all previous tokens in the sequence — computing weighted relationships without recurrence — is called _____.#
Show answer
Modern decoder-only LLMs such as GPT and Llama are built on the Transformer architecture. The core mechanism that lets every token attend to all previous tokens in the sequence — computing weighted relationships without recurrence — is called self-attention.
Self-attention (often implemented as multi-head scaled dot-product attention) is the defining mechanism of the Transformer. It computes a weighted sum of value vectors for every token pair, enabling each position to incorporate information from all other positions in a single forward pass — replacing the sequential recurrence of RNNs/LSTMs.
AI Engineering/llm-foundations
In the scaled dot-product attention used by transformer-based LLMs, the query–key dot product is divided by the square root of _____ before the softmax is applied. This scaling prevents the dot products from growing large in magnitude, which would otherwise push the softmax into regions with extremely small _____.#
Show answer
In the scaled dot-product attention used by transformer-based LLMs, the query–key dot product is divided by the square root of d_k before the softmax is applied. This scaling prevents the dot products from growing large in magnitude, which would otherwise push the softmax into regions with extremely small gradients.
Scaled dot-product attention computes Attention(Q, K, V) = softmax(QKᵀ / √d_k) V. The factor √d_k — the square root of the key (head) dimension — counteracts the fact that dot products of high-dimensional vectors have large variance, which would push the softmax into saturation (outputs near 0 or 1) and consequently produce vanishingly small gradients during backpropagation, making training unstable.
AI Engineering/llm-foundations
In an autoregressive LLM that samples tokens via softmax over logits, what is the effect of setting the temperature parameter to a value approaching 0 (e.g., 0.01)?#
Options
Show answer
Setting temperature near 0 makes the softmax distribution increasingly peaked on the highest-logit token, which is effectively greedy (argmax) decoding. The model still responds differently to different prompts, but the sampled token becomes almost entirely deterministic.
Temperature divides each logit by T before softmax. As T → 0, the largest logit dominates exponentially more strongly, so the distribution concentrates virtually all probability mass on the single highest-scoring token — behavior equivalent to greedy (argmax) decoding. The model's output still depends on the prompt (ruling out option a); temperature 1 means no scaling (ruling out option c); and near-zero temperature makes output deterministic, not random (ruling out option d).
AI Engineering/llm-foundations
In a decoder-only transformer (e.g., GPT-style architecture), what is the primary purpose of causal (masked) self-attention during training?#
Options
Show answer
Causal masking in decoder-only transformers ensures each token attends only to itself and earlier positions, preserving the autoregressive property so predictions never depend on future tokens. It prevents a train/inference mismatch but does not reduce O(n²) attention complexity, address gradient issues, or serve as regularization.
Causal masking sets the attention scores for all future positions to negative infinity (zero after softmax), so each token at position i can only attend to positions 0 through i. This preserves the autoregressive property: during training, the model never 'sees the future,' which matches the left-to-right generation used at inference time. Without causal masking, there would be a train/inference mismatch. Causal masking does not change the O(n²) complexity of attention, does not address vanishing gradients (that is the role of residual connections and layer norm), and is not a form of regularization like dropout.
AI Engineering/llm-foundations
During autoregressive inference in transformer-based LLMs, the _____ cache stores the projected key and value tensors from all previously processed token positions, allowing each newly generated token to attend to the full context without re-running earlier tokens through the attention layers. This reduces the per-step cost of generating one additional token from O(n²) (full recompute) to O(n) in sequence length.#
Show answer
During autoregressive inference in transformer-based LLMs, the KV cache stores the projected key and value tensors from all previously processed token positions, allowing each newly generated token to attend to the full context without re-running earlier tokens through the attention layers. This reduces the per-step cost of generating one additional token from O(n²) (full recompute) to O(n) in sequence length.
The KV (key-value) cache is a fundamental optimization in autoregressive LLM inference. After each token is processed through the attention layers, its projected key and value vectors are cached. When the next token is generated, only the new token's query, key, and value need to be computed; the new query attends to the cached keys and values of all previous positions. This avoids recomputing attention over the entire prefix for every new token, reducing per-step complexity from O(n²) to O(n). The trade-off is memory: the cache grows linearly with sequence length and number of layers, which is a key consideration in serving LLMs at scale.
AI Engineering/llm-foundations
The code below counts the total trainable parameters in a small decoder-only transformer model. The model uses tied input/output embeddings (the output projection reuses the embedding weight matrix, so the embedding parameters are counted only once). Each transformer layer contains four attention projection matrices (Q, K, V, O) and a two-layer feed-forward network — no biases, no layer-norm parameters. Trace the code and determine the printed output.#
d_model = 512
d_ff = 2048
n_layers = 6
vocab_size = 10000
# Per-layer params (no biases, no layer-norm params)
# Attention: Q, K, V, O = 4 * d_model^2
# FFN: 2 * d_model * d_ff
per_layer = 4 * d_model * d_model + 2 * d_model * d_ff
# Embedding + output projection (tied weights -> counted once)
embedding = vocab_size * d_model
total = n_layers * per_layer + embedding
print(total)Show answer
23994368
Tracing the code step by step: d_model = 512, d_ff = 2048, n_layers = 6, vocab_size = 10000. Per layer, attention contributes 4 × 512 × 512 = 1,048,576 parameters and the FFN contributes 2 × 512 × 2048 = 2,097,152, so per_layer = 3,145,728. Across 6 layers that is 6 × 3,145,728 = 18,874,368. The tied embedding matrix (counted once because weights are shared between input embeddings and output projection) adds 10,000 × 512 = 5,120,000. total = 18,874,368 + 5,120,000 = 23,994,368, which is what print(total) outputs.
AI Engineering/llm-foundations
Rotary Position Embeddings (RoPE) are used in many modern decoder-only LLMs (e.g., LLaMA, Mistral). Which of the following statements about RoPE are correct? Select all that apply.#
Options
Pick every one that applies.
Show answer
The correct statements are that RoPE encodes relative position through position-dependent rotations applied to both query and key vectors, and that the resulting query–key dot product depends only on the relative distance between tokens. RoPE uses no learnable parameters — the rotations are fixed functions of position and a base frequency — and it is applied to both queries and keys, not queries alone.
Statement (a) is correct: RoPE multiplies each pair of dimensions in the query and key by a 2-D rotation whose angle is determined by the token's absolute position index, so the rotations themselves are position-dependent. Statement (b) is correct and is the defining property of RoPE — because the same rotation scheme is applied to both Q and K, the inner product <R_m·q, R_n·k> simplifies to a function of (m − n), the relative position, making the attention scores translation-invariant. Statement (c) is wrong: RoPE uses fixed rotation matrices derived from a base frequency θ (e.g., 10 000) and the position index; there are no learnable position-embedding parameters. Statement (d) is wrong: RoPE is applied to both queries and keys — rotating only one would not produce the relative-position property in the dot product.
AI Engineering/llm-foundations
Rotary Position Embeddings (RoPE) encode relative position by rotating query and key vectors in 2D subspaces at different frequencies θ_i, where the relative rotation between positions m and n in dimension i is (m−n)·θ_i. A well-known issue is that RoPE generally fails to extrapolate to sequence lengths beyond those seen during training. Techniques such as NTK-aware interpolation and YaRN aim to fix this.#
Options
Show answer
RoPE fails to extrapolate beyond training lengths because its lowest-frequency dimensions encounter relative rotation angles at long distances that were never seen during training, creating an out-of-distribution regime. High-frequency dimensions wrap periodically and stay in-distribution, but low-frequency ones grow monotonically with distance. NTK-aware scaling and YaRN address this by rescaling the frequency base to compress those low-frequency angles back into the trained range.
The correct answer is (a). RoPE rotates each 2D subspace at frequency θ_i; the relative rotation between positions m and n is (m−n)·θ_i. For high-frequency dimensions (large θ_i), the angle wraps around periodically, so every possible relative angle was already encountered during training. For low-frequency dimensions (small θ_i), the relative angle grows monotonically with distance and does not wrap within the training range — so at distances beyond training, those dimensions see relative rotation angles never observed before, producing out-of-distribution attention patterns. NTK-aware scaling and YaRN mitigate this by rescaling the frequency base (or per-dimension frequencies) to compress low-frequency rotation angles back into the trained range. (b) is wrong because the failure is distributional, not a floating-point precision issue. (c) incorrectly describes absolute additive position encodings (e.g., sinusoidal PE), not RoPE's rotary multiplication. (d) is wrong because each dimension's rotational contribution is bounded in [−1, 1] via cosine/sine; the logits do not grow polynomially with distance.
AI Engineering/llm-foundations
The code below implements a single head of causal (decoder-only) self-attention with numerical stability (max-subtraction before softmax). Trace the computation by hand and determine the exact value printed.#
import numpy as np
Q = np.array([[1, 0],
[0, 1],
[1, 1]])
K = np.array([[1, 0],
[0, 1],
[1, 1]])
V = np.array([[1, 2],
[3, 4],
[5, 6]])
d_k = 2
scores = Q @ K.T / np.sqrt(d_k)
# Causal mask: upper-triangular set to -1e9
mask = np.triu(np.ones((3, 3)), k=1) * -1e9
scores = scores + mask
# Numerically stable softmax
scores = scores - scores.max(axis=-1, keepdims=True)
exp_scores = np.exp(scores)
attn = exp_scores / exp_scores.sum(axis=-1, keepdims=True)
output = attn @ V
print(np.round(output[2].sum(), 4))Show answer
8.0208
We need output[2], the attention output for the third token (index 2), which can attend to all three positions under causal masking.
Step 1 — Raw dot products (Q[2] · K[j]): Q[2] = [1, 1]. • Q[2]·K[0] = 1·1 + 1·0 = 1 • Q[2]·K[1] = 1·0 + 1·1 = 1 • Q[2]·K[2] = 1·1 + 1·1 = 2
Step 2 — Scale by 1/√d_k = 1/√2: scores[2] = [0.70711, 0.70711, 1.41421]
Step 3 — Causal mask (row 2 is all zeros, no masking): No change.
Step 4 — Stable softmax (subtract max = 1.41421): Shifted: [−0.70711, −0.70711, 0] exp: [0.49307, 0.49307, 1.0] Sum = 1.98614 attn[2] = [0.24826, 0.24826, 0.50347]
Step 5 — Weighted sum of V: output[2] = 0.24826·[1,2] + 0.24826·[3,4] + 0.50347·[5,6] = [0.24826 + 0.74479 + 2.51736, 0.49653 + 0.99306 + 3.02083] = [3.51042, 4.51042]
Step 6 — Sum: 3.51042 + 4.51042 = 8.02083 → rounded to 4 decimals = 8.0208.
Related interview questions
The other 74 questions
This page shows 10. A free account opens the rest and marks your answers, which a page of answers can't do. Miss one and it returns a few days later, then a week after that, until you stop missing it.
Free · the whole bank · 100 graded answers per 30 days · written feedback and job-ad quizzes on the paid plan