AI Engineering Interview Questions — LLM Model Selection

Quantization is a post-training compression technique, distinct from distillation (which trains a new, smaller model to mimic a teacher) or pruning (which removes weights entirely). It matters for selection because it widens the shortlist: instead of choosing between 'the big accurate model' and 'the small cheap model,' a quantized variant of the big model is often a third option that gets most of the accuracy at close to the small model's cost.

Parameter count is not the sole determinant of output quality. A smaller model that has been fine-tuned on domain-specific data can outperform a larger general-purpose model on that domain. Additionally, instruction-tuning, RLHF, quantization trade-offs, and the quality of training data all heavily influence real-world performance. Benchmarks repeatedly show task-specific smaller models (e.g., fine-tuned 7B models) beating larger base models on targeted tasks.

The 'context window' (also called context length) determines how much text — including the system prompt, conversation history, and user message — can be processed in one API call. Models with larger context windows (e.g., 128k tokens) are better suited for tasks like document summarisation or long conversations. 'Inference cost' (charged per token) directly impacts budget planning; models like GPT-4 cost significantly more per token than smaller alternatives, making it a critical factor during model selection.

When selecting an LLM for a production use case, several factors critically constrain model choice: (1) context window size matters when inputs are long (e.g., entire documents); (2) inference latency and cost per token directly impact feasibility at scale; (3) licensing terms (e.g., commercial use restrictions on some open-weight models like LLaMA 2's acceptable-use policy) can disqualify a model legally; and (4) demonstrated performance on benchmarks relevant to the task domain matters. However, the number of attention heads is an internal architectural detail that has no direct bearing on model selection for a downstream task — it affects implementation details but is not a practical selection criterion on its own.

Hard constraints go first because they are cheap, binary, and eliminate the most candidates for free — there is no point running an expensive quality eval on a model that cannot legally or technically be used. Quality on your own data comes next, since generic benchmarks don't tell you how a model performs on your task. Only among candidates that are both eligible and good enough do cost and latency at your real volume become the deciding factor. A shadow or canary pilot catches gaps between the eval set and live traffic before full commitment, and documenting the fallback up front means a later swap is a known procedure, not a scramble.

Line 2's filter condition is inverted: contextWindow < requiredContext keeps only the models whose context is too small, which is exactly backwards — it should be >= requiredContext to keep models that meet or exceed what's needed. That is why the function recommends undersized models: the 'eligible' set it builds is precisely the set that should have been excluded. The rest of the function is correct — line 3's null return for an empty eligible set is the right behavior when nothing qualifies, and the reduce on lines 4-6 correctly finds the minimum-cost model by keeping m over best only when m is cheaper.

The exercise tests whether a candidate treats data policy and required output shape as disqualifying filters applied before any quality comparison, rather than folding them into a single weighted score where a great answer from a non-compliant vendor could still 'win.' It also tests whether cost and latency get grounded in the feature's actual volume rather than compared as abstract per-token prices, and whether the plan includes a real-traffic pilot rather than jumping straight to full rollout.

Rotary Position Embeddings (RoPE) encode position by rotating query and key vectors in pairs within the attention computation. The key property is that the dot-product between a query at position m and a key at position n depends only on the relative offset (m − n), not their absolute positions — this emerges naturally from the rotation mathematics. This is why RoPE-based models generalize better to sequence lengths beyond their training context: the relative distances are inherently represented. ALiBi (Attention with Linear Biases) achieves a similar effect differently — it subtracts a linear position bias directly on attention logits rather than modifying Q/K vectors. The critical distinction is that RoPE operates in the Q/K embedding space, while ALiBi operates on the attention score matrix post-QK dot-product. Extended context methods like YaRN and LongRoPE scale the RoPE frequency base (theta) — NOT the embedding dimension — to accommodate longer sequences without full fine-tuning.

Speculative decoding uses a small, fast 'draft' model to propose multiple candidate tokens in parallel, then the large 'verifier' model checks them in a single forward pass. The acceptance criterion is designed so that the output distribution is identical to sampling from the verifier alone — this is the key correctness guarantee. The speedup comes from the fact that verifying K tokens costs only slightly more than verifying 1 (batch dimension), while the draft model is cheap. Critically, speculative decoding does NOT change the output distribution of the target model; it only accelerates sampling. The acceptance rate (α) depends on how well the draft model's distribution matches the verifier's. If the draft model is too mismatched (low α), the overhead of running both models exceeds the gain, negating the speedup. Self-speculative decoding (e.g., Medusa heads, EAGLE) avoids the need for a separate draft model by using auxiliary prediction heads on the base model itself.