AI Engineering interview questions
Reviewed by Mark Dickie · Last updated
AI engineering is the discipline of building, deploying, and maintaining AI systems that work reliably in production environments. For interviews, expect questions on model serving patterns, latency and throughput tradeoffs, monitoring for data and concept drift, pipeline reproducibility, and incident response when model performance degrades. You should be able to explain the difference between batch and real-time inference, how to version datasets and models, and how to structure CI/CD for ML. Production AI topics also cover cost management, guardrails for LLMs, A/B testing, and safety evaluation.
| Topic area | What comes up in interviews |
|---|---|
| Model serving | Batch vs. real-time inference, latency budgets, autoscaling, model registries |
| Monitoring | Data drift, concept drift, prediction quality metrics, alerting thresholds |
| MLOps pipelines | Experiment tracking, dataset versioning, CI/CD for ML, reproducibility |
| LLM production | Guardrails, prompt versioning, eval suites, token cost control, caching |
| Reliability | Rollbacks, shadow deployment, canary releases, incident response |
What does an AI engineering interview test?
Interviewers want to see whether you can take a model from a notebook to a system that serves users at scale. That means they probe three areas in roughly equal measure:
- How do you deploy a model so it stays available under load, and what do you do when traffic spikes?
- How do you detect that a production model has degraded, and what is your rollback path?
- How do you keep training and serving pipelines reproducible so a bug can be traced to a specific data or code version?
How should I prepare for production AI questions?
Work through each layer of the stack: data ingestion, training, evaluation, deployment, and monitoring. For LLM-specific roles, add prompt engineering, evaluation frameworks, and guardrail design. Practice articulating tradeoffs out loud: when you would choose a microservice over an embedded model, when batch scoring beats real-time inference, and where caching belongs in your architecture. The quiz below draws from real interview rounds and lets you check your answers immediately.
Key facts
- Tarmac has 140 AI Engineering interview questions on this topic, 10 of them on this page, at difficulty 1–5 of 5.
- Tarmac last reviewed these AI Engineering interview questions on 19 August 2026.
At a glance
| Questions | 10 shown · 140 in the bank |
|---|---|
| Difficulty | 1–5 of 5 |
| Formats | Multiple choice, Flashcard, Fill in the blank, Short answer, Find the bug, Design exercise |
What you'll review
- ai production
Practice questions
AI Engineering/ai-production
In a production LLM application, what does the temperature sampling parameter primarily control?#
Options
Show answer
The temperature parameter controls the randomness of token selection during generation. A temperature of 0 makes the model deterministic, while higher values increase variability and creativity by flattening the probability distribution over tokens. It does not affect token limits, speed, or concurrency.
Temperature scales the logits before the softmax in LLM decoding. A value of 0 makes the model deterministic (always pick the highest-probability token), while higher values flatten the distribution, increasing randomness and variability in the output. It does not control token limits, generation speed, or concurrency.
AI Engineering/ai-production
What is an LLM hallucination in the context of a production AI application?#
Show answer
A hallucination occurs when the model generates text that sounds plausible and confident but is factually incorrect or entirely fabricated — e.g., inventing citations, misstating facts, or fabricating API behavior. Production systems mitigate this with retrieval grounding (RAG), output validation, and guardrails.
Hallucination is a foundational concept in AI production engineering. It refers to the model producing fluent but ungrounded or false output. Recognizing it is the first step toward deploying mitigation strategies such as RAG, fact-checking layers, and structured output validation.
AI Engineering/ai-production
Your production service calls an LLM API and starts receiving repeated HTTP 429 (Too Many Requests) responses under load. Which retry strategy is the established best practice for handling this?#
Options
Show answer
The established best practice is exponential backoff with jitter, while honoring any Retry-After header the API returns. HTTP 429 means the provider is throttling requests, so progressively increasing the delay between retries—jittered to avoid synchronized client retries—relieves pressure without wasting calls. Ignoring the header or retrying immediately just deepens the rate-limit condition.
HTTP 429 signals rate limiting. The standard production pattern is exponential backoff (doubling wait time between retries) combined with jitter (randomized offsets) to avoid thundering-herd effects among concurrent clients. Respecting the Retry-After header lets the client honor the server's explicit guidance on when to retry. Retrying immediately worsens the load, switching models does not address rate limits, and increasing max_tokens is unrelated to throttling.
AI Engineering/ai-production
In a Retrieval-Augmented Generation (RAG) pipeline, retrieved documents are typically split into smaller, often overlapping pieces of text before being embedded or passed to the LLM. This process is called _____.#
Show answer
In a Retrieval-Augmented Generation (RAG) pipeline, retrieved documents are typically split into smaller, often overlapping pieces of text before being embedded or passed to the LLM. This process is called chunking.
Chunking is the standard term for breaking long documents into smaller, manageable text segments (chunks) so that each piece fits within embedding model input limits and the LLM's context window. Overlapping chunks help preserve context continuity across boundaries. This is a foundational step in building RAG systems for production.
AI Engineering/ai-production
In a production LLM inference server, continuous (dynamic) batching improves GPU throughput over static (request-level) batching primarily because:#
Options
Show answer
Continuous batching keeps the GPU saturated by admitting new requests into the active batch as soon as earlier requests finish generating, eliminating the idle time that static batching incurs while waiting for the longest sequence in a fixed batch to complete. It operates at the token-iteration level rather than the request level.
Static batching waits until a full batch is ready, then processes all requests together and must wait for the longest output to finish before freeing those slots — leaving GPU capacity idle whenever sequences have variable length. Continuous batching (also called iteration-level or in-flight batching) performs the forward pass at the iteration (token) level: as soon as any request in the batch finishes, its slot is immediately filled with a queued request. This keeps the GPU saturated across heterogeneous completion times. Weight compression (a), activation quantization (c), and tensor parallelism (d) are orthogonal optimizations that do not describe the batching mechanism.
AI Engineering/ai-production
You deploy a supervised model trained on historical data into production. Over time the live input feature distribution begins to diverge from the training distribution, even though labels are not immediately available. What is the standard term for this phenomenon, and name two statistical tests or metrics commonly used to detect it?#
Show answer
The phenomenon is called data drift (or covariate drift / input drift). Two commonly used detection methods are the Kolmogorov-Smirnov (KS) test and the Population Stability Index (PSI). Other valid examples include the Wasserstein distance, KL divergence, or Jensen-Shannon divergence.
When the distribution of input features in production shifts away from the training distribution — without the labels necessarily being available yet — this is called data drift (also covariate drift or input drift). Common detection techniques compare the statistical distribution of a baseline (training) sample against a recent live sample: the Kolmogorov-Smirnov (KS) test for continuous features, the Population Stability Index (PSI), and information-theoretic measures such as KL divergence, Jensen-Shannon divergence, or the Wasserstein distance. The answer must name the phenomenon and at least two of these methods.
AI Engineering/ai-production
In a production LLM inference server using continuous batching (iteration-level / dynamic batching), how is the KV cache managed when requests with very different sequence lengths are coalesced in the same batch?#
Options
Show answer
In continuous batching, each request maintains its own independently growing KV cache, and the scheduler can admit or evict requests at any single decoding-iteration boundary without recomputing prior token KV states. This per-request, per-iteration KV management is what lets short and long sequences coexist in the same batch without padding waste or batch-level synchronization.
Continuous batching (also called iteration-level or dynamic batching) processes requests at the granularity of a single decoding iteration. Each request owns its own KV cache allocation, which grows by one token's worth of KV state per iteration. Because admission and eviction happen at iteration boundaries—not batch boundaries—the scheduler can inject a new request into a running batch the moment a slot frees up, without recomputing KV tensors for any in-flight request. Option (a) and (c) describe static batching, where the entire batch is padded to the longest sequence and all KV caches are held until every request finishes. Option (d) conflates continuous batching with vLLM's PagedAttention: prefix deduplication is a memory-level optimization specific to certain implementations, not an inherent property of continuous batching, and it does not deduplicate at the attention computation level.
AI Engineering/ai-production
The following async function streams a chat completion from the OpenAI API. In production it intermittently raises TypeError: sequence item 0: expected str instance, NoneType found. Which line is the bug?#
import asyncio
from openai import AsyncOpenAI
client = AsyncOpenAI()
async def stream_chat(messages: list[dict], max_retries: int = 3) -> str:
"""Stream a chat completion with retries on failure."""
for attempt in range(max_retries):
try:
stream = await client.chat.completions.create(
model="gpt-4o",
messages=messages,
stream=True,
)
chunks = []
async for chunk in stream:
chunks.append(chunk.choices[0].delta.content)
return "".join(chunks)
except Exception:
if attempt == max_retries - 1:
raise
await asyncio.sleep(2 ** attempt)Show answer
The bug is on line 17.
Line 17 appends chunk.choices[0].delta.content directly. In the OpenAI streaming protocol, the first chunk typically carries only the role field with content set to None, and the final chunk carries finish_reason with content also None. When any of these None values lands in the chunks list, the "".join(chunks) call on line 18 raises TypeError. The fix is to guard against None, e.g. chunks.append(chunk.choices[0].delta.content or ""). None of the other lines contain a defect: the retry loop, backoff, and streaming setup are all correct.
AI Engineering/ai-production
In production LLM inference, the decode phase of autoregressive generation (batch size = 1, single new token per step) is memory-bandwidth bound rather than compute-bound. Which statement gives the fundamental reason?#
Options
Show answer
During single-token decode, the GPU must load all model weights from HBM to produce one token, giving an arithmetic intensity of roughly 1 FLOP/byte — about 200× below a modern GPU's roofline ridge point (~208 FLOP/byte on an A100). The operation is therefore memory-bandwidth bound, not compute bound. This is why continuous batching and speculative decoding help: they amortize the weight-load cost across more useful work.
During decode with batch size 1, the model performs one forward pass that touches every weight tensor to emit a single token. The arithmetic intensity is approximately (2 × num_params) FLOPs divided by (2 × num_params) bytes loaded ≈ 1 FLOP/byte (FP16). An A100's ridge point sits at roughly 312 TFLOPS ÷ 1.5 TB/s ≈ 208 FLOP/byte, so the actual intensity is ~200× below the ridge — the GPU is starved for data, not computation. This is why techniques like continuous batching (increasing batch size raises arithmetic intensity by amortizing weight loads) and speculative decoding (running multiple forward passes' worth of work per weight load) improve decode throughput. Option b is incorrect: per-step decode attention is O(n) (the new token attends to all n previous positions), not O(n²); the O(n²) cost applies to prefill. Option c is incorrect because the KV cache is small relative to model weights and its growth does not cause cache-line stalls in the architectural sense. Option d is incorrect because tensor cores can still be used; the bottleneck is data movement, not tile-size limitations.
AI Engineering/ai-production
You are building the inference serving platform for a company that deploys a 70B-parameter decoder-only LLM (80 transformer layers, GQA with 8 KV heads, head_dim 128, FP16 weights ≈ 140 GB). The system must meet these SLOs:#
Show answer
Architecture Overview
1. Iteration-Level Continuous Batching
Each GPU worker runs an iteration-level scheduler (as in vLLM/TGI). At every decode step (every ~20–40 ms), the scheduler:
- Evicts sequences that have emitted EOS or hit max tokens.
- Admits queued requests up to the limit imposed by available KV cache blocks.
- Issues a single fused forward pass for the entire active batch.
This contrasts with static batching, where a batch is formed at request time and all slots are occupied until the longest sequence finishes — leaving GPU capacity idle on short sequences. Continuous batching keeps the batch full at every step, maximizing the arithmetic intensity of the decode phase (which is memory-bandwidth bound at batch size 1 but improves as batch size grows because weight loads are amortized).
2. KV Cache Memory Management (PagedAttention)
The KV cache is the primary throughput constraint. Per-sequence KV cache size for this model:
- 80 layers × 2 (K+V) × 8 KV heads × 128 head_dim × 2,048 tokens × 2 bytes (FP16) = 80 × 2 × 8 × 128 × 2,048 × 2 ≈ 6.7 GB per sequence.
Across 5,000 concurrent sessions, the total KV cache demand is enormous (tens of TB), far exceeding GPU HBM. We use PagedAttention: the KV cache is divided into fixed-size blocks (e.g., 16 tokens/block). A block table maps each sequence's logical block indices to non-contiguous physical HBM blocks. This eliminates internal fragmentation (sequences only allocate the blocks they need) and enables prefix sharing: sequences with identical system prompts share the same physical KV cache blocks via copy-on-write reference counting. On 8× H100 nodes (640 GB HBM total), we can hold roughly 640 GB ÷ 6.7 GB ≈ 95 concurrent full-length sequences per node — so we need horizontal scaling across many nodes, each running replicas.
3. Prefill/Decode Disaggregation
Prefill is compute-bound (large GEMMs, high arithmetic intensity); decode is memory-bandwidth bound (small per-token GEMMs). Mixing a long prefill with active decodes in the same iteration stalls the decode batch: the GPU spends 200+ ms on the prefill while every active decode token waits, blowing the 50 ms inter-token SLO.
I disaggregate into two worker pools:
- Prefill workers (TP=8 within a node): handle prompt encoding, produce KV cache, then transfer it to decode workers.
- Decode workers (TP=8): run continuous batching over active sequences only.
Alternatively, chunked prefill splits a long prompt into chunks (e.g., 512 tokens) and interleaves each chunk as one "sequence" in the continuous batching loop, so no single prefill monopolizes an iteration. This is simpler operationally and I'd start with chunked prefill, moving to full disaggregation if TTFT SLOs are not met.
4. SLO-Aware Autoscaling and Admission Control
The admission controller estimates per-request TTFT (based on prompt length, current queue depth, and prefill worker utilization) before enqueuing. If the estimate exceeds 800 ms, the request is either queued (with a position estimate returned to the client) or rejected with an HTTP 429 + Retry-After.
Autoscaling is keyed on KV cache block utilization and estimated TTFT distribution, not raw GPU utilization. When KV cache utilization exceeds 80% or p90 TTFT exceeds 600 ms, the scaler provisions new replica groups. For bursty traffic, I maintain a warm pool of 2 idle replica groups (pre-loaded model weights, zero active sequences) that can absorb a 10× spike within seconds. Predictive scaling uses historical traffic patterns to pre-warm additional groups 15 minutes before expected spikes.
Graceful degradation under overload: shed load by (a) reducing max output tokens per request, (b) routing overflow traffic to a smaller fallback model (e.g., 13B), and (c) returning 429s with backoff headers.
5. Model Parallelism and Fault Tolerance
The 140 GB model requires TP=8 within a single H100 node (NVLink provides 900 GB/s inter-GPU bandwidth, minimizing all-reduce latency). With 8 nodes, we run 8 independent TP replicas, each serving ~95 concurrent sequences → ~760 concurrent sessions; further horizontal scaling (more nodes, more replicas) handles the 5,000-session target.
Pipeline parallelism (PP) is used only if a single replica cannot fit the model within one node's HBM — for a 70B model in FP16 (140 GB) across 8×80GB GPUs (640 GB), TP=8 within one node suffices. PP would be needed for larger models (e.g., 405B).
Fault tolerance: each TP replica group runs a health check (heartbeat + dummy inference every 5 seconds). On GPU failure within a TP group, the entire group is marked unhealthy; the load balancer stops routing to it, and in-flight requests on that group fail and are retried on a healthy replica (with the original prompt re-prefilled; KV cache for in-progress decodes is lost). A replacement group is launched from the warm pool. At 99.9% availability, we tolerate occasional request retries — the client SDK uses idempotent request IDs to handle duplicates.
Putting It Together
Client → API Gateway (rate limit, auth)
→ SLO-Aware Router (estimates TTFT, routes to replica)
→ Replica Group (TP=8, H100 node)
→ Chunked-Prefill Scheduler (interleaves prefill chunks with decode)
→ PagedAttention KV Cache Manager (block allocation, prefix sharing)
→ Continuous Batching Engine
← Streaming SSE response
Autoscaler watches KV cache util + TTFT p90 → scales warm pool
This is an open-ended design exercise scored against five weighted criteria: (c1) iteration-level continuous batching vs static batching, (c2) KV cache as the primary throughput bottleneck with paged/block allocation and prefix sharing, (c3) prefill/decode disaggregation or chunked-prefill scheduling to protect inter-token latency, (c4) SLO-aware admission control and autoscaling keyed on KV cache pressure rather than generic GPU utilization, and (c5) tensor parallelism within a single node with a justified size comparison showing pipeline parallelism is unnecessary for 140 GB across 640 GB HBM, plus replica-level failover. The sample answer addresses every criterion.
Related interview questions
The other 130 questions
This page shows 10. A free account opens the rest and marks your answers, which a page of answers can't do. Miss one and it returns a few days later, then a week after that, until you stop missing it.
Free · the whole bank · 100 graded answers per 30 days · written feedback and job-ad quizzes on the paid plan