AI Engineering Interview Questions
Reviewed by Mark Dickie · Last updated
AI engineering is the discipline of building, evaluating, and shipping software systems that use large language models and other ML models as core components. For interviews, you should know how LLMs generate text (tokenization, context windows, decoding strategies), how to design retrieval-augmented generation (RAG) pipelines end to end, how to evaluate model outputs systematically, and how to fine-tune or adapt models for specific tasks. You should also be comfortable with vector databases, embedding models, guardrails for hallucination and safety, and the production concerns of latency, cost, and observability.
| Topic Area | What gets tested |
|---|---|
| LLM fundamentals | Tokenization, context window limits, temperature/sampling, instruct vs. chat vs. base models |
| RAG systems | Chunking strategies, embedding model choice, vector search vs. hybrid search, reranking |
| Fine-tuning & adaptation | LoRA/QLoRA, instruction tuning vs. continued pretraining, when to fine-tune vs. prompt |
| Evaluation | BLEU/ROUGE limitations, human eval, LLM-as-judge, golden datasets, A/B testing |
| Deployment & ops | Batching, caching, streaming, rate limiting, cost monitoring, latency budgets |
| Safety & alignment | Jailbreak prevention, system prompts, content filtering, red-teaming basics |
What does an AI engineering interview actually cover?
Most interviews split into two phases: a systems-design conversation and a coding or applied-knowledge round. The design round often asks you to architect an LLM-powered feature (a chatbot, a document Q&A tool, a content moderation pipeline) and defend your choices for model selection, retrieval strategy, and evaluation plan. The applied round tends to probe your understanding of failure modes.
- Design a RAG pipeline that handles 1M+ documents with sub-second latency.
- Explain how you would evaluate whether a fine-tuned model is better than the base model with prompting.
- Build a guardrail system that blocks prompt injection attempts.
- Choose between gpt-4o, a fine-tuned Llama 3, and a distilled model for a classification task, and justify the tradeoffs.
- Diagnose why a RAG system returns irrelevant chunks and propose fixes.
How should I prepare for the systems-design round?
Start by mapping out a reference RAG architecture and knowing each component well enough to defend or swap it. Know your embedding model's dimensionality and max sequence length. Know the difference between approximate nearest neighbor search and exact search, and when each is appropriate. Be ready to discuss chunk overlap, chunk size, and how they affect retrieval quality. Practice whiteboarding the full request flow: query arrives, gets embedded, vector search runs, results get reranked, context goes into the prompt, model generates, output gets filtered.
For evaluation specifically, you should be able to explain why offline metrics like BLEU and ROUGE are weak proxies for LLM output quality, and what you would use instead. The strongest answers name a specific evaluation method (golden-set accuracy, pairwise human comparison, LLM-as-judge with a rubric) and describe how you would set up the test.
Key facts
- Tarmac has 647 AI Engineering interview questions on this topic, 10 of them on this page, at difficulty 1–5 of 5.
- Tarmac last reviewed these AI Engineering interview questions on 23 August 2026.
At a glance
| Questions | 10 shown · 647 in the bank |
|---|---|
| Difficulty | 1–5 of 5 |
| Formats | Ordering, Multiple choice, Multiple answer, Fill in the blank, True / false, Flashcard, Code output, Short answer, Find the bug, Design exercise |
What you'll review
- agent loops
- guardrails
- latency cost
- embeddings
Practice questions
AI Engineering/agents/agent-loops
Arrange the steps of a standard LLM agent loop in the correct execution order, starting from the beginning of one iteration.#
Put these in order
Show answer
In a standard LLM agent loop, the correct execution order is: (1) the LLM reasons over the current context and selects an action, (2) that action or tool call is executed, (3) the agent receives the observation or tool result, (4) a stopping condition is checked to see if the task is complete, and (5) if not done, the observation is appended to context and the next iteration begins. This repeating think-act-observe cycle is the core pattern behind frameworks like ReAct.
An agent loop (also called a ReAct loop or think-act loop) follows a repeated cycle: the agent observes the current state/context, reasons or thinks about what to do, takes an action (e.g., calls a tool), receives an observation back, and repeats until a stopping condition is met. This is the fundamental pattern behind virtually all LLM-based agent frameworks.
AI Engineering/agents/agent-loops
In a typical LLM-based agent loop, which component is responsible for actually executing a tool call (e.g., running a web search or calling an API)?#
Options
Show answer
The agent runtime (or orchestrator) that wraps the LLM is responsible for actually executing tool calls like web searches or API requests. The LLM itself only reasons about which action to take and what arguments to pass — it cannot run code or call external services directly. The runtime performs the action, then feeds the result back to the LLM as an observation to continue the loop.
In an agent loop the LLM does NOT execute code or call APIs directly — that is the job of the surrounding runtime/executor. The LLM's role is limited to reasoning and deciding which action to take and what arguments to pass. The runtime then actually performs the action and returns the result (observation) to the LLM.
AI Engineering/agents/agent-loops
Which of the following are valid stopping conditions that can terminate an LLM agent loop? Select all that apply.#
Options
Pick every one that applies.
Show answer
Valid stopping conditions for an LLM agent loop include the LLM emitting a designated "Final Answer" signal, a cap on the maximum number of steps or iterations being reached, and a programmatic check detecting that the task objective is satisfied. Any of these terminates the loop, which would otherwise run indefinitely. Reaching a minimum token count, by contrast, is not a practical or meaningful stopping criterion.
Without a stopping condition an agent loop would run forever. Common termination criteria are: the LLM emits a special 'Final Answer' token/signal, a maximum number of steps/iterations is reached, or the task objective is detected as satisfied. Reaching a minimum token count is not a meaningful stopping condition and is not used in practice.
AI Engineering/agents/agent-loops
To prevent an agent loop from running indefinitely, developers should set a _____ limit on the number of iterations, and the loop should also exit when the agent emits a _____ action (e.g., finish or final_answer).#
Show answer
To prevent an agent loop from running indefinitely, developers should set a maximum-step limit on the number of iterations, and the loop should also exit when the agent emits a terminal action (e.g., finish or final_answer).
A maximum-step (or max-iterations) guard is essential to prevent an agent loop from running forever when the LLM never produces a terminal action or gets stuck in a tool-call cycle. Without it, the loop can consume unbounded tokens and API calls. The other options — temperature, embedding size, and chunk size — do not control loop termination.
AI Engineering/evaluation-safety/guardrails
A prompt injection attack occurs when a malicious user embeds instructions inside their input (e.g., inside a document the LLM is asked to summarize) that override or hijack the system prompt's intended behavior.#
Options
Show answer
True. Prompt injection is the attack where malicious instructions are embedded in user-supplied content (e.g., a document the model is asked to summarize), causing the model to treat those instructions as authoritative and override the legitimate system prompt. It is a key concern in LLM safety because models do not inherently distinguish between trusted instructions and data they are processing.
This statement is true. Prompt injection is precisely the attack class where adversarial content—often hidden in user-supplied data such as documents, web pages, or tool outputs—contains instructions that the model interprets as authoritative, potentially causing it to ignore safety rules, leak information, or take unintended actions.
AI Engineering/ai-production/latency-cost
In LLM production systems, what does TTFT stand for, and why is it a distinct metric from total response latency?#
Show answer
TTFT = Time To First Token. It measures the elapsed time from when a request is sent until the client receives the first generated token. It is distinct from total latency because total latency covers the full generation of all tokens, whereas TTFT captures only the initial delay (network round-trip + prefill/prompt processing). In streaming UIs, a low TTFT makes the system feel fast even if total generation takes several more seconds.
Time To First Token (TTFT) measures the delay from sending a request until the very first token of the response is received by the client. It is distinct from total response time (which covers all tokens) and throughput (tokens per second). TTFT is especially important in streaming UX because users see the response start arriving sooner even if total generation time is the same. It is NOT the same as total latency.
AI Engineering/llm-foundations/embeddings
This computes the cosine similarity between two embedding vectors and prints it rounded to 4 decimals. What does it print?#
import math
a = [1, 2, 2]
b = [2, 0, 1]
dot = sum(x * y for x, y in zip(a, b))
norm_a = math.sqrt(sum(x * x for x in a))
norm_b = math.sqrt(sum(x * x for x in b))
print(round(dot / (norm_a * norm_b), 4))Show answer
0.5963
Cosine similarity is dot(a, b) / (||a|| * ||b||). The dot product is 1*2 + 2*0 + 2*1 = 4; ||a|| = sqrt(1+4+4) = 3 and ||b|| = sqrt(4+0+1) = sqrt(5) ≈ 2.2360679.... So 4 / (3 * 2.236...) ≈ 0.59628..., which rounds to 0.5963. This is the core ranking primitive behind vector search — the actual embeddings just have hundreds or thousands of dimensions, but the arithmetic is identical.
AI Engineering/evaluation-safety/guardrails
In a production LLM application with a RAG pipeline, your security review flags two distinct failure modes:#
Show answer
A single input guardrail catches malicious user queries (prompt injection) but cannot intercept PII that the model extracts from retrieved documents and embeds in its response — the violation happens after the LLM call. Conversely, a single output guardrail can redact PII in responses but allows the injected prompt to reach the model, risking system-prompt override or data exfiltration during generation. The minimum architecture requires: (1) an input guardrail applied before the LLM call that detects prompt-injection patterns, validates query intent, and sanitises user input; and (2) an output guardrail applied after the LLM call that scans the generated response for PII (using regex, NER, or a classifier), policy violations, and hallucinated citations before returning to the user. Both layers must be independent so a bypass of one does not compromise the other.
This question tests deep knowledge of layered guardrail architecture. Input guardrails run before the LLM call to block malicious or off-topic prompts. Output guardrails run after the LLM response to catch hallucinations, PII leakage, or policy violations before delivery to the user. A correct defence-in-depth pipeline therefore needs both layers. Relying only on input guardrails misses prompt-injection-induced unsafe outputs; relying only on output guardrails still allows prompt injection to reach the model, wasting compute and risking data exfiltration. Grounding checks (RAG relevance) are an output-side concern. Rate limiting is an infrastructure concern orthogonal to content safety. A well-designed system applies: (1) input validation → (2) LLM call → (3) output validation, with each stage having independent failure modes.
AI Engineering/agents/agent-loops
The following Python snippet implements one iteration of an OpenAI function-calling agent loop. There is exactly one bug that will cause the agent loop to silently malfunction (wrong role/message structure fed back to the model). Identify the buggy line number.#
import json, openai
def run_agent_step(messages, tools, tool_registry):
response = openai.chat.completions.create(
model="gpt-4o",
messages=messages,
tools=tools,
tool_choice="auto",
)
msg = response.choices[0].message
messages.append(msg)
if msg.tool_calls:
for tc in msg.tool_calls:
fn_name = tc.function.name
fn_args = json.loads(tc.function.arguments)
result = tool_registry[fn_name](**fn_args)
messages.append({"role": "assistant", "content": str(result)})
return messagesShow answer
The bug is on line 17.
The bug is on line 17. The code appends the tool result to messages as a dict with role assistant, but tool/function call results must be appended with role tool and must reference the tool_call_id from the assistant's message (e.g., {"role": "tool", "tool_call_id": tc.id, "content": str(result)}). Appending the result as an assistant message (a) loses the tool_call_id linkage and (b) causes the model to treat the observation as its own prior utterance rather than an external tool response, breaking the agent loop. All other lines are structurally correct for a minimal function-calling agent loop.
AI Engineering/agents/agent-loops
Design the agent loop behind a support assistant that answers billing questions. It can read invoices, read the customer's plan, and issue refunds up to a limit. Runs take anywhere from 2 to 40 iterations. Cover how the loop terminates, what happens when a tool fails, how context is kept under the window, how refunds are controlled, and how you would debug a bad run a week later. State your assumptions.#
Show answer
Assume a few thousand runs a day, a refund ceiling of $200, and 30-day trace retention. Most tickets follow a handful of shapes, so I would route: a classifier sends the common ones down fixed workflows and only the tail reaches the agent, which cuts cost and shrinks the surface where things can go wrong.
The loop terminates on four conditions, not one: the model returns a final answer that passes a goal check; a cumulative token budget for the run is exhausted; a step cap fires as a backstop; or a repeated-call detector sees the same (tool, arguments) hash twice with nothing new in context. Budget is the primary control because steps are a poor proxy for cost — each iteration re-sends the transcript, so late steps cost far more than early ones. On any stop without an answer the agent hands the ticket to a human with its partial findings attached, rather than replying with something it is not sure of.
Tool failures go back into the context as tool results flagged as errors, so the model can adapt — try a different lookup, or tell the user. Retry policy is per-tool, not global: reads retry with exponential backoff, and the refund call never retries blindly, because a 5xx there is ambiguous rather than failed. Refunds carry an idempotency key so a duplicate collapses server-side.
For a 40-iteration run I pin the system prompt, tool schemas and goal, summarise older turns into a running recap once the transcript passes a threshold, clear stale tool results while keeping the calls that produced them, and keep large invoice payloads in storage behind a reference the agent re-reads on demand.
Refunds are gated in code. The loop intercepts the call before execution, verifies the amount against the invoice server-side, and requires human approval above a threshold — the model's authority is whatever the loop grants it, and the system prompt is a hint, not a control. Invoice text is untrusted: it reaches the context as data, and instructions inside it have no privilege, which is exactly why enforcement cannot live in the prompt.
Every run writes a trace keyed by run id and ticket id, one span per iteration, capturing the rendered prompt, the tool calls and arguments, results, token counts, and the stop reason. A week later I can open the ticket, find the run, and see which iteration went wrong and what the model was looking at when it did.
The main tradeoff is autonomy against friction: a lower approval threshold means fewer bad refunds and more human load. I would start conservative, measure the approval queue, and raise the threshold only against evidence.
This exercise separates people who have run an agent in production from people who have read about them. The tells are consistent. A weak answer treats max_steps as termination, wraps every tool in one retry, plans to 'use a bigger context window', and puts the refund limit in the system prompt. A strong answer knows that a step cap does not bound cost, that retry policy is a property of the tool rather than of the loop, that trimming has to pin the instructions it must never drop, and above all that a prompt instruction is not an authorization boundary — the loop executes tools, so the loop is the only place a refund limit can be enforced. The prompt-injection probe is the sharpest discriminator: any design where the model's instructions are the sole control fails it, because the invoice is attacker-influenced text arriving in the same context. The debuggability criterion is what makes the whole thing operable — without per-iteration traces tied to a run id, nobody can answer 'why did it do that' a week later, and every incident becomes a guess.
Related interview questions
The other 637 questions
This page shows 10. A free account opens the rest and marks your answers, which a page of answers can't do. Miss one and it returns a few days later, then a week after that, until you stop missing it.
Free · the whole bank · 100 graded answers per 30 days · written feedback and job-ad quizzes on the paid plan