System Design interview questions: AI system architecture practice

Combine three system-level levers: stream tokens over SSE to cut time-to-first-token, cache responses to repeated or similar prompts to skip inference entirely, and route simple requests to a small model while escalating only when needed. These change how often you invoke the model, which model you invoke, and how you deliver the result. Adding replicas, blanket CDN caching, or raising the output token limit each fail here.

True. Streaming changes delivery, not work. Sending tokens as they are produced lets the user see output begin in a few hundred milliseconds, dramatically improving perceived time-to-first-token. But the model still generates the same number of tokens through the same number of forward passes, so the compute — and therefore the cost — is unchanged. Cutting cost requires doing less inference.

LLM serving is GPU-bound: capacity is driven by tokens/sec and concurrent requests rather than raw RPS, and continuous (dynamic) batching of in-flight requests on the inference server is the main throughput lever. A single generation holds the GPU for its whole token stream, so a request isn't a uniform unit of work like a CRUD call. Adding stateless replicas without GPUs does nothing for a GPU-bound workload.

False. RAG reduces hallucinations by grounding responses in retrieved source material, but it does not eliminate them. The model can still misread, mis-cite, or confabulate even when relevant context is present, and it hallucinates when retrieval itself fails on ambiguous queries or off-topic chunks. Mitigation needs retrieval-quality tuning, citation enforcement, and a confidence-based fallback to admitting it doesn't know.

First, a latency budget: the inference call dominates, so I'd stream tokens over SSE to cut perceived time-to-first-token and set explicit p95 targets. Second, cost control: prompt/semantic caching to skip repeat inference, model routing (small model first, escalate only when needed), and output token limits to bound spend. Third, safety and abuse protection: input/output guardrails and content moderation, plus per-user rate limiting and quotas so one user can't run up the bill or degrade others. Fourth, production observability and ongoing evaluation: log prompts/responses/tokens/latency, trace requests, and run quality evals on sampled traffic to catch regressions. Fifth, resilience: the model API is a third-party dependency, so a circuit breaker that fails fast plus a degraded fallback (a cheaper/local model, a cached answer, or a graceful 'try again' message) keeps the product usable when the provider is down.

At query time the user's question is embedded into a vector, a similarity search retrieves the top-k most relevant document chunks from the vector store, and those chunks are prepended to the LLM's prompt as context before generation. Retrieval is the critical bottleneck because the LLM can only synthesise information that appears in its context window — it cannot reason about chunks it never received. A good retriever with a mediocre model often outperforms a great model with a poor retriever. Common failure modes: semantic mismatch between query and chunk embedding spaces, chunks too large (diluting signal) or too small (losing context), and missing reranking to promote the most relevant results within the top-k.

The sound choices are dynamic request batching so GPU compute is shared across in-flight requests, separating the prefill and decode phases onto different node pools because their resource profiles differ sharply, routing short low-complexity requests to a smaller cheaper model (model tiering), and using a KV-cache plus prompt prefix caching so shared prefixes aren't recomputed. Always running the largest model for every request blows cost and latency budgets without commensurate quality gain.

Treat the model as untrusted and never give it direct authority to execute side effects. Put the privilege at the tool layer, not in the prompt: each tool runs with least-privilege, scoped credentials tied to the actual user's permissions, so even a hijacked agent can only do what that user could already do. Validate and constrain tool arguments before executing — schema-check them, allowlist destinations (e.g. only internal recipients), and bound parameters — rather than trusting whatever the model emitted. Keep a human-in-the-loop confirmation for high-blast-radius or irreversible actions (sending external email, deleting data, moving money). Defend specifically against prompt injection from retrieved/tool content by separating untrusted data from instructions and not letting fetched text silently escalate what the agent may do. Wrap it all in rate limits and quotas to bound damage and abuse, and log/trace every tool call so actions are auditable and reversible where possible. The principle: the model proposes, a guarded execution layer with real authorization disposes.

A RAG request flows from query to response in this order:

1. Encode the user query into a vector embedding using an embedding model.
2. Run an approximate nearest-neighbour search against the vector store to retrieve top-k chunks.
3. Re-rank or filter the retrieved chunks for relevance (optional but common).
4. Assemble a prompt that injects the retrieved context alongside the user query.
5. Send the assembled prompt to the LLM and stream the generated response to the client.

Requirements. This is grounded generation (RAG), not free-form chat: every answer must be backed by the company's own content, cited, and must never invent policy. Hard constraints: strict per-tenant isolation (no cross-account leakage), human escalation when the model isn't confident, and acceptable chat latency. LLM calls dominate cost and latency, so much of the design is about doing fewer/cheaper/cached calls.

Retrieval (RAG) pipeline. Offline, an indexing pipeline chunks the ~500K docs, embeds each chunk, and writes vectors + metadata (tenant, doc id, version, last-updated) into a vector store. It runs continuously so edited docs are re-embedded and the index stays fresh — that's how a policy reworded last week is reflected. At query time: embed the question → hybrid retrieval (vector similarity + keyword) for the top-k relevant chunks → assemble a prompt: system instructions + retrieved chunks (with their source ids) + the conversation → call the LLM, which answers from the provided context and returns citations.

Serving, cost & latency. The LLM is the bottleneck, so: cache responses (a semantic/embedding cache so near-duplicate questions reuse an answer); stream tokens so the user sees output immediately; tier models — easy/FAQ-matched queries get a small model or a direct retrieved-answer, only hard ones hit the large model; keep prompts tight (top-k, not the whole KB); and queue/batch under load to ride out the 100 QPS peak. These cut both spend and tail latency.

Grounding, hallucination & isolation. The system prompt instructs the model to answer only from the retrieved context and to say it doesn't know (and escalate) when the context doesn't cover the question — this is the main hallucination guard, reinforced by always returning citations the user can check. Tenant isolation is enforced at retrieval: the vector query is filtered by the requesting user's tenant_id (a metadata filter), so account-specific chunks for another tenant are never even candidates for the prompt. Enforcing it at retrieval (not by hoping the model behaves) is what makes leakage structurally impossible.

Failure handling & escalation. LLM calls get timeouts and retries; on provider outage we fail over to a fallback model or degrade gracefully to 'let me connect you to a human', never an indefinite hang. Low-confidence answers — weak retrieval scores, or the model signalling uncertainty — are escalated to a human queue with the conversation context attached, rather than guessing at policy.

Evaluation & scaling. Quality is tracked with an offline eval set (known Q→A pairs) run on each prompt/model change, LLM-as-judge plus human spot-checks on sampled live answers, and logging of every answer with its retrieved context and the user's thumbs feedback to find regressions and gaps. The serving tier is stateless and scales horizontally behind a load balancer; the vector store and indexing pipeline scale independently of it.