API Gateway System Design Interview Questions

True. In a monolithic architecture, all components—UI, business logic, and data access—are packaged into one deployable unit. A change to any component requires rebuilding and redeploying the entire application. This coupling is a primary reason teams adopt microservices, which allow individual services to be deployed independently.

Complete the two key concepts: A load balancer balances incoming network traffic across multiple servers to improve availability and throughput. When user session data must remain on the same server for the duration of a session, this is called sticky persistence.

An API gateway is a single client-facing entry point that routes and aggregates requests and centralises cross-cutting concerns — authentication, TLS termination, rate limiting — so services don't each reimplement them. It complements a load balancer rather than replacing it, holds no application or session state, is not a database proxy, and is not a build tool that merges services into one artifact.

Vector clocks (or version vectors) with application-level or CRDT-based merging are the best fit. They capture causal relationships between writes across regions, allowing the system to detect true conflicts (concurrent writes with no causal ordering) and apply deterministic merge logic — without requiring cross-region coordination on the write path. LWW risks silent data loss due to clock skew, 2PC sacrifices availability, and read repair only acts on the read path.

The correct order is: Detect the partition → Identify majority vs. minority nodes → Determine CP vs. AP SLA → Enforce the policy (CP: reject minority writes outright; AP: accept all writes and track divergence) → Reconcile after healing (AP: conflict resolution; CP: minority re-syncs from the majority log). Identifying the majority/minority split must precede enforcement, and true CP systems reject — not queue — minority-side writes.

The correct statements are the ones about message queues, stateful services, and read replicas. A message queue like Kafka decouples producers and consumers and smooths traffic spikes. Stateful services require externalised or sharded state before horizontal scaling; just adding instances risks split-brain. Read replicas offload read traffic from the primary. Placing a CDN in front of the application tier is wrong because CDNs cache HTTP responses at the edge — they cannot introspect or cache arbitrary database query results.

Adding read replicas does NOT mitigate a write-heavy hot partition. Read replicas offload read traffic to secondary copies, but every write still routes to the primary shard — so the write bottleneck is completely unchanged. Strategies that actually reduce write pressure include re-sharding with virtual nodes, write-behind/coalescing caches, and splitting the hot shard into finer-grained child shards.

The recommended evolutionary order is: (1) Monolith, (2) Message queue / microservice decomposition, (3) Saga orchestration, (4) CQRS, (5) Event sourcing, (6) Geo-replication. Microservice decomposition immediately removes ACID guarantees, so saga orchestration must come next — before CQRS or event sourcing — as Richardson's Microservices Patterns argues. CQRS then prepares the write/read split that makes event sourcing lower-risk, and geo-replication is safe only once the event-sourced write model is stable.

Goals & when to split. The pains are concrete: coupled deploys, no fault isolation, and an inability to scale checkout independently of browsing (which is ~50× the traffic). Those justify splitting. But microservices aren't free — they trade in-process calls and ACID transactions for network hops, partial failure, and distributed-transaction headaches. So I split deliberately, by business value, not everywhere at once.

Service boundaries. I decompose by bounded context / business capability: catalog, cart, orders, payments, inventory, shipping. Each is high-cohesion and owns its domain logic and its data. I avoid the anti-pattern of slicing along database tables into chatty anaemic services — boundaries follow how the business reasons about the domain, so services stay loosely coupled.

Database decomposition. Each service gets its own datastore (database-per-service) — this is what actually breaks the shared-DB scaling ceiling and SPOF. Services no longer cross-join each other's tables; instead they expose APIs and emit events. The orders service stores the product id (and denormalises the few fields it needs) and fetches the rest from the catalog API, or subscribes to catalog change events. What we give up is the convenient cross-table join and strong cross-entity consistency — replaced by API calls and eventual consistency.

Communication & gateway. An API gateway fronts everything: it authenticates, routes to the right service, rate-limits, and can fan out/aggregate. Between services I pick per interaction: synchronous REST/gRPC for a read I need right now (get product), asynchronous events for 'order placed' / 'inventory changed' so producers don't block on consumers. Every synchronous call is wrapped with timeouts, retries (idempotent), and a circuit breaker so a slow dependency fails fast instead of hanging the caller.

Cross-service correctness (place order). Placing an order spans orders + inventory + payments — three databases, so no single ACID transaction. I use a saga: a sequence of local transactions with compensating actions. Orchestrated, an order orchestrator does: reserve inventory → charge payment → confirm order; if payment fails after the reservation, it fires the compensation release inventory. The order sits in a PENDING state until the saga completes, and the user sees pending → confirmed (or failed). This is eventually consistent but never permanently leaks reserved stock.

Migration & new failure modes. No big-bang. I use the strangler-fig pattern: keep the monolith running behind the gateway, peel out one service plus its data at a time, route that capability's traffic to the new service (feature-flagged), dual-write / shadow-read to validate, then cut over and delete the monolith's copy. New failure modes — partial failure (one service down), cascading failure (a slow dependency dragging callers down), and added latency from hops — are contained with circuit breakers, bulkheads (isolate resource pools per dependency), idempotent retries with backoff, and distributed tracing so a request can be followed across services.

Requirements. This is grounded generation (RAG), not free-form chat: every answer must be backed by the company's own content, cited, and must never invent policy. Hard constraints: strict per-tenant isolation (no cross-account leakage), human escalation when the model isn't confident, and acceptable chat latency. LLM calls dominate cost and latency, so much of the design is about doing fewer/cheaper/cached calls.

Retrieval (RAG) pipeline. Offline, an indexing pipeline chunks the ~500K docs, embeds each chunk, and writes vectors + metadata (tenant, doc id, version, last-updated) into a vector store. It runs continuously so edited docs are re-embedded and the index stays fresh — that's how a policy reworded last week is reflected. At query time: embed the question → hybrid retrieval (vector similarity + keyword) for the top-k relevant chunks → assemble a prompt: system instructions + retrieved chunks (with their source ids) + the conversation → call the LLM, which answers from the provided context and returns citations.

Serving, cost & latency. The LLM is the bottleneck, so: cache responses (a semantic/embedding cache so near-duplicate questions reuse an answer); stream tokens so the user sees output immediately; tier models — easy/FAQ-matched queries get a small model or a direct retrieved-answer, only hard ones hit the large model; keep prompts tight (top-k, not the whole KB); and queue/batch under load to ride out the 100 QPS peak. These cut both spend and tail latency.

Grounding, hallucination & isolation. The system prompt instructs the model to answer only from the retrieved context and to say it doesn't know (and escalate) when the context doesn't cover the question — this is the main hallucination guard, reinforced by always returning citations the user can check. Tenant isolation is enforced at retrieval: the vector query is filtered by the requesting user's tenant_id (a metadata filter), so account-specific chunks for another tenant are never even candidates for the prompt. Enforcing it at retrieval (not by hoping the model behaves) is what makes leakage structurally impossible.

Failure handling & escalation. LLM calls get timeouts and retries; on provider outage we fail over to a fallback model or degrade gracefully to 'let me connect you to a human', never an indefinite hang. Low-confidence answers — weak retrieval scores, or the model signalling uncertainty — are escalated to a human queue with the conversation context attached, rather than guessing at policy.

Evaluation & scaling. Quality is tracked with an offline eval set (known Q→A pairs) run on each prompt/model change, LLM-as-judge plus human spot-checks on sampled live answers, and logging of every answer with its retrieved context and the user's thumbs feedback to find regressions and gaps. The serving tier is stateless and scales horizontally behind a load balancer; the vector store and indexing pipeline scale independently of it.