AI Engineering Interview Questions — Streaming in Production AI

Streaming in an LLM API call means the server sends output tokens incrementally as they are generated, letting the client start displaying or processing text before the full response is complete. This reduces time-to-first-token and improves perceived latency, unlike batching, polling, or compressing the whole response before sending.

The term is streaming (or token streaming). The most common HTTP-based transport mechanism used to implement it is Server-Sent Events (SSE), where the server keeps a single long-lived HTTP response open and sends chunks (e.g., individual tokens) as they are produced, so the client sees incremental output without waiting for the entire response to finish.

Streaming a model response means sending tokens (or small token chunks) to the client incrementally as the model generates them, rather than waiting for the full response to finish. This is commonly implemented with Server-Sent Events or chunked transfer encoding and reduces perceived latency because the user sees text appear in real time.

The temperature parameter controls the randomness of token selection during generation. A temperature of 0 makes the model deterministic, while higher values increase variability and creativity by flattening the probability distribution over tokens. It does not affect token limits, speed, or concurrency.

A hallucination occurs when the model generates text that sounds plausible and confident but is factually incorrect or entirely fabricated — e.g., inventing citations, misstating facts, or fabricating API behavior. Production systems mitigate this with retrieval grounding (RAG), output validation, and guardrails.

Streaming sends tokens to the client as they're generated rather than waiting for the full completion, which dramatically lowers perceived latency — the user sees the first token (time-to-first-token) in a fraction of a second instead of staring at a spinner for the whole response. What it does not change is the total time to finish or the total cost: the same number of tokens are generated and billed, so streaming improves perceived responsiveness, not throughput or price.

False. Streaming changes when the user sees the first token, not when the run finishes: the same model calls happen, the same tools run, the same total time elapses. It moves perceived latency rather than real latency — often the best change available, since a user watching text appear waits more patiently than one watching a spinner. Measure it honestly as two numbers, time-to-first-token and time-to-final-answer, or it will look like a fix for a latency problem it has not touched.

Streaming LLM responses primarily reduces time-to-first-token: the client begins receiving partial output as soon as the model produces its first token, rather than waiting for the entire response to be generated. This improves perceived latency. It does not reduce total token cost, extend the context window, or remove the need for rate limiting.

['Hello world.', ' How are you?']

The established best practice is exponential backoff with jitter, while honoring any Retry-After header the API returns. HTTP 429 means the provider is throttling requests, so progressively increasing the delay between retries—jittered to avoid synchronized client retries—relieves pressure without wasting calls. Ignoring the header or retrying immediately just deepens the rate-limit condition.

In a Retrieval-Augmented Generation (RAG) pipeline, retrieved documents are typically split into smaller, often overlapping pieces of text before being embedded or passed to the LLM. This process is called chunking.

Tool-call arguments arrive as fragments, so the JSON is incomplete until the block finishes. The loop must accumulate fragments per tool-call id and parse only once that call is complete — parsing a partial buffer throws, and a buffer that happens to parse can execute a truncated argument. Because events interleave across parallel calls, fragments must be keyed by call id rather than appended to one buffer. The loop's shape is unchanged: you still wait for the full assistant turn before executing. Streaming buys the first visible token, not a shorter run.

Use Server-Sent Events with text/event-stream, flush each token chunk immediately rather than buffering, and rely on chunked transfer encoding (HTTP/1.1) or HTTP/2 stream frames for partial delivery. Buffering the entire response defeats streaming, and switching to raw TCP is not standard practice — it removes encryption and authentication for negligible gain.

The bug is on line 11.

The sky [FILTERED] is blue

The phenomenon is called data drift (or covariate drift / input drift). Two commonly used detection methods are the Kolmogorov-Smirnov (KS) test and the Population Stability Index (PSI). Other valid examples include the Wasserstein distance, KL divergence, or Jensen-Shannon divergence.

The streamed completion lifecycle runs in this order:

1. Client sends the request; server opens a streaming connection to the model API
2. Model prefills the prompt and emits the first token (time-to-first-token)
3. Server relays incremental token deltas to the client as they arrive
4. Client appends each delta, progressively rendering the answer
5. Stream terminates with a stop/finish event and usage totals

Perceived responsiveness in streaming LLM applications is dominated by time to first token (TTFT), the gap between sending the request and receiving the first output token; once streaming starts, the ongoing delivery rate is measured in tokens per second (TPS).

In the Server-Sent Events (SSE) wire format defined by the HTML5 specification, the HTTP response must set the Content-Type header to **text/event-stream**. When an OpenAI-compatible API streams chat completions over SSE, each event payload is prefixed with the literal data: . Per the OpenAI Chat Completions streaming protocol, the server signals that no more tokens will be sent by emitting a final data: **[DONE]** sentinel, after which the client closes the stream.

Nginx's default proxy_buffering on accumulates the upstream response before forwarding it to the client, causing the batched delivery. To stream SSE tokens through Nginx incrementally, either set proxy_buffering off for the streaming location or have the backend send the X-Accel-Buffering: no response header, which Nginx honors per-request. The EventSource API itself has no batching timer, and HTTP/2 has no inherent 30-second flush interval.

The bug is on line 17.

False. Prefill computes attention as a parallel matrix–matrix operation across all prompt tokens, whereas decode computes matrix–vector attention for a single new token against the KV cache. Batching both phases in one forward pass (as vLLM's chunked prefill does) requires specialized attention kernels that handle these distinct patterns together — it cannot be done without modifying the attention computation.

Streaming Architecture for a Multi-Tenant LLM Gateway

Transport & Event Framing (c1)

Browser clients connect via Server-Sent Events (SSE) over HTTPS. SSE is chosen over WebSocket because the data flow is unidirectional (server→client) for token streaming, SSE works through all standard HTTP infrastructure (CDNs, reverse proxies, load balancers) without protocol upgrades, and the browser EventSource API provides built-in reconnection with Last-Event-ID. For clients needing bidirectional communication (e.g., audio input for speech models), a WebSocket upgrade path is available but is not the default.

Each SSE frame is a JSON object with a type discriminator:

event: delta
data: {"type":"text_delta","content":"Hel","index":42,"request_id":"req_abc"}

event: delta
data: {"type":"tool_call_delta","tool_call_id":"call_1","arguments":"{\"loc"}

event: usage
data: {"type":"usage","prompt_tokens":120,"completion_tokens":43,"model":"gpt-4o"}

event: done
data: {"type":"done","request_id":"req_abc"}

The gateway assigns a monotonically increasing id: field (SSE event ID) to every frame so the browser's EventSource automatically sends Last-Event-ID on reconnection. The index field gives the client a per-request token sequence number for gap detection.

Backpressure & Upstream Cancellation (c2)

Each client connection has a bounded write buffer (e.g., 64 KB ring buffer) on the gateway. When the gateway reads a token from the upstream provider stream, it attempts to write it to the client SSE connection with a write deadline of 2 seconds. If the client is not reading (flaky mobile connection), the write blocks until the buffer fills or the deadline expires:

1. If the buffer is full or the write deadline fires, the gateway closes the client connection and marks the stream as cancelled.
2. The gateway then propagates cancellation upstream: it calls abort() on the fetch Response body (for HTTP-based providers like OpenAI/Anthropic) or cancels the gRPC client stream (for vLLM's gRPC interface). This causes the provider to stop generating tokens and free the KV-cache slot on the GPU.
3. For self-hosted vLLM, the gateway sends an HTTP request abort, which vLLM's async streaming handler catches, cancels the generation task, and releases the GPU memory.

The key invariant: a stalled client never causes unbounded memory growth on the gateway, and a cancelled stream always frees upstream GPU resources within seconds.

Stream Resumability & Idempotency (c3)

The gateway maintains a short-lived append log per request backed by Redis Streams (TTL: 5 minutes). Every token frame forwarded to the client is also appended to the Redis stream with its sequential event ID. The full generated output is thus persisted server-side for the TTL window without re-invoking the model.

On reconnection:

1. The browser EventSource reconnects automatically and sends Last-Event-ID: 41.
2. The gateway looks up the request by request_id, reads events 42+ from the Redis stream, and replays them to the client.
3. If the upstream generation is still in progress (the client disconnected but the gateway did not cancel upstream — see below), the gateway continues forwarding live tokens after the replayed ones.
4. If the generation already completed, the gateway replays all remaining buffered events and sends done.

Critical design decision: when a client disconnects, the gateway does NOT immediately cancel the upstream. Instead, it enters a grace period (e.g., 10 seconds) during which it continues consuming the upstream stream into the Redis log. If the client reconnects within the grace period, it gets a seamless resume. If the client does not reconnect within the grace period, the gateway cancels the upstream to free resources. This trades a small amount of wasted GPU time for a much better reconnection UX on flaky networks.

Idempotency: the request_id and monotonic event IDs ensure the client can deduplicate any replayed frames. The client applies tokens by index, so out-of-order or replayed frames are handled gracefully.

Multi-Provider Race & Partial Failure (c4)

For the speculative fan-out scenario, the gateway opens two upstream SSE streams concurrently. The design is first-token-wins with immediate loser cancellation:

1. Both upstream streams are opened. The gateway holds a mutex on the client-facing stream.
2. The first provider to emit a text_delta acquires the mutex and becomes the winner. Its stream is forwarded to the client.
3. The gateway immediately aborts the losing provider's stream (closes the HTTP connection, cancels the gRPC stream) to stop its generation and free resources.
4. Race handling: if the losing provider emits a token between the winner's first token and the abort being processed, that token is silently discarded — it is never forwarded to the client. The mutex ensures only one provider's deltas reach the client at a time.
5. Partial failure: if the winning provider's stream errors mid-generation, the gateway sends an error event to the client ({"type":"error","code":"provider_error","message":"..."}) followed by closing the connection. The client can retry. The gateway does NOT silently switch to the losing provider (which was already cancelled and may have partially consumed GPU resources).
6. If both providers fail before any token is emitted, the gateway returns an error to the client without any deltas.

The client never receives interleaved or duplicated deltas because the mutex + winner-takes-all design guarantees a single source of truth for the duration of the stream.

Streaming Observability & Real-Time Cost (c5)

Metrics are emitted via OpenTelemetry spans and Prometheus histograms:

• TTFT: a span is started when the gateway receives the client request. The span's first child event is recorded when the first text_delta is forwarded to the client. TTFT = first_delta_time - request_received_time. This is exported as a Prometheus histogram (llm_gateway_ttft_seconds) labeled by model, provider, and tenant.
• Inter-token latency (ITL): the gateway records the wall-clock delta between consecutive text_delta forwards. Exported as a histogram (llm_gateway_inter_token_latency_seconds) with the same labels.
• Throughput: tokens/sec computed at stream close as total_tokens / (stream_end - first_delta_time).

Cost attribution operates in two phases:

1. Real-time estimation: the gateway maintains a running token counter per request, incremented on each text_delta. It multiplies this by the model's per-token rate (from a pricing config table) and adds the prompt cost (known at request time). This running estimate is checked against the tenant's spend limit on every token. If the estimate exceeds 90% of the limit, the gateway sends a warning event to the client; at 100%, it cancels the stream and returns a spend_limit_exceeded error.
2. Final reconciliation: when the upstream stream closes, the provider sends a final usage event with the exact token counts. The gateway computes the true cost, writes it to the billing ledger (attributed to the tenant via the request context), and adjusts the tenant's spend counter. Any discrepancy between the estimate and the actual is reconciled here.

For the spend-limit overshoot problem: the gateway applies a safety margin (e.g., 95% of the limit triggers cutoff) so that even with estimation jitter, the actual cost rarely exceeds 100%. When it does (because the final usage event reveals more tokens than estimated), the overage is logged and the tenant's next request is rejected until the balance is settled. This is a deliberate trade-off: hard real-time enforcement would require provider-side pre-checks that don't exist in the streaming API, so a small, bounded overage is accepted.

I would expose the model through an HTTP/2 API gateway that delivers tokens via Server-Sent Events (SSE). SSE is unidirectional (server-to-client), which matches our use case since the prompt is sent once in the initial POST and only tokens flow back. Each token is framed as a data: line carrying a JSON object with fields like {"token": "...", "index": N, "request_id": "...", "finish_reason": null}, terminated by a data: [DONE] sentinel. SSE gives us automatic browser-level reconnection via the Last-Event-ID header, and HTTP/2 multiplexing avoids head-of-line blocking across concurrent streams. I would avoid WebSocket here because we don't need client-to-server frames after the initial request, and the extra connection-state management isn't justified.

On the server side, the API gateway forwards each request to a continuous-batching engine (like vLLM). The engine runs an iteration loop: each step, it selects a batch of active requests, runs a single forward pass, and produces one new token per request. Each active request holds a set of KV-cache blocks (managed via paged attention at the block level, not contiguous allocation). When a new request arrives, the scheduler allocates KV-cache blocks for its prefill, runs prefill (possibly batched with other prefills), then admits it to the decode batch. When a request finishes (EOS or max_tokens), its blocks are freed and it exits the batch. This means requests at different generation stages — some on token 5, some on token 200 — coexist in the same forward pass, and the batch composition changes every iteration. The gateway receives each token from the engine and pushes it into a bounded per-connection channel for that client's SSE stream.

Backpressure is the critical design point. Each client connection gets a bounded output buffer (a fixed-capacity queue, say 64 tokens). Under normal operation, the gateway writes tokens into this queue and the SSE writer drains it to the client. If a client reads slower than the GPU generates tokens, the queue fills up. When the queue is full, I suspend that request in the scheduler: the engine marks it as inactive (removes it from the decode batch for the next iteration) but retains its KV-cache blocks so it can resume without recompute. The request stays suspended until the client drains enough tokens to open space, at which point it's re-admitted to the batch. If a client remains suspended beyond a configurable deadline (e.g., 30 seconds), I terminate the stream entirely, free the KV-cache, and return an error — this prevents a permanently-stuck consumer from hoarding GPU memory. This design ensures a slow consumer only stalls its own generation; the GPU continues serving other requests in the batch at full throughput. The trade-off is that suspending a request wastes its KV-cache memory while idle, so I'd set the suspension capacity as a fraction of total cache and evict with recompute if under memory pressure.

On failure: if the connection drops mid-generation, the gateway detects the closed socket (either via a write failure or a keepalive timeout), immediately removes the request from the batching engine's active set, and frees its KV-cache blocks. Generation does NOT continue server-side — there's no point spending GPU cycles for a client that isn't there. The client observes an abrupt stream termination. Because the model is stateless across requests, there is no automatic server-side resume. The client must replay the request with the same prompt; to avoid redundant computation, the server's prefix cache (if enabled) will recognize the prompt prefix and skip re-prefilling the shared portion. The client should include a unique request_id; if it reconnects with SSE's Last-Event-ID carrying the last token index it received, the gateway can (best-effort, not guaranteed) resume from that point if the KV-cache hasn't been evicted — but this is an optimization, not a correctness guarantee. The guaranteed contract is: either the client receives a complete response (terminated by [DONE]) or it receives a partial response followed by an error/disconnect, in which case it must replay. Deduplication of tokens on replay is the client's responsibility using the token index field.

Architecture Overview

1. Iteration-Level Continuous Batching

Each GPU worker runs an iteration-level scheduler (as in vLLM/TGI). At every decode step (every ~20–40 ms), the scheduler:

• Evicts sequences that have emitted EOS or hit max tokens.
• Admits queued requests up to the limit imposed by available KV cache blocks.
• Issues a single fused forward pass for the entire active batch.

This contrasts with static batching, where a batch is formed at request time and all slots are occupied until the longest sequence finishes — leaving GPU capacity idle on short sequences. Continuous batching keeps the batch full at every step, maximizing the arithmetic intensity of the decode phase (which is memory-bandwidth bound at batch size 1 but improves as batch size grows because weight loads are amortized).

2. KV Cache Memory Management (PagedAttention)

The KV cache is the primary throughput constraint. Per-sequence KV cache size for this model:

• 80 layers × 2 (K+V) × 8 KV heads × 128 head_dim × 2,048 tokens × 2 bytes (FP16) = 80 × 2 × 8 × 128 × 2,048 × 2 ≈ 6.7 GB per sequence.

Across 5,000 concurrent sessions, the total KV cache demand is enormous (tens of TB), far exceeding GPU HBM. We use PagedAttention: the KV cache is divided into fixed-size blocks (e.g., 16 tokens/block). A block table maps each sequence's logical block indices to non-contiguous physical HBM blocks. This eliminates internal fragmentation (sequences only allocate the blocks they need) and enables prefix sharing: sequences with identical system prompts share the same physical KV cache blocks via copy-on-write reference counting. On 8× H100 nodes (640 GB HBM total), we can hold roughly 640 GB ÷ 6.7 GB ≈ 95 concurrent full-length sequences per node — so we need horizontal scaling across many nodes, each running replicas.

3. Prefill/Decode Disaggregation

Prefill is compute-bound (large GEMMs, high arithmetic intensity); decode is memory-bandwidth bound (small per-token GEMMs). Mixing a long prefill with active decodes in the same iteration stalls the decode batch: the GPU spends 200+ ms on the prefill while every active decode token waits, blowing the 50 ms inter-token SLO.

I disaggregate into two worker pools:

• Prefill workers (TP=8 within a node): handle prompt encoding, produce KV cache, then transfer it to decode workers.
• Decode workers (TP=8): run continuous batching over active sequences only.

Alternatively, chunked prefill splits a long prompt into chunks (e.g., 512 tokens) and interleaves each chunk as one "sequence" in the continuous batching loop, so no single prefill monopolizes an iteration. This is simpler operationally and I'd start with chunked prefill, moving to full disaggregation if TTFT SLOs are not met.

4. SLO-Aware Autoscaling and Admission Control

The admission controller estimates per-request TTFT (based on prompt length, current queue depth, and prefill worker utilization) before enqueuing. If the estimate exceeds 800 ms, the request is either queued (with a position estimate returned to the client) or rejected with an HTTP 429 + Retry-After.

Autoscaling is keyed on KV cache block utilization and estimated TTFT distribution, not raw GPU utilization. When KV cache utilization exceeds 80% or p90 TTFT exceeds 600 ms, the scaler provisions new replica groups. For bursty traffic, I maintain a warm pool of 2 idle replica groups (pre-loaded model weights, zero active sequences) that can absorb a 10× spike within seconds. Predictive scaling uses historical traffic patterns to pre-warm additional groups 15 minutes before expected spikes.

Graceful degradation under overload: shed load by (a) reducing max output tokens per request, (b) routing overflow traffic to a smaller fallback model (e.g., 13B), and (c) returning 429s with backoff headers.

5. Model Parallelism and Fault Tolerance

The 140 GB model requires TP=8 within a single H100 node (NVLink provides 900 GB/s inter-GPU bandwidth, minimizing all-reduce latency). With 8 nodes, we run 8 independent TP replicas, each serving ~95 concurrent sequences → ~760 concurrent sessions; further horizontal scaling (more nodes, more replicas) handles the 5,000-session target.

Pipeline parallelism (PP) is used only if a single replica cannot fit the model within one node's HBM — for a 70B model in FP16 (140 GB) across 8×80GB GPUs (640 GB), TP=8 within one node suffices. PP would be needed for larger models (e.g., 405B).

Fault tolerance: each TP replica group runs a health check (heartbeat + dummy inference every 5 seconds). On GPU failure within a TP group, the entire group is marked unhealthy; the load balancer stops routing to it, and in-flight requests on that group fail and are retried on a healthy replica (with the original prompt re-prefilled; KV cache for in-progress decodes is lost). A replacement group is launched from the warm pool. At 99.9% availability, we tolerate occasional request retries — the client SDK uses idempotent request IDs to handle duplicates.

Putting It Together

Client → API Gateway (rate limit, auth)
       → SLO-Aware Router (estimates TTFT, routes to replica)
       → Replica Group (TP=8, H100 node)
           → Chunked-Prefill Scheduler (interleaves prefill chunks with decode)
           → PagedAttention KV Cache Manager (block allocation, prefix sharing)
           → Continuous Batching Engine
       ← Streaming SSE response

Autoscaler watches KV cache util + TTFT p90 → scales warm pool