AI Engineering Interview Questions: Latency & Cost in Production

LLM APIs charge per token (input + output) and latency scales with the number of tokens generated. Capping max_tokens directly limits how many output tokens the model can produce, which simultaneously lowers the per-call cost and reduces time-to-last-token. Larger models increase both cost and latency, temperature does not affect token count or pricing, and retry logic only adds overhead when errors occur.

Prompt or semantic caching stores the result of a previously seen query and returns it immediately on a cache hit, bypassing the LLM API entirely. This eliminates the token cost for repeated queries and returns the response in milliseconds instead of seconds, directly improving both dimensions: cost and latency.

TTFT stands for Time To First Token — it measures the latency from sending a request until the first token of the response is produced, which captures perceived responsiveness in streaming chat interfaces. TPOT stands for Time Per Output Token — it measures the average time taken to generate each subsequent output token after the first one, capturing steady-state generation throughput. Together, TTFT and TPOT are the two standard, unambiguous latency metrics used across LLM inference benchmarking (e.g., NVIDIA, vLLM, LLMPerf).

Caching identical prompts returns pre-computed responses instantly, eliminating model inference entirely and thus reducing both latency and cost. Increasing model size raises both latency and cost. Streaming improves perceived latency but does not reduce total compute cost. Logging adds negligible overhead but doesn't reduce latency or cost on its own.

Token count is the primary driver of both input cost (charged per input token) and output cost (charged per output token) in LLM APIs. Prompt engineering to reduce unnecessary context, truncating conversation history, and compressing system prompts all reduce token usage. Request timestamp and the user's geographic region are not factors in token-based billing. Response streaming affects perceived latency but does not reduce the number of tokens generated or their cost.

Time To First Token (TTFT) measures the delay from sending a request until the very first token of the response is received by the client. It is distinct from total response time (which covers all tokens) and throughput (tokens per second). TTFT is especially important in streaming UX because users see the response start arriving sooner even if total generation time is the same. It is NOT the same as total latency.

The string is 49 characters; ceil(49 / 4) = ceil(12.25) = 13. The ~4 chars/token rule is a quick estimate for budgeting context windows and cost, not an exact count — real tokenizers (BPE) split on sub-word units, so whitespace, punctuation, code, and non-English text all change the ratio. For anything that must be exact (e.g. enforcing a hard context limit) you should call the model's real tokenizer rather than rely on this heuristic.

Streaming is purely a latency-perception win: tokens are flushed incrementally so the user gets immediate feedback (low time-to-first-token), which matters a lot for long generations. It does not reduce total generation time, total tokens, or cost — those are fixed by the output length. It does add engineering cost: you must handle partial output, parse incrementally, and deal with mid-stream errors and cancellation.

Streaming changes only the delivery pattern — how and when tokens reach the client — not how many tokens the model generates. The same output is produced and billed either way; streaming's actual benefit is perceived responsiveness (a much lower time-to-first-token, so the user sees output start almost immediately instead of staring at a blank screen for the full generation time), not a reduction in total latency, token count, or price.

Model routing (a cascade) matches each request to the cheapest model that can handle it: a fast classifier sends the easy 90% to a small model and escalates only the hard 10% to the expensive one, so you pay the premium where it actually buys quality (d). Small-model-for-everything (a) sacrifices the 10% that needed the big model. Capping max_tokens (b) just truncates outputs — it hurts exactly the open-ended cases that need room. Caching (c) only helps repeated inputs and does nothing for the volume of distinct queries.

Streaming (a) doesn't change total compute but dramatically improves perceived speed by showing the first tokens immediately. Model routing / cascading (b) sends the bulk of easy traffic to a cheaper model, cutting average cost and latency while preserving quality on the hard tail. Caching (c) avoids paying for work you've already done. Trimming context and bounding output length (e) reduces both input and output tokens, which is where the bill and the time go. Indiscriminately padding every prompt with more few-shot examples (d) does the opposite — it inflates input tokens (cost and latency) on every call, and beyond a point adds no accuracy, so it is a regression, not an optimization.

Summarising history (a) directly shrinks input tokens for every subsequent turn in a long conversation. Setting a task-appropriate max_tokens cap (b) prevents the model from generating a 2,000-token essay when you need a 50-token answer — the difference is billed output tokens. Structured output (c) is a meaningful lever: a JSON object with three fields uses far fewer output tokens than the same information in a conversational paragraph, and it also eliminates post-processing. Limiting retrieved passages to the genuinely useful few (e) reduces input tokens proportionally to what you cut. Switching to a larger-context model (d) does not reduce token spend — it only raises the ceiling on how many tokens you can use; without actively trimming the prompt, it is neutral at best.

Input tokens and output tokens are priced and computed differently. Output tokens are generated auto-regressively — one forward pass per token — so generating 900 output tokens requires 900 sequential decoding steps, which dominates both latency and compute cost. Input tokens are processed in parallel in a single prefill pass that is far cheaper per token. Most providers price output tokens at 3–5× the input rate to reflect this. In latency-sensitive applications, techniques like constrained decoding, structured output (to avoid verbose free-text), and speculative decoding target output-token efficiency precisely because that is where most of the cost and time lives.

Line 7 continues immediately with no delay, so all five attempts fire back-to-back in microseconds. On a 429 the provider is telling you to slow down; retrying instantly hammers the endpoint and can extend or worsen the rate-limit window — a self-inflicted thundering herd. The fix is exponential backoff with jitter (e.g. await sleep(base * 2 ** i + random)), and ideally honoring the Retry-After header the API returns. Tight retry loops without backoff are a frequent cause of cascading failures under load.

Caching works on a prefix match: the provider stores the key/value attention state for an unchanged leading span and reuses it on subsequent calls, cutting both cost and time-to-first-token. The practical rule is order-dependent — keep the big invariant blob (system prompt, tool definitions, long context document, few-shot exemplars) at the start and anything that changes per request at the tail, otherwise the cache misses on every call.

TTFT and TBT measure different bottlenecks: prefill vs decode. Interactive UIs are TTFT-sensitive (perceived snappiness); batch pipelines care about throughput. Knowing which metric to target determines whether you shorten the prompt or reduce output length.

Trace the four iterations: the loop bills 1000, then 1500, then 2000, then 2500, summing to 7000, while context ends at 1000 + 4·500 = 3000. The point is the gap between those two numbers. The transcript grew to 3000 tokens, but the run was billed for 7000 — more than twice the final size — because every earlier token is paid for again on every later step. Option (b) is what you get by counting only the tokens each step adds, which is the intuition most people carry over from single-call pricing and the reason agent bills surprise teams. Option (c) mistakes the final context size for the amount billed. Extend the loop to 10 steps and the sum reaches 32,500 against a 6000-token transcript: the growth is quadratic, which is why trimming what you carry forward is the strongest cost lever in an agent loop.

Line 7's bare catch { swallows every error indiscriminately and falls through to the cheaper model. So a non-transient failure — a malformed request, an auth error, a content-policy block, or a bug in your own code — is silently downgraded to a weaker model (and reported only as the generic 'all models failed' if the whole chain is exhausted) instead of being surfaced. Worse, because the bare catch discards the error object, the code cannot implement the very policy it was written for: 'fall back only on 429/503.' The fix is to inspect the error and fall back only on transient/overload statuses, rethrowing everything else: catch (err) { if (isRetryable(err)) continue; throw err; }.

Prompt caching reuses computed KV state across calls. The key discipline is prefix stability: put everything that doesn't change early and everything that varies late. Inserting dynamic content early is the most common reason caches miss.

Streaming improves perceived latency: the client opens a stream, the model prefills and emits the first token (TTFT — the metric users feel most), the server relays token deltas (commonly over SSE), the client appends and renders them incrementally, and the stream closes with a finish event carrying the stop reason and token usage. Note that total generation time is unchanged — streaming just surfaces tokens as they're produced instead of waiting for the whole completion.

Observability is layered bottom-up: first instrument individual LLM calls to capture the raw signal (prompt, completion, latency, token counts); then propagate a trace-id so multi-step agent calls form a coherent trace rather than isolated records; next export to a backend that can store and query spans at scale; then surface dashboards and alerts for operationally critical metrics; finally configure sampling because capturing every token of every call at volume is prohibitively expensive — sampling keeps costs manageable while preserving statistical coverage of failures.

TTFT (time-to-first-token) is the latency a user feels before anything appears on screen — driven by prompt-processing (prefill) time, which scales with input token count. TPS (tokens per second) governs how fast the stream fills in afterward — driven by the decode phase and is roughly constant per model/hardware configuration. Optimising for TTFT often means shorter prompts or prompt caching; optimising TPS means smaller models or speculative decoding.

Paged attention's block table design deliberately makes KV pages non-contiguous, so the classic OS external-fragmentation narrative (needing a large contiguous region) does not apply to vLLM's allocator. The real mechanisms behind high TTFT despite apparent available memory are: (a) internal fragmentation — each sequence wastes slots in its last partially filled block, inflating allocated-byte counts without providing usable capacity; and (b) vLLM's watermark-based scheduler policy, which queues or preempts requests when the free-block count drops below a threshold, even if raw byte utilization looks low. Effective mitigations therefore target internal fragmentation (smaller block size) or the admission policy (watermark tuning), both of which are orthogonal levers available at the serving layer without model changes.

The correct sequence is: (1) Quantize the model weights first (AWQ INT4) because all downstream hardware and serving decisions depend on the resulting memory footprint and accuracy profile. (2) Choose the GPU SKU and tensor-parallel degree once you know how large the quantized checkpoint is—this determines per-device memory and compute budget. (3) Configure the serving engine's continuous batching and batch-token limits against the now-fixed hardware topology. (4) Add semantic routing to split traffic between the large model and a smaller distilled model; routing policies build on a stable serving stack. (5) Finally enable prefix caching, which requires knowing the routing strategy to predict which prefix groups will recur frequently enough to be worth caching.