LLM Caching Interview Questions — AI Engineering Practice Quiz

Prompt caching keys on a prefix of the request, so a long, unchanging prefix (system instructions, few-shot examples, a fixed knowledge block) can be cached once and reused — cache reads are billed at a steep discount and skip recomputation, lowering latency. The catch: ordering matters. Anything that varies must come after the stable prefix, otherwise it breaks the cacheable prefix and you pay full price every call. Structure prompts stable-first, variable-last to maximize hits.

This is a foundational design question that tests whether the candidate understands the basic mechanics of an LLM response cache: interception before the LLM call with proper hit/miss behavior, key derivation from normalized input, bounding cache size with an eviction policy, and freshness via TTL or explicit invalidation. The original rubric paid twice for the same miss/hit design point (c1 and c3 overlapped); they are now merged into a single criterion, and the freed weight goes to cache eviction/bounding — a concrete mechanism a shallow answer would miss.

On the first call get_response("hello") the cache is empty, so the if prompt in cache condition is False. The function executes cache["hello"] = "response_for_" + "hello", storing "response_for_hello", and returns that string. The second call get_response("world") likewise misses the cache, stores "response_for_world", and returns it. The third call get_response("hello") now finds "hello" in cache, so it returns cache["hello"] + " (cached)", which is "response_for_hello" + " (cached)", yielding "response_for_hello (cached)". The three print statements each emit one line, producing the expected output.

Line 6 builds the cache key from hash(prompt) alone. Because model and temperature are not part of the key, a second call with the same prompt text but a different model or temperature hits the stale entry and returns the previously cached response instead of calling the LLM with the new configuration. The fix is to include all parameters that affect the output in the key, e.g. hash((prompt, model, temperature)).

Caching works on a prefix match: the provider stores the key/value attention state for an unchanged leading span and reuses it on subsequent calls, cutting both cost and time-to-first-token. The practical rule is order-dependent — keep the big invariant blob (system prompt, tool definitions, long context document, few-shot exemplars) at the start and anything that changes per request at the tail, otherwise the cache misses on every call.

Prompt caching stores the processed key/value state for a stable prefix (your unchanging system prompt + policy doc); on later requests the model reuses it instead of re-processing those tokens, which lowers input cost and time-to-first-token while leaving outputs identical — ideal when a large prefix is constant and only the tail varies. Changing temperature (b) affects randomness, not cost. Retrieving the system prompt via vector search (c) adds machinery and a retrieval step to fetch text you already have verbatim — pure overhead. Raising max_tokens (d) only sets an upper bound on output length; it does not reduce the cost of re-reading the same input every call and can increase cost if it lets responses grow.

Prompt caching reuses a saved KV state for the part of the prompt that didn't change. Placing static content first (a) and appending variable content after (c) is the structural requirement — the cache key is a prefix, so any change before the cached region breaks the hit. Cached tokens are processed at a fraction of the normal input-token cost and with lower latency (b) — that is the entire economic rationale for the feature. Long, stable content (e) — a 10,000-token system prompt reused across thousands of requests — is the highest-leverage use case because the savings multiply per request. Prompt caching does not touch output tokens (d): output is always freshly generated from the current conversation state; caching only affects input-prefix processing costs.

Optimising blindly wastes effort — profile first to know what is actually slow (TTFT implies compute bottleneck; generation length implies output tokens; high network overhead implies infrastructure). Then trim the prompt because every input token costs money and adds prefill latency. Enable caching on the stable prefix to amortise that cost across repeated calls — the savings are proportional to how much of the prompt is reused. Downsize the model only after the prompt is already lean; routing a bloated prompt to a cheaper model often produces lower quality than a tight prompt to a frontier model. Throughout, measure after each change so you attribute gains correctly and avoid stacking optimisations that cancel each other.

Prompt caching (available on Anthropic, OpenAI, and others) stores the KV-cache entries for a stable prompt prefix server-side. Requests that hit the cache skip the prefill computation for those tokens, cutting TTFT and charging a lower per-token price for cache hits. It is most valuable when the system prompt or retrieved context is large and reused across many requests — the typical pattern in RAG and agent loops with a fixed system prompt.

Two things trip loops up here. The first is a silent invalidator in the stable block — a rendered timestamp, a tool list built from an unordered map, a user id interpolated into the system prompt — which means the cache never hits and no error is raised. The second is the tension with context trimming: summarising the oldest turns is the single most destructive edit to a cached prefix, because it rewrites the front. Batch trims so one invalidation is amortised over many iterations, and check the usage numbers after any change to how the prompt is assembled.