AI Engineering Interview Questions: Model Routing in Production

Model routing (a cascade) matches each request to the cheapest model that can handle it: a fast classifier sends the easy 90% to a small model and escalates only the hard 10% to the expensive one, so you pay the premium where it actually buys quality (d). Small-model-for-everything (a) sacrifices the 10% that needed the big model. Capping max_tokens (b) just truncates outputs — it hurts exactly the open-ended cases that need room. Caching (c) only helps repeated inputs and does nothing for the volume of distinct queries.

Model routing is the application-layer decision of which model answers a request, distinct from picking a model once at build time. It matters because LLM pricing and latency vary by 10-100x across tiers (a mini/flash model versus a frontier model), and most production traffic isn't uniformly hard — an FAQ lookup and a multi-step reasoning task don't need the same model. Routing is what lets a team serve both cheaply and reliably at scale instead of over-paying for every call or under-serving the hard ones.

A classifier router inspects the incoming request up front — prompt length, a lightweight classifier's difficulty score, task type — and sends it to exactly one model: a single round trip either way, but a misclassified hard request has no automatic recovery unless a separate fallback is layered on top. A cascade instead always pays for the cheap tier's call first, evaluates that actual output against a quality or confidence threshold, and only escalates to the expensive model when the cheap tier's real response falls short — grounding the decision in observed evidence rather than a prediction, at the cost of a second, sequential round trip on the requests that do escalate. Neither pattern fans requests out to every model in parallel (b), provider hosting model is unrelated to which pattern you use (c), and neither requires a human in the loop (d) — cascades and routers are both automated decision points.

OpenRouter's documented default behavior does exactly this: it first prioritizes providers that have not seen a significant outage in the last 30 seconds, then among the remaining stable candidates it weights selection by the inverse square of price — so, in OpenRouter's own example, a provider at $1/million tokens is roughly 9x more likely to be chosen than one at $3/million tokens, not split 50/50. This price-weighted default only applies when you leave routing unconfigured; explicitly setting a fixed provider order or a sort preference (e.g. for throughput) turns the automatic load balancing off and tries providers in the order/priority you specified instead.

Model routing decisions don't exist in isolation from caching — a per-call-optimal choice can be a per-conversation loss. The interview signal here is recognizing that prompt/prefix caching is provider- and model-scoped, so hopping providers turn-by-turn resets it every time, and that the fix (sticky routing / session affinity) is a standard pattern in LLM gateways for exactly this reason.

A cascade always pays for the cheap tier first rather than predicting difficulty up front. It then scores that real response against a quality/confidence threshold — a validator, a judge, self-consistency, or the model's own confidence — and if the score falls short, the identical request is escalated to the next, more capable and more expensive tier, whose response is what actually gets returned. Grounding the escalation decision in the cheap tier's observed output, rather than a prediction made before calling anything, is what distinguishes a cascade from a single-shot classifier router.

Effective routing reduces cost and latency without sacrificing quality. Complexity-based routing (a) is the foundational pattern: easy queries (FAQ, classification, extraction) rarely need the largest model. Speculative / cascade routing (b) — try small first, escalate if needed — is a principled way to get quality guarantees while paying small-model prices on the majority of traffic. Capability-based routing (d) sends work to the model best fit for the task domain. A fallback chain (e) is essential production hygiene: no single provider has 100% uptime. Fan-out to all models on every request (c) is not a routing strategy — it multiplies cost by the number of models and is only justified for evaluations or tie-breaking in offline settings, not latency-sensitive production traffic.

Line 7's bare catch { swallows every error indiscriminately and falls through to the cheaper model. So a non-transient failure — a malformed request, an auth error, a content-policy block, or a bug in your own code — is silently downgraded to a weaker model (and reported only as the generic 'all models failed' if the whole chain is exhausted) instead of being surfaced. Worse, because the bare catch discards the error object, the code cannot implement the very policy it was written for: 'fall back only on 429/503.' The fix is to inspect the error and fall back only on transient/overload statuses, rethrowing everything else: catch (err) { if (isRetryable(err)) continue; throw err; }.

Model routing matches model to request difficulty instead of paying frontier prices for trivial work. Cheap/fast models handle the easy, high-volume majority (classification, extraction, formatting); the large model is reserved for hard or high-stakes calls. Implement it with rules (task type, length, calling feature), a lightweight router/classifier, or a cascade that starts small and escalates on a failed confidence/validation check. Average cost and latency drop because most traffic is easy. The guardrail is quality: the small model can silently regress on the requests it now handles, so you need per-route evals, monitoring of the escalation/fallback rate, and a tuned threshold — not a blanket assumption that the cheaper model suffices.

Line 6 always calls PROVIDERS[0]. The loop varies attempt, and attempt is used correctly on line 9 to grow the backoff delay, but it is never used to select a different entry from PROVIDERS. So if primary-a is down, all six attempts retry the exact same dead provider with growing delays; primary-b and backup-c are configured but never called once, and the whole request fails even though healthy backups exist. Retry-with-backoff (line 9) is doing its job for a transient blip on one provider, but it is not a substitute for actually walking the fallback list. The fix is to index into PROVIDERS by attempt (e.g. PROVIDERS[attempt % PROVIDERS.length]), or, more cleanly, to separate the two concerns: retry the current provider a bounded number of times, then advance to the next provider in the chain.