AI Engineering Interview Questions: LLM Observability

Temperature scales the logits before the softmax in LLM decoding. A value of 0 makes the model deterministic (always pick the highest-probability token), while higher values flatten the distribution, increasing randomness and variability in the output. It does not control token limits, generation speed, or concurrency.

Hallucination is a foundational concept in AI production engineering. It refers to the model producing fluent but ungrounded or false output. Recognizing it is the first step toward deploying mitigation strategies such as RAG, fact-checking layers, and structured output validation.

Request-scoped logging is the default in most services and it is the wrong grain for an agent — it produces per-call records with no way to say which user action they belonged to. Almost every operational question about an agent is run-shaped: what did this run cost, why did it stop, which step went wrong, what did the model see at that point. All of them need the id, and threading it in later means backfilling every call site.

Chunking is the standard term for breaking long documents into smaller, manageable text segments (chunks) so that each piece fits within embedding model input limits and the LLM's context window. Overlapping chunks help preserve context continuity across boundaries. This is a foundational step in building RAG systems for production.

The same request can yield different outputs across time — sampling is non-deterministic, a model alias can be repointed to a new version, and retrieval depends on an index that changes — so the only way to replay a past response is to have logged the exact inputs that produced it: the resolved prompt (post-templating, post-retrieval), the model and version, the sampling params (temperature/top_p/seed), and the raw completion (b). Throughput (a) and IP/session data (c) are operational and UX metrics that never let you re-run a single call. Option (d) throws away the input — and the model is not stateless with respect to its prompt: the prompt is the input, so without it there is nothing to reproduce.

A trace in LLM observability follows the distributed-tracing model: the root span represents the full user request, and child spans represent each discrete step — retrieval, reranking, each LLM call, each tool invocation. Every span records its own start time, duration, inputs, outputs, token counts, and error state (b). This hierarchy is what flat logging cannot give you: with flat events you can total latency but not tell whether 2 seconds were spent in retrieval or in the LLM call. Structured traces let you immediately pinpoint which stage is slow, which step hallucinated, or where the pipeline branched unexpectedly. The trace is not a single log line (a), does not omit internal steps (c), and has nothing to do with deduplication (d).

Application logs capture what went in and came out, but quality regressions in LLM systems are rarely diagnosable from raw text alone. You need structured LLM observability: traced spans that record the full prompt, the retrieved chunks and their scores, the model version and parameters, token counts, and a per-turn cost breakdown — plus the ability to replay or diff traces across prompt versions. Without span-level tracing you cannot answer "which retrieval step returned bad context?", "did the model ignore the instruction?", or "did a model version change break this?" Tools like LangSmith, Phoenix, or OpenTelemetry LLM semantic conventions provide this layer.

Line 7 continues immediately with no delay, so all five attempts fire back-to-back in microseconds. On a 429 the provider is telling you to slow down; retrying instantly hammers the endpoint and can extend or worsen the rate-limit window — a self-inflicted thundering herd. The fix is exponential backoff with jitter (e.g. await sleep(base * 2 ** i + random)), and ideally honoring the Retry-After header the API returns. Tight retry loops without backoff are a frequent cause of cascading failures under load.

Cost attribution needs per-request token accounting: input vs output tokens (output is typically billed at a higher rate), the model, cached vs uncached input tokens, and the derived dollar cost. Tag each record with feature/endpoint, user/tenant, prompt version, trace id, and latency so you can aggregate and slice. Those slices surface the expensive paths — an over-stuffed retrieval context, an agent that loops and re-invokes the model, a feature using an overpowered model, or a workload that should fall back to something cheaper. Without input/output separation and per-dimension tagging, a rising bill is a mystery; with them it's a query.

Traces decompose a request into spans, and each span must capture inputs, outputs, timing, and token counts to be actionable. Without structured tracing, post-mortem debugging of LLM failures is guesswork — you can't reconstruct the exact prompt the model saw.

Evaluation starts by defining what good means (criteria/metrics), then assembling a representative dataset of inputs with expected outcomes. You score candidate variants against it, compare and select the winner, and finally monitor in production, routing real-world failures back into the dataset so the eval set hardens over time. Skipping the dataset step is the classic mistake — you can't iterate on quality you don't measure.

Borrowed from distributed systems (OpenTelemetry), a trace captures the full lifecycle of a request, while nested spans represent individual operations — an LLM inference call, a vector search, a tool execution. LLM observability tools (LangSmith, Langfuse, Phoenix, etc.) attach token counts, latency, and model parameters to each span, enabling you to pinpoint which stage is slow or expensive without guessing.

When the distribution of input features in production shifts away from the training distribution — without the labels necessarily being available yet — this is called data drift (also covariate drift or input drift). Common detection techniques compare the statistical distribution of a baseline (training) sample against a recent live sample: the Kolmogorov-Smirnov (KS) test for continuous features, the Population Stability Index (PSI), and information-theoretic measures such as KL divergence, Jensen-Shannon divergence, or the Wasserstein distance. The answer must name the phenomenon and at least two of these methods.

Cost is proportional to total tokens (input + output); the observability data shows input tokens spiked 4×, not output tokens. In a RAG pipeline, input tokens are dominated by the retrieved context chunks stuffed into the prompt. A 4× jump in input tokens with no code change almost always means retrieval is now returning many more chunks — a shifted similarity threshold (e.g. a higher top_k or a lower score cutoff), a reindexing that changed result counts, or a misconfigured reranker (b). Provider price changes (a) would not change token counts. A missing max_tokens cap (c) affects output tokens, not input. User query length (d) has a negligible effect on total input tokens when the bulk of the prompt is retrieved context.

User feedback is a leading indicator, not ground truth. The 60% false-negative rate on flagged traces shows that users are clicking thumbs-down for reasons other than factual incorrectness — unhelpful tone, unexpected format, a correct answer that contradicts what the user believed, or accidental clicks. Taking all thumbs-down responses as negative training examples (a) would introduce 60% clean examples as false negatives, directly degrading the fine-tuning signal. Acting on thumbs-downs without a human review step (c) has the same flaw and ignores the 40% that are real errors. Industry benchmarks (d) are irrelevant here — the finding is about signal quality, not the rate itself. The right action is to put flagged traces through a human (or LLM-judge) review layer that separates actual quality failures from user-preference mismatches before using the data for evaluation or training (b).

Useful LLM telemetry is the data that explains quality and cost. TTFT-vs-total latency (a) separates a slow-to-start model from a long generation. Token counts (b) are the unit of both cost and context pressure. The agent's tool steps and the exact retrieved chunks (c) are what you replay to debug a wrong answer in a RAG/agent flow. The finish reason (d) distinguishes a clean completion from a truncated or filtered one — often the root cause of a 'cut off' answer. The GPU temperature and scheduling (e) belong to the provider's hardware that you neither see nor control on a hosted API, and tell you nothing about output quality or your bill.

A trace ID that spans every pipeline step (a) is the foundation — without it, individual span logs are islands you cannot correlate. Logging the exact injected prompt and raw completion (b) is what lets you replay and debug a specific bad output; PII redaction is the compliance control, not a reason to omit the log. Per-span latency and token counts (c) let you identify whether the bottleneck is retrieval, the reranker, or generation. Version metadata on spans (e) is critical for before/after comparisons when a prompt template or model changes. Sampling 100% in all environments (d) is not a best practice — it is prohibitively expensive at production scale; the standard approach is high sampling in dev/staging and tail-based or probabilistic sampling in production, with 100% capture for errors.

One agent request is really a tree of LLM calls, tool invocations, and retrievals, so request-level logging tells you it failed but not where. Trace/span instrumentation models the run as a top-level trace with nested spans per step, capturing each prompt+completion, tool call arguments and results, retrieval queries and returned passages, plus per-step token counts, latency, and cost. That granularity is exactly what agent debugging requires — you can inspect intermediate outputs, localise the failing step (bad tool call, empty retrieval, a loop), and attribute latency/cost. The same traces feed eval datasets and regression detection across prompt/model changes.

Full eval coverage on production traffic is cost-prohibitive, so effective monitoring stacks cheap always-on heuristics, sampled judge scoring, and implicit user feedback — each catching different failure modes at different fidelities.

Observability is layered bottom-up: first instrument individual LLM calls to capture the raw signal (prompt, completion, latency, token counts); then propagate a trace-id so multi-step agent calls form a coherent trace rather than isolated records; next export to a backend that can store and query spans at scale; then surface dashboards and alerts for operationally critical metrics; finally configure sampling because capturing every token of every call at volume is prohibitively expensive — sampling keeps costs manageable while preserving statistical coverage of failures.

A repeated call with identical arguments and an identical result adds no information, so the next iteration faces the same context and makes the same decision — a fixed point the model cannot escape by itself. Hashing the call signature detects exactly that: same tool, same arguments, nothing new in between. What you do on detection is a design choice (break with a partial answer, inject a nudge that the tool has been tried, escalate to a human), but you cannot act until you detect. Temperature 0 (b) makes the loop more deterministic and so more likely to repeat — a genuinely tempting wrong answer, because low temperature is the right default for most tool-calling. Raising the step cap (c) buys more identical iterations at linear cost. A bigger window (d) misdiagnoses the problem: the observations are already visible; that is why the model keeps concluding the same thing.

The four that belong are the ones that answer the questions you will actually have. Calls and arguments (a) show what the model decided; results (b) show what it learned, truncated because one large payload should not blow up your trace store; token counts (c) turn 'this run was expensive' into a specific iteration; and a shared run id (d) is what makes the spans a run rather than a pile of unrelated events, and what lets you get from a support ticket to the trace. Option (e) is where teams get into trouble: prompts contain whatever the run touched — customer records, documents, credentials pasted into a chat — so indefinite full-fidelity retention turns your observability stack into an unclassified copy of production data with its own retention and access-control obligations. Store a hash or a truncated rendering by default, keep full prompts behind a short window and explicit access, and redact at capture rather than intending to clean up later.

The instinct to reproduce a past run is the wrong reflex, and the question is designed to catch it. Four independent things drift — provider-side sampling non-determinism, model updates, your own deploys, and the underlying data the tools read — so a re-run months or even minutes later is evidence about today, not about Tuesday. Anything that matters after the fact has to have been captured at the time. The model version belongs in the trace for exactly this reason, and it is the field teams most often omit until an incident makes them wish they had it. There is a compliance edge worth knowing too: recorded prompts and results contain whatever the run touched, so this trace is production data with retention and access obligations, not just debugging output.

Triage moves from the coarse signal to the atomic cause: isolate the cohort (time window, model version, experiment arm) from traces so you are not analysing noise; find the stage where the failure is concentrated — retrieval misses and generation errors need different fixes; classify the failure mode by sampling actual bad completions, because the same symptom can have multiple causes; reproduce locally to confirm the root cause in a controlled environment before writing any fix; and verify the fix in production by confirming the alert clears — a fix that only passes local evals but leaves the alert firing means something else is wrong.