DevOps Observability Interview Questions: Logs, Metrics, and Traces

A correlation ID is a unique identifier minted at the edge of a request (an API gateway or the first service it hits) and threaded through every downstream call, usually as a header (e.g. X-Correlation-ID, or the trace ID a distributed-tracing system generates, which serves the same purpose). Every service that touches the request logs that ID alongside its own log lines, so a single search or filter on the ID reconstructs the request's full path through the system regardless of how many services it crossed. Searching each service's logs by timestamp alone is unreliable — under real concurrent traffic, many unrelated requests log in the same millisecond, and clock skew across hosts makes timestamp correlation worse, not better. Relying on each service's own locally-generated request ID gets the mechanism backwards: a request ID generated independently inside each service is, by definition, a different value per hop unless it is explicitly passed in and reused — that's exactly the bug the correlation-ID pattern exists to prevent. The claim that there is no standard mechanism for this is false; this is a solved, standard practice, and its absence is a common root cause of 'we can't tell what happened' during an incident.

The three pillars are logs, metrics, and traces, and each answers a different question. Metrics are cheap aggregate time-series — great for 'is latency up?' and dashboards/alerts, but they aggregate away the per-request path, so they tell you that something is slow, not where in a multi-service call. Logs are discrete events rich in detail, but on their own they are per-service and don't automatically reconstruct one request's journey across services. Distributed tracing is purpose-built for this: a trace ties together the spans of a single request as it crosses service boundaries, each span timed, so you can read off exactly which downstream hop dominates the latency. The claim that a single latency gauge tells you exactly which service is slow overstates what a single metric can do; the claim that grepping each service's log lines reconstructs the full causal request path automatically assumes logs auto-correlate across services (they don't without a shared trace/correlation id); the claim that alerting is the fourth pillar and pinpoints the slow hop directly is wrong — alerting is built on metrics/logs, it is not a fourth pillar and does not localize a slow span.

Line B is structured logging: a machine-readable object with named, typed fields. A log platform parses it into indexed fields (user_id as a number, amount as a number), so a query like event:checkout AND amount > 100 is exact and survives wording changes. Line A is a human sentence — to extract the amount you'd need a fragile regex that breaks the moment someone changes 'checked out for $' to 'purchased', drops the dollar sign, or localizes the message, and the value arrives as text, not a comparable number. That brittleness is exactly why production systems standardize on structured logs with consistent field names (and a correlation/trace id) across services. The claim that free-text logs are easier for machines to query has it backwards; prose is easy for humans, hard for machines. The claim that structure makes no difference is wrong because structure is precisely what makes reliable field queries possible. The claim that numeric thresholds can never be evaluated from logs is false — you absolutely can threshold over a parsed numeric log field; metrics are often derived from such logs, but the log field itself is queryable.

The Google SRE book draws this distinction precisely: an SLI is a quantitative measure of some aspect of service behavior, defined from the user's perspective — request success rate, a latency percentile, throughput, freshness, or durability are all valid SLIs. An SLO is a target value or range set on top of an SLI over a defined compliance window (e.g. 99.9% success over 28 days), and it is what turns a raw measurement into an accountability contract between the teams building and operating the service. The claim that an SLI is a target set by the business and an SLO is whatever the monitoring system happens to measure inverts the relationship — the SLI is the measurement, not a business-set target, and the SLO is the deliberately chosen target, not an incidental readout. The claim that SLI and SLO are interchangeable terms for the same threshold collapses a meaningful distinction; they are different objects (measurement vs. target on that measurement), not synonyms. The claim that an SLI is always an availability percentage and an SLO is always a latency number is a common misconception: neither term is tied to a fixed dimension — an SLI can just as easily be a latency percentile as an availability ratio, and an SLO can be set on any of them.

False. Per the Google SRE book, an SLI is any carefully defined quantitative measure of some aspect of the level of service, and the canonical examples explicitly include request latency (commonly a percentile like p95 or p99), throughput, error rate, system durability, and correctness — availability is only one of several common choices, not a definitional limit. A latency SLI ('95th percentile response time under 300ms') and a throughput SLI ('requests served per second above X') are both completely standard. Teams choose the SLI that best represents what users actually experience for a given service — a batch pipeline might care about freshness or correctness, an interactive API about latency, a payments service about both availability and correctness — so treating 'up vs. down' as the only valid shape for an SLI would rule out most of the SLIs real SRE teams actually track.

Metrics are pre-aggregated numeric time series bucketed at fixed intervals, so their storage grows with the number of buckets over time, not with raw request volume, and evaluating a threshold against a rolled-up counter is cheap even a year later — which is precisely why alerting rules and long-retention trend dashboards are built on metrics. The description of logs is the anti-pattern: logs are high-volume, often high-cardinality, and expensive to retain in full and re-scan on every evaluation; using raw logs as the alerting primary doesn't scale and is why metrics exist as a separate, cheaper signal. Traces are wrong for two reasons — traces are typically sampled (often well under 100%) specifically because storing every one is too costly, so a sampled set gives an estimate, not the exact rate, and traces are optimized for explaining one request's latency/causality, not aggregating a year of trend data. The claim that none of the three work is false: metrics, via Prometheus, a hosted time-series database, or an APM platform's own metric pipeline, are the standard mechanism for exactly this use case.

OpenTelemetry (a CNCF project) provides one vendor-neutral API/SDK surface across traces, metrics, and logs, so instrumentation is written once against the OpenTelemetry API rather than against a specific vendor's proprietary client. The mechanism that turns independently-created spans on separate services into a single coherent trace is context propagation: the active span's context is serialized into a standard header on outgoing calls and extracted on the receiving side to seed a child span with the correct parent — this is exactly what the W3C Trace Context standard formalizes, and it's the core concept enabling distributed tracing at all. Because the instrumentation API is decoupled from the destination, switching backends is normally an exporter/collector configuration change, not an application rewrite — this is the practical payoff of vendor neutrality and the main reason teams adopt it. The claim that OpenTelemetry itself replaces the need for an observability backend is false: OpenTelemetry is the instrumentation, collection, and export layer; it deliberately does not include a storage/visualization backend, so you still need something like Prometheus, Jaeger, Tempo, or a commercial APM to store and view the data. The claim that OpenTelemetry only covers distributed tracing is false: traces, metrics, and logs are all first-class OpenTelemetry signals, not an out-of-scope extra.

Alert fatigue is precisely the desensitization pattern described: once responders are paged too often for things that turn out not to matter, they start treating every page — including the ones that do matter — as noise to dismiss, which is how real incidents get missed. The two standard fixes: alert on symptoms tied to what users actually experience (error rate, latency, SLO burn rate) rather than on every internal resource metric that merely correlates with a problem — a service can run hot on CPU with zero user impact, or fail users while every resource metric looks nominal, so causes alone are poor alerting signal; and hold every alert to the bar of 'what does a human do when this fires?' — if nothing, or if the response is fully automatable, it shouldn't be paging anyone. The claim that the fix is to lower every alert's threshold so issues are caught earlier is the opposite of a fix: lowering thresholds increases alert volume and false-positive rate, which is the direct driver of fatigue, not a cure for it. The claim that muting the noisy alert channel is an acceptable long-term fix treats a symptom-suppressing workaround as a solution — muting a noisy channel just hides the signal-to-noise problem instead of fixing it, and it reintroduces the exact risk of missing a real incident that alert fatigue already causes.

The interview signal here is whether the candidate understands OpenTelemetry as an instrumentation-layer standard, not a product. Before it existed, every vendor's proprietary SDK created switching costs proportional to how much code touched it, since instrumentation is spread throughout an application. OpenTelemetry's contribution is standardizing the API/SDK surface across all three signals (traces, metrics, logs) so instrumentation code becomes vendor-agnostic, and pushing the vendor-specific decision down to a swappable exporter/collector configuration. A strong answer names the lock-in problem being solved, states that OTel covers traces + metrics + logs (not tracing alone), and explains the practical consequence: changing backends becomes a configuration change rather than a re-instrumentation project. Candidates who only say 'it's for tracing' or 'it collects data' are missing the vendor-neutrality point the question is actually probing.

This is foundational SRE vocabulary, and interviewers listen for the stack order rather than just the definitions in isolation: measure (SLI) → set a target on the measurement (SLO) → derive operational slack from the target (error budget). A strong answer also grounds it with a concrete numeric example (a 99.9% SLO on 1,000,000 requests over 28 days permits ~1,000 failures) and names the operational consequence of the budget — it isn't just an accounting artifact, it's the number that decides whether the team ships fast or slows down to firefight. Candidates who define all three correctly but can't say how spending/exhausting the budget changes team behavior are missing the practical half of the concept.

This is the context-propagation mechanism that OpenTelemetry describes as the core concept enabling distributed tracing at all. It starts when a request arrives with no existing trace context, so the first service acts as the entry point: it mints a new trace ID and opens the root span. Before that service calls anything downstream, it serializes its current span's identifiers into the outgoing request — standardized today as the W3C Trace Context 'traceparent' header — so the propagation format is consistent across languages and vendors. The receiving service extracts that header, and instead of starting an unrelated new trace, it opens a child span that records the caller's span ID as its parent, which is what stitches the two services' work into one causal chain (this inject/extract pair repeats at every hop in a multi-service call). Each service exports its own finished spans, independently and typically batched, to a collector or backend — export doesn't wait for the whole request to finish. Only at the backend does assembly happen: every span sharing the trace ID gets grouped and ordered by its parent/child links into the waterfall view engineers actually read to see which hop dominated the latency. Getting inject/extract in the wrong order relative to the call, or forgetting that assembly is a backend-side step rather than something each service does locally, are the common misconceptions this ordering surfaces.

A Prometheus time series is identified by its metric name plus its full set of label key/value pairs; every distinct combination that has ever been observed becomes its own series, held in memory (and on disk) for as long as it's retained. Bounded labels like method or status_code take a handful of values, so they multiply the series count by a small, known factor. user_id is effectively unbounded — one new value per user, potentially millions — so attaching it to a metric multiplies the series count by that unbounded factor, which is exactly the 'cardinality explosion' that exhausts memory and slows every query that has to scan more series. The fix is architectural: keep metric labels to bounded, low-cardinality dimensions, and route anything you need to look up per-entity (a specific user, request, or session) to logs or traces, which are indexed for point lookups over high-cardinality fields. The fixed-size label table claim misdescribes the mechanism — memory cost scales with distinct label values, not with label count alone. The claim that it is self-resolving is false; Prometheus doesn't quietly evict live series to relieve memory pressure the way that claim implies — teams that hit the ceiling end up dropping metrics or shortening retention themselves, losing exactly the data they need. Blaming the Counter instead of the label is a distractor: Counter vs. Gauge is unrelated to cardinality.

Head-based sampling makes its keep/drop decision early — typically a consistent hash of the trace ID compared against a target rate, decided at or near the root span, before most of the downstream calls have even happened. Because that decision is made before the request has run its course, it cannot be conditioned on an outcome (like an eventual error) that hasn't happened yet. Tail-based sampling defers the decision until the full trace has been collected — its spans buffered, typically at a collector — which lets it apply outcome-aware policies such as 'always retain traces that contain an error, sample the rest at 1%.' The cost is real: tail-based sampling requires buffering infrastructure and adds latency before spans can be exported, since the collector must wait to see the whole trace. The claim that head-based sampling can conditionally keep only error traces is wrong by definition — head-based sampling can't conditionally keep only error traces, because the decision point predates knowledge of the outcome. The claim that the two approaches are the same algorithm under different names is false; the two approaches differ specifically in when the decision is made, which is the whole reason they support different guarantees. The claim that neither approach can retain 100% of error traces while sampling successes at a lower rate is false — combining a floor rate for successes with guaranteed retention for errors is precisely the standard tail-based sampling use case.

False — correlating a log line with the active trace requires the OpenTelemetry logging signal or bridge to read the active span's context and inject trace_id/span_id into the emitted record; it doesn't happen just because tracing and logging code run in the same process. In practice this is often lightweight — the .NET SDK enables logs-to-activity correlation with no extra user code once OpenTelemetry logging is wired up, and Python needs an environment variable (OTEL_PYTHON_LOG_CORRELATION=true) plus adding %(otelTraceID)s/%(otelSpanID)s to the log formatter — but it is always a deliberate integration step. A bare console.log/print call bypasses that pipeline entirely and emits an ordinary line with no trace fields, regardless of which span happens to be active at that moment. This is exactly why 'log-trace correlation' is documented as its own setup step by every OpenTelemetry language SDK and observability vendor, rather than being an automatic side effect of co-located code.

This question separates candidates who've only used a metrics dashboard from those who understand how a time-series database actually stores data. The interview point is the mechanism: a metric's identity is its name plus its full label set, so cardinality is combinatorial — each additional label multiplies the series count by the number of distinct values that label can take, not adds to it. A strong answer names the mechanism (unique label-value combination = new series = new memory/storage cost), gives the canonical bad examples (user_id, request_id, session_id, raw/unnormalized URLs), contrasts them with safe bounded labels (status_code, method, region), and states the production consequence (memory exhaustion, storage growth, slow queries, and the operational trap of teams dropping retention right when they need the history most). The fix — keep unbounded fields out of metric labels and use logs/traces for per-entity lookups — shows the candidate understands why the three observability signals are architecturally different tools, not interchangeable ones.

This is the 'what vs. why' distinction the SRE monitoring literature treats as one of the most important calls in writing alerts with high signal and low noise: 'what's broken' is the symptom, 'why' is the (possibly intermediate) cause. A strong answer gives the canonical concrete example — a team doesn't alert on 'CPU at 80%,' it alerts when that CPU spike is burning through the monthly error budget faster than expected, because CPU usage is the cause while error-budget burn is the user-facing symptom — and connects the practice back to alert fatigue: cause-based paging generates volume that doesn't reliably indicate user harm, training responders to ignore pages. Weaker answers can define 'symptom vs. cause' abstractly but can't produce a concrete before/after example or connect it to why noisy cause-based alerting burns out on-call.

The bug is cardinality explosion, and it's caused by the path label. req.originalUrl is the raw, unnormalized URL — it includes path parameters (/users/48213/...) and the query string, so nearly every request produces a URL that has never been seen before. Because a Prometheus time series is identified by metric name plus its full label set, every one of those distinct path values creates a brand-new, permanent time series that has to be indexed and held in memory — so memory grows roughly linearly with traffic instead of being bounded by the small number of actual endpoints the service exposes. The fix is to label with the matched route template the router resolved the request to (Express exposes this as req.route.path, giving a bounded value like /users/:id/orders/:id regardless of which user or order is requested) and to never include the query string in a label at all. The general rule this illustrates: any field whose value-space scales with users, requests, or time — IDs, session tokens, raw URLs, full timestamps — does not belong in a metric label; that kind of high-cardinality lookup is exactly what logs and traces are architected for, while a metrics store is architected for bounded aggregation. The claim that switching to res.on('close', ...) fixes the unbounded memory growth is a red herring — the finish-vs-close event timing has nothing to do with unbounded label growth. The claim that Prometheus counters leak memory by design and that calling .reset() on the counter on a timer fixes it misdiagnoses the failure as a client-library leak and proposes periodically discarding real data instead of fixing the label; it wouldn't even work, since new unique paths would keep arriving between resets and the underlying cause is untouched. Moving new prom.Counter(...) inside the app.use callback makes it worse: creating a new Counter object per request breaks the entire point of a counter (a single accumulating series) and does nothing about the unbounded path values driving the memory growth.