System Design Observability Interview Questions: SLOs, Metrics, Logs, Tracing

The sidecar pattern decouples telemetry infrastructure from application code: a proxy or collector container (e.g. an OpenTelemetry Collector sidecar or an Envoy proxy) is injected alongside each pod — typically via a Kubernetes mutating admission webhook — and handles trace propagation, log scraping, and metric collection transparently. Application teams write their service; the platform team ships the sidecar. This is how service meshes (Istio, Linkerd) and auto-instrumentation platforms achieve zero-code-change observability at scale. A shared library requires every team to adopt, upgrade, and initialise the library — no uniformity guarantee and direct code coupling. Polling log files on disk is fragile, adds latency, and misses structured in-process spans. Requiring each team to expose a /metrics endpoint shifts the work onto the application teams, exactly what the question wants to avoid.

The three pillars of observability are metrics, logs, and traces — and in a microservices system all three are necessary. RED/USE metrics with SLO-based alerting give you the service-level signal to know when users are affected. Correlated structured logs let you follow a request across services using a shared trace ID, essential for debugging distributed failures. Distributed traces give you the call graph and latency breakdown that individual service logs cannot — you can see which downstream hop is slow and where errors originate. Manual SSH checks are not observability: they are reactive, don't scale, and miss failures that resolve before someone looks. CPU-threshold alerts are notorious for both noisy false alarms (high CPU often means healthy load) and missed real user pain (a slow database doesn't spike your CPU); SLO-based alerts on error rate and latency are far more actionable and correlated with user experience.

Metrics tell you that something is slow and which service is involved, but they aggregate across all requests and lose per-request causality. To identify why a specific request was slow across a chain of services you need distributed tracing — spans that carry a shared trace ID, capture cross-service call trees, and expose exactly where latency was spent. Logs add per-event context that metrics discard. The three pillars of observability — metrics, logs, and traces — are complementary; none substitutes for the others.

An error budget translates the SLO into a concrete allowance of acceptable unreliability — here, about 43 minutes of downtime per month. It creates a shared language between product and reliability teams: while budget remains, teams may take on risk (feature launches, risky migrations); when it is exhausted (either by incidents or planned work), reliability investment takes precedence and risky changes are gated. This is the mechanism that prevents the SLO from being aspirational — it has teeth. MTTR is a recovery speed metric, not a risk-tolerance budget; a fast recovery doesn't make the outage free if the budget is already spent. An SLA is the contractual consequence threshold, typically more lenient than the SLO — it does not tell you whether a given action is prudent. Deployment frequency is a DORA metric about change cadence, not about whether any particular risky change is safe to make when reliability headroom is low.

They operate at different traffic boundaries. The API gateway is the north-south edge: it handles the public API surface — authentication, TLS termination, routing, rate limiting, and request aggregation for external callers. The service mesh is east-west: it intercepts every inter-service call via sidecar proxies (e.g. Envoy) to provide mTLS between pods, circuit breaking, retries, load balancing, and per-call telemetry without any changes to application code. These two layers complement each other — the gateway cannot see internal pod-to-pod calls, and the mesh is not designed as an external ingress. A gateway does not subsume mesh features; they have different scopes. Sidecars handle inter-service calls, not external ingress. A service mesh has nothing to do with database connection pooling — that is managed by the application or a separate proxy like PgBouncer.

SLI/SLO/SLA form a hierarchy of reliability accountability. An SLI is the raw measurement — success rate, latency percentile, availability — that expresses what you actually observe. An SLO is your internal target: an aspirational but achievable bound on that measurement that defines what 'working well' means. An SLA is the external, legally or commercially binding commitment; SLOs are typically set stricter than SLAs to leave a buffer. Error budgets are the complement of SLOs (e.g. 0.1% of requests may fail over 28 days) and are the key SRE mechanism: you can spend them on velocity (risky deploys, experiments) and must freeze risky changes when the budget is gone. Setting SLOs at 100% is an anti-pattern: it leaves no error budget, incentivises no risky but valuable changes, is practically unachievable, and means any single failure is by definition an SLO violation.

The three pillars answer different questions. Metrics are cheap aggregated time series (rates, error %, latency percentiles) — best for dashboards and alerting: they tell you that something is wrong but lose per-request detail. Logs are rich discrete event records — best for the what of a specific event/error, but high-volume and blind to cross-service flow. Traces stitch a single request's spans across services via a propagated trace id — best for where the latency/failure is. You need all three because they operate at different altitudes: metrics detect and alert, traces localize to a hop, logs explain the root cause. Relying on one leaves a blind spot — metrics alone can't tell you why; logs alone can't show the request's path.

Error budgets reframe reliability as a resource to be managed rather than maximised, aligning product velocity and operational stability. They depend on good observability — the SLIs (latency, error rate, availability) must be measured from the user's perspective for the SLO and budget to mean anything.

Observability flows from collection to action. You must instrument the service before there is anything to ship; you must have data centralised before you can visualise it; you need dashboards to understand normal behaviour before you can set meaningful SLOs; and SLOs must be in place before you can write alerts that measure the right thing rather than arbitrary thresholds. Alerting on error-budget burn rate (step 5) rather than raw error counts ensures alerts are proportional to user impact.

Distributed tracing stitches together the spans of one request as it crosses service boundaries via a propagated trace context, so you can see exactly which hop consumed the time — the precise need when each service looks fine on its own. Aggregate metrics tell you that p95 is high but average away the individual slow request and cannot attribute latency across the chain. Logs are per-service and unordered relative to each other; without a shared trace id you cannot reliably reconstruct one request's journey across all six. A health check only answers up/down liveness and says nothing about per-request latency distribution. Metrics, logs, and traces are complementary, but tracing is the one built for cross-service request attribution.