DevOps Interview Questions: Reliability & Incident Response

True. A blameless postmortem is an incident review that targets systemic and process causes rather than blaming individuals. The idea, common in SRE practice, is that people act reasonably given the information they had, so fixing the system prevents recurrence and encourages honest, thorough reporting.

An SLO is a target reliability level that a service team commits to, measured as a specific availability or latency threshold over a defined time window, such as 99.9% availability over a rolling 30-day period.

In SRE practice, if a service's SLO targets 99.9% availability, the remaining 0.1% represents the service's error budget — the tolerance for unplanned downtime within the SLO period.

A blameless postmortem is a retrospective written after a production incident. Its goals are to reconstruct the incident timeline, identify systemic (non-human) root causes, and produce concrete corrective action items that make the same failure less likely to recur, all without assigning individual blame.

False. A blameless post-mortem's primary purpose is to understand the systemic and process failures that allowed an incident to happen, not to assign individual blame or discipline. Blamelessness encourages honest sharing of details, so the team can fix alerts, procedures, and testing gaps and prevent future incidents.

An error budget is the maximum allowable amount of unreliability, such as downtime or failed requests, that a service can have over a period before it breaches its SLO. When the budget is used up, the team shifts its effort from shipping new features to reliability work.

An SLI is the measured metric of service health, an SLO is the internal target set for that metric, and an SLA is the external contract with consequences when the target is missed. Together they run from raw measurement, to internal goal, to customer commitment, so the three terms are not interchangeable.

About 43.2 minutes (roughly 43 minutes). The error budget is the 0.1% of the month that the SLO permits to be unavailable: 43,200 × 0.001 = 43.2 minutes.

A blameless postmortem should include: a summary of the incident, a detailed timeline of events, the impact on users and business metrics, the root cause or contributing factors (without attributing blame to individuals), what went well during the response, what went poorly, and a prioritized list of action items with assigned owners and target dates.

The primary objective is to uncover the systemic causes and contributing factors behind the incident, such as process gaps, tooling deficiencies, and missing safeguards, so the team can prevent similar incidents. It focuses on how the system allowed the failure rather than assigning blame to individuals.

A blameless postmortem reviews an incident by focusing on the systems, processes, and contributing causes that let the failure happen, never on punishing the individual who happened to trigger it. It assumes people act reasonably given the information, tools, and pressures they had at the time, so the question is 'what about the system made this mistake easy and its consequences severe?' rather than 'who screwed up?'. The payoff is psychological safety: when engineers know they won't be blamed, they report incidents and near-misses honestly and in full detail, which surfaces the real root and contributing causes instead of a sanitized story. That honest signal is what lets the team fix the system — add guardrails, automation, better alerts, safer defaults — and file durable, tracked action items that prevent recurrence. Blame does the opposite: it drives reporting underground, so the same latent failure stays hidden and recurs. Blamelessness is about accountability to fixing the system, not the absence of accountability.

A well-run incident moves through five phases in order. Detect: an alert tied to an SLI, or a user report, signals the service is out of bounds. Triage and declare: assess severity, declare an incident, and assign an incident commander. Mitigate: restore service by the fastest safe lever — roll back, fail over, or flip a flag — before chasing root cause. Resolve: confirm the SLI is healthy and stand the incident down. Postmortem: run a blameless review, find contributing causes, and file durable action items so it can't recur.

Freeze new feature launches and redirect engineering effort toward reliability work, such as fixing bugs, improving alerting, and adding redundancy, until the error budget is replenished or the SLO window resets.

An SLI (Service Level Indicator) is a quantitative measure of service health — for example, the ratio of HTTP requests returning a 2xx status code within 200 ms to all requests over a given period. An SLO (Service Level Objective) is a target value or range for that SLI over a specific time window, such as '99.9% of requests succeed within 200 ms over any rolling 30-day window.' An SLA (Service Level Agreement) is a contractual agreement between a service provider and its customers that specifies remedies or penalties if the agreed service level is not met — for example, 'if monthly uptime falls below 99.5%, affected customers receive service credits.' The SLI is the raw measurement; the SLO is the internal target set on that measurement; and the SLA is the external, business-facing contract that is typically set at a lower (more lenient) availability level than the SLO so that internal SLO breaches give the team a buffer before contractual obligations are triggered.

The burn rate is 1.5. The full 30-day error budget is 0.1% of 43,200 minutes, or 43.2 minutes, so the first 10 days allow 43.2 × (10/30) = 14.4 minutes. The observed 21.6 minutes divided by 14.4 minutes gives 1.5, meaning the budget is being consumed 50% faster than the SLO permits.

Detection and alerting: The 99.9% SLO over 30 days gives an error budget of about 43 minutes (0.1% of 43,200 minutes). The 3% error rate is a burn rate of 30x (3% divided by 0.1%). Page on error-budget burn rate using multi-window, multi-burn-rate rules (a short window paired with a longer window, so acute outages page quickly and slow sustained burns page too). Route pages to the on-call engineer through PagerDuty with a linked runbook and a Grafana dashboard scoped to the affected service and its dependencies. Keep non-SLO signals such as CPU and queue depth on dashboards rather than paging. Triage and command: Acknowledge the page promptly, assess severity (this is Sev-1: customer-facing and burning budget at 30x), and declare an incident in the incident-management tool, which provisions a channel, bridge, and timeline. The on-call becomes Incident Commander (IC), who coordinates and makes decisions rather than debugging hands-on. The IC assigns a Scribe (timeline), a Comms Lead (customer and stakeholder updates), and Subject Matter Experts (SMEs) for the failing services, and escalates to a second on-call or engineering manager if not mitigated within a set time. Mitigation: First check the deploy timeline. The canary deploy 25 minutes ago is the prime suspect, so halt the canary and roll back the affected pods immediately, because rollback is fast, reversible, and low-risk compared with debugging under active customer impact. If rollback does not resolve it, work a time-boxed playbook: check traces for a slow or failing downstream dependency, enable circuit breakers or fail fast to shed load, apply rate limiting to protect core paths, and shift traffic away from a degraded service via the service mesh or load balancer. Each step gets roughly 5 minutes to show effect, and the Scribe records actions and outcomes. Budget math for urgency: 31 remaining budget minutes at 30x burn (about 0.03 budget-minutes per wall-minute) lasts roughly 1,000 minutes, about 17 hours, so the budget would be exhausted in well under a day at this pace. Post-incident review: Hold a blameless postmortem within about 48 hours, facilitated by someone not involved in the incident. Document the timeline, customer impact, SLO budget consumed, contributing factors, and root causes. File action items as tracked tickets with owners and due dates, track completion as a reliability metric, escalate overdue items in the weekly reliability review, and publish the postmortem to the wider engineering org. Tooling: Prometheus SLI metrics with burn-rate alert rules, per-service Grafana dashboards with SLO and dependency panels, distributed tracing for cross-service latency, centralized logs for error correlation, incident tooling that auto-creates the bridge and timeline, and a runbook catalog linked from each alert route.

Requiring both windows means the alert fires only when the error rate is elevated right now (short window) and has stayed elevated long enough to matter (long window). The long window acts as a persistence filter: it suppresses brief, self-resolving spikes that consume little error budget and do not warrant a page. The short window preserves fast detection once the elevated burn is sustained. Alerting on the short window alone would page on transient blips, producing false positives.

Define four severity levels with measurable criteria: SEV1 for a full outage or critical data loss affecting a majority of users pages the primary on-call and an incident commander (IC) immediately; SEV2 for major feature degradation pages the primary on-call; SEV3 is minor degradation handled via Slack/ticket; SEV4 is cosmetic or internal, handled in daytime tickets. Escalation is time-based and automatic: if a SEV1/SEV2 is not mitigated within 15 minutes, the on-call escalates to the IC, who can page other service on-calls; at 60 minutes, engineering managers/directors are paged. This removes the social friction of asking for help. Create a dedicated IC rotation of senior engineers from different teams, separate from service on-call. The IC's sole job is coordination: assembling responders, keeping the incident timeline, and communicating with stakeholders. On SEV1 declaration, the IC pages all potentially implicated services at once using a service dependency map and pre-built paging tree, instead of serial investigate-then-escalate. For observability, add a centralized SLO burn-rate dashboard, distributed tracing (e.g., OpenTelemetry) across service boundaries, and grouped alerts so identifying the faulty service becomes a dashboard lookup. For review, every SEV1/SEV2 gets a blameless postmortem within 5 business days using a standard template (timeline, impact, root cause, contributing factors, action items). The rule is that we examine systems and processes, not individuals. Action items have owners and due dates, are tracked on a shared dashboard with weekly review, and a quarterly review aggregates postmortems to find systemic patterns.

The long window acts as a persistence filter: it suppresses pages for brief, self-resolving error spikes that consume negligible error budget, while the short window preserves fast detection for sustained incidents. The dual requirement balances detection speed against noise reduction.

Detection & Alerting: Use Prometheus with Alertmanager for metrics and Loki for log-based alerts. Define per-service SLOs with multi-window, multi-burn-rate alerting: fast-burn (e.g., 2% budget in 1 hour) triggers a page; slow-burn (5% in 6 hours) creates a ticket. Correlate alerts by service topology — Alertmanager grouping rules use the dependency graph exported by the service mesh (Istio) so a single root-cause cascading across downstream services produces one grouped alert, not fifty. Add a 30-minute suppression window on non-critical alerts after each deployment to reduce deploy-induced noise. SEV1 alerts page the primary on-call; SEV2 alerts post to a Slack channel with a one-click escalate button. Escalation & On-Call Routing: PagerDuty schedules with follow-the-sun rotations: each region's team covers a 12-hour day shift, 1-week rotation. Escalation policy has two tiers: primary on-call is paged at T+0; if unacknowledged within 5 minutes, the secondary is paged; if still unacknowledged at T+10, the team lead is paged. A final fallback at T+20 pages the global SRE lead and broadcasts a SEV1-wide Slack message to the entire engineering org. Automated Context Assembly: A PagerDuty webhook triggers an automated context bundle on incident creation. The bundle is posted to the PagerDuty incident and the dedicated Slack incident channel. It includes: (1) a pre-filtered Grafana dashboard link scoped to the affected service(s) showing the last 30 minutes of metrics; (2) the top 5 slowest distributed traces from Jaeger for the affected service; (3) error-level logs from Loki filtered by service name and time window; (4) the last 2 hours of deployment events from ArgoCD for the affected service; (5) a link to the service's runbook from a centrally indexed runbook repository. Post-Incident Learning Loop: A blameless postmortem is scheduled within 5 business days for SEV1 and 10 for SEV2, using a structured template (timeline, root cause, contributing factors, action items). Each action item becomes a Jira ticket with a named owner, priority, and due date. A quarterly reliability review tracks action-item completion rate; items open longer than 90 days are flagged as stale and discussed with the owning team's lead. Reliability KPIs (MTTR, incident frequency per service, action-item completion rate) are reported monthly to engineering leadership.

(1) Cascade isolation: Deploy timeout-based circuit breakers (e.g., Resilience4j TimeLimiter + CircuitBreaker) on every synchronous downstream call. Set a 300 ms per-call timeout; in this incident, the payment-gateway's 2x latency pushed slow calls past this timeout, converting the partial degradation into detectable timeout failures. Configure the breaker with a 25% failure-rate threshold over a 10-second sliding window and a 30-second half-open state allowing 3 probe requests — since 30% of calls were timing out (above the 25% threshold), the breaker trips and calls return a cached fallback instead of holding upstream threads. This is critical: a breaker that only counts explicit HTTP 5xx failures would miss this scenario entirely, because the gateway was returning 200s — just slowly. Apply bulkhead isolation: each downstream gets its own bounded thread pool (e.g., 20 threads for payment-gateway) separate from the service's main pool so a saturated dependency cannot starve other workloads. Set per-hop timeouts at 300 ms with a 1.5-second end-to-end budget propagated via headers; when any downstream thread-pool utilization exceeds 80%, shed non-critical traffic via feature flag (e.g., disable recommendation enrichment) to free capacity. (2) Detection & localization: Emit RED metrics (Rate, Errors, Duration) per service per downstream dependency. Track p99 latency against an EWMA baseline computed over a 1-hour rolling window and alert when p99 exceeds 3x baseline for more than 2 consecutive minutes — this catches partial degradation that simple error-rate alerts miss, since the gateway's error rate was near zero while latency spiked. Use distributed tracing (OpenTelemetry) with critical-path span analysis; when an alert fires, the tracing UI highlights the slowest hop and its upstream blast radius. Maintain a real-time service dependency graph so that when a leaf service degrades, the alert includes all affected upstreams, collapsing 47 individual alerts into one root-cause alert. (3) MTTR reduction: Circuit breakers trip autonomously based on their own internal sliding-window metrics (timeout counts and failure counts) — they are always-on safety devices that react independently of any external alerting or human action, so the initial isolation of the failing dependency happens in seconds with no operator involvement. Each alert links to a pre-built remediation runbook (e.g., 'scale payment-gateway DB connection pool by 2x', 'restart affected pods', 'switch to cached fallback responses'). Non-destructive actions (pool scaling, pod restarts) execute automatically after the circuit breaker has already isolated the dependency; state-changing actions such as regional failover to us-west-2 require human approval based on a severity assessment by the on-call engineer, because a full regional failover is a heavyweight operation with its own risk profile and should not be triggered by a single metric threshold. (4) Systemic postmortem learning: Conduct a blameless postmortem using a structured template covering incident timeline, root cause, contributing factors, and action items with named owners and 30-day due dates. Require Five-Whys analysis that drills from the immediate cause (degraded connection pool) to the systemic gap (no per-dependency bulkheads, no timeout-based circuit breaking that catches slow-but-not-failing dependencies, no partial-degradation alerting). Run quarterly catastrophe drills injecting the same failure class into staging to verify fixes hold. Enforce a CI compliance check that blocks deployment of any service lacking timeout-based circuit breakers and bulkheads on all synchronous dependencies — this ensures the 47 unprotected services from this incident are hardened going forward and prevents the same gap from recurring in new services.

Our system has five components. (1) Evidence correlation engine: during an incident, a tool built on top of our paging system automatically collects all alerts that fired in the affected services' alert streams, all deploys to those services in the preceding 72 hours from CI/CD, all config changes from our config-management system in the same window, and dependency state changes from our distributed-tracing backend. These are joined on overlapping timestamp windows to produce a draft incident timeline; the incident commander reviews and annotates it during the incident. Automation does the collection, the human adds context. (2) Action item lifecycle: every postmortem action item is filed as a ticket with a single accountable owner (an individual, not a team), a due date, and a severity-linked SLA: severity-1 items due in 2 weeks, severity-2 in 4 weeks, severity-3 in 8 weeks. A daily job checks for SLA breaches. On breach the owner's manager is notified; after 7 days it escalates to the director; after 14 days to the VP of Engineering. An item can be closed as 'risk accepted' but requires director sign-off and is surfaced in the quarterly review. (3) Blamelessness safeguards: the postmortem template uses the 5 Whys method, but every question is framed in terms of system behavior, not 'who made the mistake' but 'what about our deployment process allowed this mistake to reach production.' A rotating senior-engineer reviewer checks each postmortem for personal attribution language and flags it for revision. The template includes a section called 'What surprised us about our systems?' rather than 'What went wrong?' (4) Cross-incident pattern detection: every postmortem is tagged with one or more contributing factors from a fixed taxonomy (e.g., 'insufficient capacity headroom,' 'missing alert on key metric,' 'unbounded retry storm,' 'dependency circular failure,' 'insufficient integration test coverage'). A dashboard shows the frequency of each contributing-factor tag over rolling 6-month windows; any factor appearing in 3+ postmortems in a rolling quarter is auto-flagged for the quarterly architecture review as a candidate for a deep infrastructure investment proposal. (5) System effectiveness metrics: we measure the postmortem system itself with action-item completion rate within SLA (target >85%), mean time from incident resolution to postmortem completion (target <48 hours), recurrence rate (the percentage of incidents in a quarter sharing a contributing-factor tag with an incident in the previous quarter, target trending down), and the action-item 'risk accepted' rate (watched but not targeted, since accepting risk is sometimes the correct call).

Detection: I would replace raw threshold alerts with SLO-based multi-window, multi-burn-rate alerts (for example, a page when 2% of the 30-day error budget is consumed within 1 hour, and a page when 5% is consumed within 6 hours, with a ticket-level alert at 10% over 3 days), implemented in Prometheus with recording rules or an SLO platform. Each alert routes to the owning service's PagerDuty schedule, not a shared pool, and I track alert precision (actionable alerts divided by total) weekly, re-tuning or retiring any alert whose no-action acknowledgments exceed 50%. This targets the 62% acknowledge-without-action rate and the 9-minute MTTD, since burn-rate alerts fire on user-facing error rates and fast-burn events are caught in minutes. Mitigation: (1) For the retry storm, I would add circuit breakers at every synchronous dependency, using a service mesh such as Istio or Linkerd or a library such as Resilience4j, with fallbacks to cached or degraded responses, plus per-dependency bulkhead concurrency limits and retry budgets with jittered exponential backoff so retries cannot multiply load. (2) For the bad config deploy, I would move configs behind feature flags so they can change without a deploy, and use progressive delivery with canaries whose analysis is driven by the same SLO burn signals, so an automated rollback triggers within the canary window instead of after 8 minutes of elevated errors. (3) For the replication lag spike, I would make reads lag-aware: each region publishes replication lag as a metric, and when lag exceeds a threshold such as 500 ms the read path routes to the primary region or serves from cache with a staleness marker, so stale reads stop reaching users; an alert fires on lag before that threshold is breached. Escalation: I would define severity tiers tied to user impact (for example, SEV-1 means more than 10% of users degraded), require a 5-minute acknowledgment for SEV-1 pages with automatic escalation to the backup on-call and then the team lead after 10 minutes total, and make every acknowledgment carry an action note. Target MTTM (22 minutes) with runbooks linked from each alert and an incident commander role for SEV-1. Post-incident learning: a blameless postmortem within 48 hours for every SEV-1 and any SEV-2 over 30 minutes, covering timeline, root causes, contributing factors, and user impact. Action items are categorized as mitigation (MTTM), prevention (recurrence), or detection (MTTD), each with an owner, due date, and priority, tracked on a Jira board with a 2-week completion SLA for SEV-1-derived items. A monthly reliability review examines open action items, error-budget burn trends, and alert precision, closing the loop from incidents to fewer incidents and lower MTTD and MTTM.