System Design Interview Questions: Reliability, Failover, and SLOs

While adding parallel replicas generally improves availability because the component only fails if ALL replicas fail simultaneously, it does not help indefinitely. In practice, there are diminishing returns — the improvement becomes negligible after a few replicas. More importantly, if the replicas share a common failure domain (e.g., same power supply, same network switch, same availability zone), a correlated failure can bring all of them down at once, making redundancy ineffective. The statement is therefore false in general.

Standard SRE incident response is governed by 'stop the bleeding first,' but initial communication is embedded in the declaration step rather than treated as a separate later phase. Step 1: monitoring fires, an incident is declared, an incident commander is assigned, and an immediate 'we are investigating' notice goes out to stakeholders — communication begins here, not after mitigation. Step 2: the team applies the fastest available remediation (rollback, failover, traffic rerouting) to minimise MTTR; this is the primary focus until service is restored. Step 3: with the immediate impact contained and the service stabilising, the team pivots to diagnosing the root cause; earlier exploratory diagnosis may have run in parallel, but deep analysis becomes the primary focus only after mitigation succeeds. Step 4: once the root cause is understood and fully resolved, the incident is formally closed, the status page is updated to 'resolved', and a final all-clear is sent. Step 5: after full resolution, the team conducts a blameless post-mortem, documents contributing factors, and prioritises long-term corrective actions to prevent recurrence. By folding the initial stakeholder notification into the declaration step, the ordering avoids ambiguity about where communication sits relative to mitigation and diagnosis.

99.9% ('three nines') leaves 0.1% of a 8,760-hour year unavailable, which is about 8.76 hours per year. Each additional nine cuts allowed downtime by 10x — four nines (99.99%) is ~52 minutes/year, five nines (99.999%) is ~5 minutes/year. Knowing these conversions lets you sanity-check whether a proposed SLA is realistic given the architecture's redundancy and failover design.

placeholder

Active-active multi-region failover eliminates single-region SPOFs and keeps the service up during region-level outages. Circuit breakers prevent a failing dependency from cascading and taking down the entire service. Idempotent async messaging decouples availability of the caller from the availability of downstream services, making partial outages invisible to end users. A single Redis primary is itself a SPOF—losing it loses sessions globally—so it hurts availability. A monolithic rolling restart can improve deploy uptime, but a monolith is not inherently more available than microservices and does not directly address the 99.99% target; it is neutral-to-negative for this goal.

The canonical sequence is: (1) detect failure via health-check, (2) immediately stop sending traffic to the failed node so user impact is minimised, (3) notify humans and redirect traffic to healthy capacity, (4) provision replacement and let it catch up, (5) only re-introduce the node after it is verified healthy and in sync. Skipping detection before routing removal would send requests to a broken node; re-introducing a lagging replica before confirming lag ≈ 0 can serve stale or inconsistent data.

A latency-only SLO is a classic availability blind spot: when the service is completely down, requests fail fast (often with a TCP reset or immediate 5xx) rather than timing out, so they record near-zero latency or are excluded from percentile calculations altogether. The P99 therefore looks healthy during an outage. The fix is to pair every latency SLO with an error-rate/availability SLO so that both slowness and unavailability are captured in the error budget.

Error budgets reframe reliability as a resource to be managed rather than maximised, aligning product velocity and operational stability. They depend on good observability — the SLIs (latency, error rate, availability) must be measured from the user's perspective for the SLO and budget to mean anything.

This question tests deep knowledge of availability math and SLA composition. When services are chained sequentially, their availabilities multiply: 0.9999 × 0.9995 × 0.9990 = 0.9984005. Expressed as nines, 0.9984 ≈ 99.84%, which is between two-nines (99%) and three-nines (99.9%). The composite is only ~99.84% available, meaning roughly 1.4 hours of downtime per month — far less than any single component's SLA. This is a critical trap: engineers often assume the 'best' component's SLA dominates, but in series, the worst combination wins and you always lose nines.

In a 5-node Raft cluster, a quorum of 3 nodes is required to commit any log entry. The claim that N1 can commit new entries and serve writes is wrong: N1 cannot commit new entries because it can only hear acknowledgments from N2 (2 nodes < quorum of 3); leadership alone does not grant commit authority. The claim that the {N3, N4, N5} partition elects a new leader is correct: {N3, N4, N5} form a majority of 3 and will hold an election after their election timers fire without receiving heartbeats from N1, electing a new leader capable of committing writes. The claim that N1 cannot commit new entries and may step down is correct: N1 cannot commit new entries because quorum is unreachable with only N2. The Raft paper (Figure 2) does not mandate a proactive step-down on replication failure, but many production implementations (e.g., etcd's check-quorum) do cause N1 to step down after an election timeout of missed quorum acknowledgments. The statement correctly reflects both behaviors — N1 cannot commit regardless, and the precise step-down timing is implementation-dependent. The claim that clients requiring linearizable reads receive unavailable responses is correct: clients requiring linearizable reads routed to N1 or N2 cannot be safely served; a correct implementation must return an error or unavailable response because N1 cannot confirm it holds current leadership without quorum, and serving reads without that confirmation would violate linearizability. The claim that the minority partition elects a new leader is wrong: {N1, N2} has only 2 nodes and cannot form a quorum of 3, so no leader election in that partition can succeed.