CAP Theorem & Consistency interview questions

When a network partition occurs and availability is prioritized, Consistency (C) must be sacrificed. The CAP theorem says you can only guarantee two of C, A, and P at once. Keeping all nodes available during a partition means some nodes may return stale data, violating the guarantee that every read reflects the most recent write.

False. An AP system like Cassandra does NOT guarantee that every read immediately reflects the latest write. AP systems trade away strong consistency for availability during partitions, meaning different replicas may temporarily hold different versions of data. A read from a replica that hasn't yet received the latest write will return stale data — this is called eventual consistency.

When a network partition occurs, the system must sacrifice either strong consistency or availability — it cannot maintain both. The CAP theorem is a hard theoretical bound: since partitions cannot be prevented in distributed networks, the system designer must choose between a CP design (reject requests to stay consistent) or an AP design (serve potentially stale data to stay available). There is no middle ground during an active partition.

Eventual consistency, read-your-writes consistency, and monotonic read consistency are all weaker than linearizability. Eventual consistency allows temporary divergence between replicas; read-your-writes and monotonic read are session guarantees that relax real-time global ordering to gain availability. Strict serializability is at least as strong as linearizability, and two-phase commit is a commit protocol, not a consistency model.

The correct order is: (1) Identify consistency requirements per operation, (2) Accept that partitions will occur (partition tolerance is non-negotiable in distributed systems), (3) Choose CP or AP based on business requirements, (4) Select a replication strategy (sync/async) to match the chosen model, and (5) Implement conflict-resolution or read-repair to handle divergence. This order ensures business needs drive architecture decisions before implementation details are chosen.

False. Network partitions are a fact of distributed systems, not something you can design away. CAP says that during a partition you must sacrifice either consistency (CP) or availability (AP) — you do not get to opt out of partitions. The only systems that can claim both at all times are effectively single-node.

During a network partition the nodes on either side cannot communicate, and CAP says you must choose between consistency and availability for that period. A CP system refuses or blocks requests it cannot serve correctly so it never returns stale or conflicting data, sacrificing availability. An AP system keeps accepting reads and writes on both sides to stay available, accepting that replicas will temporarily diverge and reconcile later. When the network is healthy you can have both, so the trade-off only bites under a partition.

Requirements. The limit is per API key over a rolling window (1,000/min), checked on every request. I'd confirm: is a brief burst above the steady rate acceptable (token bucket says yes, up to the bucket size), and — most importantly — if the counter store is unavailable, do we fail open (let requests through, protect availability) or fail closed (reject, protect the backend)? For a public gateway I default to fail-open with alerting, since rate limiting is a guardrail, not the product.

Algorithm. I'd use a sliding-window counter (or token bucket). Fixed-window counters are simplest but allow ~2× the limit across a boundary: a client sends 1,000 in the last second of minute N and another 1,000 in the first second of N+1. The sliding-window counter weights the previous window's count by the overlap fraction, smoothing that out at a fraction of the cost of a full per-request log. Token bucket is the equivalent framing when you want to allow controlled bursts: refill at limit/60 per second, cap the bucket at the burst size.

Distributed counter. Counters live in a shared in-memory store (Redis) so all gateway nodes see the same count. The check must be atomic — a read-modify-write from two nodes races and under-counts — so I use INCR + EXPIRE (or a small Lua script that does increment, TTL, and the limit comparison in one round trip). The API key is the Redis key, so all operations for one key land on one shard.

Accuracy vs latency. A central store is exact but adds a network hop to every request. To hold <5 ms p99, I keep the store cell-local (one Redis per region/cell, no cross-region hop) and accept that limits are enforced per-region. If even one hop is too much, the alternative is local per-node token buckets that sync counts every few hundred ms — fast and highly available, but approximate (a key can briefly exceed the global limit by roughly the number of nodes). I'd start central-per-cell and only move to local+sync if the hop proves too costly.

Failure modes. Store down: fail open, serve requests un-limited, and alert. Hot key: a single key's counter is one Redis key on one shard, so a 200K-req/sec key hammers that shard — I'd detect it and shard that key's counter into N sub-counters (key:0..N) summed on read, or shed it at the edge. Clock skew: window algorithms key off time, so I rely on the store's clock (single source) rather than each node's wall clock.

Scaling. At ~500K ops/sec the counter store is sharded by key across a Redis cluster; because each key's ops stay on its shard, adding shards scales throughput linearly. Replicas give availability, and per-cell stores keep latency flat as we add regions. The bottleneck is always the hottest single key, handled by sub-sharding as above.

An AP system with HLC-based replication can guarantee eventual consistency (all replicas converge once writes stop) and causal consistency (by propagating HLC metadata with replication messages and having each replica delay applying an update until its causal dependencies are met — no synchronous coordination needed). Monotonic reads, RYOW without sticky sessions, and linearizability all require cross-replica synchronisation that violates availability during partitions.

Problem 1 — Silent data loss: LWW silently discards one of the two legitimate debit operations. The $300 or $500 deduction is permanently lost, causing the ledger to be incorrect; the bank either under-charges or creates phantom money, violating auditability and regulatory compliance.

Problem 2 — Clock skew vulnerability: LWW relies on wall-clock timestamps, but clocks across datacenters drift and can be manipulated. Millisecond-level skew can arbitrarily determine which write 'wins', making correctness dependent on clock synchronisation rather than application semantics.

Alternative strategy — Commutative/CRDT-based delta ledger: Model the account balance as a PN-Counter (Positive-Negative CRDT) where each debit is stored as an independent, immutable delta operation (operation-based CRDT / event sourcing). Instead of storing the absolute balance, each datacenter appends signed delta events (–$500, –$300) with a unique logical ID. On merge, all deltas from all datacenters are unioned, and the balance is derived by summing them. Because union of sets is commutative, associative, and idempotent, no write is ever lost. For stronger guarantees, combine this with a per-account distributed lock or a saga/2PC pattern for high-value transactions.