System Design Interview Questions: Scalability Fundamentals

Vertical scaling is bounded by the physical and economic limits of hardware. At some point, no single machine can be made powerful enough, or the cost becomes prohibitive. This is one of the core reasons distributed, horizontally-scaled architectures are favored for large-scale systems. There is always a ceiling on how much a single machine can be upgraded.

The typical evolution of scalability in a growing system follows this path: (1) First, vertically scale the existing server since it requires no architectural change. (2) Once vertical limits are hit, introduce horizontal scaling with a load balancer to add more app servers. (3) Add a caching layer to relieve read pressure on the database. (4) Finally, shard the database once write volume and data size exceed what a single database node can handle — this is the most complex step and is deferred as long as possible.

Load balancing is the practice of distributing incoming network requests across a pool of servers (the 'server farm' or 'backend pool'). This prevents any one server from becoming overwhelmed and is the foundational technique for horizontal scalability. Database indexing speeds up queries on a single node, data compression reduces payload size, and SSL termination offloads encryption — none of those distribute request load across servers.

The definitions are reversed. Vertical scaling ('scaling up') means upgrading the resources of an existing machine — more CPU cores, more RAM, faster storage — on the same single node. Horizontal scaling ('scaling out') means adding more machines (nodes) to the system and distributing the workload across them. Because the statement swaps the two definitions, it is false.

Caching is the technique of storing the results of expensive or frequently repeated operations (such as database queries) in a fast, temporary storage layer so that subsequent requests can be served without hitting the database again. The second blank is pinned to Redis specifically because the prompt qualifies it as supporting rich data structures (strings, lists, sets, hashes, and more) beyond simple key-value pairs — a distinguishing feature of Redis that does not apply to Memcached, which supports only plain string key-value storage. Redis is one of the most widely adopted open-source in-memory data stores used for caching in distributed systems.

Horizontal scaling (scaling out) means adding more machines/instances to share the load, as opposed to vertical scaling (scaling up), which means upgrading the existing machine's CPU, RAM, or storage. Using a load balancer to distribute traffic across multiple servers is the canonical example of horizontal scaling. Database sharding and cache invalidation are separate, more specific techniques.

Vertical scaling requires no changes to application architecture — you simply upgrade the server's CPU, RAM, or storage — making it straightforward to implement. However, hardware has a physical ceiling: there is a maximum amount of RAM or number of CPU cores you can add to one machine. Horizontal scaling removes this ceiling by adding more machines, but it introduces complexity such as load balancing, distributed state management, and network partitioning concerns.

Teams typically start with the easiest change: vertically scaling the existing server requires zero code changes. Next, adding a caching layer (like Redis) dramatically reduces database read pressure with minimal architectural change. After that, horizontal scaling of the application tier is introduced behind a load balancer to handle more concurrent users. Database sharding is the most complex and disruptive step, requiring data partitioning strategy and application-level routing, so it is tackled last.

The root problem is that the app nodes are stateful: each holds session data only it can see. Externalising state to a shared store makes the nodes interchangeable, which is the precondition for horizontal scaling and for free rolling deploys and failover. Sticky sessions are a workaround, not a fix — they pin users to a node, so a deploy or crash still drops their session and they defeat even load distribution. Vertical scaling (more RAM) raises the ceiling but keeps the state trapped on one box, so the cross-node inconsistency remains. A CDN caches public, cacheable responses; per-user session data is neither, so it does nothing here.

8,640,000 / 86,400 seconds = 100 requests per second average. Capacity must be provisioned for peak, not average, because real traffic is bursty — a common heuristic is to assume peak is several times the mean (e.g. 2-10x) and size headroom accordingly. Estimating QPS, then storage and bandwidth from request size, is the standard opening of a capacity-estimation exercise.

Horizontal scaling (scaling out) adds more machines/instances to distribute load, making it well-suited for stateless services. Vertical scaling (scaling up) increases the resources of a single machine. A load balancer distributes traffic but does not itself replicate application logic. A CDN caches static content closer to users. Only horizontal scaling directly handles increased read throughput on stateless web servers by adding replicas behind a load balancer.

Read replicas (a) directly improve read scalability by creating additional database nodes that serve SELECT queries, relieving pressure on the primary and allowing the system to handle more concurrent reads horizontally. Read-through caching (b) (e.g., Redis) directly improves read scalability by intercepting repeated read requests and returning results from memory, drastically reducing the number of queries that ever reach the database. Sharding (e) directly improves read scalability by partitioning data across nodes so each node owns and serves a subset of the data — queries for a given user hit only one shard, reducing per-node dataset size and concurrent read contention. All three are well-accepted, direct read-scalability techniques. Increasing connection pool threads (c) is tempting, but the question specifies doing so without changing the database server's own max_connections or hardware: the database itself still becomes the bottleneck once its connection limit is saturated, so this manages client-side overhead rather than increasing the database's fundamental read throughput. Rewriting stored procedures in the application layer (d) is an architectural refactoring that does not inherently increase the database's capacity to serve more reads.

According to canonical system design literature (e.g., Alex Xu's System Design Interview), the standard incremental scaling progression is: (1) Start with a single-server monolith to ship and validate quickly. (2) Move the database to a dedicated server — this is done early to eliminate resource contention between the app and the DB on the same machine, and it is a prerequisite for meaningful horizontal app-server scaling. (3) Add a load balancer with multiple stateless app servers now that the DB is independently reachable by all instances. (4) Introduce a caching layer (e.g., Redis/Memcached) to absorb hot reads and reduce DB load. (5) Shard the database to distribute write load and data volume at massive scale. This C → E → B → A → D order is the consensus ordering in the most widely cited system design references.

A queue decouples producers from consumers: checkout returns as soon as the event is enqueued (a), the queue acts as a buffer that smooths spikes so consumers process at a sustainable rate (b), and durable queues retain messages so a consumer that was down can drain the backlog on recovery (c). The two wrong options reflect common misconceptions. Async processing does not lower per-request end-to-end latency (d) — the downstream work still happens, just later; you trade latency-to-completion for responsiveness and resilience. And most queues offer at-least-once delivery, so consumers must be idempotent to tolerate duplicates (e); the queue does not remove that obligation.

Horizontal scale-out requires that any instance can handle any request. Stateless instances (a) make this trivially true — session data lives in a shared external store, so routing a request to a different instance is transparent. A health-checking load balancer (b) ensures the new capacity is used immediately on startup and that unhealthy nodes aren't given traffic. Graceful shutdown (c) means scale-in operations don't cause request errors — a critical production concern. Sticky sessions (d) are the opposite of scale-friendly: they reintroduce instance affinity, so the failure or scale-in of one node loses that user's session. Local filesystem storage for uploads (e) has the same problem: files written to one instance are invisible to all others, breaking the assumption that any instance can serve any request.

Statelessness is a prerequisite for horizontal scale. The 12-Factor App principle 'processes are stateless and share-nothing' captures this precisely. In practice, teams often accidentally bake state in (in-memory caches, local disk uploads) and discover the problem only when load balancing or autoscaling is introduced.

Token buckets (a) are the canonical burst-tolerant algorithm: tokens accumulate at the refill rate up to bucket capacity, so a burst of capacity tokens is permitted before the rate kicks in. The sliding window log (b) solves the boundary problem by tracking exact timestamps, so a burst at the end of one period plus the start of the next can't exceed the limit. The fixed window boundary exploit (c) is a real and well-documented flaw: if the limit is 100/minute and 100 requests arrive in the last second of window 1 and 100 in the first second of window 2, you've accepted 200 in two seconds. Distributed coordination (e) is essential in multi-node gateways — without a shared counter each node enforces the limit independently, effectively multiplying it by the number of nodes. Option (d) is wrong: token bucket explicitly allows bursts; the algorithm that spaces requests exactly 1/rate apart is the leaky bucket (or its fixed-rate output variant), not the token bucket.

Stateless services are trivially horizontally scalable: any node can handle any request because no local state distinguishes one node from another. Stateful services carry session data (auth tokens, cart contents, in-progress computations) that is tied to a specific node. Scaling them horizontally requires either sticky sessions (which re-introduce the node affinity problem and make load balancing uneven), or externalising the state to a shared store like Redis so that any node can reconstruct it. The added complexity — and the external store becoming a bottleneck — is why statelessness is the first design goal for horizontally scalable systems.

Both offload the primary but differently. Read replicas add queryable copies and scale diverse read traffic across nodes (writes stay on the primary); their hazard is replication lag — a replica trails the primary, so a freshly written value can read back stale (read-after-write inconsistency), fixed by routing critical reads to the primary or using read-your-writes routing. A cache fronts the DB with in-memory hot results and is unbeatable for repeated reads of a small key set; its hazard is staleness / invalidation — cached data drifts from the source of truth, so you need TTLs or explicit invalidation, the genuinely hard part. They're complementary: cache the hot path, replicate the long tail.

Always measure before acting so you scale the actual bottleneck, not a guess. Caching is the cheapest lever because it removes work entirely; adding stateless app instances is straightforward when the tier holds no session state; resharding the data tier is last because it is the most disruptive and hardest to reverse. The ordering reflects rising blast radius and engineering cost.

In a horizontally scaled stateless service behind a load balancer, session affinity (sticky sessions) pins a user's requests to a single instance. This re-introduces a single point of failure per user and prevents full horizontal elasticity. The correct alternative is to externalise session state into a shared distributed store (e.g., Redis) so any instance can serve any request. Rate-limiting at the LB, distributing static assets via CDN, and database read replicas are all orthogonal scalability improvements that do not conflict with statelessness.

This question probes deep knowledge of the CAP theorem's practical implications for scalable distributed databases. A network partition means nodes cannot communicate, so a CP system (e.g., HBase, Zookeeper) will refuse writes or reads to avoid returning stale data, sacrificing Availability. An AP system (e.g., Cassandra, DynamoDB in eventual-consistency mode) will continue serving reads and accepting writes on both sides of the partition, accepting that data may diverge and require later reconciliation. CA systems cannot exist in a distributed setting when partitions are possible — the theorem asserts you must choose between C and A when P occurs. The reconciliation mechanism for AP systems is typically vector clocks, last-write-wins, or CRDTs.

This question tests nuanced understanding of database sharding strategies and their scalability trade-offs. Range-based sharding on a monotonically increasing key (e.g., timestamp or auto-increment ID) concentrates all writes on the last shard ('hot shard' or 'write hotspot'), making it effectively unscalable for write-heavy workloads. Consistent hashing distributes keys uniformly across nodes and, crucially, minimises key remapping when nodes are added or removed — only ~K/n keys need to move (K = total keys, n = node count). Directory-based sharding with a lookup service adds a network hop and a single point of failure unless the directory itself is highly available. Composite/hierarchical sharding (e.g., hash on user ID, then range on timestamp within shard) is common in practice but adds operational complexity.

Horizontal scaling works because any instance can serve any request, letting the load balancer spread traffic and autoscaling add/remove nodes freely — and that interchangeability requires the tier to be stateless. Local per-user session memory breaks it: you're forced into sticky sessions / session affinity, which unbalances load and defeats free scheduling, and when an instance dies or scales down its in-memory sessions vanish (users logged out, work lost). The fix is to externalize state — a shared session store (Redis/DB) or self-contained signed tokens (JWT) — so no instance holds anything another can't reconstruct. 'Make it stateless, push state to a shared store' is the canonical scaling move for the app tier.

Scaling a read-heavy service is the canonical caching + replication problem, and the lever that makes it tractable is a precise freshness stance: because most catalog fields tolerate a few seconds of staleness, you can absorb the overwhelming bulk of reads in a multi-tier cache (in-process → Redis → CDN edge) and fan the rare misses across read replicas, leaving the single write primary to handle only the sparse writes. The consistency answer is deliberately per-field — eventual consistency for browsing, but the price/availability shown at checkout is read fresh (primary or revalidated) so no one buys at a stale price (a real CAP trade-off, not a global toggle). The two failure modes that distinguish a strong answer are the hot key (a flash-sale item on one cache node — spread it across nodes/edge/local caches) and the thundering herd / cache stampede on expiry (request coalescing / single-flight + jittered TTLs + serve-stale-while-revalidate). Globally, edge caches and regional replicas meet the latency budget, and read-your-writes is handled by routing a writer's own reads to the primary.