System Design Interview Questions: Caching Strategies

The correct expectations are that cached assets are served from edge PoPs close to users (cutting round-trip latency), origin load and egress drop because the CDN absorbs repeat requests for cacheable content, and Cache-Control/TTL headers govern how long edges serve content before revalidating. Highly personalized per-user responses are not trivially cacheable at a shared edge, and edges do not instantly reflect origin changes — stale objects live until TTL or an explicit purge.

Use a write-through cache: write to the cache and the database synchronously on every write. The just-written value is already in cache for the next read, giving fresh reads immediately at the cost of a slightly slower write — exactly the trade described. Cache-aside only populates on a read miss, and TTL or write-behind strategies both permit a window of staleness.

Switch to LFU (least-frequently-used). The nightly scan's one-time keys each have a frequency of 1, so LFU evicts them first and protects the frequently-hit working set. This is the classic cache-pollution problem and exactly LRU's weakness: the flood of fresh keys are the most-recently-used, so LRU evicts the genuinely hot keys instead. Scan-resistant variants like LRU-K, ARC, and W-TinyLFU blend recency and frequency.

Combine three system-level levers: stream tokens over SSE to cut time-to-first-token, cache responses to repeated or similar prompts to skip inference entirely, and route simple requests to a small model while escalating only when needed. These change how often you invoke the model, which model you invoke, and how you deliver the result. Adding replicas, blanket CDN caching, or raising the output token limit each fail here.

With cache-aside the application reads from the cache and, on a miss, loads from the database and populates the cache itself; writes update the database and invalidate or delete the cached entry. Its failure mode is a stale read: between a write that fails to invalidate (or two concurrent reads racing a write) the cache can serve outdated data. With write-through the application writes through the cache layer, which synchronously writes to the database before acknowledging, so the cache is always consistent on write. Its failure mode is added write latency and the fact that data nobody reads still gets cached, wasting memory.

In cache-aside the application code owns the cache: on a read it checks the cache first, and on a miss it loads from the database, populates the cache, and returns. Writes go to the database and then invalidate (or update) the cache entry. Its main failure mode is the thundering herd / cache stampede — when a hot key expires, many concurrent requests all miss and hammer the database at once. Mitigate with request coalescing (single-flight), a short lock per key, or probabilistic early refresh.

A cacheable read passes through progressively narrowing caches before reaching the data:

1. The client request hits the nearest CDN / edge point of presence.
2. On a CDN miss it forwards to the regional load balancer.
3. The load balancer routes to a healthy application server.
4. The app checks the in-memory cache (e.g. Redis) for the key.
5. On a cache miss it queries the database of record.

A write-through cache writes both stores before acknowledging:

1. The application issues a write for a given key.
2. The cache layer writes the new value into the cache.
3. The same write is synchronously persisted to the database.
4. Only after both succeed is the write acknowledged to the application.

Requirements. Two operations dominate: create (write a mapping, optionally with a custom alias and expiry) and redirect (look up a code and 30x to the long URL). The system is overwhelmingly read-heavy (100:1), so the redirect path is what we optimise. I'd confirm: are custom aliases required (yes), do links expire (optional TTL), and how precise must click counts be (approximate/eventually-consistent is fine). I'd use a 302 (temporary) redirect so we keep serving redirects through our system and can still count clicks; a 301 would let browsers cache and bypass us.

Capacity. ~40 writes/sec average, ~200/sec peak; ~4K reads/sec average, ~20K peak. Storage: ~6B links over 5 years; at ~500 bytes/row (code, URL, metadata) that's ~3 TB — comfortably shardable. Keyspace: base62 with 7 characters gives 62⁷ ≈ 3.5×10¹² codes, far more than 6B, so 7 chars (often padded to a fixed length) is plenty. The hot working set (recently/ popularly accessed links) is a small fraction of 6B, so a cache of tens of GB covers the bulk of redirect traffic.

Data model. A single mapping table/collection keyed by the short code: code (PK) → long_url, created_at, expires_at, owner_id. Point lookups by primary key suit a key-value store (DynamoDB/Redis-backed) or a sharded relational table sharded by code. Click counts live separately — incrementing a counter on every redirect would put write load on the read path — so clicks are emitted as events and aggregated asynchronously.

Short-code generation. Generate a unique 64-bit id (a distributed counter handed out in ranges per server, or a Snowflake-style id) and base62-encode it to get the short code; this guarantees uniqueness with no collision checks and no hotspot from a single shared counter. Custom aliases are written directly with a uniqueness check and stored in the same table, reserving that code. (A hash-of-URL scheme is the alternative but needs collision handling and breaks idempotency for duplicate URLs.)

Read/write paths. Write: allocate id → base62 → insert mapping → return the short URL. Redirect: a load balancer / CDN fronts stateless app servers; the server does a cache-aside lookup (code → long_url) in Redis, falling back to the store on a miss and populating the cache, then returns a 302. This keeps p99 well under 100 ms for cache hits. Expiry is enforced by TTL on both the row and the cache entry. Each redirect fires a lightweight click event onto a queue (Kafka) that a consumer aggregates into per-link counts.

Bottleneck & scaling. The bottleneck is the redirect read path at peak. We scale it with cache layers (most reads never touch the store), read replicas, and sharding the mapping store by code so lookups stay single-shard. Id generation scales by handing each server an independent id range (or using Snowflake), avoiding a single global counter as a SPOF. At 10× traffic, the cache absorbs most of it; we add cache nodes and replicas and, if needed, push redirects further to the edge.

Requirements. The limit is per API key over a rolling window (1,000/min), checked on every request. I'd confirm: is a brief burst above the steady rate acceptable (token bucket says yes, up to the bucket size), and — most importantly — if the counter store is unavailable, do we fail open (let requests through, protect availability) or fail closed (reject, protect the backend)? For a public gateway I default to fail-open with alerting, since rate limiting is a guardrail, not the product.

Algorithm. I'd use a sliding-window counter (or token bucket). Fixed-window counters are simplest but allow ~2× the limit across a boundary: a client sends 1,000 in the last second of minute N and another 1,000 in the first second of N+1. The sliding-window counter weights the previous window's count by the overlap fraction, smoothing that out at a fraction of the cost of a full per-request log. Token bucket is the equivalent framing when you want to allow controlled bursts: refill at limit/60 per second, cap the bucket at the burst size.

Distributed counter. Counters live in a shared in-memory store (Redis) so all gateway nodes see the same count. The check must be atomic — a read-modify-write from two nodes races and under-counts — so I use INCR + EXPIRE (or a small Lua script that does increment, TTL, and the limit comparison in one round trip). The API key is the Redis key, so all operations for one key land on one shard.

Accuracy vs latency. A central store is exact but adds a network hop to every request. To hold <5 ms p99, I keep the store cell-local (one Redis per region/cell, no cross-region hop) and accept that limits are enforced per-region. If even one hop is too much, the alternative is local per-node token buckets that sync counts every few hundred ms — fast and highly available, but approximate (a key can briefly exceed the global limit by roughly the number of nodes). I'd start central-per-cell and only move to local+sync if the hop proves too costly.

Failure modes. Store down: fail open, serve requests un-limited, and alert. Hot key: a single key's counter is one Redis key on one shard, so a 200K-req/sec key hammers that shard — I'd detect it and shard that key's counter into N sub-counters (key:0..N) summed on read, or shed it at the edge. Clock skew: window algorithms key off time, so I rely on the store's clock (single source) rather than each node's wall clock.

Scaling. At ~500K ops/sec the counter store is sharded by key across a Redis cluster; because each key's ops stay on its shard, adding shards scales throughput linearly. Replicas give availability, and per-cell stores keep latency flat as we add regions. The bottleneck is always the hottest single key, handled by sub-sharding as above.