System Design Interview Questions: Message Queues

Within a consumer group each partition is assigned to at most one consumer, so parallelism is capped at the partition count — here 4 — and the extra 4 instances get no partition and sit idle as standbys. The unit of parallelism is the partition, not the consumer. To raise consumer throughput you must add partitions, not consumers. Idle standbys are still useful for fast failover.

A dead-letter queue (DLQ) breaks the loop: after a configured number of delivery attempts the broker routes the failing message to the DLQ for inspection, freeing the main queue to keep processing healthy messages. Engineers can then diagnose the bad payload at their own pace. Switching to at-most-once or a short TTL silently drops the message, and adding consumers just spreads the same deserialization crash to more nodes.

The real benefits are that checkout no longer blocks on slow consumers (it returns once the event is enqueued), the queue absorbs traffic spikes and buffers work so consumers drain at their own pace, and a temporarily down consumer can recover and process the backlog without losing events. It does not guarantee lower per-request end-to-end latency, and it does not remove the need to handle duplicate deliveries.

You add a message queue when the producer should not be coupled to the consumer's availability or speed — for example offloading slow work (sending email, generating a report, running an inference job) from the request path. The queue decouples the two services so the producer can keep accepting work even if the consumer is down, it absorbs traffic spikes by buffering, and it smooths load so a slow consumer can drain the backlog asynchronously. It also enables retries and fan-out, at the cost of eventual consistency and harder end-to-end tracing.

A dead-letter queue is a separate queue where messages are routed after they fail processing a configured number of times. It solves the poison-message problem: a single malformed or un-processable message that always throws. Without a DLQ, an at-least-once queue keeps redelivering that message, the consumer keeps failing on it, and it blocks (head-of-line) or endlessly burns the consumer's capacity — and in an ordered partition it can stall everything behind it. Sending it to the DLQ after N attempts gets the bad message out of the main flow so healthy messages keep moving, while preserving the failed message (with its error context) so an operator can inspect, fix, and replay it instead of silently dropping it. So a DLQ contains the blast radius and makes failures observable, where retry-forever just amplifies the outage.

A queue decouples producer and consumer in time and rate. It absorbs traffic spikes by buffering (load levelling / backpressure), so a burst doesn't overwhelm a slow downstream; it lets consumers scale independently and process at their own pace; it improves resilience because a consumer outage just lets the backlog grow instead of dropping work; and it enables async, fire-and-forget workflows so the producer returns quickly. The cost is added latency, at-least-once delivery semantics (requiring idempotent consumers), and operational complexity.

Use the transactional outbox: write the event to an outbox table in the same local DB transaction as the business change, then have a relay process read and publish pending outbox rows. The business update and the outbox row both commit or both roll back, and relay-side retries make the publish at-least-once — no distributed transaction needed. Dual-write or write-broker-first both leave a gap where one side can be lost.

Apply backpressure: have the broker or consumer signal the producer to slow down (or have the producer check queue depth before publishing) so ingest rate is bounded by consumer capacity. The root cause is a sustained producer-consumer rate mismatch, and backpressure closes the feedback loop. Raising the queue limit only delays the OOM, a CDN is the wrong layer, and an in-memory broker doesn't change consumer throughput.

The correct statements are that at-least-once delivery means the consumer may see a message more than once and must be idempotent, that exactly-once is achievable in Kafka with idempotent producers and transactions but adds latency and throughput overhead, and that consumer-side idempotency (storing a processed event id) yields an effectively-exactly-once outcome under at-least-once delivery. At-most-once is wrong for billing — it drops charges — and committing the offset before processing gives at-most-once, not at-least-once.

False. At-least-once delivery guarantees every message arrives at least once but explicitly permits duplicates — a broker re-queues a message whenever an acknowledgement is lost or delayed, no matter how fast the consumer acks. A non-idempotent handler will corrupt data on duplicate delivery. The only safe pairing is an idempotent consumer that detects and ignores redeliveries via a dedup key or upsert.

False. Kafka assigns at most one consumer per partition within a consumer group, so with 4 partitions and 6 consumers only 4 receive messages while the other 2 sit idle. True parallelism is bounded by the partition count, not the consumer count. To increase throughput beyond it you must add partitions — a one-way, non-trivial change that affects key ordering.

At-least-once delivery happens because the broker only removes a message after the consumer acknowledges it; if the consumer processes the message but crashes (or the network drops) before the ack reaches the broker, the broker assumes failure and redelivers the message, so the same work runs twice. Exactly-once is hard and usually impractical end-to-end, so the standard answer is to make the consumer idempotent: design processing so that handling the same message twice has the same effect as handling it once. In practice that means deduplicating on a stable message id (record processed ids and skip duplicates), or making the side effect itself idempotent — an upsert keyed by the message's id rather than a blind insert, or a set rather than an increment — so a redelivery is a no-op.

A durable broker delivery flow runs in this order:

1. Producer calls send with the message payload and a target topic/queue.
2. Broker persists the message to disk and replicates it to in-sync followers.
3. Broker returns a send acknowledgement to the producer.
4. Consumer polls or receives a push notification and fetches the message.
5. Consumer processes the message and sends an ack (or commits the offset) to the broker.
6. Broker marks the message as delivered and advances the consumer's position.

A message that always throws (a 'poison pill') is requeued endlessly with no attempt limit, so it loops forever at the head of the queue and starves every other message; it needs a redelivery counter that routes to a dead-letter queue after N attempts

Requirements. A single send API fans to four channels, each fronted by a third-party provider with its own rate limit and failure profile. Two priority tiers: transactional (OTP, < 5 s) and marketing (batchable). First-class requirements beyond delivery: idempotency (never double-send), preferences/opt-out, retry, and an audit trail.

Pipeline. The send API does almost nothing synchronously — it validates and enqueues the event onto a message queue, then returns. Per-channel workers consume from their queue and call the provider. This decoupling is the backbone: it absorbs the 30K/sec peaks, and a slow provider only backs up its queue, not the others. Templating (render the message body from a template + data) happens in the worker just before send.

Dedup & idempotency. Each event carries an idempotency key (event_id:user_id:channel). Before sending, the worker does an atomic check-and-set against an idempotency store; if the key already exists, it's a duplicate and is dropped. This turns the queue's at-least-once delivery into at-most-once per recipient per channel, so an upstream retry can't double-send.

Reliability. Transient provider errors are retried with exponential backoff + jitter. A message that exhausts its retries (or is malformed/poison) goes to a dead-letter queue for inspection rather than blocking the worker. Because each channel has its own queue and worker pool, a failing SMS provider can't stall push or email.

Preferences. Before dispatch, the worker checks the user's preferences/opt-out for that channel and any per-user frequency cap (so we don't spam someone with ten pushes in a minute), plus compliance rules (quiet hours, unsubscribe for email). A suppressed notification is recorded as suppressed in the audit, not sent.

Scaling & provider limits. Each channel scales its worker pool independently. Workers pace themselves with a token bucket matched to the provider's rate limit, applying backpressure to the queue rather than overrunning the provider. Priority queues (a separate high-priority lane per channel) let an OTP jump ahead of a 50M-message marketing backlog so it still meets its < 5 s budget.

Requirements. Crawl ~1B pages in a month while being a good citizen: obey robots.txt, never overload a host, skip already-seen URLs, and re-crawl by freshness. Two dedup problems exist — exact URL dedup (don't fetch the same link twice) and near-duplicate content (mirrors/boilerplate shouldn't be stored repeatedly). Crawler traps (infinite URL spaces) are a known hazard.

URL frontier & politeness. The heart of the design. The frontier is a set of queues feeding the fetchers, with two axes: priority (important/fresh pages first) and politeness. For politeness, URLs are routed to per-host queues, and a host's queue is drained at a capped rate (with a delay between requests) — so even if a million links point at one domain, we crawl it steadily without flooding it. Expansion is roughly BFS from the seeds.

Dedup. Tracking billions of seen URLs in an exact hash set costs too much memory, so I use a bloom filter (or a hashed, sharded seen-set): probabilistic membership at a fraction of the memory, accepting a small false-positive rate (we occasionally skip a genuinely new URL — acceptable). For content dedup, I compute a fingerprint/simhash of each page and skip storing pages whose fingerprint is near an existing one, catching mirrors and trivially-different pages.

Fetch → parse → store pipeline. Distributed fetcher workers pull URLs from the frontier. Each does DNS resolution (heavily cached — DNS is a hidden bottleneck), fetches the page, parses it to extract links, runs new links through the seen-filter and enqueues the unseen ones back into the frontier, and writes the content to storage. The stages are decoupled (queues between them) so a slow remote fetch doesn't stall parsing or storage.

Storage & scaling. ~100 TB of raw content goes to an object store, partitioned by domain hash; crawl metadata (last-crawled timestamp, content fingerprint, status) lives in a separate store keyed by URL/host. Fetcher workers scale horizontally, and the frontier is partitioned across machines (typically by host, which conveniently also enforces politeness — all of one host's URLs sit on one partition).

Robustness & traps. Infinite URL spaces (a calendar's endless 'next month') are bounded with depth limits and per-host URL-count caps plus pattern detection. Slow/malicious hosts get timeouts and are de-prioritised; failed fetches are retried a few times then skipped. The frontier and seen-set are persisted/checkpointed so a worker crash doesn't lose progress or re-crawl everything.

Requirements. The system has two halves: submit (cheap, synchronous — durably record the job and return) and execute (asynchronous, on a worker). Jobs vary wildly in duration (10 ms to minutes) and need scheduling and priority. The crucial semantic is delivery: I'll deliver at-least-once and make handlers idempotent, which gives exactly-once effect even for charge-card — true exactly-once delivery across a crash boundary is not achievable, so I don't promise it.

Queue & leasing. Jobs go into a durable, replicated broker (Kafka, SQS, or a Redis stream). A worker doesn't delete on dequeue — it leases the job with a visibility timeout (or in-flight/ack semantics): the job is hidden from other workers while being processed and is only acked (removed) on success. If the worker crashes, the lease expires, the job reappears, and another worker runs it. This is what guarantees no lost jobs on crash, at the cost of possible re-delivery.

Delivery semantics & idempotency. Because re-delivery is possible, every side-effecting handler is idempotent, keyed by a job/idempotency id. For charge-card: before charging, the worker does an atomic check-and-set on idempotency_key in a store; if it's already marked done, it skips the charge and just re-acks. So the second worker that picks up the re-delivered job sees the key, does nothing, and acks — the card is charged exactly once even though the job was delivered twice.

Retry, DLQ, scheduling, priority. Transient failures retry with exponential backoff + jitter (bounded attempts). A job that exhausts retries — or is malformed/poison — is moved to a dead-letter queue for inspection rather than looping forever. Scheduled jobs go to a delay queue or a timestamp-indexed store that a promoter moves into the ready queue when due. Priority is separate lanes (high/default/low) or a priority field, so urgent jobs jump ahead.

Scaling the queue & fleet. The broker is partitioned (by job type or key hash); throughput scales with partition count and worker count. The fleet autoscales on queue depth / consumer lag. Critically, long-running jobs get their own queues and worker pools, isolated from latency-sensitive ones — so a flood of 5-minute transcodes drains its own pool and never sits in front of send-email.

Failure modes. Broker down: it's a replicated durable log, so it tolerates node loss; producers buffer briefly. Worker crash mid-job: lease expiry → redelivery (idempotency makes that safe). Poison job: capped at N attempts then DLQ, so it can't crash workers forever. Head-of-line blocking: per-type queues plus priority lanes keep one backlog from starving the rest.

Requirements. The goal is decoupling: checkout publishes an OrderPlaced event and doesn't know or care who consumes it, so adding a loyalty service later is a pure consumer change. Ordering matters within an order (reserve → charge → ship) but not across orders. And it must be durable (no order event lost) with idempotent consumers because redelivery will happen.

Backbone & topology. I use a durable, partitioned log (Kafka-style), not a fire-and-forget bus. Producers publish to a topic per event type (orders). Each reacting service is its own consumer group with its own committed offset, so every service independently reads the whole stream. Adding the loyalty service means standing up a new consumer group subscribed to orders — the producer doesn't change at all. That's the decoupling payoff.

Ordering & partitioning. I partition by order_id. All events for one order hash to the same partition and are therefore consumed in order (reserve before charge before ship). Different orders hash to different partitions, so thousands of orders are processed in parallel across partitions. The partition key is the single knob that buys per-order ordering and cross-order throughput at the same time — global ordering would serialise everything and kill throughput, which we don't need.

No-loss delivery & idempotency. The log is replicated and durable, and a consumer commits its offset only after it has successfully processed a message — so a crash mid-processing means the message is re-read (at-least-once), never skipped. Because re-processing can happen, each consumer is idempotent: it dedups by (event_id / order_id, step) against a processed-set or makes the side effect naturally idempotent, so a redelivered charge doesn't bill twice.

Independent consumer pacing. Per-group offsets plus the log's retention mean consumers run at wildly different speeds without interfering: analytics flies ahead, the slow shipping consumer lags but its backlog just sits buffered in the log — it never slows the producer or the other consumers. To speed a slow consumer up, add partitions and consumer instances within its group so its work parallelises.

Poison events & failure. A message that crashes the shipping handler every time would, if retried in place, block its partition forever (head-of-line blocking) — every later order for that partition stuck behind it. So after bounded retries with backoff, I route the poison event to a dead-letter topic for inspection and move the offset on, freeing the partition. A fully-down consumer is fine: the log retains events, so on recovery it resumes from its last committed offset and catches up — nothing is lost.

Requirements. Two delivery modes dominate: online (push immediately over a live connection) and offline (store and deliver on reconnect). Groups are bounded (~256), so a message fans out to at most a few hundred recipients. I'd confirm ordering is per-conversation (not global), and that 'exactly once' really means at-least-once delivery plus client dedup by message id. Presence and receipts are separate, lighter sub-systems.

Connection layer. Clients hold a WebSocket to a stateless gateway server (one of thousands; at ~500K connections each, ~1,000 servers cover the peak). The hard part is routing: when A sends to B, A's gateway must find B's gateway. A connection registry (user_id → gateway_id, in Redis) updated on connect/disconnect, or a pub/sub channel per user that B's gateway subscribes to, does this. The sender's gateway looks up B, forwards the message to B's gateway, which pushes it down B's socket.

Data model. Messages are stored keyed by conversation_id and ordered by a time-sortable id, in a write-optimised, horizontally sharded store (Cassandra/HBase-style), sharded by conversation. For offline delivery each recipient has a mailbox/inbox (a queue of undelivered message ids); on reconnect the client drains it. Recent history is hot (cached / on fast storage); old history ages into cold archive.

Send → deliver flow. Send → server assigns a message id and persists to the conversation log → for each recipient, if online, push to their gateway; if offline, enqueue in their mailbox. Delivery is at-least-once; the client deduplicates by message id, giving exactly-once feel. Delivery and read receipts are just small messages flowing the other way, updating per-message state. The sender never blocks on fan-out — it returns once the message is durably persisted.

Storage & fan-out scaling. ~15 TB/day shards cleanly by conversation; group fan-out (1 send → up to 256 deliveries) is done asynchronously by workers reading the conversation log, so a large group never stalls the sender. Hot recent messages live in cache; archives compress to cheap storage.

Presence & failure. Presence is a heartbeat with a short TTL in an in-memory store — the client refreshes every ~30 s, and absence of a refresh means offline; we never write every status flip to a database. If a gateway crashes, its ~500K connections drop, clients reconnect to another gateway (the registry updates), and any messages sent meanwhile are already durably in each recipient's mailbox, so nothing is lost — they arrive on reconnect.

Requirements. The workload is ~20:1 read-heavy, so the read (feed assembly) path is what we optimise. Ordering can be reverse-chronological to start (ranking is an orthogonal layer). The defining edge case is the follower skew: a normal post reaches ~200 timelines, a celebrity post reaches up to 100M — those need different handling.

Fan-out strategy. The core decision. Fan-out-on-write (push each post id into every follower's precomputed timeline) makes reads cheap but explodes on celebrities — one post = 100M writes. Fan-out-on-read (assemble at query time by pulling recent posts from everyone you follow) makes celebrity posts free but makes every feed read expensive. I'd use a hybrid: push on write for ordinary accounts, and for celebrities skip the push and pull their recent posts at read time, merging them into the timeline. This caps write amplification while keeping normal reads cheap.

Feed storage. Each user has a precomputed timeline: a capped list of recent post ids (say the latest few hundred) in Redis. Post bodies live once in a separate post store (sharded by post id). Storing ids — not full copies — per follower keeps the timeline small and means an edited/deleted post is resolved at hydration time.

Read path. On refresh: read the user's precomputed timeline (ids from pushed accounts) → pull recent post ids from the handful of celebrities they follow → merge and sort by time → hydrate the top N bodies from the post cache → return. Almost everything is served from cache, so p99 stays under 200 ms. The celebrity pull is bounded because a user follows only a few of them.

Bottleneck & scaling. The bottleneck is fan-out write amplification: 100M posts/day × ~200 avg followers ≈ tens of billions of timeline writes/day, and the tail (celebrities) is what kills pure push. The hybrid split removes the worst offenders; the remaining fan-out runs on async workers off a queue so posting never blocks; timelines and post bodies are cached and sharded. At higher load we add fan-out workers and cache capacity, and tune the push/pull follower threshold.