Message Queues: Kafka, RabbitMQ, and the Guarantees That Matter

A user clicks "Place Order." Behind that click, several things need to happen: charge the card, reserve inventory, email a receipt, notify the warehouse, update analytics. Doing all of that synchronously makes the click slow, and one flaky downstream service — the email provider timing out — fails the whole order.

A message queue breaks that coupling. The API does the one thing the user is waiting on — record the order — then drops an OrderPlaced event on a queue and returns. Every other service reads it on its own schedule; the email service can be down for five minutes and simply catch up later. A queue turns a fragile chain of synchronous calls into independent producers and consumers that fail, retry, and scale separately.

The hard part isn't the concept — it's the guarantees, and picking the tool whose guarantees match your problem.

Two models: log vs broker#

"Message queue" covers two designs that behave differently under load — Kafka vs RabbitMQ shows it best.

Kafka is an append-only partitioned log. Producers append records to the end of a partition; consumers read forward and track their own position (an offset). Reading doesn't delete a message — it stays retained by time or size (say, 7 days), so a new consumer can rewind and replay history. Throughput scales with partition count, reaching millions of messages per second per broker.

RabbitMQ is a smart broker. A message is routed through exchanges and bindings to one or more queues, delivered to a consumer, and deleted the moment it is acknowledged. It supports rich topologies — fanout, topic, header routing, per-message TTL and priorities — optimizing for flexible delivery over raw replayable throughput.

Reach for Kafka when many consumers share a stream or replay matters. Reach for RabbitMQ when consumers pull from a shared work queue needing intricate routing or fast acknowledgement. ByteByteGo's write-up below covers this decision further.

What "delivered" actually means#

The word "delivered" hides three contracts, and picking the wrong one silently corrupts data.

• At-most-once — acknowledge before processing. A crash after the ack loses the message. Fine for metrics or log shipping where a dropped sample is invisible.
• At-least-once — acknowledge after processing succeeds. A crash before the ack means redelivery, so the consumer may process it twice. This is the practical default and demands idempotent consumers.
• Exactly-once — each message affects state precisely once despite retries. Impossible across arbitrary systems, but achievable inside a closed one.

Kafka provides exactly-once within Kafka: idempotent producers stamp a per-partition sequence number so the broker discards duplicate appends, and transactions make "commit offset + write output" atomic. A read-transform-write loop that stays inside Kafka is therefore exactly-once. The moment a side effect leaves Kafka — an HTTP call, a SQL insert — you're back to at-least-once, and the external system must dedupe itself (an idempotent upsert, or a transactional outbox).

The practical rule: assume at-least-once, and make every consumer safe to run twice on the same message. An idempotency key checked before the side effect turns "processed twice" into "processed once."

Walkthrough: an order event through a partitioned topic#

Concrete beats abstract. Take topic orders with 3 partitions and a consumer group of 3 instances, keyed by order_id so each order's events land on one partition. Watch four events arrive:

| Event | order_id | Key → partition | Offset written | Consumer | Result | | --- | --- | --- | --- | --- | | OrderPlaced | A1023 | hash → P1 | P1:40 | C1 | charge card, mark placed | | OrderPlaced | B7788 | hash → P2 | P2:88 | C2 | charge card, mark placed | | OrderPaid | A1023 | hash → P1 | P1:41 | C1 | after P1:40 — sees order exists | | OrderShipped | A1023 | hash → P1 | P1:42 | C1 | after P1:41 — valid transition |

Two things stand out. Ordering is per partition, not global: A1023's three events stay strictly ordered on P1, while B7788 on P2 runs concurrently on C2 — the parallelism you want. And if C1 crashes right after processing P1:41 but before committing its offset, it re-reads that same event on restart; since "mark paid" is a no-op if already paid, the redelivery is harmless — that idempotence is what makes at-least-once survivable.

Picking the partition key matters most: use the entity that must stay ordered (order_id, user_id, account_id). Parallelism equals partition count, so size for peak throughput plus headroom — commonly 50 to 500 per busy topic.

Durability without killing throughput#

Kafka's trick for being both durable and fast is worth knowing cold. A producer writes to the partition leader, appending to a local log that's really a page-cache write (the kernel handles the disk flush on its own timetable), then copies the record out to followers in the in-sync replica set (ISR).

The acks setting is the durability dial:

Throughput stays near 100 MB/s per partition despite replication because writes are sequential appends, sendfile() ships bytes from page cache straight to the network without a userspace copy, and batching amortizes fsync and network cost across many records. One more knob: unclean.leader.election=false refuses to promote an out-of-ISR replica, choosing brief unavailability over silent data loss.

When things go wrong: dead-letter queues#

Consumers fail, and a good policy separates two causes: a transient failure (a 503, a network blip) will succeed on retry, while a poison message (schema violation, missing account) will fail forever no matter how many retries.

Handle them differently:

A poison message that keeps retrying just adds noise, so after N attempts it moves to a dead-letter queue (DLQ), tagged with the failure reason, retry count, and original offset. A DLQ is just another topic — you can page off it, browse it in a dashboard, or resubmit once the bug is fixed. Recovery is a triage step: patch the code and replay, drop the entry if the data was simply bad, or file a ticket. Watch DLQ size like any other metric — a climbing count means the retry logic is broken, not that failures are rare. Payments need one extra guard: dedupe every replay against an idempotency key, or a resubmit could charge a card twice.

Writing to a DB and a queue atomically#

A subtle trap: saving an order to Postgres and publishing OrderPlaced to Kafka can't happen in one transaction — they're separate systems with no shared commit. Send the event first and the order save can still fail, leaving an announcement for something that never happened. Save the order first instead and the send can fail, and the event is gone for good.

The transactional outbox sidesteps this: the business logic inserts a row into an outbox table as part of the same transaction that saves the order, so both writes succeed or fail together. A separate relay then drains the outbox into Kafka on its own schedule — Debezium tailing the write-ahead log via CDC is low-latency and adds no load to the app; a plain poller is easier to build but costs roughly a second of lag and extra read pressure on the primary. Either way the relay publishes at-least-once with an idempotency key, and deletes outbox rows once Kafka confirms the publish. Partition by aggregate_id to keep per-entity ordering.

How this shows up in interviews#

Message queues appear in almost every system-design interview the moment you say "asynchronously," and the interviewer will probe the boundary between a clean diagram and an operable system:

• "Your consumer group is lagging 20M messages and growing — what do you do?" Compare consumer throughput to producer rate; scale consumers up to (never beyond) partition count, or add partitions if already there. Parallelize inside each consumer with a worker pool that commits offsets only after a batch succeeds, then chase the real bottleneck — usually a slow downstream DB fixed with bulk writes and connection pooling.
• "How do you get ordering and parallelism at the same time?" The partition-key answer from the walkthrough, plus the failure mode: a hot key overloading one partition, fixed by splitting that key across several buckets and living with looser ordering inside each one.
• "Kafka, Pulsar, or SQS?" SQS is fully managed but caps FIFO at 3k msgs/sec per group with no replay; Pulsar separates compute from storage for independent scaling; Kafka plus tiered storage is the safe default once volume gets large.

Walk one event end-to-end, name the delivery guarantee, and say which trade-off you'd reverse if the workload changed — that separates a memorized diagram from someone who has run one of these.

Further learning#

• What is a MESSAGE QUEUE and Where is it used? — Gaurav Sen's visual intro to why queues exist.
• How to Choose a Message Queue? Kafka vs RabbitMQ — ByteByteGo's log-vs-broker decision guide.
• Explore Message Queues interactively on YeetCode — step through the roadmap topic with prompts and checkpoints.
• Related roadmap topics: Consistent Hashing, API Gateway, Database Indexing, and Scaling.

FAQ#

Is Kafka or RabbitMQ better for my system?#

Neither is universally better. Lean Kafka when several services consume the same event stream, when replay matters, or once volume hits hundreds of thousands of messages per second — it retains messages and scales throughput with partitions. Lean RabbitMQ for work queues with varied consumers, complex routing, per-message priorities or TTLs, and fast acknowledgement, since its broker deletes each message on ack. Rough heuristic: log-shaped, replayable data leans Kafka; task-shaped, routed work leans RabbitMQ.

Can Kafka really guarantee exactly-once delivery?#

Only within Kafka's own boundary. Idempotent producers use per-partition sequence numbers so the broker rejects duplicate appends, and transactions atomically commit offsets alongside produced output, so a consume-transform-produce loop that stays inside Kafka is exactly-once. Once a side effect touches an external system — an HTTP endpoint, a SQL database — the guarantee no longer holds, because Kafka can't coordinate a transaction with a system it doesn't control. There you fall back to at-least-once and make the external write idempotent, or use a transactional outbox.

Why do consumers need to be idempotent?#

Because the default guarantee is at-least-once. A consumer might finish processing and then crash before committing its offset, so on restart it re-reads and reprocesses that same message. If the handler is idempotent — running it twice produces the same state as once — the redelivery is harmless. The usual fix is a dedupe check on a business key (order ID, payment ID) before the side effect, so a retry becomes a no-op instead of a double charge.

What is a dead-letter queue and when should a message go there?#

A separate queue or topic where messages land after they repeatedly fail. Transient failures — a timeout, a temporary 503 — get retried in place with backoff, not dead-lettered, or the DLQ fills with noise that would've succeeded anyway. Poison messages — a schema violation, a missing account — never succeed, so after N attempts they go to the DLQ tagged with reason, retry count, and original offset. Ops then triages: fix and replay, discard, or escalate. A growing DLQ is an alarm, not a normal state.

How does Kafka process messages in parallel while keeping order?#

Order holds only within a single partition, never across partitions, so the trick is the partition key. Key each message by the entity that must stay ordered — user ID, order ID, account ID — and every event for it lands on one partition, read in sequence. Different entities spread across partitions and process concurrently, so parallelism equals partition count. The classic pitfall is a hot key, like one very active user, overloading a single partition. The fix is to split that one key across several buckets and accept looser ordering inside each bucket, or to run an ordered stage first and fan out to parallel workers afterward.