Asynchronous message queues and event streaming brokers decouple distributed services, buffer bursty traffic, and guarantee durable event delivery across microservices. Selecting the right message broker topology—traditional AMQP message queues (RabbitMQ), append-only distributed event logs (Apache Kafka, Apache Pulsar), or in-memory publish-subscribe channels (Redis Streams)—dictates system throughput, message ordering, and partition fault-tolerance.
This guide provides an architectural comparison of modern message brokers, delivery semantics (at-least-once, exactly-once), consumer group partition rebalancing, and dead-letter queue (DLQ) retry topologies.
+-----------------------------------------------------------------------------------------+
| MESSAGE BROKER SELECTION MATRIX |
+-----------------------------------------------------------------------------------------+
| Broker / Platform | Architecture Model | Message Ordering | Throughput (msgs/sec) | Latency Profile |
+--------------------+----------------------+---------------------+-----------------------+-----------------+
| RabbitMQ (AMQP) | Push Queue / Router | Per-Queue FIFO | ~ 50,000 / sec | Ultra-Low (<2ms)|
| Apache Kafka | Append-Only Log Pull | Per-Partition Order | > 1,000,000 / sec | Low (5ms - 15ms)|
| Apache Pulsar | Multi-Tier (Bookie) | Key-Shared / Order | > 500,000 / sec | Low (5ms - 10ms)|
| Redis Streams | In-Memory Log | Stream ID Order | > 250,000 / sec | Microsec (<1ms) |
+-----------------------------------------------------------------------------------------+
Message Queue (RabbitMQ) vs. Event Log (Kafka):
Queue Pattern (Work Queue / Competing Consumers):
[ Producer ] ---> [ Exchange ] ---> [ Queue: orders ] ---> [ Consumer 1 (Pop & Ack) ]
---> [ Consumer 2 (Pop & Ack) ]
(Message deleted immediately after acknowledgment)
Log Pattern (Replayable Distributed Partition):
[ Producer ] ---> [ Partition 0 Log: [0][1][2][3][4] ] ---> [ Service A (Offset: 4) ]
---> [ Service B (Offset: 2) ]
(Messages retained on disk; consumers maintain independent read offsets)
+---------------------------+-----------------------------------+------------------------+
| Delivery Semantic | Producer / Broker Mechanism | Consumer Requirement |
+---------------------------+-----------------------------------+------------------------+
| At-Most-Once | Fire-and-forget; no retries | Tolerates data loss |
| At-Least-Once (Standard) | Retry on unacknowledged timeout | Idempotent consumers |
| Exactly-Once (EOS) | Transactional coordinator + WAL | End-to-end atomic state|
+---------------------------+-----------------------------------+------------------------+
Because network partitions cause duplicated acknowledgments in at-least-once delivery, consumers must enforce idempotency via natural deduplication keys or idempotency tables:
-- Atomic Check-and-Insert Idempotency Pattern
INSERT INTO processed_events (event_id, processed_at)
VALUES ('evt_82739182', NOW())
ON CONFLICT (event_id) DO NOTHING;
When consumer processing slows down, unmanaged message accumulation risks out-of-memory crashes.
basic.qos(prefetch_count=50), pulling messages only as processing capacity permits.