A dead letter queue (DLQ) is where messages go when they fail repeatedly. Without a DLQ, repeatedly-failing messages either retry forever (consuming resources) or get dropped silently (data loss).
The DLQ is the safety net. Done well, it surfaces failures for human investigation. Done poorly, it accumulates without anyone looking, hiding the same problems.
Queue → Worker tries to process → Failure → Retry → ... →
Max retries exceeded → Message goes to DLQ
The original queue keeps moving; the failed message is preserved separately for investigation.
Bug in worker code that always fails on this message. Retry won't help.
Worker takes too long; visibility timeout fires; eventually exhausts retries.
Worker can't parse; rejects; retries fail the same way.
Worker runs out of memory; retries hit the same wall.
AWS SQS:
Source queue → max_receives = 5 → DLQ
After 5 failed receives, message moves to DLQ automatically.
Similar in GCP Pub/Sub, Azure Service Bus.
Implement in worker code:
try:
process(message)
except Exception as e:
if message.retry_count < MAX_RETRIES:
requeue(message, delay=backoff(message.retry_count))
else:
dlq.send(message, error=str(e))
Less convenient; more flexible.
The "DLQ" is a database table for failed jobs:
INSERT INTO failed_jobs (job_data, error, attempts, failed_at)
VALUES (...);
Queryable; investigatable; retentioned.
Why did this message fail? Look at:
Common causes:
Group DLQ messages by error pattern. Many similar errors usually mean one bug; one fix resolves many messages.
After fixing the bug, re-process the messages.
for message in dlq.read_all():
main_queue.send(message)
For some failures, replay won't help (message is fundamentally bad). Discard those after deliberate decision.
Some messages should never have been processed. Discard explicitly with logging:
log.info(f"Discarding bad message: {message.id}")
dlq.delete(message)
A DLQ that's accumulating is signaling a problem. Alert when:
Alert noise: alert on patterns, not on every individual message.
Weekly or daily: how many messages in DLQ? Investigate. Fix or discard.
A DLQ left to accumulate becomes useless — too many messages to triage.
DLQ messages have storage cost. Set retention:
Without retention, DLQ grows forever.
Each DLQ has documentation: what's its source queue, what does success vs. failure look like, who owns triage?
Multiple DLQs by failure type:
Different triage process per tier.
Don't just store the failed message. Store also:
Investigation is much easier with context.
After fixing the bug, replay all messages in DLQ. Be careful: re-running them may have side effects (notifications, billing). Verify before mass replay.
Sometimes only some messages should replay. Filter:
for message in dlq.read_all():
if message.error_type == "transient_network_error":
main_queue.send(message)
else:
log.info(f"Skipping non-replayable: {message.id}")
When a new error pattern appears in DLQ, page the on-call. New errors usually mean new bugs.
For new queue-based systems: