Background jobs are tasks that don't fit in a request-response cycle: send email, generate PDF, process upload, run report. The HTTP handler enqueues the job; a worker process executes it later.
This page covers the patterns for reliable background job systems.
Email sending, file processing, ML inference. Anything that takes more than a couple seconds shouldn't block the HTTP response.
Tasks that need retries (third-party API failures, eventual consistency).
The web tier and worker tier scale independently. Heavy job processing doesn't slow web responses.
Periodic tasks: nightly cleanup, daily reports.
HTTP request → Enqueue job → 200 OK (immediate)
Background:
Worker → Pull job from queue → Execute → Mark done (or retry)
Components:
Simple; fast; relatively easy to operate. Limited durability if Redis fails.
The job table is in your application database. Simpler architecture; no extra infrastructure.
Limitations: doesn't scale to high job volume; database load.
Built for queueing. More features (routing, dead-letter, etc.). More infrastructure to run.
Managed by cloud provider. SQS for AWS; Pub/Sub for GCP; Service Bus for Azure.
For most cloud-native shops, the managed option is right.
For high throughput with replay needs. More complex than queues.
Worker process pulls job; executes; pulls next. Not per-job process.
Workers process many jobs in parallel. Configurable concurrency per worker.
Scale horizontally. Many workers consuming the same queue.
Each worker process is independent. One worker dying doesn't affect others.
Jobs may be retried. The same job running twice should produce the same result.
Use idempotency keys, dedup logic, or idempotent operations. See IdempotencyPatterns.
Job fails; retry with exponential backoff. Don't retry forever; eventually give up.
Typical: 3-5 retries; backoff 1m, 5m, 15m, 1h, 6h.
Jobs that fail all retries go to a DLQ. Investigate; either fix and re-run, or accept failure.
When a worker pulls a job, it has a timeout to complete. If timeout exceeded, job becomes available again — another worker picks it up.
Prevents lost jobs from worker crashes. But: jobs longer than the timeout get processed twice (which is why idempotency matters).
Most queues are at-least-once. Jobs may run more than once; consumers handle duplicates.
"Exactly-once" is rare and expensive. Don't promise it; design for at-least-once.
Each job does one thing. Easier to retry; easier to reason about; easier to debug.
Job: {type: "process_upload", upload_id: "abc"} not the entire upload data. The worker fetches the data fresh; data doesn't go stale in queue.
Long jobs are problematic: visibility timeouts; lost progress on crash; harder to retry.
If a job naturally takes hours, decompose into smaller jobs.
Jobs in the queue may run with newer or older worker code. Design for compatibility:
Dominant for Ruby. Redis-backed.
Modern Node.js. Redis-backed.
Celery is more feature-rich; RQ is simpler. Both Redis-backed by default.
For batch processing. See BatchProcessingPatterns.
Java scheduling. Older but mature.
AWS SQS + Lambda; GCP Pub/Sub + Cloud Functions. Serverless workers.
Frameworks define job classes:
class ProcessUploadJob
def perform(upload_id)
# work
end
end
ProcessUploadJob.perform_async(upload_id)
Cleaner than manually serializing/deserializing.
Some jobs are higher priority. Multiple queues with different priorities; workers pull from high-priority first.
Don't run 1000 video transcodes simultaneously; not 1000 emails at once. Per-job-type concurrency limits.
Before/after job hooks. Logging; metrics; cleanup.
Periodic jobs. Many job frameworks have cron-like scheduling. See ScheduledTaskManagement.
For new applications: