Scheduled tasks: scripts or jobs that run on a schedule. Daily reports, hourly cleanup, weekly billing, periodic reconciliation. Almost every system has them.
The simple version: a cron job on a server. Works for tiny systems. For real production, you need more.
# crontab -e
0 2 * * * /path/to/script.sh
Runs the script at 2am daily. Cron is fine for:
For production at scale, cron has problems:
For Kubernetes-deployed apps:
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-cleanup
spec:
schedule: "0 2 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: cleanup
image: my-cleanup:latest
restartPolicy: OnFailure
Pros: integrated with cluster; automatic restart on failure; logs available. Cons: Kubernetes-specific; cluster needs to be running.
Cloud-native cron equivalent. Triggers Lambda, Step Functions, or other AWS services.
ScheduleExpression: rate(1 hour) # or cron-like expression
Target: arn:aws:lambda:...
Serverless; managed; pay-per-invocation.
Similar; GCP-native.
Azure equivalent.
For cloud-native deploys, the cloud scheduler is usually the right choice. Less operational overhead than self-hosted.
Tasks may run twice (network retry, infrastructure restart). The same task running twice should produce the same result.
Don't:
See IdempotencyPatterns.
Some tasks should run only once even if scheduled twice. Distributed lock (Redis, database row) ensures single execution.
Logs to a central system; metrics on success/failure; alerts on missed runs.
Network blip = retry. But not all errors should retry. Decide:
If the cleanup job fails, someone needs to know. Email, Slack, page — depends on severity.
The job didn't run at all (scheduler down). Some monitoring detects this.
Job sends a heartbeat to a monitoring service after success. Service alerts if no heartbeat.
Tools: Healthchecks.io, Cronitor, Better Stack heartbeats.
Job processes a queue. Failed messages go to a dead letter queue for investigation.
Multiple instances might try to run the job. Lock prevents duplicate execution:
with redis_lock("nightly-cleanup", timeout=300):
do_cleanup()
Define schedules in deployable code (Terraform, Kubernetes manifests). Not in someone's crontab on a specific server.
Cron schedules in what timezone? Server time? UTC? Match user expectations or document explicitly.
Crontab on one server. Server dies; tasks stop. Nobody notices for weeks.
Job fails silently. Real impact only visible when something downstream breaks.
Retries cause duplicates. Daily report sent twice.
Job runs for 2 hours; fails 1.5 hours in; restart from scratch. With checkpointing, restart from where it failed.
Job scheduled every 5 minutes; sometimes takes 10. Multiple copies run simultaneously; conflict.
"Did the cleanup run yesterday?" — nobody knows.
When a job fails, manual "kick it off again." Should be one click; ideally automatic.
For new scheduled tasks:
For an existing chaotic cron-based setup, migrate one task at a time to a structured framework.