On-call: someone is responsible for production at all times. When something breaks, they're paged. They diagnose; they mitigate; they involve others if needed.
Done well, on-call is sustainable. Done poorly, it's miserable and people quit.
This page covers the practices that work.
The on-call's job:
The first priority is mitigation. Root cause investigation comes after.
One person at a time. Common for small teams.
Pros: simple; one point of contact. Cons: long shifts hurt; primary alone if complex incident.
Primary takes pages; secondary backs up. Both rotate.
Pros: complex incidents have backup; reduced pressure on primary. Cons: more rotation slots needed.
Different time zones cover different hours. EU primary 8am-5pm; US primary 5pm-2am; APAC overnight.
Pros: no one paged at 3am. Cons: requires global team; handoffs are weak points.
For most teams, primary + secondary is the right model.
Sunday to Sunday, or Monday to Monday. Common.
Each person on call for one day. Reduces fatigue but more handoffs.
Two-week rotations. Spreads incidents across more time but increases per-rotation burden.
For most teams, weekly is the sweet spot.
The hardest part. Alarms must:
If automation can handle it (auto-scaling, auto-restart), let automation handle it. Don't page humans for things they can't actually do.
Each alert has a runbook. The on-call knows what to do.
Page-worthy: real customer impact; high severity. Non-paging: warning; investigate during business hours. Tickets: low priority; backlog.
The default should be ticket; escalate as needed.
Each alert has an owning team. Stray alerts that nobody owns get ignored.
Alarms that fire often without action become noise. Tune until each alert is actionable.
The single biggest on-call failure mode. Symptoms:
Causes:
Fix: ruthless tuning. Every fired alert should have led to action. If not, remove or downgrade.
The 80/20 rule: a few alarm types cause most of the noise. Eliminate them and on-call quality dramatically improves.
Each alert points to a runbook. The runbook says: when this happens, here's how to respond.
Good runbooks:
See RunbookAutomation.
When to escalate:
Escalation paths defined in advance: secondary on-call, manager, specific subject-matter experts. Not "Bob, but he's on vacation."
Common mitigation tools:
Practice these during quiet times. Don't first try them during incidents.
For high-severity incidents:
Owns the incident. Coordinates. Not the same as the technical lead.
Periodic updates (every 15-30 min for active incidents). Even if nothing has changed: "still investigating." Silence is worse than slow progress.
Status page; sometimes targeted emails. See StatusPageBestPractices.
Slack channel for the incident. Everyone involved. After-action report.
After any meaningful incident:
Focus on system causes, not individual blame. People made decisions with the information they had. The system shouldn't have allowed the failure.
What happened, when, in what order. Reconstruct.
Not just "Bob deployed the bad code." Why did the bad code pass review? Why didn't tests catch it? Why was monitoring late?
Specific changes with owners. Not "we should improve testing"; specific tests, specific tools, specific timeline.
Postmortems shared widely. Other teams learn from the incident.
On-call is real work; compensate appropriately.
Companies that don't compensate on-call lose engineers to companies that do.