Toil, in SRE terminology: manual, repetitive, automatable operational work that doesn't scale. The manual database failover; the weekly backup verification; the incident-after-incident with the same root cause.
Reducing toil is a primary SRE practice. Without it, operational load grows with the size of the system. With it, the team scales beyond linear.
The Google SRE definition. Toil is work that's:
If your service handles 10× more traffic, does this work increase 10×? If yes, it's toil.
What's NOT toil:
These have enduring value or aren't repetitive.
Without toil reduction, headcount grows with system size. Eventually unsustainable.
Manual repetitive work is unfulfilling. Engineers burn out. See BurnoutPreventionInTech.
Toil consumes time that could be spent on improvements. Operations falls behind.
Manual work has more errors than automation. Toil produces incidents.
Google SRE: limit operational load to 50% of an SRE's time. The other 50% is engineering — building tools, reducing toil.
If toil exceeds 50%, the team can't keep up. Either:
The 50% target is a forcing function. When toil rises above it, prioritize reduction.
What did the on-call do this week? Categorize:
The toil category gets attention.
Same alert fires monthly with the same response. That's toil — the response should be automated.
Manual production deploys are pure toil. Automate.
Capacity adjusted by humans on schedule. Auto-scaling.
Manual checks that things are running. Automated monitoring with alarming.
Manual setup of new accounts, access, etc. Self-service or automated.
The on-call follows the runbook; automation can follow the same runbook. See RunbookAutomation.
Instead of tickets to the platform team for common requests (new database, scaling change, access grant), self-service portals.
Tools that make manual operations one-click instead of multi-step. Click-to-deploy, click-to-rollback, click-to-scale.
The best toil reduction is making the failure unnecessary. Database failover toil? Multi-master so failover is automatic. Backup verification toil? Continuous verification.
Some operational complexity reflects product complexity. Simplifying the product reduces toil.
The math: cost of automation vs. recurring cost of manual work.
Plus second-order effects: error reduction, speed improvement, engineer satisfaction. Often makes automation worth more than the raw math.
Once-a-year manual work. Cost of automation exceeds savings.
Decisions requiring expertise. Automation paging a human is fine; automation deciding is risky.
Drop database, kill cluster, etc. Even if rare and repetitive, the failure mode is too severe.
The work hasn't stabilized. The "right" automation is still emerging. Premature automation locks in the wrong pattern.
Reducing toil sometimes means:
These can be politically charged. Frame as: "we're freeing up capacity for higher-value work" rather than "your job is being automated."
Manual capacity changes → automatic based on metrics.
Manual restart of stuck services → liveness probes + automatic restart.
Tickets for new resources → Terraform module that engineers run.
Manual cert installs → ACME (Let's Encrypt) or AWS Certificate Manager.
Manual backup verification → continuous validation.
For teams looking to reduce toil:
This is ongoing work, not a project. Toil constantly emerges; reduction must be constant.