What Is Toil in SRE? Spotting and Reducing Operational Toil
Toil is manual, repetitive, automatable operations work that grows with your service. How SRE teams define it, measure it, and engineer it away.
In site reliability engineering, toil is the kind of operational work that is manual, repetitive, automatable, and scales linearly as a service grows — while producing no lasting improvement. Restarting the same stuck process every week, hand-approving routine access requests, or copying values between dashboards to build a status report are all toil. The concept matters because unchecked toil quietly consumes an operations team’s time until there is none left for the engineering that would make the system better.
Toil is not the same as “work you dislike”
The term comes from the SRE discipline, and it has a specific meaning. Toil is not just boring work, and it is not all operational work. Writing a design doc, attending a planning meeting, or debugging a novel outage can be tedious, but none of them are toil. The SRE definition describes toil by a set of characteristics. The more of these a task has, the more confidently you can call it toil:
- Manual. A human has to run it, even if “running it” means executing a script by hand.
- Repetitive. You have done it before and will do it again. A one-off task is not toil.
- Automatable. A machine could do it just as well. If the task genuinely requires human judgment, it is not toil.
- Tactical. It is interrupt-driven and reactive rather than planned and strategic.
- No enduring value. After you finish, the system is in the same state it was before. Nothing has permanently improved.
- Scales with the service. If traffic, users, or hosts double, the amount of this work roughly doubles too.
That last property is the dangerous one. Engineering work scales sublinearly: one well-built automation handles ten servers or ten thousand. Toil scales at least linearly, so a growing service with a toil-heavy operating model eventually needs headcount growth just to stand still.
Toil vs overhead vs engineering
It helps to place toil alongside the other buckets of an operations team’s time.
| Toil | Overhead | Engineering | |
|---|---|---|---|
| Example | Manually rotating certificates | Team meetings, HR tasks | Building automated cert rotation |
| Directly tied to running the service | Yes | No | Yes |
| Repetitive | Yes | Often | No |
| Leaves the system better afterward | No | No | Yes |
| Scales with service growth | Linearly or worse | Roughly flat | Sublinearly |
Overhead is administrative work that comes with having a job. It is not toil, though it also does not improve the service. Engineering is work that produces a permanent improvement — reducing future toil, raising reliability, or adding capability.
Why a toil budget exists
Google’s SRE practice popularized the idea of capping toil, with a commonly cited guideline that SREs should spend no more than half their time on operational work, keeping the rest for engineering. The exact number matters less than the principle: toil should be a budgeted, measured quantity, not whatever is left after the pager stops ringing.
A cap creates a feedback loop. When a team’s toil exceeds its budget, that is a signal to redirect effort — push work back to development teams, pause feature launches, or prioritize automation. It plays a role similar to an error budget, which ties release velocity to the reliability targets defined by SLOs and SLIs.
Excessive toil has costs beyond lost engineering time:
- Burnout and attrition. Repetitive, interrupt-driven work is draining, and skilled engineers leave teams where it dominates.
- Slower progress. Automation projects never get started because there is never a quiet week to start them.
- More errors. Humans repeating the same steps make mistakes that scripts do not.
- Bad precedent. If ops absorbs every manual request, product teams have no incentive to build self-service tooling.
How to measure toil
You cannot reduce what you do not track. A few practical approaches:
- Tag tickets and pages. Mark each operational task as toil or not when it is closed. Over a few weeks the pattern becomes obvious.
- Time-box surveys. Ask engineers to estimate the share of their week spent on toil. Imprecise, but cheap and surprisingly informative.
- Mine on-call data. Alerts that are acknowledged and resolved with the same manual action every time are toil hiding in your on-call rotation.
- Track frequency and duration. A task that takes five minutes but happens forty times a week deserves more attention than a two-hour task done once a quarter.
The goal is a ranked list of the biggest toil sources by total time consumed, so automation effort goes where it pays back fastest.
Strategies for reducing toil
Not all toil should be automated immediately. Some is rare enough that automating it costs more than doing it by hand. For the rest, there is a common progression:
- Document it first. Write a clear runbook. A good runbook is often the first step toward a script, because it forces you to make every step explicit.
- Script the steps. Turn the runbook into a tool an engineer runs with one command. Still manual, but faster and less error-prone.
- Make it self-service. Let the people who request the work trigger it themselves, with guardrails, instead of filing a ticket.
- Automate the trigger. Let the system detect the condition and run the fix on its own, with alerting only when the automation fails.
- Eliminate the cause. The best outcome is removing the need entirely: fix the memory leak instead of automating restarts.
Many modern infrastructure practices are, at heart, toil-reduction tools. Infrastructure as code replaces click-through provisioning. GitOps turns deployments into reviewed pull requests rather than manual commands. A Kubernetes operator encodes an operator’s routine actions as a control loop that runs continuously. Scheduled maintenance can move from someone’s calendar into a cron job — and our cron parser helps verify the schedule says what you meant.
Watch for automation that becomes toil
Automation is software, and software needs maintenance. A pile of brittle scripts that break every time an API changes can generate as much toil as the manual work they replaced. Treat automation with the same care as production code: version control, tests, ownership, and monitoring. Post-incident reviews, especially blameless postmortems, are a good place to notice when an automation has started causing more work than it saves.
The takeaway
Toil is manual, repetitive, automatable work that keeps a service running without making it better, and it grows in step with the service. It is different from overhead and from genuine engineering. Measure it, give it a budget, rank its sources by total time consumed, and work through a progression from runbook to script to self-service to full automation — or better, remove the root cause. Teams that keep toil in check have the time to build systems that need less of it.
Tagged
Keep reading
Chisato · · 4 min read What Is a Cloud Landing Zone?
A cloud landing zone is a pre-configured, secure baseline environment for provisioning new cloud accounts and workloads at scale. How it works.
The Lycoris Team · · 4 min read What Is an Availability Zone? Cloud Regions Explained
An availability zone is an isolated data center (or cluster of them) within a cloud region, designed so one zone's failure doesn't take down another.
Chisato · · 4 min read RPO vs RTO: Setting Disaster Recovery Targets
RPO limits how much data you can afford to lose; RTO limits how long you can afford to be down. How the two targets shape a disaster recovery plan.