Margin of Safety
Build in buffer for what you can't predict. The future is usually worse than the plan.
To an executive
Plans assume things go well, and they rarely do. Spare capacity, time and budget turn surprises into minor events instead of crises.
To an engineer
Leave headroom in capacity, deadlines and error budgets. Size for load and failure beyond the forecast, and treat running at full utilisation as a risk.
The idea
Bridge engineers design for loads well beyond the traffic they expect. Benjamin Graham applied the same idea to investing: buy with a buffer between price and value, so that your inevitable errors of judgement do not ruin you.
In systems, margin of safety means headroom: capacity, time, budget and error-budget buffers that absorb surprises without failure. Google's SRE practice makes one version explicit. An error budget is the amount of unreliability a service can absorb in a period, and while some budget remains the team can keep shipping.
Why it works
Estimates are wrong, and usually in the same direction. Load grows faster than forecast, migrations uncover more data than expected, and integrations take longer than planned. A buffer turns those surprises into inconveniences.
Systems behave badly near their limits. Queues and latency grow sharply as utilisation approaches 100%, so a system at 85% has far less real room than the remaining 15% suggests. Headroom keeps you on the flat part of the curve.
Buffers buy time to react. Alerts that fire well before a hard limit give people hours to act instead of seconds.
Where it applies
- Capacity planning for compute, databases, network links and quotas.
- Project and migration plans, especially integration and data-migration phases.
- Reliability targets, where an error budget makes the buffer explicit and shared.
- Vendor limits and contracts, such as API rate limits, licence counts and quotas needed for failover.
How to apply
- Find the real limit. Load test to the breaking point rather than trusting a datasheet.
- Set a target utilisation well below that limit, based on how spiky the load is and how fast you can add capacity.
- Size the margin to the uncertainty and the cost of failure. A payment system deserves more headroom than an internal report.
- Alert on the margin, not the limit. Page people when headroom shrinks, not when it is gone.
- Protect the buffer. Review what is consuming it, and resist quietly spending it on new features.
Example scenario
A database ran at 85% CPU on a normal day. Nobody worried, because it had never fallen over. Then a routine marketing email brought a burst of visitors, the database hit 100%, queries piled up and the site went down.
The post-incident review set a target of 60% peak utilisation, added read replicas, and moved the alert threshold to 70%. A larger campaign the following quarter caused a visible bump on the graphs and nothing else.
Questions to ask
- How much headroom do we have today, and what is eating into it?
- How wrong could our load or effort estimate be, and in which direction?
- What happens at twice the expected load, or with half the expected time?
When not to use it
- When capacity is elastic and fast. If you can scale out in seconds and pay per use, a large standing buffer may be wasted money; keep the margin in limits and quotas instead.
- Cheap, reversible experiments, where the cost of failure is low and speed matters more than safety.
How it misleads you
Excessive buffers everywhere lead to waste, and padding every estimate hides real uncertainty instead of exposing it. Size the margin to the uncertainty and the cost of failure. The opposite trap is treating a buffer as spare capacity and spending it, until the day the surprise arrives and nothing is left.
In the wild
Further reading
Last reviewed · Suggest an improvement