All playbooks
Playbook · Design and build · 8 steps
Designing for failure
Decide how the system fails before production decides for you.
“…complex systems run as broken systems.”
- 1Everything Fails, All the TimeList what will fail — disks, networks, regions, dependencies — and assume each will.At scale, failure is the normal state. Design for it rather than hoping to prevent it.Open
- 2Single Point of FailureWhere does one failure stop everything, and which of those do we accept?Any component whose failure stops the whole system is a risk you've chosen, knowingly or not.Open
- 3Circuit BreakerWhich dependencies need a timeout, a breaker, and a fallback?Stop calling a failing dependency for a while, so its failure doesn't become yours.Open
- 4BackpressureWhen a consumer falls behind, how does it slow the producer down?When a consumer cannot keep up, it must be able to slow the producer — or the system falls over.Open
- 5IdempotencyCan every retry and redelivery happen safely twice?Design operations so doing them twice has the same effect as doing them once.Open
- 6Margin of SafetyHow much headroom do we keep for the load we didn't forecast?Build in buffer for what you can't predict. The future is usually worse than the plan.Open
- 7Normal AccidentsWhere is the system tightly coupled enough that small faults could combine?In tightly coupled, interactively complex systems, accidents are inevitable — they are normal.Open
- 8Antifragility and Chaos EngineeringWhich failures will we inject on purpose to prove all of this works?Some systems get stronger from stress. Inject failure deliberately to find weaknesses before they find you.Open