Skip to content
Latticework
Esc
Try
All playbooks

Playbook · Run and scale · 7 steps

Handling an outage

Stabilise first, understand second, improve third.

“The cost of failure is education.”

Start with step 1 All 27 models for incident
  1. 1Occam's RazorCheck recent changes and common causes before exotic theories.Among designs or explanations that fit the facts, prefer the one with the fewest assumptions.Open
  2. 2Blast RadiusContain the impact: roll back, isolate, or shed load.Limit how much can break when any one thing fails.Open
  3. 3Circuit BreakerCut off the failing dependency so its failure stops spreading to callers.Stop calling a failing dependency for a while, so its failure doesn't become yours.Open
  4. 4Feedback LoopsIs a retry storm or other loop amplifying the failure?Outputs that circle back as inputs either stabilise a system (balancing) or amplify it (reinforcing).Open
  5. 5Graceful DegradationWhich non-essential features can we switch off to protect the core?When parts fail, keep the core experience working instead of failing completely.Open
  6. 6Swiss Cheese ModelIn the review, ask which defences failed and why they lined up.Accidents happen when holes in several imperfect defences line up at the same moment.Open
  7. 7Hanlon's RazorLook for systemic causes, not someone to blame.Never attribute to malice what can be explained by missing information, constraints, or honest mistakes.Open