Normal Accidents
In tightly coupled, interactively complex systems, accidents are inevitable — they are normal.
To an executive
In complicated, tightly linked operations, serious failures will happen however careful people are. Plan for recovery and limit the damage instead of promising none.
To an engineer
Reduce tight coupling by adding slack, buffers and isolation between parts. Be wary of safety features that add complexity, since they can cause new kinds of failure.
The idea
Charles Perrow, a sociologist, studied accidents such as Three Mile Island and argued that some failures are not the result of a bad component or a careless operator. They are a property of the system itself.
He described systems along two dimensions. Interactions can be linear, where steps follow a predictable sequence, or complex, where parts affect each other in unexpected ways. Coupling can be loose, with slack and time between steps, or tight, where one event immediately triggers the next. When a system is both complex and tightly coupled, small failures combine in ways nobody anticipated and spread faster than people can understand them. In such systems, accidents are "normal": not frequent, but inevitable.
Perrow also warned that safety features add complexity of their own, and can become the source of new kinds of accident.
Why it works
You cannot foresee every interaction. In a system with many interacting parts, the number of possible combinations of failures outgrows anyone's ability to imagine them. Some will only be discovered when they happen.
Tight coupling removes the time to respond. Synchronous chains, shared state and automatic retries mean a fault propagates in milliseconds. There is no pause in which a person or a safeguard can intervene.
It redirects effort from prevention alone to recovery. If some accidents are inevitable, the useful questions become how to contain them and how fast you can recover, not only how to stop them happening.
Where it applies
- Microservice estates with long synchronous call chains and shared infrastructure.
- Cloud control planes and automation, where automated systems react to each other faster than humans can follow.
- Incident reviews, especially when no single component had a bug.
- Safety and resilience investments, to check whether a new safeguard adds more complexity than it removes.
How to apply
- Place the system on the grid. Ask how complex its interactions are and how tightly its parts are coupled.
- Loosen coupling where it is cheap. Replace synchronous chains with queues, add timeouts and buffers, and avoid shared mutable state.
- Simplify before adding safeguards. Each new protective mechanism is another component that can interact badly.
- Invest in detection and recovery as much as prevention: observability, fast rollback and practised incident response.
Example scenario
A cloud region outage began with a routine capacity change. The change caused a burst of retries between internal services, the retries overloaded a shared coordination service, and that service's slowness caused yet more retries. No single component had a bug; each behaved as designed.
The review did not end with "be more careful with capacity changes". The team added backoff and jitter to retries, introduced rate limits on the shared service, and split it so one client's storm could not starve the others. The next unusual interaction stayed contained to one service.
Questions to ask
- If one part of this fails, how quickly does the failure reach the next part?
- Could two small faults interact in a way nobody has designed for?
- How fast can we detect, contain and recover from something we did not foresee?
When not to use it
- Simple, linear systems. A batch job with a few well-understood steps can be made safe through careful design and testing; it does not need a theory of inevitable accidents.
- As an explanation for a plain bug. Not every incident is systemic. Sometimes the fix really is a single line of code.
How it misleads you
Fatalism. Accepting that accidents are normal should drive investment in loose coupling and fast recovery, not resignation or a shrug in the post-incident review. It can also become an excuse to skip the search for a specific cause that does exist.
In the wild
Further reading
Last reviewed · Suggest an improvement