Everything Fails, All the Time

The idea

Werner Vogels has long stressed that with enough components, something is always failing: disks, networks, servers, even whole data centres. The question is not whether, but how often and how badly.

Looking back on ten years of AWS, he described the lesson as building systems that embrace failure as a natural occurrence. At scale, even very rare faults become routine, and many cannot be anticipated. The design goal shifts from preventing every fault to containing faults and recovering from them automatically, while limiting the blast radius so the rest of the system stays healthy.

Why it works

Rare events become daily events at scale. A fault that hits one machine in a million a day will hit someone every day once you run enough machines. Planning for "if" becomes planning for "when".

Prevention has limits; recovery does not. You cannot foresee every failure, but you can design systems that detect, isolate and recover from failures you did not foresee.

Rehearsed recovery is fast recovery. Failover that runs automatically and is tested regularly works in minutes or seconds. Failover that exists only in a runbook tends to fail when it is finally needed.

Where it applies

• Distributed and cloud systems, where dependencies span networks, zones and services.
• Design reviews, to check every dependency has a failure mode that has been thought through.
• Incident follow-up, to move from "prevent this" towards "survive anything like this".
• Capacity and resilience planning for peak loads.

How to apply

1. List every dependency and ask what users see when each one fails, slows down or returns rubbish.
2. Contain failures: timeouts, circuit breakers, bulkheads and cells, so one fault does not cascade.
3. Recover automatically, with health checks, redundancy across zones and automated failover.
4. Rehearse regularly, through game days or fault injection, and fix what the rehearsal exposes.
5. Track time to recovery as closely as uptime.

Example scenario

A team designed around a database failover that would "never happen". It happened during peak trading, and the manual runbook took 90 minutes, much of it spent finding the right people and credentials.

Afterwards they automated failover, added health checks that triggered it, and rehearsed it every month in production hours. The next real failure was handled in under a minute, and most users did not notice.

Questions to ask

When not to use it

• Small or internal systems where a brief outage is acceptable and much cheaper than redundancy.
• Batch work that can simply be rerun, where retrying later is a perfectly good recovery strategy.

How it misleads you