Blast Radius
Limit how much can break when any one thing fails.
To an executive
When something fails, the question is how many customers it hurts. Designing so faults stay local turns a company-wide outage into a contained incident.
To an engineer
Avoid shared components that can take everything down. Use cells, shards and staged rollouts so any one fault or bad change reaches only a small slice of users.
The idea
Explosives engineers contain blast radius with distance and barriers. Systems contain it with isolation: when something fails, the damage should be local, not global.
You cannot stop every failure, but you can choose how far each one spreads. Every shared component, such as a database, a configuration system, an identity provider or a deployment pipeline, widens the potential blast radius of a single fault. Ships use bulkheads so one breach floods one compartment; architectures use cells, shards and staged rollouts for the same reason.
Why it works
Failures you cannot prevent, you can contain. Bad deployments, poison requests and runaway clients will happen. If a workload is split into ten independent cells, a fault in one leaves the other nine untouched.
Isolation turns outages into partial degradations. Losing a slice of customers for minutes is a bad day; losing all of them is a headline. Keeping faults small also keeps recovery simple.
Staged change catches mistakes while they are cheap. Many large outages start with a change pushed everywhere at once. Rolling out to one cell or region first means a bad change hurts a few users and triggers rollback before it reaches the rest.
Where it applies
- Multi-tenant platforms, where one noisy or malicious tenant should not affect the others.
- Deployment and configuration pipelines, which can push a single mistake to every server in minutes.
- Shared control planes, such as identity, DNS, feature flag and secrets services that everything depends on.
- Regional architecture, where independence between regions must be real, not just on the diagram.
How to apply
- List what is shared. Trace each request path and mark every component that all customers or regions depend on.
- Choose a partition key. Split the workload into cells along its natural grain, such as customer or tenant ID, and keep state from crossing cell boundaries.
- Keep the shared parts thin. The router in front of the cells is still shared, so keep it simple and free of business logic.
- Stage every change. Deploy and reconfigure one cell or region at a time, with automatic health checks and rollback.
- Size cells deliberately. Set a maximum cell size so the worst single failure affects an acceptable fraction of users.
Example scenario
A global service pushed configuration to every region at once from a single pipeline. One malformed value took down all regions simultaneously, and recovery was slow because every team was firefighting the same outage.
The team changed the pipeline to deploy region by region, starting with a small canary region, with automatic rollback when error rates rose. The next bad configuration reached only the canary, affected a small share of users for a few minutes, and was rolled back before anyone was paged.
Questions to ask
- If this fails at 3am, which customers are affected, and how many?
- Which shared components could still cause a global outage?
- How do we roll this change out to a small fraction first?
When not to use it
- Small systems with few users, where cells and staged rollouts add more operational cost than the risk they remove.
- Work that is inherently global, such as a single ledger that must be consistent everywhere; isolate around it instead of splitting it.
How it misleads you
Isolating so heavily that the operational cost and duplication outweigh the risk reduction. A second trap is isolation on paper only: cells that look independent but share a database, a certificate or a deployment pipeline have the same blast radius as one big system. Test the boundaries, do not assume them.
In the wild
- AWS S3 outage, February 2017A mistyped command removed far more S3 servers than intended in US-EAST-1, forcing full restarts; AWS responded by slowing capacity removal, adding minimum-capacity safeguards and accelerating work to split S3 into cells to reduce blast radius.Amazon Web Services · 2017 (opens in a new tab)
- CrowdStrike Falcon content update, July 2024A faulty content update, live for 78 minutes, crashed every online Windows host on recent sensor versions that received it; CrowdStrike's remediation includes canary and staggered deployment so future updates reach the fleet gradually.CrowdStrike · 2024 (opens in a new tab)
Further reading
- REL10-BP03 Use bulkhead architectures to limit scope of impactAWS Well-Architected Framework · Website(opens in a new tab)
- Workload isolation using shuffle-shardingColm MacCárthaigh · Article · 2019(opens in a new tab)
Last reviewed · Suggest an improvement