Emergence

The idea

Ant colonies, traffic jams and markets all show behaviour that no individual part intends. No single car causes a phantom traffic jam; it comes from how drivers react to each other. The whole has properties the parts do not.

Distributed systems are emergent in the same way. Each service can pass its health checks while the system as a whole fails, through queue build-ups, cascading timeouts, retry storms or metastable states, where a system stays stuck in overload even after the trigger has gone. Richard Cook makes the same point about safety: it is a property of the whole system, not of any single component or person.

Why it works

Interactions, not parts, cause many failures. Timeouts that are each reasonable can combine badly. Looking at parts one at a time cannot reveal that.

Scale changes behaviour. Effects that are negligible with a few nodes or users, such as coordination traffic or contention, can dominate at scale.

Naming it changes where you look. Accepting that system behaviour is emergent moves attention from component dashboards to end-to-end signals and the links between parts.

Where it applies

• Incident analysis, where a single "root cause" rarely explains what happened.
• Observability design, where user-facing service level objectives matter more than per-service health.
• Capacity and scaling work, where load tests at realistic scale find behaviour that unit tests cannot.
• Organisational change, where team behaviour emerges from incentives and structure, not from any one person.

How to apply

1. Measure the whole. Alert on end-to-end outcomes, such as successful checkouts, not only component metrics.
2. Test interactions under load. Run load tests and fault-injection experiments at production-like scale.
3. Look for loops and amplifiers. Retries, autoscaling and caches can all turn a small trigger into a large failure.
4. Review incidents for combinations. Ask which several conditions had to line up, rather than which one component broke.

Example scenario

Every service dashboard was green, yet checkout success had dropped sharply. Each of four services in the checkout path had a sensible timeout on its own, but the caller's timeout was shorter than the sum of the timeouts beneath it. Under real traffic, requests were abandoned at the top while work carried on below, and retries doubled the load.

The team added an end-to-end checkout success objective with its own alert, aligned timeouts so each caller waited longer than its callees, and added a retry budget. The next time latency rose in one service, the alert fired on the user impact within minutes, and the retries no longer amplified it.

Questions to ask

When not to use it

• Simple, linear failures, such as a full disk or an expired certificate, where a direct cause is the honest explanation.
• As an excuse for not understanding a system; "it is emergent" is the start of an investigation, not the end.

How it misleads you