Idempotency

Design operations so doing them twice has the same effect as doing them once.

With an idempotency key, a retried request returns the original result instead of repeating its effect.

To an executive

Retries are unavoidable, and without safeguards they cause double charges, duplicate orders and angry customers. Safe repeats protect revenue and trust.

To an engineer

Make operations safe to repeat. Have callers send a unique request identifier, record it atomically with the change, and return the original result on any retry.

The idea

In mathematics, an operation is idempotent if applying it twice gives the same result as applying it once: f(f(x)) = f(x). In distributed systems, retries are unavoidable. Networks time out, replies get lost and brokers redeliver messages, and the caller often cannot tell whether the first attempt succeeded.

Exactly-once delivery is practically impossible. An exactly-once effect is achievable, through idempotency keys, deduplication and operations that are naturally idempotent.

AWS's Builders' Library describes the common pattern: the caller sends a unique request identifier, the service records it atomically with the change it makes, and any retry with the same identifier returns a response with the same meaning as the original instead of repeating the work.

Why it works

It makes retries safe. If repeating a call cannot cause harm, callers can retry freely after timeouts, and transient faults stop becoming incidents.

It removes the "did it work?" problem. The caller does not need to know whether the first attempt succeeded; it just tries again with the same key.

It simplifies consumers. Message consumers that tolerate duplicates do not need fragile exactly-once machinery.

Where it applies

Payment, order and booking APIs, where duplicates cost real money or stock.
Event and message consumers, which will see redelivered messages sooner or later.
Resource creation, such as provisioning infrastructure, where a retry must not create a second resource.
Batch jobs and data pipelines, which should be safe to rerun after a partial failure.

How to apply

Prefer naturally idempotent operations. "Set the status to shipped" is safe to repeat; "add 1 to the count" is not.
Require a client-generated key for operations that are not naturally idempotent, and reuse it on every retry.
Store the key with the effect, atomically. Record the key and make the change in one transaction, and keep keys for longer than any retry window.
Return the original outcome. On a duplicate, return the same result, and reject a reused key that comes with different parameters.

Example scenario

A mobile app retried a payment request after a timeout on a poor connection. The first request had actually succeeded; only the reply was lost. The retry charged the customer a second time, and support had to issue refunds by hand.

The team made the app generate an idempotency key when the customer pressed "pay" and send it on every attempt. The payment service stored the key in the same transaction as the charge, and returned the stored result for any repeat. Retries became safe, and duplicate charges stopped.

Questions to ask

What happens if this request or message is processed twice?
Where is the idempotency key, who generates it, and how long do we keep it?
Can we make this operation naturally idempotent, so it needs no key at all?

When not to use it

Read-only operations, which are already safe to repeat.
Operations where each call is meant to be distinct, such as sending a new message each time a user presses "send again"; use a new key for each intended action.

How it misleads you

Assuming a broker's "exactly-once" setting removes the need for idempotent consumers. Such guarantees usually hold only within the broker, not across your database and external calls. Another trap is keeping keys for less time than clients may retry, so a late retry slips through as a new request.