Idempotency

Concepts Covered

• Retry uncertainty
• Duplicate side effects
• Idempotency keys
• Natural idempotency
• State transition guards
• Idempotent responses
• Retention windows
• Downstream deduplication

Definition

Idempotency means applying the same logical operation more than once has the same intended effect as applying it once.

In backend systems, idempotency exists because clients and services often do not know whether a request failed before or after the server committed work.

The client sees a timeout, but the server may have already inserted the row, charged the card, sent the message, or created the like.

Idempotency turns that uncertainty into retry safety.

The Pain That Forces Idempotency

Distributed systems fail in awkward places. A request does not simply "succeed" or "fail." It can succeed on the server while the client sees failure.

Example timeline:

1. User taps Like.
2. Client sends POST /likes.
3. Server inserts the like.
4. Server starts sending the response.
5. Mobile network drops.
6. Client never receives the response.
7. Client retries POST /likes.

From the client's perspective, retrying is reasonable. From the server's perspective, the like may already exist.

Without idempotency, that retry can create duplicate side effects.

The Naive Version

A naive API treats every request as new work:

POST /likes
insert into post_likes(user_id, post_id)
increment like_count
publish LikeCreated
return success

That looks fine until the client retries.

If the server runs the same logic again, it may insert another row, increment the counter twice, and publish two events.

The bug is not "the client retried." Retrying is necessary in unreliable systems. The bug is that the server did not know how to recognize repeated intent.

Mental Model

Idempotency is not:

do nothing if this exact HTTP packet appears again

It is:

identify the user's logical intent
remember the result of processing it
make repeated attempts converge on the same outcome

For a like, the logical intent is:

user_7 wants post_42 to be liked

For a chat message, the logical intent is:

device_d1 wants to send client_message_id cmsg_99 once

For a payment, the logical intent might be:

merchant_m1 wants to charge order_o7 once

The idempotency boundary must match the business operation.

Common Strategies

Strategy	How it works	Example
Natural uniqueness	A unique constraint prevents duplicate state	One active like per (user_id, post_id)
Idempotency key	Client sends a stable key for one logical operation	sender_id + device_id + client_message_id
State transition guard	Server only performs valid transitions	pending -> paid, not paid -> paid as a new charge
Upsert	Repeated writes set the same final state	PUT /profile/avatar

These strategies can be combined. A chat system might use an idempotency key to deduplicate sends and a unique delivery record to deduplicate downstream delivery.

Idempotency Key Flow

A practical command flow:

1. Client generates an idempotency key before sending.
2. Server receives the request.
3. Server checks whether that key was already processed.
4. If yes, server returns the original result.
5. If no, server performs the operation and stores the result with the key.
6. Future retries receive the same logical response.

Example record:

idempotency_keys
- key: device_d1:cmsg_99
- operation: send_message
- status: completed
- result_message_id: msg_123
- created_at
- expires_at

The response matters. A duplicate retry should not merely return "already exists" if the client needs the original message_id. It should return enough information for the client to reconcile local state.

What Idempotency Guarantees

Idempotency can guarantee that repeated attempts of the same logical operation do not create repeated business effects.

It helps with:

• client retries after timeout
• service retries after transient failure
• duplicate form submissions
• worker reprocessing
• message broker redelivery

It does not automatically guarantee:

• exactly-once execution at every layer
• no duplicate network packets
• no duplicate events unless downstream consumers deduplicate too
• no race conditions unless the storage model enforces uniqueness
• infinite retry safety unless keys are retained forever

Operational Reality

Idempotency requires storage and policy.

Important questions:

• What is the unit of deduplication?
• Who generates the key: client, server, or both?
• How long are keys retained?
• What happens if the same key is reused with a different payload?
• Should the duplicate return the original response or a simplified status?
• Are downstream events and consumers also idempotent?

Important signals:

• duplicate request rate
• idempotency key storage growth
• key reuse with payload mismatch
• retry rate by endpoint
• unique constraint conflicts
• downstream duplicate event rate

Failure modes:

• Key retention is too short, so a late retry creates duplicate work.
• The key is scoped too broadly and blocks legitimate operations.
• The key is scoped too narrowly and misses duplicates.
• Duplicate requests return different responses and confuse clients.
• Downstream consumers process duplicate events even though the API was idempotent.

Related Topics

• Instagram Likes System
• WhatsApp-Style Messaging System
• Delivery Guarantees
• Idempotent Consumers
• Retry With Backoff And Jitter
• Outbox Pattern