Idempotent Consumers

Concepts Covered

• Duplicate messages
• At-least-once delivery
• Consumer retries
• Deduplication tables
• Event IDs
• Safe side effects
• Reprocessing
• Retention windows

Definition

An idempotent consumer can process the same event more than once while producing the same final intended result.

This matters because many event systems use at-least-once delivery. At-least-once delivery means the system tries hard not to lose events, but it may deliver the same event multiple times.

The consumer must be designed for that reality.

The Pain That Forces Idempotent Consumers

Imagine a like event:

LikeCreated(event_id=evt_123, post_id=42, user_id=7)

A notification consumer receives the event and creates a notification:

user_42 received a like from user_7

Now suppose the worker crashes after writing the notification but before acknowledging the event to the broker.

From the broker's perspective, the event was not completed. So it sends the event again.

If the consumer blindly processes the event again, the recipient may receive two notifications for one like.

The broker did not do anything wrong. The consumer was not safe to retry.

Mental Model

Event consumers should assume:

I may see this event again.

The job is not to prevent duplicate delivery at every layer. That is often impossible or too expensive. The job is to make duplicate delivery harmless at the business side-effect layer.

For each consumer, ask:

What side effect does this event create?
How do I know whether I already created it?

The answer depends on the side effect: notification, counter update, email, search index update, inbox projection, billing entry, or analytics aggregate.

Common Strategies

Strategy	How it works	Example
Event ID deduplication	Store processed event IDs and skip repeats	processed_events(event_id)
Natural unique key	Enforce uniqueness on the side effect	one notification per (recipient, actor, post, type)
State-based update	Apply only if event advances a version	update projection from version 7 to 8
Upsert	Repeated writes converge to same row	set latest delivery state
Idempotent aggregate	Add only if operation ID was not applied	counter shard stores applied event IDs

The safest systems often combine database constraints with application-level checks.

Deduplication Table Flow

A consumer can use a transaction:

BEGIN;

INSERT INTO processed_events (event_id, consumer_name)
VALUES ('evt_123', 'notification-consumer');

INSERT INTO notifications (...);

COMMIT;

If the event is delivered again, the insert into processed_events fails because the event was already processed by that consumer. The worker can skip the side effect.

The key detail is that the dedupe marker and the side effect should commit together when possible. If the marker commits but the side effect does not, the system may think work happened when it did not.

Natural Idempotency

Sometimes the business model already provides a unique key.

For example:

unique(recipient_id, actor_id, post_id, notification_type)

This prevents duplicate like notifications even if the same event arrives twice.

Natural uniqueness is often better than a generic event table because it protects the actual business invariant. But it only works when the side effect has a clear unique identity.

What Idempotent Consumers Do Not Solve

Idempotent consumers do not guarantee that events arrive in order. They do not guarantee that all consumers are up to date. They do not remove the need for retries, monitoring, or reconciliation.

They solve a narrower but critical problem:

Duplicate delivery should not create duplicate business effects.

Operational Reality

Important signals:

• duplicate event rate
• dedupe table growth
• skipped duplicate count
• consumer retry rate
• dead-lettered events
• side-effect uniqueness violations
• replay duration
• old event retention window

Deduplication storage needs retention policy. Keeping every processed event forever may be expensive. Dropping records too early can make very late retries unsafe.

The retention window should match how long events can realistically be redelivered or replayed.

Related Topics

• Instagram Likes System
• WhatsApp-Style Messaging System
• Event Streams
• Idempotency
• Idempotent Consumer Pattern