Idempotent Consumer

Concepts Covered

• Duplicate message delivery
• At-least-once delivery
• Event IDs
• Deduplication stores
• Natural uniqueness
• Safe retries
• Consumer side effects
• Replay safety

1. Intent

The Idempotent Consumer pattern makes an event consumer safe to run more than once for the same message.

This matters because many queues and event streams use at-least-once delivery. At-least-once delivery means a message should not be lost, but it may be delivered multiple times.

The intent is to make duplicate delivery harmless at the business side-effect layer.

2. The Problem Without This Pattern

Suppose a notification worker receives LikeCreated.

It writes a notification, then crashes before acknowledging the event. The broker retries. Without idempotency, the worker writes another identical notification.

The same issue can:

• inflate counters
• duplicate emails
• send duplicate push notifications
• charge customers twice
• replay old state over newer state
• create duplicate delivery records

The broker did not fail by redelivering. The consumer failed by treating every delivery as new business intent.

3. How The Pattern Works

There are several common implementations.

Strategy	How it works	Example
Processed-event table	Store event_id after processing and skip repeats	processed_events(event_id, consumer_name)
Unique side-effect key	Use a unique constraint on the effect itself	one notification per (recipient, actor, post, type)
Version guard	Apply only events newer than current state	update projection only if event_version increases
Upsert	Make repeated writes converge to the same value	set delivery state to delivered

A common flow uses a transaction:

BEGIN;

INSERT INTO processed_events (event_id, consumer_name, processed_at)
VALUES ('evt_123', 'notification-worker', now());

INSERT INTO notifications (...);

COMMIT;

If the same event arrives again, the processed_events insert fails, and the consumer skips the side effect.

The dedupe marker and the side effect should commit together when possible. If the marker commits but the side effect fails, the consumer may think work happened when it did not.

4. When To Use It

Use this pattern when:

• the message broker may redeliver events
• the publisher may publish duplicates
• the consumer performs side effects
• retries are necessary for reliability
• replay is possible
• correctness matters more than raw simplicity

Good examples:

• notification workers
• counter projection workers
• message delivery workers
• analytics aggregators that require accurate counts
• outbox event consumers

5. When Not To Use It

You may not need heavy deduplication when:

• the side effect is naturally idempotent
• duplicate processing is harmless
• the stream is purely analytical and sampling is acceptable
• the consumer can rebuild state from scratch regularly
• the cost of dedupe exceeds the cost of duplicates

Even then, be explicit. "Duplicates are acceptable" should be a product and engineering decision, not an accident.

6. Data And Operational Model

Idempotent consumers often need:

• stable event IDs
• event version or timestamp
• deduplication retention window
• unique constraints for side effects
• metrics for duplicate rate
• replay policy
• dead-letter behavior for invalid events

Deduplication data cannot always be kept forever. Systems often retain processed IDs for a window that matches replay and retry expectations.

Operators should monitor:

• duplicate event rate
• dedupe table growth
• skipped duplicate count
• uniqueness violations
• replay success rate
• old-event processing attempts

7. Failure Modes

• Deduplication key is too broad and drops valid work.
• Deduplication key is too narrow and misses duplicates.
• Processed-event table grows without retention.
• Consumer records an event as processed before side effects complete.
• Old events overwrite newer state.
• Replay re-triggers side effects because dedupe records expired too early.
• Different consumers incorrectly share the same dedupe namespace.

8. Tradeoffs

Benefit	Cost
Makes retries safer	Requires dedupe state or careful write design
Prevents duplicate side effects	Adds storage and complexity
Works well with at-least-once brokers	Retention window must be chosen
Improves operational confidence	Bugs can hide if dedupe keys are wrong
Supports replay	Can block valid reprocessing if scoped poorly

This pattern is the practical partner of at-least-once delivery. If the broker may redeliver, the consumer must know how to converge.

9. Related Systems And Concepts

• Instagram Likes System
• WhatsApp-Style Messaging System
• Idempotent Consumers
• Event Streams
• Transactional Outbox
• Dead-Letter Queue