Analytics Pipelines

Concepts Covered

• Event capture
• Async ingestion
• Stream processing
• Batch aggregation
• Data freshness
• Duplicate events
• Late events
• Product metrics

Definition

An analytics pipeline collects product events and transforms them into useful information.

Examples:

• clicks per short link
• likes per post
• active users
• conversion rates
• abuse signals
• experiment metrics
• dashboard time series

The pipeline is usually separate from the user-facing request path. The product action happens first. Analytics processing happens asynchronously.

The Pain That Forces Analytics Pipelines

Product events can be extremely high volume.

For a URL shortener, every redirect can produce a click event:

short_code
timestamp
country
device
referrer
user_agent

If the redirect service writes all analytics synchronously before redirecting the user, analytics becomes part of the critical path.

That creates a reliability problem:

analytics database slows down
  -> redirect latency increases
  -> users wait longer
  -> retries increase
  -> click ingestion gets even more traffic

Analytics pipelines exist because event recording is important, but it should not usually block the product action.

Mental Model

Analytics is a second system built from facts emitted by the first system.

The product system says:

This thing happened.

The analytics system later answers:

How many times did it happen?
Where did it happen?
What pattern does it reveal?
Should another system react to it?

This separation lets the product stay fast while analytics workers process, aggregate, and store data at their own pace.

Typical Pipeline Shape

application event
  -> queue or event stream
  -> stream processors or workers
  -> raw event storage
  -> aggregate tables
  -> dashboards, alerts, or product features

Each stage has a job:

• capture the event reliably
• buffer bursts
• process events into useful forms
• store raw history for reprocessing
• serve dashboards or derived metrics

The queue or stream is the shock absorber. It lets ingestion absorb bursts even when processors temporarily fall behind.

Example: Click Analytics

A redirect request should be fast:

1. Resolve short code.
2. Return redirect.
3. Emit click event asynchronously.

Workers can later consume click events and update:

• total click count
• clicks by hour
• clicks by country
• referrer summaries
• abuse signals
• customer dashboards

If the analytics workers are slow, redirects can continue as long as the event buffer has capacity.

The user-facing system trades immediate analytics freshness for lower redirect latency and better isolation.

Freshness Versus Cost

Not every metric needs to be instant.

Use case	Freshness need
Fraud or abuse signal	Often seconds or near real time
User-facing live counter	Product-dependent
Customer dashboard	Seconds to minutes may be acceptable
Monthly reporting	Batch processing may be fine

Freshness has a cost. Lower latency usually means more infrastructure, more operational sensitivity, and less tolerance for delayed processing.

The right question is not "can this be real time?" It is "what decision becomes worse if this is delayed?"

Duplicates And Late Events

Analytics pipelines must expect imperfect input.

Events can be duplicated when producers retry or consumers reprocess. Events can arrive late because mobile clients were offline, queues backed up, or processing jobs failed.

If a click event is processed twice, dashboards may overcount. If a late event is ignored, dashboards may undercount.

Common strategies include:

• event IDs for deduplication
• idempotent aggregation
• watermarking for late events
• periodic reconciliation jobs
• raw event storage for reprocessing

Operational Reality

Important signals:

• ingestion rate
• queue or stream lag
• consumer error rate
• duplicate event rate
• processing latency
• dashboard freshness
• dropped events
• schema compatibility failures
• storage growth
• backfill duration

Analytics looks like "reporting" from the outside, but it behaves like a distributed data system internally. It needs buffering, retries, idempotency, schema discipline, and clear freshness promises.

Related Topics

• URL Shortener System
• Instagram Likes System
• Event Streams
• Backpressure
• Derived Projections