Paged Attention

Definition

Paged Attention is an LLM serving technique that manages key-value cache memory in fixed-size blocks instead of requiring each request's cache to live in one large contiguous allocation.

The plain-English version:

treat KV cache more like paged memory

It helps serving systems use GPU memory more efficiently when many requests have different prompt lengths and output lengths.

Why This Concept Exists

During autoregressive generation, each active request keeps a KV cache.

That cache grows as tokens are processed and generated.

Serving systems struggle because requests are uneven:

• one user sends a short prompt
• another sends a long document
• one answer ends after ten tokens
• another generates hundreds of tokens

If the serving system reserves memory poorly, GPU memory gets wasted by fragmentation, over-allocation, or duplicated prompt state.

Paged Attention exists because KV cache memory is large, dynamic, and central to batching throughput.

The Beginner Mental Model

A beginner may think:

The model just stores each request's cache somewhere in memory.

That skips the allocation problem.

A better mental model is:

request has logical cache blocks
runtime maps those blocks to physical memory blocks
blocks can be allocated as needed

This is inspired by virtual memory ideas from operating systems. The point is not that LLM serving becomes an operating system. The point is that indirection helps manage uneven memory demand.

How The Block Idea Helps

Instead of reserving one large contiguous region for a request, the system divides KV cache into blocks.

A request may have logical blocks:

block 1 -> prompt tokens
block 2 -> more prompt tokens
block 3 -> generated tokens

Those logical blocks can map to physical blocks that are not contiguous.

As the request generates more tokens, the system allocates more blocks when needed.

This reduces waste because memory can grow with the request instead of being over-reserved up front.

A Concrete Example

Imagine three users arrive:

• user A asks a short question
• user B pastes a long document
• user C asks for three alternative drafts from the same prompt

Without careful cache management, the system may reserve too much memory for A, fragment memory for B, and duplicate prompt cache for C's alternatives.

With a paged approach:

• A uses a small number of blocks
• B receives blocks as needed
• C can share prompt blocks across alternatives until outputs diverge

That can let the serving system batch more active sequences in the same memory budget.

Cache Sharing

Paged Attention can also support efficient sharing.

For example, when multiple outputs are generated from the same prompt, their prompt-side cache can be shared at first.

When one sequence diverges, the system can allocate separate blocks for the new generated tokens.

The useful product implication:

some expensive prompt work can be reused across related generations

This matters for sampling multiple candidates, beam-like workflows, and agent systems that branch from shared context.

Product And Infrastructure Pressure

Paged Attention matters when LLM serving needs:

• high throughput
• long prompts
• many concurrent users
• dynamic output lengths
• memory-efficient batching
• multiple generations from shared prompts

In a chat assistant, it helps active conversations share limited GPU memory more efficiently.

In a coding assistant, long file context can create large caches.

In a document assistant, retrieved chunks inflate prompt length and cache memory.

Common Confusions

Paged Attention is not the same thing as Flash Attention.

Flash Attention focuses on executing attention computation with less memory movement. Paged Attention focuses on managing KV cache memory for serving.

Paged Attention is not model memory.

It manages temporary inference state, not permanent knowledge about a user.

Paged Attention does not make generation non-sequential.

It improves memory management. The decode loop still depends on generated tokens.

Paged Attention is not only about long prompts.

It also helps with many uneven concurrent requests and shared prompt scenarios.

What This Does Not Mean

Paged Attention does not make GPU memory infinite.

It reduces waste and improves the serving operating point, but model size, prompt length, output length, batch size, and traffic shape still matter.

It also does not decide whether the answer is correct. It makes serving more efficient; evals still judge behavior.

Related Topics

• KV Cache
• LLM Inference Serving
• Flash Attention
• Quantization