Definition
Paged Attention is an LLM serving technique that manages key-value cache memory in fixed-size blocks instead of requiring each request's cache to live in one large contiguous allocation.
The plain-English version:
treat KV cache more like paged memory
It helps serving systems use GPU memory more efficiently when many requests have different prompt lengths and output lengths.
Why This Concept Exists
During autoregressive generation, each active request keeps a KV cache.
That cache grows as tokens are processed and generated.
Serving systems struggle because requests are uneven:
- one user sends a short prompt
- another sends a long document
- one answer ends after ten tokens
- another generates hundreds of tokens
If the serving system reserves memory poorly, GPU memory gets wasted by fragmentation, over-allocation, or duplicated prompt state.
Paged Attention exists because KV cache memory is large, dynamic, and central to batching throughput.
The Beginner Mental Model
A beginner may think:
The model just stores each request's cache somewhere in memory.
That skips the allocation problem.
A better mental model is:
request has logical cache blocks
runtime maps those blocks to physical memory blocks
blocks can be allocated as needed
This is inspired by virtual memory ideas from operating systems. The point is not that LLM serving becomes an operating system. The point is that indirection helps manage uneven memory demand.
How The Block Idea Helps
Instead of reserving one large contiguous region for a request, the system divides KV cache into blocks.
A request may have logical blocks:
block 1 -> prompt tokens
block 2 -> more prompt tokens
block 3 -> generated tokens
Those logical blocks can map to physical blocks that are not contiguous.
As the request generates more tokens, the system allocates more blocks when needed.
This reduces waste because memory can grow with the request instead of being over-reserved up front.
A Concrete Example
Imagine three users arrive:
- user A asks a short question
- user B pastes a long document
- user C asks for three alternative drafts from the same prompt
Without careful cache management, the system may reserve too much memory for A, fragment memory for B, and duplicate prompt cache for C's alternatives.
With a paged approach:
- A uses a small number of blocks
- B receives blocks as needed
- C can share prompt blocks across alternatives until outputs diverge
That can let the serving system batch more active sequences in the same memory budget.
Cache Sharing
Paged Attention can also support efficient sharing.
For example, when multiple outputs are generated from the same prompt, their prompt-side cache can be shared at first.
When one sequence diverges, the system can allocate separate blocks for the new generated tokens.
The useful product implication:
some expensive prompt work can be reused across related generations
This matters for sampling multiple candidates, beam-like workflows, and agent systems that branch from shared context.
Product And Infrastructure Pressure
Paged Attention matters when LLM serving needs:
- high throughput
- long prompts
- many concurrent users
- dynamic output lengths
- memory-efficient batching
- multiple generations from shared prompts
In a chat assistant, it helps active conversations share limited GPU memory more efficiently.
In a coding assistant, long file context can create large caches.
In a document assistant, retrieved chunks inflate prompt length and cache memory.
Common Confusions
Paged Attention is not the same thing as Flash Attention.
Flash Attention focuses on executing attention computation with less memory movement. Paged Attention focuses on managing KV cache memory for serving.
Paged Attention is not model memory.
It manages temporary inference state, not permanent knowledge about a user.
Paged Attention does not make generation non-sequential.
It improves memory management. The decode loop still depends on generated tokens.
Paged Attention is not only about long prompts.
It also helps with many uneven concurrent requests and shared prompt scenarios.
What This Does Not Mean
Paged Attention does not make GPU memory infinite.
It reduces waste and improves the serving operating point, but model size, prompt length, output length, batch size, and traffic shape still matter.
It also does not decide whether the answer is correct. It makes serving more efficient; evals still judge behavior.