Optional

KV Cache

Understand how key-value caching makes autoregressive LLM inference faster by reusing attention work from previous tokens.

4 min readUpdated May 2026

After this, you will understand

How KV Cache helps you see why the model and retrieval machinery behaves the way it does, and which tradeoffs it creates.

Article guideprerequisites, mental models, and concepts

intermediateMechanicsInferenceOperations

Three useful mental models

In plain terms

Treat the idea as a definition to memorize.

Production pressure

Real systems force the idea to handle KV Cache, Key-Value Cache, and Autoregressive Inference.

Better reasoning

Use the concept to decide what the system guarantees, what it risks, and what it costs to operate.

Think before reading

Where would KV Cache appear in a real production system, and what failure or bottleneck would it help you reason about?

As you read, look for the pressure that creates the idea first. The mechanics matter more once the reason is clear.

Optional depth

Follow these links when you want to understand the internals more deeply.

1LLM Inference ServingUnder The Hood

Concepts Covered

KV cache
Key-value cache
Autoregressive inference
Prefill phase
Decode phase
Attention keys and values
Cache memory growth
Serving latency tradeoffs

Definition

A KV cache stores the key and value tensors produced by attention layers for previous tokens during autoregressive generation.

The plain-English version:

do not recompute the attention keys and values for old tokens every time
store them once and reuse them for the next token

It is a serving optimization for transformer language models that generate one token at a time.

Why This Concept Exists

LLMs generate text step by step.

Suppose the model has already processed:

The database connection timed out because

To generate the next token, the model needs context from those previous tokens.

After it generates one more token, it still needs context from the same previous tokens again.

Without caching, the model would repeatedly redo attention-related work for tokens it already processed. KV cache exists because that repeated work becomes expensive during generation.

The Beginner Mental Model

A beginner may think:

The model reads the prompt once and then just writes the answer.

That hides the serving loop.

For many language models, generation looks more like:

read current context
predict one token
append that token
repeat

The KV cache is one of the tricks that makes that loop fast enough to serve real users.

What Gets Cached

Attention uses queries, keys, and values.

During generation, the current token produces a new query. It attends over keys and values from the current and previous tokens.

The useful observation is:

past keys and values do not need to be rebuilt from scratch every step

So the serving system stores them per layer.

A simplified step is:

new token -> compute current key and value
cache <- append current key and value
current query attends over cached keys and values

The exact tensor shapes are implementation details, but the engineering idea is stable: reuse past attention state.

Prefill And Decode

LLM serving often talks about two phases.

Prefill processes the prompt:

prompt tokens -> build initial model state and KV cache

Decode generates new tokens:

use cache + current token -> predict next token
append new key and value to cache
repeat

Prefill can process prompt tokens together. Decode is more sequential because each new token depends on the previous generated token.

That is one reason long prompts and long outputs create different performance pressures.

Why KV Cache Helps Latency

Without a cache, every new token step would spend work recomputing old keys and values.

With a cache, the model still attends to previous context, but it can reuse stored keys and values.

That improves time per generated token.

The tradeoff is memory:

more tokens + more layers + more heads + larger head dimension -> larger cache

So KV cache reduces repeated computation, but it increases memory pressure.

Product And Infrastructure Pressure

In a ChatGPT-style assistant, every active generation request may hold a KV cache.

For an AI coding assistant, a long file context and long completion can make the cache large.

For a document assistant, retrieved context increases prompt length, which increases prefill work and cache size.

This is why LLM serving teams care about:

batching active requests
prompt length
output length
GPU memory
cache eviction or paging
quantized caches
attention kernels

KV cache is not a tiny implementation detail. It shapes real serving cost and latency.

Common Confusions

KV cache is not model memory.

It does not mean the model permanently remembers a user. It is temporary inference state for the current generation.

KV cache is not the same as retrieval.

Retrieval fetches external information to place into context. KV cache stores attention state for tokens already inside the current context.

KV cache does not remove attention cost entirely.

The model still has to attend over available cached context. The cache avoids recomputing past keys and values.

KV cache is mostly an inference concern.

Training has different parallelism and memory patterns. Caching previous token states is mainly useful for autoregressive decoding.

What This Does Not Mean

KV cache does not make long context free.

Longer context still consumes memory and can create latency, batching, and quality challenges. The cache helps reuse work, but the serving system still has to store and manage the growing state.

Finished reading?

Your reading history is saved in this browser so you can continue later.

Recommended Next

What Is AI?AI Foundations12 min read

Return to the recommended AI Engineering journey here. Start with the model and inference vocabulary every practical AI system depends on.

Optional exploration

These links add context, but they do not replace the recommended next lesson.