Definition
Context compression is the step of reducing retrieved material to the parts that matter for the current query.
The plain-English version:
retrieve enough material, then trim it to useful evidence
Compression can mean removing irrelevant chunks, extracting relevant sentences, summarizing evidence, or filtering content before prompt assembly.
Why This Concept Exists
Retrieval often optimizes recall.
It may bring back many chunks so the right evidence is not missed.
But the model's context window is not a junk drawer. Extra text can:
- distract the model
- consume tokens
- increase cost
- slow inference
- introduce contradictions
- make citations weaker
Context compression exists because RAG systems need enough evidence, not every retrieved word.
The Beginner Mental Model
A beginner may think:
If retrieval found it, put it in the prompt.
That is risky.
A better mental model:
retrieval finds candidates
compression chooses usable evidence
prompt assembly organizes the evidence
The compression step sits between retrieval and generation.
Common Compression Shapes
Context compression can happen in several ways.
Document filtering: remove entire chunks that are not relevant.
Sentence extraction: keep only sentences that support the query.
LLM extraction: ask a model to extract relevant statements from retrieved documents.
Summarization: compress long retrieved material into a shorter query-specific summary.
Metadata filtering: remove stale, unauthorized, or low-confidence material.
Deduplication: remove repeated chunks from the same source.
Each strategy has different quality and reliability risks.
A Concrete Example
A retriever returns a long policy section about company equipment.
Only two sentences answer the user's question:
Contractors are not eligible for home office equipment reimbursement.
Exceptions require a written client agreement before purchase.
Context compression can extract those two sentences instead of inserting the full section about laptops, monitors, chairs, procurement, and shipping.
The prompt becomes smaller and sharper.
Compression Risk
Compression can remove important context.
For example, if a compressor keeps:
Contractors are not eligible...
but drops:
unless a written client agreement says otherwise
the final answer can become overconfident and wrong.
So compression needs evals that check whether the compressed context still supports the correct answer.
Product And Infrastructure Pressure
Context compression helps with:
- long documents
- high-recall retrieval
- large candidate sets
- expensive model context
- noisy corpora
- repeated chunks
- source-backed answers
It also adds a new stage to observe and debug.
The product should log what was retrieved, what was compressed away, what entered the prompt, and what the answer claimed.
Common Confusions
Context compression is not the same thing as summarizing everything.
Sometimes the right move is extraction or filtering, not summary.
Compression is not guaranteed safe.
It can remove exceptions, definitions, dates, or conditions.
Compression does not replace reranking.
Reranking orders candidates. Compression trims or transforms selected material.
More compression is not always better.
Shorter context can be cheaper but less grounded.
What This Does Not Mean
Context compression does not make RAG automatically faithful.
The system still needs source preservation, citation checks, answer grounding, and evals on compressed context.