Re-ranking

Definition

Re-ranking is the step where a retrieval system takes an initial candidate set and reorders it with a stronger relevance signal.

The plain-English version:

retrieve broadly first
then spend more work ordering the best candidates

This is common because the first retrieval stage must be fast, while the second stage can afford to be slower on a smaller list.

Why This Concept Exists

The fastest retrieval method is not always the most precise.

Vector search may retrieve chunks that are semantically related but not answer-bearing.

Keyword search may retrieve exact terms but miss intent.

Approximate indexes may return good candidates quickly but not in the best final order.

Re-ranking exists because retrieval systems often need two different jobs:

candidate retrieval -> find enough possibly useful results
reranking -> order those results by task relevance

The Beginner Mental Model

A beginner may think:

The vector database returns the best documents.

Sometimes it returns useful results. But "nearest vector" and "best evidence for this answer" are not the same thing.

A better mental model:

retriever optimizes recall under scale
reranker improves precision on a smaller set

Retrieval gets candidates into the room. Re-ranking decides who should sit near the front.

How It Works At A Practical Level

A common pipeline looks like this:

query
  -> retrieve top 50 or top 100 candidates
  -> score each candidate with a reranker
  -> keep top 5 or top 10
  -> assemble context

The first-stage retriever may be:

• keyword search
• vector search
• hybrid search
• metadata-filtered retrieval

The reranker may be a cross-encoder or another scoring model that reads the query and candidate together.

That joint reading is often more precise, but too expensive to run over millions of documents.

A Concrete Example

A user asks:

Can contractors expense home office monitors?

The retriever returns several chunks:

• employee equipment policy
• contractor reimbursement policy
• office furniture purchase process
• remote work setup guide
• monitor installation instructions

All are related. Only one or two may answer the question.

A reranker can score query-document pairs and move the contractor reimbursement policy above the more generally similar office-equipment chunks.

What Re-ranking Improves

Re-ranking often helps with:

• query-document relevance
• answer-bearing evidence
• citation quality
• choosing fewer but stronger chunks
• reducing noisy prompt context
• handling approximate retrieval candidates

For RAG, this matters because weak context can make the generated answer weak or misleading.

Failure Modes

Re-ranking can fail too.

• the first retriever fails to include the right candidate
• the reranker is trained on a different task
• latency is too high for the product path
• reranking ignores permissions or metadata
• top-ranked chunks are relevant but redundant
• scores are treated as truth instead of ranking signals

The most important rule:

a reranker cannot rescue a candidate that was never retrieved

So the first stage still needs enough recall.

Common Confusions

Re-ranking is not the same thing as retrieval.

Retrieval finds candidates. Re-ranking reorders candidates.

A reranker is not always an LLM.

It can be a cross-encoder, learned ranker, rules plus features, or another model.

Re-ranking does not eliminate the need for chunking quality.

If the evidence is split badly, reranking may still choose weak context.

Re-ranking is not free quality.

It adds latency, cost, and operational complexity.

What This Does Not Mean

Re-ranking does not guarantee factual answers.

It improves candidate ordering. The RAG system still needs context assembly, generation constraints, citations, evals, and refusal behavior when evidence is missing.

Related Topics

• Search Execution Flow
• Hybrid Search
• RAG In Plain English
• LLM Evaluation