Search Execution Flow

Concepts Covered

• Search request lifecycle
• Query embedding
• Metadata filters
• ANN candidate retrieval
• Payload hydration
• Reranking
• Context assembly
• Latency budget
• Retrieval observability

Definition

A search execution flow is the ordered runtime path that turns a user query into retrieved results or model context.

For vector-backed retrieval, the path is usually more than:

query -> vector database -> answer

A more honest shape is:

query
  -> embed
  -> scope and filter
  -> retrieve candidates
  -> hydrate payloads
  -> rerank or blend
  -> return results or assemble context

Understanding that sequence makes retrieval failures easier to locate.

Why The Flow Matters

Search quality is created across stages.

If the query embedding is weak, the index is searching the wrong neighborhood.

If filters are wrong, good candidates may be hidden or forbidden candidates may leak.

If payload hydration is slow, the ANN lookup can look fast while the user still waits.

If reranking is missing, approximate nearest vectors may arrive in an order that is acceptable for candidate generation and weak for final context.

Execution flow turns "retrieval is bad" into a debuggable pipeline.

Stage 1: Query Understanding

The request begins before the index.

The system may:

• normalize input
• identify tenant or user scope
• decide whether keyword, vector, or hybrid retrieval is needed
• embed the query
• attach structured filters

For a RAG question, this stage determines the query representation that will search the vector space. For product search, it may also preserve exact facets such as category or availability.

Stage 2: Candidate Retrieval

The search service receives a query vector and constraints.

It chooses the configured search path:

• exact comparison for a small candidate set
• ANN traversal over an index
• partition probing
• graph navigation
• compressed approximate comparisons

The output of this stage is often candidate IDs plus scores, not yet final product truth.

index search -> candidate set

Stage 3: Hydration, Refinement, And Reranking

Candidates need usable payloads.

The system may fetch:

• chunk text
• document metadata
• product fields
• code snippets
• high-precision vectors

Then it may refine or rerank.

Reranking is useful when the first stage is optimized for cheap candidate discovery and a later stage can spend more work on a smaller set. Hybrid search may also blend lexical and vector signals here or earlier depending on architecture.

Stage 4: Downstream Use

Search results do not always end at a results page.

For RAG:

retrieved chunks -> context selection -> prompt assembly -> generation

For recommendations:

candidates -> ranker -> feed assembly

For coding assistance:

retrieved code context -> model reasoning or edit workflow

The retrieval contract should match that downstream consumer. A chunk that is "related" may still be too vague for answer grounding.

Latency Budget

Each stage spends time.

query embedding
filters
index lookup
payload reads
reranking
context packing

The runtime question is not only "how fast is the vector database?" It is "which stage owns p95 and p99 latency for the end-user path?"

That budget often decides whether you:

• reduce candidate counts
• move work offline
• add caches
• use lighter reranking
• tighten chunk payloads
• choose a different index operating point

Observability And Failure Handling

Useful retrieval telemetry includes:

• query volume
• embedding latency
• filter selectivity
• candidate count
• index latency
• payload-hydration latency
• rerank latency
• recall or relevance evals
• empty-result rate
• freshness lag

With that view, teams can tell whether a failure came from representation, search infrastructure, data freshness, filtering, or downstream context selection.

Related Topics

• Vector Search
• ANN Indexes
• Indexing Techniques For Vector Search
• RAG In Plain English