ANN Indexes

Concepts Covered

• Nearest-neighbor search
• Approximate nearest neighbors
• ANN indexes
• Exact scans
• Recall and latency
• Search-time tuning
• HNSW
• Candidate generation
• Index build cost

Definition

An approximate nearest-neighbor index, often shortened to ANN index, is a search structure that tries to find very good nearby vectors without exhaustively comparing the query vector with every stored vector.

The approximation is the trade:

skip some comparisons
  -> search faster or cheaper
  -> accept that the result may miss a true nearest neighbor

In retrieval systems, that is often acceptable when the index returns strong candidates quickly enough for a later reranker, prompt builder, or product ranking stage.

The Pressure Behind Approximation

Vector search starts with a clean idea:

compare query vector with every candidate vector
sort by similarity
return top-k

That exact scan has a useful baseline property. Under the metric and candidate set you chose, it knows it checked everything.

But as vector count, dimensionality, query throughput, and latency pressure grow, exhaustive comparisons become expensive. The product does not only need the mathematically closest result. It needs useful candidates on a live path that may still have payload fetches, reranking, prompt construction, and model inference after search.

ANN indexes exist because production retrieval often wants a controllable speed-quality tradeoff.

Mental Model

Think of an ANN index as a shortcut map through a large vector collection.

Instead of walking every item:

query -> all vectors -> exact top-k

the search path uses structure built ahead of time:

query -> index navigation -> candidate neighborhood -> approximate top-k

The index may partition space, connect vectors through a graph, compress vectors, or combine several ideas. The common goal is to reduce search work while preserving enough recall for the product.

Recall Is The Cost Of The Shortcut

Recall asks whether the search path found relevant nearest neighbors that a stronger baseline would have found.

If a search system gets faster by skipping useful candidates, quality can fall. If it hunts too widely for candidates, latency and memory work can rise.

This is why ANN tuning usually lives on a curve:

• lower latency
• higher recall
• lower memory
• faster build
• fresher updates

You rarely maximize every one at once.

HNSW As A Concrete ANN Shape

Hierarchical Navigable Small World, or HNSW, is a graph-based ANN approach.

At a concept level, it builds links among stored vectors and searches through a hierarchy of graph neighborhoods. The upper levels help move broadly through the collection; lower levels refine the local neighborhood.

The point for Arcflow is not to memorize HNSW parameters yet. The point is to see a graph index as a different search shape from scanning every vector or probing vector buckets.

Graph indexes can offer strong recall-latency operating points, but they also bring memory, build, update, and tuning considerations.

Engineering Tradeoffs

When an ANN index enters the system, engineers now own:

• index build and rebuild time
• insertion and deletion behavior
• memory overhead
• query-time search depth
• recall measurement
• interactions with metadata filtering
• fallback or exact checks for small filtered candidate sets

For example, an approximate path may be excellent on the full corpus and awkward after a very narrow tenant or permission filter. The product query shape matters as much as the index benchmark.

Failure Modes

Common failures include:

• tuning for a benchmark distribution that does not match real queries
• celebrating search latency while recall drops on rare but important cases
• using approximate candidates where an exact filtered subset would be simpler
• index rebuild or freshness work falling behind content changes
• forgetting that search quality also depends on embeddings, chunking, filters, and reranking

ANN is a retrieval-system tradeoff, not a magic speed flag.

Related Topics

• Vector Search
• Vector Databases
• Indexing Techniques For Vector Search
• Search Execution Flow