Perplexity-Style RAG Search System

1. Introduction

A Perplexity-style answer engine combines search and generation. The user asks a question. The system searches for evidence, selects sources, builds context, and asks a language model to produce a useful answer with citations.

The naive mental model is:

search the web -> give results to the model -> answer

The real engineering problem is sharper. Search results are noisy. Sources disagree. The freshest page is not always the most authoritative. A top-ranked result may mention the right words without answering the question. A model can sound confident even when the evidence is weak. The system must therefore treat retrieval as an evidence pipeline, not a magic truth pipe.

This page uses a familiar product shape, but does not claim private details about any specific company.

2. Product Requirements

Functional Requirements

• Users can ask natural-language questions.
• The system can retrieve relevant sources from web pages, indexes, or curated corpora.
• The answer can cite sources used for important claims.
• The system can handle ambiguous, fresh, and multi-hop questions.
• The system can refuse or qualify answers when evidence is insufficient.
• The system can show source metadata such as title, URL, date, or snippet.
• The system can log retrieval and generation traces for evaluation.

Non-Functional Requirements

• Answers should arrive with low enough latency for interactive search.
• Retrieval should balance semantic relevance, keyword precision, authority, freshness, and diversity.
• Citation links should point to sources that actually support the answer.
• The system should avoid leaking private or unauthorized documents.
• The architecture should tolerate failed crawls, unavailable sources, and model errors.
• Quality should be measured end to end, not only by search rank or model fluency.

3. Core Engineering Challenges

The system fails when it optimizes a single layer in isolation. A fast retriever that returns weak sources leads to confident wrong answers. A model that writes beautifully can hide evidence gaps. A citation UI can make unsupported text look trustworthy.

4. High-Level Architecture

flowchart LR
  User[User] --> QueryAPI[Question API]
  QueryAPI --> Planner[Query planner]
  Planner --> WebSearch[Keyword and web search]
  Planner --> VectorSearch[Vector retrieval]
  WebSearch --> CandidatePool[Candidate pool]
  VectorSearch --> CandidatePool
  CandidatePool --> Reranker[Re-ranker]
  Reranker --> ContextBuilder[Context builder]
  ContextBuilder --> Generator[LLM generator]
  Generator --> CitationChecker[Citation and support checker]
  CitationChecker --> Answer[Answer with sources]
  Answer --> User
  CitationChecker --> TraceStore[(Trace and eval store)]

The planner may rewrite the query, expand terms, or split the question into subquestions. Retrieval produces candidates. Re-ranking selects the most useful sources. The context builder extracts passages and manages token budget. The generator writes the answer under citation instructions. A support checker or evaluator inspects whether important claims are backed by sources.

5. Core Components

Question API: receives the question, applies account and policy checks, and starts a trace.

Query planner: decides whether the question needs fresh search, curated retrieval, query expansion, or multiple subqueries.

Retrievers: include keyword search, vector search, hybrid search, web indexes, curated source indexes, and sometimes specialized APIs.

Candidate pool: merges results while preserving source metadata, score, freshness, and permission information.

Re-ranker: scores candidates more carefully against the user question. It is slower than initial retrieval but runs over fewer items.

Context builder: chooses passages, removes duplicates, compresses text, and reserves room for answer instructions.

Generator: writes the final answer from supplied context. It should be instructed to cite evidence and qualify uncertainty.

Citation checker and eval sink: inspect final claims, source usage, and trace data for offline analysis.

6. Data Modeling

The system has both online request data and indexed source data.

source(id, url, title, domain, fetched_at, authority_signals)
document(id, source_id, content_hash, language, published_at)
passage(id, document_id, text, offset, embedding_version)
retrieval_trace(id, query_id, retriever, score, rank, passage_id)
answer_trace(id, question_id, model_id, prompt_version, cited_source_ids)

For web-like content, freshness and provenance matter. A passage should retain its source URL, fetch time, publication hints, and content hash. For private enterprise content, tenant ID and permission metadata matter even more than public freshness.

Indexes may include an inverted index for keywords, a vector index for semantic similarity, and metadata indexes for date, domain, language, or tenant filtering. The answer should never cite an index entry without a path back to the source text.

7. Request Lifecycle

1. The user asks a question.
2. The system classifies the question: factual, fresh, exploratory, local-document, or ambiguous.
3. The planner may rewrite the query or create subqueries.
4. Retrieval runs across keyword, vector, and specialized sources.
5. Candidates are deduplicated and filtered for permissions and freshness constraints.
6. A re-ranker selects passages most likely to answer the question.
7. The context builder extracts, compresses, and orders evidence.
8. The model generates an answer with citation instructions.
9. A support check verifies whether citations align with claims.
10. The response and trace are logged for evaluation.

If retrieval is weak, the system should say so. A grounded answer engine earns trust by refusing to overstate evidence.

8. Scaling Problems

RAG search systems face both search-scale and model-scale bottlenecks.

Initial retrieval may fan out to multiple indexes. Re-ranking adds latency because it compares candidates more deeply. Context assembly may fetch full documents or passages from storage. Generation latency depends on prompt size and output length.

Common scaling problems include:

• duplicate content wasting context budget
• slow web fetches blocking interactive answers
• fresh indexes lagging behind source updates
• vector search returning semantically related but unsupported passages
• re-ranking becoming the latency bottleneck
• citation checks adding cost to every response
• high-cardinality traces overwhelming analytics storage

The architecture needs tiered work. Some questions can use cached search results. Some need fresh retrieval. Some should run deeper multi-step research only after the product decides the user can tolerate slower output.

9. Distributed Systems Concepts

Fan-out: the planner may query multiple retrievers in parallel, then merge results.

Eventual consistency: crawled or embedded content may lag behind source updates.

Read amplification: one user question can create many reads across indexes, document stores, and model-serving systems.

Backpressure: search and model queues need limits when traffic spikes.

Deduplication: repeated or mirrored content should not dominate the candidate set.

Caching: query results, page fetches, passage extraction, and answer fragments may be cached with careful freshness rules.

10. Reliability & Failure Handling

Failures should be visible in the answer contract.

If one retriever fails, the system may continue with degraded evidence but should log the missing source. If fresh search times out, the answer should avoid claiming current certainty. If citation support is weak, the product can ask a clarifying question, show sources without a synthesized answer, or explicitly state that evidence is insufficient.

Operational metrics include retrieval latency by source, top-k diversity, re-ranker latency, citation support rate, empty-result rate, freshness lag, model refusal rate, and user correction signals.

The hardest failures are quality failures: answers that look grounded but cite sources that do not support the claim. Those require eval cases, human review, and trace inspection.

11. Real-World Company Approaches

A company at this scale might combine web search, curated indexes, freshness pipelines, semantic retrieval, passage re-ranking, and model-based answer generation. It might cache popular queries, maintain source quality signals, and treat citations as product-critical rather than decorative.

Public behavior of answer engines shows a common shape: search first, read sources, synthesize, and cite. The private details vary. The reusable pattern is evidence retrieval plus controlled generation.

12. Tradeoffs & Alternatives

The best design depends on whether the product optimizes for speed, freshness, authority, depth, or enterprise permissions.

13. Evolution Path

1. Start with keyword search plus answer generation.
2. Add source snippets and citation display.
3. Add vector retrieval for semantic recall.
4. Add hybrid search and re-ranking.
5. Add context compression and duplicate removal.
6. Add freshness-aware indexing and source quality signals.
7. Add multi-step planning for complex questions.
8. Add groundedness evaluation and citation support checks.

Each stage responds to a failure in the previous version: weak recall, poor citations, stale evidence, or answers that sound better than their sources.

14. Key Engineering Lessons

• RAG does not make answers automatically true; it supplies evidence to a generator.
• Search quality, source quality, context quality, and generation quality are separate layers.
• Citations are only useful when they actually support the claims near them.
• Freshness, authority, and permissions are part of retrieval design.
• End-to-end evals must inspect retrieved sources, not only final text.

15. Related Topics

• RAG In Plain English
• Hybrid Search
• Re-ranking
• Query Expansion
• Context Compression
• Agentic RAG