Document Q&A System

1. Introduction

A document Q&A system lets users ask questions over files such as PDFs, docs, policies, tickets, wikis, contracts, or support articles. The promise sounds simple: upload documents, ask a question, get an answer. The engineering reality is a pipeline that must parse messy files, preserve permissions, build searchable chunks, retrieve evidence, and prevent the model from answering beyond what the documents support.

The naive implementation is:

put the whole document in the prompt and ask the model

That fails when documents are large, many users upload many files, permissions differ by user, and answers need citations. A production system needs retrieval because the model cannot read every document on every request.

2. Product Requirements

Functional Requirements

• Users can upload or connect documents.
• The system can parse documents into text and metadata.
• The system can chunk documents for retrieval.
• Users can ask questions over allowed documents.
• Answers can cite document passages.
• Document updates and deletes affect retrieval.
• Permissions are enforced before any content reaches the model.
• The system can report when the documents do not contain enough evidence.

Non-Functional Requirements

• Ingestion should be reliable for common document formats.
• Queries should feel interactive for normal document collections.
• Access control should be strict and auditable.
• Retrieval should return relevant passages rather than only related passages.
• Citations should point to source locations users can inspect.
• The system should tolerate parse failures and partial indexing.
• Quality should be evaluated with real user questions.

3. Core Engineering Challenges

The hardest part is that every mistake looks like model failure to the user. If parsing drops a table, retrieval cannot find it. If permissions are wrong, the model may see private text. If chunks are poor, the answer may cite half a sentence without its parent section.

4. High-Level Architecture

flowchart LR
  Uploader[Uploader] --> IngestAPI[Ingest API]
  IngestAPI --> Parser[Parser and normalizer]
  Parser --> Chunker[Chunker]
  Chunker --> Embedder[Embedding worker]
  Embedder --> VectorStore[(Vector store)]
  Parser --> DocumentStore[(Document store)]

  User[User] --> QAAPI[Question API]
  QAAPI --> AuthZ[Permission filter]
  AuthZ --> Retriever[Retriever]
  Retriever --> VectorStore
  Retriever --> DocumentStore
  Retriever --> Reranker[Re-ranker]
  Reranker --> ContextBuilder[Context builder]
  ContextBuilder --> Generator[LLM answer generator]
  Generator --> User

Ingestion and querying are separate. Ingestion turns source files into searchable records. Querying retrieves only allowed evidence and generates an answer from that evidence.

5. Core Components

Ingest API: accepts files, validates type and size, records ownership, and starts asynchronous processing.

Parser and normalizer: extracts text, structure, page numbers, headings, tables, and source offsets. It should preserve enough location metadata for citations.

Chunker: creates retrieval units. It may use headings, page boundaries, semantic sections, overlap, or parent-child relationships.

Embedding worker: generates vectors for chunks and records embedding version.

Document store: keeps canonical text, metadata, permissions, and source location maps.

Vector store: supports semantic retrieval over chunks while preserving tenant and permission metadata.

Retriever and re-ranker: find candidates, filter them, and select evidence most likely to answer the question.

Answer generator: writes from supplied evidence and cites source locations.

6. Data Modeling

The system needs source, chunk, vector, permission, and trace records.

document(id, tenant_id, owner_id, title, source_uri, version, status)
document_acl(document_id, principal_id, permission)
section(id, document_id, heading, page_start, page_end)
chunk(id, document_id, section_id, text, start_offset, end_offset)
embedding(chunk_id, embedding_version, vector_ref)
qa_trace(id, user_id, question, retrieved_chunk_ids, answer_id)
answer_citation(answer_id, document_id, chunk_id, page, span)

The chunk is the retrieval unit, but the document or section may be the display unit. Parent-child retrieval often searches small chunks and then returns a larger parent section to preserve context.

Document versioning matters. If a file is updated, old chunks should either be retired or tied to the old version. The answer should cite the version it actually used.

7. Request Lifecycle

1. A user uploads a document.
2. The ingest API records the document and starts parsing.
3. The parser extracts text, structure, and source offsets.
4. The chunker creates retrieval chunks and parent section links.
5. The embedding worker embeds chunks and writes them to the vector store.
6. The user asks a question.
7. The system checks which documents the user may access.
8. Retrieval searches only allowed chunks.
9. The re-ranker selects evidence and the context builder assembles passages.
10. The model answers from evidence and emits citations.
11. The trace records retrieval, citations, and final answer for evaluation.

If ingestion is incomplete, the product should make that visible. A half-indexed document should not create fake confidence.

8. Scaling Problems

Document Q&A scales along ingestion volume, corpus size, query rate, document length, permission complexity, and citation quality.

Common scaling problems include:

• parsing workers falling behind large uploads
• embedding costs spiking during bulk imports
• vector indexes containing stale chunks after document updates
• permission filters slowing retrieval
• large tenants dominating shared indexes
• duplicated documents wasting storage and context
• citation spans drifting after re-processing
• tables and images producing weak text representations

At larger scale, ingestion needs queues, retries, dead-letter handling, and idempotent processing. Query serving needs caches, per-tenant limits, and permission-aware retrieval strategies.

9. Distributed Systems Concepts

Asynchronous processing: document ingestion is usually too slow for the upload request path.

Idempotency: retrying parse or embedding jobs should not create duplicate chunks.

Eventual consistency: uploaded documents may become searchable after a delay.

Access control: permission checks must apply before model context assembly.

Derived data: chunks, embeddings, and indexes are derived from canonical documents.

Backpressure: bulk uploads should not overload query-serving capacity.

10. Reliability & Failure Handling

The system needs clear document processing states: uploaded, parsing, parsed, embedding, indexed, failed, deleted. Users should know whether a document is searchable.

If parsing fails, the system can show the file as failed with a reason and retry option. If embedding fails, the document remains stored but not searchable. If a delete occurs, retrieval must stop returning chunks quickly, even if physical cleanup happens later.

Important metrics include ingestion lag, parse failure rate, embedding failure rate, searchable-document count, permission-filter miss rate, retrieval hit rate, citation support rate, and answer refusal rate.

The most dangerous reliability issue is unauthorized context. A model should never receive text the user is not allowed to see.

11. Real-World Company Approaches

Companies building document assistants often use asynchronous ingestion pipelines, separate durable document stores, vector or hybrid retrieval, permission filters, and answer traces. Enterprise systems usually invest heavily in access control because the retrieval layer can accidentally bypass application permissions if metadata is weak.

Common public patterns include chunking documents, embedding chunks, searching semantically, re-ranking evidence, and generating answers with citations. The quality of the system depends heavily on the less glamorous ingestion and permission layers.

12. Tradeoffs & Alternatives

Document Q&A should optimize for grounded usefulness, not answer volume.

13. Evolution Path

1. Start with manual upload and simple text extraction.
2. Add chunking and vector retrieval.
3. Add citations with page and span mapping.
4. Add permission filtering and tenant isolation.
5. Add hybrid search and re-ranking.
6. Add parent-child retrieval and context compression.
7. Add document update and delete workflows.
8. Add groundedness evals and ingestion quality monitoring.

The system matures when it treats documents as governed data, not prompt decoration.

14. Key Engineering Lessons

• The answer can only be as good as parsing, chunking, retrieval, and evidence selection.
• Permissions must be enforced before context reaches the model.
• Chunks are retrieval units, not necessarily explanation units.
• Citations require source-location metadata from ingestion onward.
• Stale embeddings and deleted documents are product trust issues.

15. Related Topics

• RAG In Plain English
• Parent-Child Retrieval
• Multi-Vector Retrieval
• Context Compression
• Re-ranking
• Hallucinations