Agentic RAG

Definition

Agentic RAG is a RAG workflow where a model-backed controller can make decisions during retrieval.

The plain-English version:

the system can decide when to retrieve, how to rewrite, what to inspect, and when to stop

It extends a fixed retrieve-then-generate pipeline into a controlled loop.

Why This Concept Exists

Basic RAG is often:

question -> retrieve -> generate answer

That works for many direct questions.

But some questions need more judgment:

• Should the system retrieve at all?
• Is the first retrieval result relevant?
• Should the query be rewritten?
• Are the sources contradictory?
• Should another tool be used?
• Should the system stop and say the answer is not supported?

Agentic RAG exists because retrieval can be an interactive decision process, not one fixed step.

The Beginner Mental Model

A beginner may think:

Agentic RAG means an autonomous agent searches until it finds an answer.

That is too loose.

A better mental model:

bounded retrieval loop
with tool choices, observations, grading, and stop rules

The agentic part is the controller behavior around retrieval, not permission to wander forever.

A Practical Flow

An agentic RAG workflow might run:

receive question
decide whether retrieval is needed
rewrite query if needed
retrieve candidates
grade whether candidates are relevant
retry with another query or retriever if weak
assemble context
generate grounded answer
stop or ask clarification

The system may expose retrieval as a tool, and the model-backed controller chooses when to call it.

A Concrete Example

A user asks:

Does the new contractor rule affect equipment reimbursement?

A fixed RAG pipeline might retrieve general contractor policy.

An agentic RAG workflow could:

1. retrieve contractor policy
2. judge that it does not mention "new rule"
3. rewrite the query to search policy-change notes
4. retrieve the update memo
5. compare old and new policy context
6. answer only if the evidence supports the claim

The value comes from inspecting retrieval quality before answering.

Control Boundaries

Agentic RAG needs boundaries.

Important controls include:

• max retrieval attempts
• allowed retrievers and tools
• query rewrite policy
• document grading rubric
• stop conditions
• permission checks
• trace logging
• fallback to clarification

Without boundaries, the system may loop, drift, over-retrieve, or answer from weak evidence.

Failure Modes

Agentic RAG can fail when:

• the controller rewrites away from the user's intent
• the model grades bad documents as relevant
• retrieval loops spend cost without improving evidence
• tool choices leak unauthorized data
• the system overfits to finding an answer instead of refusing
• traces are too weak to debug failures
• evals only test final answers and ignore retrieval paths

Agentic RAG increases power and failure surface at the same time.

Product Examples

In a research assistant, the workflow may search, inspect sources, refine the query, and search again.

In a support assistant, the system may retrieve policy, fetch account state, and ask for clarification if evidence conflicts.

In a coding assistant, the workflow may retrieve code context, inspect files, run tests, and revise the retrieval target.

In an enterprise assistant, the system may choose between document search, graph search, and tool calls under permission rules.

Common Confusions

Agentic RAG is not always better than fixed RAG.

Fixed pipelines are simpler, cheaper, and easier to evaluate when the task is predictable.

Agentic RAG is not permissionless browsing.

Retrieval tools still need scopes, filters, and auditability.

Document grading is not truth.

A model can misjudge whether a document supports the answer.

More retrieval attempts do not guarantee better grounding.

They can increase drift, latency, and cost.

What This Does Not Mean

Agentic RAG does not remove the need for evals.

It makes evals more important because you need to inspect decisions, tool calls, retrieved documents, and stopping behavior, not only final text.

Related Topics

• Agent Loops
• Query Expansion
• Knowledge Graph RAG
• LLM Evaluation