Hallucinations

Concepts Covered

• Hallucinations
• Fluent but wrong output
• Unsupported claims
• Grounding
• Context gaps
• Retrieval
• Verification
• Product risk
• Why hallucination is not lying
• How AI products reduce damage

1. Plain-English Definition

In AI discussions, a hallucination is output that sounds plausible but is wrong, invented, or unsupported for the task.

For example, a model might:

• invent a policy that is not in the document
• cite a source it did not actually have
• give the wrong account balance
• explain code that it never saw
• answer confidently when the right answer should be "I do not know"

The important beginner lesson is:

fluent output is not the same thing as verified truth

2. Why This Idea Exists

Generative models produce outputs. They do not automatically run a truth checker over every sentence they emit.

That becomes dangerous because language quality is persuasive.

A broken SQL query looks broken. A broken UI may show a visible error. A hallucinated answer can look polished.

If the model is missing context, if the task is underspecified, if the product asks for facts outside available evidence, or if the model follows a likely language pattern that does not match reality, the output can drift away from truth.

The word hallucination exists because teams needed a name for this failure mode.

3. The Beginner Mental Model

Think of a hallucination as a confident guess that escaped the product boundary.

The model receives a request. It generates the most suitable output it can under its learned behavior and visible context. If that visible context is weak, the output may still continue in a believable direction.

missing evidence + pressure to answer -> plausible-looking wrong output

This mental model helps software engineers ask better questions:

• Did the model have the right evidence?
• Did we ask it to answer even when evidence was missing?
• Did we validate the output before showing it?

4. What That Mental Model Misses

Calling it a "guess" is useful, but it can oversimplify the failure.

First, hallucinations are not only caused by an empty prompt. A model can hallucinate even with context if the context is ambiguous, noisy, contradictory, or used poorly.

Second, not every wrong answer has the same cause. The product might retrieve the wrong document, parse a tool result incorrectly, use stale data, or ask the model for a claim it cannot support.

Third, hallucination is not identical to uncertainty. A model may be uncertain and still produce a cautious answer. It may also sound certain while being wrong.

Fourth, the term should not make the model sound human. The model is not necessarily "seeing" something that is not there like a person. It is generating output that fails the truth or support requirement.

5. A Concrete Example

Imagine an internal HR assistant.

The user asks:

Can contractors expense home office equipment?

The product retrieves a policy excerpt about full-time employees only. The model still answers:

Yes. Contractors can expense up to $300 for home office equipment.

That answer may sound helpful. It is also dangerous if the retrieved policy never said it.

A better product path might say:

The provided policy excerpt only covers full-time employees. I could not verify the contractor rule.

The difference is not politeness. The difference is grounding the answer in available evidence.

6. How It Works At A Practical Level

At a practical level, hallucination risk appears when generated output crosses a trust boundary.

model completion -> product displays claim -> user acts on claim

Product teams reduce the risk with techniques that fit the use case:

• retrieve relevant source material before generation
• include tool results for fresh or private facts
• instruct the model to state when evidence is missing
• constrain outputs into formats software can validate
• show citations or supporting snippets where appropriate
• use deterministic checks for facts software already knows
• test behavior with evals and failure cases
• add human review for high-risk actions

None of these makes every answer perfect. They create evidence and guardrails around a model that generates language.

7. Where You See This In Real AI Products

In a document Q&A product, hallucination risk appears when the answer mentions details missing from the retrieved document chunks.

In a search answer product, the risk appears when a summary overstates what sources support.

In a coding assistant, the risk appears when it invents APIs, file names, tests, or repo behavior.

In a support draft tool, the risk appears when it promises refunds or account actions that policy and account data do not allow.

In an agent workflow, the risk can become more serious if a wrong generated assumption leads to a tool call or a sequence of bad steps.

8. Common Confusions

Hallucination does not mean the model is lying.

Lying implies intent. The engineering problem is unsupported or incorrect generated output.

Hallucination is not fixed by longer prompts alone.

Better instructions can help, but evidence, retrieval, tools, validation, and evaluation often matter more.

Retrieval does not remove all hallucinations.

It can provide grounding context. The model can still misread, overstate, or ignore it.

A refusal is not always a failure.

For missing evidence or high-risk tasks, "I cannot verify that" may be the correct product behavior.

9. What This Does Not Mean

This does not mean generative models are useless.

It means generated language needs reliability design that matches the stakes.

This does not mean every output needs the same verification burden.

A brainstorming draft and a medical instruction do not carry the same risk.

This does not mean the only fix is training a new model.

Many product improvements come from better context, better retrieval, tool access, output checks, and evaluation.

10. What To Learn Next

Learn how outside information can be brought into a workflow in Retrieval In Plain English.

Then learn the retrieval-plus-generation pattern in RAG In Plain English.