What Is AI Hallucination?

An AI hallucination is output from a language model that sounds confident and plausible but is false or not supported by any source: an invented fact, a quote nobody said, a court case that does not exist, a wrong figure, or a software function that was never written. The model is not lying; it produces the most likely-sounding continuation, and sometimes that continuation is wrong. Hallucinations are a property of how these models work, so systems are designed to catch and contain them rather than assume they will disappear.

NIST’s Generative AI risk profile uses the term “confabulation” for the same behaviour (NIST AI 600-1).

Why Language Models Hallucinate

A large language model learns to predict likely text. It has no built-in record of where each fact came from, and it has no reliable internal signal that says “I do not know this”. Several factors make errors more likely:

  • Rare facts. Things that appear seldom in training data (a small company’s founding date, a niche regulation) are recalled poorly.
  • Questions outside the training period. The model does not know about events after its training data ends.
  • Pressure to answer. A 2025 paper by Kalai and colleagues argues that training and common benchmarks reward guessing over admitting uncertainty, because a guess sometimes scores and “I don’t know” never does (Why Language Models Hallucinate, arXiv 2509.04664).
  • Missing or noisy context. When retrieval returns the wrong document, or none, the model fills the gap from memory.
  • Long, cluttered inputs. Accuracy drops as the context fills with irrelevant material.

Types of Hallucination

  • Factual: a wrong date, name, number or event.
  • Fabricated sources: citations, links or quotes that do not exist.
  • Unfaithful to context: the answer contradicts or goes beyond the documents the model was given, a key failure in retrieval-augmented generation (RAG).
  • Reasoning errors: each step looks plausible but the conclusion does not follow, or arithmetic is wrong.
  • Tool and code hallucination: calling a tool that does not exist, inventing parameters, or importing a package that is not real.

How to Reduce Hallucinations

No single fix removes them. Production systems layer several controls:

  1. Ground answers in sources. Retrieve the relevant documents, instruct the model to answer only from them, and require citations.
  2. Allow “not found”. Make declining to answer an explicit, acceptable outcome, and test that the system actually does it.
  3. Compute numbers in code. Totals, prices and metrics come from a database query or tested code, with the model only presenting the result.
  4. Constrain the output. Structured outputs and schema validation catch invented fields and malformed values.
  5. Verify after generation. Check that every cited passage exists and supports the claim; check that referenced IDs, links and functions are real.
  6. Measure it. Track faithfulness and factual accuracy with LLM evaluation on real questions, before release and in production.
  7. Keep people on high-impact decisions. Anything that changes money, records or customer commitments is reviewed by a person.

A larger or newer model usually hallucinates less on general knowledge, but switching models is not a control. It changes the error rate without telling you where the errors are.

Why It Matters for a Business

A hallucinated answer in a customer-facing assistant is a statement made by the company. A wrong refund policy, an invented product capability or a made-up legal reference can become a liability, and in regulated areas such as health, finance and law the cost of one confident error is high. Risk frameworks treat it accordingly: NIST lists confabulation among the risks that generative AI introduces or makes worse, and proposes concrete actions for managing it.

Hallucination is the main reason grounding, AI guardrails and evaluations exist. Good context engineering lowers the rate; human-in-the-loop review limits the damage when one gets through.

Hallucination Control in Practice

Hallucination control begins with a list of the kinds of claims a system will make, sorted by what a wrong one would cost. Figures and dates are pulled from a database and inserted into the answer; free-text claims carry a citation to the passage they rest on, and the model is instructed to say it does not know when no passage supports an answer. The test set deliberately includes questions the documents cannot answer, because a system that never declines those is guessing on the others too. See how this works in a text-to-SQL reporting assistant.

Back to Glossary

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected