Hallucination
An AI agent producing output that sounds confident but is factually wrong, typically because the model invented information not present in its inputs.
Also known as: confabulation · fabrication
What is Hallucination?
Hallucination is when an AI agent produces output that sounds confident but is factually wrong, usually because the LLM invented information that was not in its inputs. In production it clusters around missing knowledge-base data, ambiguous requests and niche topics. theagency47 limits it with strict retrieval-augmented generation, escalation rules that send uncertain cases to a human, and adversarial eval cases.
Christos Papadimitriou, theagency47 · Updated May 15, 2026Why hallucinations happen
LLMs predict the next token based on statistical patterns from training data. When asked something they do not know, they default to what sounds plausible, which is often wrong but never marked as low-confidence. The model has no native “I do not know” output unless explicitly trained to produce one.
In production AI agents, three patterns produce most hallucinations:
- Missing knowledge base content. Agent is asked about a topic the KB does not cover; the LLM fills the gap from training data, which is generic and often wrong for the client’s specifics.
- Ambiguous user input. Agent gets a vague question and the LLM picks one of several plausible interpretations without checking.
- Adversarial input. A user crafts a question designed to elicit a confident-sounding wrong answer.
How theagency47 prevents hallucinations
Three layers of defense:
- Strict RAG. Every claim the agent makes must be traceable to a source in the knowledge base. No source → escalate, do not guess.
- Confidence-based escalation. Below a configurable threshold, agent flags for human review instead of replying.
- Eval suite adversarial coverage. 15% of test cases (the adversarial bucket) explicitly probe for hallucination patterns. Pre-launch.
Practical result: when the agent cannot find a source, it escalates instead of guessing, so unsupported answers go to a human rather than to the customer.
FAQ
Is hallucination unique to LLMs?
Not exactly, any ML system can be wrong. But LLMs are uniquely prone to confident-sounding errors because their output is fluent language, which masks low-confidence reasoning.
Does using a bigger model fix hallucinations?
Partially. Larger models hallucinate less on general knowledge. They do not solve business-specific hallucinations, those require RAG and escalation discipline.
Related terms
Knowledge Base (RAG)
The collection of documents an AI agent retrieves from at decision time. The mechanism is called retrieval-augmented generation (RAG).
Eval Suite
A structured set of test cases with known expected outputs that verifies an AI agent's behavior before deployment and after every change.
Human-in-the-Loop
A deployment pattern where every AI agent action is reviewable by a human before downstream effects, used during the first weeks of production.
