Retrieval-Augmented Generation (RAG)
A pattern where an AI agent retrieves relevant documents from a knowledge base before generating a response, grounding its output in specific data rather than only training memory.
Also known as: RAG · retrieval augmented generation · vector search
What is Retrieval-Augmented Generation (RAG)?
Retrieval-Augmented Generation (RAG) is the pattern where an AI agent fetches relevant documents from a knowledge base (usually a vector database) and includes them in the model's context before generating a response. This grounds the answer in specific, up-to-date data rather than the model's training memory, which is frozen at a cutoff date and prone to hallucination on facts the model never saw. RAG is the dominant pattern for enterprise AI agents in 2026, but it is no longer the only option: as frontier model context windows pass 1 million tokens, smaller knowledge bases can be loaded directly without retrieval, and agentic RAG (where the agent loops over retrieval until confident) is replacing single-shot RAG for complex queries.
Christos Papadimitriou, theagency47 · Updated May 18, 2026Why RAG matters
A foundation model’s knowledge ends at its training cutoff date. If a customer asks about something that happened last week, or references a document the model has never seen, the model will either hallucinate confidently or refuse. RAG fixes this by giving the model the relevant document at query time, the model sees the source, cites it, and stays grounded.
For business use cases this is the difference between a chatbot that makes up answers and an agent that gives correct, source-cited responses based on the client’s own data.
How theagency47 uses RAG in agent builds
Most of our Workforce Starter and Workforce Pro builds use RAG over the client’s knowledge base, usually internal docs, support tickets, product specs, or contract templates. The agent’s system prompt tells it to retrieve before answering, and the eval suite includes “no-hallucination” test cases to catch the model improvising past its sources.
For smaller knowledge bases (under 200 pages or ~150K tokens), we increasingly skip the vector database entirely and load the whole corpus into the context window. Simpler, fewer moving parts, and a 2026 model can attend to all of it.
FAQ
Does RAG eliminate hallucination?
No, but it reduces it sharply on questions where the answer is in the retrieved documents. Agents without RAG hallucinate roughly 35% more on tasks that need fresh information. RAG is a hallucination control, not a hallucination cure, proper evals still matter.
When should I not use RAG?
When the knowledge base is small enough to fit in the context window directly (under ~150K tokens for current Claude models). RAG adds complexity (a vector store, an embedding model, a retrieval step) and that complexity earns its keep only when the corpus is too large to load.
What is “agentic RAG”?
The 2026 evolution: instead of one retrieval step before the answer, the agent loops, retrieves, reasons, retrieves again with refined queries, critiques its own draft, retrieves more, and stops when confident. Higher quality on complex queries, higher cost per query.
Related terms
Knowledge Base (RAG)
The collection of documents an AI agent retrieves from at decision time. The mechanism is called retrieval-augmented generation (RAG).
Large Language Model (LLM)
A neural network trained on large text corpora that takes text input and produces text output. The underlying engine inside an AI agent, not the agent itself.
Hallucination
An AI agent producing output that sounds confident but is factually wrong, typically because the model invented information not present in its inputs.