Context Window
The maximum number of tokens an LLM can attend to in a single request, system prompt, conversation history, retrieved documents, and the new user message all count toward it.
Also known as: context length · token budget · context size · context limit
What is Context Window?
A context window is the maximum number of tokens an LLM can process in a single request, including the system prompt, conversation history, retrieved documents, tool outputs, and the new user message. By 2026, frontier models like Claude and Gemini support context windows of 1 million tokens or more, roughly 750,000 words, or about 1,500 pages of plain text. Larger context windows change the economics of AI agents: knowledge bases that previously required retrieval can now be loaded whole, agents can hold longer conversations without losing earlier context, and the cost-per-query rises proportionally to how much of the window is filled. The context window is the single most important sizing constraint when designing an agent's memory, retrieval, and conversation strategy.
Christos Papadimitriou, theagency47 · Updated May 18, 2026Why context window matters
Every token an agent receives (instructions, history, retrieved documents, tool outputs) costs money to process and counts against the window. Run out of room and the model starts forgetting earlier turns of the conversation, dropping retrieved documents, or refusing the request entirely.
For agent builders, the context window is the budget. Cramming a 50-page contract plus 30 turns of conversation plus tool definitions into a model with a 200K context window is fine. Doing the same on a model with an 8K context window is impossible.
How theagency47 uses context window in agent builds
We pick the model first, then design backwards from the context budget. A Workforce Pro deployment with 5 agents and complex knowledge base integration usually runs on Claude with a 200K-1M token window. A Spark build with one tight workflow might run on a smaller, cheaper model with 32K-100K, every token saved is margin preserved.
For cost modeling we estimate the typical context fill rate (~30-60% on most agent workloads) and multiply by the model’s price per million tokens. A bigger window does not automatically mean a bigger bill, it means more headroom, used when needed.
FAQ
Is a bigger context window always better?
No. Bigger windows cost more per token-processed and can introduce “lost in the middle” effects where the model attends less reliably to information in the middle of a long context. For most production agents in 2026, 200K is the sweet spot.
How is context window different from memory?
Context window is short-term, everything the model sees in this single request. Long-term memory is something we build separately (vector stores, summarization, structured state) and load into the context window at the right moment. The window itself does not remember anything between requests.
Can I just load everything into a 1M context window and skip RAG?
For small knowledge bases (under ~150K tokens, roughly 200 pages), yes, and we often do. For larger corpora, you still need retrieval because loading the whole thing into every query is wasteful and slow.
Related terms
Large Language Model (LLM)
A neural network trained on large text corpora that takes text input and produces text output. The underlying engine inside an AI agent, not the agent itself.
System Prompt
The instructional text that defines an AI agent's role, voice, boundaries, and decision rules. The single most important customization layer of an agent.
Knowledge Base (RAG)
The collection of documents an AI agent retrieves from at decision time. The mechanism is called retrieval-augmented generation (RAG).