What is Context Window?
Context Window: The maximum number of tokens a model can consider in one request, covering instructions, documents, history, tool results, and its own reply.
The context window is the model's working memory for a single request. Everything the model uses to answer has to fit inside it: the system prompt, the user's message, retrieved documents, previous turns, tool definitions and results, and the output being generated. Current frontier models offer windows from hundreds of thousands of tokens up to a million or more.
A bigger window does not mean the model uses all of it equally well. Quality on details buried in the middle of very long inputs can drop, and every token in the window costs money and adds latency. That is why most production systems still retrieve only the relevant pieces of information instead of loading everything, and why agents that run for many steps need a strategy for summarizing or trimming old context.
Why it matters when you build or buy AI
Long context windows tempt teams to skip retrieval and paste the whole knowledge base into every prompt. It demos well and scales badly: costs climb, latency grows, and accuracy can fall. For anything with more than a few dozen documents or high request volume, plan for retrieval and context management from day one.
Example
An internal assistant has 4,000 policy documents. Instead of loading them all, it retrieves the five most relevant passages per question and keeps each request under 10,000 tokens.
Related terms
More in AI & LLMs
Book 20 minutes, or send a brief
A former CTO replies. You leave with a written plan and a fixed estimate.
Prefer LinkedIn? Connect with Harish.