New: outcomized services. Pay per result, with a monthly cap you set. See pricing →
AI & LLMs

What is Context Window?

Context Window: The maximum number of tokens a model can consider in one request, covering instructions, documents, history, tool results, and its own reply.

The context window is the model's working memory for a single request. Everything the model uses to answer has to fit inside it: the system prompt, the user's message, retrieved documents, previous turns, tool definitions and results, and the output being generated. Current frontier models offer windows from hundreds of thousands of tokens up to a million or more.

A bigger window does not mean the model uses all of it equally well. Quality on details buried in the middle of very long inputs can drop, and every token in the window costs money and adds latency. That is why most production systems still retrieve only the relevant pieces of information instead of loading everything, and why agents that run for many steps need a strategy for summarizing or trimming old context.

Why it matters when you build or buy AI

Long context windows tempt teams to skip retrieval and paste the whole knowledge base into every prompt. It demos well and scales badly: costs climb, latency grows, and accuracy can fall. For anything with more than a few dozen documents or high request volume, plan for retrieval and context management from day one.

Example

An internal assistant has 4,000 policy documents. Instead of loading them all, it retrieves the five most relevant passages per question and keeps each request under 10,000 tokens.

Want one assistant that knows your business?Answers from your docs and tools, with citations and permissions. From $7,500.

Related terms

More in AI & LLMs

Next step

Book 20 minutes, or send a brief

A former CTO replies. You leave with a written plan and a fixed estimate.

No spam. One reply within one business day. Or skip the form and book a call.

Thank you. Your brief is on its way. Grab a time now so we can talk it through.

Prefer LinkedIn? Connect with Harish.

Book a callFree lead test