What Is a Context Window?

A context window is the maximum amount of text a language model can take into account in a single request. It is measured in tokens, the word pieces a model reads and writes (a token is often a short word or part of a longer one). Everything counts toward it: the system instructions, the documents and conversation history you send, the definitions of any tools, and the answer the model writes.

The context window is the model’s working memory for one request. It is separate from what the model learned in training. Anything outside the window, the model simply does not see.

What Counts Toward the Context Window

Vendor documentation is specific about this. Anthropic’s, for example, states that the system prompt, every message including tool results, images and documents, the tool definitions, and the generated output including any reasoning (“thinking”) tokens all count toward the limit (Anthropic context windows). In practice the budget is split between:

  • instructions and tool definitions (fixed for each call),
  • retrieved documents or files,
  • tool results from earlier steps (often the largest part in agents),
  • conversation history,
  • the space reserved for the answer, which is a separate output limit in most APIs.

When input exceeds the window, the request fails or the application has to cut something. Chat products often drop the oldest messages silently, which is why a long chat can “forget” what was agreed at the start.

How Big Context Windows Are

Window sizes have grown fast, and several current commercial models accept around one million tokens in a single request, enough for several long books or a medium-sized codebase. Checked in November 2026. Sizes change with every model release, so check the vendor’s model table rather than relying on a number in an article.

Bigger Is Not Automatically Better

A large window means the model can accept a lot of text. It does not mean the model uses all of it equally well.

  • Research known as “Lost in the Middle” found that models answer best when the relevant information is near the start or end of the input and worse when it is in the middle (arXiv 2307.03172).
  • Chroma’s 2025 study of 18 models found performance becomes less reliable as input length grows, even on simple tasks, and that a focused prompt of about 300 tokens beat the same question asked inside about 113,000 tokens of mostly irrelevant history (Chroma context rot).
  • Every token costs money and adds latency. Filling a window on every call is expensive at scale, even with prompt caching.

This decline is commonly called context rot, and it is the reason context engineering exists as a discipline.

Long Context vs Retrieval

The two approaches complement each other:

ApproachWorks well when
Put everything in the windowThe material is small and stable, and the task needs a view of all of it at once (a contract review, a single codebase module)
Retrieval-augmented generation (RAG)The material is large, changes often, has access rules, or answers must cite a specific passage

Many systems combine them: retrieval narrows the material, and a large window lets the model read the retrieved documents in full rather than in small fragments.

Common Failure Modes

  • Silent truncation: the application cuts input to fit and the model answers from incomplete data with full confidence.
  • History bloat: agents that keep every tool result until the window fills, then degrade or fail mid-task.
  • Lost instructions: constraints given early in a long session stop being followed.
  • Cost surprises: a feature that sends the full document set on every request.
  • Output squeezed out: so much input that the model has too little room left for its answer or reasoning.

These failures come from the size limit itself. Problems with what is chosen to fill the window, such as stuffing it “just in case” or letting untrusted text carry instructions, are covered under context engineering.

The context window sets the limit; context engineering decides what fills it. Prompt caching lowers the cost of repeated content but does not free space, since cached tokens still count. Long-running AI agents use compaction (summarising earlier work) and external memory to work beyond a single window.

Context Windows in Practice

The window is planned as a budget before the first prompt is written: so many tokens for instructions and tool definitions, so many for retrieved documents, and a fixed reserve for the answer and any reasoning. The token count of each part is then measured on real requests in tracing tools such as LangSmith, because the largest consumer is often a tool result nobody expected to be long. Before a model is chosen for its window size, the team runs the actual task at the input length it will see in production, since a model that accepts a million tokens may answer worse at that length. When content has to be cut, the system says so, and an answer built on partial data is flagged as partial.

Back to Glossary

Let's talk about your project

We'll help you harness AI and ML technologies to create innovative software solutions

File requirements: pdf, doc, docx, rtf, ppt, pptx — max 15 MB

No file selected