Context Window

The context window is the working memory of a single call. The context window of a single call holds the instructions, the conversation so far, documents and results from tools, and depending on the model the output tokens as well. Whatever is not in it, the model does not know in that call, unless it is part of its training knowledge. If a framework agreement is not in the call, the model cannot reliably tell you its notice period.

The context window is measured in tokens, and its size depends on the model. It has grown substantially with each model generation. Anthropic states a context window of one million tokens for its current models and 200,000 tokens for older ones (documentation, as of November 2026). A million tokens is the amount of text in several long books. The model retains nothing between two calls: a chat application resends the history with every message.

For executives, the context window decides two things: what an AI application can handle in one go and what each call costs. A large context window does not guarantee that every piece of information in it gets used reliably. The real work is choosing what goes in.

What goes into the context window.

A call to a language model consists of several parts that together have to fit into the context window. First come the application's instructions, often a system prompt that sets the task, role and rules. If tools are connected, such as a search or database access, their descriptions are added. Then come the conversation so far, attached documents, images or files, the results of tools that have already run and finally the current question.

Depending on the model, the answer has to fit within this limit too. Anthropic's documentation states that everything in a request counts toward the context window, including tool definitions, and that the generated answer, including its thinking, counts as well.

The model's training knowledge is not part of the context window. It lives in the model's weights and is the same for every call. The context window is the part the application assembles for this one call.

That leads to a practical rule: every connected tool, every long instruction and every attached document takes up space, even if the specific question does not need it.

What happens when the window is full.

What happens when the window fills up depends on how the model is used. Through an API, the provider rejects a request whose input alone is already too large. If there is room for the input but not for the full answer, the answer stops at the limit.

In chat interfaces, users often notice none of this. Older parts of a long conversation are dropped or automatically summarized so the conversation can continue. That is convenient, but it has a consequence: instructions from early in a long conversation, such as a preferred tone or an important restriction, can quietly get lost. Answers then become less accurate without any error being reported.

For long pieces of work, a simple approach works well. Large tasks are split into sections. The progress so far is summarized, and a new conversation starts from that summary. Important instructions belong in the fixed instructions, not in an early message that eventually drops out of the window.

More context does not mean better answers.

A large context window means more room, not better selection. Anthropic's documentation describes accuracy and recall degrading as the token count grows and calls the phenomenon context rot. What is in the window, it says, matters as much as how much space is available.

A widely cited study by Nelson Liu and colleagues (Stanford University and others, published in the journal TACL in 2024) showed another pattern. The models tested used information at the beginning and end of a long input much better than information in the middle. The paper is titled "Lost in the Middle." Newer models do better on such tests, but the underlying problem is well known.

There is also a content risk. The more material sits in the window, the more likely it contains contradictions: an old and a new version of the same rule, a draft next to the current document. The model then has to decide which passage to follow, and it does not always choose correctly.

In practice, that means putting only what the answer needs into the window. Key instructions and the central material belong at the beginning or the end. And an answer drawn from a full window deserves no more credibility than one drawn from a lean one.

Large window or targeted search.

Large context windows raise a basic question: do you give the model entire documents, or does a search step pick the relevant passages first? The second approach is called RAG, retrieval-augmented generation.

Passing everything along has the edge when the task involves one long document or a few documents and connections across the whole text matter. A contract whose clauses refer to each other, or a report that has to be assessed as a whole, belongs in the window in one piece. That removes the risk of a search missing the decisive passage.

Search has the edge when the collection is large or changes constantly, when access rights apply or when many questions need answering. An assistant then receives only the relevant sections of the documentation instead of the entire archive. Each question costs less, and contradictory material is more likely to stay out. In exchange, there is a new risk: if the search step picks the wrong passage, the right one stays invisible because the rest of the archive is outside the window.

In practice, many applications combine both. A search makes the selection, and the large window allows that selection to be generous, passing along whole chapters instead of single paragraphs.

What the context window means for cost.

You do not pay for the size of the context window but for what a call actually loads into it and gets back. Billing is based on input and output tokens, usually at different rates. A call with an entire manual in context therefore costs many times more than a call with three selected passages from it.

Full windows have a second drawback beyond price: they make answers slower, because the model first has to process the entire input.

Prompt caching is an important remedy for both. The provider stores an unchanging beginning of the input, such as the system prompt or a fixed set of rules, and charges much less when it is read again. Anthropic points out that cached content still occupies the context window; caching changes the price, not the space used.

Some providers also charge more per token for very long inputs, others do not. Which rule applies is in the price list for each model and should be checked before committing to an application that works with long documents.

What a good implementation looks like.

The practice of deliberately assembling the window is called context engineering. Good implementations treat the window as a scarce resource, even when it is large.

That means each task has a defined set of content that belongs in the window and content that does not. Long histories are summarized, and old tool results are removed once they are no longer needed. Only the tools the task requires are connected. Key instructions sit in the fixed instructions, not somewhere in the middle.

Good applications log the input and output tokens for every call. They define what happens on overflow and test with both short and very long inputs to see whether quality holds.

When buying or commissioning an application, ask about exactly these points: How is it decided what goes into the context window? What happens when it is full? How was it tested that information from long documents is used reliably? If the answer to the last question is only the size of the window, the question has not been answered.

Where the term stops working

A large context window does not guarantee that every piece of information in it gets used reliably, and it is not long-term memory: once the call is over, its contents are gone. It is distinct from the token, its unit of measurement, from training knowledge, which lives in the model itself, from RAG, which selects what goes into the window, and from application memory features that store content outside and insert it again.