A context window refers to all the text a language model can reference when generating a response, including the response itself, according to Anthropic’s Claude platform documentation. The window is a working memory for the model, separate from the large corpus of data it was trained on.
The generative-AI sense applies throughout this entry: The pool of text a model can see when it answers. Corpus linguistics and word-embedding training use the same term for a fixed span of neighboring words, which is a different concept. A larger context window allows the model to handle more complex and lengthy prompts, which makes curating what is in context just as important as how much space is available.
Key Takeaways
- The context window holds all the text a language model can reference when generating a response, including the response itself, and works as a working memory rather than the corpus the model trained on, according to Anthropic’s Claude platform documentation. The unit is the token, and 100 tokens is approximately 75 words of English, per OpenAI.
- Capacity runs up to 1 million tokens, depending on the model, and the window holds the conversation history plus the new output the model generates, per Anthropic’s platform documentation.
- Earlier versions of generative models were only able to process 8,000 tokens at a time; newer models pushed this further by accepting 32,000 or even 128,000 tokens, and Gemini is the first model capable of accepting 1 million tokens, according to Google’s Gemini API documentation.
- In practice, 1 million tokens would look like 50,000 lines of code with the standard 80 characters per line, 8 average-length English novels, or transcripts of over 200 average-length podcast episodes.
- All 17 long-context language models evaluated on 13 representative tasks claim context sizes of 32,000 tokens or greater, yet only half of them can maintain satisfactory performance at the length of 32,000 tokens, according to NVIDIA’s RULER benchmark.
How Does a Context Window Work?
Each turn has an input phase that contains all previous conversation history plus the current user message, and an output phase that generates a text response which becomes part of the input for the next turn. Three steps sit underneath that loop.
1. Text Is Split Into Tokens
Tokens are the building blocks of text that OpenAI models process, and they can be as short as a single character or as long as a full word, depending on the language and context, per OpenAI’s help documentation. Spaces, punctuation, and partial words all contribute to token counts.
Two named documents make the unit concrete: the OpenAI Charter is 476 tokens, and the US Declaration of Independence is 1,695 tokens.
2. Attention Makes Every Token in the Window Available
The Transformer is a network architecture based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. That design choice is what makes the window a single pool. The model weighs everything inside it at once instead of stepping through the text in order. That is what lets a question at the end reach material placed at the start.
3. The Window Is a Budget, and the Answer Spends From It
Everything in the request counts toward the context window: the system prompt, every message including tool results, images, and documents, and the tool definitions. The output the model generates for the turn, including its extended thinking, counts too, according to Anthropic.
A desk with a fixed surface is the closest everyday match. Reference notes spread across it compete with the page being written on. A desk buried in source material leaves less room for the answer.
| Element | Draws on the context window? | Where it is documented |
|---|---|---|
| System prompt | Yes | Anthropic, Claude platform documentation |
| Conversation history | Yes | Anthropic, Claude platform documentation |
| Tool definitions and tool results | Yes | Anthropic, Claude platform documentation |
| Attached images and documents | Yes | Anthropic, Claude platform documentation |
| The model’s own output and extended thinking | Yes | Anthropic, Claude platform documentation |
| Data the model was trained on | No | Anthropic, Claude platform documentation |
Sources: Anthropic Claude platform documentation; OpenAI help documentation.
As token count grows, accuracy and recall degrade, a phenomenon known as context rot, according to Anthropic’s platform documentation. A vendor advertising one of the largest capacities also documents where that capacity stops holding up. That is unusual, and it is worth reading closely.
How Big Is a Context Window?
Gemini is the first model capable of accepting 1 million tokens, after earlier versions of generative models were only able to process 8,000 tokens at a time and newer models pushed this further by accepting 32,000 or even 128,000 tokens, according to Google’s Gemini API documentation. Anthropic documents its own capacity as up to 1 million tokens, depending on the model.
In practice, 1 million tokens would look like 50,000 lines of code with the standard 80 characters per line, all the text messages you have sent in the last 5 years, 8 average-length English novels, or transcripts of over 200 average-length podcast episodes.
| Documented tier | Approximate token count | Roughly what fits | Documented by |
|---|---|---|---|
| Earlier generative models | 8,000 tokens | A long article | Google, Gemini API documentation |
| Mid-generation models | 32,000 to 128,000 tokens | A short book or a large file set | Google, Gemini API documentation |
| Gemini long context | 1 million tokens | 50,000 lines of code, or 8 average length English novels | Google, Gemini API documentation |
| Claude, model dependent | Up to 1 million tokens | Conversation history plus the generated output | Anthropic, Claude platform documentation |
Sources: Google Gemini API documentation; Anthropic Claude platform documentation.
Advertised ceilings move with each model release. Current per-model context window sizes are worth reading straight from a tracked column instead of a static comparison table.
Both vendors publish detailed context documentation, and they are the pair most often compared in our OpenAI and Anthropic adoption data. Their published ceilings are worded differently enough that merging them into one industry number misreads both.
Why Does a Context Window Matter?
A larger context window allows the model to handle more complex and lengthy prompts, but more context is not automatically better, according to Anthropic’s platform documentation. That sentence is the whole reason the advertised number needs a second source.
Despite achieving nearly perfect accuracy in the vanilla needle-in-a-haystack test, almost all of the 17 long-context language models evaluated in the RULER benchmark exhibit large performance drops as the context length increases, per NVIDIA’s published results. While these models all claim context sizes of 32,000 tokens or greater, only half of them can maintain satisfactory performance at the length of 32,000 tokens.
The Lost in the Middle analysis adds a second variable: Position. Performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models. The finding comes from two tasks that require identifying relevant information in the input context: multi-document question answering and key-value retrieval.
The advertised number is a ceiling rather than a service level. It states what a model will accept, not what it will reliably use. The distance between those two readings is wide enough that a published benchmark exists to measure it.
A retrieval miss inside a long window is a different failure from a fabricated fact. The second pattern is what our LLM hallucination rate data tracks. Curating what goes into the window helps reduce retrieval failures, though nothing in the published evidence supports the claim that it prevents them.
Pros, Cons, and Risks
Advantages
- A larger context window allows the model to handle more complex and lengthy prompts.
- More limited context windows often require strategies like arbitrarily dropping old messages, summarizing content, using RAG with vector databases, or filtering prompts to save tokens, so an extensive context window invites a more direct approach: providing all relevant information upfront.
- Those techniques remain valuable in specific scenarios, according to Google’s Gemini API documentation.
- Gemini models demonstrate powerful in-context learning because they were purpose-built with massive context capabilities.
Trade-offs and Risks
- Accuracy and recall degrade as token count grows, a phenomenon known as context rot.
- Almost all of the 17 models evaluated in the RULER benchmark exhibit large performance drops as the context length increases.
- Performance can degrade significantly when changing the position of relevant information, indicating that current language models do not robustly make use of information in long input contexts.
- Cost and latency track what goes into the window. A padded prompt is billed and processed even when the extra material never gets used.
Curation is the practical lever, and it is the same discipline measured in our prompt engineering adoption data. Shorter, better-ordered prompts help reduce the failure rate. They do not eliminate it.
Context Window vs Memory vs Training Data
The context window is different from the large corpus of data the language model was trained on, and instead represents a working memory for the model, according to Anthropic’s Claude platform documentation. Training data is absorbed during training. A reader cannot see it, edit it, or count it in tokens per request.
The context window is rebuilt every turn, because the input phase contains all previous conversation history plus the current user message and the output phase generates a response that becomes part of the input for the next turn. Everything in it is counted, and the count resets against the same ceiling each time.
Product memory features are a third thing again. They live at the application layer and work by re-injecting earlier content back into the window. That means they spend the same budget instead of adding to it. None of the primary documentation cited here describes a specific vendor memory product, so the framing here is ours and carries no vendor endorsement.
Real-World Applications
Learning a Language From Documents Placed in the Window
Using only in-context instructional materials (a 500-page reference grammar, a dictionary, and 400 parallel sentences), Gemini learned to translate from English to Kalamang, a Papuan language with fewer than 200 speakers, with quality similar to a human learner using the same materials, according to Google’s Gemini API documentation.
Nothing was retrained for that result. The teaching materials simply sat in the window, which is what in-context learning means in practice.
Reading a Whole Codebase in One Request
1 million tokens is roughly 50,000 lines of code with the standard 80 characters per line, per Google’s Gemini API documentation. Where the relevant information sits in that span still matters, because performance is often highest when relevant information occurs at the beginning or end of the input context.
A function the model skims past becomes a code-review problem first. The downstream shape of that problem sits in our AI-generated code vulnerability data.
Long Agent Sessions and Tool Output
Tool definitions, tool results, images, and documents all count toward the context window, and the output the model generates for the turn, including its extended thinking, counts too, per Anthropic’s platform documentation.
Agent runs therefore reach the ceiling faster than the length of the original instruction suggests. Every tool call spends budget twice, once for the definition and once for the result.
What Happens When You Exceed the Context Window?
The request stops fitting, so something has to leave the window before the model can answer. The context window holds the conversation history plus the new output the model generates, and everything in the request counts toward it.
More limited context windows often require strategies like arbitrarily dropping old messages, summarizing content, using RAG with vector databases, or filtering prompts to save tokens. Which of those a given product applies varies by vendor, so read the documentation instead of assuming truncation works the same way everywhere.
Is a Bigger Context Window Always Better?
No. More context is not automatically better, and as token count grows, accuracy and recall degrade, a phenomenon known as context rot, according to Anthropic’s platform documentation.
The models evaluated in the RULER benchmark all claim context sizes of 32,000 tokens or greater, yet only half of them can maintain satisfactory performance at the length of 32,000 tokens. A bigger window buys headroom, which is real and useful. It does not buy accuracy at the far end of that headroom.
Conclusion
All 17 long-context language models evaluated on 13 representative tasks in the RULER benchmark claim context sizes of 32,000 tokens or greater, and only half of them can maintain satisfactory performance at that length. That gap is the practical definition of the term. The advertised size states what a model will accept, and a published evaluation states what it will hold.
Advertised ceilings keep climbing, and the reading skill has to climb with them. Treat the vendor number as the outer edge and check where published evaluations put the working range. Place the material that matters at the start or the end of the prompt.
