Resource

Context windows.
“Fits” is not “works”.

Context windows are the unit in which AI systems are budgeted, and the most commonly misunderstood part of the stack. Two facts do most of the work: tokens are not words, and filling a window is not the same as being understood.

All Resources →

The practical facts

A token is a text fragment.

For ordinary English prose, roughly three-quarters of a word on average — but the average conceals wide variation. Code, unusual names, non-Latin scripts and numbers all tokenise less efficiently, sometimes dramatically so. If your content is not English prose, measure rather than assume, because a budget built on the rule of thumb can be out by a factor of two.
The window covers everything.

System prompt, conversation history, retrieved passages, tool definitions, tool results and the output being generated all share it. A large window fills faster than people expect, and the first thing to overflow is usually the retrieved context that the answer depended on.
Input and output are priced differently

, usually with output several times more expensive. A system that sends large context and generates short answers has a very different cost profile from one that does the reverse, and optimising the wrong side is common.
Conversations grow quadratically in cost.

Each turn re-sends the whole history. A twenty-turn conversation is not twenty times the cost of one turn; it is considerably more. Summarising or truncating history is a design decision, and leaving it to the default is how a bill surprises somebody.

The long-context trap

Very large context windows invite an appealing architecture: skip retrieval, put the whole corpus in the prompt, let the model find what it needs. It works in a demonstration and it has three problems in production.

Recall across a very long context is uneven. Information in some positions is attended to more reliably than in others, and the pattern is model- and task-specific. A system that works on a fifty-page document may quietly degrade on a five-hundred-page one, without any error to indicate it.

You pay for every token on every call. A retrieval system sends the relevant few thousand tokens; a long-context system sends everything, every time. At any volume the difference is large and permanent.

And you lose citation. Retrieval knows which passage it used and can show it. A model reading everything cannot tell you where its answer came from, which in any regulated or research setting is disqualifying.

The reasonable use is exploratory and one-off work, where the engineering saving is real and the volume is low. For a system that runs continuously, retrieval remains the right architecture and long context is a convenience rather than a replacement.

Questions

Questions about Tokens & Context Windows.

How do we estimate cost before building?

Tokenise a representative sample of your real content with the tokeniser for the model you intend to use, multiply by realistic volume, and account for the conversational growth if there is a chat interface. That estimate is usually within a reasonable margin, and it is a half-day of work rather than a guess.

What is prompt caching?

Providers can cache a stable prefix so repeated content is cheaper and faster on a hit. It requires designing your context so the stable part genuinely is a prefix — system prompt and fixed reference material first, variable content last. That is an architectural decision worth making before integration rather than retrofitting.

Should we summarise long conversations?

Usually, past some length. Summarise older turns and keep recent ones verbatim — it controls cost and it often improves quality, because a very long history dilutes attention on what actually matters now.