Context windows.
“Fits” is not “works”.

A token is a text fragment. For ordinary English prose, roughly three-quarters of a word on average — but the average conceals wide variation. Code, unusual names, non-Latin scripts and numbers all tokenise less efficiently, sometimes dramatically so. If your content is not English prose, measure rather than assume, because a budget built on the rule of thumb can be out by a factor of two.

The window covers everything. System prompt, conversation history, retrieved passages, tool definitions, tool results and the output being generated all share it. A large window fills faster than people expect, and the first thing to overflow is usually the retrieved context that the answer depended on.

Input and output are priced differently, usually with output several times more expensive. A system that sends large context and generates short answers has a very different cost profile from one that does the reverse, and optimising the wrong side is common.

Conversations grow quadratically in cost. Each turn re-sends the whole history. A twenty-turn conversation is not twenty times the cost of one turn; it is considerably more. Summarising or truncating history is a design decision, and leaving it to the default is how a bill surprises somebody.

Very large context windows invite an appealing architecture: skip retrieval, put the whole corpus in the prompt, let the model find what it needs. It works in a demonstration and it has three problems in production.

Recall across a very long context is uneven. Information in some positions is attended to more reliably than in others, and the pattern is model- and task-specific. A system that works on a fifty-page document may quietly degrade on a five-hundred-page one, without any error to indicate it.

You pay for every token on every call. A retrieval system sends the relevant few thousand tokens; a long-context system sends everything, every time. At any volume the difference is large and permanent.

And you lose citation. Retrieval knows which passage it used and can show it. A model reading everything cannot tell you where its answer came from, which in any regulated or research setting is disqualifying.

The reasonable use is exploratory and one-off work, where the engineering saving is real and the volume is low. For a system that runs continuously, retrieval remains the right architecture and long context is a convenience rather than a replacement.