RAG or long context?

We build retrieval systems, so start with the argument against ourselves. If your corpus fits in the window, putting it in the window is simpler, and simpler is worth a great deal. There is no chunking strategy to tune, no embedding model to choose, no vector store to operate and back up, no reranker, no relevance floor, and — the big one — no class of failure where the correct answer was in your data and the system never saw it.

For a genuinely large set of real problems that is now the right answer. A contract of eighty pages, a quarter's board papers, one long support conversation, a single codebase module, a policy handbook: these fit, and a retrieval layer in front of them is engineering that earns nothing. We have talked clients out of building one on exactly this basis, and the honest version of a comparison page says so before it says anything else.

The question is what happens when the corpus is bigger than the window, or grows, or is not the same corpus for every user. That is where the two approaches stop being alternatives with different ergonomics and start having genuinely different properties.

Retrieval fails by omission. The answer existed and the search did not surface it — the query used different vocabulary from the document, the chunk boundary split the relevant sentence, the reranker demoted it, the relevance floor cut it. This is frustrating and it has one large compensating property: it is directly measurable. Recall@k on a labelled query set tells you the rate, component evaluation separates it from generation quality, and every fix has an observable effect on a number. A failure you can put a metric on is a failure that gets fixed.

Long context fails by inattention. The material was in the window and the answer did not reflect it. Published work has repeatedly found that information placed in the middle of a long context is used less reliably than material at either end, and that performance degrades as the window fills — the effect varies substantially by model, which is itself the problem, since it means the behaviour is a property of the model version rather than of your system. Nothing is missing when this happens. There is no recall number to look at. You find it only through answer-level evaluation, and only if your evaluation set happens to contain a case that depends on something buried in the middle.

That asymmetry is underrated in the comparison. The long-context approach removes a failure mode that is easy to see and introduces one that is hard to see, and the second is worse to own in production even where it occurs less often. If you go long-context, the mitigation is deliberate ordering — put what matters at the start or the end rather than wherever concatenation happened to place it — and testing at the length you actually intend to run at, not at a convenient shorter one.

This is the row that ends the discussion in most enterprises, and it rarely appears in the benchmark comparisons because it is not a capability question.

In any organisation of size, two people asking the same question are entitled to different answers, because they are entitled to see different documents. A retrieval layer handles this natively: the search is filtered by the requesting user's entitlements before anything reaches the model, so the board papers are not in the context of a query from the graduate scheme. The filter is ordinary access-control code, it is testable, and it fails closed.

Long context has no equivalent mechanism. Whatever you assembled is visible to the model and therefore potentially reflected in the answer. You can of course assemble a per-user context — but building the set of documents this user may see, for this request, is retrieval. You have reimplemented the selection layer without the index, and usually without the query-time filtering that made it safe.

The same argument applies to deletion and residency. When a person exercises a right to erasure, deleting a record from an index and re-indexing is a defined operation. Knowing every place a document has been pasted into a context is not.

The cost shape is the part teams model last and regret first. A retrieval system sends a few thousand tokens of selected passages per query, roughly independent of how large the corpus is. A long-context system sends the whole assembled context on every request, so the per-query cost scales with the size of the material rather than with the difficulty of the question. At one query a day the difference is a rounding error. At tens of thousands, it is the line item somebody in finance asks about, and the ratio between the two is roughly the ratio between the assembled context and the selected passages.

Prompt caching changes this, conditionally. Where a large prefix is stable and reused — the same handbook, the same codebase, the same policy set, for many queries — caching substantially reduces both the cost and the time to first token on a hit, and it makes long context far more defensible than the raw arithmetic suggests. The conditions are the whole story: the prefix must be stable, so a context reassembled per user or per query gets few hits; caches expire, so low-traffic paths pay full price most of the time; and the cacheable material must sit at the front, which constrains how you order context at exactly the moment the previous section told you to order it by importance. Confirm the current cache lifetimes and pricing against the provider's current terms rather than a figure from a blog post, including ours.

Latency runs the same way in both directions and is worth measuring rather than assuming. Retrieval adds a search round trip, typically tens of milliseconds against a well-built index. Long context removes that and adds prefill time over a much larger prompt. Which is faster depends on your corpus size, your index and whether you are getting cache hits — so it belongs in a benchmark on your own data, not in an argument.

They are complementary, and the strongest systems we build use both: retrieval to select, and a large window to reason over what was selected. Long context did not remove the retrieval step; it made it much more forgiving. When a window was small, retrieval had to return the exactly right paragraph, which is a hard search problem. When the window is large, retrieval can return forty candidate passages and whole documents rather than fragments, and let the model sort it out — which means simpler chunking, less aggressive reranking, and a much lower penalty for a mediocre relevance score.

So the working rule is not “which one”, it is “how much selection”. Corpus that fits, one permission set, low volume: no retrieval layer, and do not let anyone sell you one. Corpus larger than the window, or growing, or per-user, or needing citations, or running at volume: retrieval, with a generous context budget behind it. Most real systems start at the first and arrive at the second, and the migration is much easier if the evaluation set was built at the start — because it is what tells you the day the simple version stopped being good enough.