Technology

Context engineering
for LLM systems.

The prompt is a small part of what determines output quality. What dominates is everything else in the window: what was retrieved, what was kept from earlier, what was summarised away, and what was quietly dropped.

WHY THE NAME CHANGED

"Prompt engineering" described a real activity when the interesting variable was the instruction. Today, in any production system, the instruction is a stable few hundred tokens and the interesting variable is the several thousand tokens of retrieved documents, conversation history, tool output and structured state surrounding it.

That is an information-retrieval and state-management problem, not a writing problem. Teams that treat it as a writing problem spend weeks rewording instructions to compensate for the fact that the right document was never retrieved — a fix that cannot work, applied with great diligence.

Long context windows changed the shape of this without removing it. When everything fits, the constraint stops being capacity and becomes attention: models can still under-use information placed mid-context — the effect varies by model, so test at the length you intend to use, and cost and latency scale with what you include. "Put it all in" is a strategy that works in a demo and gets expensive and unreliable in production.

THE FOUR LEVERS

Retrieval — what gets fetched

Hybrid search (dense plus keyword) has beaten pure vector search on most of the real corpora we have measured, because exact identifiers, product codes and names are exactly what embeddings handle worst. Reranking a wide candidate set with a cross-encoder has, in our experience, outperformed trusting top-k similarity.

Selection — what makes the cut

Retrieved is not the same as included. Deduplicate near-identical passages, drop anything below a relevance floor, and put the most important material at the start or end where attention is strongest — position within the window measurably affects use.

Compaction — what gets compressed

Long conversations need summarising, and every summary discards something. Compact structurally — preserve decisions, entities, constraints and open questions verbatim; compress narrative. A naive "summarise the conversation" loses precisely the specifics that were load-bearing.

Memory — what persists

Durable facts about a user or account belong in a database with explicit read and write points, not in an ever-growing context. "Memory" implemented as unbounded accumulation becomes both costly and noisy within weeks.

THE FAILURE MODES

Lost in the middle. Information placed mid-context is used less reliably than material at either end. If something is critical, position it deliberately rather than wherever the concatenation happened to put it.

Context poisoning. One wrong retrieved passage can anchor an entire response, and downstream reasoning inherits the error confidently. A relevance floor that excludes weak matches beats including them "just in case".

Distraction. More context is not monotonically better. Padding a window with marginally relevant material measurably degrades performance on the actual task, as well as costing money.

Silent truncation. The failure that hurts most in production: context exceeds the limit, something gets dropped, and nothing reports it. Instrument context assembly, log what was excluded, and alert on truncation rather than discovering it through mysterious quality complaints.

QUESTIONS

QUESTIONS — 4

No, and the reasons are practical rather than theoretical. Cost scales with tokens, so stuffing a large corpus into every request is expensive at volume. Latency scales too. Attention degrades over very long contexts. And most corpora are far larger than any window. Long context does remove the need for aggressive chunking in mid-sized cases, which is a genuine simplification — but retrieval remains the mechanism that keeps cost and quality manageable.

Chunk on structure rather than length wherever the documents have structure — sections, clauses, table boundaries. Where they do not, 400–800 tokens with modest overlap is a reasonable default. Treat it as a parameter to tune against your retrieval eval set, not a setting to get right once.

Structured compaction: keep the system instruction and the most recent turns verbatim, and compress the middle into a structured summary that preserves decisions, entities and constraints explicitly. Store the full transcript separately so nothing is genuinely lost and the agent can retrieve back into it if needed.

Yes, wherever a large stable prefix is reused — system instructions, few-shot examples, a fixed document. It reduces both cost and latency substantially on cache hits. It does constrain how you order context, since the cacheable part must be a stable prefix, so it is worth designing for rather than adding afterwards.

Keep reading

Architecture

Enterprise RAG Systems

→

Technology

Choosing a Vector Database

30-minute technical call · no deck