The prompt is a small part of what determines output quality. What dominates is everything else in the window: what was retrieved, what was kept from earlier, what was summarised away, and what was quietly dropped.
All Technology & Model Stack →
"Prompt engineering" described a real activity when the interesting variable was the instruction. Today, in any production system, the instruction is a stable few hundred tokens and the interesting variable is the several thousand tokens of retrieved documents, conversation history, tool output and structured state surrounding it.
That is an information-retrieval and state-management problem, not a writing problem. Teams that treat it as a writing problem spend weeks rewording instructions to compensate for the fact that the right document was never retrieved — a fix that cannot work, applied with great diligence.
Long context windows changed the shape of this without removing it. When everything fits, the constraint stops being capacity and becomes attention: models can still under-use information placed mid-context — the effect varies by model, so test at the length you intend to use, and cost and latency scale with what you include. "Put it all in" is a strategy that works in a demo and gets expensive and unreliable in production.
No, and the reasons are practical rather than theoretical. Cost scales with tokens, so stuffing a large corpus into every request is expensive at volume. Latency scales too. Attention degrades over very long contexts. And most corpora are far larger than any window. Long context does remove the need for aggressive chunking in mid-sized cases, which is a genuine simplification — but retrieval remains the mechanism that keeps cost and quality manageable.
Chunk on structure rather than length wherever the documents have structure — sections, clauses, table boundaries. Where they do not, 400–800 tokens with modest overlap is a reasonable default. Treat it as a parameter to tune against your retrieval eval set, not a setting to get right once.
Structured compaction: keep the system instruction and the most recent turns verbatim, and compress the middle into a structured summary that preserves decisions, entities and constraints explicitly. Store the full transcript separately so nothing is genuinely lost and the agent can retrieve back into it if needed.
Yes, wherever a large stable prefix is reused — system instructions, few-shot examples, a fixed document. It reduces both cost and latency substantially on cache hits. It does constrain how you order context, since the cacheable part must be a stable prefix, so it is worth designing for rather than adding afterwards.
It is usually retrieval. Measuring the two separately is the fastest way to find out.
30-minute technical call · no deck
What we built, what stage it is at, and a link to the live thing wherever there is one.
yourOS is an operating system with an AI at its heart. You talk to it like a person, out loud or typed. It makes your calls, plans your days and works on your projects with a crew of agents, on your own private server. Our own product, built from the first commit; the waitlist is open.
Website, backend and app assets for Sonora, a free AI sound-therapy app on iOS and Android published by Cognitive AI.
Brand, website, prototype, apps on five platforms, the AI backend and the algorithm-intelligence pipeline for Podswap, a social growth engine that runs without a human in the loop.
In closed beta: Roxy, a private AI assistant with permanent memory running on infrastructure the user owns. Brand, site, product, backend and conversational design.
Live at coherence.com: brand identity generated with AI agents we directed, the motion system and website for the group behind Sonora, AboutAI, WakeUp and MudShadows. Agent work remains in build.
Live at domainbooth.com for DomainBooth, a premium domain brokerage: a new animated mark, every page redesigned and rebuilt with designed mobile layouts, and a custom CMS for its sales record, featured names, press and enquiries.
Apps, platforms and campaigns for these teams, long before the AI work. Every one is written up on the work page — what we did, and where our part ended.
