Most AI projects that stall do not stall on the model. They stall on a document set that two departments maintain differently, a permissions model nobody can express, or the absence of anyone who can say which of two contradictory answers is correct. Here is how to find that out in a fortnight rather than in month six.
All Resources →
The failure that surprises people is not the model inventing an answer. It is the model giving a correct answer that the business disagrees with, because two of its own documents say different things and the retrieval step happened to find one of them.
The 2019 policy and the 2024 policy are both in the share drive, both look official, and neither says which supersedes the other. The onboarding guide and the intranet page describe different approval thresholds. The product specification and the support macro give different return windows. Every organisation over a certain age has hundreds of these, and nobody notices because a human reader applies context that is not written down anywhere: they know which team maintains which document, they know the 2019 one is dead, they have heard the corridor conversation.
A retrieval system has none of that context, and it will confidently surface the wrong one for the same reason a new joiner would. Worse, it does it consistently, at volume, and with a citation that makes the answer look verified.
There is a useful reframing here. A system that surfaces contradictions is doing you a service, and the fortnight of measurement below usually produces a list of them as its first real output. That list is an asset — it is the documentation debt made visible for the first time, and it is often worth more to the organisation than the system that found it. What it is not is something the model can resolve. Deciding which document is authoritative is an act of ownership, and it needs a person with the standing to make it.
Practically, this means one thing for scoping: prefer a corpus with a live owner and a maintenance habit over a larger one without. A thousand pages someone maintains beats forty thousand nobody does, and it is a much better first system.
Score each row honestly, and note where the honest answer is “I would have to ask”. A row you cannot answer is not neutral — it is a red row that has not been checked yet.
| Criterion | Ready | Workable | Stop and fix first |
|---|---|---|---|
| Access | You can produce an extract this week | Access needs a request and a fortnight | It lives in a system with no export |
| Enumeration | A list of what is in scope exists | A list can be built from a crawl | Nobody can say what is in scope |
| Permissions | Expressible as rules against identities | Expressible for most, with known exceptions | Lives in folder structure and memory |
| Ground truth | Adjudicated outcomes already recorded | An expert can label 200 cases | No one can say what the right answer was |
| Freshness | Automated update path exists | Manual refresh with a named owner | Last updated at some point, unclear when |
| Contradiction | Superseded versions are retired | Duplicates exist and are identifiable | Multiple current versions, no owner |
| Format | Text or clean structured records | Mixed, with scanned documents present | Spreadsheets used as databases |
| Deletion | A propagation path is documented | Deletion is manual but traceable | Copies exist in places nobody tracks |
| Ownership | A named person decides what is current | Ownership is shared but reachable | No owner, and no candidate |
Two or three rows in the right-hand column is normal and not a reason to stop; it is a reason to narrow the scope until those rows fall outside it. A red under permissions, ground truth or ownership is different in kind, because each of those blocks something the engineering cannot supply — a lawful retrieval boundary, a way to know whether the system works, and a person able to resolve what it finds.
The temptation at this point is a data readiness programme: eighteen months, a catalogue, a governance board, and an AI project at the end of it. We would generally advise against that as a precondition, because the programme tends to outlive the enthusiasm that funded it, and because the fastest way to find out which data problems actually matter is to try to use the data for something specific.
The alternative is a fortnight of measurement against one narrow question. Take the real corpus, not a curated sample. Write forty to a hundred questions a person actually asks, with their correct answers agreed by someone with the standing to agree them. Run retrieval against the corpus and score how often the right passage is found at all. That number — retrieval hit rate, before any generation — tells you more about feasibility than any survey of your data estate, and it takes days rather than quarters.
What comes back is usually threefold: a realistic accuracy ceiling for this corpus, a list of contradictions worth resolving regardless of whether anything is ever built, and a clear view of whether the permissions model can be expressed. Any of those three justifies the fortnight on its own. And if the answer is that the corpus cannot support the system you wanted, that is a good outcome purchased cheaply — it is far better to learn it from an evaluation set than from a pilot that has already been announced internally.
Two further notes. First, this measurement is also the baseline you will need later: a claim about improvement requires a dated before, and the ROI page sets out why an unmeasured baseline makes every later return unfalsifiable. Second, the fortnight is scoped work with a written output, and we run it as a paid engagement rather than as a free consultation, precisely so it is done properly and belongs to you at the end. The readiness assessment page describes the shape.
For a retrieval system, far less than people expect — retrieval works over documents you already have, and a well-maintained thousand pages is a better starting corpus than a neglected forty thousand. For fine-tuning, the question is different and the honest answer is that you need a labelled set with a defensible ground truth, which is usually the constraint rather than the volume. In both cases, the binding number is not the size of the corpus but the size of the evaluation set, and forty to a hundred well-chosen questions is enough to start.
Only the part you are about to use, and only once you know which mess matters. A general cleaning programme ahead of a specific use is expensive and tends to tidy the wrong things. Running a narrow evaluation against the real corpus tells you which problems block the system and which are merely untidy, and that list is a much better cleaning brief than a data quality audit.
Then a person has to decide which is authoritative, because no model can. Expect this: it is the most common blocker we meet, and it is the reason a project can be technically sound and still produce answers the business rejects. The useful reframe is that the system found a contradiction that was already costing you, silently, in the form of staff giving customers different answers.
Yes. That constraint changes the deployment model rather than the diagnostic: open-weight models on your own infrastructure, and in some cases an air-gapped environment. It also changes the capability ceiling and the cost, and we would rather state that trade-off during scoping than let it be discovered after a pilot on a hosted model that was never going to be allowed in production.
Yes, and it happens. The most common cases are an absent ground truth, a permissions model that cannot be expressed, and a corpus with no owner — in each of those the engineering is not the constraint and building would produce a system nobody could trust. It is a better conversation to have in the first fortnight than in the second quarter.
What we built, what stage it is at, and a link to the live thing wherever there is one.
yourOS is an operating system with an AI at its heart. You talk to it like a person, out loud or typed. It makes your calls, plans your days and works on your projects with a crew of agents, on your own private server. Our own product, built from the first commit; the waitlist is open.
Website, backend and app assets for Sonora, a free AI sound-therapy app on iOS and Android published by Cognitive AI.
Brand, website, prototype, apps on five platforms, the AI backend and the algorithm-intelligence pipeline for Podswap, a social growth engine that runs without a human in the loop.
In closed beta: Roxy, a private AI assistant with permanent memory running on infrastructure the user owns. Brand, site, product, backend and conversational design.
Live at coherence.com: brand identity generated with AI agents we directed, the motion system and website for the group behind Sonora, AboutAI, WakeUp and MudShadows. Agent work remains in build.
Live at domainbooth.com for DomainBooth, a premium domain brokerage: a new animated mark, every page redesigned and rebuilt with designed mobile layouts, and a custom CMS for its sales record, featured names, press and enquiries.
One narrow question, the real corpus, and a hundred questions with agreed answers. That is enough to know whether there is a system here.
30-minute technical call · no deck