Resource

Is your data ready for AI?

Most AI projects that stall do not stall on the model. They stall on a document set that two departments maintain differently, a permissions model nobody can express, or the absence of anyone who can say which of two contradictory answers is correct. Here is how to find that out in a fortnight rather than in month six.

THE DIAGNOSTIC

Does it exist, and can you get at it

Two separate questions. Data that exists inside a vendor system with no export, a database only one team may query, or a format that needs a licence you do not hold is data you do not have for this purpose. Answer the access question with an actual extract, not with an assurance from the system owner.

Is it findable

Can you enumerate it? A share drive where the authoritative version of a document is whichever copy the last person edited is not a corpus, it is a search problem wearing a corpus costume. If nobody can produce a list of what is in scope, that list is the first deliverable.

Can you express who may see what

This is where more projects die than any other technical cause. If permissions live in people’s heads, in folder structures that drifted, or in a rule like “the finance team, except contractors, except during audit”, then a retrieval system cannot enforce them — and a system that answers from a document the asker was not entitled to read is a data breach with a chat interface.

Volume versus quality

Teams overvalue volume. A hundred thousand tickets of which most are one-line non-answers is worse than four hundred well-resolved ones, because the first set teaches a system what your organisation does badly. Sample fifty records at random and read them properly before believing any count.

Is there a ground truth

For any task where correctness matters, somebody has to be able to say what the right answer was. Sometimes it exists already — the adjudicated claim, the accepted invoice, the resolved ticket. Sometimes it must be created by an expert labelling a few hundred cases. If neither is possible, you cannot evaluate the system, and you should not build it.

Freshness and the update path

How stale is the data now, and what happens when it changes? A corpus that is refreshed by a person remembering to re-run an export will be wrong within a quarter. The update path is part of the system, and costing it is part of the estimate — including the reindexing, which is a running cost rather than a one-off.

Duplication and contradiction

Two documents that disagree, both current, both authoritative-looking. This is the most common real blocker we meet, and it is treated at length below.

Structured, unstructured, or worse

Structured data is easy to query and rarely holds the answer. Unstructured prose holds it and needs retrieval. The awkward middle — spreadsheets used as databases, PDFs of tables, forms scanned as images — needs specific handling, and it is usually the largest category by volume in an established organisation.

Can a deletion propagate

If a person exercises their right to erasure, or a document is withdrawn, what has to happen? The source record, the extracted copy, the embedding in the vector store, the cache, the logs. If you cannot describe that path today, design it before you build the index rather than after a request arrives with a statutory clock on it.

Is anyone accountable for it

A corpus with no owner degrades. Someone must be able to decide what is authoritative, retire what is not, and be interrupted when the system finds a contradiction. If no name can be put against that role, the readiness problem is organisational, and no amount of engineering will fix it.

THE CONTRADICTION PROBLEM

The failure that surprises people is not the model inventing an answer. It is the model giving a correct answer that the business disagrees with, because two of its own documents say different things and the retrieval step happened to find one of them.

The 2019 policy and the 2024 policy are both in the share drive, both look official, and neither says which supersedes the other. The onboarding guide and the intranet page describe different approval thresholds. The product specification and the support macro give different return windows. Every organisation over a certain age has hundreds of these, and nobody notices because a human reader applies context that is not written down anywhere: they know which team maintains which document, they know the 2019 one is dead, they have heard the corridor conversation.

A retrieval system has none of that context, and it will confidently surface the wrong one for the same reason a new joiner would. Worse, it does it consistently, at volume, and with a citation that makes the answer look verified.

There is a useful reframing here. A system that surfaces contradictions is doing you a service, and the fortnight of measurement below usually produces a list of them as its first real output. That list is an asset — it is the documentation debt made visible for the first time, and it is often worth more to the organisation than the system that found it. What it is not is something the model can resolve. Deciding which document is authoritative is an act of ownership, and it needs a person with the standing to make it.

Practically, this means one thing for scoping: prefer a corpus with a live owner and a maintenance habit over a larger one without. A thousand pages someone maintains beats forty thousand nobody does, and it is a much better first system.

THE SELF-ASSESSMENT

Score each row honestly, and note where the honest answer is “I would have to ask”. A row you cannot answer is not neutral — it is a red row that has not been checked yet.

	Ready	Workable	Stop and fix first
Access	You can produce an extract this week	Access needs a request and a fortnight	It lives in a system with no export
Enumeration	A list of what is in scope exists	A list can be built from a crawl	Nobody can say what is in scope
Permissions	Expressible as rules against identities	Expressible for most, with known exceptions	Lives in folder structure and memory
Ground truth	Adjudicated outcomes already recorded	An expert can label 200 cases	No one can say what the right answer was
Freshness	Automated update path exists	Manual refresh with a named owner	Last updated at some point, unclear when
Contradiction	Superseded versions are retired	Duplicates exist and are identifiable	Multiple current versions, no owner
Format	Text or clean structured records	Mixed, with scanned documents present	Spreadsheets used as databases
Deletion	A propagation path is documented	Deletion is manual but traceable	Copies exist in places nobody tracks
Ownership	A named person decides what is current	Ownership is shared but reachable	No owner, and no candidate

A readiness self-assessment. Qualitative by design — these are questions to answer about your own corpus, not a score with a threshold.

Two or three rows in the right-hand column is normal and not a reason to stop; it is a reason to narrow the scope until those rows fall outside it. A red under permissions, ground truth or ownership is different in kind, because each of those blocks something the engineering cannot supply — a lawful retrieval boundary, a way to know whether the system works, and a person able to resolve what it finds.

WHAT TO DO INSTEAD OF A PROJECTION

The temptation at this point is a data readiness programme: eighteen months, a catalogue, a governance board, and an AI project at the end of it. We would generally advise against that as a precondition, because the programme tends to outlive the enthusiasm that funded it, and because the fastest way to find out which data problems actually matter is to try to use the data for something specific.

The alternative is a fortnight of measurement against one narrow question. Take the real corpus, not a curated sample. Write forty to a hundred questions a person actually asks, with their correct answers agreed by someone with the standing to agree them. Run retrieval against the corpus and score how often the right passage is found at all. That number — retrieval hit rate, before any generation — tells you more about feasibility than any survey of your data estate, and it takes days rather than quarters.

What comes back is usually threefold: a realistic accuracy ceiling for this corpus, a list of contradictions worth resolving regardless of whether anything is ever built, and a clear view of whether the permissions model can be expressed. Any of those three justifies the fortnight on its own. And if the answer is that the corpus cannot support the system you wanted, that is a good outcome purchased cheaply — it is far better to learn it from an evaluation set than from a pilot that has already been announced internally.

Two further notes. First, this measurement is also the baseline you will need later: a claim about improvement requires a dated before, and the ROI page sets out why an unmeasured baseline makes every later return unfalsifiable. Second, the fortnight is scoped work with a written output, and we run it as a paid engagement rather than as a free consultation, precisely so it is done properly and belongs to you at the end. The readiness assessment page describes the shape.

QUESTIONS

QUESTIONS — 5

For a retrieval system, far less than people expect — retrieval works over documents you already have, and a well-maintained thousand pages is a better starting corpus than a neglected forty thousand. For fine-tuning, the question is different and the honest answer is that you need a labelled set with a defensible ground truth, which is usually the constraint rather than the volume. In both cases, the binding number is not the size of the corpus but the size of the evaluation set, and forty to a hundred well-chosen questions is enough to start.

Only the part you are about to use, and only once you know which mess matters. A general cleaning programme ahead of a specific use is expensive and tends to tidy the wrong things. Running a narrow evaluation against the real corpus tells you which problems block the system and which are merely untidy, and that list is a much better cleaning brief than a data quality audit.

Then a person has to decide which is authoritative, because no model can. Expect this: it is the most common blocker we meet, and it is the reason a project can be technically sound and still produce answers the business rejects. The useful reframe is that the system found a contradiction that was already costing you, silently, in the form of staff giving customers different answers.

Yes. That constraint changes the deployment model rather than the diagnostic: open-weight models on your own infrastructure, and in some cases an air-gapped environment. It also changes the capability ceiling and the cost, and we would rather state that trade-off during scoping than let it be discovered after a pilot on a hosted model that was never going to be allowed in production.

Yes, and it happens. The most common cases are an absent ground truth, a permissions model that cannot be expressed, and a corpus with no owner — in each of those the engineering is not the constraint and building would produce a system nobody could trust. It is a better conversation to have in the first fortnight than in the second quarter.

Keep reading

Enterprise

AI Readiness Assessment

Modelling the Cost of an AI System

→

30-minute technical call · no deck