Every vendor in your shortlist can show you something impressive in twenty minutes. These are the questions that tell you which of them has run a system in production for a year afterwards — including several we would find uncomfortable.
All Resources →
Any accuracy figure quoted before a vendor has seen your data is about someone else’s data. The question that separates the field is: how will we know this works, on our documents, before we depend on it?
A good answer describes an evaluation set built from your own cases, who agrees the correct answers, what is measured, and what the pass threshold is. A weak answer is a benchmark score, a demo, or a promise to iterate.
Not whether it will be. Ask what the system does when it is: does the error reach a customer, sit in a queue, or get caught by a validation rule? Who finds out, how quickly, and from what?
A vendor who has run something in production answers this immediately and with examples. A vendor who has not will reach for the reliability of the underlying model, which is an answer to a different question.
All three, separately, in writing. Code ownership is usually addressed and the other two often are not, which is how a client ends up owning a repository they cannot operate because the prompts sit in a vendor platform and the test set was never handed over.
The evaluation set is the most valuable artefact the project produces. It outlives the model, the vendor and usually the system.
Which providers, in which regions, under which agreement, with what retention and what position on training on inputs. Ask what happens on deprecation, because hosted models are retired and each migration costs an evaluation run and a period of reduced confidence.
Third-party terms are the provider’s to set and they change, so ask for them as of a stated date and confirm against the provider’s current terms yourself.
Inference is the visible line and rarely the largest. Ask for retries, evaluation traffic, reindexing, logging and storage, and the engineering time to maintain prompts, watch quality and handle provider changes.
A running cost with no human line in it has not been run. The omitted lines are set out on our cost model page if you want a checklist to score answers against.
Not a logo, not an anonymised case study, not a testimonial in the deck. A named person at a named organisation who ran the system after the vendor left, and who you may ask what broke.
Where a vendor genuinely cannot name a client — some sectors forbid it — ask instead for a technical walkthrough of a real system with the client details removed. What you are testing is whether the operational detail exists.
Suppose the engagement ends in twelve months. What exists, where does it run, who can maintain it, and what would your own team need to learn? Ask for the handover artefacts by name: architecture document, runbooks, evaluation sets, deployment scripts.
A vendor whose commercial model depends on being irreplaceable will answer this vaguely, and the vagueness is the signal.
The most informative question in the list, and the one most likely to produce silence. Every experienced team has a list: fully automated decisions on regulated outcomes, systems with no ground truth to evaluate against, agents with unsupervised write access to production, projects where the data does not support the promise.
A vendor who would build anything you can pay for has told you how the difficult conversations will go.
The distinction that matters is rarely between a good answer and a bad one. It is between an answer with operational detail in it and an answer that is fluent about outcomes. This table is what we listen for.
| Criterion | A shipped answer sounds like | A demo answer sounds like |
|---|---|---|
| On accuracy | We will build an evaluation set from your cases and report per-field results before scoping | We typically achieve high accuracy on documents like these |
| On failure | It lands in a typed queue, this person sees it, here is the alert | The model is very reliable and we monitor closely |
| On cost | Here are seven lines including retries, evaluation and maintenance days | Here is the price per thousand tokens |
| On ownership | Code, prompts and evaluation sets are yours; here is where each lives | You own your data |
| On providers | These providers, these regions, these terms as of this date | We are model-agnostic |
| On timelines | Integration is the long pole and here is why | Six weeks to production |
| On handover | These artefacts, this training, your team can run it without us | We offer ongoing support packages |
| On refusal | We would not build this part, and here is what we would do instead | We can do all of it |
A fixed price against a vague scope is a warning sign for both sides, and it is worth being explicit about why, because procurement processes are often designed to produce exactly that.
For the buyer, the risk is that the price was set by guessing, and a supplier who has guessed low will protect the margin somewhere: fewer evaluation cycles, less integration testing, a narrower definition of done, or a change-request conversation in month three about work you assumed was included. For the supplier, the risk is the mirror image — a scope that grows through a hundred reasonable clarifications until the engagement is unprofitable and the relationship is adversarial. Neither party gets a good system out of that.
The specific difficulty in AI work is that the integration surface and the data quality are the two largest cost drivers, and neither is knowable from the outside. Nobody can price extraction over your document set without seeing a representative sample of it, and any supplier who says otherwise has priced the demo.
The shape that works is a small, genuinely fixed-price first phase — the evaluation, the sample, the architecture, the honest cost model — with the build priced afterwards against what that phase found. It gives you a real number and a real supplier assessment for a bounded outlay, and it gives you the option to stop. Our own version of that is on the pricing page, and the sequence is described on how it works. Where a framework obliges you to take a fixed price for the whole thing, ask each supplier to state the assumptions the price rests on, and score the assumptions rather than the number.
It would be dishonest to publish this without saying that we benefit from you running a process like it. A buyer who asks for an evaluation plan, a running cost with the people in it and a handover definition is a buyer whose process rewards the way we already work, and it disadvantages suppliers who compete on the polish of a demonstration.
So here are the questions on this page that are awkward for us. The reference question is the hardest: our published client work is on the work page and it is a short list, because we do not invent case studies and we do not publish clients who have not agreed to it. If your process weights reference count heavily, we will score badly against larger firms, and that is a fair reflection of size rather than of the work. The certification question is the second: we hold none, and if your scoring matrix requires a certified supplier we would rather tell you in the first conversation. The fixed-price question is the third: we will resist quoting a full build price before an evaluation phase, and where a process demands one, we will either state the assumptions loudly or decline to bid.
The reason to publish it anyway is that the alternative is worse for us. The projects that go wrong are the ones scoped on optimism, and a buyer who has asked these questions arrives with a realistic timeline, a defined success measure and an accurate expectation of what maintenance costs. That is a better project to work on, and a much better one to still be running in two years.
For a first AI project, often not. A formal process written before anyone understands the data tends to specify a solution rather than a problem, and it selects for the supplier best at responding to documents. A short paid evaluation phase with two or three suppliers, judged on what they find and how they report it, gives you far better information for a comparable outlay. Where procurement rules require a formal process, write the requirement as an outcome with a measurement attached rather than as a technical specification.
What would you refuse to build? It cannot be answered from a template, it requires the vendor to have met a project they should have declined, and the specificity of the answer tells you how the difficult conversations in your own engagement will go.
By normalising what is included rather than by comparing totals. Ask every bidder for the same seven cost lines — inference, retries, evaluation traffic, storage and logging, reindexing, integration, and the maintenance days — and for their assumptions about volume. Most large gaps between bids turn out to be scope differences hiding in the integration line, and the cheapest bid is frequently the one that has not yet discovered the integration work.
Only with a pass threshold agreed before it starts and real data inside it. A proof of concept on curated examples with no defined success criterion proves that the technology exists, which was not in doubt. Define what result would cause you to proceed, what result would cause you to stop, and who decides, then run it against traffic that looks like production.
Who specifically will do the work, how much of their time, and what have they personally shipped that is still running. Agencies of any size can present senior people at the pitch and staff the delivery differently, so it is a fair question to ask for named individuals in the contract, and a fair one to ask of us.
Bring the list. We would rather answer the reference and certification questions in the first thirty minutes than in the third meeting.
30-minute technical call · no deck
What we built, what stage it is at, and a link to the live thing wherever there is one.
yourOS is an operating system with an AI at its heart. You talk to it like a person, out loud or typed. It makes your calls, plans your days and works on your projects with a crew of agents, on your own private server. Our own product, built from the first commit; the waitlist is open.
Website, backend and app assets for Sonora, a free AI sound-therapy app on iOS and Android published by Cognitive AI.
Brand, website, prototype, apps on five platforms, the AI backend and the algorithm-intelligence pipeline for Podswap, a social growth engine that runs without a human in the loop.
In closed beta: Roxy, a private AI assistant with permanent memory running on infrastructure the user owns. Brand, site, product, backend and conversational design.
Live at coherence.com: brand identity generated with AI agents we directed, the motion system and website for the group behind Sonora, AboutAI, WakeUp and MudShadows. Agent work remains in build.
Live at domainbooth.com for DomainBooth, a premium domain brokerage: a new animated mark, every page redesigned and rebuilt with designed mobile layouts, and a custom CMS for its sales record, featured names, press and enquiries.
Apps, platforms and campaigns for these teams, long before the AI work. Every one is written up on the work page — what we did, and where our part ended.
