Enterprise

Why most AI pilots never ship.

The pilot worked. Everyone saw the demo. Eighteen months later it is still a demo. The reasons are consistent, unglamorous, and almost never about the model.

THE PATTERN

A pilot is built to prove that a thing is possible. Production requires proving that it is reliable, supportable, permissioned, monitored, affordable and better than what it replaces. Those are different projects, and the second is larger than the first — which is the opposite of how the budget is usually split.

The failure is structural rather than technical. A pilot is optimised for speed of demonstration: hardcode the credentials, use the export rather than the API, skip the edge cases, run it on the ten documents that work. Every one of those shortcuts is correct for a pilot and fatal as a foundation. When the pilot succeeds, the organisation naturally asks to "just productionise it" — and the honest answer is that most of the work has not started.

Below are the five gaps we see most often, in roughly the order they kill projects — and a sixth that is worth naming.

FIVE GAPS, AND A SIXTH

Nobody agreed what 'good enough' means

Without an evaluation set defined before building, quality is decided by anecdote. One bad output in a steering meeting outweighs a thousand good ones nobody saw. Close it by agreeing a labelled evaluation set and a threshold during discovery — while expectations are still negotiable.

The integration was never scoped

The pilot read a CSV. Production needs authenticated, permissioned, rate-limited access to the system of record — plus whatever the security team requires before granting it. Provisioning that access is commonly the longest single item on the critical path, and it is almost never on the plan.

There is no owner after go-live

An AI system needs someone watching quality drift, refreshing content, tuning thresholds and handling exceptions. If that role is not named and resourced, the system degrades quietly until people stop trusting it. "The vendor will handle it" is not an operating model.

Unit economics were never calculated

Pilot inference costs are trivial at pilot volume. At production volume, with retries, longer contexts and higher traffic, the bill can exceed the cost of the process being replaced. Model this before committing to an architecture, because it changes which architecture you pick.

The workflow around it did not change

An agent that drafts a response still needs someone to have stopped writing that response by hand. Without the process change, you have added a step rather than removed one — and adoption quietly reverts to the old way.

The honest sixth: it should not have been built

Some pilots stall because the underlying problem did not warrant an AI system, and the pilot was the discovery. That is a legitimate and cheap outcome — provided someone is willing to say it before the production budget is committed.

WHAT CHANGES IF YOU PLAN FOR IT

	Typical pilot	Pilot designed to ship
Data access	Export or sample file	Authenticated API against real systems
Quality bar	Decided in review meetings	Labelled eval set agreed up front
Security review	After the build	Constraint on the initial design
Failure handling	Happy path only	Typed exception routes with owners
Cost model	Not calculated	Modelled at production volume
Ownership	Unassigned	Named, with a runbook
Typical outcome	Stalls after the demo	Reaches production

The difference is not effort, it is sequencing. Everything in the right column is cheap to decide early and expensive to retrofit.

QUESTIONS

QUESTIONS — 4

Yes, but scope it to answer a specific question rather than to impress. A good pilot has a written hypothesis (“an agent can resolve at least 30% of tier-1 tickets at human-level accuracy”), a labelled evaluation set, and a pre-agreed decision on what happens at each possible outcome — including stopping. A pilot with no defined failure condition cannot fail, which means it also cannot succeed.

As a rule of thumb, roughly a quarter. If a pilot takes six weeks, budget the production path at eighteen to twenty-four more — integration, evaluation infrastructure, monitoring, exception handling, security sign-off and the process change. Organisations that budget production as a small increment on the pilot are the ones that stall.

Often, and it is a common starting point. The first step is an honest architectural review: some pilots are a solid foundation with gaps, and some are a prototype whose shortcuts are load-bearing. We will tell you which, including when the answer is that rebuilding is cheaper than remediating — that answer costs us revenue in the short term and saves it for you.

Build the evaluation set. Before the architecture, before the model choice. A hundred labelled examples of what a correct output looks like, produced by the people who will judge the system, converts every downstream argument from opinion into measurement — and it frequently reveals that different stakeholders wanted different things, which is much cheaper to discover on day three than in month six.

Keep reading

Technology

Evaluating AI Systems

AI Readiness Assessment

→

30-minute technical call · no deck