Why Most AI Pilots Never Reach Production

Every organisation that has tried AI at any scale has a pilot in its history that worked and went nowhere. The demo was convincing. The accuracy number was good. Someone presented it to a steering committee, everyone agreed it was promising, and eighteen months later nothing is in production and nobody can quite explain why.

This is worth being precise about, because the usual explanations, "the technology wasn't ready", "the business wasn't ready", are not diagnoses. They are ways of not having one. In our experience the same six failures account for most of it, and five of them are decided before a line of model code is written.

1. The pilot was scored on the wrong number

Accuracy on a held-out test set is the easiest thing to measure and often the least useful. It tells you how the model performs on data that resembles the data you already had, at a threshold nobody has argued about, with a cost matrix that treats every error as equivalent.

Production does none of those things. A fraud model at 99% accuracy on a dataset where 0.4% of transactions are fraudulent may be worse than useless. An inspection model that is 96% accurate but concentrates its errors on your most expensive defect class will not survive its first quarter.

If you cannot state what a false positive costs and what a false negative costs, in the units your business already uses, the accuracy figure is decoration.

The fix is unglamorous: agree the operating metric before the pilot starts. Not "accuracy" but "alerts per analyst per day at a recall of 0.9", or "escapes per million units at a false-reject rate the line will tolerate". Those numbers force the conversation about thresholds and costs into week one, where it is cheap, rather than into the deployment review, where it is fatal.

2. The pilot data was not the production data

Pilot datasets are almost always cleaner than reality. Someone exported them deliberately, from one system, over a period they chose, and often removed the rows that looked broken. Production data arrives continuously, from several systems that disagree, with missing fields, duplicate identifiers, timezone confusion and a schema that changed last March.

Three specific gaps recur:

• Leakage. A feature that is available in the training export but not at decision time, a status field populated later in the process, an outcome flag, a timestamp that encodes the answer. Leakage is the single most common reason a pilot number cannot be reproduced.
• Distribution shift between sites. The model learned one plant's lighting, one region's customer mix, one lab's scanner. It is not a general model; it is a model of that site.
• Label quality. Historical labels are often operational artefacts rather than ground truth, what an operator recorded when they were busy, not what was actually true.

A data readiness audit before modelling costs a fraction of what discovering these things after a build costs. It also occasionally produces the most valuable possible outcome: the recommendation not to proceed.

3. Nobody owned the integration

This is the one that kills the most pilots, and it is almost never on the pilot's scope. A model that produces a prediction is not a system. It becomes a system when its output reaches the place where a decision is made, the PLC, the case management queue, the ERP, the dispatcher's screen, inside the time window in which that decision is still useful.

That path is usually owned by a different team, running software with a release cadence measured in quarters, and it was not consulted during the pilot. The integration is then discovered to be six months of work against a system that nobody wants to touch, and the business case that justified the model does not survive the revised estimate.

Bring the owner of the destination system into the discovery phase. Not for approval, for a realistic estimate of what writing into their system actually involves.

4. There was no plan for being wrong

Every model is wrong some of the time, and a production system needs a defined behaviour for that case. Who reviews low-confidence outputs? What happens when the model is unavailable? Can a human override it, and is the override captured as training signal or lost?

Pilots skip this because in a pilot a human is always in the loop by default, the data scientist. In production that person is gone, and the absence of a fallback path is discovered by an operator at 2am.

5. The pilot was never going to be maintainable

Pilot code is exploratory by nature: notebooks, hardcoded paths, a preprocessing step that exists only in someone's shell history. That is fine. That is what exploration looks like. The failure is treating the pilot artefact as the starting point for the production system rather than as the specification for it.

What has to exist before deployment, and rarely does after a pilot:

• A reproducible training pipeline that runs end to end from raw data
• Versioned data, features and model artefacts, so a result can be traced
• An evaluation suite that can be re-run against any candidate model
• Monitoring on input distributions, not just on outputs
• A documented retraining trigger and procedure

None of this is exotic. All of it takes longer than people expect, and none of it is visible in a demo, which is precisely why it gets deferred and then never funded.

6. Success was never defined as a decision

The most avoidable failure of all. A pilot should have been commissioned with an explicit statement of what result would cause the organisation to proceed, and what result would cause it to stop. Without that, a pilot cannot succeed. It can only produce numbers that everyone interprets according to what they already wanted.

Write the go/no-go criteria down before you start, get the person who controls the build budget to agree to them, and the pilot becomes a decision-making instrument rather than a science project.

What this looks like done properly

A discovery phase that produces a data readiness audit, a feasibility result measured against an agreed operating metric, a target architecture that names the systems it has to integrate with, and a cost model, takes a few weeks and answers the question the pilot was actually asked.

It is also, frequently, the cheapest way to find out that the answer is no. That is not a failed engagement. A defensible no in week three is worth considerably more than an ambiguous maybe in month nine.