For leaders
Why most AI pilots fail
Pilots don't fail on the model. They fail because nobody mapped where it was supposed to go. Five recurring causes, and what to do about each of them.
Over the past two years almost every larger company has started some kind of AI experiment. A chatbot on customer service, document processing in finance, report generation in controlling. The widely cited MIT survey found that the overwhelming majority of these generative AI pilots never reach production — and what we see in the field matches.
The interesting question isn’t how many fail. It’s why they always fail at the same few points.
1. The team built on the documented process
This is the most common one. Someone sits down with a subject-matter expert for an hour, asks how the work happens, and gets a clean six-step answer. They start building against it.
The reality is that the six-step process is thirty steps, and half of them are exceptions. “An email arrives” — from forty senders, no two formatted alike, half the time the number that matters is inside a PDF or a screenshot, and the rule for where each case gets routed lives in one colleague’s head. They never wrote it down, because nobody ever sat next to them for eight hours.
What you build in that situation works perfectly in the test environment and stops in its first live week.
In a one-hour interview you get what your colleague thinks their job is. In a full working day you get what the job is.
2. Everything got handed to the model
There was a period when the strategy was “the models are good enough, give
them everything”. Two things followed: token costs ran away, and the system
hallucinated at steps where a simple if would have been far more reliable.
In a ten-step workflow, typically two or three steps genuinely need judgement. The rest is API calls, data validation and branching. Put those on a model too and you get a system that is more expensive, slower and less reliable than the one you’d have got from deterministic code.
Deciding this — step by step, which is which — is the single most important piece of work a forward deployed engineer does.
3. There was no measurement, only impressions
“Looks good” is not a result. When the project comes up for defence, the CFO will ask what percentage it gets right and what happens to the remainder. If there’s no number for that, the project stops.
The eval set — the golden dataset — isn’t a luxury. It’s a precondition for deployment. Forty-one out of fifty runs passed; of the nine failures, five had missing source data and four pulled the wrong record. That’s a sentence you can work from. “I think it’s good enough” isn’t.
4. Someone proposed a system migration to a company that migrated two years ago
If a business spent two years and a large budget rolling out an ERP, then any AI proposal that starts with “first we’d need to replace that” is dead on arrival.
What works is building on top. On the existing system’s APIs, the existing permission model, the existing process. Not because that’s the more elegant engineering choice, but because it’s the only one the organisation can absorb.
5. Nobody accounted for the fact that you are the risk
This is the one people say out loud least often. Your internal sponsor — the department head who brought you in — is not primarily shopping for innovation. They would like to keep their job, and they would like to be promoted.
If you do nothing, the status quo holds and nothing bad happens to them. If they bring you in and the project fails, that sits next to their name. Which means your mere presence is already a risk to them.
Every practical decision about deployment follows from that:
- Discovery should come first, not the build — because discovery is a valuable deliverable in its own right, even if nothing follows it.
- The system should run in shadow mode alongside the team before it runs instead of them.
- Every step should be logged and reviewable.
- Success should be stated in numbers, so your sponsor can put it in their own performance review.
What to do instead
One process, all the way through. Not five in parallel.
- Discovery. Someone sits with the team and records how the work actually happens — exceptions included.
- Decision. Step by step, you decide what stays code, what becomes a model call, and where a human stays in the loop.
- Measurement. A golden dataset is built from your own historical cases, and you find out where the system stands.
- Shadow mode. It runs, but it doesn’t decide. The team sees what it proposes.
- Staged go-live. High-confidence categories first, the rest after.
This looks slower than building immediately. In practice it’s the faster route, because you don’t have to throw everything away halfway.
If you’re curious where things stall at your company, let’s talk.
Questions on this topic
How long before you know a pilot is a dead end?
With an eval set, two to three weeks. Without one, usually month six — when somebody has to show a result.
Should we run several pilots at once?
No. Take one process all the way through discovery, evals and go-live. The first complete loop teaches the organisation how to run the second one; started in parallel, that learning never happens.
What does this look like at your company?
If this problem sounds familiar, let's start with one process. Tell us which department burns the most manual hours — we'll come back with a concrete proposal.