Deployment
Shadow mode: going live without anyone carrying the risk
The system runs but doesn't decide. Why this is the most useful six weeks of a deployment, what to measure during it, and when you can move on.
Go-live is the riskiest moment of a deployment. Not technically — organisationally. It’s the day somebody has to sign that from now on, the system does it.
Shadow mode breaks that day into six weeks.
What shadow mode is
The system runs in full: it receives the same inputs, walks the same steps, makes the same decisions.
It just doesn’t write anything back.
Meanwhile your colleague works exactly as before. At the end you have two results for the same case: theirs and the system’s. That comparison is the product.
Why it’s the most useful six weeks
Because it measures on real data, not a sample. The golden dataset holds historical cases we selected. Shadow mode measures what actually arrives — including the things nobody thought of.
Because there’s no risk. If it’s wrong, nothing happens. It’s the only phase where mistakes are free, and that should be exploited.
Because the team gets used to it. Over six weeks the system becomes visible: it’s on screen, proposing, and the colleague can see whether the proposals are any good. By go-live it isn’t a stranger.
Because the sponsor gets a number. Not a promise — a measurement on their own data. That’s what they can take into a leadership meeting.
What to measure
Agreement, per category. Not aggregate. The lesson is almost always that one segment is excellent and another is weak.
The direction of disagreements. When they differ, who was right? This matters more than you’d expect: in a share of cases it turns out the system was right and the manual processing was wrong. At one customer, the first month of shadow mode surfaced eight previously undetected misclassifications.
Coverage. How many cases would it carry through, and how many would go to the exception queue? This, not accuracy, gives you the real savings number.
Latency and cost. How long a case takes, what it costs. That produces the monthly cost estimate.
What we fix during it
Shadow mode isn’t passive observation. There’s a weekly loop:
- Review disagreements by category.
- Pick the largest cluster.
- Find its cause — typically missing context, not “a bad model”.
- Fix it, and check next week whether it moved.
Step four is why the eval set exists: without it you don’t know whether the fix broke something else.
When to move on
Not when a number crosses a line. When three conditions hold per category:
- agreement is stably above the threshold, not just in one good week,
- the remaining disagreements are known and explainable,
- and the error in that category is reversible.
Where all three hold, that category can go live. Where they don’t, it stays in shadow or on human approval — possibly permanently.
That staged go-live is the whole point. You’re not switching the system on; you’re switching on the first category.
The most common mistake
Impatience. After two good weeks the pressure arrives: “this is working, turn it on.” In week three it turns out the first two were good because it was mid-month and the closing wave hadn’t arrived.
Which is why it’s worth watching at least two full business cycles. For a monthly process that’s two months. It sounds long — right up until you compare it with how long it takes to rebuild trust after a botched go-live.
Related: evals and human in the loop.
Questions on this topic
How long should shadow mode run?
Until you have enough runs across every important case type — typically 4-8 weeks. For monthly-cycle processes, watch at least two full cycles, because start-of-month and end-of-month behave differently.
Isn't it wasteful to run the model for weeks like that?
Inference in this phase typically costs about a day or two of salary. Against that, finding a systematic error before go-live rather than after is worth orders of magnitude more.
What does this look like at your company?
If this problem sounds familiar, let's start with one process. Tell us which department burns the most manual hours — we'll come back with a concrete proposal.