Evals: how to measure whether it actually works

When conventional software is done you run the tests and get green or red. With a system built on a model that doesn’t work: the same input can produce two different outputs on two runs, and both may be acceptable.

It’s still measurable. Just differently.

What an eval is

An eval is a measurement set: a collection of inputs, the output we consider correct for each, and a rule that decides whether the system’s answer qualifies.

This is not the same as a unit test. A unit test wants character equality. An eval asks: would this answer be acceptable to the person who does this by hand today?

The golden dataset

The set is built from your own historical data. Not synthetic examples, not demo data: real cases somebody already resolved, where we know how.

Three things matter when assembling it.

Coverage, not count. A good set isn’t one with five thousand cases, it’s one with every case type in it. If 12% of real cases come from foreign suppliers with foreign-currency invoices, the set needs that too — otherwise the measurement will lie.

Exceptions over-represented. Rare cases cause the trouble in production. We deliberately include more of them than occur naturally, and weight them back down when scoring.

A domain expert states the correct answer. Not the developer, not the model. The colleague who does this work today. This is the most time-consuming part, and it can’t be skipped.

What we measure

It depends on the task, but typically at three levels:

1. Is all the data there? Did the system extract what’s needed? The easiest layer to measure.
2. Did it perform the required steps? Did it query the vendor master, check the framework agreement? This needs step-level logs.
3. Does it match the expert’s decision? Same classification, same proposed cost centre?

We separately measure whether it’s safe to let run: did it do anything it shouldn’t have, and did it flag when it was uncertain?

The measurement report

The output isn’t a percentage. It looks like this:

Run: 2026-05-14 · 50 cases · model: [X] · version: 0.7.2

Passed:   41 / 50   (82%)
Failed:    9 / 50   (18%)

Failure breakdown:
  missing data in the source        5   → not the system's fault, needs discovery
  pulled the wrong record           3   → vendor-master matching to fix
  correct, but didn't flag doubt    1   → threshold tuning

By category:
  domestic, HUF          28/29  (97%)
  foreign, FX             9/14  (64%)   ← the problem is here
  prepayment invoice      4/7   (57%)   ← and here

The conclusion is rarely “the model is bad”. Much more often it’s that the model is bad in one specific category — and that category needs its own rule or its own context. The 64% on FX cases isn’t a call for a model change; it’s a call to hand the system the exchange-rate table.

The threshold

The measurement produces the most important operational decision: what runs unattended and what goes to a human.

That isn’t one number. It’s set per category. In the example above, domestic HUF cases can run automatically at 97%; FX cases all go to a person for now. As the measurement improves the threshold moves — but on measurement, not impatience.

Why the set stays yours

This set is the most valuable thing left behind by the engagement.

While you have it, every future decision is a measurement. A cheaper model appears? Run it. The vendor updates their system? Run it. Someone rewrites a prompt? Run it, and see what they broke.

Without it, every change is an article of faith.

---

Evals are a precondition for deployment here. Related reading: building a golden dataset.