Building a golden dataset in practice

A measurement set is worth exactly as much as it reflects reality. That sounds like a platitude until you look at how these sets usually get assembled: someone exports a thousand rows, takes the first hundred, done.

The first hundred rows are usually from the most recent month, from a single clerk, and contain not one exception.

What belongs in it

Every case type, at real proportions

Start by writing down the categories. For invoice processing, say: domestic HUF, domestic FX, EU, non-EU, prepayment, credit note, consolidated invoice, framework drawdown.

Then check what proportion each represents in reality — and make sure all of them appear. The category that isn’t in the set is the one that will fall over in production.

Exceptions over-represented

If credit notes are 3% of cases, a random 100-item sample gets you three. You cannot tell from three cases whether the system handles them.

So we deliberately include more than occur naturally, and weight them back down when scoring. That gives a meaningful read on rare cases and a realistic aggregate at the same time.

From several people

Take one clerk’s cases and you’re measuring their habits, not the process. It’s worth gathering from at least three or four people — and where they differ from each other, that’s a finding in itself: there is no real rule there, only habit.

Spread over time

A sample from one month bakes in that period’s peculiarities: a campaign, a close, a supplier switch. Cover at least 12 months.

What “the correct answer” means

Everything turns on this. We distinguish three levels.

Unambiguous. There is one right value. The invoice total is HUF 1,240,000. Machine comparison works.

An acceptable set. Several answers are fine. The cost centre could be “Marketing” or “Marketing / Events” — both acceptable. Here the expert supplies a list, not a value.

Judgement. There’s no objective right answer, only better and worse. A summary, for instance. Here we write down criteria (“contains the deadline”, “contains no claim absent from the source”) and score against them. More on that: evaluating open-ended tasks.

Labelling is always done by a domain colleague. Not the developer, and not a model. Label with a model and you’ll be measuring your own mistakes.

The most common mistakes

The set changes along with development. If you rewrite the expected answer after every failure, the measurement will always read 100% and mean nothing. Changing the set is a separate decision, with a reason, versioned.

Only the happy path. If it contains no case with missing data, a typo, or two attachments, those are what will catch you in production.

One aggregate number. “87%” on its own says nothing. Break it down by category, or you won’t see that one segment is at 98% and another at 41%.

The set leaks into the prompt. If the cases end up in the system’s context, they stop measuring anything. Keep them strictly separated.

How much is enough

For a narrow, well-defined task, 50–100 cases already give a usable signal. Above 300–500 the per-category numbers stabilise. Past a thousand, it’s more useful to broaden coverage than to add rows.

If you’re starting now: start small, but per category. An 80-item set covering every type beats a 2,000-item one that only knows the typical case.

---

The set stays yours — it’s what makes switching to a model that ships six months from now a measurement rather than an article of faith. Let’s talk.