Proving AI Output Quality Without a Lab: Forty Archived Cases and a Replay Gate

What does "the system works" mean operationally?

When someone asks you to prove the AI system works, they are not asking for a model score. They are asking whether the system still produces the outcomes you already accepted, and how you would know if it stopped.

Those are different questions, and only the second has an answer that fits on a slide. Model quality is a property of a model; output quality is a property of your workflow against your cases. A vendor benchmark cannot tell you whether a margin exception on a renewal quote is still being flagged, because the vendor has never seen your rate card. What can tell you is a fixed set of cases from your own history whose correct outcomes are already known, replayed after every change.

You do not need a research lab to measure output quality; you need forty archived cases with known-good outcomes, replayed on every change, which is a regression suite rather than an eval framework.

The word "framework" is where budgets die. A framework implies graders, rubrics, judge models and a labelled dataset of ten thousand rows, and it produces a number that nobody can act on. A suite implies forty files in a repository and a command that either exits zero or fails the build.

Why can the pilot not answer the question it was funded to answer?

Most pilot reports contain two numbers: a time saving per item, from a stopwatch or a survey, and an accuracy figure with no denominator. Neither survives contact with an operations director, because neither is falsifiable. Ask what the accuracy figure was measured against and the honest answer is usually "the reviewer's judgement on about thirty items, in week two". The reviewer has since moved to another project, and the thirty items were never written down.

The things that move under a working pilot are also the things the pilot never pins. A provider can update a model behind an unchanged alias. A supplier changes payment terms and the retrieved rate card rotates the same day. Someone edits the prompt to fix one complaint and no version number changes. The workflow behaves differently on Tuesday than it did on Monday, and no artefact in the company records what Monday looked like. The pattern I see is consistent: the team can describe the change and cannot demonstrate it.

How do forty archived cases cover a workflow that handles thousands?

They cover it the way a tax audit covers a company: by sampling the population that carries the money and the ambiguity, not by volume. Forty is enough when each case is selected for a reason and the reasons are written down.

The selection rule I use is four buckets. Ten cases are the highest-value transactions in the last ninety days, because that is where a wrong output is measured in money rather than tokens. Ten are known failures that were already corrected by a human, because those are the only cases where you have a documented correct answer. Ten are routine items from the middle of the distribution, which is what proves you did not break the happy path while fixing the edge. Ten are adversarial: a missing reference number, a supplier name that exists twice, a currency with no rate on file, an attachment that says one thing and a line item that says another. The third and fourth buckets matter more than teams expect. Almost every regression I have seen introduced by a "quality improvement" shows up in the routine bucket, not in the cases the change was designed for.

Forty also fits a working week: two people, roughly four hours each, a hundred and sixty cases reviewed and thrown away, forty kept. That is the whole data-labelling cost, paid once.

How much of the grading can machines do, and how much needs a person?

Most of it, but only if you refuse to grade the prose. Grade the decision the workflow produced, in system language, against a rule.

Roughly thirty of the forty cases can be graded by code with no judgement at all: a required field is present, the total matches the line items, the approval level corresponds to the threshold, the cited clause exists in the current policy and is the one that applies. Six need a rule with a tolerance band — a date within two working days, a quantity within a stated percentage — and those tolerances belong in the suite, not in a reviewer's head. About six genuinely require a human, because the correct answer is contested: two people at the client would disagree about what should have happened. Those six are the most valuable cases in the set, because they mark the boundary where you should route to a person instead.

The failure mode to design against is the rubric. A rubric with five dimensions and a one-to-five scale feels rigorous and produces scores that cannot fail a build, because every dimension regresses a little and the average holds. A pass or fail against a written expectation, with the expectation stored next to the case as one sentence, can fail a build.

What has to be recorded for a replayed case to mean anything?

A case is an input plus an expectation plus an environment. Miss the third and you are comparing two different experiments.

Score a case against the live model alone and the result tells you the model moved, not whether your code broke. That is why the suite runs in two passes: pass one replays the stored model responses and tests your logic, prompts, tool schemas and routing, which is what actually changes when engineers ship; pass two re-issues live model calls and tests the model itself. Pass one should be deterministic and fast. Pass two costs money and is the one that catches a provider-side change.

That split only works if every run is recorded at the level of detail replay requires — the rendered prompt and its content hash, the model version string returned by the provider, each tool response or its hash, and the terminal state — which is what a run record built for deterministic replay and per-case cost measurement has to contain. Without it, the suite cannot separate "our change broke it" from "the model changed under us", and you are back to an opinion.

How do you keep the suite from turning into a report nobody reads?

Put it in front of the change, and make it print the same three numbers every time.

The gate is a command in the pipeline: if any case that passed at the previous release now fails, the release does not ship, and the failing case is named in one line with its expectation. Twelve minutes for the forty-case suite is realistic for a workflow of twelve steps; if yours takes an hour, the cases are too broad or the environment is not frozen. The second number is cost per case, so a prompt edit that triples input tokens is visible as money rather than as a diff. The third is the count of cases whose expectation has been revised, and that count is the one to watch, because revising an expectation is also how a team quietly accepts a regression.

Those numbers are my own estimates from the workflows I have shipped, not a benchmark. The harness is the only line that is real engineering, and it is smaller than the reporting someone will otherwise build by hand in a spreadsheet.

Where does this approach break, and what do you do about it?

Three places, and all three are answerable.

Forty cases is not a statistical sample. It cannot tell you your true error rate, and any percentage computed from it carries a confidence interval so wide that quoting it would be dishonest. Do not quote it. Use the suite to detect regressions and demonstrate coverage, and take the rate from production: total items, human overrides, escalations and complaints, counted weekly. The suite decides whether a change ships; production data tells you how the workflow behaves.

The suite ages. A case selected in February encodes February's rate card, product mix and policy wording. Replaying it in August proves consistency, not correctness, so the selection has to be refreshed each quarter and retired cases archived with a note explaining why. I have watched a suite become a comfort blanket: everything green, override rate climbing, because the cases had frozen while the business moved.

And the suite is not a substitute for the one case you care about most. Keep a copy of the worst output the workflow has ever produced, the one that reached a customer, and replay it before every release. That single case has ended more arguments than any dashboard, because the person asking you to prove the system works recognises the incident, not the metric.

Where to put the first forty cases

Start with the workflow that has the largest financial exposure and the smallest number of steps — usually the quote, pricing or margin exception path, where a wrong output is measured in contracts rather than tokens. Ask the person who reviews those outputs today for the ten that worried them most in the last quarter, add the ten routine ones and the ten that already failed, and write one sentence of expectation next to each. Then wire the replay into the pipeline so a regression fails the build while the change is still cheap to reverse. If you cannot find forty cases with known-good outcomes, that is itself the finding: your workflow has been operating without a definition of correct, and the first thing to fund is the person who can write those forty sentences, not an evaluation platform.