How to Verify a Delivered AI System in One Day: Repo, Tools, Cases, Run History

Why does a successful demo prove nothing about a delivery?

Because a demo runs on the vendor's environment, with the vendor's data, with the person who built the system watching. Acceptance is a different question: what can your team do with this on an ordinary Monday, six months from now, when the vendor's engineers are busy and the person who championed the purchase has changed role?

I write acceptance tests for the systems I hand over, and I have also been the person called in to run one against somebody else's delivery. The test that finds problems is not a longer demo. It is four artifacts, exercised in a fixed order, by someone who did not build the system.

The delivery is not the software. It is the repository, the tool definitions, the evaluation set and the run history — and if any of the four is missing, what was demoed was a capability, not an asset.

Everything below runs in a day. Budget four hours for a system of moderate size, plus an hour to write up the verdict. Do it before the final payment, and do it with a laptop that has never had the vendor's VPN on it.

What are the four artifacts, and what does each one decide?

Ask for all four in writing before the acceptance day, as deliverables rather than as a demonstration. A vendor that has shipped this kind of system has them; a vendor that has only demoed will negotiate, and that negotiation is the finding. The buying-side version of this argument — which artefacts decide whether you own the system or rent it — is in the four boundaries of a build-versus-buy decision for agents.

Step 1 — Can it run from nothing, without help?

Start on a clean machine with no vendor access. The point is not that it works on yours; it is that the instructions are sufficient.

git clone <vendor-repo> agent && cd agent

git log --oneline | head -20        # is this the history you were shown, or a squashed export?
cat .env.example                    # every secret the system needs, named, none of them real values
grep -rn "localhost\|TODO\|FIXME" --include='*.ts' --include='*.py' . | head -30

pnpm install --frozen-lockfile
docker compose up -d                # database, queue, object store: whatever the system needs
pnpm test                           # unit tests, if any
pnpm dev                            # does it start without a call to the vendor's cloud?

Three failures matter here, and each means something specific. A squashed repository with one commit means you cannot see what changed or why, which makes the code read-only in practice. A .env.example that omits a required secret means the documented setup is incomplete and you will spend the day guessing — the same class of problem as a runbook that only works for its author. A start-up path that requires the vendor's hosted control plane means the orchestration is rented even if the code is on your disk, and the acceptance test has already found the answer to the question the contract was supposed to settle.

Write down every step you had to guess. That list is the handover document you are buying, and the omissions are the cheapest thing to have fixed while the invoice is still open.

Step 2 — Read the tool boundary instead of trusting the diagram

The tool definitions are where your business semantics live, so read them from the code rather than from a slide. For an MCP-based system, list the tools the server actually exposes and inspect the schemas:

curl -sS "$AGENT/mcp" -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | jq '.result.tools[] | {name, description}'

{
  "name": "create_draft_quote",
  "description": "Creates a draft quote for an existing customer. Never sends it. Fails if the customer is on credit hold.",
  "inputSchema": {
    "type": "object",
    "required": ["customer_id", "line_items", "idempotency_key"],
    "properties": {
      "customer_id": { "type": "string", "description": "Internal customer id, not the email" },
      "line_items": { "type": "array", "items": { "type": "object",
        "required": ["sku", "quantity", "unit_price_cents"],
        "properties": { "sku": { "type": "string" }, "quantity": { "type": "integer", "minimum": 1 },
                        "unit_price_cents": { "type": "integer", "minimum": 0 } } } },
      "idempotency_key": { "type": "string", "description": "Caller-supplied key; a repeat with the same key must not create a second quote" }
    }
  }
}

Check four things on every write-capable tool, because each one is a class of production incident:

1. A written contract. The description states what the tool does and what it refuses. "Creates a draft quote" with no mention of credit hold is a description that will be discovered by an operator at the worst moment.
2. Types that are narrow enough to reject nonsense. If quantity is a string and unit_price is a float, nobody is validating anything and the model is the only thing standing between a wrong number and your ERP.
3. An explicit failure mode. The tool must be able to say "I cannot do this", rather than returning an empty success. Idempotency, timeouts and partial writes belong in the contract, not in a comment.
4. An idempotency key on every write. An agent that retries after a timeout will duplicate a write unless the tool refuses to. Idempotency for agentic writes is the full treatment; the acceptance question is only whether the key exists and whether a duplicate call is actually refused.

Then run the test that no diagram survives: ask the vendor to add a trivial read-only tool in front of you, or ask who can. A system whose tool boundary takes a day and a code review to change is yours; a system where the answer is "we will submit it to the platform team" is a feature request channel.

Step 3 — Replay the recorded cases and compare the numbers

Pull your own cases from the run history rather than accepting the set you were given, then score both. Forty to sixty cases is enough to see the shape, and the query is unremarkable:

-- Cases that a human overrode: the ground truth is the correction, not the model output.
select r.id,
       r.created_at,
       r.model_version,
       r.decision,
       r.input_json,
       r.output_json,
       o.corrected_output,
       o.reviewer_role
from runs r
join human_overrides o on o.run_id = r.id
where r.created_at >= now() - interval '90 days'
order by r.duration_ms desc
limit 60;

If that query cannot run — no overrides table, no model_version column, no accessible run history — the delivery has failed the acceptance test at step three, and the rest of the day is spent documenting it rather than testing.

Score in three buckets, not one. Handled without review is the number that pays for the system. Routed to a human is the cost line that finance will ask about. Confidently wrong is the acceptance criterion, because it is the failure that reaches a customer without anyone noticing. A system at ninety-four per cent accuracy whose six per cent of errors are all confident is worse than one at eighty-eight per cent that abstains on the cases it does not understand, and only this third bucket tells the two apart.

Then re-run the same set against the current model version and against the previous one if the vendor still has it configured. The difference is what a model upgrade costs you, measured rather than feared, and it is the single most valuable artifact you can walk away with.

Step 4 — Reconstruct one run from its identifier

Pick a run at random from three months ago and rebuild what happened: the input as the system received it, the prompt version, the model version, which tools were called with which arguments, what the system decided, which gate applied, and what the human did next.

If that reconstruction requires the vendor's console, the audit trail belongs to them. If it takes more than a few minutes, investigations will not happen at all — and an incident you cannot investigate gets resolved by switching the feature off, which is how automation quietly dies. What a usable run record contains, and how to store it without a platform team, is covered in AI workflow observability; acceptance only requires that you can do the reconstruction unaided.

Step 5 — Break it on purpose

Five deliberate failures, all of them things that happen in production within the first quarter. Run them on your infrastructure, not on the vendor's.

# 1. The write that must not happen twice: same idempotency key, twice.
curl -sS -X POST "$AGENT/runs" -H 'content-type: application/json' \
  -d @fixtures/draft-quote.json | jq '.id'   # run it a second time; one record must exist, not two

# 2. The upstream that fails slowly: a 429 and a read timeout, injected at the model call.
#    Does the run fail safe, or leave a half-written record behind?

# 3. Malformed input the schema should reject: a negative quantity, a null customer id.
#    Which layer rejects it — the tool schema, or the accounting team next week?

# 4. A revoked credential mid-run. Does the operator see a clear error, or a silent no-op?

# 5. The case the model is confidently wrong about, from step three. Does the gate catch it,
#    or does it reach the customer?

Record the behaviour of each. A system that fails safe is worth accepting even with rough edges; a system that fails open is not worth accepting with any number of features, because the failure arrives as a wrong quote or a duplicated invoice rather than as an error message somebody can fix.

Step 6 — Export everything, then do it again without them

Finally, run the documented export: run history, the evaluation set, the tool and schema definitions, and the configuration. Then load them somewhere of your own — a laptop is sufficient — and check that the exports are usable rather than merely present. A JSON dump with no schema and no join keys is a copy of your data, not a transfer of your system.

Two conditions finish the test. The first is that no step above required the vendor's console. The second is that the person who ran it was not the person who built it. Both conditions exist to answer one question, and it is worth asking out loud at the end of the day: if their team disappeared tomorrow, what would we be unable to do? The honest answer is the acceptance result.

What does an acceptance failure actually mean?

Three different verdicts, and mixing them up is how buyers end up accepting a rental and calling it a delivery.

Missing artifacts are scope, not bugs. A missing evaluation set, no run history, a repository without history: these are things to be handed over, and the fix is a dated commitment before final payment rather than a defect report. If the vendor cannot produce them at all, the system was never architected to be handed over, and that is a different conversation about price.

Failing safe is negotiable; failing open is not. A missing retry, an ugly error message, a slow queue: these are remediable, and I would accept with a written date. A write that can duplicate, a gate that lets confident errors through, credentials with broader scope than the workflow needs: these go back.

A number that cannot be reproduced is a claim, not a result. If your replay of their own cases does not match the claimed figure, the difference is the finding — and the useful next question is not "why were you wrong" but "which of our cases does your set omit", because that answers whether the claim was optimistic or the process changed underneath it.

The acceptance sheet

Copy this into the handover document and fill it in during the day. It is deliberately short enough to complete.

Ten lines, four hours, and it decides whether what you bought is an asset with a maintenance plan or a rental with an invoice. The refusal to run it — or the fiddling with environments to make it possible — is the most informative answer you will get in the whole engagement, and it arrives a great deal cheaper than the alternative, which is finding out in year two when the invoice is still arriving and the person who understood it has left.