An AI Vendor Risk Assessment Template You Can Actually Send

Why do vendor risk questionnaires fail?

Because they ask for assurances and receive them. "Do you use our data for training?" returns "No" — a word nobody can check, from a person whose answer is not a contract. The questionnaire did its administrative job and produced no evidence, which is why the same questions get asked again a year later with the same result.

The fix is mechanical rather than clever: every row names the artifact that answers it. Not the topic — the thing. The clause, the subprocessor appendix, the exported run history, the sample export file, the written per-case result. A row that cannot name an artifact is a row that will be answered with an adjective.

Ask for the artifact in the same sentence as the question, and the questionnaire stops being a ritual. Vendors who have shipped produce it in an afternoon; vendors who have demoed ask whether a summary would do.

I write these for procurement teams, and the version below is the one I would send. It is deliberately shorter than the templates in circulation, because thirty-two rows that get answered with documents beat two hundred that get answered with prose. If you want the prompt mechanics for auditing what comes back — claim-by-claim, with the missing denominators flagged — that is the prompt pack that does the clerical work; this page is the finished artifact itself.

How should the template be used?

Five rules, and they matter more than the row wording.

1. Send it as a spreadsheet, not a PDF. The response column has to hold links to artifacts, and a PDF turns those into attachments in an email thread.
2. Require one artifact per row. "See attached security page" is not responsive; a link to a page that can change without notice is not either.
3. Score the artifacts, not the answers. A written "no" with a clause number scores better than an enthusiastic "absolutely".
4. Record refusals as findings, not as zeros. A refusal is information: it names a boundary, and the row it refused tells you which boundary matters to your risk.
5. Keep the completed file. The person who inherits the vendor relationship will need the refusals more than the reassurances.

Section A — Accuracy and failure behaviour

Unacceptable answers: a percentage without a denominator, "configurable", "full audit trail" as a noun, and any accuracy figure quoted from a public benchmark. Rows A5 and A7 are the two refusals that should stop a purchase: a vendor who will not show a failure case, or will not run yours, has told you what the first year will be like.

Section B — Cost and metering

Why B3 is the decisive commercial row: the human tail usually exceeds the model cost by an order of magnitude, and it is the line every quote leaves out because it is not the vendor's to price. Asking who carries it forces the answer into the contract, even when the answer is you.

Section C — Data handling and subprocessors

Unacceptable answers: a compliance badge as a response to C1, "industry standard" for C3, and "our logs are available in the console" for C5 — a console is not an export.

Section D — Exit and ownership

Why D2 is usually the finding: orchestration held by the vendor means the decision you made was not a purchase but a transfer of your workflow boundary. Silence on any row resolves in the vendor's favour later, which is the argument behind the lock-in analysis.

Section E — Production evidence and references

E3 is uncomfortable to ask and cheap to answer for anyone who has run a system for a year. The response is more informative than the references, because it is the only row that cannot be prepared in advance by a marketing team.

How do you score the responses?

Four response types, and one score each — not to produce a number, but to make the difference visible to people who will not read thirty-two rows.

Weight the sections by your own exposure rather than uniformly: for a read-only workflow, section C dominates and A matters less; for anything that writes to a system of record, A and D decide the outcome. A uniform total hides exactly the thing procurement is trying to see.

What this template cannot do

It measures answers, not the future. A vendor can score well and still fail in year two, because the questionnaire captures the state of the company that answered it, not the one that will hold the contract.

It does not replace running your own cases. This is the section A rows turned into work: sixty to a hundred recorded cases, sent before commercial terms, with results returned in writing. That single exercise generates more information than the rest of the file, and it is the difference between evaluating and interviewing.

It does not assess model quality or fit. Whether the system is good enough for your process is a question only your evaluation set can answer; the questionnaire asks whether the vendor is capable of proving anything at all, which is a different and prior question.

The row that matters most

If the file has to be reduced to one question, it is A5: show me a case you got confidently wrong — what did the operator see, and what happened next?

That single answer reveals whether the system can tell that it does not know, whether the review interface shows evidence or only a verdict, whether a per-case record exists at all, who carries the cost of a failure, and whether the vendor has been close enough to production to have seen one. Six rows of the document are answered by that one artifact, which is why the strongest questionnaire in an evaluation is the one you already know the shape of before it is returned.