AI Prompts for Vendor Evaluation: Turning a Demo Into Evidence

What is the difference between a question and a prompt?

A question invites an adjective. A prompt specifies an artifact.

That distinction is the whole reason vendor evaluations produce nothing you can act on. "How accurate is it?" is a question, and it will be answered with a word — strong, high, production-grade — which is noise with good manners. "List every claim in this response, and for each one state the artifact I would have to see to check it" is a prompt, and it comes back as a table where half the rows are empty. The empty rows are the finding.

I have been on both sides of these evaluations: building the workflow, and called in as the second opinion on someone else's shortlist. The buyers who get to a defensible decision are not the ones who ask better questions. They are the ones who arrive with a written list of the evidence they will accept, and who use a language model to do the clerical part of enforcing it — extracting claims, normalising quotes, hunting for the missing denominator — instead of asking it for an opinion.

Ask a language model to judge a vendor and you get a confident summary of the vendor's own framing. Ask it what artifact would prove each claim, and you get the shopping list for the rest of the process.

Eight prompts carry most of that work. Each one below is the actual text, followed by what comes back and how to read it. Nothing here needs a special tool: a long-context model and your own documents are enough.

Prompt 1 — Build the process register before you talk to anyone

The first artifact of a vendor evaluation is not a comparison. It is a description of the process you are about to buy for, in units that can be counted. Without it you are comparing vendors to each other, which is how you buy the best-presented system rather than the right one.

You are helping me prepare an AI vendor evaluation. I will paste raw notes about
what my team does by hand. Produce a table, one row per process, with exactly
these columns:
- process name
- the decision rule a person applies, in one sentence, in their words
- who owns it today (role, not name)
- monthly volume, in units
- minutes of human time per unit
- how often the correct answer is "send this to a human", as a share of units
- what breaks downstream when this process is late
- what data it reads, and what it writes back

Do not invent numbers. Where a number is missing, write "unknown" and list the
questions I need to answer to fill it.

Read the "decision rule" column first. If the rule cannot be stated in one sentence, no vendor can automate it, and the work in front of you is documentation rather than procurement — which is what writing the process down before automating it is about. The "unknown" rows are your agenda for the vendor calls: they are the questions you genuinely need answered, as opposed to the ones on the standard RFP template.

Prompt 2 — Turn your requirements into a per-case test protocol

Most buyers give a vendor a scope paragraph and get a demo back. The alternative is to give them a protocol and require a written result per case, before commercial terms are discussed.

Turn the following process description into a test protocol I can send a vendor,
with no prose beyond what is needed to run it:
- a table of 20 case slots: case id, what the input is, the outcome that is
  correct, the outcome that must route to a human, and what counts as a
  confident error
- the exact format required in their written response: per case, pass / fail /
  route to human, the reasoning, and a run identifier I can ask to see
- a scoring rule in which declining to run the cases is recorded as a failure
  to respond
- the two sentences to include in the covering note that explain why a decline
  is scored that way

Then replace the twenty invented slots with sixty to a hundred real cases from last quarter, deliberately weighted towards the ones that took longest, escalated, or were later reversed. What you learn is not only an accuracy figure on your own traffic; it is how the vendor behaves when the inputs are not theirs to choose. The seven procurement questions cover the rest of the first round, and the per-case run is the one item on that list that cannot be answered with a slide.

Prompt 3 — Audit the vendor's written response for unfalsifiable claims

This is the prompt that earns its place. Paste the vendor's RFP response, or the email the sales engineer sent after the call, and ask for a claim-by-claim audit.

Below is a vendor's written response to our requirements. Extract every factual
claim into a table with these columns, one row per claim:
| the claim, quoted verbatim | type: accuracy, cost, security, timeline, or
capability | what artifact would prove it | is it falsifiable as written? |
the missing denominator, unit or period | the one question that would settle it |

Then, separately, list the five sentences that read as claims but contain no
falsifiable content, and say for each what it is doing rhetorically.

Do not assess whether any claim is true. Tell me only what I would have to see
in order to check it.

The "missing denominator" column is the deliverable. A claim of ninety-five per cent accuracy without a denominator, a period, and a case population is not a claim about your process; it is a claim about a dataset you will never see. The second list matters for a different reason: it is where the sentence "our platform is built for enterprise scale" ends up, and seeing five of them side by side in one output is more persuasive to a buying committee than any argument you could make in a meeting.

Keep the last instruction. Without it, the model drifts into evaluating the vendor, and you end up reading a summary of the response's own framing — which is the exact failure the prompt exists to prevent.

Prompt 4 — Convert a benchmark into a statement about your volume

Vendors price the part they control and stay silent about the part you carry. This prompt makes the arithmetic show which side of that line a claim sits on.

A vendor claims: "[paste the claim]".
My process runs [4,200] units a month, with [12%] requiring human handling, at
[6] minutes each, with a fully loaded rate of [$62] an hour.

1. State the three things that would have to be true for this claim to hold at
   my volume.
2. Compute the monthly cost of the human-handling tail under the claim, and
   under a claim five percentage points worse.
3. List the questions whose answers would change the number most, in order of
   how much they move it.

The output is not the number, which is arithmetic you could do yourself. The output is the third list, and the ratio in the second: on most workflows I have scoped, the human tail costs an order of magnitude more than the model calls, so the claim that moves the invoice is the one about how often the system sends work to a person — not the one about model accuracy. The per-token cost model is how you translate a claim into a line your finance team can review.

Prompt 5 — Generate the failure classes from your own worst cases

Generic systems fail generically, and the failures cluster. If you have twenty hard cases from last quarter, you already have the shape of what will break.

Here are 20 cases from last quarter that took the longest, were escalated, or
were later reversed. Group them into failure classes that a system trained on
typical traffic would get confidently wrong. For each class, state:
- the signal in the input that a system could plausibly learn to catch
- the check a person has to make at that point, and what they need to see
- the question I should put to the vendor about this class

Send the classes, not the cases, into the vendor conversation. A vendor who handles the classes has a real answer, and a vendor who does not will tell you so in the language of "we can configure that", which is itself a useful answer.

Prompt 6 — Normalise three quotes onto one table

Quotes are not comparable documents, because each one is written to compare favourably with the other two. Normalise the terms before anyone produces a spreadsheet that makes them look equivalent.

Here are three vendor quotes. Produce one table with a row per quote and these
columns: cost at 1x, 3x and 10x my current volume; whether retries and failed
attempts are metered; what a process change costs in year two; who pays for
human handling of exceptions; exit cost and export format; implementation time
with the assumptions stated.

Where a quote is silent on a column, write "silent". Never write "0" for silence.

The "silent" values are the comparison. A quote that is silent about year-two change requests and another that is silent about exception handling are not comparable on price, and the table is what makes that visible to people who have not read either document. This is the same discipline an AI ROI model that survives finance applies to your own numbers.

Prompt 7 — Extract what you keep when you stop paying

The exit is decided by the contract, not by the relationship, and nobody reads it carefully at the point when everyone is enthusiastic.

From the following order form and terms, list what we keep if we stop paying:
repository and source of the workflow, tool and schema definitions, prompts,
the evaluation set, historical run records, configuration, and any integration
code touching our systems.

For each item, quote the clause that grants it, or write "silent".
Then list the four clauses that would have to change for us to hold a real exit.

"Silent" is the default in a template agreement, and silence is resolved in the vendor's favour later. If orchestration lives with the vendor, what you are buying is not a tool but a transfer of your workflow boundary — the lock-in analysis is the longer version of that argument, and the artifact list above is what an exit requires in practice.

Prompt 8 — Write the one-day acceptance test for the handover

Evaluation ends with an acceptance test, and the test should be something a person who did not build the system can run without a briefing.

Write a one-day acceptance test for the handover of this system, as a checklist
for someone who is not a specialist. It must cover: cloning and running the
repository on a clean machine; executing the recorded evaluation set and
comparing the result to the claimed numbers; triggering the human-review path on
a known exception; replaying one historical run and reconstructing what the
system saw; and exporting everything in the documented format.

For each step give the command, the expected output, and what a failure means
about the handover rather than about the software.

That last clause changes the output from a QA script into a handover verdict. The concrete mechanics of making such a test real — what to check in the repo, the schemas, the evaluation set and the run history — are in how to verify a delivered AI system in one day.

What if procurement needs a written due diligence questionnaire?

Most organisations end up with one anyway, usually assembled by somebody in risk or legal from a template for software in general. The prompts above are the raw material for a better version, and generating it takes one more pass. If you would rather start from a finished document than generate one, the thirty-two-row template I send is the same material tightened into a questionnaire procurement can own.

Turn the questions and artifacts above into a due diligence questionnaire for our
procurement file. One row per item, with these columns:
| # | question, phrased so it can be answered in writing | the artifact that
answers it | which of our findings depends on it | who at the vendor signs it |
requested by date |

Group the rows: accuracy and failure behaviour · cost and metering · data
handling and subprocessors · exit and ownership · references and production
evidence. Mark any row that is a risk assessment item rather than an evaluation
item. Do not add questions I have not asked, and do not soften the wording.

Two things about this artefact are worth understanding before it goes out. The scoring column is what turns it from a questionnaire into a shortlisting tool: a vendor who answers every accuracy question and leaves the exit row blank has told you where the risk sits. And the refused rows are answers too — a refused artifact request, in writing, with a name against it, is the cheapest evidence you will ever get that the system is not portable. Keep the refusals in the file rather than dropping them; the person who inherits this decision will need them more than they need the answers.

What these prompts cannot do

They cannot verify anything. Every prompt above produces a table of things to go and look at, and the table is worth nothing until a human being hands you the artifact in the right-hand column. A model asked to summarise a vendor's answer will summarise the vendor's framing; a model asked what evidence would settle a claim will tell you — and only the second one is doing diligence.

Three rules keep this honest:

Never let the model grade the answer. Replace "is this response good?" with "what is missing from this response?" the moment you notice yourself wanting the first one.

Keep the do-not-invent instructions. They are not politeness. A model that is asked to fill a table will fill a table, and a fabricated denominator is worse than an empty row because it survives review.

Do not skip the two weeks. Every prompt here is clerical work around a decision. The decision itself is made by running sixty to a hundred of your own recorded cases through the system and reading the three numbers that come back: the share handled without review, the cost per run, and the cases where it is confidently wrong. No prompt produces those numbers, and no demo substitutes for them.

The order that matters

Run prompts one and two before the first vendor meeting, so you are testing against a written register instead of a feeling. Run three, four and five against the first written response, and treat the empty cells as the shortlist filter. Run six and seven before anything is signed, because both are cheaper before signature by a wide margin. Run eight at handover, and refuse to accept the handover until a non-specialist has run it end to end.

The uncomfortable part is that this list applies to anyone selling AI work, including the person who wrote it. Ask me for the failure cases, the denominators and the exit terms, and you will get specific answers — including the ones that are inconvenient. A buyer's guide whose author is exempt from it is a sales document in a checklist costume, and the vendors worth hiring are the ones who hand you the table without being asked twice.