When Not to Use AI: If a Rule Solves It, a Model Is a Downgrade

Most AI proposals are requests for a deterministic system

When someone asks me to put a model on a process, my first question is never about models. It is: what rule would a competent new hire be told to follow? If that rule fits in a paragraph, the cheapest correct system is the paragraph, written as code — a conditional, a database constraint, or a scheduled job.

When a deterministic rule, a database constraint or a scheduled job produces the right answer, a model is a downgrade: you trade a failure mode you can explain for one you cannot, and you attach a per-run bill to work that used to be free.

I have been paid to build the model version of a rule more than once. The builds worked. That is the uncomfortable part. They worked well enough that nobody asked whether a cheaper system would have been more accurate, and the bill arrived every month, and no one in the room could name which three per cent of the outputs were wrong.

What does the cheaper option actually cost?

Three deterministic mechanisms cover most of what I am asked to build.

A conditional. If the field is empty, reject the record. If the amount exceeds the threshold, route to approval. Written once, tested once, unchanged for years, and readable by anyone who reviews it.

A database constraint. A unique index on a supplier's tax identifier ends duplicate vendor records more thoroughly than any classifier, because it refuses the insert at the moment the error happens instead of noticing it later. The database cannot be argued with.

A scheduled job. A nightly run that finds deals with no activity in fourteen days and queues a follow-up. AWS lists Lambda requests at $0.20 per million invocations on its published pricing page at the time of writing, and a job that runs once a night sits far below that line. Check the figure yourself; the point is not the number but that the number is knowable in advance and does not move with the volume of judgement required.

Now the comparison that actually decides the question, and it is not the per-run price. Take 10,000 items a month. A deterministic system that handles 96% of them and raises an explicit exception on the other 4% hands you 400 items to review, and those 400 identify themselves. A model that is right 96% of the time hands you 10,000 items, of which 400 are wrong and none of which announce the fact. Reviewing all 10,000 at twenty seconds each is about 55 hours of somebody's month. Reviewing 400 is a little over two hours. The gap is not the token bill. It is the review labour that a probabilistic system forces you to spread across everything, because it does not know which of its outputs are the bad ones.

Why does the model get proposed anyway?

Four mechanisms, roughly in the order I encounter them.

The budget line. "AI" is often the only version of the project that gets funded. The same work described as fixing the vendor master data competes with a hundred other operational chores; described as an AI initiative it acquires an executive sponsor and a quarter of protected time. This is a naming problem, and it is the most common one. It is also not dishonest — it is how capital allocation works in most companies.

Nobody wants to own the rule. A rule is a decision with a name attached. When it fires wrongly, the person who wrote it is accountable for the outcome. "The model returned that" distributes the accountability across a vendor, a prompt and a temperature setting. I have watched this trade be made deliberately, in rooms where several people knew the rule was available and none of them wanted to be the one who wrote it.

The rule cannot be written down. This is the legitimate version of the argument, and it is worth taking seriously. If two experienced people apply the same written policy to the same ten real cases and disagree on three of them, the variance lives in the process rather than in the model, and no architecture will repair it. That condition deserves a diagnosis before you choose a tool, which is the argument I made in automating the wrong process first.

The input is genuinely unstructured. A scanned supplier invoice, a customer email, a page of a contract. There is no conditional that reads handwriting, and no constraint that reconciles a totals row printed in a different currency format.

Only the last two are technical reasons. The first two are procurement behaviour, and they are expensive, because they buy the wrong system with the right label.

What does the proposal look like when it is written down?

This is roughly the table I fill in during a scoping conversation, using the requester's own words in the left column. The verdict column is mine, and I have had to defend the bottom two rows more often than the top four.

The pattern in the top four rows is not that models cannot do those jobs. It is that the deterministic version is already correct, already auditable, and already cheap, so the model version adds only variance.

What does a deterministic system fail at?

I do not want to overstate the case, so here are the limits as I have hit them.

Non-enumerable exceptions. A rule with two exceptions is a rule. A rule with forty exceptions, each learned from a different escalation two years ago, is a lookup table nobody maintains. Past roughly a page, maintaining the rule costs more than a model that approximates it.

Unstructured input. Text, scans, audio, free-form email. The extraction step is where models earn their place, and it is a narrow place: extract the fields, then apply the rule to the extracted fields. Most of the value is still in the deterministic part.

Judgement as the output. Drafting, summarising, explaining a discrepancy to a customer. Correctness is a matter of taste, so there is no rule to write and no constraint to enforce.

Silent distribution shift. A rule breaks when the input changes shape; you see it in the error rate. A model degrades gradually as the world moves away from its training distribution, and it degrades without announcing itself. A rule is the better alarm system, which is a reason to keep one in the pipeline even after a model arrives.

Why build the rule first even when a model is the destination?

Three reasons, and they are all practical.

The rule gives you a labelled dataset for free. Every case the rule handles cleanly is a correct example; every exception it raises is a case worth labelling. Teams that start with a model spend the first month of the project manufacturing what a rule would have produced as a by-product.

The rule gives you a baseline. "The model agrees with the rule on 91% of the 4% residual" is a sentence a finance committee can act on. "The model seems good" is not.

The rule is the fallback. When the model provider has an outage, changes a model version, or raises prices, the rule still runs. In two systems I maintain, the deterministic path handles the bulk of volume and the model is called only for the residual, which keeps the per-item cost low and the review queue short enough for one person to clear before lunch.

There is also a governance argument that is narrower than it is usually stated. Most regulators do not forbid statistical models. They ask who can explain a decision to a customer or an auditor. Explaining a predicate takes a minute and a code reference. Explaining a model output takes the input, the prompt, the model version and a conversation about confidence that most operations teams are not equipped to have at nine in the morning.

What I would do with the next AI proposal on your desk

Ask for the rule. Write it down together, apply it by hand to two hundred past cases, and count how many it gets right. If it covers ninety per cent or more, ship the rule this month, route the remainder to a named human queue, and revisit the model in two quarters with a baseline in hand. If it covers less, you have found the genuine candidates, and you now know exactly which part of the work is unstructured — which is the only part worth paying a model for. Either way you will have spent a week and learned something the vendor's demo would not have told you.