AI System Maintenance Is a Quarterly Replay, Not a Model Upgrade

Why does an AI system that still works stop working?

Nothing about an AI workflow announces its own decline. There is no error state for "the answers are now slightly worse than they were in October", which is why maintenance on these systems fails: the failure looks like calm, and calm does not get escalated. The instrument that catches it needs no lab, no benchmark subscription and no vendor evaluation, only your own past cases, replayed.

An AI workflow does not fail loudly; it drifts, and a quarterly replay of archived cases against the current system is the cheapest early-warning instrument you can run.

I have watched this on systems I built and on systems I inherited. The one that cost the most was not the one that crashed. A document workflow kept returning clean, well-formed output for four months after a prompt edit narrowed what the model treated as a complete answer. It surfaced when a customer's compliance team noticed a missing clause.

What changes in a quarter when nobody changed anything?

Four things move, and only one of them is the model.

The input distribution. A new product line, a new region, a new form field, a supplier that changed its template. Your prompt still parses the old shape, and it will do something plausible with the new one, which is worse than failing.

The volume mix. Total cases flat, hard cases up. If the workflow is measured as an aggregate accuracy figure, a rising share of edge cases reads as a small dip everyone attributes to noise.

The surrounding process. An upstream field becomes optional. A review step gets shortened to meet a deadline. The person who used to catch one specific error class is promoted.

The prompt and the glue. Somebody adds a sentence to a template. A retry path fires more often than it used to. A longer document quietly exceeds a context limit and the tail is dropped, with no error and no log line.

None of these produce an exception, and all of them move output quality in the same direction: down, slowly, in the cases that are hardest to see.

Why does the model behave differently when the code never changed?

Two arrangements, two different problems. If your integration calls a floating alias, a pointer that resolves to whatever the provider currently serves as its default, you receive behaviour change with no deployment and no trace in your build history. If you pin a specific version instead, the provider publishes a retirement date for it, usually months ahead, and on that date the endpoint stops answering or moves under you. Neither is maintenance-free. Pinning is still the better choice, because it converts a quiet drift into a dated event, and a dated event goes on a calendar.

The model endpoint is not the only versioned dependency. If your retrieval layer uses a hosted embedding model, a version change on the provider's side makes previously stored vectors non-comparable with newly generated ones. Similarity scores shift, thresholds that used to be correct start picking up the wrong neighbours, and nothing raises an error. The same holds for a reranker, a transcription model, or a small classifier that routes cases before the language model sees them.

An AI system has more external dependencies than a normal service, and most are owned by someone else.

How do you make a replay cheap enough to actually run?

The reason teams skip this is not disagreement: "evaluate the system" sounds like a project. It is not, if you structure it as a diff.

Re-run a sample of archived cases through the current system. Compare the new output against the output you stored at the time. Adjudicate only the cases that differ.

That third line is where the economics live: on a stable workflow most reruns reproduce the earlier output closely enough that no human needs to look, so review time goes to the disagreements rather than the sample. In the replays I have run, two hundred cases took one person most of a day to set up and diff, and two to three hours of adjudication once the disagreements were separated.

Cost the re-run as cases times average input tokens times your contracted rate. Two hundred cases at four thousand input tokens each is eight hundred thousand tokens of input, a quantity you can price against your own invoice in under a minute. Where that bill is not trivial, you chose the sample size. Sixty stratified cases will not catch a rare regression, but it will catch a systematic shift. If the harder question is how you score those cases, there is a longer treatment of how to score an AI workflow you cannot test in a lab.

What has to be in the archive before any of this works?

Six fields per case, and the sixth is the one people skip:

• The input exactly as the system received it.
• The output exactly as the system produced it, before any human edit.
• The model version, prompt version and retrieval index version.
• The timestamp.
• What the reviewer actually did: accepted, edited, rejected, plus the final version that shipped.
• How long the reviewer spent.

That last field tells you whether your ground truth is real. A case accepted in four seconds by someone working through a queue of two hundred is not evidence that the output was correct. It is evidence that the reviewer was tired. Treat those cases as unlabelled rather than as passing, or you will spend a quarter proving that a rubber stamp is consistent.

Keep at least a few cases where the human call was wrong; without known-bad outputs in the set, you cannot distinguish a regression from a correction.

Storage collides with retention rules. Pseudonymise identifiers rather than deleting them where the domain allows, because a case you cannot re-run is not an archived case, and decide the retention question while you still have the data. If you genuinely cannot keep certain inputs, keep a synthetic twin that reproduces the structure and say so in the report: it detects input-shape drift and nothing about content.

What does the report say, and who acts on it?

Not a score: a short list where every disagreement carries a cause and an owner.

• Input drift. The upstream system changed. Fix it at the source, or add validation that refuses the new shape instead of guessing at it.
• Dependency change. Dated and attributable. Re-test against the pinned version, or accept the new behaviour and re-baseline deliberately.
• Prompt or code change. Someone's edit. Check whether it did what they believed it did.
• Business rule change. The world moved and the archive is now wrong. Re-adjudicate that subset and update the reference set.

The distribution across those four buckets is the management signal. If most disagreements land in the fourth, your system is fine and your test set is stale. If the first dominates, you have an integration problem that is being blamed on the model.

The owner should be whoever carries the outcome, not whoever built the pipeline: builders have a bias toward their own prompt and read a disagreement as a labelling error. A vendor cannot run this for you either, since a provider's replay uses the provider's sample.

How do you tell that a replay is measuring nothing?

Zero disagreements quarter after quarter is not a good result. It usually means the sample never reaches the hard cases, or you are comparing post-processed outputs, or somebody re-baselines the archive after each run, making last quarter's output this quarter's reference. If most cases differ, you are not measuring drift either: the task changed, or the archive predates a rule change.

Never let the replay tune the system automatically. If the model's own outputs become the reference, the next replay measures agreement with the previous run, and drift becomes invisible by construction. The reference has to contain human decisions, including the human mistakes.

Does every AI workflow need a quarterly replay?

No, and the honest version of this argument says which ones do not. If volume is low enough that a person reads every output before it leaves the building, the human is the maintenance system and drift is caught for free. That is the correct design for plenty of workflows, and a replay on top of it is bureaucracy. The replays earn their cost where two conditions hold together: volume is high enough that human review has quietly become sampling, and the cost of a wrong output lands outside your building, on a customer, a regulator or a supplier. That combination is what makes silent degradation expensive. Everything else can wait.

Watch for the moment the first condition flips: a workflow fully reviewed at two hundred cases a month gets sampled at two thousand, and nobody announces the change because the queue still clears daily.

What to do this quarter

Open your logs and ask one question: from what I stored, could I reconstruct the exact input that produced this output six months ago, along with the model version and what the reviewer did with it? If yes, you already own the expensive half of maintenance, and the replay is a few hours of one person's time per quarter. If no, archiving is the work this quarter, and the first replay tells you whether you stored the right fields. The arithmetic is simple: an early-warning system costing one day per quarter is worth running even in the quarters where it finds nothing, because the quarter where it finds something is the one where you find it instead of your customer.