Why AI Knowledge Capture Fails When You Archive Answers Instead of Reasons

The part that walks out the door is the reason, not the answer

Knowledge loss inside a company is usually described as a filing problem, as though the documents exist and nobody can find them. The loss that costs money is narrower. It is the reasoning behind a decision that nobody wrote down because it did not look like documentation at the time, and an AI system can capture it if you point the system at decisions rather than at files. Most knowledge projects do the opposite, then report that the model was not useful.

The most valuable thing an AI system captures is not the task it performs but the decision rationale behind it, which is normally lost the moment the person who held it leaves.

I have rebuilt this after the fact twice for clients, and in both cases the expensive missing item was not a procedure. It was the set of conditions under which a procedure was allowed to be ignored.

Why is the rationale gone before the person gives notice?

Rationale decays earlier than the person does. Once a decision works twice, the reason compresses into "that is how we do it here", and the boundary that made it safe stops being spoken out loud. By the time someone resigns, they can still tell you what they do; they are much less reliable on which cases they would refuse today.

The record your systems keep makes this worse. An approval trail stores who approved, when, and the resulting value. It rarely stores the trade-off. The trade-off happened somewhere the system does not model: a call, a corridor, a message thread, a meeting where somebody said the customer would otherwise leave.

So rationale clusters where the system has no rule, which is exactly where a human overrode the default. Discount approvals, credit decisions, exceptions to a payment term, escalation of a stuck case, the first twenty minutes of an incident, change requests that skip a test. Those decisions carry judgement. The routine ones can be automated, and automating them is not where the value was.

There is also a social reason it goes unwritten. Some reasons are political, some are embarrassing: we did it this way because the client threatened to leave, because the previous attempt failed in a way nobody wants on the record, because the person who proposed the alternative is the one who signs the budget. None of that goes into a wiki page. It does go into a decision log read by the people making the same call next quarter, which is a different kind of object.

What does an AI pointed at outcomes get wrong?

Two systems, same question. An account manager asks whether a thirty-day payment term can be accepted for a mid-sized client.

A retrieval system over policy documents answers with the clause: standard terms are fourteen days, exceptions require finance approval. Correct, and useless.

A retrieval system over decisions answers differently. It finds the four times the term was extended in the last two years and reports the condition behind each one. Two were year-end deals with a public-sector buyer whose payment cycle is statutory. One was a client with nine years of payments and no late days. The fourth went ninety days overdue, and that case is why finance approval is now mandatory. It also surfaces the failed attempt, which is what makes the answer usable rather than merely true.

The failure mode without rationale is worse than unhelpful. Given only outcomes, a model will recite the policy confidently and, asked why, generate a plausible justification that was never anyone's reason. That output cannot be audited, because there is nothing to check it against. In a regulated process an explanation you cannot trace is a liability, not an answer.

Where is the raw material already sitting?

You are not starting from nothing. Approval trails, override dialogs, ticketing escalations, CRM notes, change requests and incident write-ups already emit one line per decision. What they lack is the two fields that carry the reasoning, and adding those is a much smaller change than a documentation programme. The longer version of that argument is in treating the records you already produce as the retrieval corpus.

Two fields do the work:

• The condition that made this the right call.
• The condition that would have made it the wrong call.

The second is the one people skip and the one that matters, because it bounds the precedent. Without it, the log records that an exception happened. With it, the log records when the exception is still available to you. A model handed three precedents and no boundary will generalise from them, and that is precisely how you get confident advice that is wrong on the next case.

Log the attempts that failed as well. Negative knowledge is the highest-value and lowest-supply item in most organisations, because nobody writes down a loss and the person who does looks worse for a quarter. One line is enough: what we tried, what broke, what it cost.

Then keep the raw note. Ask people to write reasons and they will write the clean version. Ask them to paste the thread or the original note next to the tidy summary, and keep the messy one as the ground truth. The summary is what retrieval matches against; the messy version is what a human needs when the summary is too smooth to trust.

Which of the three ways to capture this is worth funding?

The middle row is what most companies buy, and it has the worst ratio of cost to transferable reasoning. People are generous with what they did and vague about what they would refuse, which is the field the next person needs.

What does the arithmetic look like?

Payroll will tell you the cost of the person. Three lines: gross salary, employer contributions and benefits, and the last recruiter fee you actually paid. Most agencies in most markets quote 20 to 25 percent of first-year salary for a specialist placement, and you can read the real one off your own last hire.

Payroll will not tell you the cost of the dependency, which is the number that matters. Measure it as the queue. For one approval owner I tracked, two weeks of leave moved the median case from under three days to over three weeks, and the backlog took another month to clear. Those figures came from queue timestamps rather than an estimate, which is what makes them usable in front of a CFO: that is the cost of a partial version of the loss you are avoiding.

The volume needed for a usable log is arithmetic rather than ambition. If a queue handles forty cases a week and one in eight involves a judgement call, ninety days yields roughly one hundred and thirty precedents. That is enough for retrieval to answer the recurring questions, and small enough to still read.

How do you know it worked?

Start with a baseline you can reproduce. Take twenty closed decisions from the past year and give them to someone who was not involved, with no log available. Record whether they reach the same call, how long they take, and how many they escalate. Then instrument one queue and repeat the exercise after ninety days.

The number to move is second-person agreement without escalation. That beats documents captured, which counts activity, or benchmark accuracy, which measures something nobody in the process is paid to care about. A companion metric is simpler: for each question routed to the log, did it contain a precedent, yes or no. If the hit rate is low, the log is missing the decisions that block people, which usually means you instrumented the visible process instead of the override path.

What this does not fix

Rationale is portable; judgement under genuinely novel conditions is not. A log helps the second time the organisation meets a problem and does nothing the first time. Anyone selling institutional memory as a substitute for a senior hire is selling you the second occurrence and calling it the first.

Captured reasons are cleaned reasons, and a model reading clean reasons will sound more certain than the evidence supports. The counterfactual field and the raw note are the two guards against that, and neither holds if updating the log stays optional. The most common way this fails is not technical. The log becomes a compliance artifact, read once a year by an auditor, and it will be written for that reader rather than for the next person making the call.

The honest ceiling: few decisions have a single reason. Most have three, and one of them is usually that we ran out of time. A log that records the time constraint is more useful than one that records a tidy account of strategy, because the next person will also run out of time.

What to do in the next ninety days

Choose the one queue where a single long-tenured person is the gate, and log only the overrides for one quarter: the condition, the counterfactual, and the failures. The first ninety days are note-taking rather than software, and the entire cost is a few minutes a day from people who are already making those calls. Only then decide whether the corpus is worth wiring into a retrieval system, because at that point you are testing a return against your own hundred and thirty precedents instead of a vendor's case study. Waiting has a cost too, but it is smaller than most people assume: the reasoning does not disappear faster because you spent a quarter writing it down. What you cannot do is buy it back after the person has left, and I have never seen a platform licence do it.