Idempotent Agent Writes: Derive the Key From the Inputs, Never From the Attempt

Why is a duplicate effect a retry problem rather than a bug?

An agent action that touches money or state needs an idempotency key derived deterministically from its semantic inputs, because the retry that produces the duplicate is the transport working as designed. The same logical action will be attempted more than once in routine operation: a client SDK retries a timeout, a queue redelivers an unacknowledged message, an MCP client reconnects and replays a tool call, an agent loop re-plans the step, a human presses Approve again because the spinner did not move.

None of that is an incident. The reason duplicates cost so much is that a duplicated effect reports success twice. Both attempts return 200, both write a row, both send a receipt, and nothing in your logs is red. The defect is invisible in the failure path and visible only in the ledger of the system you wrote to, usually when a customer reconciles an invoice. I have stopped filing duplicate writes under incidents and started filing them under missing keys, because in every case I have debugged the handler was correct and the key was absent, non-deterministic, or derived from the wrong thing.

Every agent action that touches money or state needs an idempotency key derived deterministically from its inputs, because retries are the normal case and a duplicated effect returns 200 OK on both attempts.

That distinction decides where you spend the engineering. Hunting the retry is unbounded work against a transport you do not control. Removing the retry's ability to produce a second effect is a table and a hash.

How many times does one logical action actually execute?

More than the number of retries you configured, because the layers multiply rather than add.

A four-attempt HTTP retry budget behind a queue that delivers up to five times is twenty executions of one intent, and that is before the agent loop re-plans. Multiply that by a write path that moves forty thousand RMB and the arithmetic stops being theoretical.

The distribution matters more than the mean, and this is from my own logs rather than a benchmark: on the invoice-matching workflow I run, the downstream pricing call answers in under half a second at the median but crosses a thirty-second client timeout on roughly two calls in a thousand, and the client reports those as failures after the write has committed. At three thousand write calls a month, two in a thousand is six duplicate payments a month. Your tail percentage will differ; the shape will not, because every timeout you set is a bet that the other side is slower than you are patient.

What makes a key deterministic enough to dedupe on?

The key has to be a pure function of what the action means. In practice that requires canonicalising the arguments before hashing them: sort object keys, normalise strings to NFC and trim them, hold money as integer minor units, pin timestamps to UTC, and apply optional-field defaults before hashing rather than after validation drops them. Two spellings of one action must produce one key. Two meanings must produce two.

import { createHash } from "node:crypto";

type Json = string | number | boolean | null | Json[] | { [key: string]: Json };

function canonical(value: Json): string {
  if (value === null || typeof value !== "object") return JSON.stringify(value);
  if (Array.isArray(value)) return `[${value.map(canonical).join(",")}]`;
  const keys = Object.keys(value).sort();
  return `{${keys.map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(",")}}`;
}

// scope keeps two tenants, or two currencies, from sharing a keyspace
export function deriveKey(scope: string, args: Json): string {
  const body = canonical({ scope, args });
  return createHash("sha256").update(body, "utf8").digest("hex").slice(0, 32);
}

The boring parts are where the bugs live. JSON.stringify alone is not canonical: { a: 1, b: 2 } and { b: 2, a: 1 } serialise differently, so the same payment intent arrives with two keys and executes twice. 40000 and "40000.00" are different keys even when your validator coerces both to the same Money type, so canonicalise after coercion, never before. A UUID generated per attempt dedupes nothing at all, and a key that includes the attempt id, the trace id or a timestamp is worse than no guard, because it looks like one.

Where is the key computed, and what names the intent?

At the tool boundary, in your server, from validated arguments. Not in the prompt, and never by the model. An LLM asked to emit an idempotency key produces a different string on a re-plan, which reintroduces the duplicate; and a key the model can choose is a key that prompt-injected content can choose to collide with an unrelated action.

Two levels are usually needed, because intent and payload diverge exactly when retries happen. The intent is named by something stable outside the model's prose: the plan id, the step index, plus a normalised identifier for the target resource. The payload is a hash of the canonical arguments.

If the model re-plans and emits 40000.00 instead of 40000, or Acme Limited instead of Acme Ltd, you must not silently create a second effect. Same intent plus a different payload fingerprint is a conflict: reject the call with a 409, return the discrepancy to the loop as a validation error, and let the agent or a reviewer resolve it. A conflict is cheap. A second payment is not.

MCP makes this your problem rather than the protocol's. A JSON-RPC id correlates a response with a request inside one session, and a client that reconnects after a dropped stream issues a new one, so it cannot serve as a dedupe key. If your tool server mutates state, the ledger is the part you own.

In what order do the claim and the side effect run?

Claim, then effect, then commit. Never the other way round, and never record after the fact.

create table effect_ledger (
  intent_key          text primary key,
  request_fingerprint text not null,
  scope               text not null,
  status              text not null check (status in ('claimed', 'committed')),
  attempt_count       integer not null default 1,
  result              jsonb,
  lease_expires_at    timestamptz not null,
  committed_at        timestamptz
);

create index effect_ledger_reclaim_idx
  on effect_ledger (lease_expires_at)
  where status = 'claimed';

insert into effect_ledger (intent_key, request_fingerprint, scope, status, lease_expires_at)
values ($1, $2, $3, 'claimed', now() + interval '60 seconds')
on conflict (intent_key) do update
  set attempt_count = attempt_count + 1,
      lease_expires_at = excluded.lease_expires_at
  where effect_ledger.status = 'claimed'
    and effect_ledger.lease_expires_at < now()
returning status, attempt_count, result;

The unique constraint is the arbiter. Two workers racing on one key: one inserts, the other matches the conflict, fails the lease predicate and receives zero rows. A select followed by an insert in application code has a race window between the two statements; the database does not. Zero rows returned is not an error, it means the action is already in flight or already done, so read the row to find out which.

type ClaimResult =
  | { kind: "acquired" }
  | { kind: "replay"; result: unknown }
  | { kind: "in_flight" }
  | { kind: "fingerprint_conflict" };

export async function runEffect<T>(
  intentKey: string,
  fingerprint: string,
  effect: () => Promise<T>,
  record: (result: T) => Promise<void>,
): Promise<T | null> {
  const claim = await claimEffect(intentKey, fingerprint);

  if (claim.kind === "replay") return claim.result as T;
  if (claim.kind === "in_flight") return null; // the original attempt owns this key
  if (claim.kind === "fingerprint_conflict") throw new IntentConflict(intentKey);

  try {
    const result = await effect();
    await record(result);
    return result;
  } catch (err) {
    if (isDefinitelyPreCommit(err)) await releaseClaim(intentKey);
    throw err; // ambiguous errors keep the lease for the reconciler
  }
}

The catch block is the part teams get wrong. Separate errors you know happened before any byte was written, such as validation or a refused connection, from ambiguous ones, such as a read timeout after the request body was sent. Release the lease for the first so a retry can acquire it; leave it claimed until the lease expires for the second, and let the reconciler query the downstream by the reference you passed. Treating a timeout as "failed and retryable" is how one intent becomes two payments.

What do you do when the downstream has no idempotency key?

Email, SMS, a partner webhook, an ERP you cannot change. Here idempotency is not available and it is worth saying so out loud rather than putting a per-attempt UUID on the request and calling it guarded. What you can build is recoverability: write the intent and a business reference into an outbox, send with that reference embedded in the payload, then confirm. A crash leaves a row without confirmation, and the reconciler asks the downstream whether reference INV-2291 arrived. The reference is what makes the question answerable; without it, the only recovery is to ask a human.

If the downstream cannot be queried by a business key at all, the honest options narrow to two: keep the effect behind an explicit human confirmation, or batch it into a file with a checksum so a re-send is detectable. Both are slower than a retry loop. Both are cheaper than a duplicate you cannot see.

When is an idempotency ledger the wrong investment?

Read-only tool calls do not need it. A duplicate read costs tokens, not money, and the fix there is caching rather than a ledger. Writes that are naturally idempotent by construction, such as PUT of a full document or setting a status field to a constant, do not need it either: the second write lands on the same state, and a ledger would only add a row.

When the downstream already has a first-class key, pass yours through instead of inventing a second scheme, but still record the effect locally. Without a local row you cannot attribute the charge to the intent that caused it, which is the same observability gap you were trying to close. Costs are real but small: one row per intent plus an update per attempt, roughly two hundred bytes each, so three thousand writes a day is about 1.1 million rows and 220 MB a year at a twenty-four-month retention. That is not the constraint.

The constraint is the retention window. If your keys expire after twenty-four hours and a queue backlog can replay an action after seventy-two, you have a forty-eight-hour duplicate window and a guard that fails silently on the days it matters most. Set the window from your longest realistic retry horizon plus your reconciler's lookback, and write that number down where the on-call engineer can find it. And no ledger helps if side effects leave your process before you commit the claim, or if the key is derived from free text the model can rewrite; in those two cases the design is wrong rather than under-configured.

What should you change in your highest-value write path this week?

Pick the one write path in your system that moves money or changes a record a customer will notice, and add three things: a ledger table whose primary key is the idempotency key, a key derived at the tool boundary from canonicalised arguments rather than from the attempt or the model, and a claim inserted before the effect runs. Then write the test that runs the handler twice concurrently with identical arguments and asserts a single row and a single downstream call; it is roughly fifteen lines, and it is the cheapest evidence you will ever have that the path is safe to leave unsupervised. That test is also one of the items on the checklist a workflow has to pass before it leaves its pilot, which is the argument to make when someone wants to defer the work to the next quarter. The retry storm is not scheduled. It arrives with the first elevated error rate, and by then the duplicate is already committed.