LLM Token Cost per Workflow Needs Three Terms, and Retry Rate Is the One People Omit

What does one workflow cost before you have built it?

The number finance needs is cost per successful run, not cost per thousand tokens, and it is the product of three volume terms and two rate cards. Cost per token is an input to that model. It is not the model.

I priced a five-step quoting workflow twice: once on paper, once from thirty days of production logs. The rate cards were right and the per-step token estimates were within 12 percent. The paper version was still 18 percent low, because roughly one step in six gets executed twice and I had not given that a column.

A per-token cost model survives a finance review only when it multiplies three measured terms — input growth, output growth and retry rate — and retry rate is the one almost everyone leaves at zero.

What follows is the arithmetic of those terms, the ledger that measures them, and two cases where the model is not worth building.

Which three terms actually move the number?

Input growth is not a property of your prompt. It is a property of the loop. In a tool-calling workflow, step n sends the brief plus the results of steps 1 through n-1, so the volatile middle of the context grows with every call even when the cacheable prefix stays byte-identical. Across five-step workflows I have instrumented, peak input lands between 1.4 and 2.2 times first-step input. Below 1.4 the loop is probably not carrying tool results forward; above 2.2 something is dumping raw payloads into context.

Output growth is smaller and less obvious. Later steps have more evidence to summarise, and a schema-repair turn produces more output than a clean first pass. I use 1.4 as a default for extraction and quoting workflows. That is my own log figure, not a benchmark, so I re-measure it per workflow rather than carrying it across.

Retry rate multiplies the other two, so omitting it costs more than omitting either growth term. It is also the term nobody measures, for a structural reason: if your observability records runs instead of attempts, every failed attempt is already invisible in the data you have.

The largest single input-growth event I have measured came from a Model Context Protocol tool that returned a full quote history as unshaped JSON: 22,000 tokens in one result. These payloads come from servers I wrote, so there is nobody else to blame. Because growth is per step, that result is re-sent on each of the four remaining calls, which is 88,000 tokens at $2.50 per million, or $0.22 for that one response. The workflow it sits inside costs $0.097 per run once all three terms are in the model. One undeclared tool response cost 2.3 times the workflow containing it. Projecting the result down to the six fields the model actually reads brought it to about 300 tokens and removed the problem entirely.

How do you write that model so it survives scrutiny?

// lib/cost/model.ts
type TokenRates = {
  freshInputPerMTok: number;
  cachedInputPerMTok: number;
  outputPerMTok: number;
};

type WorkflowShape = {
  stablePrefixTokens: number;  // system prompt, rate card, tool schemas: cacheable
  volatileInputTokens: number; // brief, retrieved rows, tool results
  outputTokens: number;
  steps: number;               // model calls per run
};

type CostModel = {
  inputGrowth: number;  // input tokens on the last step / first step
  outputGrowth: number; // output tokens on the last step / first step
  retryRate: number;    // non-ok attempts / all attempts
};

const usd = (tokens: number, perMTok: number) => (tokens / 1_000_000) * perMTok;

export function costPerSuccessfulRun(
  shape: WorkflowShape,
  model: CostModel,
  rates: TokenRates,
): number {
  const attempts = 1 + model.retryRate;
  const freshPerStep = shape.volatileInputTokens * model.inputGrowth;
  const cachedPerStep = shape.stablePrefixTokens;
  const outputPerStep = shape.outputTokens * model.outputGrowth;

  const perStep =
    usd(freshPerStep, rates.freshInputPerMTok) +
    usd(cachedPerStep, rates.cachedInputPerMTok) +
    usd(outputPerStep, rates.outputPerMTok);

  return perStep * shape.steps * attempts;
}

Three details in that function carry the argument. Retry is applied to the whole per-step cost rather than to a separate error budget, because a failed attempt that hits a schema violation has already generated and been billed for its output tokens. The cached prefix is multiplied by step count, because prompt caching saves money per call and the benefit therefore scales with the loop. And the growth terms sit on per-step volume, not the run total, because that is where they are incurred.

Run the same shape with the terms switched on one at a time, using 8,000 cached prefix tokens, 1,500 volatile input tokens, 600 output tokens and five steps at $0.25, $2.50 and $10.00 per million tokens respectively:

The three-term model is 65 percent above the one-term model on the same traffic. Any of the three terms will move a budget line more than a model downgrade will.

A cost model is only half of the case it sits inside, and an understated cost line is the fastest way to lose a reviewer who was prepared to believe the benefit. The benefit side has its own failure modes, which is why the costing work here pairs with the AI ROI model that survives finance review rather than standing alone.

How do you measure the retry rate instead of asserting it?

You cannot compute retry rate from a table of finished runs. The unit you log has to be the attempt, not the run, and the token counts have to be written on the failure path as well as the success path — the provider reports usage on partial output too.

-- migrations/0007_workflow_run.sql
create table workflow_run (
  run_id         uuid        primary key,
  attempt        smallint    not null default 1,
  workflow       text        not null,
  step_index     smallint    not null,
  outcome        text        not null
    check (outcome in ('ok', 'schema_error', 'rate_limited', 'timeout', 'refusal')),
  input_tokens   integer     not null,
  cached_tokens  integer     not null default 0,
  output_tokens  integer     not null,
  started_at     timestamptz not null default now()
);

create index workflow_run_window_idx on workflow_run (workflow, started_at);

Insert on every call, including the calls that throw. I write the row in the same finally block that emits the error to the client stream, so a crashed attempt still has a cost attached. If your ledger keys on run_id alone with an upsert, the failed attempt overwrites its own evidence and the retry term stays invisible forever.

With the ledger in place, all three terms come from one query:

-- sql/three_terms.sql
select
  workflow,
  round(
    (count(*) filter (where outcome <> 'ok'))::numeric / nullif(count(*), 0),
    3
  ) as retry_rate,
  round(
    max(input_tokens - cached_tokens)::numeric
      / nullif(max(input_tokens - cached_tokens) filter (where step_index = 0), 0),
    2
  ) as input_growth,
  round(
    max(output_tokens)::numeric
      / nullif(max(output_tokens) filter (where step_index = 0), 0),
    2
  ) as output_growth
from workflow_run
where started_at >= now() - interval '30 days'
group by workflow
order by retry_rate desc;

Both ratios are peak-to-first, which is the conservative reading: it takes the widest context the workflow reaches and divides by the cheapest. The input ratio subtracts cached_tokens, because that is the volume the growth term scales in the model; leaving the constant prefix in the numerator would damp every ratio toward 1.0 and hide exactly the growth you are looking for. Feeding those three numbers into the model turns it from an assumption into a measurement, and re-running the query monthly tells you when it has expired. My threshold is five points of retry rate; past that I re-derive the rate cards too, because a retry surge usually means the output contract changed.

What breaks the model once real traffic arrives?

Three things, in the order they have cost me money.

Cache misses. The stable prefix above is 8,000 tokens per step, so 40,000 tokens per run. At $0.25 per million that is $0.010; at $2.50 fresh input rates it is $0.100. A single interpolated timestamp near the top of the system prompt is enough to break the byte-identical prefix, and the nine cents per run that follows is $900 a month at 10,000 runs — larger than the retry term it was supposed to sit beside. Volatile data goes after the cacheable block, always.

A retry rate that is not stationary. A model calibrated in March at 0.18 read 0.31 in May after a provider version bump began rejecting a field that had previously arrived as a number or a string. Nothing in my code changed and no dashboard showed an incident, because a retry that eventually succeeds looks like a slow request. Retry rate is a distribution over input types, not a constant.

Failure classes with different costs. Not every retry costs a full step, and averaging them hides where the money goes:

Those shares come from two workflows I have instrumented, not a survey; treat them as a shape, not a benchmark. The 429 row is the one worth engineering around, because a rejection before generation is nearly free, so a quarter of failures can be made almost irrelevant with backoff that honours the Retry-After header.

The last piece is reconciliation. Model and invoice drift apart silently, so compare them over the same period:

// lib/cost/reconcile.ts
type BilledRun = { workflow: string; billedUsd: number };

type Drift = {
  workflow: string;
  modelledUsd: number;
  billedUsd: number;
  errorPct: number;
};

export function reconcile(
  modelled: ReadonlyMap<string, number>,
  billed: readonly BilledRun[],
): Drift[] {
  const actual = new Map<string, number>();
  for (const run of billed) {
    actual.set(run.workflow, (actual.get(run.workflow) ?? 0) + run.billedUsd);
  }

  const rows: Drift[] = [];
  for (const [workflow, modelledUsd] of modelled) {
    const billedUsd = actual.get(workflow) ?? 0;
    if (billedUsd === 0) continue;
    rows.push({
      workflow,
      modelledUsd,
      billedUsd,
      errorPct: ((modelledUsd - billedUsd) / billedUsd) * 100,
    });
  }

  return rows.sort((a, b) => Math.abs(b.errorPct) - Math.abs(a.errorPct));
}

Sort by absolute error and fix the top row first. It is never the model that was hardest to price; it is the workflow where someone changed a prompt and did not touch the spreadsheet.

When is a per-token cost model the wrong approach?

This model is built for multi-step workflows with tool calls. It is the wrong instrument in four situations.

A single-call workflow has no loop, so input growth is 1.0 by definition, and a retry is a user pressing a button again. One multiplication is enough. Do not build a ledger for it.

A review-bound workflow is dominated by human time, and I have watched teams optimise the small number. Four minutes of review at a loaded rate of $58 an hour, which is $120,000 a year over 2,080 hours, is $3.85. Inference at $0.097 is 2.5 percent of the cost of one run. Cutting tokens there is a rounding exercise; the queue depth in front of the reviewer is the actual lever.

A flat-rate or seat-priced product turns cost into a capacity question. If inference is included in a monthly fee, the constraint is concurrency and rate limits, so model requests per minute and queue wait, not tokens per run. The three terms still tell you whether the flat rate is profitable at your usage, but they are not the operating model.

And at low volume the measurement costs more than it returns. Under a few hundred runs a month, the engineer time to build and maintain the attempt ledger exceeds the value of the visibility. Use conservative defaults — retry 0.25, growth 2.0 — label them as guesses in the document, and revisit when volume makes the ledger worth building.

What is the next concrete step?

Build the attempt ledger before the cost model, because the retry term cannot be recovered from data you did not collect, and every month without it produces a number you have to argue about instead of cite. Log one row per model call with step index, outcome, and token counts on both success and failure paths, then run the three-term query for thirty days before proposing anything.

When the numbers come back, put them in the model from this article, present cost per successful run rather than cost per token, and reconcile modelled against billed at the end of each month so drift has a named owner. The reviewer's real question is not whether tokens are cheap. It is whether your number holds at ten times the volume and half the retry rate, and a model with three visible terms answers that question in the meeting, while a model with one term answers it in month four as a budget overrun.