Observability Without a Platform Team: Three Signals in One Postgres Table

Why does observability get scoped as a platform-team project?

Three signals carry most of the value in production telemetry: a trace per request, a cost counter tied to the unit of work, and a failure record you can replay. None of the three requires a platform team, and the version I ship runs all of them in one Postgres table with two TypeScript helpers and no agent process.

A trace per request, a cost counter per unit of work and one replayable failure record answer most production questions, and a team of five can operate all three inside the database it already backs up.

Most observability writing is addressed to organisations with an on-call rotation and a telemetry budget line. The tooling is shaped accordingly: an agent per host, cardinality-based pricing, a query language to learn, a collector to deploy and upgrade. A small team adopts the free tier, instruments half of one service, hits the cardinality limit, and stops. Six months later the dashboards are stale and the only telemetry anyone actually reads is the provider invoice.

The failure is not the tooling. It is that a team of five copied the shape of an organisation of two hundred without its staff. What a small team needs is not coverage, it is answerability: which run produced this quote, what did last week's change do to cost per order, and can I make this failure happen again on purpose. Those are three questions, and each one maps to one signal.

What are the three signals, and what does each one answer?

The distinction that matters is not logs versus metrics versus traces. It is the question class each signal can close a week later, when nobody remembers the deploy.

Rows one and two are cheap enough that there is no reason to do them late. Row three is the one teams skip, and it is the one that decides whether a change can be released after an incident.

What does the storage actually look like?

One table, one row per span, and a run-level row whose name is run. Span rows are immutable once written, which means no update path, no lock contention, and a table you can back up like any other.

create table run_span (
  run_id         uuid        not null,
  span_id        uuid        not null,
  parent_span_id uuid,
  tenant_id      text        not null,
  name           text        not null,
  kind           text        not null check (kind in ('run', 'llm', 'tool', 'db', 'http')),
  status         text        not null check (status in ('ok', 'error', 'timeout')),
  attempt        smallint    not null default 1,
  workflow       text        not null,
  prompt_version text,
  input_tokens   integer     not null default 0,
  cached_tokens  integer     not null default 0,
  output_tokens  integer     not null default 0,
  cost_micros    integer     not null default 0,
  error_code     text,
  payload_ref    text,
  started_at     timestamptz not null,
  duration_ms    integer     not null,
  primary key (run_id, span_id)
);

create index run_span_tenant_time_idx on run_span (tenant_id, started_at desc);
create index run_span_failures_idx on run_span (workflow, started_at desc) where status <> 'ok';

create view run_cost as
select
  date_trunc('day', started_at) as day,
  workflow,
  count(*) filter (where kind = 'run' and status = 'ok') as successful_runs,
  count(*) filter (where kind = 'llm' and attempt > 1) as retried_attempts,
  sum(cost_micros) as cost_micros,
  sum(cost_micros) filter (where kind = 'run' and status = 'ok') as cost_micros_on_success
from run_span
group by 1, 2;

The index on failures is partial because failures are a rounding error in volume and the query that matters most is always "what broke this week". Cost is stored as micro-USD in an integer, never as a float, so the sums that finance audits are exact rather than approximately right. Inputs and model responses are not in the table: payload_ref points at a compressed object in storage you already pay for, which keeps the database small enough to restore in minutes.

With 9,000 runs a month at roughly 14 spans per run, that is 126,000 rows a month, about 1.5 million rows after a year. At a few hundred bytes per row including indexes that is under a gigabyte, and a nightly delete of rows older than 90 days keeps it flat forever.

How do you trace without threading a context through every call?

You keep the run context in async-local storage and wrap the calls that can fail. The wrapper is the only place that writes spans, so no call site has to know the run id and no background job can silently write orphan rows.

import { AsyncLocalStorage } from 'node:async_hooks';
import { randomUUID } from 'node:crypto';

interface RunContext { runId: string; tenantId: string; attempt: number; spanStack: string[] }

const run = new AsyncLocalStorage<RunContext>();

export async function trace<T>(
  name: string,
  kind: 'llm' | 'tool' | 'db' | 'http',
  body: (addCost: (micros: number) => void) => Promise<T>,
  explicitParentId?: string,
): Promise<T> {
  const ctx = run.getStore();
  if (!ctx) throw new Error(`trace("${name}") ran outside a run: no context to attach it to`);

  const spanId = randomUUID();
  const parentSpanId = explicitParentId ?? ctx.spanStack[ctx.spanStack.length - 1] ?? null;
  ctx.spanStack.push(spanId);

  let costMicros = 0;
  let status: 'ok' | 'error' | 'timeout' = 'ok';
  const startedAt = performance.now();

  try {
    return await body((micros) => { costMicros += micros; });
  } catch (error) {
    status = error instanceof Error && error.name === 'AbortError' ? 'timeout' : 'error';
    throw error;
  } finally {
    ctx.spanStack.pop();
    await insertSpan({
      runId: ctx.runId, spanId, parentSpanId, name, kind, status, attempt: ctx.attempt,
      costMicros, durationMs: Math.round(performance.now() - startedAt),
    });
  }
}

The failure path is the part worth reading twice. The span is written once, in finally, after catch has recorded the status, and the original error is rethrown untouched so the caller's retry logic still sees it. A naive version that inserts in both branches double-counts every failure and inflates your error rate by exactly the number of errors you have.

Nesting gives you the tree. A run span wraps the workflow, and each llm or db call inside it is a child with the same run_id and its own span_id, so a failed step can be walked back to the decision that produced it. insertSpan is one parameterised insert into run_span, batched per request if you are writing more than fifty spans a run.

The span stack has one real limitation, and it is the part of this helper I rewrite most often. A single mutable stack is correct while children run one after another and wrong the moment you fan out with Promise.all, because two concurrent children read the same parent and the resulting tree claims a step was nested inside a sibling. When I know a run fans out, I pass an explicit parentSpanId down instead of reading the stack, and the wrapper takes it as an optional fourth argument. A trace with a slightly wrong parent is worse than no parent at all, because it is the one defect a reader will never question.

How do you count cost per unit of work?

Attach tokens to the span, convert to integer micro-USD at the boundary, and sum per run. The conversion has a property that makes the whole thing auditable: one US dollar per million tokens is exactly one micro-USD per token, so tokens multiplied by a per-million price is already micro-USD.

type Tier = 'fast' | 'deep';

interface TokenUsage {
  readonly tier: Tier;
  readonly freshInputTokens: number;
  readonly cachedInputTokens: number;
  readonly outputTokens: number;
}

/** Illustrative USD per million tokens. Substitute your provider's current rates. */
const RATE_PER_MTOK: Record<Tier, { freshInput: number; cachedInput: number; output: number }> = {
  fast: { freshInput: 0.25, cachedInput: 0.025, output: 2 },
  deep: { freshInput: 3, cachedInput: 0.3, output: 15 },
};

export function costMicros(usage: TokenUsage): number {
  const rate = RATE_PER_MTOK[usage.tier];
  return Math.round(
    usage.freshInputTokens * rate.freshInput +
      usage.cachedInputTokens * rate.cachedInput +
      usage.outputTokens * rate.output,
  );
}

Using those illustrative rates, a run that sends 8,114 fresh input tokens, 32,000 cached tokens and 1,180 output tokens on the fast tier costs 2,029 + 800 + 2,360 = 5,189 micro-USD, just over half a cent. At 9,000 successful runs a month the happy path is about $46.70.

Now count attempts rather than runs. If 15 percent of runs take one extra attempt, that is 1,350 extra attempts, the invoice reads $53.71, and cost per successful run is $0.00597 rather than the $0.00519 a per-run average implies. Both numbers are arithmetic on stated inputs, and the second is the one that survives a review, because it holds the failed attempts in the numerator where they belong. I have watched a workflow retry a schema violation for eight days and reach 3.4 times its intended per-unit cost, invisible in every aggregate anyone was reading, because failing attempts were never counted as work.

The reconciliation is a single query against the view: sum cost_micros for a month and compare it with the provider invoice. A gap of a few percent is rounding and tokenizer drift. A gap of thirty percent is an unmetered call path, and you want to find that before finance does.

What makes a failure replayable rather than merely logged?

A stack trace tells you where the code stopped. A replay record tells you what the system was looking at when it stopped, which is the only version of the answer that lets you change one variable and re-run.

{
  "runId": "8f1c0b6e-6a5d-4a3f-9a0f-2a1b7c9d4e10",
  "tenantId": "acme-eu",
  "workflow": "quote-draft",
  "promptVersion": "quote-draft@12",
  "model": "fast-tier",
  "status": "error",
  "errorCode": "SchemaValidationError",
  "attempt": 2,
  "startedAt": "2026-02-01T09:14:07.221Z",
  "usage": { "freshInputTokens": 8114, "cachedInputTokens": 32000, "outputTokens": 1180 },
  "costMicros": 5189,
  "steps": [
    {
      "spanId": "b2a7c1d0-51f4-4c8a-9a3e-0d6f2b7e4411",
      "name": "retrieve-rate-card",
      "kind": "db",
      "status": "ok",
      "rowCount": 42,
      "rowsRef": "replay/2026-02-01/8f1c0b6e/rate-card.json.gz",
      "queryFingerprint": "sha256:71c0a4f9"
    },
    {
      "spanId": "c9e4a8b3-7d21-4f6b-8e02-9b3c5a1d7f60",
      "name": "draft-quote",
      "kind": "llm",
      "status": "error",
      "errorCode": "SchemaValidationError",
      "attempt": 2,
      "responseRef": "replay/2026-02-01/8f1c0b6e/draft-quote.2.json"
    }
  ]
}

Four fields do the work. prompt_version pins the instruction text, because a prompt edited on Tuesday makes Monday's failure meaningless. queryFingerprint pins the retrieval, so a changed where clause shows up as a different fingerprint rather than as a mysterious behaviour change. rowsRef stores the rows the model actually saw, which is what makes the reproduction exact on a database that has moved on since. responseRef keeps the provider's raw response, so replay can serve it from the archive instead of paying for a live call you cannot reproduce anyway.

This is also the record a reviewer signs rather than a dashboard, which is why an untraceable workflow cannot pass a release gate: the reviewer's question is never "is it working now", it is "how do you know the fix holds for the input that broke it". The argument for treating replay as the release artifact rather than as a monitoring feature is one I have made at length in why an unreproducible failure cannot be signed off after a change, and the engineering consequence here is small: one extra column and an archive write on the paths that can fail.

When are three signals the wrong approach?

This design fails in four situations, and I would rather name them than defend it everywhere.

Two more honest limits. Postgres aggregates over tens of millions of spans stop being interactive, and the fix is materialised rollups, which is a platform again with a different name; below about 50 million rows in the window you care about, it is not a problem. And if your traffic is a few hundred runs a day that one person reads from a terminal, this is over-engineering: start with the replay record alone, because that is the one signal you cannot reconstruct after the fact. Traces and cost can be backfilled from logs and invoices. A payload you did not capture is gone.

What should you build first?

Create run_span, wrap your LLM call and your retrieval call in one trace helper, and let a run without context throw at the call site so you find the uninstrumented path in a test rather than in an incident. Add the micro-USD price table and one integer column, then post the cost-per-successful-run query where finance can see it, because that single number turns every later conversation about model choice into arithmetic instead of taste. Capture the payload on failure paths only, which is a fraction of your storage and all of your diagnostic value. Then trigger one real failure on purpose, replay it, and fix it: the first time that loop closes in an afternoon, the argument for a telemetry platform stops being a question of principle and becomes a question of what you would actually buy with the money.