@ultimat3/jobs 2.0.0 → 4.0.0

package/CLAUDE.md

@@ -35,16 +35,35 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   A look-alike never registers. Deliberately not a registry lookup — the registry is what
   registration rewrites.
 - `idempotencyKey` is NON-OPTIONAL in `JobDefinition`. Never relax it, never default it.
-- **The idempotency namespace is `(name, idempotency_key)`, never the key alone** (`As of
-  2026-08`). It was the key alone, and that is silent data loss with no error anywhere: two jobs
-  that derive the same natural key from the same input — `sendWelcomeEmail` and
-  `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so the second enqueue hit
-  `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found the FIRST job's row and
-  returned `{ id: <A's>, deduped: true }`. The workspace was never provisioned and `x jobs ls`
-  showed one healthy job. Three places have to agree and a test pins them together: the index, the
-  conflict target, and the live-row lookup. `x_jobs` SHIPPED, so the DDL `drop index if exists`es
-  the old one — left in place it would keep enforcing exactly the collision this fixes. The
-  scheduler's occurrence key already prefixes the task name and is unaffected.
+- **The idempotency namespace is `(name, coalesce(tenant_id, ''), idempotency_key)`, never the key
+  alone and never name-only** (`As of 2026-08`). Two rounds of the same defect, and both are silent
+  data loss with no error anywhere.
+
+  It was the key alone: two jobs deriving the same natural key from the same input —
+  `sendWelcomeEmail` and `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so
+  the second enqueue hit `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found
+  the FIRST job's row and returned `{ id: <A's>, deduped: true }`. The workspace was never
+  provisioned and `x jobs ls` showed one healthy job.
+
+  Then it was name-only, while the row already carried `tenant_id` as `$9` of the same insert.
+  Every natural key the docs suggest is unique only WITHIN a tenant — `` `invoice:${input.invoiceId}` ``,
+  `` `order:${input.orderNumber}` `` — so tenant B enqueuing while tenant A held that key deduped
+  into tenant A's row: B's work never ran AND B's caller received A's job id, which is valid on
+  every id-addressed surface (`cancelJob(driver, jobId)` takes an id with no tenant predicate, so
+  an app wiring the returned id to a cancel button gave B cancellation of A's job). The sibling
+  projection in `@ultimat3/action` (`idempotency-key.ts`) had folded the actor's org in all along.
+  `coalesce`, not the bare column: a null `tenant_id` compares unequal to every other null under a
+  unique index, so a tenantless queue would lose its dedupe entirely — all tenantless rows share
+  one namespace instead, which is what they had before tenancy existed.
+
+  Three places have to agree and a test pins them together: the index, the conflict target, and the
+  live-row lookup. The conflict target must spell the index EXPRESSION exactly —
+  `(name, (coalesce(tenant_id, '')), idempotency_key)` — or Postgres cannot infer the index at all.
+  `driver-memory.ts` mirrors it with `(record.tenantId ?? '')`, which is the parity that turned a
+  gap into a confirmed one rather than catching it. `x_jobs` SHIPPED, so the DDL
+  `drop index if exists`es BOTH superseded indexes — each is strictly narrower than its successor,
+  so either left in place would keep enforcing exactly the collision this fixes. The scheduler's
+  occurrence key already prefixes the task name and is unaffected.
 - **`SQL_JOBS_TABLE` is the ONE install point, and every durable table this package owns is in
   it** (`As of 2026-08`): `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_scheduler_state`,
   `x_scheduler_leader`, `x_job_leases`, `x_job_events`. Four of those were subsystems that shipped
@@ -237,6 +256,30 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   so, not a job that failed: nacking it re-runs completed work and records `retried` in
   `jobs_total{outcome}` for a failure that never happened. It propagates instead, and the worker's
   `jobs.worker.settle-failed` plus the lapsing lease are the honest answer.
+- **The retry decision reads the ERROR as well as the attempt count — added 2026-08.**
+  `executeJob` decided a retry from `nextRetry(handle.retry, attempt)` alone, so every `terminal`
+  classification in the framework was decorative on the job path: an `X_SCRAPE_AUTH_FAILED` from a
+  rotated password burned the whole policy, which at a site that locks an account after three wrong
+  passwords makes the framework's retry the thing that destroys the account.
+  `retry-classification.ts` composes AROUND `nextRetry` — the backoff arithmetic stays in one
+  place — and `nextRetryForError` is the only caller `execute.ts` has.
+
+  **`classifyThrown` must never read `error.retry` on its own.** That field is
+  `init.retry ?? retryFor(code)` and `retryFor` FAILS CLOSED, so every unclassified `UltimateError`
+  already carries `terminal`; reading it would dead-letter the first attempt of every job in every
+  app whose codes nobody has classified. Hence core's `declaredErrorRetry(code)`, which answers
+  `undefined` where `retryFor` answers the default — and hence the one case that is knowingly
+  under-read: an instance `retry: 'terminal'` on an UNREGISTERED code is indistinguishable from the
+  default and is treated as unclassified. Register the code; that is the one way.
+  `retry-after` reuses the delay the nack already takes (`meta.retryAfterSeconds`, clamped by the
+  policy's `maxDelay`) rather than a second suspension mechanism — `StepSuspension` stays the only
+  way to park a run, and unlike a suspension a retry-after DOES burn an attempt, because the work
+  failed. The ceiling outranks every classification but `terminal`.
+
+  **The verdict is published, not inferred**: `jobs.attempt.failed` and `reportError` carry
+  `stop`, `JobExecution` carries `stopReason`, and `recordedFailure` appends the terminal verdict to
+  the nack's `error` — `lastError` is the ONE failure field a row has, so without it `x jobs show`
+  renders a dead letter at attempt 1 of 5 as a silent early stop.
 - **The claim loop re-arms on the PASS, never on the jobs.** A slot belongs to its own job and is
   free the moment it settles, so `claimRound` starts what it claimed and returns the promises —
   ending the pass on `Promise.allSettled([...inFlight])` made the pool as slow as its slowest
@@ -401,8 +444,80 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   A SIGTERM landing between `driver.enqueue` and `markPublished` returned to a caller that then
   closed the database under the row it was about to mark. `OutboxRelay.stop(): Promise<void>` — a
   caller that does not await gets what it always got (the chain carries its own `catch`), so the
-  join is only as good as the `await`: **`packages/cli/src/dev-roles.ts` still calls it without
-  one**, in both teardown paths.
+  join is only as good as the `await`. Both of `packages/cli/src/dev-roles.ts`'s paths take it
+  today (`:306` returns `() => relay.stop()` so the rollback awaits it, `:356` is
+  `await relay?.stop()`); this file claimed the opposite until 2026-08.
+- **The outbox claim is a LEASE, and one statement is what makes it one** (`As of 2026-08`).
+  `SQL_OUTBOX_CLAIM` was a bare `select ... for update skip locked` run on the POOLED executor, and
+  those row locks last only for their own statement — under autocommit they are gone before
+  `claim()` resolves, and `x_outbox` had no claimed column, so nothing fenced the batch at all. Two
+  relays 200ms apart read the identical rows and both published them. **The idempotency key does
+  not collapse that in general**: `SQL_ENQUEUE`'s conflict target is the PARTIAL index over
+  `('ready','delayed','running','suspended')`, so a repeat landing after the first job reached a
+  terminal state inserts a second row and the handler runs twice. The mechanism is Postgres
+  semantics; the second-publish-after-terminal ordering was argued, never reproduced — say it that
+  way, in a comment or a changelog. The claim now stamps `claimed_at`/`claimed_by` in the same
+  statement that locks (the CTE shape `SQL_CLAIM` already used), and the outer
+  `select ... order by staged_at, id` is load-bearing: `update ... returning` has no defined row
+  order and the relay publishes in the order it is handed rows. `claimed_at` is a LEASE and not a
+  flag — without a reclaim window a relay that died mid-batch strands its rows forever — and
+  `OutboxStore.release` (optional, so a store written before this still compiles) hands back the
+  batch a failed publish stopped, or one pool blip would park committed work for a whole lease
+  window instead of one poll interval. `createMemoryOutboxStore` answers the SAME question, on an
+  injected clock, and `outbox-claim.test.ts` pins the two side by side.
+- **The lease is fenced on EVERY outbox mutation, and the sort key is TOTAL** (`As of 2026-08`).
+  Two holes the first version of the lease left open, both reachable without a second relay
+  process. `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED` matched on `id` alone, so a relay
+  that stalled past its own lease still spoke for rows another relay had reclaimed: its late
+  `release` unclaimed a batch mid-publish (a third relay claims it, publishes it again — the
+  duplicate the lease exists to prevent, reached the long way round) and its late `markPublished`
+  retired a row nobody had published, losing the job with nothing to notice. Both now carry
+  `and claimed_by = $n`, `claim()` hands the token back as `OutboxRecord.claimedBy`, and the relay
+  passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
+  first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
+  way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
+  ONE store, and a per-store id could not tell them apart. **An absent token is NOT one rule in
+  both**: `createMemoryOutboxStore`'s `owns(id, undefined)` answers `true` unconditionally, so a
+  caller with no token really is unfenced there — while `createPgOutboxStore` substitutes
+  `claimant ?? relayId` into `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED`, both of which
+  carry `and claimed_by = $n`, so a token-less call fences on THIS store's relay id and no-ops
+  against a row some other relay holds. `outbox-pg.ts:159-165` is the honest comment. Neither store
+  refuses such a caller, which is the shared half: a caller holding no token is one written before
+  the fence.
+  **`order by staged_at` was not a total order**: every row staged in one transaction shares a
+  `staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
+  order they publish, differed between two relays and between two runs of one. `, id` fixes it in
+  the CTE and in the projection, and needed NO DDL: `id` is a UUIDv7 minted by `uuid()`, monotonic
+  and already the primary key, so the tiebreak IS stage order. `byClaimOrder` in `outbox.ts` is the
+  memory store's copy of that key.
+- **`claimLeaseMs` is normalised in ONE place — `outbox-lease.ts`** (`As of 2026-08`). Both stores
+  call `resolveClaimLeaseMs`, which owns `DEFAULT_OUTBOX_CLAIM_LEASE_MS` and refuses anything that
+  is not a positive whole number of ms with `X_INVARIANT` (the generic, no new code: same borrow
+  `@ultimat3/db` makes). A memory default and a pg default that could drift are two answers to
+  "how long is a claim mine for", and the shorter one publishes a row twice. `0` expires before
+  `claim()` resolves and `Infinity` never expires, so both are refused at CONSTRUCTION, not at the
+  first tick where the only trace is a log line.
+- **A renewal is decided against `stopped()`, not only against the interval** (`As of 2026-08`).
+  `renewal-timer.ts` is the one shape, read by `heartbeat.ts` and `worker-fleet-slots.ts`, and it
+  exists because both files reported a LOSS for a job that had finished cleanly: `stop()` cleared
+  the interval, which does nothing to the request already on the wire, so the fenced statement came
+  back `false` — the row left `running` when `executeJob` acked it — and that answer took the loss
+  branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
+  queue re-delivered a job this process was still running, so a false one is a page for a
+  non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
+  await and inside the reporter, the way `settleWithin`'s `decided` does in core.
+- **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
+  are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
+  implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
+  only production construction passed neither and `JobDefinition` had no field that could. Same
+  verdict as `job.concurrency`: a documented guarantee that silently does nothing is the worst of
+  the three options, so it is threaded rather than deleted. Both are refused at declaration when
+  non-positive, because `withStepTimeout` reads `<= 0` as "no ceiling at all".
+- **`registeredJobs()`/`registeredTasks()` sort by CODE UNITS, never `localeCompare`.** The list is
+  projected by `describeJobs()` into `x.manifest.json`, which both tracked apps commit and
+  `x verify`'s `drift` step diffs byte for byte; `localeCompare` with no locale argument answers
+  from the runtime's ICU default and collation version. Same rule as `@ultimat3/http`'s
+  `describeRoutes`, restated locally rather than imported — `http` is not below this package.
 - **A driver's semantics are pinned in ONE test with the pg statement beside them.**
   `driver-parity.test.ts` asserts the memory driver's behaviour and the SQL that has to mean the
   same thing in a single test, so neither side can move alone. `introspect.list` answered
@@ -438,8 +553,20 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   `backfill-pass-fixture.ts` raises `BackfillHandleFailure`, a plain `Error` subclass on purpose:
   a backfill `handle` is app code and the pass propagates what it threw, so a framework code there
   would exercise a path no app takes.
-- Suspension is control flow: `StepSuspension` -> `nack({ countsAsAttempt: false })`.
-  Never log it as an error, never let it burn an attempt.
+- **Suspension is control flow, and a SHED is not a suspension** (`As of 2026-08`).
+  `StepSuspension` -> `nack({ countsAsAttempt: false, park: true })`; never log it as an error,
+  never let it burn an attempt. The two facts were ONE flag until 2026-08: a limiter shed and a
+  `job.concurrency` shed both handed the job back with `countsAsAttempt: false`, and both drivers
+  derived `deadLetter ? 'dead' : counts ? 'ready' : 'suspended'` — so a job that is merely WAITING
+  was filed beside a 3-day sleep. `SQL_STATS` and the memory `stats()` then counted it out of
+  `ready` and out of `oldest_ready_ms`, which `worker.ts` publishes as `queue_depth` and
+  `queue_oldest_ready_seconds`: 20 jobs at `concurrency: 10` behind `createLimiter({ global: 1 })`
+  read as a depth of 10 with 19 waiting, and under sustained overload the shed fraction approaches
+  100%, so the HPA signal and the "oldest job older than 5 minutes" page both go quiet exactly
+  when the queue is saturated. `park` is now the state and `countsAsAttempt` is the counter, only.
+  The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
+  that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
+  where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
 - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
 - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
 - Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
@@ -514,11 +641,13 @@ picture from the other side.
 | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
 | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
 | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
-| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements |
+| `backfill-errors.ts` | the seven `X_BACKFILL_*` classes — split out of `errors.ts`, which was over the 500-line ceiling. The codes themselves stay declared in `errors.ts`: one registry, one place |
+| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements. Skips a non-job in silence — a module namespace is full of helpers — EXCEPT an `@ultimat3/action` projection (`kind: 'action-job'`), which is `X_ACTION_JOB_UNBRIDGED` |
 | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
 | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
 | `outbox.ts` | staging in a `Tx`, the relay, the ambient `JobsFacade` slot |
 | `outbox-pg.ts` | `createPgOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
+| `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
 | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
 | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
 | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark) + `createPgLeaseLeader` |
@@ -530,8 +659,10 @@ picture from the other side.
 | `driver-memory.ts` | `x dev` / tests |
 | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
 | `retry.ts` | backoff arithmetic, dead-letter decision |
+| `retry-classification.ts` | the OTHER half of that decision: what the thrown error says, and the stop reason the row and the log carry |
 | `execute.ts` | `executeJob` — one claimed job run and settled, and the run's deadline/cancel |
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
+| `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
 | `worker.ts` | `worker` role, claim loop, drain |
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |

package/README.md

@@ -315,6 +315,20 @@ Nothing can kill a body that ignores the signal, so the durable state is fenced:
 cancel every step write is refused with `X_ABORTED`, and a run that finishes anyway is logged
 as `jobs.timeout.abandoned` — the one way to find a handler that never reads `ctx.signal`.
 
+Three ceilings, declared on the job and nowhere else (`As of 2026-08` — `stepTimeout` and
+`eventPoll` had been implemented in the step runner since 1.0 with no declaration able to reach
+them, so no `job()` could ask for either):
+
+| Field | Bounds | Absent |
+|---|---|---|
+| `timeout` | the whole attempt — aborts `ctx.signal`, then fails it | no attempt deadline |
+| `stepTimeout` | ONE `step.run` — aborts that step's signal, then fails the step | no per-step ceiling |
+| `eventPoll` | how long a `step.waitForEvent` parks between polls | 30s |
+
+A zero or negative `stepTimeout` / `eventPoll` is refused at declaration, the way `concurrency: 0`
+is: `withStepTimeout` reads `<= 0` as "no ceiling at all", which is the opposite of what the author
+wrote.
+
 ## The transactional outbox
 
 ```ts
@@ -334,10 +348,15 @@ it after commit. The bug class this removes:
 
 Both are load-dependent, both pass every test you would write, and both produce "the email
 went out but the order isn't in the database". Joining the transaction closes the window.
-The relay publishes *then* marks published, so a crash re-publishes — collapsed by the
-idempotency key. A publish that FAILS stops the batch rather than letting later rows overtake it:
-`claim()` returns rows in `staged_at` order, so an app that stages `createInvoice` then
-`chargeCard` in one transaction must never have the charge run first. Set `mode: 'required'` to
+The relay publishes *then* marks published, so a crash re-publishes — and **that repeat is
+collapsed only while the first job is still live** (`As of 2026-08`): `SQL_ENQUEUE`'s conflict
+target is a partial index over `ready`/`delayed`/`running`/`suspended`, so a re-publish landing
+after the first job reached a terminal state inserts a second row and the handler runs again.
+**Handlers are at-least-once. Write them idempotent** — that is the standing contract, not a
+caveat on this paragraph. A publish that FAILS stops the batch rather than letting later rows
+overtake it: `claim()` returns rows in `staged_at, id` order — total, so two relays compose the
+same batch in the same order — and an app that stages `createInvoice` then `chargeCard` in one
+transaction must never have the charge run first. Set `mode: 'required'` to
 make an enqueue outside a transaction an `X_OUTBOX_NO_TX` error instead of a direct publish.
 
 **It is not on by default, and it is not on until you install it** (`As of 2026-08`). Three
@@ -355,6 +374,34 @@ transactional: `stage()` runs on the CALLER'S connection, never the pool. With n
 publishes straight to the driver — deliberate, so a script and a test enqueue with no wiring, but
 it is a fallback and not the guarantee.
 
+`claim()` is a **claim, not a read** (`As of 2026-08`). `for update skip locked` in a bare select
+holds its row locks only until that statement ends — under autocommit, before `claim()` even
+resolves — so two relays polling 200ms apart read the same unpublished rows and both publish them.
+`SQL_ENQUEUE` collapses that repeat only while the first job is still LIVE, because its conflict
+target is a partial index over the live states: a second publish landing after that job finished
+inserts a second row and **the handler runs twice**. So the claim stamps `claimed_at` in the same
+statement that locks the row, and that stamp is a lease — `claimLeaseMs` (a positive whole number
+of ms, default 30s; anything else is `X_INVARIANT` at construction) is how long the rows of a relay
+that DIED mid-batch wait before any relay may take them again. A batch a failed publish stopped is
+handed back at once through `release`, so a pool blip still costs one poll interval and not a lease
+window.
+
+**Every outbox mutation is fenced on the claimant, not just the claim** (`As of 2026-08`).
+`release` and `markPublished` both match on `claimed_by`, and `claim()` hands the token back on
+each record as `claimedBy`. A relay that stalls past its lease wakes up owning nothing: its late
+`release` would otherwise unclaim rows the relay that reclaimed them is mid-publish on (a third
+relay claims and republishes them), and its late `markPublished` would retire a row nobody has
+published yet — losing the job outright. Both are no-ops now, in the pg store and in the memory
+store alike.
+
+What the lease buys, precisely:
+
+| It stops | It does not stop |
+|---|---|
+| two relays holding one batch — a committed row cannot be claimed twice inside its lease | the handler running twice |
+| a lapsed claimant releasing or retiring a newer claimant's rows | a crash between publish and `markPublished` re-publishing after the first job is terminal |
+| a relay that died mid-batch stranding its rows forever | anything a **non-idempotent** handler does on its second run |
+
 The memory store (`createMemoryOutboxStore`, `x dev` and tests) **drops** a published row —
 `retained()` is the relay's backlog, not a running total; the pg store keeps `published_at` as
 the audit trail this map is not. A relay pass that throws is logged as `jobs.outbox.tick-failed`
@@ -447,6 +494,28 @@ retrySchedule({ attempts: 5, backoff: 'exponential', delay: 1000 })
 // => [1000, 2000, 4000, 8000]
 ```
 
+**The error decides too, not only the attempt count** (`As of 2026-08`). `executeJob` reads the
+thrown error's retry classification — `@ultimat3/core`'s `registerErrorRetry`, the same table
+`--json` and an HTTP client read — and a code nobody classified keeps the attempt-count path
+exactly as it had before.
+
+| Thrown | What the queue does |
+|---|---|
+| a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
+| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay` — instead of the backoff. Still an attempt, still under the ceiling |
+| a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
+| an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
+
+Classify your app's codes beside the module that declares them — that import IS the registration:
+
+```
+registerErrorRetry({ X_INVOICE_REJECTED: 'terminal', X_GATEWAY_BUSY: 'retry-after' });
+```
+
+Why a job stopped is on the row and in the log, never inferred: `jobs.attempt.failed` carries
+`stop: 'terminal' | 'attempts-exhausted'`, `JobExecution.stopReason` carries the same, and a
+terminal dead letter appends its verdict to `lastError` so `x jobs show` explains an attempt 1 of 5.
+
 ## Limits
 
 Two layers, and the difference matters:
@@ -486,6 +555,13 @@ still running it — at-least-once turning into twice. Alert on any non-zero rat
 measured from the last renewal that **landed**, on this process's clock, so a driver whose
 heartbeat hangs is caught the same as one that rejects.
 
+Neither fires for a job that finished (`As of 2026-08`). `stop()` is terminal for the renewal
+already **on the wire**, not only for the next one: a clean completion acks the row out of
+`running`, so the fenced UPDATE already in flight comes back `false` — and reported, that was
+`jobs.lease.lost` at error plus the counter, a page for a non-event, on every completed job whose
+pool was slow enough. The fleet slot's `jobs.worker.slot-lost` had the same shape and the same
+fix (`renewal-timer.ts`).
+
 ## Introspection
 
 `inspectQueues`, `inspectJob` (per-step trace), `inspectDeadLetters`, `retryFromStep`,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "2.0.0",
+  "version": "4.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "2.0.0",
-    "@ultimat3/entity": "2.0.0",
-    "@ultimat3/schema": "2.0.0",
-    "@ultimat3/time": "2.0.0"
+    "@ultimat3/core": "4.0.0",
+    "@ultimat3/entity": "4.0.0",
+    "@ultimat3/schema": "4.0.0",
+    "@ultimat3/time": "4.0.0"
   }
 }

package/src/backfill-errors.ts

@@ -0,0 +1,137 @@
+// The seven `X_BACKFILL_*` codes, apart from `errors.ts` for the reason `driver-pg-rows.ts` is
+// apart from `driver-pg.ts`: one file, one job, and `errors.ts` was over the 500-line ceiling
+// `x verify`'s `filesize` step enforces. The registry stays there — `JOB_OWNED_ERROR_CODES`,
+// `JOB_ERROR_TITLES` and the single `registerErrorCodes()` call — because a package's codes are
+// declared in ONE place; only the classes that throw them live here, beside `backfill-ledger.ts`,
+// `backfill-pending.ts` and `backfill-registry.ts`.
+
+import { UltimateError } from '@ultimat3/core';
+import { docsFor } from './errors';
+
+/**
+ * The seven backfill codes below all answer one question — "why is this sweep not running?" — and
+ * each is here because it sends the reader somewhere different: run it, force it, change
+ * environment, migrate first, wait, fix the predicates, or fix the name. A code that shared a fix
+ * line with another one would be code inflation, which is why `X_BACKFILL_WRITE_UNCONFIRMED` was
+ * considered and rejected: a dry run that wrote nothing did exactly what it was asked to.
+ *
+ * Every `fix` below is ONE line something can execute — a shell command, or the edit to make.
+ * Never a command with prose appended: a `fix:` is copied and run verbatim, so a trailing clause
+ * turns a working command into a syntax error at the one moment the reader is following it
+ * literally. Explanations belong in `cause`, which is read and never run.
+ */
+
+/**
+ * Declared and never completed. The alarm the framework did not have: an author could
+ * `x g backfill`, merge and deploy, and nothing anywhere said the pass had not run.
+ */
+export class BackfillPendingError extends UltimateError {
+  constructor(input: { backfill: string; environment: string }) {
+    super({
+      code: 'X_BACKFILL_PENDING',
+      cause: `backfill "${input.backfill}" is declared and x_backfills holds no completed pass for it in ${input.environment}`,
+      fix: `x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_PENDING'),
+    });
+  }
+}
+
+/** Already swept. A rerun is legitimate, so this names the flag rather than refusing outright. */
+export class BackfillAppliedError extends UltimateError {
+  constructor(input: { backfill: string; runId: string; completedAt: string }) {
+    super({
+      code: 'X_BACKFILL_APPLIED',
+      cause: `backfill "${input.backfill}" completed as run ${input.runId} at ${input.completedAt}; a forced rerun writes a NEW ledger row and never edits that one`,
+      fix: `x db backfill ${input.backfill} --write --force --json`,
+      docs: docsFor('X_BACKFILL_APPLIED'),
+    });
+  }
+}
+
+/**
+ * The declaration names the environments it belongs to and this is not one. Declared DATA, never a
+ * hardcoded "cleanups are production": a staging rehearsal is correct practice, so which
+ * environments a sweep belongs to is the app's convention and this is only the mechanism carrying
+ * it (axiom 8).
+ */
+export class BackfillEnvironmentError extends UltimateError {
+  constructor(input: { backfill: string; environment: string; declared: readonly string[] }) {
+    // The first declared environment, because the fix has to be ONE runnable line and the list is
+    // ordered by the author. The empty case cannot arise from `checkBackfillEnvironment`, which
+    // treats an empty list as "every environment" — but this constructor is public, so it answers
+    // with the command that lists what IS declared rather than an `ULTIMATE_ENV=undefined`.
+    const target = input.declared[0];
+    super({
+      code: 'X_BACKFILL_ENVIRONMENT',
+      cause: `backfill "${input.backfill}" declares environments: ${input.declared.join(', ')} and this process resolved ${input.environment} — add "${input.environment}" to that list if this deploy should sweep too`,
+      fix:
+        target === undefined
+          ? 'x db backfill --pending --json'
+          : `ULTIMATE_ENV=${target} x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_ENVIRONMENT'),
+    });
+  }
+}
+
+/**
+ * `requires` names a migration the ledger has not applied. Checked where `x_migrations` is
+ * readable — this package holds no `@ultimat3/db` dependency and growing one to read a ledger
+ * would put the migration engine on the tier-3 queue's import graph.
+ */
+export class BackfillMigrationPendingError extends UltimateError {
+  constructor(input: { backfill: string; migration: string }) {
+    super({
+      code: 'X_BACKFILL_MIGRATION_PENDING',
+      cause: `backfill "${input.backfill}" requires migration ${input.migration}, which x_migrations does not record as applied`,
+      fix: 'x db migrate --json',
+      docs: docsFor('X_BACKFILL_MIGRATION_PENDING'),
+    });
+  }
+}
+
+/**
+ * The enqueue deduped: one live pass per name, so this run is the one already going. Distinct from
+ * `X_BACKFILL_APPLIED`, which is a pass that finished — the response there is `--force`, and the
+ * response here is to look at the run that is holding the key.
+ */
+export class BackfillRunningError extends UltimateError {
+  constructor(input: { backfill: string; jobId: string }) {
+    super({
+      code: 'X_BACKFILL_RUNNING',
+      cause: `backfill "${input.backfill}" already has a live pass queued as ${input.jobId}, and one name holds one live pass; its step trace names the batch it is on, and a pass that is not advancing is a worker that lost its lease`,
+      fix: `x jobs show ${input.jobId} --json`,
+      docs: docsFor('X_BACKFILL_RUNNING'),
+    });
+  }
+}
+
+/**
+ * The source ran out of rows and the declaration's own `count()` still matches some. Two
+ * predicates that disagree is an authoring bug in any business — the sweep reported success over
+ * rows it never visited — so the pass fails rather than writing a completed row nobody can trust.
+ */
+export class BackfillStalledError extends UltimateError {
+  constructor(input: { backfill: string; remaining: number; swept: number }) {
+    super({
+      code: 'X_BACKFILL_STALLED',
+      cause: `backfill "${input.backfill}" swept ${input.swept} rows, exhausted its source, and count() still matches ${input.remaining} — a WHERE the sweep narrows and the count does not is what leaves rows behind`,
+      fix: `make count() select on exactly what source() selects on in backfill("${input.backfill}")`,
+      docs: docsFor('X_BACKFILL_STALLED'),
+    });
+  }
+}
+
+/** A name no declaration in this app carries — a typo, or a backfill whose module was deleted. */
+export class BackfillUnknownError extends UltimateError {
+  constructor(input: { backfill: string; known: readonly string[] }) {
+    super({
+      code: 'X_BACKFILL_UNKNOWN',
+      cause:
+        input.known.length === 0
+          ? `no backfill named "${input.backfill}" is declared, and this app declares none at all`
+          : `no backfill named "${input.backfill}" is declared (declared: ${input.known.join(', ')})`,
+      fix: 'x db backfill --pending --json',
+      docs: docsFor('X_BACKFILL_UNKNOWN'),
+    });
+  }
+}

package/src/backfill-gate.ts

@@ -8,15 +8,15 @@
 // convention rather than a rail (axiom 3).
 
 import type { Environment, UltimateError } from '@ultimat3/core';
-// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
-// own row shape, which would make this a second reader of `x_backfills`.
-import type { BackfillProgress } from './backfill-inspect';
-import type { BackfillDeclaration } from './backfill-registry';
 import {
   BackfillAppliedError,
   BackfillEnvironmentError,
   BackfillMigrationPendingError,
-} from './errors';
+} from './backfill-errors';
+// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
+// own row shape, which would make this a second reader of `x_backfills`.
+import type { BackfillProgress } from './backfill-inspect';
+import type { BackfillDeclaration } from './backfill-registry';
 
 export type BackfillGate =
   | { readonly run: true }

package/src/backfill-pass.ts

@@ -16,13 +16,13 @@
 import { appVersion, assert, logger, resolveEnvironment } from '@ultimat3/core';
 import type { BatchIterator } from '@ultimat3/entity';
 import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill';
+import { BackfillStalledError } from './backfill-errors';
 import { checkBackfillEnvironment } from './backfill-gate';
 import type { BackfillLedger, BackfillRun } from './backfill-ledger';
 import { decideBackfill } from './backfill-ledger';
 import type { Pacer } from './backfill-rate';
 import { withBackfillScope } from './backfill-scope';
 import { jobDriver } from './driver';
-import { BackfillStalledError } from './errors';
 import type { JobRunArgs } from './job';
 import { isStepSuspension } from './steps';
 

package/src/describe.ts

@@ -14,10 +14,18 @@ export interface JobDescriptor {
   readonly queue: string;
   readonly retry: { readonly attempts: number; readonly backoff: BackoffStrategy };
   readonly steps: readonly string[];
+  /**
+   * Whether a replayed attempt is safe to run — `job()` REQUIRES an `idempotencyKey` and refuses
+   * a definition without one (`X_IDEMPOTENCY_REQUIRED`), so this is `true` for every registered
+   * job. That is the point: the guarantee, published where an operator asks the question, rather
+   * than left as prose in a doc. The KEY itself never crosses — it is computed from an input and
+   * is app data, so a descriptor carrying it would put customer ids in `x.manifest.json`.
+   */
+  readonly idempotent: boolean;
 }
 
 /**
- * Narrower than `JobHandle` on purpose: the projection reads four declared fields, so keeping
+ * Narrower than `JobHandle` on purpose: the projection reads five declared fields, so keeping
  * it structural means it never has to carry — or vary with — the handle's input generic.
  */
 export interface DescribableJob {
@@ -25,6 +33,13 @@ export interface DescribableJob {
   readonly queue: string;
   readonly retry: RetryPolicy;
   readonly input: unknown;
+  /**
+   * `JobHandle.idempotencyKeyFor`, read only for its presence — `unknown` because the real
+   * signature is `(input: I) => string` and this shape is deliberately free of the generic.
+   * Required, not optional: a descriptor built without it would publish `idempotent: false`,
+   * which is the exact wrong answer the `/_x` jobs panel used to give for every job.
+   */
+  readonly idempotencyKeyFor: unknown;
 }
 
 export function describeJob(handle: DescribableJob): JobDescriptor {
@@ -39,6 +54,7 @@ export function describeJob(handle: DescribableJob): JobDescriptor {
     // Empty by design: step names are chosen inside `run()` at execution time, so they are
     // not statically knowable. `inspect(name)` reports the steps an actual run recorded.
     steps: [],
+    idempotent: typeof handle.idempotencyKeyFor === 'function',
   };
 }