@ultimat3/jobs 3.0.0 → 4.1.0

package/CLAUDE.md

@@ -35,16 +35,35 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   A look-alike never registers. Deliberately not a registry lookup — the registry is what
   registration rewrites.
 - `idempotencyKey` is NON-OPTIONAL in `JobDefinition`. Never relax it, never default it.
-- **The idempotency namespace is `(name, idempotency_key)`, never the key alone** (`As of
-  2026-08`). It was the key alone, and that is silent data loss with no error anywhere: two jobs
-  that derive the same natural key from the same input — `sendWelcomeEmail` and
-  `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so the second enqueue hit
-  `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found the FIRST job's row and
-  returned `{ id: <A's>, deduped: true }`. The workspace was never provisioned and `x jobs ls`
-  showed one healthy job. Three places have to agree and a test pins them together: the index, the
-  conflict target, and the live-row lookup. `x_jobs` SHIPPED, so the DDL `drop index if exists`es
-  the old one — left in place it would keep enforcing exactly the collision this fixes. The
-  scheduler's occurrence key already prefixes the task name and is unaffected.
+- **The idempotency namespace is `(name, coalesce(tenant_id, ''), idempotency_key)`, never the key
+  alone and never name-only** (`As of 2026-08`). Two rounds of the same defect, and both are silent
+  data loss with no error anywhere.
+
+  It was the key alone: two jobs deriving the same natural key from the same input —
+  `sendWelcomeEmail` and `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so
+  the second enqueue hit `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found
+  the FIRST job's row and returned `{ id: <A's>, deduped: true }`. The workspace was never
+  provisioned and `x jobs ls` showed one healthy job.
+
+  Then it was name-only, while the row already carried `tenant_id` as `$9` of the same insert.
+  Every natural key the docs suggest is unique only WITHIN a tenant — `` `invoice:${input.invoiceId}` ``,
+  `` `order:${input.orderNumber}` `` — so tenant B enqueuing while tenant A held that key deduped
+  into tenant A's row: B's work never ran AND B's caller received A's job id, which is valid on
+  every id-addressed surface (`cancelJob(driver, jobId)` takes an id with no tenant predicate, so
+  an app wiring the returned id to a cancel button gave B cancellation of A's job). The sibling
+  projection in `@ultimat3/action` (`idempotency-key.ts`) had folded the actor's org in all along.
+  `coalesce`, not the bare column: a null `tenant_id` compares unequal to every other null under a
+  unique index, so a tenantless queue would lose its dedupe entirely — all tenantless rows share
+  one namespace instead, which is what they had before tenancy existed.
+
+  Three places have to agree and a test pins them together: the index, the conflict target, and the
+  live-row lookup. The conflict target must spell the index EXPRESSION exactly —
+  `(name, (coalesce(tenant_id, '')), idempotency_key)` — or Postgres cannot infer the index at all.
+  `driver-memory.ts` mirrors it with `(record.tenantId ?? '')`, which is the parity that turned a
+  gap into a confirmed one rather than catching it. `x_jobs` SHIPPED, so the DDL
+  `drop index if exists`es BOTH superseded indexes — each is strictly narrower than its successor,
+  so either left in place would keep enforcing exactly the collision this fixes. The scheduler's
+  occurrence key already prefixes the task name and is unaffected.
 - **`SQL_JOBS_TABLE` is the ONE install point, and every durable table this package owns is in
   it** (`As of 2026-08`): `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_scheduler_state`,
   `x_scheduler_leader`, `x_job_leases`, `x_job_events`. Four of those were subsystems that shipped
@@ -457,8 +476,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
   first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
   way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
-  ONE store, and a per-store id could not tell them apart. `undefined` is unfenced in both, for the
-  reason `release` is optional: a caller holding no token is one written before the fence.
+  ONE store, and a per-store id could not tell them apart. **An absent token is NOT one rule in
+  both**: `createMemoryOutboxStore`'s `owns(id, undefined)` answers `true` unconditionally, so a
+  caller with no token really is unfenced there — while `createPgOutboxStore` substitutes
+  `claimant ?? relayId` into `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED`, both of which
+  carry `and claimed_by = $n`, so a token-less call fences on THIS store's relay id and no-ops
+  against a row some other relay holds. `outbox-pg.ts:159-165` is the honest comment. Neither store
+  refuses such a caller, which is the shared half: a caller holding no token is one written before
+  the fence.
   **`order by staged_at` was not a total order**: every row staged in one transaction shares a
   `staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
   order they publish, differed between two relays and between two runs of one. `, id` fixes it in
@@ -528,8 +553,20 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   `backfill-pass-fixture.ts` raises `BackfillHandleFailure`, a plain `Error` subclass on purpose:
   a backfill `handle` is app code and the pass propagates what it threw, so a framework code there
   would exercise a path no app takes.
-- Suspension is control flow: `StepSuspension` -> `nack({ countsAsAttempt: false })`.
-  Never log it as an error, never let it burn an attempt.
+- **Suspension is control flow, and a SHED is not a suspension** (`As of 2026-08`).
+  `StepSuspension` -> `nack({ countsAsAttempt: false, park: true })`; never log it as an error,
+  never let it burn an attempt. The two facts were ONE flag until 2026-08: a limiter shed and a
+  `job.concurrency` shed both handed the job back with `countsAsAttempt: false`, and both drivers
+  derived `deadLetter ? 'dead' : counts ? 'ready' : 'suspended'` — so a job that is merely WAITING
+  was filed beside a 3-day sleep. `SQL_STATS` and the memory `stats()` then counted it out of
+  `ready` and out of `oldest_ready_ms`, which `worker.ts` publishes as `queue_depth` and
+  `queue_oldest_ready_seconds`: 20 jobs at `concurrency: 10` behind `createLimiter({ global: 1 })`
+  read as a depth of 10 with 19 waiting, and under sustained overload the shed fraction approaches
+  100%, so the HPA signal and the "oldest job older than 5 minutes" page both go quiet exactly
+  when the queue is saturated. `park` is now the state and `countsAsAttempt` is the counter, only.
+  The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
+  that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
+  where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
 - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
 - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
 - Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
@@ -604,7 +641,8 @@ picture from the other side.
 | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
 | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
 | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
-| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements |
+| `backfill-errors.ts` | the seven `X_BACKFILL_*` classes — split out of `errors.ts`, which was over the 500-line ceiling. The codes themselves stay declared in `errors.ts`: one registry, one place |
+| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements. Skips a non-job in silence — a module namespace is full of helpers — EXCEPT an `@ultimat3/action` projection (`kind: 'action-job'`), which is `X_ACTION_JOB_UNBRIDGED` |
 | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
 | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
 | `outbox.ts` | staging in a `Tx`, the relay, the ambient `JobsFacade` slot |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "3.0.0",
+  "version": "4.1.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "3.0.0",
-    "@ultimat3/entity": "3.0.0",
-    "@ultimat3/schema": "3.0.0",
-    "@ultimat3/time": "3.0.0"
+    "@ultimat3/core": "4.1.0",
+    "@ultimat3/entity": "4.1.0",
+    "@ultimat3/schema": "4.1.0",
+    "@ultimat3/time": "4.1.0"
   }
 }

package/src/backfill-errors.ts

@@ -0,0 +1,137 @@
+// The seven `X_BACKFILL_*` codes, apart from `errors.ts` for the reason `driver-pg-rows.ts` is
+// apart from `driver-pg.ts`: one file, one job, and `errors.ts` was over the 500-line ceiling
+// `x verify`'s `filesize` step enforces. The registry stays there — `JOB_OWNED_ERROR_CODES`,
+// `JOB_ERROR_TITLES` and the single `registerErrorCodes()` call — because a package's codes are
+// declared in ONE place; only the classes that throw them live here, beside `backfill-ledger.ts`,
+// `backfill-pending.ts` and `backfill-registry.ts`.
+
+import { UltimateError } from '@ultimat3/core';
+import { docsFor } from './errors';
+
+/**
+ * The seven backfill codes below all answer one question — "why is this sweep not running?" — and
+ * each is here because it sends the reader somewhere different: run it, force it, change
+ * environment, migrate first, wait, fix the predicates, or fix the name. A code that shared a fix
+ * line with another one would be code inflation, which is why `X_BACKFILL_WRITE_UNCONFIRMED` was
+ * considered and rejected: a dry run that wrote nothing did exactly what it was asked to.
+ *
+ * Every `fix` below is ONE line something can execute — a shell command, or the edit to make.
+ * Never a command with prose appended: a `fix:` is copied and run verbatim, so a trailing clause
+ * turns a working command into a syntax error at the one moment the reader is following it
+ * literally. Explanations belong in `cause`, which is read and never run.
+ */
+
+/**
+ * Declared and never completed. The alarm the framework did not have: an author could
+ * `x g backfill`, merge and deploy, and nothing anywhere said the pass had not run.
+ */
+export class BackfillPendingError extends UltimateError {
+  constructor(input: { backfill: string; environment: string }) {
+    super({
+      code: 'X_BACKFILL_PENDING',
+      cause: `backfill "${input.backfill}" is declared and x_backfills holds no completed pass for it in ${input.environment}`,
+      fix: `x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_PENDING'),
+    });
+  }
+}
+
+/** Already swept. A rerun is legitimate, so this names the flag rather than refusing outright. */
+export class BackfillAppliedError extends UltimateError {
+  constructor(input: { backfill: string; runId: string; completedAt: string }) {
+    super({
+      code: 'X_BACKFILL_APPLIED',
+      cause: `backfill "${input.backfill}" completed as run ${input.runId} at ${input.completedAt}; a forced rerun writes a NEW ledger row and never edits that one`,
+      fix: `x db backfill ${input.backfill} --write --force --json`,
+      docs: docsFor('X_BACKFILL_APPLIED'),
+    });
+  }
+}
+
+/**
+ * The declaration names the environments it belongs to and this is not one. Declared DATA, never a
+ * hardcoded "cleanups are production": a staging rehearsal is correct practice, so which
+ * environments a sweep belongs to is the app's convention and this is only the mechanism carrying
+ * it (axiom 8).
+ */
+export class BackfillEnvironmentError extends UltimateError {
+  constructor(input: { backfill: string; environment: string; declared: readonly string[] }) {
+    // The first declared environment, because the fix has to be ONE runnable line and the list is
+    // ordered by the author. The empty case cannot arise from `checkBackfillEnvironment`, which
+    // treats an empty list as "every environment" — but this constructor is public, so it answers
+    // with the command that lists what IS declared rather than an `ULTIMATE_ENV=undefined`.
+    const target = input.declared[0];
+    super({
+      code: 'X_BACKFILL_ENVIRONMENT',
+      cause: `backfill "${input.backfill}" declares environments: ${input.declared.join(', ')} and this process resolved ${input.environment} — add "${input.environment}" to that list if this deploy should sweep too`,
+      fix:
+        target === undefined
+          ? 'x db backfill --pending --json'
+          : `ULTIMATE_ENV=${target} x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_ENVIRONMENT'),
+    });
+  }
+}
+
+/**
+ * `requires` names a migration the ledger has not applied. Checked where `x_migrations` is
+ * readable — this package holds no `@ultimat3/db` dependency and growing one to read a ledger
+ * would put the migration engine on the tier-3 queue's import graph.
+ */
+export class BackfillMigrationPendingError extends UltimateError {
+  constructor(input: { backfill: string; migration: string }) {
+    super({
+      code: 'X_BACKFILL_MIGRATION_PENDING',
+      cause: `backfill "${input.backfill}" requires migration ${input.migration}, which x_migrations does not record as applied`,
+      fix: 'x db migrate --json',
+      docs: docsFor('X_BACKFILL_MIGRATION_PENDING'),
+    });
+  }
+}
+
+/**
+ * The enqueue deduped: one live pass per name, so this run is the one already going. Distinct from
+ * `X_BACKFILL_APPLIED`, which is a pass that finished — the response there is `--force`, and the
+ * response here is to look at the run that is holding the key.
+ */
+export class BackfillRunningError extends UltimateError {
+  constructor(input: { backfill: string; jobId: string }) {
+    super({
+      code: 'X_BACKFILL_RUNNING',
+      cause: `backfill "${input.backfill}" already has a live pass queued as ${input.jobId}, and one name holds one live pass; its step trace names the batch it is on, and a pass that is not advancing is a worker that lost its lease`,
+      fix: `x jobs show ${input.jobId} --json`,
+      docs: docsFor('X_BACKFILL_RUNNING'),
+    });
+  }
+}
+
+/**
+ * The source ran out of rows and the declaration's own `count()` still matches some. Two
+ * predicates that disagree is an authoring bug in any business — the sweep reported success over
+ * rows it never visited — so the pass fails rather than writing a completed row nobody can trust.
+ */
+export class BackfillStalledError extends UltimateError {
+  constructor(input: { backfill: string; remaining: number; swept: number }) {
+    super({
+      code: 'X_BACKFILL_STALLED',
+      cause: `backfill "${input.backfill}" swept ${input.swept} rows, exhausted its source, and count() still matches ${input.remaining} — a WHERE the sweep narrows and the count does not is what leaves rows behind`,
+      fix: `make count() select on exactly what source() selects on in backfill("${input.backfill}")`,
+      docs: docsFor('X_BACKFILL_STALLED'),
+    });
+  }
+}
+
+/** A name no declaration in this app carries — a typo, or a backfill whose module was deleted. */
+export class BackfillUnknownError extends UltimateError {
+  constructor(input: { backfill: string; known: readonly string[] }) {
+    super({
+      code: 'X_BACKFILL_UNKNOWN',
+      cause:
+        input.known.length === 0
+          ? `no backfill named "${input.backfill}" is declared, and this app declares none at all`
+          : `no backfill named "${input.backfill}" is declared (declared: ${input.known.join(', ')})`,
+      fix: 'x db backfill --pending --json',
+      docs: docsFor('X_BACKFILL_UNKNOWN'),
+    });
+  }
+}

package/src/backfill-gate.ts

@@ -8,15 +8,15 @@
 // convention rather than a rail (axiom 3).
 
 import type { Environment, UltimateError } from '@ultimat3/core';
-// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
-// own row shape, which would make this a second reader of `x_backfills`.
-import type { BackfillProgress } from './backfill-inspect';
-import type { BackfillDeclaration } from './backfill-registry';
 import {
   BackfillAppliedError,
   BackfillEnvironmentError,
   BackfillMigrationPendingError,
-} from './errors';
+} from './backfill-errors';
+// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
+// own row shape, which would make this a second reader of `x_backfills`.
+import type { BackfillProgress } from './backfill-inspect';
+import type { BackfillDeclaration } from './backfill-registry';
 
 export type BackfillGate =
   | { readonly run: true }

package/src/backfill-pass.ts

@@ -16,13 +16,13 @@
 import { appVersion, assert, logger, resolveEnvironment } from '@ultimat3/core';
 import type { BatchIterator } from '@ultimat3/entity';
 import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill';
+import { BackfillStalledError } from './backfill-errors';
 import { checkBackfillEnvironment } from './backfill-gate';
 import type { BackfillLedger, BackfillRun } from './backfill-ledger';
 import { decideBackfill } from './backfill-ledger';
 import type { Pacer } from './backfill-rate';
 import { withBackfillScope } from './backfill-scope';
 import { jobDriver } from './driver';
-import { BackfillStalledError } from './errors';
 import type { JobRunArgs } from './job';
 import { isStepSuspension } from './steps';
 

package/src/describe.ts

@@ -14,10 +14,18 @@ export interface JobDescriptor {
   readonly queue: string;
   readonly retry: { readonly attempts: number; readonly backoff: BackoffStrategy };
   readonly steps: readonly string[];
+  /**
+   * Whether a replayed attempt is safe to run — `job()` REQUIRES an `idempotencyKey` and refuses
+   * a definition without one (`X_IDEMPOTENCY_REQUIRED`), so this is `true` for every registered
+   * job. That is the point: the guarantee, published where an operator asks the question, rather
+   * than left as prose in a doc. The KEY itself never crosses — it is computed from an input and
+   * is app data, so a descriptor carrying it would put customer ids in `x.manifest.json`.
+   */
+  readonly idempotent: boolean;
 }
 
 /**
- * Narrower than `JobHandle` on purpose: the projection reads four declared fields, so keeping
+ * Narrower than `JobHandle` on purpose: the projection reads five declared fields, so keeping
  * it structural means it never has to carry — or vary with — the handle's input generic.
  */
 export interface DescribableJob {
@@ -25,6 +33,13 @@ export interface DescribableJob {
   readonly queue: string;
   readonly retry: RetryPolicy;
   readonly input: unknown;
+  /**
+   * `JobHandle.idempotencyKeyFor`, read only for its presence — `unknown` because the real
+   * signature is `(input: I) => string` and this shape is deliberately free of the generic.
+   * Required, not optional: a descriptor built without it would publish `idempotent: false`,
+   * which is the exact wrong answer the `/_x` jobs panel used to give for every job.
+   */
+  readonly idempotencyKeyFor: unknown;
 }
 
 export function describeJob(handle: DescribableJob): JobDescriptor {
@@ -39,6 +54,7 @@ export function describeJob(handle: DescribableJob): JobDescriptor {
     // Empty by design: step names are chosen inside `run()` at execution time, so they are
     // not statically knowable. `inspect(name)` reports the steps an actual run recorded.
     steps: [],
+    idempotent: typeof handle.idempotencyKeyFor === 'function',
   };
 }
 

package/src/driver-memory.ts

@@ -38,7 +38,18 @@ export interface MemoryDriverOptions {
 
 const LIVE_STATES = new Set(['ready', 'delayed', 'running', 'suspended']);
 
-export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver {
+/**
+ * The in-memory driver's own type: `JobDriver` with `close` REQUIRED.
+ *
+ * `JobDriver.close` is optional because a driver may hold nothing to release. This one always
+ * does — it clears the job map — and every wrapper in the test suite delegates through
+ * `base.close()`. Declaring it here is what makes that delegation a CHECKED call: against a plain
+ * `JobDriver` the only way to write it is `base.close?.()`, which a driver that quietly stopped
+ * shipping a `close` would satisfy in silence.
+ */
+export type MemoryJobDriver = JobDriver & { close(): Promise<void> };
+
+export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJobDriver {
   const clock = options.clock ?? systemClock;
   const steps = options.steps ?? createMemoryStepStore();
   const backfills = options.backfills ?? createMemoryBackfillLedger(clock);
@@ -46,12 +57,20 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
     options.leases ?? createMemoryLeaseStore(options.clock === undefined ? {} : { clock });
   const jobs = new Map<string, JobRecord>();
 
-  // Keyed by NAME and key, exactly as `x_jobs_name_idempotency_live_idx` is. A global key
-  // namespace let two unrelated jobs that derived the same natural key dedupe against each
-  // other: the second enqueue returned the first's id and its work never ran.
-  const liveByKey = (name: string, key: string): JobRecord | undefined => {
+  // Keyed by NAME, TENANT and key, exactly as `x_jobs_name_tenant_idempotency_live_idx` is. A
+  // global key namespace let two unrelated jobs that derived the same natural key dedupe against
+  // each other: the second enqueue returned the first's id and its work never ran. A tenant-blind
+  // one did the same ACROSS tenants, where the id handed back belongs to somebody else and is
+  // valid on every id-addressed surface. `?? ''` mirrors the index's `coalesce`, so all tenantless
+  // rows share one namespace rather than each becoming its own.
+  const liveByKey = (name: string, key: string, tenantId?: string): JobRecord | undefined => {
     for (const record of jobs.values()) {
-      if (record.name === name && record.idempotencyKey === key && LIVE_STATES.has(record.state)) {
+      if (
+        record.name === name &&
+        record.idempotencyKey === key &&
+        (record.tenantId ?? '') === (tenantId ?? '') &&
+        LIVE_STATES.has(record.state)
+      ) {
         return record;
       }
     }
@@ -124,7 +143,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
     introspect,
 
     enqueue(request: EnqueueRequest): Promise<EnqueueResult> {
-      const existing = liveByKey(request.name, request.idempotencyKey);
+      const existing = liveByKey(request.name, request.idempotencyKey, request.tenantId);
       if (existing !== undefined) {
         if (request.onConflict === 'error') {
           throw new JobDuplicateError({
@@ -206,7 +225,14 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
       const at = nowMs(clock);
       const counts = nackOptions.countsAsAttempt !== false;
       const patch: Partial<JobRecord> = {
-        state: nackOptions.deadLetter === true ? 'dead' : counts ? 'ready' : 'suspended',
+        // `park`, never `counts`: parking is what leaves the ready bucket, and burning an attempt
+        // is a separate fact. A shed sets neither and stays `ready`, which is what it is.
+        state:
+          nackOptions.deadLetter === true
+            ? 'dead'
+            : nackOptions.park === true
+              ? 'suspended'
+              : 'ready',
         runAt: at + nackOptions.delayMs,
         // A suspension must not burn an attempt, or a 3-day sleep dead-letters the run. Floored
         // where `SQL_NACK` floors it (`greatest(attempt - 1, 0)`): the fence above is what keeps

package/src/driver-nats.ts

@@ -16,10 +16,13 @@ import type {
 import { JobsNotImplementedError } from './errors';
 import type { StepRecord, StepStore } from './steps';
 
-// Names the config edit that actually removes the stub, plus the runnable command for whatever
-// is already queued. The nats driver lands in v2; there is no flag that turns this one on.
+// Names the seam that actually replaces the stub, plus the runnable command for whatever is
+// already queued. NOT `jobs: { driver }` in app.config.ts, which this line said until 2026-08-20:
+// `JobsConfig.driver` has no reader anywhere (see `driver.ts`'s header), so that edit repairs
+// nothing and the reader is sent back to the same throw. #223 removes the field.
+// The nats driver lands in v2; there is no flag that turns this one on.
 const FIX =
-  "set jobs: { driver: 'postgres' } in app.config.ts, then: x jobs drain --to memory --json";
+  'call setJobDriver(createPgDriver()) at boot instead of this driver, then move what is already queued: x jobs drain --to memory --json';
 
 const unavailable = (method: string): never => {
   throw new JobsNotImplementedError({ feature: `nats jobs driver (${method})`, fix: FIX });

package/src/driver-pg-ddl.ts

@@ -39,19 +39,33 @@ alter table x_jobs add column if not exists traceparent text;
 
 alter table x_jobs add column if not exists enqueued_by text;
 
--- Partial unique index: one LIVE job per (name, idempotency key). Completed rows stay for
+-- Partial unique index: one LIVE job per (name, tenant, idempotency key). Completed rows stay for
 -- history, so re-running the same work tomorrow is allowed and re-delivering it today is not.
 --
 -- The NAME is in the key, and its absence was silent data loss: two jobs that happened to derive
 -- the same natural key from the same input ("user:42") shared one namespace, so the second
 -- enqueue deduped against the FIRST jobs row and returned its id. The work never ran, no error
--- was raised, and the queue showed one healthy job. The old index is dropped rather than left
--- beside the new one — it is strictly narrower, so keeping it would keep enforcing exactly the
--- collision this fixes.
+-- was raised, and the queue showed one healthy job.
+--
+-- The TENANT is in the key for that same argument with one word substituted, and it is the worse
+-- half. Every natural key an app writes is unique only WITHIN a tenant, invoice:1001 and
+-- order:5540 being the shapes the docs suggest, so tenant B enqueuing while tenant A held that
+-- key deduped into tenant A row: tenant B work never ran, and tenant B caller received tenant A
+-- job id, which is valid on every id-addressed surface. Cancel takes an id with no tenant
+-- predicate.
+--
+-- coalesce rather than the bare column, because a null tenant_id compares unequal to every other
+-- null under a unique index and a tenantless queue would lose its dedupe entirely. All tenantless
+-- rows share one namespace instead, which is exactly what they had before tenancy existed.
+--
+-- Each superseded index is dropped rather than left beside the new one. Each is strictly narrower,
+-- so keeping it would keep enforcing exactly the collision this fixes.
 drop index if exists x_jobs_idempotency_live_idx;
 
-create unique index if not exists x_jobs_name_idempotency_live_idx
-  on x_jobs (name, idempotency_key)
+drop index if exists x_jobs_name_idempotency_live_idx;
+
+create unique index if not exists x_jobs_name_tenant_idempotency_live_idx
+  on x_jobs (name, (coalesce(tenant_id, '')), idempotency_key)
   where state in ('ready', 'delayed', 'running', 'suspended');
 
 create index if not exists x_jobs_claim_idx

package/src/driver-pg-rows.ts

@@ -3,9 +3,10 @@
 // decoding a row is not control flow — every number arrives as `number | string` (a bigint is a
 // string in every client) and every absent column as `null`, and that translation is its own job.
 
-import type { BackfillRun, BackfillStatus } from './backfill-ledger';
-import type { JobRecord } from './driver';
-import type { StepRecord } from './steps';
+import { BACKFILL_STATUSES, type BackfillRun, isBackfillStatus } from './backfill-ledger';
+import { isJobState, JOB_STATES, type JobRecord, type JobState } from './driver';
+import { JobRowStatusUnknownError } from './errors';
+import { isStepStatus, STEP_STATUSES, type StepRecord, type StepStatus } from './steps';
 
 export interface StepRow {
   readonly run_id: string;
@@ -54,6 +55,21 @@ export interface JobRow {
   readonly enqueued_by?: string | null;
 }
 
+/**
+ * The one narrowing for all three status columns. `as` is not a check, and these rows cross a
+ * process boundary — a queue row was written by whatever build was deployed when the job was
+ * enqueued, which on a rolling deploy is not this one. `isBackfillStatus`'s own doc already stated
+ * the rule ("Never a cast — the list decides") and this file was the caller ignoring it.
+ */
+const statusIn = <T extends string>(
+  known: readonly T[],
+  is: (value: string) => value is T,
+  input: { table: string; column: string; value: string },
+): T => {
+  if (is(input.value)) return input.value;
+  throw new JobRowStatusUnknownError({ ...input, known });
+};
+
 export const num = (value: number | string | null | undefined): number =>
   value === null || value === undefined ? 0 : Number(value);
 
@@ -71,7 +87,11 @@ export function toJobRecord(row: JobRow): JobRecord {
     runId: row.run_id,
     attempt: row.attempt,
     maxAttempts: row.max_attempts,
-    state: row.state as JobRecord['state'],
+    state: statusIn<JobState>(JOB_STATES, isJobState, {
+      table: 'ultimate_jobs',
+      column: 'state',
+      value: row.state,
+    }),
     runAt: num(row.run_at),
     createdAt: num(row.created_at),
     updatedAt: num(row.updated_at),
@@ -94,7 +114,11 @@ export function toStepRecord(row: StepRow): StepRecord {
   return {
     runId: row.run_id,
     name: row.name,
-    status: row.status as StepRecord['status'],
+    status: statusIn<StepStatus>(STEP_STATUSES, isStepStatus, {
+      table: 'ultimate_job_steps',
+      column: 'status',
+      value: row.status,
+    }),
     output: row.output,
     startedAt: num(row.started_at),
     attempts: row.attempts,
@@ -112,7 +136,11 @@ export function toBackfillRun(row: BackfillRow): BackfillRun {
     runId: row.run_id,
     name: row.name,
     checksum: row.checksum,
-    status: row.status as BackfillStatus,
+    status: statusIn(BACKFILL_STATUSES, isBackfillStatus, {
+      table: 'ultimate_backfills',
+      column: 'status',
+      value: row.status,
+    }),
     appVersion: row.app_version,
     // `rows_processed` is a bigint, which every Postgres client hands back as a string.
     rows: num(row.rows_processed),

package/src/driver-pg-sql.ts

@@ -17,17 +17,26 @@ values
   ($1, $2, $3, $4::jsonb, $5, $6, $7,
    case when to_timestamp($8 / 1000.0) > now() then 'delayed' else 'ready' end,
    to_timestamp($8 / 1000.0), $9, $10, $11)
-on conflict (name, idempotency_key)
+on conflict (name, (coalesce(tenant_id, '')), idempotency_key)
   where state in ('ready', 'delayed', 'running', 'suspended')
   do nothing
 returning id, run_id
 `.trim();
 
-/** Scoped by NAME as well as key — the index is, so a lookup that was not would find a stranger. */
+/**
+ * Scoped by NAME and TENANT as well as key, because the index is — a lookup narrower than the
+ * index it reads answers with whichever stranger holds the key, and `{ deduped: true, id: <another
+ * tenant's> }` is a cross-tenant handle, not just a missed run.
+ *
+ * `coalesce` on both sides, matching the index expression exactly: a bare `tenant_id = $3` never
+ * matches a null row, so every tenantless enqueue would fall through to the "rejected but no live
+ * row holds its idempotency key" refusal instead of finding its own live job.
+ */
 export const SQL_FIND_LIVE_BY_KEY = `
 select id, run_id from x_jobs
  where name = $1
    and idempotency_key = $2
+   and coalesce(tenant_id, '') = coalesce($3::text, '')
    and state in ('ready', 'delayed', 'running', 'suspended')
  limit 1
 `.trim();

package/src/driver-pg.ts

@@ -271,19 +271,22 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
         return { id: inserted.id, runId: inserted.run_id, deduped: false };
       }
 
-      // `do nothing` fired: a live job OF THIS NAME already owns this idempotency key. The name
-      // is in the lookup because it is in the index — without it this returned whichever other
-      // job happened to derive the same natural key, and the caller's work silently never ran.
+      // `do nothing` fired: a live job OF THIS NAME, IN THIS TENANT, already owns this idempotency
+      // key. Both are in the lookup because both are in the index — without the name this returned
+      // whichever other job derived the same natural key; without the tenant it returned another
+      // TENANT's row, so the caller's work silently never ran AND the caller was handed an id it
+      // has no right to, on a surface (`cancel`) that takes an id with no tenant predicate.
       const existing = await exec().query<{ id: string; run_id: string }>(SQL_FIND_LIVE_BY_KEY, [
         request.name,
         request.idempotencyKey,
+        request.tenantId ?? null,
       ]);
       const found = existing[0];
       if (found === undefined) {
         throw new DriverUnavailableError({
           driver: 'pg',
           cause: `enqueue of "${request.name}" was rejected but no live row holds its idempotency key`,
-          fix: 'x db migrate   # reapplies SQL_JOBS_TABLE, whose x_jobs_name_idempotency_live_idx is what this lookup reads',
+          fix: 'x db migrate   # reapplies SQL_JOBS_TABLE, whose x_jobs_name_tenant_idempotency_live_idx is what this lookup reads',
         });
       }
       if (request.onConflict === 'error') {
@@ -321,7 +324,14 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
 
     async nack(jobId: string, nackOptions: NackOptions): Promise<void> {
       const counts = nackOptions.countsAsAttempt !== false;
-      const state = nackOptions.deadLetter === true ? 'dead' : counts ? 'ready' : 'suspended';
+      // The same three-way the memory driver takes, and it reads `park` rather than `counts`: the
+      // attempt counter and the ready bucket are two facts, and a shed only ever meant the first.
+      const state =
+        nackOptions.deadLetter === true
+          ? 'dead'
+          : nackOptions.park === true
+            ? 'suspended'
+            : 'ready';
       await exec().query(SQL_NACK, [
         jobId,
         state,

package/src/driver-redis.ts

@@ -19,10 +19,13 @@ import type {
 import { JobsNotImplementedError } from './errors';
 import type { StepRecord, StepStore } from './steps';
 
-// Names the config edit that actually removes the stub, plus the runnable command for whatever
-// is already queued. The redis driver lands in v2; there is no flag that turns this one on.
+// Names the seam that actually replaces the stub, plus the runnable command for whatever is
+// already queued. NOT `jobs: { driver }` in app.config.ts, which this line said until 2026-08-20:
+// `JobsConfig.driver` has no reader anywhere (see `driver.ts`'s header), so that edit repairs
+// nothing and the reader is sent back to the same throw. #223 removes the field.
+// The redis driver lands in v2; there is no flag that turns this one on.
 const FIX =
-  "set jobs: { driver: 'postgres' } in app.config.ts, then: x jobs drain --to memory --json";
+  'call setJobDriver(createPgDriver()) at boot instead of this driver, then move what is already queued: x jobs drain --to memory --json';
 
 const unavailable = (method: string): never => {
   throw new JobsNotImplementedError({ feature: `redis jobs driver (${method})`, fix: FIX });