@ultimat3/jobs 3.0.0 → 4.0.0

package/CLAUDE.md

@@ -35,16 +35,35 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   A look-alike never registers. Deliberately not a registry lookup — the registry is what
   registration rewrites.
 - `idempotencyKey` is NON-OPTIONAL in `JobDefinition`. Never relax it, never default it.
-- **The idempotency namespace is `(name, idempotency_key)`, never the key alone** (`As of
-  2026-08`). It was the key alone, and that is silent data loss with no error anywhere: two jobs
-  that derive the same natural key from the same input — `sendWelcomeEmail` and
-  `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so the second enqueue hit
-  `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found the FIRST job's row and
-  returned `{ id: <A's>, deduped: true }`. The workspace was never provisioned and `x jobs ls`
-  showed one healthy job. Three places have to agree and a test pins them together: the index, the
-  conflict target, and the live-row lookup. `x_jobs` SHIPPED, so the DDL `drop index if exists`es
-  the old one — left in place it would keep enforcing exactly the collision this fixes. The
-  scheduler's occurrence key already prefixes the task name and is unaffected.
+- **The idempotency namespace is `(name, coalesce(tenant_id, ''), idempotency_key)`, never the key
+  alone and never name-only** (`As of 2026-08`). Two rounds of the same defect, and both are silent
+  data loss with no error anywhere.
+
+  It was the key alone: two jobs deriving the same natural key from the same input —
+  `sendWelcomeEmail` and `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so
+  the second enqueue hit `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found
+  the FIRST job's row and returned `{ id: <A's>, deduped: true }`. The workspace was never
+  provisioned and `x jobs ls` showed one healthy job.
+
+  Then it was name-only, while the row already carried `tenant_id` as `$9` of the same insert.
+  Every natural key the docs suggest is unique only WITHIN a tenant — `` `invoice:${input.invoiceId}` ``,
+  `` `order:${input.orderNumber}` `` — so tenant B enqueuing while tenant A held that key deduped
+  into tenant A's row: B's work never ran AND B's caller received A's job id, which is valid on
+  every id-addressed surface (`cancelJob(driver, jobId)` takes an id with no tenant predicate, so
+  an app wiring the returned id to a cancel button gave B cancellation of A's job). The sibling
+  projection in `@ultimat3/action` (`idempotency-key.ts`) had folded the actor's org in all along.
+  `coalesce`, not the bare column: a null `tenant_id` compares unequal to every other null under a
+  unique index, so a tenantless queue would lose its dedupe entirely — all tenantless rows share
+  one namespace instead, which is what they had before tenancy existed.
+
+  Three places have to agree and a test pins them together: the index, the conflict target, and the
+  live-row lookup. The conflict target must spell the index EXPRESSION exactly —
+  `(name, (coalesce(tenant_id, '')), idempotency_key)` — or Postgres cannot infer the index at all.
+  `driver-memory.ts` mirrors it with `(record.tenantId ?? '')`, which is the parity that turned a
+  gap into a confirmed one rather than catching it. `x_jobs` SHIPPED, so the DDL
+  `drop index if exists`es BOTH superseded indexes — each is strictly narrower than its successor,
+  so either left in place would keep enforcing exactly the collision this fixes. The scheduler's
+  occurrence key already prefixes the task name and is unaffected.
 - **`SQL_JOBS_TABLE` is the ONE install point, and every durable table this package owns is in
   it** (`As of 2026-08`): `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_scheduler_state`,
   `x_scheduler_leader`, `x_job_leases`, `x_job_events`. Four of those were subsystems that shipped
@@ -457,8 +476,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
   first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
   way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
-  ONE store, and a per-store id could not tell them apart. `undefined` is unfenced in both, for the
-  reason `release` is optional: a caller holding no token is one written before the fence.
+  ONE store, and a per-store id could not tell them apart. **An absent token is NOT one rule in
+  both**: `createMemoryOutboxStore`'s `owns(id, undefined)` answers `true` unconditionally, so a
+  caller with no token really is unfenced there — while `createPgOutboxStore` substitutes
+  `claimant ?? relayId` into `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED`, both of which
+  carry `and claimed_by = $n`, so a token-less call fences on THIS store's relay id and no-ops
+  against a row some other relay holds. `outbox-pg.ts:159-165` is the honest comment. Neither store
+  refuses such a caller, which is the shared half: a caller holding no token is one written before
+  the fence.
   **`order by staged_at` was not a total order**: every row staged in one transaction shares a
   `staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
   order they publish, differed between two relays and between two runs of one. `, id` fixes it in
@@ -528,8 +553,20 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   `backfill-pass-fixture.ts` raises `BackfillHandleFailure`, a plain `Error` subclass on purpose:
   a backfill `handle` is app code and the pass propagates what it threw, so a framework code there
   would exercise a path no app takes.
-- Suspension is control flow: `StepSuspension` -> `nack({ countsAsAttempt: false })`.
-  Never log it as an error, never let it burn an attempt.
+- **Suspension is control flow, and a SHED is not a suspension** (`As of 2026-08`).
+  `StepSuspension` -> `nack({ countsAsAttempt: false, park: true })`; never log it as an error,
+  never let it burn an attempt. The two facts were ONE flag until 2026-08: a limiter shed and a
+  `job.concurrency` shed both handed the job back with `countsAsAttempt: false`, and both drivers
+  derived `deadLetter ? 'dead' : counts ? 'ready' : 'suspended'` — so a job that is merely WAITING
+  was filed beside a 3-day sleep. `SQL_STATS` and the memory `stats()` then counted it out of
+  `ready` and out of `oldest_ready_ms`, which `worker.ts` publishes as `queue_depth` and
+  `queue_oldest_ready_seconds`: 20 jobs at `concurrency: 10` behind `createLimiter({ global: 1 })`
+  read as a depth of 10 with 19 waiting, and under sustained overload the shed fraction approaches
+  100%, so the HPA signal and the "oldest job older than 5 minutes" page both go quiet exactly
+  when the queue is saturated. `park` is now the state and `countsAsAttempt` is the counter, only.
+  The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
+  that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
+  where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
 - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
 - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
 - Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
@@ -604,7 +641,8 @@ picture from the other side.
 | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
 | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
 | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
-| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements |
+| `backfill-errors.ts` | the seven `X_BACKFILL_*` classes — split out of `errors.ts`, which was over the 500-line ceiling. The codes themselves stay declared in `errors.ts`: one registry, one place |
+| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements. Skips a non-job in silence — a module namespace is full of helpers — EXCEPT an `@ultimat3/action` projection (`kind: 'action-job'`), which is `X_ACTION_JOB_UNBRIDGED` |
 | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
 | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
 | `outbox.ts` | staging in a `Tx`, the relay, the ambient `JobsFacade` slot |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "3.0.0",
+  "version": "4.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "3.0.0",
-    "@ultimat3/entity": "3.0.0",
-    "@ultimat3/schema": "3.0.0",
-    "@ultimat3/time": "3.0.0"
+    "@ultimat3/core": "4.0.0",
+    "@ultimat3/entity": "4.0.0",
+    "@ultimat3/schema": "4.0.0",
+    "@ultimat3/time": "4.0.0"
   }
 }

package/src/backfill-errors.ts

@@ -0,0 +1,137 @@
+// The seven `X_BACKFILL_*` codes, apart from `errors.ts` for the reason `driver-pg-rows.ts` is
+// apart from `driver-pg.ts`: one file, one job, and `errors.ts` was over the 500-line ceiling
+// `x verify`'s `filesize` step enforces. The registry stays there — `JOB_OWNED_ERROR_CODES`,
+// `JOB_ERROR_TITLES` and the single `registerErrorCodes()` call — because a package's codes are
+// declared in ONE place; only the classes that throw them live here, beside `backfill-ledger.ts`,
+// `backfill-pending.ts` and `backfill-registry.ts`.
+
+import { UltimateError } from '@ultimat3/core';
+import { docsFor } from './errors';
+
+/**
+ * The seven backfill codes below all answer one question — "why is this sweep not running?" — and
+ * each is here because it sends the reader somewhere different: run it, force it, change
+ * environment, migrate first, wait, fix the predicates, or fix the name. A code that shared a fix
+ * line with another one would be code inflation, which is why `X_BACKFILL_WRITE_UNCONFIRMED` was
+ * considered and rejected: a dry run that wrote nothing did exactly what it was asked to.
+ *
+ * Every `fix` below is ONE line something can execute — a shell command, or the edit to make.
+ * Never a command with prose appended: a `fix:` is copied and run verbatim, so a trailing clause
+ * turns a working command into a syntax error at the one moment the reader is following it
+ * literally. Explanations belong in `cause`, which is read and never run.
+ */
+
+/**
+ * Declared and never completed. The alarm the framework did not have: an author could
+ * `x g backfill`, merge and deploy, and nothing anywhere said the pass had not run.
+ */
+export class BackfillPendingError extends UltimateError {
+  constructor(input: { backfill: string; environment: string }) {
+    super({
+      code: 'X_BACKFILL_PENDING',
+      cause: `backfill "${input.backfill}" is declared and x_backfills holds no completed pass for it in ${input.environment}`,
+      fix: `x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_PENDING'),
+    });
+  }
+}
+
+/** Already swept. A rerun is legitimate, so this names the flag rather than refusing outright. */
+export class BackfillAppliedError extends UltimateError {
+  constructor(input: { backfill: string; runId: string; completedAt: string }) {
+    super({
+      code: 'X_BACKFILL_APPLIED',
+      cause: `backfill "${input.backfill}" completed as run ${input.runId} at ${input.completedAt}; a forced rerun writes a NEW ledger row and never edits that one`,
+      fix: `x db backfill ${input.backfill} --write --force --json`,
+      docs: docsFor('X_BACKFILL_APPLIED'),
+    });
+  }
+}
+
+/**
+ * The declaration names the environments it belongs to and this is not one. Declared DATA, never a
+ * hardcoded "cleanups are production": a staging rehearsal is correct practice, so which
+ * environments a sweep belongs to is the app's convention and this is only the mechanism carrying
+ * it (axiom 8).
+ */
+export class BackfillEnvironmentError extends UltimateError {
+  constructor(input: { backfill: string; environment: string; declared: readonly string[] }) {
+    // The first declared environment, because the fix has to be ONE runnable line and the list is
+    // ordered by the author. The empty case cannot arise from `checkBackfillEnvironment`, which
+    // treats an empty list as "every environment" — but this constructor is public, so it answers
+    // with the command that lists what IS declared rather than an `ULTIMATE_ENV=undefined`.
+    const target = input.declared[0];
+    super({
+      code: 'X_BACKFILL_ENVIRONMENT',
+      cause: `backfill "${input.backfill}" declares environments: ${input.declared.join(', ')} and this process resolved ${input.environment} — add "${input.environment}" to that list if this deploy should sweep too`,
+      fix:
+        target === undefined
+          ? 'x db backfill --pending --json'
+          : `ULTIMATE_ENV=${target} x db backfill ${input.backfill} --write --json`,
+      docs: docsFor('X_BACKFILL_ENVIRONMENT'),
+    });
+  }
+}
+
+/**
+ * `requires` names a migration the ledger has not applied. Checked where `x_migrations` is
+ * readable — this package holds no `@ultimat3/db` dependency and growing one to read a ledger
+ * would put the migration engine on the tier-3 queue's import graph.
+ */
+export class BackfillMigrationPendingError extends UltimateError {
+  constructor(input: { backfill: string; migration: string }) {
+    super({
+      code: 'X_BACKFILL_MIGRATION_PENDING',
+      cause: `backfill "${input.backfill}" requires migration ${input.migration}, which x_migrations does not record as applied`,
+      fix: 'x db migrate --json',
+      docs: docsFor('X_BACKFILL_MIGRATION_PENDING'),
+    });
+  }
+}
+
+/**
+ * The enqueue deduped: one live pass per name, so this run is the one already going. Distinct from
+ * `X_BACKFILL_APPLIED`, which is a pass that finished — the response there is `--force`, and the
+ * response here is to look at the run that is holding the key.
+ */
+export class BackfillRunningError extends UltimateError {
+  constructor(input: { backfill: string; jobId: string }) {
+    super({
+      code: 'X_BACKFILL_RUNNING',
+      cause: `backfill "${input.backfill}" already has a live pass queued as ${input.jobId}, and one name holds one live pass; its step trace names the batch it is on, and a pass that is not advancing is a worker that lost its lease`,
+      fix: `x jobs show ${input.jobId} --json`,
+      docs: docsFor('X_BACKFILL_RUNNING'),
+    });
+  }
+}
+
+/**
+ * The source ran out of rows and the declaration's own `count()` still matches some. Two
+ * predicates that disagree is an authoring bug in any business — the sweep reported success over
+ * rows it never visited — so the pass fails rather than writing a completed row nobody can trust.
+ */
+export class BackfillStalledError extends UltimateError {
+  constructor(input: { backfill: string; remaining: number; swept: number }) {
+    super({
+      code: 'X_BACKFILL_STALLED',
+      cause: `backfill "${input.backfill}" swept ${input.swept} rows, exhausted its source, and count() still matches ${input.remaining} — a WHERE the sweep narrows and the count does not is what leaves rows behind`,
+      fix: `make count() select on exactly what source() selects on in backfill("${input.backfill}")`,
+      docs: docsFor('X_BACKFILL_STALLED'),
+    });
+  }
+}
+
+/** A name no declaration in this app carries — a typo, or a backfill whose module was deleted. */
+export class BackfillUnknownError extends UltimateError {
+  constructor(input: { backfill: string; known: readonly string[] }) {
+    super({
+      code: 'X_BACKFILL_UNKNOWN',
+      cause:
+        input.known.length === 0
+          ? `no backfill named "${input.backfill}" is declared, and this app declares none at all`
+          : `no backfill named "${input.backfill}" is declared (declared: ${input.known.join(', ')})`,
+      fix: 'x db backfill --pending --json',
+      docs: docsFor('X_BACKFILL_UNKNOWN'),
+    });
+  }
+}

package/src/backfill-gate.ts

@@ -8,15 +8,15 @@
 // convention rather than a rail (axiom 3).
 
 import type { Environment, UltimateError } from '@ultimat3/core';
-// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
-// own row shape, which would make this a second reader of `x_backfills`.
-import type { BackfillProgress } from './backfill-inspect';
-import type { BackfillDeclaration } from './backfill-registry';
 import {
   BackfillAppliedError,
   BackfillEnvironmentError,
   BackfillMigrationPendingError,
-} from './errors';
+} from './backfill-errors';
+// `BackfillProgress`, the one ledger projection every surface already reads — never the driver's
+// own row shape, which would make this a second reader of `x_backfills`.
+import type { BackfillProgress } from './backfill-inspect';
+import type { BackfillDeclaration } from './backfill-registry';
 
 export type BackfillGate =
   | { readonly run: true }

package/src/backfill-pass.ts

@@ -16,13 +16,13 @@
 import { appVersion, assert, logger, resolveEnvironment } from '@ultimat3/core';
 import type { BatchIterator } from '@ultimat3/entity';
 import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill';
+import { BackfillStalledError } from './backfill-errors';
 import { checkBackfillEnvironment } from './backfill-gate';
 import type { BackfillLedger, BackfillRun } from './backfill-ledger';
 import { decideBackfill } from './backfill-ledger';
 import type { Pacer } from './backfill-rate';
 import { withBackfillScope } from './backfill-scope';
 import { jobDriver } from './driver';
-import { BackfillStalledError } from './errors';
 import type { JobRunArgs } from './job';
 import { isStepSuspension } from './steps';
 

package/src/describe.ts

@@ -14,10 +14,18 @@ export interface JobDescriptor {
   readonly queue: string;
   readonly retry: { readonly attempts: number; readonly backoff: BackoffStrategy };
   readonly steps: readonly string[];
+  /**
+   * Whether a replayed attempt is safe to run — `job()` REQUIRES an `idempotencyKey` and refuses
+   * a definition without one (`X_IDEMPOTENCY_REQUIRED`), so this is `true` for every registered
+   * job. That is the point: the guarantee, published where an operator asks the question, rather
+   * than left as prose in a doc. The KEY itself never crosses — it is computed from an input and
+   * is app data, so a descriptor carrying it would put customer ids in `x.manifest.json`.
+   */
+  readonly idempotent: boolean;
 }
 
 /**
- * Narrower than `JobHandle` on purpose: the projection reads four declared fields, so keeping
+ * Narrower than `JobHandle` on purpose: the projection reads five declared fields, so keeping
  * it structural means it never has to carry — or vary with — the handle's input generic.
  */
 export interface DescribableJob {
@@ -25,6 +33,13 @@ export interface DescribableJob {
   readonly queue: string;
   readonly retry: RetryPolicy;
   readonly input: unknown;
+  /**
+   * `JobHandle.idempotencyKeyFor`, read only for its presence — `unknown` because the real
+   * signature is `(input: I) => string` and this shape is deliberately free of the generic.
+   * Required, not optional: a descriptor built without it would publish `idempotent: false`,
+   * which is the exact wrong answer the `/_x` jobs panel used to give for every job.
+   */
+  readonly idempotencyKeyFor: unknown;
 }
 
 export function describeJob(handle: DescribableJob): JobDescriptor {
@@ -39,6 +54,7 @@ export function describeJob(handle: DescribableJob): JobDescriptor {
     // Empty by design: step names are chosen inside `run()` at execution time, so they are
     // not statically knowable. `inspect(name)` reports the steps an actual run recorded.
     steps: [],
+    idempotent: typeof handle.idempotencyKeyFor === 'function',
   };
 }
 

package/src/driver-memory.ts

@@ -38,7 +38,18 @@ export interface MemoryDriverOptions {
 
 const LIVE_STATES = new Set(['ready', 'delayed', 'running', 'suspended']);
 
-export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver {
+/**
+ * The in-memory driver's own type: `JobDriver` with `close` REQUIRED.
+ *
+ * `JobDriver.close` is optional because a driver may hold nothing to release. This one always
+ * does — it clears the job map — and every wrapper in the test suite delegates through
+ * `base.close()`. Declaring it here is what makes that delegation a CHECKED call: against a plain
+ * `JobDriver` the only way to write it is `base.close?.()`, which a driver that quietly stopped
+ * shipping a `close` would satisfy in silence.
+ */
+export type MemoryJobDriver = JobDriver & { close(): Promise<void> };
+
+export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJobDriver {
   const clock = options.clock ?? systemClock;
   const steps = options.steps ?? createMemoryStepStore();
   const backfills = options.backfills ?? createMemoryBackfillLedger(clock);
@@ -46,12 +57,20 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
     options.leases ?? createMemoryLeaseStore(options.clock === undefined ? {} : { clock });
   const jobs = new Map<string, JobRecord>();
 
-  // Keyed by NAME and key, exactly as `x_jobs_name_idempotency_live_idx` is. A global key
-  // namespace let two unrelated jobs that derived the same natural key dedupe against each
-  // other: the second enqueue returned the first's id and its work never ran.
-  const liveByKey = (name: string, key: string): JobRecord | undefined => {
+  // Keyed by NAME, TENANT and key, exactly as `x_jobs_name_tenant_idempotency_live_idx` is. A
+  // global key namespace let two unrelated jobs that derived the same natural key dedupe against
+  // each other: the second enqueue returned the first's id and its work never ran. A tenant-blind
+  // one did the same ACROSS tenants, where the id handed back belongs to somebody else and is
+  // valid on every id-addressed surface. `?? ''` mirrors the index's `coalesce`, so all tenantless
+  // rows share one namespace rather than each becoming its own.
+  const liveByKey = (name: string, key: string, tenantId?: string): JobRecord | undefined => {
     for (const record of jobs.values()) {
-      if (record.name === name && record.idempotencyKey === key && LIVE_STATES.has(record.state)) {
+      if (
+        record.name === name &&
+        record.idempotencyKey === key &&
+        (record.tenantId ?? '') === (tenantId ?? '') &&
+        LIVE_STATES.has(record.state)
+      ) {
         return record;
       }
     }
@@ -124,7 +143,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
     introspect,
 
     enqueue(request: EnqueueRequest): Promise<EnqueueResult> {
-      const existing = liveByKey(request.name, request.idempotencyKey);
+      const existing = liveByKey(request.name, request.idempotencyKey, request.tenantId);
       if (existing !== undefined) {
         if (request.onConflict === 'error') {
           throw new JobDuplicateError({
@@ -206,7 +225,14 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): JobDriver
       const at = nowMs(clock);
       const counts = nackOptions.countsAsAttempt !== false;
       const patch: Partial<JobRecord> = {
-        state: nackOptions.deadLetter === true ? 'dead' : counts ? 'ready' : 'suspended',
+        // `park`, never `counts`: parking is what leaves the ready bucket, and burning an attempt
+        // is a separate fact. A shed sets neither and stays `ready`, which is what it is.
+        state:
+          nackOptions.deadLetter === true
+            ? 'dead'
+            : nackOptions.park === true
+              ? 'suspended'
+              : 'ready',
         runAt: at + nackOptions.delayMs,
         // A suspension must not burn an attempt, or a 3-day sleep dead-letters the run. Floored
         // where `SQL_NACK` floors it (`greatest(attempt - 1, 0)`): the fence above is what keeps

package/src/driver-pg-ddl.ts

@@ -39,19 +39,33 @@ alter table x_jobs add column if not exists traceparent text;
 
 alter table x_jobs add column if not exists enqueued_by text;
 
--- Partial unique index: one LIVE job per (name, idempotency key). Completed rows stay for
+-- Partial unique index: one LIVE job per (name, tenant, idempotency key). Completed rows stay for
 -- history, so re-running the same work tomorrow is allowed and re-delivering it today is not.
 --
 -- The NAME is in the key, and its absence was silent data loss: two jobs that happened to derive
 -- the same natural key from the same input ("user:42") shared one namespace, so the second
 -- enqueue deduped against the FIRST jobs row and returned its id. The work never ran, no error
--- was raised, and the queue showed one healthy job. The old index is dropped rather than left
--- beside the new one — it is strictly narrower, so keeping it would keep enforcing exactly the
--- collision this fixes.
+-- was raised, and the queue showed one healthy job.
+--
+-- The TENANT is in the key for that same argument with one word substituted, and it is the worse
+-- half. Every natural key an app writes is unique only WITHIN a tenant, invoice:1001 and
+-- order:5540 being the shapes the docs suggest, so tenant B enqueuing while tenant A held that
+-- key deduped into tenant A row: tenant B work never ran, and tenant B caller received tenant A
+-- job id, which is valid on every id-addressed surface. Cancel takes an id with no tenant
+-- predicate.
+--
+-- coalesce rather than the bare column, because a null tenant_id compares unequal to every other
+-- null under a unique index and a tenantless queue would lose its dedupe entirely. All tenantless
+-- rows share one namespace instead, which is exactly what they had before tenancy existed.
+--
+-- Each superseded index is dropped rather than left beside the new one. Each is strictly narrower,
+-- so keeping it would keep enforcing exactly the collision this fixes.
 drop index if exists x_jobs_idempotency_live_idx;
 
-create unique index if not exists x_jobs_name_idempotency_live_idx
-  on x_jobs (name, idempotency_key)
+drop index if exists x_jobs_name_idempotency_live_idx;
+
+create unique index if not exists x_jobs_name_tenant_idempotency_live_idx
+  on x_jobs (name, (coalesce(tenant_id, '')), idempotency_key)
   where state in ('ready', 'delayed', 'running', 'suspended');
 
 create index if not exists x_jobs_claim_idx

package/src/driver-pg-rows.ts

@@ -3,9 +3,10 @@
 // decoding a row is not control flow — every number arrives as `number | string` (a bigint is a
 // string in every client) and every absent column as `null`, and that translation is its own job.
 
-import type { BackfillRun, BackfillStatus } from './backfill-ledger';
-import type { JobRecord } from './driver';
-import type { StepRecord } from './steps';
+import { BACKFILL_STATUSES, type BackfillRun, isBackfillStatus } from './backfill-ledger';
+import { isJobState, JOB_STATES, type JobRecord, type JobState } from './driver';
+import { JobRowStatusUnknownError } from './errors';
+import { isStepStatus, STEP_STATUSES, type StepRecord, type StepStatus } from './steps';
 
 export interface StepRow {
   readonly run_id: string;
@@ -54,6 +55,21 @@ export interface JobRow {
   readonly enqueued_by?: string | null;
 }
 
+/**
+ * The one narrowing for all three status columns. `as` is not a check, and these rows cross a
+ * process boundary — a queue row was written by whatever build was deployed when the job was
+ * enqueued, which on a rolling deploy is not this one. `isBackfillStatus`'s own doc already stated
+ * the rule ("Never a cast — the list decides") and this file was the caller ignoring it.
+ */
+const statusIn = <T extends string>(
+  known: readonly T[],
+  is: (value: string) => value is T,
+  input: { table: string; column: string; value: string },
+): T => {
+  if (is(input.value)) return input.value;
+  throw new JobRowStatusUnknownError({ ...input, known });
+};
+
 export const num = (value: number | string | null | undefined): number =>
   value === null || value === undefined ? 0 : Number(value);
 
@@ -71,7 +87,11 @@ export function toJobRecord(row: JobRow): JobRecord {
     runId: row.run_id,
     attempt: row.attempt,
     maxAttempts: row.max_attempts,
-    state: row.state as JobRecord['state'],
+    state: statusIn<JobState>(JOB_STATES, isJobState, {
+      table: 'ultimate_jobs',
+      column: 'state',
+      value: row.state,
+    }),
     runAt: num(row.run_at),
     createdAt: num(row.created_at),
     updatedAt: num(row.updated_at),
@@ -94,7 +114,11 @@ export function toStepRecord(row: StepRow): StepRecord {
   return {
     runId: row.run_id,
     name: row.name,
-    status: row.status as StepRecord['status'],
+    status: statusIn<StepStatus>(STEP_STATUSES, isStepStatus, {
+      table: 'ultimate_job_steps',
+      column: 'status',
+      value: row.status,
+    }),
     output: row.output,
     startedAt: num(row.started_at),
     attempts: row.attempts,
@@ -112,7 +136,11 @@ export function toBackfillRun(row: BackfillRow): BackfillRun {
     runId: row.run_id,
     name: row.name,
     checksum: row.checksum,
-    status: row.status as BackfillStatus,
+    status: statusIn(BACKFILL_STATUSES, isBackfillStatus, {
+      table: 'ultimate_backfills',
+      column: 'status',
+      value: row.status,
+    }),
     appVersion: row.app_version,
     // `rows_processed` is a bigint, which every Postgres client hands back as a string.
     rows: num(row.rows_processed),

package/src/driver-pg-sql.ts

@@ -17,17 +17,26 @@ values
   ($1, $2, $3, $4::jsonb, $5, $6, $7,
    case when to_timestamp($8 / 1000.0) > now() then 'delayed' else 'ready' end,
    to_timestamp($8 / 1000.0), $9, $10, $11)
-on conflict (name, idempotency_key)
+on conflict (name, (coalesce(tenant_id, '')), idempotency_key)
   where state in ('ready', 'delayed', 'running', 'suspended')
   do nothing
 returning id, run_id
 `.trim();
 
-/** Scoped by NAME as well as key — the index is, so a lookup that was not would find a stranger. */
+/**
+ * Scoped by NAME and TENANT as well as key, because the index is — a lookup narrower than the
+ * index it reads answers with whichever stranger holds the key, and `{ deduped: true, id: <another
+ * tenant's> }` is a cross-tenant handle, not just a missed run.
+ *
+ * `coalesce` on both sides, matching the index expression exactly: a bare `tenant_id = $3` never
+ * matches a null row, so every tenantless enqueue would fall through to the "rejected but no live
+ * row holds its idempotency key" refusal instead of finding its own live job.
+ */
 export const SQL_FIND_LIVE_BY_KEY = `
 select id, run_id from x_jobs
  where name = $1
    and idempotency_key = $2
+   and coalesce(tenant_id, '') = coalesce($3::text, '')
    and state in ('ready', 'delayed', 'running', 'suspended')
  limit 1
 `.trim();

package/src/driver-pg.ts

@@ -271,19 +271,22 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
         return { id: inserted.id, runId: inserted.run_id, deduped: false };
       }
 
-      // `do nothing` fired: a live job OF THIS NAME already owns this idempotency key. The name
-      // is in the lookup because it is in the index — without it this returned whichever other
-      // job happened to derive the same natural key, and the caller's work silently never ran.
+      // `do nothing` fired: a live job OF THIS NAME, IN THIS TENANT, already owns this idempotency
+      // key. Both are in the lookup because both are in the index — without the name this returned
+      // whichever other job derived the same natural key; without the tenant it returned another
+      // TENANT's row, so the caller's work silently never ran AND the caller was handed an id it
+      // has no right to, on a surface (`cancel`) that takes an id with no tenant predicate.
       const existing = await exec().query<{ id: string; run_id: string }>(SQL_FIND_LIVE_BY_KEY, [
         request.name,
         request.idempotencyKey,
+        request.tenantId ?? null,
       ]);
       const found = existing[0];
       if (found === undefined) {
         throw new DriverUnavailableError({
           driver: 'pg',
           cause: `enqueue of "${request.name}" was rejected but no live row holds its idempotency key`,
-          fix: 'x db migrate   # reapplies SQL_JOBS_TABLE, whose x_jobs_name_idempotency_live_idx is what this lookup reads',
+          fix: 'x db migrate   # reapplies SQL_JOBS_TABLE, whose x_jobs_name_tenant_idempotency_live_idx is what this lookup reads',
         });
       }
       if (request.onConflict === 'error') {
@@ -321,7 +324,14 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
 
     async nack(jobId: string, nackOptions: NackOptions): Promise<void> {
       const counts = nackOptions.countsAsAttempt !== false;
-      const state = nackOptions.deadLetter === true ? 'dead' : counts ? 'ready' : 'suspended';
+      // The same three-way the memory driver takes, and it reads `park` rather than `counts`: the
+      // attempt counter and the ready bucket are two facts, and a shed only ever meant the first.
+      const state =
+        nackOptions.deadLetter === true
+          ? 'dead'
+          : nackOptions.park === true
+            ? 'suspended'
+            : 'ready';
       await exec().query(SQL_NACK, [
         jobId,
         state,

package/src/driver.ts

@@ -17,15 +17,22 @@ import type { StepStore } from './steps';
  * a cancellation is work an operator stopped on purpose and `x jobs retry` must not resurrect by
  * accident. It appears in no claim predicate, so the queue never hands a cancelled row out again.
  */
-export type JobState =
-  | 'ready'
-  | 'delayed'
-  | 'running'
-  | 'suspended'
-  | 'done'
-  | 'failed'
-  | 'dead'
-  | 'cancelled';
+export const JOB_STATES = [
+  'ready',
+  'delayed',
+  'running',
+  'suspended',
+  'done',
+  'failed',
+  'dead',
+  'cancelled',
+] as const;
+
+export type JobState = (typeof JOB_STATES)[number];
+
+/** Narrows a state read back off a queue row. Never a cast — the list decides. */
+export const isJobState = (value: string): value is JobState =>
+  (JOB_STATES as readonly string[]).includes(value);
 
 export interface JobRecord {
   readonly id: string;
@@ -114,10 +121,23 @@ export interface NackOptions {
   readonly delayMs: number;
   readonly error?: string;
   /**
-   * False for a suspension (`step.sleep`): parking a run is not a failure and must not burn
-   * a retry attempt, or a 3-day sleep would dead-letter the job.
+   * The ATTEMPT COUNTER, and nothing else. False for a suspension and for a shed alike: neither is
+   * a failure, and a 3-day sleep that burned an attempt would dead-letter the job.
    */
   readonly countsAsAttempt?: boolean;
+  /**
+   * True for a SUSPENSION — `step.sleep`, or a name this deploy does not know — which leaves the
+   * ready bucket and is counted `suspended`. Absent for a limiter or `job.concurrency` shed, which
+   * is a job still WAITING to run.
+   *
+   * The two were one flag until 2026-08: `countsAsAttempt: false` decided the state as well as the
+   * counter, so a shed was filed beside a 3-day sleep and `stats()` excluded it from `ready` and
+   * from `oldestReadyMs` — the two numbers the worker publishes as `queue_depth` and
+   * `queue_oldest_ready_seconds`. Under sustained overload the shed fraction approaches 100%, so
+   * the HPA signal and the "oldest job older than 5 minutes" page both went quiet exactly when the
+   * queue was saturated.
+   */
+  readonly park?: boolean;
   readonly deadLetter?: boolean;
 }
 
@@ -128,7 +148,11 @@ export interface QueueStats {
   readonly running: number;
   readonly suspended: number;
   readonly dead: number;
-  /** Age in ms of the oldest claimable job — the number that decides autoscaling. */
+  /**
+   * Age in ms of the oldest job that is READY and due — the number that decides autoscaling.
+   * Not "claimable": a `suspended` row is claimable once its `runAt` passes and is deliberately
+   * excluded here, which is exactly why a limiter shed may not be filed as a suspension.
+   */
   readonly oldestReadyMs: number;
 }
 

package/src/errors.ts

@@ -23,6 +23,8 @@ export const JOB_OWNED_ERROR_CODES = [
   'X_BACKFILL_RUNNING',
   'X_BACKFILL_STALLED',
   'X_BACKFILL_UNKNOWN',
+  'X_JOB_ROW_STATUS_UNKNOWN',
+  'X_ACTION_JOB_UNBRIDGED',
 ] as const;
 
 /**
@@ -58,6 +60,8 @@ export const JOB_ERROR_TITLES: Readonly<Record<JobOwnedErrorCode, string>> = {
   X_BACKFILL_RUNNING: 'a pass under this name is already live',
   X_BACKFILL_STALLED: 'the sweep ended with rows its own count still matches',
   X_BACKFILL_UNKNOWN: 'no declaration carries this backfill name',
+  X_JOB_ROW_STATUS_UNKNOWN: 'a queue row carries a status this build does not know',
+  X_ACTION_JOB_UNBRIDGED: 'an action projection was registered as a job',
 };
 
 // One unconditional call, so a second package claiming one of jobs' codes throws
@@ -89,7 +93,8 @@ registerErrorRetry({
   X_BACKFILL_APPLIED: 'terminal',
 });
 
-const docsFor = (code: JobErrorCode): string => `https://ultimate.dev/errors/${code}`;
+/** Shared with `backfill-errors.ts`, which holds the seven `X_BACKFILL_*` classes. */
+export const docsFor = (code: JobErrorCode): string => `https://ultimate.dev/errors/${code}`;
 
 /** An enqueue collided with a live job holding the same idempotency key under `onConflict: 'error'`. */
 export class JobDuplicateError extends UltimateError {
@@ -124,6 +129,57 @@ export class JobNameTakenError extends UltimateError {
   }
 }
 
+/**
+ * A `text` status column holds a value outside the vocabulary this build compiled.
+ *
+ * Refused rather than passed through, because the alternative is what used to happen: the three
+ * decoders in `driver-pg-rows.ts` cast the column, and `stepRun`'s `existing?.status ===
+ * 'completed'` then read false for the laundered value and RE-EXECUTED the step. A second charge
+ * is a worse answer than a failed attempt, and "an unrecognised fact is never a satisfied one" is
+ * the rule the rest of the framework already follows.
+ *
+ * Almost always a NEWER deploy's row, not corruption: a status string only reaches the table
+ * because some version of this framework wrote it. A rolling deploy that only ADDS a status is
+ * safe in the normal direction — the new build knows every old value — and it is the old build
+ * reading the new build's row that lands here, on that one job, loudly.
+ */
+export class JobRowStatusUnknownError extends UltimateError {
+  constructor(input: { table: string; column: string; value: string; known: readonly string[] }) {
+    super({
+      code: 'X_JOB_ROW_STATUS_UNKNOWN',
+      cause:
+        `${input.table}.${input.column} holds "${input.value}", which this build does not know — ` +
+        `it reads ${input.known.join(', ')}`,
+      fix: `x jobs show --json   # then drain the older workers: a status this build cannot read was almost certainly written by a newer deploy`,
+      docs: docsFor('X_JOB_ROW_STATUS_UNKNOWN'),
+    });
+  }
+}
+
+/**
+ * `registerJobs()` was handed `someAction.job()`.
+ *
+ * That call answers an `ActionJobHandle` — `kind: 'action-job'`, deliberately a different literal
+ * from `'job'` — which is the four fields `job()` takes, not a job. It cannot be one: `action` and
+ * `jobs` are both tier 3, so neither may import the other, and only `job()` seats a handle the
+ * queue, the worker and the manifest accept.
+ *
+ * Refused BY NAME rather than skipped, which is what used to happen. `registerJobs(module)` is
+ * handed a whole module namespace, so silently ignoring a constant or a helper exported beside a
+ * job is right — but ignoring this one meant `registerJobs({ publishPost: publishPost.job() })`
+ * registered nothing, returned `[]`, and the job never ran, with nothing failing anywhere.
+ */
+export class ActionJobUnbridgedError extends UltimateError {
+  constructor(input: { export: string; job: string }) {
+    super({
+      code: 'X_ACTION_JOB_UNBRIDGED',
+      cause: `export "${input.export}" is the action projection "${input.job}", which is not a job handle and cannot be registered as one`,
+      fix: `wrap it: agentJob(${input.export}, { name: '${input.export}', tenant, retry }) from @ultimat3/ai — that composes job() and returns a handle the queue accepts`,
+      docs: docsFor('X_ACTION_JOB_UNBRIDGED'),
+    });
+  }
+}
+
 /** Two steps in one run share a name, so replay cannot tell their persisted results apart. */
 export class StepDuplicateError extends UltimateError {
   constructor(input: { job: string; step: string }) {
@@ -334,134 +390,6 @@ export class OutboxNoTxError extends UltimateError {
   }
 }
 
-/**
- * The seven backfill codes below all answer one question — "why is this sweep not running?" — and
- * each is here because it sends the reader somewhere different: run it, force it, change
- * environment, migrate first, wait, fix the predicates, or fix the name. A code that shared a fix
- * line with another one would be code inflation, which is why `X_BACKFILL_WRITE_UNCONFIRMED` was
- * considered and rejected: a dry run that wrote nothing did exactly what it was asked to.
- *
- * Every `fix` below is ONE line something can execute — a shell command, or the edit to make.
- * Never a command with prose appended: a `fix:` is copied and run verbatim, so a trailing clause
- * turns a working command into a syntax error at the one moment the reader is following it
- * literally. Explanations belong in `cause`, which is read and never run.
- */
-
-/**
- * Declared and never completed. The alarm the framework did not have: an author could
- * `x g backfill`, merge and deploy, and nothing anywhere said the pass had not run.
- */
-export class BackfillPendingError extends UltimateError {
-  constructor(input: { backfill: string; environment: string }) {
-    super({
-      code: 'X_BACKFILL_PENDING',
-      cause: `backfill "${input.backfill}" is declared and x_backfills holds no completed pass for it in ${input.environment}`,
-      fix: `x db backfill ${input.backfill} --write --json`,
-      docs: docsFor('X_BACKFILL_PENDING'),
-    });
-  }
-}
-
-/** Already swept. A rerun is legitimate, so this names the flag rather than refusing outright. */
-export class BackfillAppliedError extends UltimateError {
-  constructor(input: { backfill: string; runId: string; completedAt: string }) {
-    super({
-      code: 'X_BACKFILL_APPLIED',
-      cause: `backfill "${input.backfill}" completed as run ${input.runId} at ${input.completedAt}; a forced rerun writes a NEW ledger row and never edits that one`,
-      fix: `x db backfill ${input.backfill} --write --force --json`,
-      docs: docsFor('X_BACKFILL_APPLIED'),
-    });
-  }
-}
-
-/**
- * The declaration names the environments it belongs to and this is not one. Declared DATA, never a
- * hardcoded "cleanups are production": a staging rehearsal is correct practice, so which
- * environments a sweep belongs to is the app's convention and this is only the mechanism carrying
- * it (axiom 8).
- */
-export class BackfillEnvironmentError extends UltimateError {
-  constructor(input: { backfill: string; environment: string; declared: readonly string[] }) {
-    // The first declared environment, because the fix has to be ONE runnable line and the list is
-    // ordered by the author. The empty case cannot arise from `checkBackfillEnvironment`, which
-    // treats an empty list as "every environment" — but this constructor is public, so it answers
-    // with the command that lists what IS declared rather than an `ULTIMATE_ENV=undefined`.
-    const target = input.declared[0];
-    super({
-      code: 'X_BACKFILL_ENVIRONMENT',
-      cause: `backfill "${input.backfill}" declares environments: ${input.declared.join(', ')} and this process resolved ${input.environment} — add "${input.environment}" to that list if this deploy should sweep too`,
-      fix:
-        target === undefined
-          ? 'x db backfill --pending --json'
-          : `ULTIMATE_ENV=${target} x db backfill ${input.backfill} --write --json`,
-      docs: docsFor('X_BACKFILL_ENVIRONMENT'),
-    });
-  }
-}
-
-/**
- * `requires` names a migration the ledger has not applied. Checked where `x_migrations` is
- * readable — this package holds no `@ultimat3/db` dependency and growing one to read a ledger
- * would put the migration engine on the tier-3 queue's import graph.
- */
-export class BackfillMigrationPendingError extends UltimateError {
-  constructor(input: { backfill: string; migration: string }) {
-    super({
-      code: 'X_BACKFILL_MIGRATION_PENDING',
-      cause: `backfill "${input.backfill}" requires migration ${input.migration}, which x_migrations does not record as applied`,
-      fix: 'x db migrate --json',
-      docs: docsFor('X_BACKFILL_MIGRATION_PENDING'),
-    });
-  }
-}
-
-/**
- * The enqueue deduped: one live pass per name, so this run is the one already going. Distinct from
- * `X_BACKFILL_APPLIED`, which is a pass that finished — the response there is `--force`, and the
- * response here is to look at the run that is holding the key.
- */
-export class BackfillRunningError extends UltimateError {
-  constructor(input: { backfill: string; jobId: string }) {
-    super({
-      code: 'X_BACKFILL_RUNNING',
-      cause: `backfill "${input.backfill}" already has a live pass queued as ${input.jobId}, and one name holds one live pass; its step trace names the batch it is on, and a pass that is not advancing is a worker that lost its lease`,
-      fix: `x jobs show ${input.jobId} --json`,
-      docs: docsFor('X_BACKFILL_RUNNING'),
-    });
-  }
-}
-
-/**
- * The source ran out of rows and the declaration's own `count()` still matches some. Two
- * predicates that disagree is an authoring bug in any business — the sweep reported success over
- * rows it never visited — so the pass fails rather than writing a completed row nobody can trust.
- */
-export class BackfillStalledError extends UltimateError {
-  constructor(input: { backfill: string; remaining: number; swept: number }) {
-    super({
-      code: 'X_BACKFILL_STALLED',
-      cause: `backfill "${input.backfill}" swept ${input.swept} rows, exhausted its source, and count() still matches ${input.remaining} — a WHERE the sweep narrows and the count does not is what leaves rows behind`,
-      fix: `make count() select on exactly what source() selects on in backfill("${input.backfill}")`,
-      docs: docsFor('X_BACKFILL_STALLED'),
-    });
-  }
-}
-
-/** A name no declaration in this app carries — a typo, or a backfill whose module was deleted. */
-export class BackfillUnknownError extends UltimateError {
-  constructor(input: { backfill: string; known: readonly string[] }) {
-    super({
-      code: 'X_BACKFILL_UNKNOWN',
-      cause:
-        input.known.length === 0
-          ? `no backfill named "${input.backfill}" is declared, and this app declares none at all`
-          : `no backfill named "${input.backfill}" is declared (declared: ${input.known.join(', ')})`,
-      fix: 'x db backfill --pending --json',
-      docs: docsFor('X_BACKFILL_UNKNOWN'),
-    });
-  }
-}
-
 export class JobsNotImplementedError extends UltimateError {
   constructor(input: { feature: string; fix: string }) {
     super({

package/src/execute.ts

@@ -1,5 +1,7 @@
 // One claimed job run to completion, suspension or failure and settled with the driver — the
-// single execution path the worker loop and `x jobs run` share. It owns the run's deadline, and
+// single execution path, shared by the worker loop (`worker-run.ts`) and by the one caller outside
+// this package, `@ultimat3/testing`'s job fixture. There is no `x jobs run` to share it with: the
+// subcommands are `ls`, `show`, `retry`, `cancel`, `drain`. It owns the run's deadline, and
 // a deadline here means CANCEL: the nack that follows makes the job claimable again, so a body
 // still running past it would be a second copy of one job, racing the attempt that replaced it.
 
@@ -87,7 +89,8 @@ export interface ExecuteJobOptions {
 
 /**
  * Run one claimed job to completion, suspension or failure, and settle it with the driver.
- * Shared by the worker loop and `x jobs run` so both take exactly the same code path.
+ * Shared by the worker loop and by `@ultimat3/testing`'s job fixture, so a job under test takes
+ * exactly the code path the worker takes.
  */
 export async function executeJob(options: ExecuteJobOptions): Promise<JobExecution> {
   const { driver, claimed, handle } = options;
@@ -159,8 +162,10 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
   } catch (error) {
     if (isStepSuspension(error)) {
       const delayMs = Math.max(0, error.resumeAt - nowMs(options.clock));
-      // countsAsAttempt: false — parking a run is not a failure.
-      await driver.nack(claimed.id, { delayMs, countsAsAttempt: false });
+      // `park: true` is the suspension itself — the row leaves the ready bucket — and
+      // `countsAsAttempt: false` only says not to burn an attempt on it. A limiter shed passes the
+      // second and not the first: it is a job still waiting, and it belongs in `queue_depth`.
+      await driver.nack(claimed.id, { delayMs, countsAsAttempt: false, park: true });
       return settle({
         outcome: 'suspended',
         jobId: claimed.id,
@@ -200,8 +205,8 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
     // This package's ONE error-reporting call site, and it is here rather than in the loop because
     // this is the only frame that still holds the thrown value — the loop sees a message string.
     // A retry is a failure the framework recovered from, so it is a `warning`; a dead letter is
-    // one nobody recovered from. `x jobs run` takes this path too, which is the point: one
-    // execution path means one place a failed job can become visible.
+    // one nobody recovered from. A job driven by `@ultimat3/testing`'s fixture takes this path
+    // too, which is the point: one execution path means one place a failed job becomes visible.
     reportError(error, {
       source: 'job',
       severity: decision.retry ? 'warning' : 'error',

package/src/index.ts

@@ -17,6 +17,15 @@ export type {
   BackfillReport,
 } from './backfill';
 export { backfill, DEFAULT_BACKFILL_BATCH } from './backfill';
+export {
+  BackfillAppliedError,
+  BackfillEnvironmentError,
+  BackfillMigrationPendingError,
+  BackfillPendingError,
+  BackfillRunningError,
+  BackfillStalledError,
+  BackfillUnknownError,
+} from './backfill-errors';
 export type { BackfillGate, BackfillGateInput } from './backfill-gate';
 export { checkBackfillEnvironment, gateBackfill } from './backfill-gate';
 export type { BackfillProgress } from './backfill-inspect';
@@ -77,11 +86,13 @@ export type {
 export {
   DEFAULT_QUEUE,
   DEFAULT_VISIBILITY_TIMEOUT_MS,
+  isJobState,
+  JOB_STATES,
   jobDriver,
   resetJobDriver,
   setJobDriver,
 } from './driver';
-export type { MemoryDriverOptions } from './driver-memory';
+export type { MemoryDriverOptions, MemoryJobDriver } from './driver-memory';
 export { createMemoryDriver } from './driver-memory';
 export type { NatsDriverOptions } from './driver-nats';
 export { createNatsDriver } from './driver-nats';
@@ -121,13 +132,7 @@ export type { RedisDriverOptions } from './driver-redis';
 export { createRedisDriver } from './driver-redis';
 export type { JobErrorCode } from './errors';
 export {
-  BackfillAppliedError,
-  BackfillEnvironmentError,
-  BackfillMigrationPendingError,
-  BackfillPendingError,
-  BackfillRunningError,
-  BackfillStalledError,
-  BackfillUnknownError,
+  ActionJobUnbridgedError,
   CancelUnsupportedError,
   ConcurrencyUnenforceableError,
   DriverUnavailableError,
@@ -139,6 +144,7 @@ export {
   JobMaxAttemptsError,
   JobNameTakenError,
   JobNotCancellableError,
+  JobRowStatusUnknownError,
   JobSlotLostError,
   JobsNotImplementedError,
   JobTenantRequiredError,
@@ -248,8 +254,10 @@ export type {
 export {
   createMemoryStepStore,
   createStepRunner,
+  isStepStatus,
   isStepSuspension,
   MAX_TRACE_NAMES,
+  STEP_STATUSES,
   StepSuspension,
 } from './steps';
 export type {

package/src/metrics.ts

@@ -18,7 +18,7 @@ import { gauge } from '@ultimat3/core';
 /** Seconds and not milliseconds: every Prometheus duration is seconds, and the alert is `> 300`. */
 export const queueOldestReady: Gauge = gauge('queue_oldest_ready_seconds', {
   unit: 's',
-  description: 'Age of the oldest claimable job, by queue — 0 when the queue is empty',
+  description: 'Age of the oldest job that is ready and due, by queue — 0 when none is',
 });
 
 export const queueDeadJobs: Gauge = gauge('queue_dead_jobs', {

package/src/register.ts

@@ -5,9 +5,24 @@
  */
 
 import { type RegisteredPrimitive, registerPrimitiveRegistrar } from '@ultimat3/core';
+import { ActionJobUnbridgedError } from './errors';
 import { isJobHandle, registerJob } from './job';
 import { isTaskHandle, registerTask } from './task';
 
+/**
+ * `@ultimat3/action`'s job PROJECTION, recognised structurally because that package is this tier
+ * and may never be imported here. `kind: 'action-job'` is a literal chosen to be distinguishable
+ * from `'job'` rather than a near-miss (`packages/action/src/job-handle.ts` says so), which is
+ * precisely what makes this check possible without an import.
+ */
+const isActionProjection = (
+  value: unknown,
+): value is { readonly kind: 'action-job'; readonly name: string } =>
+  typeof value === 'object' &&
+  value !== null &&
+  'kind' in value &&
+  (value as { readonly kind: unknown }).kind === 'action-job';
+
 /** `registerJobs(await import('./jobs'))` — export names become job names. */
 export function registerJobs(
   module: Readonly<Record<string, unknown>>,
@@ -15,7 +30,16 @@ export function registerJobs(
   const registered: RegisteredPrimitive[] = [];
   for (const name of Object.keys(module).sort()) {
     const value = module[name];
-    if (isJobHandle(value)) registered.push(registerJob(name, value));
+    if (isJobHandle(value)) {
+      registered.push(registerJob(name, value));
+      continue;
+    }
+    // Everything else is skipped in silence — a module namespace is full of constants, types and
+    // helpers exported beside the jobs. Everything else EXCEPT this one, which is unambiguously
+    // someone trying to queue an action and getting nothing at all.
+    if (isActionProjection(value)) {
+      throw new ActionJobUnbridgedError({ export: name, job: value.name });
+    }
   }
   return registered;
 }

package/src/retry.ts

@@ -1,6 +1,7 @@
-// Retry arithmetic, kept pure so the schedule is testable and printable. `x jobs schedule`
-// renders `retrySchedule()` verbatim — an agent should be able to see when attempt 5 lands
-// without running the queue.
+// Retry arithmetic, kept pure so the schedule is testable and printable. `x jobs show <id>`
+// renders `retrySchedule()` verbatim as `JobTrace.retryDelaysMs` — an agent should be able to see
+// when attempt 5 lands without running the queue. There is no `x jobs schedule`: the subcommands
+// are `ls`, `show`, `retry`, `cancel`, `drain`.
 
 import type { DurationInput } from './clock';
 import { toMs } from './clock';

package/src/steps.ts

@@ -12,7 +12,20 @@ import type { DurationInput } from './clock';
 import { nowMs, toMs } from './clock';
 import { JobAbortedError, JobTimeoutError, StepDuplicateError } from './errors';
 
-export type StepStatus = 'completed' | 'sleeping' | 'waiting' | 'failed';
+/**
+ * The runtime list is the declaration and `StepStatus` is derived from it, the shape
+ * `BACKFILL_STATUSES` and `PRIMITIVE_KINDS` already have. A bare union cannot narrow a `text`
+ * column, so `driver-pg-rows.ts` cast one instead — and a cast that lands an unknown status on a
+ * record makes `stepRun`'s `existing?.status === 'completed'` false, which RE-EXECUTES a step
+ * this file promises runs once.
+ */
+export const STEP_STATUSES = ['completed', 'sleeping', 'waiting', 'failed'] as const;
+
+export type StepStatus = (typeof STEP_STATUSES)[number];
+
+/** Narrows a status read back out of a store. Never a cast — the list decides. */
+export const isStepStatus = (value: string): value is StepStatus =>
+  (STEP_STATUSES as readonly string[]).includes(value);
 
 export interface StepRecord {
   readonly runId: string;

package/src/task.ts

@@ -54,6 +54,7 @@ export interface TaskDefinition {
    */
   enqueue: (occurrenceMs: number) => readonly TaskEnqueueEntry[];
   readonly catchUp?: CatchUpPolicy;
+  /** Whole occurrences per round, one or more. Zero fires nothing at all, and `task()` refuses it. */
   readonly maxCatchUp?: number;
 }
 
@@ -98,6 +99,9 @@ export interface TaskHandle {
 const registry = new Map<string, TaskHandle>();
 let anonymous = 0;
 
+/** Occurrences one round may fire when neither the declaration nor a catch-up says otherwise. */
+const DEFAULT_MAX_CATCH_UP = 10;
+
 /** Job's store, for tasks: proof `task()` built the handle, plus whether it named itself. */
 interface TaskOrigin {
   readonly declaredName: boolean;
@@ -127,13 +131,26 @@ export function task(definition: TaskDefinition): TaskHandle {
     `use the full zone id on task("${name}"), e.g. tz: 'America/Bogota' — list the valid ones with: bun -e "console.log(Intl.supportedValuesOf('timeZone').join('\\n'))"`,
   );
 
+  // `maxCatchUp: 0` is not "no ceiling" — `occurrencesSince` walks
+  // `for (let i = 0; i < handle.maxCatchUp; i += 1)`, so zero (and any negative, and any fraction
+  // below one) returns an empty list on every round and the task NEVER fires: no error, no log
+  // line, no queue row, forever. Refused where it is written, exactly as `job()` refuses
+  // `concurrency: 0` and `createPacer` refuses `rate: 0`. `Number.isInteger` covers `NaN` and
+  // `Infinity` in the same predicate — an unbounded catch-up is a burst nobody declared.
+  assert(
+    definition.maxCatchUp === undefined ||
+      (Number.isInteger(definition.maxCatchUp) && definition.maxCatchUp >= 1),
+    `task "${name}" declares maxCatchUp ${String(definition.maxCatchUp)}, so no occurrence can ever fire`,
+    `set a whole maxCatchUp of 1 or more on task("${name}"), or omit the field for the default of ${DEFAULT_MAX_CATCH_UP}`,
+  );
+
   const handle: TaskHandle = {
     kind: 'task',
     name,
     cron: definition.cron,
     tz: definition.tz,
     catchUp: definition.catchUp ?? 'skip',
-    maxCatchUp: definition.maxCatchUp ?? 10,
+    maxCatchUp: definition.maxCatchUp ?? DEFAULT_MAX_CATCH_UP,
     // `nowMs()` and not `Date.now()`: every reading of time in this package goes through a
     // Clock so a frozen one cannot be bypassed.
     entries: (occurrenceMs: number = nowMs()) => definition.enqueue(occurrenceMs),

package/src/worker-run.ts

@@ -48,10 +48,13 @@ export async function runClaimedJob(options: RunClaimedOptions): Promise<JobExec
   const handle = getJob(claimed.name);
   if (handle === undefined) {
     // Park it, do not burn attempts: the job may well be registered by the pod next to this one.
+    // A genuine park — no worker in this deploy can run it — so it leaves the ready bucket, unlike
+    // a limiter shed, which is a job this fleet will pick up on its next pass.
     await driver.nack(claimed.id, {
       delayMs: 30_000,
       error: `no job registered as "${claimed.name}"`,
       countsAsAttempt: false,
+      park: true,
     });
     return unknownJob(claimed);
   }

package/src/worker.ts

@@ -161,6 +161,26 @@ export function createWorker(options: WorkerOptions): Worker {
       ...(options.events === undefined ? {} : { events: options.events }),
     });
 
+  /**
+   * A claimed job handed straight back over a cap. It is NOT a suspension and NOT a failure: no
+   * `park`, so the row stays where `queue_depth` and `queue_oldest_ready_seconds` can see it, and
+   * no `error`, so `x jobs show` does not report a `lastError` for a job that never ran. It was
+   * both of those until 2026-08 — parked beside a 3-day `step.sleep`, and stamped with a failure
+   * it never had — which is why the two sheds go through one function now.
+   */
+  const shed = async (
+    claimed: ClaimedJob,
+    detail: { readonly queue: string; readonly reason: string },
+  ): Promise<void> => {
+    logger.debug('jobs.worker.shed', {
+      workerId,
+      job: claimed.name,
+      jobId: claimed.id,
+      ...detail,
+    });
+    await options.driver.nack(claimed.id, { delayMs: pollIntervalMs, countsAsAttempt: false });
+  };
+
   /** The drain's one question: may this worker still take work off the queue? */
   const claiming = (): boolean => state !== 'draining' && state !== 'stopped';
 
@@ -196,11 +216,17 @@ export function createWorker(options: WorkerOptions): Worker {
           ...(job.tenantId === undefined ? {} : { tenantId: job.tenantId }),
         });
         if (lease === undefined) {
-          // Over a tenant/queue/global cap: hand it straight back for another worker.
-          await options.driver.nack(job.id, {
-            delayMs: pollIntervalMs,
-            countsAsAttempt: false,
-            error: `limited: ${limiter.blockedBy({ queue, ...(job.tenantId === undefined ? {} : { tenantId: job.tenantId }) }) ?? 'unknown'}`,
+          // Over a tenant/queue/global cap: hand it straight back for another worker. No `park`
+          // and no `error` — the row stays in the ready bucket the depth gauge reads, and nothing
+          // about this job failed, so `x jobs show` must not report a `lastError` for it. The
+          // reason is a log FIELD instead, where it costs nothing when nobody is asking.
+          await shed(job, {
+            queue,
+            reason:
+              limiter.blockedBy({
+                queue,
+                ...(job.tenantId === undefined ? {} : { tenantId: job.tenantId }),
+              }) ?? 'unknown',
           });
           continue;
         }
@@ -223,10 +249,9 @@ export function createWorker(options: WorkerOptions): Worker {
         }
         if (!granted) {
           lease.release();
-          await options.driver.nack(job.id, {
-            delayMs: pollIntervalMs,
-            countsAsAttempt: false,
-            error: `limited: job concurrency (${getJob(job.name)?.concurrency ?? 0})`,
+          await shed(job, {
+            queue,
+            reason: `job concurrency (${getJob(job.name)?.concurrency ?? 0})`,
           });
           continue;
         }