@ultimat3/jobs 7.0.0 → 9.0.0

package/CLAUDE.md

@@ -334,6 +334,29 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   jumps over a page the live iteration never read. A checkpoint READ back is checked rather than
   trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`,
   so the pass would silently reopen the source at the top and walk the whole table again.
+- **`purge()` is a FACTORY over `job()` too, and it is the ONE caller every `purgeExpired()` in
+  the framework was missing** (`As of 2026-08-22`). Three stores shipped one —
+  `postgresIdempotencyStore` (`x_idempotency`), `postgresRateLimitStore` (`x_rate_limit`) and
+  `postgresAuthLimiter` (`x_auth_failures`/`x_auth_lockouts`) — each documented as "an app runs
+  this from a `task`", and a task only ENQUEUES, so there was no job for one to enqueue and every
+  row written was a row kept. `x_rate_limit` takes one upsert per HTTP request the web role serves,
+  assets included.
+
+  `PurgeTarget` is STRUCTURAL (`{ name, purgeExpired(nowMs) }`) for the reason `PgExecutor` is: two
+  of those three packages are below this one and one is beside it, and a sweep that needed their
+  types would put the HTTP pipeline on this package's import graph. `targets()` is a THUNK, read
+  once per attempt: a host declares the sweep at boot and the auth limiter does not exist yet —
+  `defineAuth` builds it when the app's modules import. One table per `step.run`, so a killed
+  attempt resumes at the table it stopped on; a purge is idempotent by nature, so the replay that
+  at-least-once guarantees deletes rows that are already gone. **One clock reading for the whole
+  pass**, handed to every target: `postgresRateLimitStore.purgeExpired(nowMs)` requires the
+  CALLER's clock, and reading the server's computed a 20,000,000-second refill against a frozen
+  test clock and deleted a bucket holding 0 of 4 tokens — a free limit reset, handed out by the
+  cleanup. Two targets under one name are refused (`X_INVARIANT`) before the first delete rather
+  than discovered as `X_STEP_DUPLICATE` after one table is already empty.
+
+  It declares no schedule of its own: `DEFAULT_PURGE_CRON` is the hourly cron a host's `task()`
+  uses, and `@ultimat3/cli`'s `dev-purge.ts` is the one that declares both halves at boot.
 - **`handle` is AT LEAST ONCE, and the ordering that makes it so is deliberate.** The body runs
   inside the step and the record is written after it returns, so an attempt killed, cancelled or
   lease-expired between the two hands that page to the next attempt — which is why the doc comment,
@@ -567,6 +590,25 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
   that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
   where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
+- **No read returns a WHOLE row, and `driver-pg-sql.test.ts` is what enforces it** (`As of
+  2026-08-22`). `PgExecutor` is an injected seam over any client that speaks `(text, values)`, and a
+  client with no type map decodes `timestamptz` as TEXT — so `toJobRecord`/`toStepRecord` read
+  `Number('2026-01-01 00:00:00+00')` and answered `NaN`. Six statements were `select *` /
+  `returning *` and shipped that way: `pgStepStore.list`, `introspect.job`, `introspect.list`,
+  `introspect.deadLetters`, `introspect.requeue` and `SQL_CANCEL`. Every one of them feeds a
+  surface an operator reads — `x jobs ls`, `x jobs show`, `x jobs cancel` — and `SQL_CLAIM` had
+  projected epoch ms all along, so the driver disagreed with itself. The guard is a scan of every
+  production file in this directory, DISCOVERED rather than listed and with comments stripped: a
+  `select *` anywhere in them, in either case, is a failing test rather than a review note. It
+  reads the directory because the hand-kept list it replaces had already missed one
+  (`driver-pg-ddl.ts`) — a registry a new SQL source opts out of by simply not joining it is a
+  guard with the shape of a rule and none of the force.
+- **A run's acquisitions are handed back even when the wiring throws.** `worker-run.ts` starts the
+  lease heartbeat first, so every line between it and the `try` can leak an interval renewing the
+  lease of a job that never ran, with nothing left holding a reference to stop it. `context()` was
+  moved ABOVE the heartbeat for that reason; `createRunSignal` and `fleetSlots.startRenewal` — the
+  second an injected seam whose production implementation reaches a lease store — are INSIDE the
+  `try`, with `undefined` meaning "never taken" and both handbacks idempotent.
 - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
 - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
 - Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
@@ -653,8 +695,9 @@ picture from the other side.
 | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark) + `createPgLeaseLeader` |
 | `events-pg.ts` | `createPgEventBus` — `step.waitForEvent` across processes |
 | `driver.ts` | `JobDriver` contract + wire records |
-| `driver-pg.ts` | default driver, real SQL constants, advisory-lock leader |
+| `driver-pg.ts` | default driver, real SQL constants, and `createPgLeader` — the advisory-lock election that is **not** what a scheduler uses; `scheduler-pg.ts` above owns the lease-row one boot wires |
 | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` + `SQL_OUTBOX_TABLE` — the schema the driver installs. Whichever file holds the DDL is the one whose comments may carry no `;` and no `'` |
+| `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and the `JOB_ROW_COLUMNS` projection they share. Split off at `driver-pg-sql.ts`'s size ceiling and re-exported from it |
 | `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
 | `driver-memory.ts` | `x dev` / tests |
 | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
@@ -667,6 +710,7 @@ picture from the other side.
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
 | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back. The claim loop asks "may I start this one?"; this answers it across the fleet |
+| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep, its structural target seam and the hourly cron a host schedules it on |
 | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
 | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
 | `limits.ts` | per-tenant / per-queue / global concurrency + rate |

package/README.md

@@ -298,6 +298,45 @@ the new pods serve, puts the sweeps on the queue and exits, and a slow UPDATE ne
 open against a database still serving the previous build. `--all` isolates per name and continues
 past a failure, so one wedged cleanup cannot block every later one forever.
 
+## Retention sweeps are jobs too
+
+`purge()` is the **second factory over `job()`**, and it exists because three framework stores
+shipped a `purgeExpired()` with no caller — `x_idempotency`, `x_rate_limit` and the auth pair each
+kept every row they ever took. `x_rate_limit` takes one upsert per HTTP request the web role
+serves, assets included, so its growth follows total traffic and not traffic that hit a limit.
+
+```ts
+import { DEFAULT_PURGE_CRON, purge, task } from '@ultimat3/jobs';
+
+declare const store: { purgeExpired(nowMs: number): Promise<number> };
+
+export const sweep = purge({
+  name: 'x.purge',
+  // Read once per ATTEMPT, never captured: a host declares the sweep at boot, and some of the
+  // stores behind it are built later.
+  targets: () => [{ name: 'x_rate_limit', purgeExpired: (nowMs) => store.purgeExpired(nowMs) }],
+});
+
+export const hourly = task({
+  name: 'x.purge.hourly',
+  cron: DEFAULT_PURGE_CRON,
+  tz: 'UTC',
+  enqueue: () => [[sweep, {}]],
+});
+```
+
+| Rule | Why |
+|---|---|
+| `PurgeTarget` is structural | the stores live in `@ultimat3/action`, `@ultimat3/http` and `@ultimat3/auth`; importing them would put the HTTP pipeline on this package's graph |
+| one `step.run` per target | a killed attempt resumes at the table it stopped on, not at the first |
+| one clock reading per pass | `postgresRateLimitStore.purgeExpired(nowMs)` needs the CALLER's clock — the server's read a 20,000,000-second refill against a frozen one and deleted a live bucket |
+| at least once is safe here | a replayed delete removes rows that are already gone, and a row a purge deleted answers exactly as one that was never there |
+| two targets under one name | `X_INVARIANT`, before the first delete — `step.run` would raise `X_STEP_DUPLICATE` after one table was already empty |
+
+`@ultimat3/cli`'s boot declares both halves over the three tables it owns, so an app gets the sweep
+without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
+deployment with neither has no background work at all, and this is one more thing it does not do.
+
 ## The deadline cancels
 
 A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "7.0.0",
+  "version": "9.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "7.0.0",
-    "@ultimat3/entity": "7.0.0",
-    "@ultimat3/schema": "7.0.0",
-    "@ultimat3/time": "7.0.0"
+    "@ultimat3/core": "9.0.0",
+    "@ultimat3/entity": "9.0.0",
+    "@ultimat3/schema": "9.0.0",
+    "@ultimat3/time": "9.0.0"
   }
 }

package/src/driver-pg-jobs-sql.ts

@@ -0,0 +1,45 @@
+// Every statement returning a WHOLE `x_jobs` row, and the one column list they share. They ask
+// Postgres for epoch ms because `select *` left the decoding to the client's type map: one with
+// none decodes `timestamptz` as TEXT, so `toJobRecord` read `Number('2026-01-01 00:00:00+00')`
+// and `x jobs ls` / `show` / `cancel` printed `NaN` for every timestamp.
+
+/**
+ * The `JobRow` shape as a projection. Asking Postgres for epoch ms is what makes the decoding
+ * independent of the client's type map — never `select *`, whose correctness is the driver's
+ * opinion about `timestamptz` rather than this statement's.
+ */
+export const JOB_ROW_COLUMNS = `id, name, queue, input, idempotency_key, run_id, attempt,
+       max_attempts, state, tenant_id, last_error, claimed_by,
+       traceparent, enqueued_by,
+       (extract(epoch from run_at)     * 1000)::bigint as run_at,
+       (extract(epoch from visible_at) * 1000)::bigint as visible_at,
+       (extract(epoch from created_at) * 1000)::bigint as created_at,
+       (extract(epoch from updated_at) * 1000)::bigint as updated_at`;
+
+export const SQL_JOB_GET = `
+select ${JOB_ROW_COLUMNS}
+  from x_jobs where id = $1
+`.trim();
+
+export const SQL_JOB_LIST = `
+select ${JOB_ROW_COLUMNS}
+  from x_jobs
+ where ($1::text is null or queue = $1)
+   and ($2::text is null or name  = $2)
+   and ($3::text is null or state = $3)
+ order by created_at desc
+ limit $4
+`.trim();
+
+export const SQL_JOB_DEAD_LETTERS = `
+select ${JOB_ROW_COLUMNS}
+  from x_jobs where state = 'dead' order by updated_at desc limit $1
+`.trim();
+
+/** `run_at = now()` makes the requeued job due immediately; the attempt counter starts over. */
+export const SQL_JOB_REQUEUE = `
+update x_jobs
+   set state = 'ready', attempt = 0, run_at = now(), updated_at = now()
+ where id = $1
+returning ${JOB_ROW_COLUMNS}
+`.trim();

package/src/driver-pg-sql.ts

@@ -8,6 +8,19 @@
 // point keeps the ONE import path every caller already uses.
 
 export { SQL_JOBS_TABLE, SQL_OUTBOX_TABLE } from './driver-pg-ddl';
+// The whole-`x_jobs`-row reads, split off at this file's size ceiling and re-exported for the
+// same reason the DDL is. `JOB_ROW_COLUMNS` is imported as a value because `SQL_CANCEL` also
+// returns a whole row and must project it identically — two spellings of one row shape is how
+// one of them goes back to `returning *`.
+export {
+  JOB_ROW_COLUMNS,
+  SQL_JOB_DEAD_LETTERS,
+  SQL_JOB_GET,
+  SQL_JOB_LIST,
+  SQL_JOB_REQUEUE,
+} from './driver-pg-jobs-sql';
+
+import { JOB_ROW_COLUMNS } from './driver-pg-jobs-sql';
 
 export const SQL_ENQUEUE = `
 insert into x_jobs
@@ -112,7 +125,7 @@ update x_jobs
    set state = 'cancelled', visible_at = null, claimed_by = null,
        last_error = coalesce($2, last_error), updated_at = now()
  where id = $1 and state <> 'done'
-returning *
+returning ${JOB_ROW_COLUMNS}
 `.trim();
 
 /**
@@ -370,6 +383,22 @@ select run_id, name, status, output, attempts, error,
   from x_job_steps where run_id = $1 and name = $2
 `.trim();
 
+/**
+ * `list`'s projection is `SQL_STEP_GET`'s, minus the name predicate. It exists as its own constant
+ * because `select *` was what `list` issued: a client that decodes `timestamptz` as text — most of
+ * them, without a type map — handed `toStepRecord` a date string, `Number()` made it `NaN`, and
+ * `x jobs show` printed one per step. Every other statement in this file asks Postgres to do the
+ * conversion; this is that rule applied to the one statement that had opted out of it.
+ */
+export const SQL_STEP_LIST = `
+select run_id, name, status, output, attempts, error,
+       (extract(epoch from started_at)   * 1000)::bigint as started_at,
+       (extract(epoch from completed_at) * 1000)::bigint as completed_at,
+       (extract(epoch from wake_at)      * 1000)::bigint as wake_at,
+       event, correlation_key
+  from x_job_steps where run_id = $1 order by started_at
+`.trim();
+
 export const SQL_STEP_PUT = `
 insert into x_job_steps
   (run_id, name, status, output, started_at, completed_at, wake_at, event,

package/src/driver-pg.ts

@@ -35,12 +35,17 @@ import {
   SQL_ENQUEUE,
   SQL_FIND_LIVE_BY_KEY,
   SQL_HEARTBEAT,
+  SQL_JOB_DEAD_LETTERS,
+  SQL_JOB_GET,
+  SQL_JOB_LIST,
+  SQL_JOB_REQUEUE,
   SQL_LEASE_ACQUIRE,
   SQL_LEASE_RELEASE,
   SQL_LEASE_RENEW,
   SQL_NACK,
   SQL_STATS,
   SQL_STEP_GET,
+  SQL_STEP_LIST,
   SQL_STEP_PUT,
   SQL_TRY_ADVISORY_LOCK,
 } from './driver-pg-sql';
@@ -102,10 +107,7 @@ function pgStepStore(exec: () => PgExecutor): StepStore {
       ]);
     },
     async list(runId) {
-      const rows = await exec().query<StepRow>(
-        `select * from x_job_steps where run_id = $1 order by started_at`,
-        [runId],
-      );
+      const rows = await exec().query<StepRow>(SQL_STEP_LIST, [runId]);
       return rows.map(toStepRecord);
     },
     async del(runId, name) {
@@ -189,27 +191,21 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
 
   const introspect: JobIntrospection = {
     async job(jobId) {
-      const rows = await exec().query<JobRow>(`select * from x_jobs where id = $1`, [jobId]);
+      const rows = await exec().query<JobRow>(SQL_JOB_GET, [jobId]);
       const row = rows[0];
       return row === undefined ? undefined : toJobRecord(row);
     },
     async list(filter: JobFilter = {}) {
-      const rows = await exec().query<JobRow>(
-        `select * from x_jobs
-          where ($1::text is null or queue = $1)
-            and ($2::text is null or name  = $2)
-            and ($3::text is null or state = $3)
-          order by created_at desc
-          limit $4`,
-        [filter.queue ?? null, filter.name ?? null, filter.state ?? null, filter.limit ?? 100],
-      );
+      const rows = await exec().query<JobRow>(SQL_JOB_LIST, [
+        filter.queue ?? null,
+        filter.name ?? null,
+        filter.state ?? null,
+        filter.limit ?? 100,
+      ]);
       return rows.map(toJobRecord);
     },
     async deadLetters(limit = 100) {
-      const rows = await exec().query<JobRow>(
-        `select * from x_jobs where state = 'dead' order by updated_at desc limit $1`,
-        [limit],
-      );
+      const rows = await exec().query<JobRow>(SQL_JOB_DEAD_LETTERS, [limit]);
       return rows.map(toJobRecord);
     },
     async requeue(jobId, requeueOptions) {
@@ -222,11 +218,7 @@ export function createPgDriver(options: PgDriverOptions = {}): JobDriver {
           ]);
         }
       }
-      const rows = await exec().query<JobRow>(
-        `update x_jobs set state = 'ready', attempt = 0, run_at = now(), updated_at = now()
-          where id = $1 returning *`,
-        [jobId],
-      );
+      const rows = await exec().query<JobRow>(SQL_JOB_REQUEUE, [jobId]);
       const row = rows[0];
       if (row === undefined) {
         throw new DriverUnavailableError({

package/src/index.ts

@@ -221,6 +221,14 @@ export { DEFAULT_OUTBOX_CLAIM_LEASE_MS } from './outbox-lease';
 export type { PgOutboxOptions } from './outbox-pg';
 export { createPgOutboxStore } from './outbox-pg';
 
+export type {
+  PurgeDefinition,
+  PurgeInput,
+  PurgeReport,
+  PurgeSweep,
+  PurgeTarget,
+} from './purge';
+export { DEFAULT_PURGE_CRON, purge } from './purge';
 export type { BackoffStrategy, Random, RetryDecision, RetryPolicy } from './retry';
 export { backoffDelayMs, DEFAULT_RETRY, nextRetry, retrySchedule } from './retry';
 export type { JobRetryDecision, JobStopReason } from './retry-classification';

package/src/purge.ts

@@ -0,0 +1,150 @@
+// `purge()` — the framework's retention sweep, declared as a `job` and NOT as a ninth primitive.
+// Deleting expired rows on a schedule is durable background work with an input schema, a retry
+// policy, an idempotency key and a queue, which is the definition of a `job` — so this file is a
+// FACTORY over `job()`, exactly as `backfill()` is one and `llm()` is one over `action()`. That is
+// what gives a retention sweep `.enqueue()`, the worker's cancellation, the dead-letter path,
+// `x jobs show` and a manifest row without a line here.
+//
+// WHY it exists: `postgresIdempotencyStore`, `postgresRateLimitStore` and `postgresAuthLimiter`
+// each shipped a `purgeExpired()` and NOTHING called any of them, so every row those three tables
+// ever took was a row kept. `x_rate_limit` takes one upsert per HTTP request a web role serves,
+// assets included, so its growth is proportional to total traffic rather than to traffic that hit
+// a limit. A `task` could not fix it: a task only ENQUEUES, which is this package's design.
+
+import type { Clock } from '@ultimat3/core';
+import { assert, logger } from '@ultimat3/core';
+import { t } from '@ultimat3/schema';
+import type { DurationInput } from './clock';
+import { nowMs } from './clock';
+import type { JobHandle } from './job';
+import { job } from './job';
+import type { RetryPolicy } from './retry';
+import { DEFAULT_RETRY } from './retry';
+
+/**
+ * One table's worth of expired rows, behind the narrowest possible seam.
+ *
+ * Structural, exactly like `JobActor` and `PgExecutor`: the three stores this was written for live
+ * in `@ultimat3/action` (this tier), `@ultimat3/http` and `@ultimat3/auth` — none of them
+ * importable here — and a sweep that needed their types would put the whole HTTP pipeline on this
+ * package's import graph. A store satisfies this by having the method it already has.
+ */
+export interface PurgeTarget {
+  /**
+   * What this sweep is called in its durable step, its log line and its report. A table name is
+   * the natural spelling (`x_rate_limit`); a target that clears a SET of tables names their common
+   * prefix (`x_auth`, for `x_auth_failures` and `x_auth_lockouts`). Unique within one definition —
+   * the name is the step key, and two steps under one name is `X_STEP_DUPLICATE` mid-run.
+   */
+  readonly name: string;
+  /**
+   * Delete every expired row and answer how many went.
+   *
+   * `nowMs` is the JOB's clock, and a store that writes its instants from the caller MUST measure
+   * against it rather than against `now()` on the server. That mismatch is not theoretical: the
+   * http store's purge read `extract(epoch from now())` against a `last_ms` written by the caller
+   * and, on a frozen test clock, computed a 20,000,000-second refill and deleted a bucket holding
+   * 0 of 4 tokens — a free limit reset, handed out by the cleanup. A store that holds its own
+   * clock (because its host handed it one) may ignore this argument; a store that holds none
+   * may not.
+   */
+  purgeExpired(nowMs: number): Promise<number>;
+}
+
+/** What one target's sweep removed. Bounded and JSON-safe, so it survives as a step's output. */
+export interface PurgeSweep {
+  readonly name: string;
+  readonly removed: number;
+}
+
+/** What one pass reports — bounded, so `x jobs show` can print it. */
+export interface PurgeReport {
+  readonly swept: readonly PurgeSweep[];
+  readonly removed: number;
+}
+
+/**
+ * A purge decides nothing, so its payload carries nothing. Deliberately not a `force` flag like
+ * `BackfillInput`'s: a backfill is a ONE-PASS sweep whose ledger says it already ran, and this is
+ * a recurring one with no ledger and nothing to override.
+ */
+export type PurgeInput = Readonly<Record<string, never>>;
+
+export interface PurgeDefinition {
+  /**
+   * Omit it and `defineApi({ jobs })` assigns the export name. A framework-owned sweep pins one,
+   * the way `mail.send` does, because the queue key is what rows already carry.
+   */
+  readonly name?: string;
+  /**
+   * The tables to sweep, read ONCE PER ATTEMPT rather than captured at declaration. Lazy because
+   * a host declares the sweep at boot and the stores behind it are not all resolved yet — an
+   * app's `defineAuth` runs after the boot that installed the limiter factory, so the auth target
+   * does not exist until later. An empty list is a pass that removes nothing, which is the honest
+   * answer for a process whose boot has already stopped.
+   */
+  targets(): readonly PurgeTarget[];
+  /**
+   * The clock every target is measured against. Defaults to the system clock, and it must be the
+   * SAME clock the stores write their instants from — see `PurgeTarget.purgeExpired`.
+   */
+  readonly clock?: Clock;
+  readonly queue?: string;
+  readonly retry?: RetryPolicy;
+  /** Per attempt. A killed attempt resumes at the first table it had not yet checkpointed. */
+  readonly timeout?: DurationInput;
+}
+
+/** The cron a framework-shipped sweep runs on when its host has no opinion. */
+export const DEFAULT_PURGE_CRON = '23 * * * *';
+
+export function purge(definition: PurgeDefinition): JobHandle<PurgeInput> {
+  const clock = definition.clock;
+
+  return job<PurgeInput>({
+    ...(definition.name === undefined ? {} : { name: definition.name }),
+    input: t.object({}),
+    // One live sweep, forever: a second enqueue while a pass is still running is the same pass,
+    // and two deletes racing over one table buy nothing but lock contention. The scheduler's own
+    // key is occurrence-scoped on top of this, so the hourly runs are still distinct.
+    idempotencyKey: () => 'purge',
+    // Framework tables, not an org's rows. Every statement behind a target is raw SQL over the
+    // whole table, so there is no tenant-scoped read here to fail closed.
+    tenant: 'none',
+    retry: definition.retry ?? DEFAULT_RETRY,
+    ...(definition.queue === undefined ? {} : { queue: definition.queue }),
+    ...(definition.timeout === undefined ? {} : { timeout: definition.timeout }),
+    async run({ step }): Promise<PurgeReport> {
+      // ONE reading for every target in the pass. Two readings would let two tables be measured
+      // against instants a round trip apart, which is the same class of mismatch as reading the
+      // server's clock — smaller, and just as unnecessary.
+      const at = nowMs(clock);
+      const targets = definition.targets();
+      const names = new Set(targets.map((target) => target.name));
+      // Refused before the first delete, not discovered at the second step: `step.run` raises
+      // `X_STEP_DUPLICATE` on the repeat, which dead-letters a sweep AFTER it has already emptied
+      // one table. The list is lazy, so this cannot be checked at declaration.
+      assert(
+        names.size === targets.length,
+        `purge targets repeat a name: ${[...names].sort().join(', ')} across ${targets.length} targets`,
+        'give every PurgeTarget its own name — the name is the durable step key, and two steps under one name is X_STEP_DUPLICATE',
+      );
+
+      const swept: PurgeSweep[] = [];
+      for (const target of targets) {
+        // One durable step per table, so a killed attempt resumes at the table it stopped on
+        // rather than sweeping the ones already done a second time. At least once either way, and
+        // a purge is idempotent by nature: a replayed delete removes the rows that are already
+        // gone, which is none, and a row this deletes answers exactly as a row that was never
+        // there — no decision anywhere changes.
+        const removed = await step.run(target.name, () => target.purgeExpired(at));
+        swept.push({ name: target.name, removed });
+      }
+      const removed = swept.reduce((total, sweep) => total + sweep.removed, 0);
+      // Ops reads this to size the cadence: a sweep that removes hundreds of thousands every hour
+      // is a table that wants a shorter window, not a longer cron.
+      if (removed > 0) logger.info('jobs.purge.swept', { removed, tables: swept.length });
+      return { swept, removed };
+    },
+  });
+}

package/src/scheduler.ts

@@ -1,7 +1,12 @@
 // The `scheduler` role: one node walks every registered task's cron, dispatches the occurrences
 // it owes and enqueues their jobs. The `task` primitive it reads lives in `task.ts`.
 //
-// Exactly one node dispatches per tick, enforced by a Postgres advisory lock. Two schedulers
+// Exactly one node dispatches per tick, enforced by leader election. Multi-node that is an
+// EXPIRING LEASE ROW in `x_scheduler_leader` (`createPgLeaseLeader`), never an advisory lock: an
+// advisory lock is held by the SESSION, not by this process — it outlives every transaction and is
+// released only by an explicit unlock, the pool's reset on release, or the connection dying, and
+// the next round may run on a different connection. So a node can neither renew it nor prove it
+// still holds one, and leadership passes to a second node while the first is still dispatching. Two schedulers
 // double-enqueue every task; the idempotency key would absorb it, but leader election means
 // the queue never sees the duplicate at all. One ROUND at a time is the same rule inside one
 // process: the loop re-arms on the round it just finished, and any other caller joins that
@@ -40,7 +45,7 @@ export interface LeaderElection {
 }
 
 /** Single-node default: always the leader. Multi-node uses `createPgLeaseLeader()` — never
- * `createPgLeader()`, whose advisory lock dies with the pooled connection that took it. */
+ * `createPgLeader()`, whose advisory lock is owned by a pooled session this process cannot name. */
 export function soleLeader(): LeaderElection {
   return {
     acquire: () => Promise.resolve(true),
@@ -276,10 +281,10 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
     timer = undefined;
     logger.info('jobs.scheduler.draining', { reason, dispatching: round !== undefined });
     try {
-      // The round this stop races runs to the end first. Releasing the advisory lock under a
+      // The round this stop races runs to the end first. Releasing the lease under a
       // live dispatch hands the next node a task this one is still enqueueing for, and both
       // then own the same occurrence — the exact double-fire leader election exists to prevent.
-      // Settled, not awaited: a round that failed is its own caller's to see, and the lock still
+      // Settled, not awaited: a round that failed is its own caller's to see, and the lease still
       // has to go back.
       await Promise.allSettled([round]);
       if (isLeader) await leader.release();

package/src/worker-run.ts

@@ -11,6 +11,7 @@ import type { JobExecution } from './execute';
 import { executeJob } from './execute';
 import { startLeaseHeartbeat } from './heartbeat';
 import { getJob } from './job';
+import type { RunSignal } from './run-signal';
 import { createRunSignal } from './run-signal';
 import type { EventLookup } from './steps';
 import type { FleetSlots } from './worker-fleet-slots';
@@ -77,33 +78,42 @@ export async function runClaimedJob(options: RunClaimedOptions): Promise<JobExec
     ...(options.clock === undefined ? {} : { clock: options.clock }),
   });
 
-  // The caller's context plus this lease's cancellation, so a job cancelled from outside — or one
-  // this worker lost the lease on — stops at the next renewal. `steps.ts` refuses every write past
-  // the signal, which is what unwinds a body that never reads it. Composed through a controller
-  // this worker owns rather than `AbortSignal.any`, for two reasons: it is handed BACK when the run
-  // settles (an app whose `context()` carries a process-lifetime signal was accumulating one
-  // composite per job), and the worker can abort it itself — which is the only way a fleet slot
-  // taken by somebody else reaches the body running under it.
-  const runSignal = createRunSignal([base.signal, heartbeat.signal]);
-  const ctx: Ctx = { ...base, signal: runSignal.signal };
+  // Everything from here on is INSIDE the `finally`, because the heartbeat is now running.
+  // `createRunSignal` and `fleetSlots.startRenewal` are ordinary calls — the second is an injected
+  // seam whose production implementation reaches a lease store — and a throw from either used to
+  // land outside the `try`, leaving an interval renewing the lease of a job that never ran with
+  // nothing holding a reference to stop it. The same defect `context()` was moved ABOVE the
+  // heartbeat for; the acquisitions after it need the `finally` rather than a reordering, since
+  // each one depends on the last. `undefined` is "never taken", and both handbacks are idempotent.
+  let runSignal: RunSignal | undefined;
+  let stopSlotRenewal: (() => void) | undefined;
 
-  // The fleet slot this job already holds, kept alive for as long as the lease is and stopped in
-  // the same `finally`: one clock for "this worker still owns the job" and "this worker still owns
-  // the slot" is one fewer way for them to disagree. A renewal answering "not yours" means another
-  // worker is already running this job under a cap of one, so it CANCELS — discarding that boolean
-  // made `job.concurrency` a number the framework prints and does not hold.
-  const stopSlotRenewal = fleetSlots.startRenewal(claimed.id, (slot) => {
-    runSignal.abort(
-      new JobSlotLostError({ job: claimed.name, jobId: claimed.id, slot: slot.slot }),
-    );
-  });
+  try {
+    // The caller's context plus this lease's cancellation, so a job cancelled from outside — or
+    // one this worker lost the lease on — stops at the next renewal. `steps.ts` refuses every write
+    // past the signal, which is what unwinds a body that never reads it. Composed through a
+    // controller this worker owns rather than `AbortSignal.any`, for two reasons: it is handed BACK
+    // when the run settles (an app whose `context()` carries a process-lifetime signal was
+    // accumulating one composite per job), and the worker can abort it itself — which is the only
+    // way a fleet slot taken by somebody else reaches the body running under it.
+    runSignal = createRunSignal([base.signal, heartbeat.signal]);
+    const signal = runSignal;
+    const ctx: Ctx = { ...base, signal: signal.signal };
 
-  // The job's span is a CHILD of the request that queued it when the row carries a trace. That
-  // link is what `04-jobs.md` promised and no column existed to hold: without it a checkout's
-  // `chargeCard` opens a fresh root two seconds later with nothing pointing back.
-  const parent = parseTraceparent(claimed.traceparent);
+    // The fleet slot this job already holds, kept alive for as long as the lease is and stopped in
+    // the same `finally`: one clock for "this worker still owns the job" and "this worker still
+    // owns the slot" is one fewer way for them to disagree. A renewal answering "not yours" means
+    // another worker is already running this job under a cap of one, so it CANCELS — discarding
+    // that boolean made `job.concurrency` a number the framework prints and does not hold.
+    stopSlotRenewal = fleetSlots.startRenewal(claimed.id, (slot) => {
+      signal.abort(new JobSlotLostError({ job: claimed.name, jobId: claimed.id, slot: slot.slot }));
+    });
+
+    // The job's span is a CHILD of the request that queued it when the row carries a trace. That
+    // link is what `04-jobs.md` promised and no column existed to hold: without it a checkout's
+    // `chargeCard` opens a fresh root two seconds later with nothing pointing back.
+    const parent = parseTraceparent(claimed.traceparent);
 
-  try {
     return await withSpan(
       `job.${handle.name}`,
       () =>
@@ -126,10 +136,10 @@ export async function runClaimedJob(options: RunClaimedOptions): Promise<JobExec
       },
     );
   } finally {
-    stopSlotRenewal();
+    stopSlotRenewal?.();
     heartbeat.stop();
     // Nothing of the caller's is held past the run: `dispose` is what makes the composition above
     // reversible, and it is the whole reason this is not `AbortSignal.any`.
-    runSignal.dispose();
+    runSignal?.dispose();
   }
 }