@ultimat3/jobs 8.0.0 → 9.0.0

package/CLAUDE.md

@@ -334,6 +334,29 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   jumps over a page the live iteration never read. A checkpoint READ back is checked rather than
   trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`,
   so the pass would silently reopen the source at the top and walk the whole table again.
+- **`purge()` is a FACTORY over `job()` too, and it is the ONE caller every `purgeExpired()` in
+  the framework was missing** (`As of 2026-08-22`). Three stores shipped one —
+  `postgresIdempotencyStore` (`x_idempotency`), `postgresRateLimitStore` (`x_rate_limit`) and
+  `postgresAuthLimiter` (`x_auth_failures`/`x_auth_lockouts`) — each documented as "an app runs
+  this from a `task`", and a task only ENQUEUES, so there was no job for one to enqueue and every
+  row written was a row kept. `x_rate_limit` takes one upsert per HTTP request the web role serves,
+  assets included.
+
+  `PurgeTarget` is STRUCTURAL (`{ name, purgeExpired(nowMs) }`) for the reason `PgExecutor` is: two
+  of those three packages are below this one and one is beside it, and a sweep that needed their
+  types would put the HTTP pipeline on this package's import graph. `targets()` is a THUNK, read
+  once per attempt: a host declares the sweep at boot and the auth limiter does not exist yet —
+  `defineAuth` builds it when the app's modules import. One table per `step.run`, so a killed
+  attempt resumes at the table it stopped on; a purge is idempotent by nature, so the replay that
+  at-least-once guarantees deletes rows that are already gone. **One clock reading for the whole
+  pass**, handed to every target: `postgresRateLimitStore.purgeExpired(nowMs)` requires the
+  CALLER's clock, and reading the server's computed a 20,000,000-second refill against a frozen
+  test clock and deleted a bucket holding 0 of 4 tokens — a free limit reset, handed out by the
+  cleanup. Two targets under one name are refused (`X_INVARIANT`) before the first delete rather
+  than discovered as `X_STEP_DUPLICATE` after one table is already empty.
+
+  It declares no schedule of its own: `DEFAULT_PURGE_CRON` is the hourly cron a host's `task()`
+  uses, and `@ultimat3/cli`'s `dev-purge.ts` is the one that declares both halves at boot.
 - **`handle` is AT LEAST ONCE, and the ordering that makes it so is deliberate.** The body runs
   inside the step and the record is written after it returns, so an attempt killed, cancelled or
   lease-expired between the two hands that page to the next attempt — which is why the doc comment,
@@ -687,6 +710,7 @@ picture from the other side.
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
 | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back. The claim loop asks "may I start this one?"; this answers it across the fleet |
+| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep, its structural target seam and the hourly cron a host schedules it on |
 | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
 | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
 | `limits.ts` | per-tenant / per-queue / global concurrency + rate |

package/README.md

@@ -298,6 +298,45 @@ the new pods serve, puts the sweeps on the queue and exits, and a slow UPDATE ne
 open against a database still serving the previous build. `--all` isolates per name and continues
 past a failure, so one wedged cleanup cannot block every later one forever.
 
+## Retention sweeps are jobs too
+
+`purge()` is the **second factory over `job()`**, and it exists because three framework stores
+shipped a `purgeExpired()` with no caller — `x_idempotency`, `x_rate_limit` and the auth pair each
+kept every row they ever took. `x_rate_limit` takes one upsert per HTTP request the web role
+serves, assets included, so its growth follows total traffic and not traffic that hit a limit.
+
+```ts
+import { DEFAULT_PURGE_CRON, purge, task } from '@ultimat3/jobs';
+
+declare const store: { purgeExpired(nowMs: number): Promise<number> };
+
+export const sweep = purge({
+  name: 'x.purge',
+  // Read once per ATTEMPT, never captured: a host declares the sweep at boot, and some of the
+  // stores behind it are built later.
+  targets: () => [{ name: 'x_rate_limit', purgeExpired: (nowMs) => store.purgeExpired(nowMs) }],
+});
+
+export const hourly = task({
+  name: 'x.purge.hourly',
+  cron: DEFAULT_PURGE_CRON,
+  tz: 'UTC',
+  enqueue: () => [[sweep, {}]],
+});
+```
+
+| Rule | Why |
+|---|---|
+| `PurgeTarget` is structural | the stores live in `@ultimat3/action`, `@ultimat3/http` and `@ultimat3/auth`; importing them would put the HTTP pipeline on this package's graph |
+| one `step.run` per target | a killed attempt resumes at the table it stopped on, not at the first |
+| one clock reading per pass | `postgresRateLimitStore.purgeExpired(nowMs)` needs the CALLER's clock — the server's read a 20,000,000-second refill against a frozen one and deleted a live bucket |
+| at least once is safe here | a replayed delete removes rows that are already gone, and a row a purge deleted answers exactly as one that was never there |
+| two targets under one name | `X_INVARIANT`, before the first delete — `step.run` would raise `X_STEP_DUPLICATE` after one table was already empty |
+
+`@ultimat3/cli`'s boot declares both halves over the three tables it owns, so an app gets the sweep
+without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
+deployment with neither has no background work at all, and this is one more thing it does not do.
+
 ## The deadline cancels
 
 A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "8.0.0",
+  "version": "9.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "8.0.0",
-    "@ultimat3/entity": "8.0.0",
-    "@ultimat3/schema": "8.0.0",
-    "@ultimat3/time": "8.0.0"
+    "@ultimat3/core": "9.0.0",
+    "@ultimat3/entity": "9.0.0",
+    "@ultimat3/schema": "9.0.0",
+    "@ultimat3/time": "9.0.0"
   }
 }

package/src/index.ts

@@ -221,6 +221,14 @@ export { DEFAULT_OUTBOX_CLAIM_LEASE_MS } from './outbox-lease';
 export type { PgOutboxOptions } from './outbox-pg';
 export { createPgOutboxStore } from './outbox-pg';
 
+export type {
+  PurgeDefinition,
+  PurgeInput,
+  PurgeReport,
+  PurgeSweep,
+  PurgeTarget,
+} from './purge';
+export { DEFAULT_PURGE_CRON, purge } from './purge';
 export type { BackoffStrategy, Random, RetryDecision, RetryPolicy } from './retry';
 export { backoffDelayMs, DEFAULT_RETRY, nextRetry, retrySchedule } from './retry';
 export type { JobRetryDecision, JobStopReason } from './retry-classification';

package/src/purge.ts

@@ -0,0 +1,150 @@
+// `purge()` — the framework's retention sweep, declared as a `job` and NOT as a ninth primitive.
+// Deleting expired rows on a schedule is durable background work with an input schema, a retry
+// policy, an idempotency key and a queue, which is the definition of a `job` — so this file is a
+// FACTORY over `job()`, exactly as `backfill()` is one and `llm()` is one over `action()`. That is
+// what gives a retention sweep `.enqueue()`, the worker's cancellation, the dead-letter path,
+// `x jobs show` and a manifest row without a line here.
+//
+// WHY it exists: `postgresIdempotencyStore`, `postgresRateLimitStore` and `postgresAuthLimiter`
+// each shipped a `purgeExpired()` and NOTHING called any of them, so every row those three tables
+// ever took was a row kept. `x_rate_limit` takes one upsert per HTTP request a web role serves,
+// assets included, so its growth is proportional to total traffic rather than to traffic that hit
+// a limit. A `task` could not fix it: a task only ENQUEUES, which is this package's design.
+
+import type { Clock } from '@ultimat3/core';
+import { assert, logger } from '@ultimat3/core';
+import { t } from '@ultimat3/schema';
+import type { DurationInput } from './clock';
+import { nowMs } from './clock';
+import type { JobHandle } from './job';
+import { job } from './job';
+import type { RetryPolicy } from './retry';
+import { DEFAULT_RETRY } from './retry';
+
+/**
+ * One table's worth of expired rows, behind the narrowest possible seam.
+ *
+ * Structural, exactly like `JobActor` and `PgExecutor`: the three stores this was written for live
+ * in `@ultimat3/action` (this tier), `@ultimat3/http` and `@ultimat3/auth` — none of them
+ * importable here — and a sweep that needed their types would put the whole HTTP pipeline on this
+ * package's import graph. A store satisfies this by having the method it already has.
+ */
+export interface PurgeTarget {
+  /**
+   * What this sweep is called in its durable step, its log line and its report. A table name is
+   * the natural spelling (`x_rate_limit`); a target that clears a SET of tables names their common
+   * prefix (`x_auth`, for `x_auth_failures` and `x_auth_lockouts`). Unique within one definition —
+   * the name is the step key, and two steps under one name is `X_STEP_DUPLICATE` mid-run.
+   */
+  readonly name: string;
+  /**
+   * Delete every expired row and answer how many went.
+   *
+   * `nowMs` is the JOB's clock, and a store that writes its instants from the caller MUST measure
+   * against it rather than against `now()` on the server. That mismatch is not theoretical: the
+   * http store's purge read `extract(epoch from now())` against a `last_ms` written by the caller
+   * and, on a frozen test clock, computed a 20,000,000-second refill and deleted a bucket holding
+   * 0 of 4 tokens — a free limit reset, handed out by the cleanup. A store that holds its own
+   * clock (because its host handed it one) may ignore this argument; a store that holds none
+   * may not.
+   */
+  purgeExpired(nowMs: number): Promise<number>;
+}
+
+/** What one target's sweep removed. Bounded and JSON-safe, so it survives as a step's output. */
+export interface PurgeSweep {
+  readonly name: string;
+  readonly removed: number;
+}
+
+/** What one pass reports — bounded, so `x jobs show` can print it. */
+export interface PurgeReport {
+  readonly swept: readonly PurgeSweep[];
+  readonly removed: number;
+}
+
+/**
+ * A purge decides nothing, so its payload carries nothing. Deliberately not a `force` flag like
+ * `BackfillInput`'s: a backfill is a ONE-PASS sweep whose ledger says it already ran, and this is
+ * a recurring one with no ledger and nothing to override.
+ */
+export type PurgeInput = Readonly<Record<string, never>>;
+
+export interface PurgeDefinition {
+  /**
+   * Omit it and `defineApi({ jobs })` assigns the export name. A framework-owned sweep pins one,
+   * the way `mail.send` does, because the queue key is what rows already carry.
+   */
+  readonly name?: string;
+  /**
+   * The tables to sweep, read ONCE PER ATTEMPT rather than captured at declaration. Lazy because
+   * a host declares the sweep at boot and the stores behind it are not all resolved yet — an
+   * app's `defineAuth` runs after the boot that installed the limiter factory, so the auth target
+   * does not exist until later. An empty list is a pass that removes nothing, which is the honest
+   * answer for a process whose boot has already stopped.
+   */
+  targets(): readonly PurgeTarget[];
+  /**
+   * The clock every target is measured against. Defaults to the system clock, and it must be the
+   * SAME clock the stores write their instants from — see `PurgeTarget.purgeExpired`.
+   */
+  readonly clock?: Clock;
+  readonly queue?: string;
+  readonly retry?: RetryPolicy;
+  /** Per attempt. A killed attempt resumes at the first table it had not yet checkpointed. */
+  readonly timeout?: DurationInput;
+}
+
+/** The cron a framework-shipped sweep runs on when its host has no opinion. */
+export const DEFAULT_PURGE_CRON = '23 * * * *';
+
+export function purge(definition: PurgeDefinition): JobHandle<PurgeInput> {
+  const clock = definition.clock;
+
+  return job<PurgeInput>({
+    ...(definition.name === undefined ? {} : { name: definition.name }),
+    input: t.object({}),
+    // One live sweep, forever: a second enqueue while a pass is still running is the same pass,
+    // and two deletes racing over one table buy nothing but lock contention. The scheduler's own
+    // key is occurrence-scoped on top of this, so the hourly runs are still distinct.
+    idempotencyKey: () => 'purge',
+    // Framework tables, not an org's rows. Every statement behind a target is raw SQL over the
+    // whole table, so there is no tenant-scoped read here to fail closed.
+    tenant: 'none',
+    retry: definition.retry ?? DEFAULT_RETRY,
+    ...(definition.queue === undefined ? {} : { queue: definition.queue }),
+    ...(definition.timeout === undefined ? {} : { timeout: definition.timeout }),
+    async run({ step }): Promise<PurgeReport> {
+      // ONE reading for every target in the pass. Two readings would let two tables be measured
+      // against instants a round trip apart, which is the same class of mismatch as reading the
+      // server's clock — smaller, and just as unnecessary.
+      const at = nowMs(clock);
+      const targets = definition.targets();
+      const names = new Set(targets.map((target) => target.name));
+      // Refused before the first delete, not discovered at the second step: `step.run` raises
+      // `X_STEP_DUPLICATE` on the repeat, which dead-letters a sweep AFTER it has already emptied
+      // one table. The list is lazy, so this cannot be checked at declaration.
+      assert(
+        names.size === targets.length,
+        `purge targets repeat a name: ${[...names].sort().join(', ')} across ${targets.length} targets`,
+        'give every PurgeTarget its own name — the name is the durable step key, and two steps under one name is X_STEP_DUPLICATE',
+      );
+
+      const swept: PurgeSweep[] = [];
+      for (const target of targets) {
+        // One durable step per table, so a killed attempt resumes at the table it stopped on
+        // rather than sweeping the ones already done a second time. At least once either way, and
+        // a purge is idempotent by nature: a replayed delete removes the rows that are already
+        // gone, which is none, and a row this deletes answers exactly as a row that was never
+        // there — no decision anywhere changes.
+        const removed = await step.run(target.name, () => target.purgeExpired(at));
+        swept.push({ name: target.name, removed });
+      }
+      const removed = swept.reduce((total, sweep) => total + sweep.removed, 0);
+      // Ops reads this to size the cadence: a sweep that removes hundreds of thousands every hour
+      // is a table that wants a shorter window, not a longer cron.
+      if (removed > 0) logger.info('jobs.purge.swept', { removed, tables: swept.length });
+      return { swept, removed };
+    },
+  });
+}