@marianmeres/workflow 2.2.0 → 3.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md CHANGED
@@ -2,24 +2,33 @@
2
2
 
3
3
  ## Quick Reference
4
4
 
5
- - **Stack**: Deno/Node.js, PostgreSQL, TypeScript
6
- - **Test**: `deno task test` (requires `TEST_PG_*` env; see [tests/_pg.ts](./tests/_pg.ts))
7
- - **Check**: `deno task check` — `deno fmt --check && deno lint && deno check` over `src/`, `tests/` and the example's server-side files
8
- - **Build (npm)**: `deno task npm:build`
9
- - **Example app**: `deno task example` (needs `EXAMPLE_PG_*`); `deno task example:build` after touching `example/src/`, because `example/dist/bundle.js` is committed. See [example/README.md](./example/README.md).
5
+ - **Stack**: TypeScript (erasable-only), Node LTS (`.node-version`), pnpm, `node:test`, Biome;
6
+ PostgreSQL through `@marianmeres/sqlkit` (the host's `pg` v8 pool or client, or a `Driver`)
7
+ - **Install**: `pnpm install` | **Test**: `pnpm test` (requires `TEST_PG_*` env, read from `.env`;
8
+ see [tests/_pg.ts](./tests/_pg.ts)) | **Check all**: `pnpm run ci` (typecheck + lint + test)
9
+ - **Type gate**: `pnpm run typecheck` — `tsc --noEmit` over `src/`, `tests/`, `scripts/` and
10
+ the example's server-side files (`example/*.ts`)
11
+ - **Build npm**: `pnpm run build` (`tsc` → `dist/`) | **Release + publish**: `pnpm run rp` / `rpm`
12
+ (human action)
13
+ - **Registry**: npm only, `dist/` (manifest `package.json`). No JSR since 3.0: sqlkit,
14
+ steve 4 and cron 4 are npm-only.
15
+ - **Example app**: `pnpm run example` (needs `EXAMPLE_PG_*`); `pnpm run example:build` after
16
+ touching `example/src/`, because `example/dist/bundle.js` is committed. See
17
+ [example/README.md](./example/README.md).
10
18
  - **Entry point**: [src/mod.ts](./src/mod.ts)
11
19
 
12
20
  ## Package Overview
13
21
 
14
22
  - **Name**: `@marianmeres/workflow`
15
23
  - **Type**: Durable workflow / orchestration framework
16
- - **Runtime**: Deno and Node.js (via npm build)
17
- - **DB**: PostgreSQL only (uses `pg` driver)
24
+ - **Runtime**: Node.js 24.20+ / 26.6+ (sqlkit's floor); npm, built `dist/`
25
+ - **DB**: PostgreSQL only, through a sqlkit `Driver` of the `pg` dialect. `pg` is the host's:
26
+ a devDependency here, never imported by `src/`
18
27
  - **License**: MIT
19
28
 
20
29
  ## Purpose
21
30
 
22
- Long-lived, persisted FSM instances with durable wake-ups (timers) and external-signal correlation (inbox + matchers). Definitions are pure data, versioned per-instance. Effects run as steve jobs (transactional outbox). Built on top of `@marianmeres/fsm` (state graph), `@marianmeres/steve` (effect execution), `@marianmeres/cron` (scheduler & correlator ticks), `@marianmeres/migrate` (schema), `@marianmeres/clog` (logging).
31
+ Long-lived, persisted FSM instances with durable wake-ups (timers) and external-signal correlation (inbox + matchers). Definitions are pure data, versioned per-instance. Effects run as steve jobs (transactional outbox). Built on top of `@marianmeres/fsm` (state graph), `@marianmeres/steve` (effect execution), `@marianmeres/cron` (scheduler & correlator ticks), `@marianmeres/sqlkit` (the SQL layer: driver contract, fragments, transactions, and the `@marianmeres/migrate` wiring for the schema), `@marianmeres/clog` (logging).
23
32
 
24
33
  **Ships no domain code.** Userland brings the definitions, handlers, and matchers.
25
34
 
@@ -58,13 +67,12 @@ src/
58
67
  ├── persistence/
59
68
  │ ├── instances.ts # CRUD + lock + seq bump + due/stale scans + find by correlation
60
69
  │ ├── inbox.ts # append, claim-unprocessed, lock-row, mark-processed
61
- │ ├── history.ts # append + read
62
- │ └── tx.ts # withTransaction helper
70
+ │ └── history.ts # append + read
63
71
  └── migrations/
64
72
  ├── 1_0_0.ts # CREATE/DROP three tables
65
73
  ├── 1_1_0.ts # rename project_id -> tenant_id (in place)
66
74
  ├── 1_2_0.ts # add seq (fence) + stale-pending partial index
67
- └── index.ts # createMigrate(pool) — @marianmeres/migrate setup
75
+ └── index.ts # createMigrate(db) — sqlkit's migrate wiring + the versions
68
76
  tests/
69
77
  ├── _pg.ts # env-driven pool + schema reset
70
78
  ├── fixtures/
@@ -73,18 +81,22 @@ tests/
73
81
  ├── admin.test.ts # cancel/retry: guards, fencing of in-flight jobs, resume
74
82
  ├── correlator.test.ts # deferral, dedupe, matcher throw/reject, delivery-in-advance
75
83
  ├── durability.test.ts # fence, preconditions, expiry re-dispatch, stale-pending re-poke
76
- ├── migrations.test.ts # schema rename/fence up/down + legacy payload drain
84
+ ├── migrations.test.ts # schema rename/fence up/down + legacy payload drain + legacy version table
85
+ ├── sqlkit.test.ts # the db option (pool / client / driver); job inserts in and out of the transaction
77
86
  ├── stock_replenishment.test.ts # happy/timeout/fail paths + validator
78
87
  └── weekly_digest.test.ts # recurring trigger via cron
79
- example/ # the interactive app — consumer code, imports by package name
88
+ example/ # the interactive app — consumer code, imports the public entry only
80
89
  ├── workflow.ts # userland: one definition + handlers + matcher
81
90
  ├── server.ts # runtime wiring + HTTP API (raw SQL for the listing views)
82
91
  ├── src/main.ts # the page (@marianmeres/vanilla); bundled to dist/bundle.js
92
+ ├── src/tsconfig.json # the page's type config (DOM, no Node); used by example:build
83
93
  ├── index.html # shell, styles, <template>s
84
- └── theme.css, reboot.css # generated / vendored; excluded from fmt
94
+ └── theme.css, reboot.css # generated / vendored; excluded from Biome
85
95
  scripts/
86
- ├── build-npm.ts
87
- └── gen-example-{version,theme}.ts # generated example assets; run by the example tasks
96
+ └── gen-example-{version,theme}.ts # generated example assets; run by the example scripts
97
+ package.json # manifest of record: version, npm exports, deps, scripts
98
+ tsconfig.json # type gate (src + tests + scripts + example/*.ts, noEmit)
99
+ tsconfig.build.json # src → dist (ESM + .d.ts) for npm
88
100
  ```
89
101
 
90
102
  ## Conceptual Model — Three Layers
@@ -125,11 +137,11 @@ Two invariants hold the durability story together. Break either one and the fail
125
137
 
126
138
  **2. Every settle-point write bumps `seq`; every job carries the `seq` it was issued against.** `expected_seq` on `AdvanceJobPayload`, `seq` on `EffectJobPayload`. On a mismatch the job is dropped:
127
139
 
128
- | Where | Check | On mismatch |
129
- | ---------------------------------- | ---------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
130
- | `runAdvance` | `row.seq !== payload.expected_seq` | debug log, no-op |
131
- | `runEffect` | `row.seq !== payload.seq` | `row.seq > payload.seq` → skip (`{ skipped: "stale" }`); `row.seq < payload.seq` → **throw**, so steve retries: the dispatching tx has not committed yet |
132
- | `failAdvanceJob` / `failEffectJob` | same | no-op — a dead job must not fail an instance that has moved on |
140
+ | Where | Check | On mismatch |
141
+ | ---------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
142
+ | `runAdvance` | `row.seq !== payload.expected_seq` | debug log, no-op |
143
+ | `runEffect` | `row.seq !== payload.seq` | `row.seq > payload.seq` → skip (`{ skipped: "stale" }`); `row.seq < payload.seq` → **throw**, so steve retries: the dispatching tx has not committed yet (only reachable with a `Jobs` on a connection of its own) |
144
+ | `failAdvanceJob` / `failEffectJob` | same | no-op — a dead job must not fail an instance that has moved on |
133
145
 
134
146
  This is what makes at-least-once job delivery safe. Duplicate deliveries are _expected_, not exceptional.
135
147
 
@@ -150,36 +162,36 @@ This is what makes at-least-once job delivery safe. Duplicate deliveries are _ex
150
162
 
151
163
  ## Public API (from `src/mod.ts`)
152
164
 
153
- | Export | Type | Purpose |
154
- | ------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
155
- | `Workflow` | class | Main driver: attaches handlers to an injected `steve.Jobs`. Exposes `create/find/cancel/retry/appendInbox/enqueueAdvance/enqueueEffect`. **No own start/stop** — consumer runs the Jobs. |
156
- | `WorkflowScheduler` | class | Attaches a wake tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
157
- | `WorkflowInboxCorrelator` | class | Attaches a match tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
158
- | `WorkflowRegistry` | class | Immutable registry of definitions + handlers + matchers; validates on construction |
159
- | `validateDefinition` | function | Standalone validator usable outside `WorkflowRegistry` |
160
- | `defKey(id, version)` | function | `"id@version"` registry key helper |
161
- | `createMigrate(pool)` | function | Builds `@marianmeres/migrate` instance pre-loaded with schema versions |
162
- | `getHistory(exec, id, limit?)` | function | Reads `__workflow_history` rows for an instance (observability) |
163
- | `effectJobType(name)` | function | Steve job-type string for an effect handler: `workflow.effect.<name>` |
164
- | `JOB_TYPE_ADVANCE` | const | `"workflow.advance"` |
165
- | `JOB_TYPE_EFFECT_PREFIX` | const | `"workflow.effect."` |
166
- | `PURE_ENTER_EVENT` | const | `"ENTER"` — synthetic event fired by driver into pure nodes |
167
- | `DEFAULT_TENANT_ID` | const | `"_default"` |
168
- | `EXECUTION_STATE` | const | `{ PENDING, RUNNING, WAITING, COMPLETED, FAILED, CANCELLED }` |
169
- | `HISTORY_EVENT` | const | Audit event-type strings (`created`, `transition`, `effect_dispatched`, ...) |
170
- | `NodeMeta` | type | Discriminated union: `pure` \| `effectful` \| `suspending` \| `terminal` |
171
- | `AdvanceKind` | type | `"start" \| "effect" \| "timeout" \| "signal"` — what produced an advance; drives its preconditions |
172
- | `SchedulerTickResult` | type | `{ woken, repoked }` — what one scheduler tick poked |
173
- | `WorkflowDefinition` | type | `{ id, version, fsm: WorkflowFSMConfig }` |
174
- | `WorkflowFSMConfig` | type | `FSMConfig` with `states: Record<TState, WorkflowStateConfig>` — narrowing only, still a valid `FSMConfig` |
175
- | `WorkflowStateConfig` | type | fsm state config with `meta: NodeMeta` required instead of `meta?: unknown` |
176
- | `Handler` | type | `(args: HandlerArgs) => Promise<HandlerResult> \| HandlerResult` |
177
- | `Matcher` | type | `(args: MatcherArgs) => boolean \| Promise<boolean>` |
178
- | `WorkflowInstanceRow` | type | Schema-aligned row type |
179
- | `InboxRow` | type | Schema-aligned row type |
180
- | `HistoryRow` | type | Schema-aligned row type |
181
- | `WorkflowContext` | type | `Record<string, unknown>` — accumulated payload |
182
- | `WorkflowSnapshot` | type | `FSMSnapshot<string, WorkflowContext>` |
165
+ | Export | Type | Purpose |
166
+ | ---------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
167
+ | `Workflow` | class | Main driver: attaches handlers to an injected `steve.Jobs`. Exposes `create/find/cancel/retry/appendInbox/enqueueAdvance/enqueueEffect`. **No own start/stop** — consumer runs the Jobs. |
168
+ | `WorkflowScheduler` | class | Attaches a wake tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
169
+ | `WorkflowInboxCorrelator` | class | Attaches a match tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
170
+ | `WorkflowRegistry` | class | Immutable registry of definitions + handlers + matchers; validates on construction |
171
+ | `validateDefinition` | function | Standalone validator usable outside `WorkflowRegistry` |
172
+ | `defKey(id, version)` | function | `"id@version"` registry key helper |
173
+ | `createMigrate(db)` | function | Builds `@marianmeres/migrate` instance (sqlkit's wiring over `db`) pre-loaded with schema versions |
174
+ | `getHistory(db, id, limit?)` | function | Reads `__workflow_history` rows for an instance (observability) |
175
+ | `effectJobType(name)` | function | Steve job-type string for an effect handler: `workflow.effect.<name>` |
176
+ | `JOB_TYPE_ADVANCE` | const | `"workflow.advance"` |
177
+ | `JOB_TYPE_EFFECT_PREFIX` | const | `"workflow.effect."` |
178
+ | `PURE_ENTER_EVENT` | const | `"ENTER"` — synthetic event fired by driver into pure nodes |
179
+ | `DEFAULT_TENANT_ID` | const | `"_default"` |
180
+ | `EXECUTION_STATE` | const | `{ PENDING, RUNNING, WAITING, COMPLETED, FAILED, CANCELLED }` |
181
+ | `HISTORY_EVENT` | const | Audit event-type strings (`created`, `transition`, `effect_dispatched`, ...) |
182
+ | `NodeMeta` | type | Discriminated union: `pure` \| `effectful` \| `suspending` \| `terminal` |
183
+ | `AdvanceKind` | type | `"start" \| "effect" \| "timeout" \| "signal"` — what produced an advance; drives its preconditions |
184
+ | `SchedulerTickResult` | type | `{ woken, repoked }` — what one scheduler tick poked |
185
+ | `WorkflowDefinition` | type | `{ id, version, fsm: WorkflowFSMConfig }` |
186
+ | `WorkflowFSMConfig` | type | `FSMConfig` with `states: Record<TState, WorkflowStateConfig>` — narrowing only, still a valid `FSMConfig` |
187
+ | `WorkflowStateConfig` | type | fsm state config with `meta: NodeMeta` required instead of `meta?: unknown` |
188
+ | `Handler` | type | `(args: HandlerArgs) => Promise<HandlerResult> \| HandlerResult` |
189
+ | `Matcher` | type | `(args: MatcherArgs) => boolean \| Promise<boolean>` |
190
+ | `WorkflowInstanceRow` | type | Schema-aligned row type |
191
+ | `InboxRow` | type | Schema-aligned row type |
192
+ | `HistoryRow` | type | Schema-aligned row type |
193
+ | `WorkflowContext` | type | `Record<string, unknown>` — accumulated payload |
194
+ | `WorkflowSnapshot` | type | `FSMSnapshot<string, WorkflowContext>` |
183
195
 
184
196
  See [API.md](./API.md) for full signatures, options, and examples.
185
197
 
@@ -206,7 +218,7 @@ Effectful node outcomes are whatever the handler returns (e.g. `LOW`, `OK`, `CON
206
218
 
207
219
  Steve and Cron own their own tables (`__job*`, `__cron*`); managed by their respective libraries.
208
220
 
209
- **Schema versions** (registered in `src/migrations/index.ts`, applied via `createMigrate(pool).up("latest")`):
221
+ **Schema versions** (registered in `src/migrations/index.ts`, applied via `createMigrate(db).up("latest")`). A run holds an advisory lock; each version is one transaction, committed with its version mark. A worker is `({ db }) => db.exec(script)`: it must not issue `BEGIN`/`COMMIT`, and cannot run a statement that refuses a transaction (`CREATE INDEX CONCURRENTLY`). The version table keeps the layout earlier releases created (`id`, `version`, `created_at`, newest row wins), so an existing one is continued as it is:
210
222
 
211
223
  | Version | Change |
212
224
  | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -221,6 +233,31 @@ Migration files are an append-only ledger: `1_0_0.ts` still creates `project_id`
221
233
  - `payloadTenantId()` falls back to a legacy `project_id` key (pre-1.1.0 payloads).
222
234
  - `advanceKind()` infers `kind` from the payload shape when it is absent (pre-fence payloads). Such a payload also has no `expected_seq`, i.e. it is **unfenced**: it runs without the fence check, and without the per-`kind` preconditions — guessing the kind must not swallow the job.
223
235
 
236
+ ## The Database Handle (sqlkit)
237
+
238
+ `Workflow.db` is a sqlkit `Driver` — `toPgDriver(options.db)`, where `options.db` is a `pg.Pool`, a connected `pg.Client` or a `Driver` of the `pg` dialect (`PgDb`). `createMigrate` and `getHistory` take the same. Not to be confused with the workflow _Driver_ of this document (the `workflow.advance` engine in `src/driver.ts`); the database one is always "the sqlkit driver" here.
239
+
240
+ Three properties of it shape the code:
241
+
242
+ 1. **One driver per pool.** `toPgDriver` keeps the driver it builds for a pool or a client, so every library converting the same object gets the same driver: the `Workflow`, steve's `Jobs`, cron's `Cron`. (Per copy of sqlkit — keep this package on the sqlkit minor steve and cron install.)
243
+ 2. **A transaction is the calling context, not a handle.** `db.transaction(async (tx) => …)` pins a connection for the callback's async context; `tx` is `db` itself. Any statement issued on that driver from inside the callback is in the transaction, _whoever issues it_ — so `jobs.create()` called there, on a `Jobs` that shares the driver, commits or rolls back with the instance write. A nested `transaction()` joins the open one, so a consumer's own transaction around `wf.create()` contains all of it.
244
+ 3. **A `Jobs` on another pool is another driver.** Its inserts are statements of their own, as all of steve's were before 3.0. Both wirings are supported and tested ([tests/sqlkit.test.ts](./tests/sqlkit.test.ts)); the fence and the `seq`-too-low throw in `runEffect` are what make this one safe, and stay.
245
+
246
+ Where each enqueue sits, and why:
247
+
248
+ | Call | Placement | Reason |
249
+ | ------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------ |
250
+ | `create()` → `start` advance | inside the transaction | row + history + job are one unit; a throwing enqueue rolls the instance back |
251
+ | advance → effect job | inside the transaction | the transactional outbox: no effect job for a step the instance did not take |
252
+ | `runEffect` → `effect` advance | no transaction | the handler's side effect is already out; the advance is fenced |
253
+ | `retry()` → `start` advance | after the commit | with a `Jobs` on its own connection, a job claimed before the commit would see the old `seq` and drop itself |
254
+ | correlator → `signal` advances | after the tick's commit | the advance blocks on the inbox row lock the tick holds; pokes need no atomicity |
255
+ | scheduler → `timeout` / `start` pokes | no transaction (read-only tick) | pokes need no atomicity |
256
+
257
+ Userland code that runs inside one of the package's transactions is a hazard on Postgres, where one failed statement aborts the whole transaction: a **matcher** reading through the shared driver runs inside the correlator tick's. Each matcher call is therefore wrapped in `tx.savepoint()`, so its failure is undone alone and the row is deferred. Effect handlers run outside any transaction; fsm guards and actions are synchronous.
258
+
259
+ Steve checks its schema lazily, on a `Jobs` instance's first call. Inside a transaction that rolls back, on a database with no steve tables yet, the check is rolled back too and not repeated — so a test (or a producer-only process) makes one steve call outside any transaction before wrapping `wf.create()` in one.
260
+
224
261
  ## Runtime Ownership
225
262
 
226
263
  The framework **does not start or stop steve.Jobs or cron.Cron.** The consumer constructs both, passes them in, and runs their lifecycle:
@@ -265,26 +302,29 @@ Rule: one `Jobs` and one `Cron` per process, and every process that _starts_ the
265
302
  7. **Correlation token is the index, matcher is the gate.** Token alone is not sufficient — the matcher does the semantic check.
266
303
  8. **Only the advance writes instance rows.** Anything else that wants an instance moved enqueues an advance and lets it re-check under the lock. A new settle-point write must pass `{ bumpSeq: true }` to `updateInstance` and a new job payload must carry the fence, or duplicate delivery stops being safe.
267
304
  9. **Fenced-out work is not an event.** Duplicate/stale jobs and pokes get a `clog.debug`, never a history row — with re-poking ticks they are routine, and history is an event log, not a trace.
305
+ 10. **SQL goes through sqlkit.** `sql` fragments and the `insert` / `update` builders, every value bound (`json()` for a jsonb column, `inList()` for a list, `raw()` only for text the code owns); transactions are `db.transaction(async (tx) => …)`. No `pg` import in `src/`. Persistence helpers take a `Driver`; the public entry points take `PgDb` and convert with `toPgDriver`.
306
+ 11. **Placing an enqueue is a durability decision.** Inside a `db.transaction()` callback it commits with the write (on a shared driver); after it, it does not. Don't move one across that line without re-reading [The Database Handle](#the-database-handle-sqlkit), and wrap any new userland callback that runs inside a transaction in `tx.savepoint()`.
268
307
 
269
308
  ## Before Making Changes
270
309
 
271
- - [ ] Check existing patterns in [src/](./src/) — small files, plain SQL, explicit transactions
272
- - [ ] Run `deno task test` (set `TEST_PG_*` env first; see `.env` example)
310
+ - [ ] Check existing patterns in [src/](./src/) — small files, SQL as sqlkit fragments, explicit `db.transaction()`
311
+ - [ ] Run `pnpm run ci` (set `TEST_PG_*` in `.env` first; see `.env.example`)
273
312
  - [ ] If adding a new history event type, add it to `HISTORY_EVENT` in [src/types.ts](./src/types.ts) and emit it consistently
274
313
  - [ ] If changing schema, write a new migration in [src/migrations/](./src/migrations/) — never edit `1_0_0.ts` in place
275
- - [ ] Changed the public API? Check [example/](./example) still compiles (`deno task check`) and still tells the truth — it is the one consumer in the repo
276
- - [ ] Touched `example/src/`? Run `deno task example:build` — `example/dist/bundle.js` is committed
314
+ - [ ] Changed the public API? Check [example/](./example) still compiles (`pnpm run typecheck`) and still tells the truth — it is the one consumer in the repo
315
+ - [ ] Touched `example/src/`? Run `pnpm run example:build` — `example/dist/bundle.js` is committed
277
316
 
278
317
  ## Dependency Versions
279
318
 
280
- | Package | Min | Why |
281
- | ---------------------- | --------- | ------------------------------------------------------------- |
282
- | `@marianmeres/fsm` | `^3.1.0` | Requires `state.meta` and `FSM.fromSnapshot` (added in 3.1.0) |
283
- | `@marianmeres/steve` | `^3.0.0` | Job queue (advance + effect dispatch) |
284
- | `@marianmeres/cron` | `^3.2.0` | Scheduler + correlator ticks |
285
- | `@marianmeres/migrate` | `^1.3.1` | Schema versioning |
286
- | `@marianmeres/clog` | `^3.21.0` | Logging |
287
- | `pg` | `^8.23.0` | PostgreSQL client |
319
+ | Package | Min | Why |
320
+ | ---------------------- | --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
321
+ | `@marianmeres/fsm` | `^3.1.0` | Requires `state.meta` and `FSM.fromSnapshot` (added in 3.1.0) |
322
+ | `@marianmeres/steve` | `^4.3.0` | Job queue (advance + effect dispatch), on sqlkit: shares the driver, and so the transactions |
323
+ | `@marianmeres/cron` | `^4.0.1` | Scheduler + correlator ticks, on sqlkit |
324
+ | `@marianmeres/sqlkit` | `^0.3.2` | Driver contract, fragments, `toPgDriver`, migrate wiring. `0.x`: a minor is breaking, so move it together with steve and cron — two copies mean two drivers per pool |
325
+ | `@marianmeres/migrate` | `^2.0.0` | The `Migrate` type `createMigrate` returns (sqlkit runs it) |
326
+ | `@marianmeres/clog` | `^3.23.1` | Logging |
327
+ | `pg` (dev only) | `^8.23.1` | Tests and the example build the pool; `src/` imports nothing from it |
288
328
 
289
329
  ## Failure Modes & Recovery
290
330
 
@@ -298,15 +338,17 @@ Rule: one `Jobs` and one `Cron` per process, and every process that _starts_ the
298
338
  | Definition missing on advance | Driver marks instance `failed`. (Pin-and-freeze: keep old versions registered until drained.) |
299
339
  | Crashed worker mid-job (advance or effect) | Steve's reaper marks the job `expired` — **terminal, never retried by steve**. `Workflow`'s `onDone` re-queues the identical payload instead, `redispatchLimit` dispatches in total (default 3), then fails the instance. Safe because the fence no-ops a re-dispatch whose original did commit. **Requires the reaper to run at all**: `new Jobs({ db, autoCleanup: true })`, or a periodic `jobs.cleanup()`. Without it the job sits in `running` forever and the instance never moves. |
300
340
  | Lost advance after effect (the side effect happened, `jobs.create('workflow.advance')` did not) | A throwing enqueue → steve retries the effect job; a dead process → the job expires and is re-dispatched. Either way the fence lets the retry through — the instance never left that `seq` — so the handler runs again and re-enqueues the advance. Handlers must be idempotent. |
301
- | Crash between `create()`'s commit and steve's job insert | The instance sits in `pending` with no job. The scheduler's second scan re-pokes it after `stalePendingSec` (default 300s; `0` disables). |
341
+ | A `pending` instance with no `start` job left to move it | `retry()` enqueues after its commit, so a crash in between leaves one. So does a `start` job consumed without effect: claimed by a handler-less `Jobs`, or (queue on a connection of its own) out of attempts before the creating transaction committed. `create()` itself cannot strand one: its job insert precedes its commit, and on a shared sqlkit driver is part of it. The scheduler's second scan re-pokes the row after `stalePendingSec` (default 300s; `0` disables). |
302
342
  | Duplicate advance / effect job | Fenced out by `seq`: debug-logged no-op, no history row. Duplicates are routine (every tick re-pokes what still looks due), so they are not events. |
303
343
  | Second `Jobs` / `Cron` without the workflow handlers | Silent loss — see [Runtime Ownership](#runtime-ownership). |
304
344
 
305
345
  ## The Example App
306
346
 
307
- [example/](./example) is consumer code, not a test: it imports `@marianmeres/workflow` by
308
- package name (mapped to `./src/mod.ts` in `deno.json`) and wires the runtime exactly the
309
- way the README does. It is also the fastest way to see the two orthogonal states, the
347
+ [example/](./example) is consumer code, not a test: it imports only the package's public
348
+ entry (`../src/mod.ts`, what `@marianmeres/workflow` resolves to) and wires the runtime
349
+ exactly the way the README does, on one `createPgDriver(pool)` handed to the migrations, the
350
+ `Jobs`, the `Cron` and the `Workflow`. Its HTTP server is `serve` from `@marianmeres/demino/node`
351
+ (a devDependency) around a plain `Request → Response` handler. It is also the fastest way to see the two orthogonal states, the
310
352
  timer and the poke model behave, so keep it working.
311
353
 
312
354
  Three things in it deviate from production defaults, all commented where they are set:
@@ -322,32 +364,47 @@ such: **listing instances and inbox rows with raw SQL** (the package exposes `fi
322
364
  `getHistory(id)` — anything that lists is an admin surface, i.e. your query), and nothing
323
365
  else. If you add a listing helper to `src/`, the example is where it should be used first.
324
366
 
325
- Touching `example/src/` means re-running `deno task example:build`; the bundle is
367
+ Touching `example/src/` means re-running `pnpm run example:build`; the bundle is
326
368
  committed so the app runs with no build step.
327
369
 
328
370
  ## Testing
329
371
 
330
- All tests run against a real PostgreSQL and skip gracefully when `TEST_PG_DATABASE` / `TEST_PG_USER` are unset — the driver is SQL-bound, so mocking it would test nothing. [stock_replenishment.test.ts](./tests/stock_replenishment.test.ts) covers happy/timeout/failure paths, [durability.test.ts](./tests/durability.test.ts) the fence and the recovery paths, [correlator.test.ts](./tests/correlator.test.ts) the signal semantics.
372
+ All tests run against a real PostgreSQL and skip gracefully when `TEST_PG_DATABASE` / `TEST_PG_USER` are unset — the driver is SQL-bound, so mocking it would test nothing. `pnpm test` runs the files one at a time (`--test-concurrency=1`, as `deno test` did), because every file drops and recreates the same tables. [stock_replenishment.test.ts](./tests/stock_replenishment.test.ts) covers happy/timeout/failure paths, [durability.test.ts](./tests/durability.test.ts) the fence and the recovery paths, [correlator.test.ts](./tests/correlator.test.ts) the signal semantics, [sqlkit.test.ts](./tests/sqlkit.test.ts) the `db` option and what a job insert does in each queue wiring.
373
+
374
+ Per-tick helpers `WorkflowScheduler.tickOnce()` and `WorkflowInboxCorrelator.tickOnce()` are exposed for tests so they don't have to wait for cron expressions. Durability behavior is generally testable without a running `Jobs`: enqueue nothing, call `runAdvance` / `runEffect` directly (they take the sqlkit driver: `toPgDriver(pool)`, or `workflow.db`) with a hand-built payload, and assert on the row + history. Assertions read through the raw `pg.Pool`, which sees only what was committed.
375
+
376
+ ## Tooling Conventions
331
377
 
332
- Per-tick helpers `WorkflowScheduler.tickOnce()` and `WorkflowInboxCorrelator.tickOnce()` are exposed for tests so they don't have to wait for cron expressions. Durability behavior is generally testable without a running `Jobs`: enqueue nothing, call `runAdvance` / `runEffect` directly with a hand-built payload, and assert on the row + history.
378
+ 1. **Runtime policy** — Node at the keyboard, npm as the only registry. No `Deno.*`,
379
+ `Bun.*`, `jsr:` or `npm:` specifiers in `src/`, `tests/`, `scripts/` or `example/`.
380
+ 2. **Erasable TypeScript** — no `enum`, `namespace`, constructor parameter properties;
381
+ `import type` for types; explicit `.ts` import extensions (rewritten to `.js` on emit).
382
+ Node strips types and never checks them: `pnpm run typecheck` is the only type gate.
383
+ `noUncheckedIndexedAccess` is off (Deno never had it, and the move off Deno changed no code).
384
+ 3. **Formatting** — tabs, 4 wide, 90 columns, double quotes (`biome.json`); manifests at 2
385
+ spaces. `pnpm run format` before committing. `docs/_archive/` (frozen) and the
386
+ generated/vendored `example/theme.css` / `reboot.css` are excluded.
387
+ 4. **Never run a script named `publish`** (npm lifecycle hook); publishing is `pnpm publish`,
388
+ run by `rp` / `rpm`.
333
389
 
334
390
  ## Glossary
335
391
 
336
- | Term | Meaning |
337
- | ----------------- | -------------------------------------------------------------------------------------------------------------------- |
338
- | Definition | Static, versioned, declarative shape of a workflow. Pure data |
339
- | Instance | Single running occurrence of a definition; one DB row |
340
- | Driver | The engine (`workflow.advance` steve handler) that advances instances |
341
- | Cursor | The instance's current node id |
342
- | Execution state | The instance's lifecycle status |
343
- | Outcome | The label a handler returns (or `MATCHED`/`TIMEOUT`/`ENTER`); fed to fsm as the event name |
344
- | Effect port | A typed interface to the outside world (email, AI, HTTP) — userland concern |
345
- | Inbox | Append-only `__workflow_inbox` intake of external signals |
346
- | Correlation token | Unique token that maps an external event back to a waiting instance |
347
- | Scheduler | Cron-driven loop that wakes time-suspended instances |
348
- | Correlator | Cron-driven loop that matches inbox signals to waiting instances |
349
- | Fence | The `seq` column. Bumped by every settle-point write; carried by every job, so a stale one is recognized and dropped |
350
- | Poke | A `workflow.advance` job enqueued by something that will not write the row itself. The advance re-checks and decides |
351
- | Settle point | The end of an advance: terminal / effectful / suspending. Where the instance row is written and `seq` bumped |
352
- | Defer | Correlator verdict: leave the inbox row unprocessed and re-examine it next tick |
353
- | Pin-and-freeze | Versioning strategy: every instance runs on its birth-version forever |
392
+ | Term | Meaning |
393
+ | ----------------- | ---------------------------------------------------------------------------------------------------------------------- |
394
+ | Definition | Static, versioned, declarative shape of a workflow. Pure data |
395
+ | Instance | Single running occurrence of a definition; one DB row |
396
+ | Driver | The engine (`workflow.advance` steve handler) that advances instances |
397
+ | sqlkit driver | The database handle: `Workflow.db`, a `Driver` from `@marianmeres/sqlkit`. Carries transactions in the calling context |
398
+ | Cursor | The instance's current node id |
399
+ | Execution state | The instance's lifecycle status |
400
+ | Outcome | The label a handler returns (or `MATCHED`/`TIMEOUT`/`ENTER`); fed to fsm as the event name |
401
+ | Effect port | A typed interface to the outside world (email, AI, HTTP) — userland concern |
402
+ | Inbox | Append-only `__workflow_inbox` intake of external signals |
403
+ | Correlation token | Unique token that maps an external event back to a waiting instance |
404
+ | Scheduler | Cron-driven loop that wakes time-suspended instances |
405
+ | Correlator | Cron-driven loop that matches inbox signals to waiting instances |
406
+ | Fence | The `seq` column. Bumped by every settle-point write; carried by every job, so a stale one is recognized and dropped |
407
+ | Poke | A `workflow.advance` job enqueued by something that will not write the row itself. The advance re-checks and decides |
408
+ | Settle point | The end of an advance: terminal / effectful / suspending. Where the instance row is written and `seq` bumped |
409
+ | Defer | Correlator verdict: leave the inbox row unprocessed and re-examine it next tick |
410
+ | Pin-and-freeze | Versioning strategy: every instance runs on its birth-version forever |
package/API.md CHANGED
@@ -10,10 +10,11 @@ Full public API for [`@marianmeres/workflow`](./README.md).
10
10
  - [WorkflowInboxCorrelator](#workflowinboxcorrelator)
11
11
  - [WorkflowRegistry](#workflowregistry)
12
12
  - [Functions](#functions)
13
- - [createMigrate](#creatempool)
13
+ - [createMigrate](#createmigratedb-options)
14
14
  - [validateDefinition](#validatedefinitiondef-available)
15
15
  - [defKey](#defkeyid-version)
16
16
  - [effectJobType](#effectjobtypename)
17
+ - [getHistory](#gethistorydb-instanceid-limit)
17
18
  - [Types](#types)
18
19
  - [WorkflowDefinition](#workflowdefinition)
19
20
  - [WorkflowFSMConfig](#workflowfsmconfig--workflowstateconfig)
@@ -49,7 +50,7 @@ new Workflow(options: WorkflowOptions)
49
50
 
50
51
  | Field | Type | Default | Description |
51
52
  | ---------------------------- | ------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
52
- | `db` | `pg.Pool` | required | PostgreSQL pool. Used for direct SQL (persistence helpers, transactions). |
53
+ | `db` | `PgDb` | required | A `pg.Pool`, a connected `pg.Client`, or a sqlkit `Driver` of the `pg` dialect. Give `jobs` the same one. See [The `db` option](#the-db-option). |
53
54
  | `jobs` | `Jobs` | required | Externally-owned `steve.Jobs` instance. Workflow registers its handlers on it via `setHandler`. The consumer runs `jobs.start()` / `jobs.stop()`. |
54
55
  | `tenantId` | `string` | `"_default"` | Multi-tenant scope. |
55
56
  | `definitions` | `WorkflowDefinition[]` | required | Definitions to register. Validated on construction. |
@@ -62,6 +63,14 @@ new Workflow(options: WorkflowOptions)
62
63
 
63
64
  Throws on construction if any definition references a handler/matcher not present in the maps, or if any other structural problem is found (unknown transition target, missing terminal state, etc.). See [`validateDefinition`](#validatedefinitiondef-available).
64
65
 
66
+ ##### The `db` option
67
+
68
+ `PgDb` is sqlkit's type for "a `pg.Pool`, a connected `pg.Client`, or a `Driver` of the `pg` dialect" (`@marianmeres/sqlkit/pg`). A pool or a client is turned into sqlkit's driver for it with `toPgDriver`, which keeps one driver per pool and per client; a `Driver` is taken as it is. Anything else is a `TypeError` from the constructor. The workflow never closes it.
69
+
70
+ The driver carries transactions in the calling context, so a `Jobs` given the same pool, client or driver shares the workflow's transactions: a job enqueued while one is open is part of it. `create()` then writes the instance and its first advance together, and an advance writes the row and the effect job it dispatches together. A `Jobs` with a database connection of its own inserts outside those transactions, and the `seq` fence covers the difference. The same holds one level up: `create()`, `appendInbox()`, `cancel()` and `retry()` called inside your own `db.transaction()` on that driver are part of it.
71
+
72
+ See [The database: a pool, a client or a driver](README.md#the-database-a-pool-a-client-or-a-driver) for what sharing a driver means in practice.
73
+
65
74
  **One `Workflow` per `Jobs`.** Steve's `setHandler` is last-writer-wins, so a second `Workflow` on the same `Jobs` would take over `workflow.advance` and fail the first one's instances with "Unknown workflow definition". Constructing it throws instead; call [`detach()`](#detach-void) on the incumbent first, or give the second one its own `Jobs`.
66
75
 
67
76
  The constructor also subscribes (`jobs.onDone`) to the terminal outcomes of `workflow.advance` and of each `workflow.effect.<name>` type:
@@ -75,7 +84,7 @@ Note that jobs only ever reach `expired` if something reaps them: run the `Jobs`
75
84
 
76
85
  | Name | Type | Description |
77
86
  | ---------- | ------------------ | -------------------------------------------------- |
78
- | `db` | `pg.Pool` | The provided pool. |
87
+ | `db` | `Driver` | The sqlkit driver for the provided `db`. |
79
88
  | `jobs` | `Jobs` | The injected `Jobs` instance. |
80
89
  | `tenantId` | `string` | Scope. |
81
90
  | `registry` | `WorkflowRegistry` | Read-only access to definitions/handlers/matchers. |
@@ -129,10 +138,12 @@ Removes everything the constructor registered on the `Jobs` instance — the `wo
129
138
 
130
139
  It does not stop the queue and does not touch running instances: jobs already enqueued stay in the table and fall back to steve's noop handler until something re-attaches. A no-op when this instance is not the attached one (already detached, or superseded).
131
140
 
132
- ##### `enqueueAdvance` / `enqueueEffect`
141
+ ##### `enqueueAdvance(payload)` / `enqueueEffect(handler, payload)`
133
142
 
134
143
  Internal `JobEnqueuer` interface methods used by the driver. Public so the scheduler and correlator can call them. You normally don't call these directly.
135
144
 
145
+ Neither takes a database handle: where the job insert lands is decided by where the call is made. Inside a `db.transaction()` callback, on a `Jobs` that shares the workflow's driver, the job is part of that transaction; anywhere else it is a statement of its own.
146
+
136
147
  ---
137
148
 
138
149
  ### `WorkflowScheduler`
@@ -277,20 +288,22 @@ Validates every definition against the registered handler/matcher names. Throws
277
288
 
278
289
  ## Functions
279
290
 
280
- ### `createMigrate(pool, options?)`
291
+ ### `createMigrate(db, options?)`
281
292
 
282
293
  Builds a `@marianmeres/migrate` instance pre-loaded with the package's schema versions. The caller drives it (`.up('latest')`, `.down(...)`, `.status()`, etc.).
283
294
 
295
+ It is sqlkit's migrate wiring over `db`: a run holds an advisory lock, so two processes booting at once wait for each other, and each version is applied in a transaction of its own, together with its version mark.
296
+
284
297
  ```typescript
285
298
  import { createMigrate } from "@marianmeres/workflow";
286
299
  ```
287
300
 
288
301
  **Parameters:**
289
302
 
290
- - `pool` (`pg.Pool`) — required
303
+ - `db` (`PgDb`) — required: a `pg.Pool`, a connected `pg.Client`, or a sqlkit `Driver` of the `pg` dialect. Anything else is a `TypeError`. Never closed here.
291
304
  - `options.logger` (`(...args: unknown[]) => void`, optional) — passed through to `Migrate`
292
305
 
293
- **Returns:** `Migrate` — see [`@marianmeres/migrate`](https://jsr.io/@marianmeres/migrate)
306
+ **Returns:** `Migrate<MigrateContext>` — see [`@marianmeres/migrate`](https://www.npmjs.com/package/@marianmeres/migrate). Workers receive `{ db }`, the driver.
294
307
 
295
308
  **Example:**
296
309
 
@@ -299,7 +312,7 @@ const migrate = createMigrate(pool);
299
312
  await migrate.up("latest");
300
313
  ```
301
314
 
302
- Active version is stored in `__workflow_migrations`. `1.0.0` creates `__workflow_instances`, `__workflow_inbox`, `__workflow_history` and their indexes; `1.1.0` renames `project_id` → `tenant_id`; `1.2.0` adds the `seq` fencing column and the partial index behind the scheduler's stale-`pending` scan. Upgrading an existing 2.0.x database needs `up("latest")` once — until then instances have `seq = 0` and every write bumps it from there.
315
+ Active version is stored in `__workflow_migrations`. `1.0.0` creates `__workflow_instances`, `__workflow_inbox`, `__workflow_history` and their indexes; `1.1.0` renames `project_id` → `tenant_id`; `1.2.0` adds the `seq` fencing column and the partial index behind the scheduler's stale-`pending` scan. Upgrading an existing 2.0.x database needs `up("latest")` once — until then instances have `seq = 0` and every write bumps it from there. A `__workflow_migrations` table created by an earlier release is read and continued as it is.
303
316
 
304
317
  ---
305
318
 
@@ -346,13 +359,13 @@ Builds the steve job-type string for an effect handler.
346
359
 
347
360
  ---
348
361
 
349
- ### `getHistory(exec, instanceId, limit?)`
362
+ ### `getHistory(db, instanceId, limit?)`
350
363
 
351
364
  Reads the append-only history rows for an instance in chronological order. Useful for `/admin/workflows`-style observability views.
352
365
 
353
366
  **Parameters:**
354
367
 
355
- - `exec` (`pg.Pool | pg.PoolClient | pg.Client`)
368
+ - `db` (`PgDb`) — a `pg.Pool`, a connected `pg.Client`, or a sqlkit `Driver` of the `pg` dialect. Called inside a `db.transaction()` callback of the same driver, the read is part of that transaction.
356
369
  - `instanceId` (`string`)
357
370
  - `limit` (`number`, optional) — Default: `200`
358
371
 
@@ -644,6 +657,7 @@ const HISTORY_EVENT = {
644
657
  - [README.md](./README.md) — overview, install, end-to-end example
645
658
  - [AGENTS.md](./AGENTS.md) — architectural deep-dive for coding agents
646
659
  - [`@marianmeres/fsm`](https://jsr.io/@marianmeres/fsm) — the underlying state-graph library
647
- - [`@marianmeres/steve`](https://jsr.io/@marianmeres/steve) — the underlying job queue
648
- - [`@marianmeres/cron`](https://jsr.io/@marianmeres/cron) — the underlying cron scheduler
649
- - [`@marianmeres/migrate`](https://jsr.io/@marianmeres/migrate) — the underlying migration runner
660
+ - [`@marianmeres/steve`](https://www.npmjs.com/package/@marianmeres/steve) — the underlying job queue
661
+ - [`@marianmeres/cron`](https://www.npmjs.com/package/@marianmeres/cron) — the underlying cron scheduler
662
+ - [`@marianmeres/sqlkit`](https://www.npmjs.com/package/@marianmeres/sqlkit) — the SQL driver layer: fragments, transactions, typed errors
663
+ - [`@marianmeres/migrate`](https://www.npmjs.com/package/@marianmeres/migrate) — the underlying migration runner