@marianmeres/workflow 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/AGENTS.md ADDED
@@ -0,0 +1,240 @@
1
+ # @marianmeres/workflow — Agent Guide
2
+
3
+ ## Quick Reference
4
+
5
+ - **Stack**: Deno/Node.js, PostgreSQL, TypeScript
6
+ - **Test**: `deno task test` (requires `TEST_PG_*` env; see [tests/_pg.ts](./tests/_pg.ts))
7
+ - **Build (npm)**: `deno task npm:build`
8
+ - **Entry point**: [src/mod.ts](./src/mod.ts)
9
+
10
+ ## Package Overview
11
+
12
+ - **Name**: `@marianmeres/workflow`
13
+ - **Type**: Durable workflow / orchestration framework
14
+ - **Runtime**: Deno and Node.js (via npm build)
15
+ - **DB**: PostgreSQL only (uses `pg` driver)
16
+ - **License**: MIT
17
+
18
+ ## Purpose
19
+
20
+ Long-lived, persisted FSM instances with durable wake-ups (timers) and external-signal correlation (inbox + matchers). Definitions are pure data, versioned per-instance. Effects run as steve jobs (transactional outbox). Built on top of `@marianmeres/fsm` (state graph), `@marianmeres/steve` (effect execution), `@marianmeres/cron` (scheduler & correlator ticks), `@marianmeres/migrate` (schema), `@marianmeres/clog` (logging).
21
+
22
+ **Ships no domain code.** Userland brings the definitions, handlers, and matchers.
23
+
24
+ ## Architecture
25
+
26
+ ```
27
+ ┌─ User application ─────────────────────────────────────────────────┐
28
+ │ WorkflowDefinition (pure data) + Handlers + Matchers │
29
+ │ steve.Jobs + cron.Cron — consumer-owned, consumer calls start/stop│
30
+ └────────────────────────────┬───────────────────────────────────────┘
31
+ ▼
32
+ ┌─ @marianmeres/workflow ────────────────────────────────────────────┐
33
+ │ Workflow WorkflowScheduler WorkflowInboxCorrelator │
34
+ │ setHandler(advance, register() attaches register() attaches │
35
+ │ effect.*) on Jobs tick on Cron tick on Cron │
36
+ │ ▼ ▼ ▼ │
37
+ │ __workflow_instances · __workflow_inbox · __workflow_history │
38
+ └────────────────────────────┬───────────────────────────────────────┘
39
+ ▼
40
+ PostgreSQL
41
+ ```
42
+
43
+ The framework owns no runtime. It attaches handlers to a `Jobs` instance and ticks to a `Cron` instance that the consumer constructs and starts. This means the consumer's other job types / cron jobs can coexist on the same shared runtime (one pollers per process instead of N).
44
+
45
+ ```
46
+ src/
47
+ ├── mod.ts # Public exports
48
+ ├── types.ts # Definition, Instance, NodeMeta, Handler, Matcher, ...
49
+ ├── definition.ts # Definition validator + defKey()
50
+ ├── registry.ts # WorkflowRegistry (validates on construction)
51
+ ├── driver.ts # runAdvance / runEffect — the core dispatch logic
52
+ ├── workflow.ts # Workflow class (attaches handlers to injected Jobs)
53
+ ├── scheduler.ts # WorkflowScheduler (attaches wake tick to injected Cron)
54
+ ├── correlator.ts # WorkflowInboxCorrelator (attaches match tick to injected Cron)
55
+ ├── log.ts # createClog("workflow")
56
+ ├── persistence/
57
+ │ ├── instances.ts # CRUD + lock + claim due wake-ups + find by correlation
58
+ │ ├── inbox.ts # append, claim-unprocessed, mark-processed
59
+ │ ├── history.ts # append + read
60
+ │ └── tx.ts # withTransaction helper
61
+ └── migrations/
62
+ ├── 1_0_0.ts # CREATE/DROP three tables
63
+ └── index.ts # createMigrate(pool) — @marianmeres/migrate setup
64
+ tests/
65
+ ├── _pg.ts # env-driven pool + schema reset
66
+ ├── fixtures/
67
+ │ └── stock-replenishment.ts # reference workflow used by tests only
68
+ └── stock_replenishment.test.ts # happy/timeout/fail paths + validator
69
+ ```
70
+
71
+ ## Conceptual Model — Three Layers
72
+
73
+ | Layer | What it is | Lives in |
74
+ |---|---|---|
75
+ | **Definition** | Static graph: nodes, transitions, handler ids (strings). Pure data, JSON-safe. | Code, registered in `WorkflowRegistry` |
76
+ | **Instance** | Live state of one running occurrence. Cursor, context, execution_state, wake_at, correlation_token. | `__workflow_instances` row |
77
+ | **Driver** | Event-driven engine that advances instances. Owns no state. | `workflow.advance` steve job |
78
+
79
+ ## Two Orthogonal States
80
+
81
+ Never conflate. Always two columns on `__workflow_instances`:
82
+
83
+ | Column | Meaning | Source of truth for |
84
+ |---|---|---|
85
+ | `cursor` | Which node am I at? | The FSM's `state` (`getSnapshot().state`) |
86
+ | `execution_state` | What's my lifecycle status? | `pending` / `running` / `waiting` / `completed` / `failed` / `cancelled` |
87
+
88
+ An instance can be at node `await_reply` (cursor) and simultaneously `waiting` (execution_state).
89
+
90
+ ## Node Taxonomy (`NodeMeta`)
91
+
92
+ Each FSM state's `meta` field carries a discriminated `NodeMeta`. The driver dispatches on `meta.kind`:
93
+
94
+ | Kind | Driver behavior | Required fields |
95
+ |---|---|---|
96
+ | `pure` | Auto-fire `ENTER` event; transition by guards. Loop inline until terminal/effectful/suspending. Max 64 hops. | none |
97
+ | `effectful` | Enqueue `workflow.effect.<handler>` steve job, flip to `running`. On completion, an `advance` is enqueued with `{ outcome, outcome_data }`. | `handler: string` |
98
+ | `suspending` | Set `execution_state='waiting'`, `wake_at` (if `timeoutSec`), keep `correlation_token`. Wakes via scheduler (`TIMEOUT`) or correlator (`MATCHED`). | optional `matcher`, optional `timeoutSec` |
99
+ | `terminal` | Set `execution_state='completed'`. | none |
100
+
101
+ ## Public API (from `src/mod.ts`)
102
+
103
+ | Export | Type | Purpose |
104
+ |---|---|---|
105
+ | `Workflow` | class | Main driver: attaches handlers to an injected `steve.Jobs`. Exposes `create/find/appendInbox/enqueueAdvance/enqueueEffect`. **No own start/stop** — consumer runs the Jobs. |
106
+ | `WorkflowScheduler` | class | Attaches a wake tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
107
+ | `WorkflowInboxCorrelator` | class | Attaches a match tick to an injected `cron.Cron`. Exposes `register/unregister/tickOnce`. **No own start/stop** — consumer runs the Cron. |
108
+ | `WorkflowRegistry` | class | Immutable registry of definitions + handlers + matchers; validates on construction |
109
+ | `validateDefinition` | function | Standalone validator usable outside `WorkflowRegistry` |
110
+ | `defKey(id, version)` | function | `"id@version"` registry key helper |
111
+ | `createMigrate(pool)` | function | Builds `@marianmeres/migrate` instance pre-loaded with schema versions |
112
+ | `getHistory(exec, id, limit?)` | function | Reads `__workflow_history` rows for an instance (observability) |
113
+ | `effectJobType(name)` | function | Steve job-type string for an effect handler: `workflow.effect.<name>` |
114
+ | `JOB_TYPE_ADVANCE` | const | `"workflow.advance"` |
115
+ | `JOB_TYPE_EFFECT_PREFIX` | const | `"workflow.effect."` |
116
+ | `PURE_ENTER_EVENT` | const | `"ENTER"` — synthetic event fired by driver into pure nodes |
117
+ | `DEFAULT_PROJECT_ID` | const | `"_default"` |
118
+ | `EXECUTION_STATE` | const | `{ PENDING, RUNNING, WAITING, COMPLETED, FAILED, CANCELLED }` |
119
+ | `HISTORY_EVENT` | const | Audit event-type strings (`created`, `transition`, `effect_dispatched`, ...) |
120
+ | `NodeMeta` | type | Discriminated union: `pure` \| `effectful` \| `suspending` \| `terminal` |
121
+ | `WorkflowDefinition` | type | `{ id, version, fsm: FSMConfig }` |
122
+ | `Handler` | type | `(args: HandlerArgs) => Promise<HandlerResult> \| HandlerResult` |
123
+ | `Matcher` | type | `(args: MatcherArgs) => boolean \| Promise<boolean>` |
124
+ | `WorkflowInstanceRow` | type | Schema-aligned row type |
125
+ | `InboxRow` | type | Schema-aligned row type |
126
+ | `HistoryRow` | type | Schema-aligned row type |
127
+ | `WorkflowContext` | type | `Record<string, unknown>` — accumulated payload |
128
+ | `WorkflowSnapshot` | type | `FSMSnapshot<string, WorkflowContext>` |
129
+
130
+ See [API.md](./API.md) for full signatures, options, and examples.
131
+
132
+ ## Outcome Convention
133
+
134
+ The driver passes outcome labels as fsm events. Three labels have special meaning:
135
+
136
+ | Label | Source | Used by |
137
+ |---|---|---|
138
+ | `ENTER` | Driver (auto) | Pure nodes — wire guarded transitions on this event |
139
+ | `MATCHED` | `WorkflowInboxCorrelator` | Suspending nodes whose matcher returns true |
140
+ | `TIMEOUT` | `WorkflowScheduler` | Suspending nodes whose `wake_at` fires |
141
+
142
+ Effectful node outcomes are whatever the handler returns (e.g. `LOW`, `OK`, `CONFIRMED`).
143
+
144
+ ## DB Schema (managed by `createMigrate`)
145
+
146
+ | Table | Purpose | Indexes |
147
+ |---|---|---|
148
+ | `__workflow_instances` | Live instance rows | `(project_id, execution_state, wake_at)` partial, `(project_id, execution_state, correlation_token)` partial |
149
+ | `__workflow_inbox` | Append-only intake of external signals | `(project_id, correlation_token)` partial WHERE unprocessed |
150
+ | `__workflow_history` | Append-only audit log | `(instance_id, at)` |
151
+ | `__workflow_migrations` | Active migration version | — |
152
+
153
+ Steve and Cron own their own tables (`__job*`, `__cron*`); managed by their respective libraries.
154
+
155
+ ## Runtime Ownership
156
+
157
+ The framework **does not start or stop steve.Jobs or cron.Cron.** The consumer constructs both, passes them in, and runs their lifecycle:
158
+
159
+ ```ts
160
+ const jobs = new Jobs({ db: pool }); // consumer-owned
161
+ const cron = new Cron({ db: pool }); // consumer-owned
162
+
163
+ const wf = new Workflow({ db: pool, jobs, ... }); // setHandler on jobs
164
+ const scheduler = new WorkflowScheduler({ cron, workflow: wf }); // register tick on cron
165
+ const correlator = new WorkflowInboxCorrelator({ cron, workflow: wf });
166
+ await scheduler.register();
167
+ await correlator.register();
168
+
169
+ await jobs.start(4); // consumer
170
+ await cron.start(2); // consumer
171
+
172
+ // ... later ...
173
+ await cron.stop(); // consumer
174
+ await jobs.stop(); // consumer
175
+ ```
176
+
177
+ The consumer's other steve job types and cron jobs coexist on the same `Jobs` / `Cron` instances — handlers are keyed by `type` (steve) / `name` (cron), and the workflow's are namespaced (`workflow.advance`, `workflow.effect.<name>`, `workflow.scheduler.<projectId>`, `workflow.correlator.<projectId>`) so collisions are unlikely. Running parallel `Jobs` instances on the same `__job` table is **unsafe**: an instance polling a type it doesn't know falls back to a noop handler that silently marks the job completed (see [steve/jobs.ts:499](file:///Users/mm/projects/@marianmeres/steve/src/steve/jobs.ts)). Always share.
178
+
179
+ ## Critical Conventions
180
+
181
+ 1. **Definitions are pure data.** Handlers and matchers are referenced by string name. Never store function pointers in a definition. Litmus test: `JSON.stringify(def)` must lose no meaning.
182
+ 2. **Version-pin instances.** Every instance carries `(definition_id, definition_version)`. Never edit an in-use definition in place — bump the version. Old versions stay registered until the last instance using them completes.
183
+ 3. **Handlers must be idempotent.** Steve may retry handlers if a worker crashes mid-execution. Side effects must be safe to repeat.
184
+ 4. **No fsm `onEnter`/`onExit` hooks for side effects.** Driver re-constructs FSMs via `FSM.fromSnapshot()`, which skips `onEnter`. Side effects belong in effectful handlers — they're the only way to get durable, retryable, observable execution.
185
+ 5. **Cursor + execution_state are separate columns.** Don't fold one into the other.
186
+ 6. **Outcome labels are fsm events.** The driver calls `fsm.transition(outcome, data)`. If fsm rejects (unknown event in current state), the instance is marked `failed` and history records `transition_rejected`.
187
+ 7. **Correlation token is the index, matcher is the gate.** Token alone is not sufficient — the matcher does the semantic check.
188
+
189
+ ## Before Making Changes
190
+
191
+ - [ ] Check existing patterns in [src/](./src/) — small files, plain SQL, explicit transactions
192
+ - [ ] Run `deno task test` (set `TEST_PG_*` env first; see `.env` example)
193
+ - [ ] If adding a new history event type, add it to `HISTORY_EVENT` in [src/types.ts](./src/types.ts) and emit it consistently
194
+ - [ ] If changing schema, write a new migration in [src/migrations/](./src/migrations/) — never edit `1_0_0.ts` in place
195
+
196
+ ## Dependency Versions
197
+
198
+ | Package | Min | Why |
199
+ |---|---|---|
200
+ | `@marianmeres/fsm` | `^3.1.0` | Requires `state.meta` and `FSM.fromSnapshot` (added in 3.1.0) |
201
+ | `@marianmeres/steve` | `^2.0.2` | Job queue (advance + effect dispatch) |
202
+ | `@marianmeres/cron` | `^2.0.1` | Scheduler + correlator ticks |
203
+ | `@marianmeres/migrate` | `^1.2.2` | Schema versioning |
204
+ | `@marianmeres/clog` | `^3.21.0` | Logging |
205
+ | `pg` | `^8.20.0` | PostgreSQL client |
206
+
207
+ ## Failure Modes & Recovery
208
+
209
+ | Failure | Behavior |
210
+ |---|---|
211
+ | Handler throws | Steve retries per `effectMaxAttempts`. On terminal failure, `Workflow`'s `onDone` listener marks the instance `failed` + appends `effect_failed` history. |
212
+ | fsm rejects outcome | Driver appends `transition_rejected` history, marks instance `failed`. |
213
+ | Pure node with no matching guard | Driver appends `transition_rejected` history, marks instance `failed`. |
214
+ | Pure-state cycle (>64 hops) | Driver appends `failed` history with "exceeded MAX_PURE_HOPS" reason. |
215
+ | Definition missing on advance | Driver marks instance `failed`. (Pin-and-freeze: keep old versions registered until drained.) |
216
+ | Crashed worker mid-effect | Steve auto-cleanup reaps stuck `running` jobs. Steve retries per `max_attempts`. |
217
+ | Lost advance after effect (process dies between handler success and `jobs.create('workflow.advance')`) | Steve retries the effect job; handler runs again; advance enqueued. Handlers must be idempotent. |
218
+
219
+ ## Testing
220
+
221
+ Integration tests in [tests/stock_replenishment.test.ts](./tests/stock_replenishment.test.ts) cover happy, timeout, and failure paths against a real PostgreSQL. Skip gracefully when `TEST_PG_DATABASE` / `TEST_PG_USER` are unset.
222
+
223
+ Per-tick helpers `WorkflowScheduler.tickOnce()` and `WorkflowInboxCorrelator.tickOnce()` are exposed for tests so they don't have to wait for cron expressions.
224
+
225
+ ## Glossary
226
+
227
+ | Term | Meaning |
228
+ |---|---|
229
+ | Definition | Static, versioned, declarative shape of a workflow. Pure data |
230
+ | Instance | Single running occurrence of a definition; one DB row |
231
+ | Driver | The engine (`workflow.advance` steve handler) that advances instances |
232
+ | Cursor | The instance's current node id |
233
+ | Execution state | The instance's lifecycle status |
234
+ | Outcome | The label a handler returns (or `MATCHED`/`TIMEOUT`/`ENTER`); fed to fsm as the event name |
235
+ | Effect port | A typed interface to the outside world (email, AI, HTTP) — userland concern |
236
+ | Inbox | Append-only `__workflow_inbox` intake of external signals |
237
+ | Correlation token | Unique token that maps an external event back to a waiting instance |
238
+ | Scheduler | Cron-driven loop that wakes time-suspended instances |
239
+ | Correlator | Cron-driven loop that matches inbox signals to waiting instances |
240
+ | Pin-and-freeze | Versioning strategy: every instance runs on its birth-version forever |