@powerhousedao/reactor-workflow 6.2.3-staging.0 → 6.3.0-dev.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -39,8 +39,9 @@ rather than restated here.
39
39
  contexts from `Store`, `ServerContext`, `FilesService`, `ConnectionsManager`,
40
40
  `FlowsContext`, `RunContext`, `TriggerHookContext` and `SetScheduleRequest`;
41
41
  the connection shapes from `AppConnectionType` and `AppConnectionValue`.
42
- `PackagePiece`, `ReactorService` and `DEDUPE_KEY_PROPERTY` are the framework's
43
- own Powerhouse half.
42
+ `PackagePiece`, `RequireReactor`, `ReactorClient`, `ReactorReadClient`, the
43
+ reactor error names and `DEDUPE_KEY_PROPERTY` are the framework's own
44
+ Powerhouse half.
44
45
  - The **enums stay strings here**. A piece bundle inlines its own copy of the
45
46
  framework, so a `PropertyType` or `TriggerStrategy` read off one shares no
46
47
  identity with ours. Every such value is compared as a string; nothing in
@@ -51,6 +52,10 @@ rather than restated here.
51
52
  `apfile://` refs, the size ceiling and a host-injected fetcher have no
52
53
  upstream equivalent. An `ApFile` a processor builds is flattened to a plain
53
54
  object at that boundary: a class instance does not survive the worker IPC.
55
+ Its `base64` is computed on first read, as upstream's getter is, so a piece
56
+ that only reads `data` holds one copy of the file. A `streaming: true` FILE
57
+ prop gets upstream's `ApStreamingFile` instead: `{ filename, extension,
58
+ size, body }`, with `body` read from the staged file or the network.
54
59
  - **Prop validation**, from the same place. Before an action's `run()` or any
55
60
  trigger hook but `onDisable`, a prop left unset takes its `defaultValue` and
56
61
  the coerced values go through the engine's `validateProperty`. A failure is
@@ -77,6 +82,94 @@ rather than restated here.
77
82
  the text of an HTML error page reach the run journal. Redaction runs last,
78
83
  over the formatter's output as well.
79
84
 
85
+ ## Placement: one reactor runs workflows
86
+
87
+ **Workflow execution is a singleton pinned to one reactor.** The engine forks
88
+ child processes, so no browser reactor can compose it; the hazard a guard is
89
+ needed for is two Node replicas over one run journal. It is not theoretical:
90
+ opening the journal fails every RUNNING run that is not in **this** process's
91
+ in-flight set, and starting the runtime admits every PENDING one — so a second
92
+ replica booting fails the first one's live runs, runs its queued ones a second
93
+ time, and then both arm every trigger and both poll it.
94
+
95
+ The guard is a durable claim on the journal's own database
96
+ (`reactor/singleton-lease.ts`, one row in `workflow_singleton`), taken by the host
97
+ **before** the runtime is built, and refused by name when another live process
98
+ holds it (`WorkflowSingletonConflictError`). The
99
+ `trigger_state.lease_owner` / `lease_expires_at` columns stay in the schema,
100
+ always null and unread: they were per trigger, the wrong granularity, since
101
+ the sweeps and the supervisor are per process.
102
+
103
+ - The lease is valid for 60s and renewed every 20s from the moment it is
104
+ claimed, so a slow compose cannot outlast it. A compose that fails after the
105
+ claim releases it, and so does `stop()`, so the next boot owns workflows
106
+ immediately instead of waiting out the TTL.
107
+ - An **expired** lease is taken over: a killed process does not lock workflows
108
+ out until a human intervenes. A claim under the holder's own owner name may
109
+ take it over earlier, once the holder's heartbeat has been silent for two
110
+ renewal periods (40s), but never from a holder that is still renewing: the
111
+ new process opening the journal would fail the old one's live runs.
112
+ - Each claim records a random **instance** token, never configured. Heartbeat
113
+ and release match on owner and instance, so during a rolling deploy under
114
+ one stable owner name the old process can neither renew nor delete the new
115
+ process's lease.
116
+ - A heartbeat that finds the lease held by another claim logs an **error**
117
+ naming the holder, stops renewing and calls `onLost`. The host shuts the
118
+ runtime down: no trigger, webhook or manual run starts after that, and the
119
+ host reports its triggers unavailable. It does not re-claim; workflows come
120
+ back on the next boot, since re-arming needs a fresh compose.
121
+ - A firing the shutdown refuses, whether queued for its concurrency slot or
122
+ still on its way in, is recorded once. Every trigger firing (an operation, a
123
+ piece trigger's items, a schedule slot, a webhook delivery) is journaled as a
124
+ PENDING run with its payload before it fires. If its source will not deliver
125
+ it again, the refusal leaves that run PENDING, and the next owner starts it;
126
+ a manual fire's caller hears the refusal, so its run is failed. If it will (a sync
127
+ webhook answered 503 for the sender to retry, or piece items whose cursor the
128
+ stopped trigger put back), the run is cancelled as redelivered. A firing
129
+ refused before its run was written is left to its source: the operation is
130
+ replayed, the sender retries, the schedule slot stays due, the trigger's
131
+ cursor is put back or never moved. A close-out write that fails leaves the
132
+ run PENDING, and the next owner starts it. Without a
133
+ working journal a firing that still runs is unrecorded, and its refusal is
134
+ logged as an error.
135
+ - A renewal that fails or hangs is retried every 5s. A holder that has gone
136
+ 30s without a renewal it knows landed reports itself lost
137
+ (`reason: "unrenewable"`), before a stale same-owner claim (40s) or expiry
138
+ (60s) could take the lease: journal writes are best-effort, so it would
139
+ otherwise keep running workflows beside the next owner. Switchboard then
140
+ goes down through its fatal shutdown, since nobody else runs workflows,
141
+ and the restarted process re-claims; an embedded host without one stays
142
+ without workflows and logs that a restart is needed.
143
+ - Over a journal no other process can open (embedded PGlite) nothing can take
144
+ the lease, so the holder never fences itself; a stalled renewal is logged.
145
+ - Nothing fences the journal itself: writes do not check the lease, so a
146
+ process keeps writing until it notices the loss.
147
+ - **A refused claim does not take the API down.** The host boots WITHOUT the
148
+ workflow runtime and warns, naming the current owner: no trigger fires here
149
+ and the workflow GraphQL face is absent, while inspection, GraphQL, sync, MCP
150
+ and every drive serve normally. The one thing this process must not do is run
151
+ workflows against a journal a live process owns; aborting the whole boot over
152
+ it turned "not allowed to run one component" into an outage — and, with a
153
+ random owner name, into a crash loop for the TTL after every unclean kill.
154
+ The host retries the claim every 20s and starts workflows once it succeeds,
155
+ so a rolling deploy whose slots have different owner names does not end
156
+ with nobody running them. This is only for a host that never composed: one
157
+ that lost the lease after composing stays without workflows until restart.
158
+ - **The default owner is stable**: `<hostname>/<fingerprint of the journal's
159
+ storage location>`. So one deployment slot restarting re-claims its OWN lease
160
+ once the killed process's heartbeat is stale (40s) rather than waiting out
161
+ the whole TTL. A genuine second replica still differs
162
+ by hostname or by the journal it points at. The storage location is hashed,
163
+ never printed: it can be a Postgres URL with credentials, and the owner name
164
+ goes into a database row and every log line about the lease. The case a stable
165
+ name cannot separate is two processes on ONE host over ONE journal, which is a
166
+ misconfiguration those two already share — and it is not silent: the loser's
167
+ heartbeat finds the lease taken and says so by name.
168
+ - `PH_WORKFLOWS_SINGLETON_OWNER` is still the operator's contract and overrides
169
+ the derived name. Set it per deployment slot when the hostname is not stable
170
+ (a fresh container id each deploy) or when two slots share a journal on
171
+ purpose.
172
+
80
173
  ## How the host composes it
81
174
 
82
175
  The engine names no host type. `WorkflowRuntimeHostDeps` (`src/reactor/host.ts`)
@@ -101,10 +194,68 @@ next boot instead of vanishing; a fresh registration starts at head, so history
101
194
  is never replayed. `onOperations` journals a matched fire before it returns, so
102
195
  the cursor never passes an event that is not yet durable.
103
196
 
197
+ **A fire that crashes the reactor is bounded.** The durable cursor is what
198
+ makes an operation written while the runtime was down catch up — and it is also
199
+ what re-delivers, on every boot, an operation whose fire takes the process down
200
+ before anything is journaled (the EPIPE boot loop). So the dedupe row counts
201
+ **deliveries**, committed before the risky work, which is the only way a crash
202
+ that leaves nothing behind can be counted at all:
203
+
204
+ - A delivery whose claim already holds a run id is an ordinary duplicate and is
205
+ suppressed, as before. So is one whose key fired with no run linked — run
206
+ without a journal row, its run erased, or written before claims existed —
207
+ which holds the `FIRED_WITHOUT_RUN_ID` marker rather than NULL.
208
+ - A delivery whose claim holds **no** run id is retried: the previous attempt
209
+ died before it journaled anything, and losing a legitimate trigger to a
210
+ transient store failure would be worse than the loop.
211
+ - Past `FIRE_CRASH_BUDGET` (3) such deliveries the fire is **abandoned**, with
212
+ a FAILED run naming the loop — visible, and rerunnable once the cause is
213
+ fixed, instead of a reactor that crashes on every boot and says nothing. The
214
+ FAILED run is linked to the key, so later deliveries are duplicates.
215
+
216
+ The **count** and the **claim** are deliberately different writes. Counting is
217
+ its own committed transaction, because a delivery that leaves nothing behind
218
+ has to be countable. Claiming is `run_id`, set under a `WHERE run_id IS NULL`
219
+ guard in the SAME transaction as the run row: the guard takes the row's lock,
220
+ so of two concurrent deliveries of one operation the second blocks until the
221
+ first commits and then matches no row — exactly one delivery fires, and a crash
222
+ in between rolls the claim back with the run it failed to journal. Were the
223
+ bump to ride along inside the claim transaction, a run insert that takes the
224
+ process down would roll the count back too and the budget could never reach its
225
+ limit.
226
+
227
+ Log writes on the piece-log and run-failure paths are truncated before the
228
+ write (`MAX_LOG_LINE_CHARS`): a piece error carrying an HTML error page is a
229
+ multi-megabyte write to a pipe that may be blocked, and bounding the write is
230
+ cheaper than handling the throw.
231
+
104
232
  A workflow document's `DELETE_DOCUMENT` disarms it as disabling does: its
105
233
  deliveries stop once the operation is indexed, a piece trigger's `onDisable`
106
234
  runs, and then its trigger row, its FLOW `ctx.store` partition, its webhook
107
- token and its dedupe keys are deleted. A restart finds nothing to re-arm.
235
+ token and its dedupe keys are deleted. A restart finds nothing to re-arm. Its
236
+ `PURGE_DOCUMENT` marker does the same whether or not the deletion was seen, and
237
+ also drops the trigger-test store partitions; the read model awaits it, so a
238
+ failure holds the cursor below the marker.
239
+
240
+ A document's purge (`PURGE_DOCUMENT`) deletes every run that carried it: the
241
+ runs `run_document` ties to it, the runs whose trigger payload names it as
242
+ `documentId`, `driveId` or `parentId`, the test runs whose sample (a step's
243
+ `output`, a payload or a list of them) names it the same way, a purged
244
+ workflow's own runs (`run.workflow_id`, test runs included), and every rerun
245
+ of those, transitively. Their
246
+ `step_execution` and `run_document` rows go with them; a `trigger_dedupe` row
247
+ they claimed keeps its key, with `run_id` set to the `FIRED_WITHOUT_RUN_ID`
248
+ marker so a redelivery is still a duplicate. A run erased while it is
249
+ still executing journals nothing more from this process. The read model's
250
+ fence is `"skip"`: the journal lives on the relational handle, not the
251
+ reactor's, so a run the purge races can still be written after the marker was
252
+ applied; a rescan from just below the marker repairs it.
253
+
254
+ Not erased, because nothing ties it to a document: another workflow's
255
+ `trigger_state.store_state` (vestigial, but an unmigrated legacy blob may
256
+ remain), `trigger_dedupe.dedupe_key` (an operation key, which can embed a
257
+ document id) and `piece_store.value` (whatever a piece stored), and a test
258
+ run's step output that names the id anywhere but those payload fields.
108
259
 
109
260
  `runsPage` pages newest first on when a run was journaled (`enqueued_at`),
110
261
  which starting a PENDING run leaves alone, so a run keeps its place between
@@ -120,6 +271,47 @@ transport finds that child's code by walking up to this package's own
120
271
  `package.json` and reading `dist/worker-entry.js`, so `pnpm build` must have run
121
272
  before anything executes a piece — including the suites here.
122
273
 
274
+ Children come from one warm pool (`worker/pool.ts`, ADR 0007). Every piece
275
+ request leases an idle child, preferring one that has the piece loaded, and
276
+ returns it when it settles: a step, a trigger hook, an editor's options or
277
+ describe. A run holds no child between steps. Trigger hooks have children of
278
+ their own, so a burst of runs can't delay a poll, and so does the editor (one
279
+ child, which exits when idle). A child is retired when it
280
+ dies or times out, past 80% of
281
+ its heap budget, after `PH_WORKFLOWS_WORKER_MAX_REQUESTS` requests, or after a
282
+ minute idle above `PH_WORKFLOWS_WORKER_MIN`.
283
+
284
+ Piece code is trusted: what a piece keeps in module state carries over to the
285
+ next request its child serves, whichever workflow sent it.
286
+
287
+ Certificate checks stay on in every child (`worker/tls-guard.ts`). Pieces
288
+ bundled with an old pieces-common set `NODE_TLS_REJECT_UNAUTHORIZED=0` before
289
+ every request, which would switch verification off process-wide; the guard
290
+ gives every `tls.connect` an explicit `rejectUnauthorized: true` unless the
291
+ caller explicitly asks for `false`, so fetch, https and the HTTP clients
292
+ pieces bundle all verify. The worker still reports a request that set the
293
+ variable (`tlsPoisoned`), but no longer retires the child for it.
294
+
295
+ That entry is `src/worker/entry.ts`: the piece worker from `src/pieces`, plus
296
+ `ctx.reactor` (`src/worker/reactor.ts`), which needs `@powerhousedao/reactor`
297
+ and so lives outside the piece layer. For an action, trigger or option resolver
298
+ that declares `requireReactor`, the host serves one `ReactorHostServer` per
299
+ request over the child's IPC channel, as `{ type: "reactor-rpc", requestId,
300
+ message }`; the worker builds a reactor RPC proxy per request and closes it when
301
+ the request settles. The boot document models reach the child on fork as
302
+ `{ type: "model-manifest", entries }`. A type the host loaded later is looked
303
+ up with a `model-entries` host call the first time a piece asks for it. Either
304
+ way the model is imported on first use.
305
+
306
+ `ctx.reactor.query` is served the same way, by the host's optional
307
+ `queryIndex` dep (`WorkflowRuntimeHostDeps`): a read of a read-model index the
308
+ host registers as a GraphQL subgraph, run in the host process as the run user,
309
+ so the index's own access rules decide as they do over HTTP. A host that hands
310
+ over no `queryIndex` refuses the method by name, and a run with no user is
311
+ refused under enforcement like every other read. Switchboard answers it from
312
+ the GraphQL manager (`executeSubgraph`), so a piece never needs the
313
+ switchboard's URL or a bearer of its own to read an index.
314
+
123
315
  ## Blocks
124
316
 
125
317
  A step names its block the way Activepieces does, with three fields:
@@ -132,7 +324,10 @@ action at another version is the same block.
132
324
  ### The core piece
133
325
 
134
326
  The engine's own blocks are the built-in piece `@powerhousedao/piece-core`, in
135
- `src/pieces/core/`. Its actions are `branch` and `assert`; its triggers are
327
+ `src/pieces/core/`. Its actions are `branch`, `router`, `loop` ("Loop on
328
+ items"), `assert`, `delay`,
329
+ `approval` ("Request approval"), `stop` ("Stop run") and `waitForWebhook`
330
+ ("Respond and wait for webhook"); its triggers are
136
331
  `manual`, `schedule` and `webhook`. It is written with `createPiece`,
137
332
  `createAction` and `createTrigger`, and its version is this package's version.
138
333
 
@@ -147,8 +342,81 @@ The engine's own blocks are the built-in piece `@powerhousedao/piece-core`, in
147
342
  - `branch` and `assert` run through `ActivepiecesBlockExecutor` like any piece
148
343
  action, with the same resolution and journaling. `branch` leaves on `true`
149
344
  or `false` by its result.
345
+ - `router` is Activepieces' Router, first match wins. Its `branches` prop is a
346
+ list; each item has an optional `name` and one condition of the same shape
347
+ and semantics as `branch` (`left`, `operator`, `right`, `caseSensitive`).
348
+ Its ports follow its config: `branch_1` … `branch_N`, then `otherwise` when
349
+ no branch holds, and `error` (`stepPorts` and `routerPorts` in
350
+ `@powerhousedao/pieces-framework/workflow`; the editor labels them with the
351
+ branch names, else "Branch N", and "Otherwise"). It outputs
352
+ `{ branch, index }`: the matched branch's name and 1-based index, both
353
+ `null` for `otherwise`. Every branch resolves before any is tested, so a
354
+ missing reference fails the step unless that branch's operator is an
355
+ existence test. An operand of the wrong type fails it naming the branch.
356
+ - `loop` is Activepieces' Loop on Items. Its `items` must resolve to a list;
357
+ anything else fails the step. Its ports are `next`, `each` and `error`. Its
358
+ body is every step reachable from its `each` edges, and must be closed: a
359
+ body step reached from outside the body, or a body step with an edge back
360
+ to the loop, fails the run before any step runs, naming the step. The
361
+ coordinator runs the body once per item, in order,
362
+ each iteration from fresh records and edge decisions; body steps read
363
+ `{{steps.<loop>.output.item}}` and `.index` (from 1). Then the loop takes
364
+ `next` with the output `{ item, index, count, iterations }`, where
365
+ `iterations` is `[{ index, item, steps: { <key>: output } }]`, kept while
366
+ the row stays under 240 KiB of UTF-8 (`iterationsTruncated` past it), and
367
+ `item` and `index` are the last item's. The row is redacted with every body
368
+ step's connection secrets too. Zero items skip the body. After the loop each
369
+ body step's scope entry and record are its last iteration's. A body step
370
+ failing with no `error` edge fails the loop and the run. Loops nest. A
371
+ step that waits inside a body fails the run: `Step "<key>" cannot wait
372
+ inside a loop yet`. A core wait, or Call Flow with Wait for Response, is
373
+ refused before it runs; any other piece that pauses is refused after it
374
+ ran, so its side effects stand.
375
+ - `delay` ("Delay for" an amount and unit, or "Delay until" a date) suspends
376
+ the run on a `DELAY` waitpoint and outputs `{ resumedAt }`; a time already
377
+ past does not wait. `approval` suspends it on a WEBHOOK waitpoint of kind
378
+ `approval` (title, Markdown details rendered when the run pauses, labels,
379
+ when a reason is required, approver DIDs, an optional expiry) and outputs
380
+ `{ approved, reason, decidedBy, decidedAt }` from the editor's decision
381
+ (`decidedAt` is when it was decided, not when the run resumed). It
382
+ gets no resume URL: a waiting step stops the leg, so no later step could send
383
+ one. To email a link, use Activepieces' approval pieces (Gmail's "Request
384
+ approval in email", or Create Approval Links then Wait for Approval). Both
385
+ throw in a single-step test, and both leave on `next`.
386
+ - `stop` ends the run `SUCCEEDED` without running the steps after it, through
387
+ the same path as `ctx.run.stop`. Its optional response (`responseType`
388
+ `json`, `raw` or `redirect`; `status`, `headers`, `body`, `text`,
389
+ `redirectUrl`) answers a held request; with none set it answers 200 with
390
+ `{}`, as `run.stop()` does. Its only port is `error`, so the editor offers
391
+ no step after it.
392
+ - `waitForWebhook` is Activepieces' "Respond and Wait for Next Webhook": it
393
+ suspends the run on a WEBHOOK waitpoint of kind `webhook` whose
394
+ `responseToSend` is the configured response (the same props as `stop`)
395
+ with the resume URL in the `x-resume-webhook-url` header and, for a JSON
396
+ object body unless `includeResumeUrl` is off, in a `resumeUrl` field. The
397
+ URL is the plain resume URL; a caller appends `/sync` to hold for the run's
398
+ answer. It outputs `{ resumeUrl }` while waiting and the resuming request's
399
+ `{ body, headers, queryParams }` after. `expiresAfter` and `expiresUnit`
400
+ end the run if nobody calls in time. It needs the host's
401
+ `waitpointBaseUrl`, throws in a single-step test, and leaves on `next`.
150
402
  - The triggers are fed by the host: `reactor/schedule.ts`, `reactor/webhook.ts`
151
403
  and the `fire` mutation. Their hooks are never called.
404
+ - `manual` takes optional `inputs`, shaped after Activepieces' web form
405
+ fields: per field a payload key (`name`), `displayName`, `type` (`text`,
406
+ `text_area`, `number`, `toggle`, `date`, `dropdown`, `json`), `required`,
407
+ `defaultValue` and `description`. A dropdown's `options` is one per line,
408
+ either a bare value or `Label = value` (split at the first `=` with a space
409
+ on each side); the Run form shows the label and the payload carries the
410
+ value, which is what the fire checks. Its `defaultValue` is a value, or an
411
+ option's label. A caller's
412
+ `fire` checks its payload against the published trigger's inputs
413
+ (`coerceManualPayload` in `@powerhousedao/pieces-framework/workflow`):
414
+ defaults fill missing fields, values are coerced (`"42"` to 42, `"true"` to
415
+ true, a date to ISO 8601), keys no field declares pass through, and a
416
+ missing required field or a bad value refuses the fire with a
417
+ `ManualPayloadError` naming each field, before any run row. Reruns and
418
+ other trigger kinds are not checked. The inputs also type
419
+ `{{trigger.payload.<name>}}` in the expression picker.
152
420
 
153
421
  ## Block resolution
154
422
 
@@ -173,7 +441,9 @@ step tests. A block whose version is not an exact semver resolves to `missing`.
173
441
  note). At most five candidates are described per resolution.
174
442
  4. **Host-bound pieces** (`@powerhousedao/piece-core`,
175
443
  `@powerhousedao/piece-reactor`) always run the installed copy, with match
176
- `installed`.
444
+ `installed`. Both ship with this runtime's packages and version with it;
445
+ `piece-reactor` reaches the reactor through `requireReactor` like any
446
+ other piece.
177
447
  5. **Missing** is the only failure: no source has the piece, or none of the
178
448
  candidates described has an action or trigger of that name.
179
449
 
@@ -203,11 +473,190 @@ passes over the `steps` array in order until a pass changes nothing:
203
473
  - **Never reached.** A step in a cycle, or fed by an edge from a step that does
204
474
  not exist, is skipped when the run ends. So is every step not yet reached
205
475
  when a step fails with no `error` edge taken.
476
+ - **Loop bodies.** The steps a loop's `each` edges reach are left out of the
477
+ outer passes. The loop step walks them by the same rules once per item,
478
+ then decides its own edges on `next`. Each iteration journals its body
479
+ steps under the path `<loop>[<i>]/<step>` (nested loops concatenate,
480
+ `i` from 1); once a top-level loop ends, the loop's row, then each body
481
+ step's last-iteration row under its own id, follow (`step_execution` is
482
+ unique per `(run_id, step_path)`). Run detail reads the first 1000
483
+ iteration rows and the total; run listings read none. Resume carry reads
484
+ only the rows under a step's own id; a resume carries a finished loop
485
+ with its body.
486
+ - **Loop reruns.** A rerun runs every loop again, and replays its
487
+ iterations step by step from the rows under their paths. An iteration
488
+ replays only while its item, as the loop row journaled it, is unchanged;
489
+ a body step replays by the same rules as a top-level one (same key and
490
+ hash, SUCCEEDED or REPLAYED, a read re-runs). The loop row's config hash
491
+ covers its body's steps and edges, so a rewired body, or a loop row
492
+ truncated past the payload cap, runs every iteration again.
206
493
 
207
494
  Which port a step takes changes which steps run, never the order steps are
208
495
  reached in. The studio's step outline (`stepOutline` in the workflow editor)
209
496
  lists steps in this order.
210
497
 
498
+ ## The workflow policy
499
+
500
+ A workflow document carries a `policy` block, and every field in it was schema
501
+ and editor only until W3.3: nothing read `concurrency`, `runTimeoutSeconds`,
502
+ `defaultRetry` or `onFailure`, so an author who set them got no behaviour and
503
+ no warning. They are enforced now, and the fields that are **not** are marked
504
+ `NOT YET ENFORCED` in the document model's own SDL rather than left to look
505
+ live. `reactor/policy.ts` resolves the block. A definition with **no** policy
506
+ at all enforces nothing, but only hand-built definitions lack one: `policy` is
507
+ non-null in the workflow model since v1, so every workflow document, old or
508
+ new, carries one.
509
+
510
+ | Field | Where | Behaviour |
511
+ | ------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
512
+ | `concurrency` | `reactor/run-gate.ts` | SINGLETON drops a firing while a run is active; QUEUE serialises; PARALLEL runs concurrently |
513
+ | `maxParallelRuns` | `reactor/run-gate.ts` | Bounds PARALLEL; null is unbounded. SINGLETON and QUEUE are 1 by definition |
514
+ | `runTimeoutSeconds` | `pieces/engine/coordinator.ts` | A run deadline from FIRING time, checked between steps and bounding every retry wait; expiry ends the run CANCELLED |
515
+ | `defaultRetry`, step `retry` | `pieces/engine/retry.ts` | Attempts, backoff, delays and `retryOn`; attempts land on the step's journal row |
516
+ | `onFailure` | `reactor/service.ts` | On a trigger's failed run: PARK parks the trigger; NOTIFY logs at error level; IGNORE does nothing. A failed manual run or rerun changes nothing |
517
+ | `maxSuspensionDays` | `reactor/waitpoint-poller.ts` | A WEBHOOK waitpoint expires this long after it was created (sooner if its block says so); expiry of the one the run waits on ends the run CANCELLED with `WaitpointExpired`. A `DELAY` longer than this is refused when created |
518
+ | `retainRunsDays` | — | **Not enforced** per workflow; `PH_WORKFLOWS_RUN_RETENTION_DAYS` is the journal-wide control |
519
+ | `journalAsDocument` | — | **Not enforced**: the journal is relational, and there is no run document model |
520
+ | step `idempotencyKeyExpression` | — | **Not enforced**: a fire dedupes on its trigger operation or a trigger item's `_dedupe_key` |
521
+
522
+ **Enforcing the policy changes every deployed workflow.** The model's initial
523
+ values are `concurrency: QUEUE`, `onFailure: PARK` and
524
+ `runTimeoutSeconds: 3600`, and documents written before enforcement carry them
525
+ too. Unless an author changed them, a workflow now:
526
+
527
+ 1. **Serialises its runs** (QUEUE): a firing waits while a run of the same
528
+ workflow executes.
529
+ 2. **Parks its trigger on the first terminal failure of a run it fired**
530
+ (PARK; a failed manual run or rerun does not count): the trigger
531
+ stops firing until the workflow is re-published or re-enabled.
532
+ 3. **Has a one-hour run deadline counted from firing** (3600s): queue wait
533
+ included; past it the run ends CANCELLED, before its next step.
534
+ 4. **Cancels firings past 100 waiters** (`PH_WORKFLOWS_MAX_QUEUED_FIRINGS`):
535
+ an overflowing firing is journaled CANCELLED instead of run.
536
+
537
+ That is what the fields have said since the first schema; what changed is that
538
+ they are true.
539
+
540
+ - **Every run is admitted twice**: by its workflow's own `concurrency`, then
541
+ across workflows (`reactor/run-dispatcher.ts`). The second admits at most
542
+ `PH_WORKFLOWS_RUNS_IN_FLIGHT` runs at once, and none while the main thread's
543
+ event-loop utilization is above `PH_WORKFLOWS_ADMIT_MAX_ELU`: that thread
544
+ caps throughput, so a run admitted past it only adds latency. One run is
545
+ always admitted when none is running. A run waiting at either gate stays
546
+ PENDING, a manual fire included, so a crash leaves it for the next boot.
547
+ - **The next boot starts what a stopped reactor journaled.** Its PENDING runs
548
+ are admitted when the runtime starts, oldest first, through both gates. A
549
+ run admitted at `FIRE_CRASH_BUDGET` (3) boots without ever starting is
550
+ failed instead, so a fire that takes the reactor down is not replayed
551
+ forever. RUNNING runs are still failed at open, and rerunnable.
552
+ - **Concurrency is process-local**, which is exactly right: workflow execution
553
+ is a singleton pinned to one reactor (see **Placement** above), so this
554
+ process is the deployment's whole run set. A firing SINGLETON drops is
555
+ journaled as a CANCELLED run rather than discarded — a firing that vanished
556
+ is indistinguishable from a trigger that never fired.
557
+ - **The QUEUE is bounded**, at `PH_WORKFLOWS_MAX_QUEUED_FIRINGS` waiting
558
+ firings per workflow (100). QUEUE means latency, not failure — but an
559
+ unbounded queue means neither: a document-event trigger on a busy type
560
+ enqueues faster than the workflow runs, every waiter holds its payload and
561
+ its promise, and the lane grows until the process dies. A firing that
562
+ overflows the depth is journaled CANCELLED exactly as a SINGLETON refusal is.
563
+ - **A sync webhook answers a refused firing without a 500.** A firing refused
564
+ as parked, by SINGLETON, or as stale after its wait gets 409; one refused by
565
+ a full queue or that waited past its deadline gets 429. A provider retries a
566
+ 500, and each retry would journal another CANCELLED run; 500 stays for real
567
+ failures.
568
+ - **A firing that waited reads the workflow again before it runs.** If it is
569
+ no longer ENABLED, was re-published, or was parked while the firing waited,
570
+ the firing is journaled CANCELLED without running a step.
571
+ - **The run deadline starts at FIRING time, not at admission.** Queue time is
572
+ part of the time the run took: a firing that waits past its
573
+ `runTimeoutSeconds` for a slot is CANCELLED without executing a single step,
574
+ rather than running its side effect long after the timeout that was supposed
575
+ to bound it. The document read counts too.
576
+ - **PARKED holds for every trigger kind, and a restart does not clear it.** The
577
+ park is a runtime override of the document's enabled-ness: parking writes a
578
+ `workflow_park` row naming the published version that failed, never the
579
+ document, so the document still says ENABLED and re-arming from it — which is
580
+ what a reboot does for every workflow it finds — would un-park the broken
581
+ workflow and resume firing it. A schedule or piece trigger's `trigger_state`
582
+ row also turns PARKED, so the supervisor stops polling it; a document-event,
583
+ document-lifecycle or webhook trigger is left unregistered; and any firing
584
+ that still arrives is journaled CANCELLED. Only two things clear a park: a
585
+ **re-publish** (the published version moves past the one that failed, trigger
586
+ changed or not), and a **disable then re-enable**. Both work across a restart.
587
+ A park blocks the registrations of the version that failed and older ones
588
+ only: a newer version always arms, and lifts the park it outlived. Every park
589
+ write runs on the workflow's trigger lane, with its enable and disable, so a
590
+ park cannot land between an enable's check and its write. Lanes are per
591
+ workflow: a slow hook delays only its own workflow, and a tick fires due
592
+ schedules first, then runs polls alongside each other, as many as
593
+ `PH_WORKFLOWS_TRIGGER_SLOTS`. An unresolvable
594
+ piece leaves a PARKED row alone as well, rather than turning it ERROR and
595
+ letting the ERROR row's own retry arm it.
596
+ - **`retryOn` empty means every error is retryable.** The schema reads "error
597
+ classes that are retryable; everything else fails terminally on attempt 1",
598
+ but the shipped default is an empty list, and taking that literally would
599
+ make every `maxAttempts` a lie. A non-empty entry matches the error's class
600
+ name exactly or appears anywhere in its message, case-insensitively, so both
601
+ `["HostCallTimeoutError"]` and `["429"]` work.
602
+ - `maxAttempts` is clamped to 10 and one backoff wait to 5 minutes: each
603
+ attempt re-runs a side effect and holds the run's worker slot.
604
+ - **Only the attempt is retried, never the resolution.** A step's input is
605
+ resolved once, before the first attempt, because resolution reads the scope
606
+ and nothing an attempt changes: an `UnresolvedReferenceError` names a key
607
+ that will not exist on attempt five either. A resolution failure fails the
608
+ step immediately, with no backoff waits spent on it.
609
+ - **A retry wait is clipped to the run deadline, and the deadline wins.** A
610
+ backoff longer than the time left is served out only as far as the deadline,
611
+ and then the step gets no further attempt: running one would be a side effect
612
+ after the run was already over. The attempt that failed is still journaled,
613
+ no error port is taken — nothing downstream may run after the run has ended —
614
+ and the run reads CANCELLED, since the clock stopped it rather than the
615
+ workflow failing.
616
+ - **A run its deadline cancelled can be rerun**, like a FAILED one: it is
617
+ journaled with the error name `RunDeadlineExceeded`, and the rerun replays
618
+ the steps that finished. A firing refused before it ran (parked, SINGLETON,
619
+ stale, queue full, expired in the queue) is CANCELLED without that name and
620
+ is not rerunnable; fire the workflow again instead.
621
+ - A step that DECLARES a `retry` block overrides `defaultRetry`, whatever the
622
+ block resolves to — `{maxAttempts: 1}` is an author saying "not this one".
623
+ - A step with no `retry` takes the workflow's `defaultRetry`, else its action's
624
+ default retry. A `defaultRetry` that retries nothing (the document's initial
625
+ `{maxAttempts: 1}`) counts as unset, so the action's applies.
626
+ - An **INDETERMINATE** step is never retried and never replayed: a retry would
627
+ be a second write. A rerun does execute it again, so it can write twice. See
628
+ **Indeterminate steps** below.
629
+
630
+ ## Indeterminate steps
631
+
632
+ A piece's call of its host is capped (`PH_WORKFLOWS_HOST_CALL_TIMEOUT_MS`, 10s,
633
+ raised to the step's own `timeoutSeconds` when that is longer; a trigger hook
634
+ or a design-time call takes the same cap, raised to its own timeout), and always
635
+ clipped to end a margin before the step's kill deadline, so the call's own
636
+ timeout is what the step reports. A **writing** call that times out —
637
+ `store.put`, `store.delete` — may well have been committed, so the step records
638
+ `INDETERMINATE` rather than FAILED: reporting a failure for a write that landed
639
+ is a claim nobody can stand behind, and it was happening (the 10s cap against a
640
+ dispatch under load). A reactor write still unfinished at the step deadline
641
+ (`ReactorJobPendingError`) is INDETERMINATE the same way. A read that times out
642
+ is an ordinary failure, and a call with no time left before the deadline is
643
+ not sent at all.
644
+
645
+ An INDETERMINATE step **takes no port**, so no error branch claims to have
646
+ handled it, and the run fails naming the state. It is not retried, and a rerun
647
+ does not replay it — only SUCCEEDED and REPLAYED steps replay. A rerun
648
+ **executes it again**, so a write that did land the first time is made a second
649
+ time; check its target before rerunning such a run. Workflow Studio renders it
650
+ in its own tone.
651
+
652
+ **Design-time tests carry it too.** `testStep` reports
653
+ `SUCCEEDED | FAILED | INDETERMINATE`, a trigger test whose hook made an
654
+ unconfirmed host call records INDETERMINATE, and such a test run is journaled
655
+ FAILED rather than green — the same answer a real run gives. An INDETERMINATE
656
+ last test is not a sample either: a draft step reading it is told to test that
657
+ block again, since it has no confirmed output to stand on. Collapsing the three
658
+ states to two is how a write-unconfirmed test came to read as a pass.
659
+
211
660
  ## Expressions
212
661
 
213
662
  Every string in a step's config is a template, nested strings in objects and
@@ -252,32 +701,86 @@ Everything the engine itself reads from the environment is prefixed
252
701
  — that manifest is the published surface a host reads, this table is the
253
702
  explanation behind it.
254
703
 
255
- | Variable | Default | What it sets |
256
- | ------------------------------------- | ------------------ | ----------------------------------------------------------------------------------------- |
257
- | `PH_WORKFLOWS_SECRETS_MASTER_KEY` | generated key file | 64 hex chars (32 bytes) encrypting connection secrets at rest (`reactor/secret-store.ts`) |
258
- | `PH_WORKFLOWS_EGRESS_ALLOW_ADDRESSES` | unset | Addresses or CIDRs a piece may reach, widening the default policy (`reactor/lib.ts`) |
259
- | `PH_WORKFLOWS_RUN_CONCURRENCY` | `4` | Runs executing at once; one forked node child each (`worker/pool.ts`) |
260
- | `PH_WORKFLOWS_RUN_QUEUE_DEPTH` | `0` | Runs that may wait for a slot before new ones are refused; `0` waits without limit |
261
- | `PH_WORKFLOWS_POLL_INTERVAL_MS` | `60000` | Cadence for a polling trigger that names none of its own |
262
- | `PH_WORKFLOWS_WEBHOOK_RECONCILE_MS` | `900000` | How often a webhook trigger re-registers with its provider |
263
- | `PH_WORKFLOWS_WEBHOOK_TIMEOUT_MS` | `30000` | How long a sync-mode delivery holds the provider's socket |
264
- | `PH_WORKFLOWS_PIECE_MAX_FILE_BYTES` | `8388608` | File-size ceiling for FILE-property hydration and `ctx.files.write` |
265
- | `PH_WORKFLOWS_RUN_RETENTION_DAYS` | unset (off) | Deletes finished runs older than this many days (`reactor/run-retention.ts`) |
704
+ | Variable | Default | What it sets |
705
+ | ------------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
706
+ | `PH_WORKFLOWS_SECRETS_MASTER_KEY` | generated key file | 64 hex chars (32 bytes) encrypting connection secrets at rest (`reactor/secret-store.ts`) |
707
+ | `PH_WORKFLOWS_EGRESS_ALLOW_ADDRESSES` | unset | Addresses or CIDRs a piece may reach, widening the default policy (`reactor/lib.ts`) |
708
+ | `PH_WORKFLOWS_RUN_CONCURRENCY` | from the host | Most piece children at once (`worker/pool.ts`); unset, sized from the cores and memory, at most 32 |
709
+ | `PH_WORKFLOWS_RUN_QUEUE_DEPTH` | `0` | Requests that may wait for a child before new ones are refused; `0` waits without limit |
710
+ | `PH_WORKFLOWS_WORKER_MEMORY_MB` | `256` | Each child's V8 heap (`--max-old-space-size`); a child past 80% of it is retired |
711
+ | `PH_WORKFLOWS_WORKER_MAX_REQUESTS` | `1000` | Requests a child serves before it is retired |
712
+ | `PH_WORKFLOWS_WORKER_MIN` | `1` | Idle children kept warm; `0` lets every idle child exit |
713
+ | `PH_WORKFLOWS_TRIGGER_SLOTS` | `1` | Children reserved for trigger hooks |
714
+ | `PH_WORKFLOWS_RUNS_IN_FLIGHT` | 8 × the pool | Runs admitted at once across workflows (`reactor/run-dispatcher.ts`) |
715
+ | `PH_WORKFLOWS_ADMIT_MAX_ELU` | `0.8` | Main-thread event-loop utilization above which no new run is admitted; one always runs |
716
+ | `PH_WORKFLOWS_MAX_QUEUED_FIRINGS` | `100` | Firings of ONE workflow that may wait for its concurrency slot; past it a firing is journaled CANCELLED (`reactor/run-gate.ts`) |
717
+ | `PH_WORKFLOWS_POLL_INTERVAL_MS` | `60000` | Cadence for a polling trigger that names none of its own |
718
+ | `PH_WORKFLOWS_WEBHOOK_RECONCILE_MS` | `900000` | How often a webhook trigger re-registers with its provider |
719
+ | `PH_WORKFLOWS_WEBHOOK_TIMEOUT_MS` | `30000` | How long a sync-mode delivery holds the provider's socket |
720
+ | `PH_WORKFLOWS_PIECE_MAX_FILE_BYTES` | `8388608` | File-size ceiling for FILE-property hydration and `ctx.files.write` |
721
+ | `PH_WORKFLOWS_HOST_CALL_TIMEOUT_MS` | `10000` | Cap on one call a piece makes of its host; raised to the step's own timeout when that is longer, clipped to the step deadline (`activepieces/context/limits.ts`) |
722
+ | `PH_WORKFLOWS_ATTACHMENT_CACHE_BYTES` | `1073741824` | Downloaded attachments kept on disk by content (`engine/attachment-cache.ts`); `0` is off |
723
+ | `PH_WORKFLOWS_RUN_RETENTION_DAYS` | `30` | Deletes finished runs older than this many days; `0`/`off` keeps everything (`reactor/run-retention.ts`) |
724
+ | `PH_WORKFLOWS_SINGLETON_OWNER` | `<host>/<journal hash>` | Names this process as the workflow singleton's owner; the default is stable per slot, so a restart re-claims once the old heartbeat is stale (`reactor/singleton-lease.ts`) |
266
725
 
267
726
  Each numeric one parses as `Number(raw) || default`: a value that is not a
268
727
  positive number falls back silently rather than failing at boot.
269
-
270
- **Run retention is off by default**: the run journal keeps every run. With
271
- `PH_WORKFLOWS_RUN_RETENTION_DAYS` set, a sweep runs when the journal opens and
272
- hourly after, deleting runs that finished before the window together with their
273
- step executions and run documents, 500 runs per transaction. Unfinished runs
274
- are never pruned. The same sweep drops trigger dedupe keys older than the
275
- longest dedupe TTL (24h); a deleted workflow's keys go when it is deleted.
728
+ `PH_WORKFLOWS_WORKER_MIN` also takes `0`.
729
+
730
+ **Run retention is ON by default, at 30 days.** It used to be opt-in, which
731
+ meant unbounded growth on every host that did not know to set the variable —
732
+ measured at 743MB in three days. A journal is diagnostic, so a default window
733
+ is the honest setting and "keep everything" is a deliberate choice:
734
+ `PH_WORKFLOWS_RUN_RETENTION_DAYS=0` (or `off`, `never`, `false`, `none`) turns
735
+ it off. A value that is not a positive number falls back to the **default**
736
+ rather than to off — a typo must not remove the bound the variable exists to
737
+ set.
738
+
739
+ A sweep runs when the journal opens and hourly after, deleting runs that
740
+ finished before the window together with their step executions and run
741
+ documents, 500 runs per transaction. Unfinished runs are never pruned. The same
742
+ sweep drops trigger dedupe keys older than the longest dedupe TTL (24h); a
743
+ deleted workflow's keys go when it is deleted.
744
+
745
+ **Row width is capped too**: a step's input and output, and a run's trigger
746
+ payload, are each truncated past `STEP_PAYLOAD_MAX_BYTES` (256KB) to a marker
747
+ holding the original byte count and the head of the serialized JSON. The marker
748
+ is keyed on a **reserved** key whose value is a versioned sentinel, so a
749
+ payload cannot be mistaken for one — the predicate used to duck-type
750
+ `{truncated, bytes, prefix}`, which is exactly the shape of a truncation report
751
+ a piece might legitimately return, and being mistaken for a marker makes a
752
+ side-effectful step re-run. Rows written before the sentinel are still read, by
753
+ their exact key set.
754
+
755
+ A SUCCEEDED step whose output was truncated **replays** on rerun rather than
756
+ re-executing: it had side effects. Its output is explicitly unavailable, so a
757
+ later step that reads it fails the rerun by name (`UnavailableValueError`)
758
+ instead of being handed a marker. A step whose piece declares a reactor
759
+ **read** has no side effect to repeat, so it re-executes and produces its
760
+ output again.
761
+
762
+ The fact **survives further reruns**. The rerun's own REPLAYED row journals the
763
+ truncation marker again (the record's `journaledOutput`), so a second rerun of
764
+ the rerun still reads a marker rather than a NULL it would take for an ordinary
765
+ replay with no output. The record handed back to a caller keeps `output`
766
+ absent either way: a marker must never sit where real data goes.
767
+
768
+ The refusal is **deep, and on every route out of resolution**. The unavailable
769
+ wrapper carries its reason on a symbol, which anything that serializes it drops
770
+ — so a path that lands one level ABOVE the wrapper (`{{steps.charge}}`, or a
771
+ bare `{{steps}}`) would have handed the piece an ordinary-looking object whose
772
+ `output` became `{}` across the worker boundary, reason and all. Resolution
773
+ therefore checks the value it is about to return at any depth, and
774
+ `resolveStepInput` checks the whole resolved input again before it crosses to
775
+ the piece.
276
776
 
277
777
  **The secrets key is not optional in production.** Unset, `loadKey` generates
278
- `./.ph/secrets.key` — relative to the working directory, like the bundle cache
279
- and the attachment staging dir. A host whose working directory does not survive
280
- a restart would come back with a new key, so the store guards against it:
778
+ `.ph/secrets.key` in the same `.ph` tree as the bundle cache and the attachment
779
+ folders: the working directory's, or the one `PH_USER_DATA_DIR` moves it to. A
780
+ tree with no key takes a copy of one left in the working directory, as a
781
+ desktop shell's stores from before it moved are encrypted with that. A host
782
+ whose tree does not survive a restart would come back with a new key, so the
783
+ store guards against it:
281
784
 
282
785
  - A host passes `secretsKeyFile: false` when its database outlives the working
283
786
  directory, and the store then requires `PH_WORKFLOWS_SECRETS_MASTER_KEY`
@@ -321,7 +824,7 @@ more trusted than one that arrived inside a package.
321
824
 
322
825
  ## OAuth2 connections
323
826
 
324
- An OAUTH2 connection brings its own app. `startOAuth` (`reactor/oauth.ts`)
827
+ An OAUTH2 connection brings its own app, or names a publisher's (below). `startOAuth` (`reactor/oauth.ts`)
325
828
  builds the provider's authorize URL from the piece's `PieceAuth.OAuth2`
326
829
  (`{prop}` placeholders filled from the connection's config, PKCE when the
327
830
  piece asks for it) and records a pending attempt in the `oauth` namespace,
@@ -332,10 +835,156 @@ set as a managed secret named `token` on the connection and runs the
332
835
  connection check. Resolution refreshes the token 15 minutes before it
333
836
  expires and rotates that secret in place.
334
837
 
838
+ A refresh the provider refuses with `invalid_grant` (the refresh token
839
+ expired or was revoked) ends the sign-in, and so does a stored token that
840
+ cannot be read. The runtime records it on the connection as a check would:
841
+ `status` ERROR, the refusal in `lastError`, once per refusal rather than once
842
+ per poll. A REVOKED connection keeps its status. A run or poll that needed the
843
+ token fails with `OAuthSignInRefused`. The stored token is still used until it
844
+ expires. A refresh that fails for another reason (a timeout, an outage)
845
+ records nothing, and the next call tries again. The next sign-in's check
846
+ records OK.
847
+
335
848
  Token requests leave from the reactor process, so they are held to the egress
336
849
  policy pieces run under: `https` to a public address, or an address named in
337
850
  `PH_WORKFLOWS_EGRESS_ALLOW_ADDRESSES`, over `http` too.
338
851
 
852
+ ### Publisher apps and the relay
853
+
854
+ A connection may instead name a publisher's app (`oauth_app` in its config),
855
+ configured once per host in `workflows.oauthApps`:
856
+
857
+ ```json
858
+ {
859
+ "id": "sektar-google",
860
+ "clients": { "desktop": "desktop.json", "web": "web.json" },
861
+ "relay": {
862
+ "clientId": "<the web client's id>",
863
+ "redirectUri": "https://sektar.io/api/oauth/callback",
864
+ "tokenUrl": "https://sektar.io/api/oauth/token"
865
+ }
866
+ }
867
+ ```
868
+
869
+ `clients` are the provider's client files (relative to the config file). A
870
+ host may pass the same list programmatically (Switchboard's
871
+ `workflows.oauthApps` option), which wins over the config file. `startOAuth`
872
+ picks, for the redirect it is given (`oauthSignInFor` in
873
+ `reactor/oauth-apps.ts`, also answered by the `oauthSignIn` query as
874
+ `DESKTOP | WEB | RELAY | NONE`):
875
+
876
+ - a loopback callback: the **desktop** client, always with PKCE;
877
+ - any other callback: the **web** client when the host has its file, sending
878
+ the provider the host's own callback, exactly as without a relay;
879
+ - any other callback, no web client file, a `relay`: the **relay**;
880
+ - otherwise none, and `startOAuth` refuses.
881
+
882
+ The relay is a token broker the publisher runs and that alone holds the web
883
+ client's secret. Its one `redirectUri` is registered with the provider, so
884
+ hosts whose own callback is not (a cloud environment per hostname) need no
885
+ provider change. A relayed sign-in:
886
+
887
+ - sends the provider `redirect_uri` = `relay.redirectUri`, `client_id` =
888
+ `relay.clientId`, PKCE whatever the piece says;
889
+ - sets `state` to `<nonce>.<base64url(own callback)>`: the nonce is the usual
890
+ base64url of 32 random bytes (no dots), the callback the redirect the host
891
+ passed, base64url without padding. The relay forwards the provider's
892
+ parameters unchanged to that callback, which runs `completeOAuth` as ever;
893
+ the whole string is the attempt's key;
894
+ - records the attempt with `relay.redirectUri` and `relay.tokenUrl` and no
895
+ client secret ref (`client_secret_ref` null);
896
+ - exchanges the code at `relay.tokenUrl` with `client_id`, `code_verifier`
897
+ and `redirect_uri`, no secret; the broker adds it and answers with the
898
+ provider's token response;
899
+ - stores the token set with `relayed: true`, its `token_url` the relay's: every
900
+ refresh posts `grant_type=refresh_token`, the refresh token and `client_id`
901
+ there, again without a secret, whatever the config says by then.
902
+
903
+ A relay needs a `clientId` and two absolute `https` URLs without credentials;
904
+ otherwise it is dropped with a warning and the app kept without it. The
905
+ relay's token URL is held to the egress policy above, which a public `https`
906
+ host passes.
907
+
908
+ ## Files and attachments
909
+
910
+ Bytes never cross the worker's IPC channel; host and worker share
911
+ `.ph/ap-attachment-staging/<uuid>`, one directory per step, removed when the
912
+ step returns. Directories a crashed host left behind are swept at startup.
913
+
914
+ - **In.** The host downloads every `attachment://` ref in a step's config into
915
+ the step's directory, counting bytes against
916
+ `PH_WORKFLOWS_PIECE_MAX_FILE_BYTES` as they arrive. Downloads land in a
917
+ content-addressed cache first (`.ph/ap-attachment-cache`), so a later step
918
+ or run reuses them; a step gets a copy (a clone where the filesystem has
919
+ them), never the cache file. A URL is fetched by the worker, and its body is
920
+ counted as it arrives too.
921
+ - **Out.** `ctx.files.write` takes a Buffer or a Readable and writes it into
922
+ the step's directory under the ceiling. After the step the host ingests each
923
+ file, hashing and uploading it from disk, and refuses any path the step did
924
+ not write: outside its directory, a symlink or a hard link.
925
+ `ctx.files.upload` stages the same way, but the host ingests the file during
926
+ the call, over the host-call channel, and returns the ref.
927
+ - **Who may read.** A run reads the refs its own steps wrote, and the refs its
928
+ journaled steps already held on a rerun or a step test. Anything else is the
929
+ host's call (`canReadAttachmentRef`), made against the documents the run was
930
+ handed: the trigger's and those its steps read. Switchboard allows a ref
931
+ when the run's user may read one of those documents and that document
932
+ references it, as its download route does; a cached copy is served only
933
+ after the same check.
934
+ - **Time.** Staging and ingest count against the step's timeout.
935
+
936
+ ## Sub-flows
937
+
938
+ Activepieces' `@activepieces/piece-subflows` runs unmodified (the editor pins
939
+ `0.7.0` and lists its blocks under Core): the **Callable Flow** trigger
940
+ ("Callable workflow"), **Call Flow** ("Call workflow") and **Return Response**
941
+ ("Return response"). Stream CSV to Subflows is not offered.
942
+
943
+ - `ctx.flows.list` answers with the Callable Flow workflows of every drive
944
+ the calling workflow is in, itself included, that the reader may read: the
945
+ design-time caller, else the calling workflow's run user (with no run user,
946
+ none while auth enforcement is on). They come as Activepieces
947
+ `PopulatedFlow` records: `id` and `externalId` are the workflow id,
948
+ `status` is `ENABLED` only for an enabled, published workflow,
949
+ `version.displayName` is its name and `version.trigger.settings` is its
950
+ runnable trigger (`pieceName`, `pieceVersion`, `triggerName`, and its
951
+ config as `input`). A workflow in no drive can list only itself. Steps, trigger hooks (but `test` and `onDisable`)
952
+ and design-time options get it; `blockOptions` takes the edited
953
+ `workflowId` for the last.
954
+ - `ctx.server.apiUrl` is `http://reactor.invalid`, an origin that is never on
955
+ the network. The worker's global `fetch` hands a request to it to the host
956
+ over the call channel, so a sub-flow call needs no public URL, no egress
957
+ rule and no replica routing, and only a run of this reactor can make one.
958
+ Another HTTP client fails to resolve it. `publicUrl` is the origin of
959
+ `waitpointBaseUrl` when there is one; `server.token` throws.
960
+ - The host origin serves `POST /v1/webhooks/<workflowId>` (Call Flow) and
961
+ `POST /v1/waitpoints/<token>` (Return Response); anything else is 404. A
962
+ call is refused unless the target is listed for the caller (404) and is
963
+ enabled with the Callable Flow trigger (409), or when its
964
+ `ap-parent-run-id` names another run (403). It goes through the trigger's
965
+ `run` hook and journals the callee's run, answering `{ runId }`.
966
+ - Call Flow's callback is a resume URL on the host origin, minted without
967
+ `waitpointBaseUrl`, so Return Response resumes the caller in process. A
968
+ callback on `waitpointBaseUrl` this host minted is mapped to the host
969
+ origin too. A response no pending waitpoint takes
970
+ answers `{ status: "gone", discarded: true }` and Return Response fails.
971
+ Switchboard's public resume route adds `discarded` to its JSON answers too.
972
+ - The callee's run records its caller (`parent_run_id`) and, when the caller
973
+ asked to fail on failure (Wait for Response), the waitpoint it waits on
974
+ (`parent_waitpoint_id`). A callee that ends `FAILED`, `CANCELLED` or
975
+ `INDETERMINATE` resolves that waitpoint with
976
+ `{ status: "error", data: { message, runId } }`, so Call Flow throws and the
977
+ caller fails. A callee that succeeds without Return Response leaves the
978
+ caller waiting until its waitpoint expires, as upstream.
979
+ - A Callable Flow trigger arms without a public webhook endpoint and is never
980
+ polled. Its workflow's public webhook URL answers 401: it is called only by
981
+ runs of this reactor.
982
+ - A trigger's `onStart` runs once per run, before its first step, with the
983
+ run's id as `ctx.run.id` and the trigger item as `payload`; it fails the run
984
+ when it throws. A rerun or a resumed leg does not run it, so a rerun called
985
+ run answers no caller. A trigger without one, or with `createTrigger`'s no-op
986
+ default, is not called.
987
+
339
988
  ## Known missing features
340
989
 
341
990
  This is what a piece can declare or call that this engine does not run. It is tracked in
@@ -377,7 +1026,8 @@ releases what an earlier enable registered. The reason reads
377
1026
  either fails or fires a spurious run (#3090).
378
1027
  - `setSchedule({ cronExpression })` is run as a fixed interval; wall-clock
379
1028
  time and timezone are lost.
380
- - `onStart` is never called. The trigger context's `server` is a throwing stub.
1029
+ - A trigger hook's `server` is the host origin (see Sub-flows), except in
1030
+ `test` and `onDisable`, where it throws.
381
1031
  - Outside a delivery a hook's `payload` is `undefined`; upstream passes `{}`
382
1032
  (#3090).
383
1033
  - A webhook payload carries no raw body, and its signature headers
@@ -419,19 +1069,101 @@ releases what an earlier enable registered. The reason reads
419
1069
 
420
1070
  **Actions**
421
1071
 
422
- - `errorHandlingOptions` (retry, continue on failure) is ignored.
1072
+ - `errorHandlingOptions.retryOnFailure` with `defaultValue: true` becomes the
1073
+ action's default retry (3 attempts, exponential from 5s) unless the action
1074
+ declares a `defaultRetry`. `continueOnFailure` is ignored.
423
1075
  - `test` is never called.
1076
+ - `run.canPause` is `true` in a run and `false` in a single-step test, where
1077
+ `createWaitpoint`, `waitForWaitpoint` and `run.pause` throw.
1078
+ - `createWaitpoint`, `waitForWaitpoint` and the deprecated `run.pause` (both
1079
+ pause shapes) suspend the run (#3218): the step and run go `WAITING` and
1080
+ the run gives up its worker, its admission slot and its `concurrency`
1081
+ slot. A resume goes through admission again, re-runs the waiting step as
1082
+ `RESUME` with `resumePayload`, and gets a fresh `runTimeoutSeconds`.
1083
+ - A step has at most one live waitpoint, as upstream: a second
1084
+ `createWaitpoint` in the step, a retry of it, or a retry of a create whose
1085
+ answer was lost gets the same waitpoint and resume URL, already resolved if
1086
+ its resume came before the step paused. Once the step resumes, it can wait
1087
+ again on a new one. Only the hash of a resume URL's token is stored, so the
1088
+ URL is re-issued only within the leg that minted it; a step that would get a
1089
+ live waitpoint from an earlier leg fails instead (unreachable today).
1090
+ - A waiting run resumes on the waitpoint its step waits on, or on a WEBHOOK
1091
+ link a step that succeeded created (Create Approval Links, then Wait for
1092
+ Approval), as upstream. When the run starts waiting, the waitpoints of
1093
+ steps that failed close `CANCELLED`; when it resumes on such a link, the
1094
+ waitpoint its step waited on closes. Only the expiry of the waitpoint it
1095
+ waits on ends the run; a link that expires just closes. A resume already
1096
+ delivered wins over that expiry.
1097
+ - Every end of a run (succeeded, failed, cancelled, expired, or failed by the
1098
+ orphan sweep) closes its waitpoints and fails its waiting step. A resume
1099
+ URL of a closed waitpoint, or of a run that has ended, answers `gone`, and
1100
+ the editor refuses to resolve it.
1101
+ - One poller on the singleton owner wakes due `DELAY` waitpoints and expires
1102
+ overdue ones, in batches, on a single timer armed for the next due time
1103
+ (re-checked at least every five minutes). An expiry and the run it ends are
1104
+ one transaction. Each pass also ends, `CANCELLED` with `WaitpointNotLive`,
1105
+ a WAITING run whose waitpoint closed without ending it.
1106
+ - Not yet: a run left `RUNNING` by a crash after it resumed is failed, not
1107
+ resumed (#3224).
1108
+ - A run resumed on a definition that was re-published meanwhile rebuilds it
1109
+ by replaying the workflow's global operations below the revision it pinned,
1110
+ from the latest `UPGRADE_DOCUMENT` snapshot, without denied or undone
1111
+ operations. This is not the reactor's own rebuild (keyframes, upgrade
1112
+ transitions); when the result does not hash to the pin, the leg fails.
1113
+ - A WEBHOOK waitpoint needs the host's `waitpointBaseUrl`; without it
1114
+ `createWaitpoint({ type: "WEBHOOK" })` throws, except in Call Flow.
1115
+ - A resumed run reads steps journaled before it waited from the journal: a
1116
+ truncated or redacted output is unavailable to later steps, and a document
1117
+ output is its reference, not the document.
424
1118
  - `requireAuth` defaults to `false` in the descriptor; upstream defaults to
425
1119
  `true`.
426
- - `run.stop`, `run.respond`, `run.pause`, waitpoints and `generateResumeUrl`
427
- throw.
428
- - `connections.get`, `tags`, `server`, `agent` and `flows.list` throw.
1120
+ - `run.respond` and `run.stop` answer the request the run holds: a
1121
+ `responseMode: "sync"` webhook delivery, or a `POST <resumeUrl>/sync` for
1122
+ the leg it resumed. The first answer wins and later ones are ignored; within
1123
+ a step the last of `respond`, `stop` and `waitForWaitpoint` wins, as
1124
+ upstream. The answer goes out when the step returns, with status 200 and
1125
+ body `{}` by default; a string body goes as text, anything else as JSON, and
1126
+ `set-cookie` and framing headers are dropped. The answer is served on the
1127
+ host's origin, so it always carries `x-content-type-options: nosniff` and
1128
+ `content-security-policy: sandbox`, over any the step sets: a step's HTML
1129
+ renders without scripts or the origin's cookies. Unlike upstream, any step may
1130
+ answer, not only one of the trigger's piece. `run.stop` ends the run
1131
+ `SUCCEEDED` and skips the steps after it.
1132
+ - A run that pauses answers the held request with its waitpoint's
1133
+ `responseToSend`; without one, a sync webhook answers 202 with the run's
1134
+ status. A `/sync` resume is held up to `PH_WORKFLOWS_WEBHOOK_TIMEOUT_MS`
1135
+ (504 after), and answered with the run's status when the leg ends or waits
1136
+ without responding. One whose waitpoint closes, or whose run ends, before a
1137
+ leg resumes on it (including a resume that fails to start) answers 409 with
1138
+ the run's status and error.
1139
+ - The deprecated `generateResumeUrl` links to the waitpoint the step's
1140
+ deprecated `run.pause` creates (one link per step). A step that pauses some
1141
+ other way, or that already made a waitpoint, leaves that link dead.
1142
+ - `connections.get`, `tags`, `agent` and `server.token` throw. `server` and
1143
+ `flows.list` are served as described in Sub-flows; a single-step test gets
1144
+ neither.
429
1145
  - `ctx.store` checks the key length on `put` only; `get` and `delete` of an
430
1146
  over-long key answer as if it were absent.
431
1147
  - `flows.current.version.id` is a constant. `project.id` is `reactor` on
432
1148
  every reactor: the reactor is the project, as it is for `ctx.store`'s
433
1149
  PROJECT scope.
434
1150
 
1151
+ **Files**
1152
+
1153
+ - Trigger hooks cannot read `attachment://` refs, and a trigger's
1154
+ `ctx.files.write` returns an inline data URI rather than a ref.
1155
+ - `ctx.files.upload` returns the attachment ref as both `id` and `url`; the
1156
+ `url` is not an HTTP URL. In a trigger hook both are the inline data URI.
1157
+ Without an attachment store an action's call throws.
1158
+ - A webhook body that is not JSON or form data is decoded as UTF-8 text, so
1159
+ binary and multipart deliveries are corrupted.
1160
+ - A step cannot read back a file it wrote: its `apfile://` token only becomes
1161
+ a ref once the step returns.
1162
+ - A Buffer or `ApFile` returned in an output is serialized as a JSON number
1163
+ array, not ingested as a file.
1164
+ - A written file's content type is inferred from ten extensions; others are
1165
+ `application/octet-stream`.
1166
+
435
1167
  **Piece**
436
1168
 
437
1169
  - `deprecated` is not in the descriptor.
@@ -469,7 +1201,7 @@ A case this engine is known to fail runs as `it.fails` and names the issue that
469
1201
  fixes it; never edit those files by hand. To regenerate them:
470
1202
 
471
1203
  ```sh
472
- pnpm --filter @powerhousedao/pieces-framework sync-upstream -- --tag 0.91.0 --from <activepieces checkout>
1204
+ pnpm --filter @powerhousedao/pieces-framework sync-upstream -- --tag d078d47750b53c13b837b2bec86b0402b4bff94f --from <activepieces checkout>
473
1205
  ```
474
1206
 
475
1207
  ## Design documents