okengine 0.18.4 → 0.19.0

package/site/content/docs/elements/flow/workflows.mdx

@@ -0,0 +1,640 @@
+---
+title: "Workflows"
+description: "Journaled multi-step Flows — fx.step replay, LIFO undo, durable sleep, and crash resume."
+icon: "GitFork"
+source: "docs/spec/unified-theory.md"
+---
+
+A durable Flow is an ordinary Flow with `durable: true`. Named `fx.step` calls journal
+their results so a crash resumes without re-running completed work — checkout, onboarding,
+anything that must not double-charge.
+
+For developers writing multi-step work on okengine — set `durable: true`, wrap side
+effects in `fx.step`, keep time on `fx.clock`.
+
+<Callout title="The one rule">
+  Wrap every side effect in a uniquely named `fx.step`. Replay returns the journaled value and never
+  re-runs the body. Register `{undo}` on steps that must reverse; sleep only with
+  `fx.clock.sleep(label, duration)` on a durable Flow.
+</Callout>
+
+<FlowDurable />
+
+## Smallest Example
+
+<Steps>
+
+<Step>
+### Define a durable Flow
+
+```typescript title="src/flows/orders/checkout.ts"
+import { on, flow, http } from "okengine";
+import { z } from "zod";
+import { db, charges, orders } from "@/schema";
+
+export const checkout = on(
+  http.post({
+    in: z.object({ userId: z.string(), sku: z.string() }),
+    out: z.object({ orderId: z.string() }),
+  }),
+  flow({
+    durable: true,
+    do: async ({ userId, sku }, fx) => {
+      const charge = await fx.step("charge", async () => {
+        const id = fx.id();
+        await fx.store(db).insert(charges).values({ id, userId, amount: 50 });
+        return { id };
+      });
+
+      const orderId = await fx.step("create-order", async () => {
+        const id = fx.id();
+        await fx.store(db).insert(orders).values({
+          id,
+          userId,
+          sku,
+          chargeId: charge.id,
+        });
+        return id;
+      });
+
+      return { orderId };
+    },
+  }),
+);
+```
+
+</Step>
+
+<Step>
+### Call the endpoint
+
+```bash
+curl -X POST http://localhost:6530/orders/checkout \
+  -H "content-type: application/json" \
+  -d '{"userId":"usr_1","sku":"sku_42"}'
+```
+
+Response:
+
+```json
+{
+  "data": { "orderId": "ord_1" },
+  "error": null
+}
+```
+
+Kill the process after `charge` persists and before `create-order` finishes.
+Boot again — `charge` replays from the journal; the card is not charged twice.
+
+</Step>
+
+</Steps>
+
+<Callout title="Not a separate species">
+  There is no workflow engine API. `durable: true` is a Flow option — HTTP, Signal, Clock, CDC, and
+  call-only Flows all journal the same way. See [Consumers](/docs/elements/flow/consumers).
+</Callout>
+
+## Progressive Patterns
+
+Explore durable Flows from a two-step journal to undo, sleep, and flow-level compensation:
+
+<Tabs items={["Minimal", "Undo", "Sleep", "Compensate"]}>
+
+<Tab value="Minimal">
+
+Two named steps. `fx.id()` lives **inside** the step so resume reuses the journaled id:
+
+```typescript title="src/flows/billing/charge.ts"
+import { on, flow, http } from "okengine";
+import { z } from "zod";
+import { db, payments } from "@/schema";
+
+export const charge = on(
+  http.post({
+    in: z.object({ userId: z.string(), amount: z.number() }),
+    out: z.object({ paymentId: z.string() }),
+  }),
+  flow({
+    durable: true,
+    do: async ({ userId, amount }, fx) => {
+      const paymentId = await fx.step("create-intent", async () => {
+        const id = fx.id();
+        await fx.store(db).insert(payments).values({ id, userId, amount });
+        return id;
+      });
+      await fx.step("confirm", async () => {
+        await fx.call(capturePayment, { paymentId });
+      });
+      return { paymentId };
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Undo">
+
+`{ undo }` receives the journaled return value. On terminal failure, completed
+undos run last-in first-out; the failed step does **not** undo:
+
+```typescript title="src/flows/orders/checkout.ts"
+import { on, flow, http } from "okengine";
+import { z } from "zod";
+import { eq } from "drizzle-orm";
+import { db, charges } from "@/schema";
+
+export const checkout = on(
+  http.post({ in: z.object({ userId: z.string() }) }),
+  flow({
+    durable: true,
+    do: async ({ userId }, fx) => {
+      const charge = await fx.step(
+        "charge",
+        async () => {
+          const id = fx.id();
+          await fx.store(db).insert(charges).values({ id, userId, amount: 50 });
+          return { id };
+        },
+        {
+          undo: async (res) => {
+            await fx.store(db).delete(charges).where(eq(charges.id, res.id));
+          },
+        },
+      );
+      await fx.step("fulfill", async () => {
+        await fx.call(fulfillOrder, { chargeId: charge.id });
+      });
+      return { chargeId: charge.id };
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Sleep">
+
+`fx.clock.sleep(label, duration)` parks the run and releases the worker. HTTP
+returns **`204 No Content`** immediately — the caller is not waiting at wake:
+
+```typescript title="src/flows/trials/reminder.ts"
+import { on, flow, http } from "okengine";
+import { z } from "zod";
+import { trialExpiringEmail } from "@/channels/trial";
+
+export const reminder = on(
+  http.post({ in: z.object({ email: z.string().email() }) }),
+  flow({
+    durable: true,
+    do: async ({ email }, fx) => {
+      await fx.step("mark-trial", async () => {
+        await fx.call(startTrial, { email });
+      });
+      await fx.clock.sleep("expiry-window", "3d");
+      await fx.step("notify", async () => {
+        await fx.send(trialExpiringEmail, { to: email });
+      });
+    },
+  }),
+);
+```
+
+Without `durable: true`, `sleep` resolves immediately and does not park.
+
+</Tab>
+
+<Tab value="Compensate">
+
+Per-step `{ undo }` runs first (LIFO), then optional `compensate` for
+cross-cutting cleanup. Manual undo work uses distinct `undo:…` step names:
+
+```typescript title="src/flows/orders/checkout.ts"
+import { on, flow, http } from "okengine";
+import { z } from "zod";
+
+export const checkout = on(
+  http.post({ in: z.object({ userId: z.string(), sku: z.string() }) }),
+  flow({
+    durable: true,
+    do: async ({ userId, sku }, fx) => {
+      await fx.step("reserve", () => fx.call(reserveStock, { sku }), {
+        undo: () => fx.call(releaseStock, { sku }),
+      });
+      await fx.step("charge", () => fx.call(chargeCard, { userId }));
+      return { ok: true as const };
+    },
+    compensate: async (ctx, fx) => {
+      await fx.step("undo:alert", async () => {
+        await fx.send(opsAlert, {
+          to: "oncall@example.com",
+          data: {
+            steps: ctx.completedSteps.join(", "),
+            error: String(ctx.error),
+          },
+        });
+      });
+    },
+  }),
+);
+```
+
+`compensate` does **not** run on success, between `retry` attempts, or on sleep park.
+
+</Tab>
+
+</Tabs>
+
+## Options Reference
+
+| Option / call                     | Type                   | Default   | Meaning                                                   |
+| --------------------------------- | ---------------------- | --------- | --------------------------------------------------------- |
+| `durable`                         | `boolean`              | `false`   | Journal `fx.step`, `fx.clock.sleep`, and gated `fx` calls |
+| `compensate`                      | `(ctx, fx) => unknown` | omitted   | After LIFO undos, before the run commits `failed`         |
+| `retry`                           | `FxRetryOptions`       | omitted   | Whole-`do` retry on the **same** journal session          |
+| `fx.step(name, fn, opts?)`        | step                   | —         | Named checkpoint; `{ undo }` is optional                  |
+| `fx.clock.sleep(label, duration)` | park                   | —         | Durable pause; two arguments (label then duration)        |
+| `fx.retry(fn, opts?)`             | inner retry            | see Retry | Put **inside** a step so a completed charge never re-runs |
+
+`compensate` context: `{ input, error, completedSteps }` — `completedSteps` are
+forward names only (`undo:…` entries are excluded).
+
+**Consequence:** `durable: true` disables automatic read-cache for that Flow.
+
+## Steps
+
+<Callout title="Detailed section">
+  If you only need a named checkpoint, jump to Replay below. Step names must be unique per run. The
+  prefix `undo:` is reserved for compensation — a forward step with that prefix throws `journal:
+  step name "…" uses reserved prefix "undo:"`.
+</Callout>
+
+Every `fx.step` persists `{ name, value }` before the next line runs. On resume the
+engine matches by name, returns the stored value, and skips the function.
+
+<Tabs items={["Replay", "At-least-once", "Inner retry"]}>
+
+<Tab value="Replay">
+
+Completed steps are skipped. Generate ids and call providers **inside** the step:
+
+```typescript
+const intent = await fx.step("create-intent", async () => {
+  const id = fx.id();
+  await fx.call(createPaymentIntent, { id, amount: input.total });
+  return { id };
+});
+```
+
+`fx.id()` outside a step mints a new id on every resume. Gated `fx` calls
+(`store`, `emit`, `send`, `ask`, `call`, `vault`) are also journaled in call
+order — named steps are the stable checkpoint when the sequence might branch.
+
+</Tab>
+
+<Tab value="At-least-once">
+
+A crash **during** a step (before persist) re-runs that function. Completed
+neighbors never re-run. Make the in-flight body safe to repeat, or persist at
+the provider first and journal only the id (the create-intent pattern).
+
+**Consequence:** two replicas will not double-run a **journaled** step; they
+may double-run the step that was in flight when the holder died.
+
+</Tab>
+
+<Tab value="Inner retry">
+
+`fx.retry` inside `fx.step` retries the provider without committing a step
+until success. Flow-level `retry` re-enters `do` on the same journal — completed
+steps still replay:
+
+```typescript
+const charge = await fx.step("charge", () =>
+  fx.retry(() => fx.call(stripeCharge, { amount: input.total }), {
+    retries: 3,
+    delay: "100ms",
+    backoff: 2,
+    jitter: true,
+  }),
+);
+```
+
+Do **not** put `fx.retry` around the whole `do` by hand — use `flow({ retry })`.
+
+</Tab>
+
+</Tabs>
+
+<Accordions>
+
+<Accordion title="Step Options">
+  Third argument to `fx.step(name, fn, options)`.
+
+| Option | Type                 | Default | Meaning                                                  |
+| ------ | -------------------- | ------- | -------------------------------------------------------- |
+| `undo` | `(value) => unknown` | omitted | Runs on terminal failure with the journaled value (LIFO) |
+
+`undo` closures are re-bound on resume by re-entering `do` without new forward
+work. Nested `{ undo }` on an `undo:…` step throws
+`journal: undo steps cannot register nested undo`.
+
+</Accordion>
+
+<Accordion title="Duplicate names">
+  A second forward step with the same name throws
+  `journal: duplicate step name "charge"`. Pick a new name (`charge-tax`) or
+  fold the work into the first step.
+
+Sleep matches by **label** from the cursor — use a distinct label per pause.
+
+</Accordion>
+
+<Accordion title="Manifest steps">
+  The compiler records `fx.step("…")` string names on the Flow as `steps`.
+  Removing a name is a Manifest contract change (`oke doctor --diff`). Adding
+  a name is recorded the same way. Names only — bodies are not in the Manifest.
+
+</Accordion>
+
+<Accordion title="fx.using is not journaled">
+  `fx.using(acquire, release, use)` is same-attempt cleanup. Do not hold a
+  connection or file handle across `fx.clock.sleep` — acquire again after wake.
+
+</Accordion>
+
+</Accordions>
+
+## Compensation
+
+<Callout title="Detailed section">
+  If you only need per-step refunds, jump to the table below. Compensation runs on throw **and**
+  `fx.fail` — after retries are exhausted, never on sleep park.
+</Callout>
+
+When a durable run fails terminally, status becomes `compensating`, then `failed`.
+
+| Step                        | State                  | Action when `fulfill` throws    |
+| --------------------------- | ---------------------- | ------------------------------- |
+| 1. `charge` with `{ undo }` | Succeeded              | `undo(journaledValue)` — refund |
+| 2. `fulfill`                | Failed (not persisted) | No undo for this step           |
+| 3. later work               | Not started            | Never executed                  |
+
+Order: reverse `{ undo }` frames, then `flow.compensate`, then commit `failed`.
+If an undo or `compensate` throws, the journal error is `compensate:{code}`.
+
+<Accordions>
+
+<Accordion title="compensate context">
+
+| Field            | Meaning                                 |
+| ---------------- | --------------------------------------- |
+| `input`          | Original validated Flow input           |
+| `error`          | Thrown value or `fx.fail` result        |
+| `completedSteps` | Forward step names (no `undo:` entries) |
+
+Use `compensate` for alerts and cross-cutting cleanup. Prefer `{ undo }` for
+the reverse of one step. Manual bodies must call `fx.step("undo:…", …)` — never
+reuse a forward name.
+
+</Accordion>
+
+<Accordion title="Orphan mid-undo">
+  A crash during compensation resumes in `compensating`. Already-journaled
+  `undo:charge` is skipped; remaining undos continue — forward `do` does not
+  re-enter. Failed/completed runs refuse resume (`journal: run is already failed`).
+
+</Accordion>
+
+<Accordion title="Retry vs undo">
+  `flow({ retry })` does **not** undo between attempts. Undos run once, after
+  the last extra attempt still throws or `fx.fail`s.
+
+</Accordion>
+
+</Accordions>
+
+## Durable Sleep
+
+<Callout title="Detailed section">
+  If you only need a pause, jump to the example below. Signature is `fx.clock.sleep(label,
+  duration)` — a single duration string is the **label**, not the wait. Durations: `"200ms"` ·
+  `"30s"` · `"2m"` · `"1h"` · `"7d"`.
+</Callout>
+
+Sleep writes a wake time, sets status `sleeping`, and **releases the run lease**
+so a parked flow does not hold a 30s lock for days. Any instance may claim the
+row when `wakeAt` is due.
+
+```typescript title="src/flows/onboarding/welcome.ts"
+import { on, flow } from "okengine";
+import { userSignedUp } from "@/signals";
+import { welcomeEmail } from "@/channels/welcome";
+
+export const sendWelcome = on(
+  userSignedUp,
+  flow("onboarding.welcome", {
+    durable: true,
+    do: async ({ email }, fx) => {
+      await fx.step("provision", async () => {
+        await fx.call(createWorkspace, { email });
+      });
+      await fx.clock.sleep("morning-window", "8h");
+      await fx.step("notify", async () => {
+        await fx.send(welcomeEmail, { to: email });
+      });
+    },
+  }),
+);
+```
+
+HTTP + sleep: the request returns `204` with an empty body. Resume is a
+scheduler job, not a second response to that client.
+
+<Accordions>
+
+<Accordion title="Duration strings">
+  Integer + unit only — no weeks. `"d"` is 86_400_000 ms, not a calendar day.
+  Unknown strings parse as `0` (wake immediately). Same grammar as
+  `fx.clock.ago` / `fromNow`.
+
+</Accordion>
+
+<Accordion title="Do not fx.call a sleeper">
+  `fx.call` waits for the callee to return. If the callee parks, the caller
+  receives `undefined` and continues; the child wakes later as its own run.
+  Sleep on the **root** durable Flow, or split with `fx.emit` to a consumer.
+
+</Accordion>
+
+<Accordion title="Non-durable sleep">
+  Without a journal, `fx.clock.sleep` resolves immediately (tests and sync
+  Flows). There is no thread sleep and no `setTimeout`.
+
+</Accordion>
+
+</Accordions>
+
+## Retry
+
+Two layers — do not mix them up.
+
+| Layer                       | Where             | Journal                          | Undo                         |
+| --------------------------- | ----------------- | -------------------------------- | ---------------------------- |
+| `fx.retry` inside `fx.step` | One provider call | Step commits once, after success | No                           |
+| `flow({ retry })`           | Whole `do`        | Same session; rewind + replay    | After the last attempt fails |
+
+| `retry` option | Default                | Meaning                                       |
+| -------------- | ---------------------- | --------------------------------------------- |
+| `retries`      | `0`                    | Extra attempts after the first                |
+| `delay`        | `50` (ms) or `"100ms"` | Initial backoff                               |
+| `backoff`      | `2`                    | Multiplier after each retry                   |
+| `jitter`       | `true`                 | Full jitter (thundering-herd)                 |
+| `when`         | thrown errors          | Skips abort and sleep park (`JournalSuspend`) |
+
+**Consequence:** put provider retries inside the step; use Flow `retry` for
+transient failures **after** a step (network to your own `fx.call`).
+
+## Journal
+
+<Callout title="Detailed section">
+  If you only need defaults, jump to the table. `drivers.journal` is `postgres` in `dev`/`prod` and
+  `memory` in `test`. Pin `file` for a single host without Postgres.
+</Callout>
+
+The journal is a driver, not an element. Runs are rows: `running` · `sleeping` ·
+`compensating` · `completed` · `failed`.
+
+| Driver     | Default env    | Best for                                             |
+| ---------- | -------------- | ---------------------------------------------------- |
+| `postgres` | `dev`, `prod`  | Shared durable runs across replicas (`DATABASE_URL`) |
+| `memory`   | `test`         | Process-local; lost on exit                          |
+| `file`     | pin explicitly | One machine — `.oke/journal.json`                    |
+
+Unknown ids throw `oke boot: unknown journal driver "…" (expected memory · file · postgres)`.
+Postgres without a URL throws `oke boot: journal driver "postgres" needs DATABASE_URL`.
+
+<Accordions>
+
+<Accordion title="Leases">
+  Default lease is **30s** (same as Signal claims). A live holder renews on
+  every journal write. Sleep and terminal commit **release** the lease.
+
+Resume that loses the race throws `journal: run "{id}" is leased by another instance`
+(`JournalLeaseBusy`). The other holder continues; this instance skips.
+
+</Accordion>
+
+<Accordion title="Orphans & ready">
+  Boot resumes `running` / `sleeping` / `compensating` rows with no live lease;
+  future sleeps stay scheduled until `wakeAt`. `GET /_/ready` stays
+  `503 { ready: false, reason: "orphan_scan" }` until that scan finishes.
+
+</Accordion>
+
+<Accordion title="Statuses">
+
+| Status         | Meaning                               |
+| -------------- | ------------------------------------- |
+| `running`      | In-flight attempt; lease held         |
+| `sleeping`     | Parked; lease released; `wakeAt` set  |
+| `compensating` | LIFO undos / `compensate` in progress |
+| `completed`    | Terminal success — resume refused     |
+| `failed`       | Terminal failure — resume refused     |
+
+</Accordion>
+
+</Accordions>
+
+## Troubleshooting
+
+<Accordions>
+
+<Accordion title='journal: duplicate step name "charge"'>
+  Two forward `fx.step("charge", …)` calls in one run. Rename one, or combine the work. Compensation
+  uses `undo:charge` automatically — do not declare a second forward step with that name.
+</Accordion>
+
+<Accordion title='journal: step name "undo:x" uses reserved prefix "undo:"'>
+  `undo:` is for the compensation phase. Forward work needs a plain name (`refund`). Inside
+  `compensate`, `fx.step("undo:alert", …)` is the intended form.
+</Accordion>
+
+<Accordion title="HTTP 204 with empty body after POST">
+  The Flow parked on `fx.clock.sleep` — success, not a missing handler. The original client is done;
+  wake continues on a worker. Return a body **before** sleep if the caller must see an id, or emit
+  to a Signal consumer for the rest.
+</Accordion>
+
+<Accordion title="Sleep returns immediately / work runs twice after wait">
+  Missing `durable: true`, or `fx.clock.sleep("8h")` with one argument — `"8h"` is the label,
+  duration is missing. Use `fx.clock.sleep("label", "8h")`. Non-durable sleep is a no-op.
+</Accordion>
+
+<Accordion title="Card charged twice after a crash">
+  The provider call was outside `fx.step`, or the crash was mid-step (at-least-once). Move
+  create-intent into a step and make confirm idempotent. `Date.now()` / `fetch` bypass the journal —
+  use `fx.clock.now()` and `fx.step`.
+</Accordion>
+
+<Accordion title="journal: run is leased by another instance">
+  Two instances claimed the same run. This is skip-not-fail: the holder continues. Shared `postgres`
+  (or `file` on one host) is required — `memory` does not coordinate across processes.
+</Accordion>
+
+<Accordion title="oke boot: journal driver postgres needs DATABASE_URL">
+  Dev/prod default is `postgres`. Set `DATABASE_URL`, or pin `drivers.journal.test` / a non-Postgres
+  map in `oke.config.ts` for local experiments without SQL.
+</Accordion>
+
+<Accordion title="compensate:{code} on the failed run">
+  An `{undo}` or `compensate` body threw. Fix the reverse path; the forward error is already
+  recorded. Forward `do` will not re-run on that run id.
+</Accordion>
+
+<Accordion title="fx.call of a durable sleeper returned undefined">
+  The callee parked. Sleep on the root Flow, or emit to a durable consumer instead of calling a
+  sleeper inline.
+</Accordion>
+
+<Accordion title="GET /_/ready is 503 reason orphan_scan">
+  Boot is resuming durable orphans. Wait — do not point a liveness probe at `/_/ready`. Use a
+  separate liveness check; readiness may stay 503 until the orphan scan finishes.
+</Accordion>
+
+</Accordions>
+
+## Learn more
+
+- [Flow](/docs/elements/flow) — `durable`, `retry`, `compensate` on the Flow options table
+- [Consumers](/docs/elements/flow/consumers) — Signal / Clock / CDC as the same species
+- [HTTP](/docs/elements/flow/http) — request envelope; `204` from `undefined`
+- [Clock · Durable Sleep](/docs/elements/clock/sleep) — pause physics
+- [fx](/docs/reference/fx) — `fx.step`, `fx.retry`, `fx.clock.sleep`, `fx.using`
+- [Configuration](/docs/reference/configuration) — `drivers.journal`
+
+## Next
+
+<Cards>
+  <Card
+    title="Durable Sleep"
+    description="Process-safe pauses that resume across reboots."
+    href="/docs/elements/clock/sleep"
+  />
+  <Card
+    title="Consumers"
+    description="Signal workers, named Clock jobs, and SQL CDC — one Flow species."
+    href="/docs/elements/flow/consumers"
+  />
+  <Card
+    title="HTTP"
+    description="Synchronous REST, QUERY, resources, and live SSE."
+    href="/docs/elements/flow/http"
+  />
+  <Card
+    title="Flow Overview"
+    description="One shape for every kind of backend behavior."
+    href="/docs/elements/flow"
+  />
+</Cards>