okengine 0.18.4 → 0.19.0

package/site/content/docs/elements/flow/consumers.mdx

@@ -0,0 +1,707 @@
+---
+title: "Consumers"
+description: "Asynchronous Flows for Signal emissions, named Clock ticks, and SQL row changes."
+icon: "Radio"
+source: "docs/spec/unified-theory.md"
+---
+
+Consumers are Flows that run when something else happens — a Signal emit, a named Clock tick, or a SQL row change — instead of waiting for an HTTP request.
+
+For developers wiring background work on okengine — bind the trigger, keep `do` on `fx`.
+
+<Callout title="The one rule">
+  Bind with `on(signal)`, `on(clockDecl)`, or `on(db.table(…).changed())`. Delivery physics live on
+  the Signal; `cron` / `every` live on the Clock; CDC input is always `{ before, after }`. World
+  access still goes through `fx`.
+</Callout>
+
+## Smallest Example
+
+<Steps>
+
+<Step>
+### Bind a Signal consumer
+
+```typescript title="src/flows/notifications/welcome.ts"
+import { on, flow } from "okengine";
+import { userSignedUp } from "@/signals";
+import { welcomeEmail } from "@/channels/welcome";
+
+export const sendWelcome = on(
+  userSignedUp,
+  flow("notifications.welcome", {
+    do: async ({ userId, email }, fx) => {
+      await fx.send(welcomeEmail, {
+        to: email,
+        data: { userId },
+      });
+    },
+  }),
+);
+```
+
+</Step>
+
+<Step>
+### Emit from any Flow
+
+```typescript
+await fx.emit(userSignedUp, { userId: "usr_123", email: "alice@example.com" });
+```
+
+The compiler records `emits: ["users.signed-up"]` on the producer. The consumer runs after the
+emit commits — the HTTP request does not wait for the welcome mail.
+
+</Step>
+
+</Steps>
+
+<Callout title="Jobs are consumers">
+  A named Clock bound with `on(clockDecl, flow)` is the same species — an asynchronous Flow. There
+  is no separate job runner. See [Clock jobs](#clock-jobs).
+</Callout>
+
+## Progressive Patterns
+
+Explore consumers from a typed queue worker to a cron job and a table-change handler:
+
+<Tabs items={["Signal", "Clock", "CDC", "Ordered"]}>
+
+<Tab value="Signal">
+
+Declare delivery physics on the Signal, then bind the worker with `on(handle, flow)`:
+
+```typescript title="src/signals/email.ts"
+import { signal } from "okengine";
+import { z } from "zod";
+
+export const emailTask = signal.once("tasks.email", {
+  schema: z.object({ to: z.string().email(), body: z.string() }),
+  retries: 3,
+  deadLetter: true,
+});
+```
+
+```typescript title="src/flows/workers/email.ts"
+import { on, flow } from "okengine";
+import { emailTask } from "@/signals/email";
+import { rawEmail } from "@/channels/email";
+
+export const processEmail = on(
+  emailTask,
+  flow("workers.email", {
+    do: async ({ to, body }, fx) => {
+      await fx.send(rawEmail, { to, body });
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Clock">
+
+Name the schedule, then bind it. Omit `timezone` for `"UTC"`:
+
+```typescript title="src/flows/metrics/cleanup.ts"
+import { on, flow, clock } from "okengine";
+import { lt } from "drizzle-orm";
+import { db, metricLogs } from "@/schema";
+
+export const cleanupClock = clock("metrics.cleanup", { every: "1h" });
+
+export const cleanupMetrics = on(
+  cleanupClock,
+  flow("metrics.cleanup", {
+    do: async (_, fx) => {
+      await fx
+        .store(db)
+        .delete(metricLogs)
+        .where(lt(metricLogs.timestamp, fx.clock.ago("7d")));
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="CDC">
+
+`db.table(handle).changed()` fires on insert, update, and delete. Input is `{ before, after }`:
+
+```typescript title="src/flows/audit/users.ts"
+import { on, flow } from "okengine";
+import { db } from "@/core";
+import { users, auditLogs } from "@/schema";
+
+export const onUserWrite = on(
+  db.table(users).changed(),
+  flow("audit.users", {
+    do: async ({ before, after }, fx) => {
+      const action = !before ? "created" : !after ? "deleted" : "updated";
+      const recordId = String(after?.id ?? before?.id ?? "");
+      await fx.store(db).insert(auditLogs).values({
+        table: "users",
+        recordId,
+        action,
+      });
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Ordered">
+
+`signal.once` plus `fx.emit(…, { key })` serializes work per key. Same key never runs
+concurrently — the in-flight visibility lease is the lock:
+
+```typescript title="src/flows/orders/ship.ts"
+import { on, flow } from "okengine";
+import { orderPlaced } from "@/signals/orders";
+
+export const shipOrder = on(
+  orderPlaced,
+  flow("orders.ship", {
+    do: async ({ orderId }, fx) => {
+      await fx.call(fulfillOrder, { orderId });
+    },
+  }),
+);
+```
+
+```typescript
+await fx.emit(orderPlaced, { orderId: "ord_1", userId: "usr_1" }, { key: "usr_1" });
+```
+
+Omit `key` for competing consumers with no ordering.
+
+</Tab>
+
+</Tabs>
+
+## Trigger Reference
+
+| Trigger    | Signature                              | Purpose                                     | `do` input          |
+| ---------- | -------------------------------------- | ------------------------------------------- | ------------------- |
+| Signal     | `on(signalHandle, flow)`               | Queue (`once`) or fan-out (`broadcast`)     | Payload (`schema`)  |
+| Clock      | `on(clockDecl, flow)`                  | Interval or cron tick                       | none (`_`)          |
+| CDC any    | `on(db.table(t).changed(), flow)`      | Every insert / update / delete              | `{ before, after }` |
+| CDC column | `on(db.table(t).changed("col"), flow)` | Same writes; column stamped on the Manifest | `{ before, after }` |
+
+`signal.live` is an HTTP SSE tape — bind it with [`http.live`](/docs/elements/flow/http#live-streams),
+not as a worker. A Flow with no trigger is [call-only](/docs/elements/flow).
+
+## Signal Consumers
+
+<Callout title="Detailed section">
+  If you only need a worker, jump to Once below. Physics live on `signal.once` / `broadcast` /
+  `live` — same idea as `http.get` / `http.post`. Defaults: `retries: 3`, `deadLetter: true`,
+  `optional: false`.
+</Callout>
+
+Each emit is handled according to the Signal helper you declared. The Flow is the subscriber.
+
+<Tabs items={["Once", "Broadcast", "Optional"]}>
+
+<Tab value="Once">
+
+Competing workers — exactly one consumer processes each message. Failed attempts retry, then
+dead-letter when `deadLetter` is true (default):
+
+```typescript title="src/signals/orders.ts"
+import { signal } from "okengine";
+import { z } from "zod";
+
+export const orderPlaced = signal.once("orders.placed", {
+  schema: z.object({
+    orderId: z.string(),
+    amount: z.number(),
+    userId: z.string(),
+  }),
+  retries: 5,
+  deadLetter: true,
+});
+```
+
+```typescript title="src/flows/orders/fulfill.ts"
+import { on, flow } from "okengine";
+import { orderPlaced } from "@/signals/orders";
+
+export const fulfill = on(
+  orderPlaced,
+  flow("orders.fulfill", {
+    do: async ({ orderId }, fx) => {
+      await fx.call(chargeAndShip, { orderId });
+    },
+  }),
+);
+```
+
+Visibility lease defaults to **30s**. An inflight worker that dies is reclaimed on the next claim.
+
+</Tab>
+
+<Tab value="Broadcast">
+
+Every subscribed Flow gets a copy. Offline listeners miss the event — there is no replay tape:
+
+```typescript title="src/signals/cache.ts"
+import { signal } from "okengine";
+import { z } from "zod";
+
+export const catalogChanged = signal.broadcast("catalog.changed", {
+  schema: z.object({ sku: z.string() }),
+});
+```
+
+```typescript title="src/flows/cache/invalidate.ts"
+import { on, flow } from "okengine";
+import { catalogChanged } from "@/signals/cache";
+
+export const invalidate = on(
+  catalogChanged,
+  flow("cache.invalidate", {
+    do: async ({ sku }, fx) => {
+      await fx.cache.delete(`sku:${sku}`);
+    },
+  }),
+);
+```
+
+A second Flow bound to the same Signal also runs. That is the fan-out.
+
+</Tab>
+
+<Tab value="Optional">
+
+Emit with zero subscribers throws **OKE1240** unless `optional: true`. Use that for live
+firehoses and hook Signals that may have no worker yet:
+
+```typescript
+export const webhook = signal.once("hooks.inbound", {
+  optional: true,
+  schema: z.object({ id: z.string() }),
+});
+```
+
+**OKE1240** cause: `Flow "{flow}" emits signal "{resource}" with no subscriber.`
+Fix: add `on(signal, …)` or mark `{ optional: true }`.
+
+</Tab>
+
+</Tabs>
+
+<Accordions>
+
+<Accordion title="Signal Options">
+  Optional second argument to `signal.once` / `broadcast` / `live`. Delivery is the helper name.
+
+| Option        | Type                     | Default   | Meaning                                                      |
+| ------------- | ------------------------ | --------- | ------------------------------------------------------------ |
+| `schema`      | Standard Schema          | omitted   | Enforced at `fx.emit` (**OKE1250** on mismatch)              |
+| `retries`     | `number`                 | `3`       | Extra attempts after the first (`retries + 1` total)         |
+| `deadLetter`  | `boolean`                | `true`    | Keep exhausted `once` messages; `false` marks them delivered |
+| `optional`    | `boolean`                | `false`   | Allow emit with zero subscribers                             |
+| `retention`   | `{ maxAge?, maxCount? }` | unbounded | **`signal.live` only** — type error on `once` / `broadcast`  |
+| `description` | `string`                 | the name  | Console / docs blurb                                         |
+
+</Accordion>
+
+<Accordion title="Ordering">
+  Pass `{ key }` on emit. No two `once` messages sharing `(signal, key)` are claimed at once.
+
+```typescript
+await fx.emit(emailTask, payload, { key: user.id });
+```
+
+Same key → FIFO. Different keys run in parallel. Omit `key` for a pure competing pool.
+
+</Accordion>
+
+<Accordion title="Retries & dead letters">
+  `once` retries then DLQ. After `retries + 1` handler invocations the message is dead when
+  `deadLetter: true`. Inspect with `fx.deadLetters(signal)`.
+
+`deadLetter: false` marks the message delivered after the last attempt — nothing lands in the DLQ.
+
+Broadcast does not use the `once` lease / DLQ path. Live uses the retained tape, not this worker.
+
+</Accordion>
+
+<Accordion title="Schema at emit">
+  Invalid payloads fail at `fx.emit` with **OKE1250** (`"{resource}": {detail}`) before any
+  consumer runs. Fix the payload or the Signal's `schema`. This is an **emit** contract — not
+  an HTTP invoke contract on `flow()`. Workers inherit the typed payload from the Signal
+  declaration; they do not declare `in` on `flow({ do })`.
+</Accordion>
+
+</Accordions>
+
+## Clock Jobs
+
+<Callout title="Detailed section">
+  If you only need an interval, jump to the example below. `clock(name)` requires `cron` or `every`
+  — missing both throws `clock("name"): require cron or every`. Bind the returned handle with
+  `on(clockDecl, flow)`.
+</Callout>
+
+Named clocks reconcile into the Store at boot. The scheduler leader-elects so N instances do not
+double-fire. `do` receives no payload — read time through `fx.clock`.
+
+<Tabs items={["Interval", "Cron", "Per-tenant"]}>
+
+<Tab value="Interval">
+
+Human durations: `"200ms"` · `"30s"` · `"5m"` · `"1h"` · `"7d"` (integer + unit, no weeks):
+
+```typescript title="src/flows/health/ping.ts"
+import { on, flow, clock } from "okengine";
+
+export const pingClock = clock("health.pingExternal", { every: "30s" });
+
+export const pingExternal = on(
+  pingClock,
+  flow("health.pingExternal", {
+    plane: "operator",
+    do: async (_, fx) => {
+      await fx.call(pingUpstream);
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Cron">
+
+Five-field cron (`m h dom mon dow`) plus an IANA `timezone` (default `"UTC"`):
+
+```typescript title="src/flows/reports/daily.ts"
+import { on, flow, clock } from "okengine";
+
+export const dailyReportClock = clock("reports.daily", {
+  cron: "0 6 * * *",
+  timezone: "Asia/Riyadh",
+});
+
+export const runDailyReport = on(
+  dailyReportClock,
+  flow("reports.runDaily", {
+    plane: "operator",
+    do: async (_, fx) => {
+      await fx.call(buildDailyReport, { at: fx.clock.now() });
+    },
+  }),
+);
+```
+
+You may set `cron` and `every` together. Extract records the cron expression as the Manifest
+trigger when both are present.
+
+</Tab>
+
+<Tab value="Per-tenant">
+
+`clock.perTenant` expands one Store row per tenant (`{name}#{tenantId}`). The bare template name
+is never ticked:
+
+```typescript title="src/clocks/invoices.ts"
+import { on, flow, clock } from "okengine";
+
+export const invoicesClock = clock.perTenant("invoices", { every: "1h" });
+
+export const runInvoices = on(
+  invoicesClock,
+  flow("billing.invoices", {
+    plane: "operator",
+    do: async (_, fx) => {
+      await fx.call(closeOpenInvoices);
+    },
+  }),
+);
+```
+
+Equivalent: `clock("invoices", { every: "1h", perTenant: true })`.
+
+</Tab>
+
+</Tabs>
+
+<Accordions>
+
+<Accordion title="Clock Options">
+  Second argument to `clock(name, options)` / `clock.perTenant(name, options)`.
+
+| Option        | Type        | Default  | Meaning                                        |
+| ------------- | ----------- | -------- | ---------------------------------------------- |
+| `cron`        | `string`    | —        | Five-field cron (`m h dom mon dow`)            |
+| `every`       | `string`    | —        | Interval (`"10s"`, `"1h"`, `"7d"`, …)          |
+| `timezone`    | IANA string | `"UTC"`  | Zone for cron (intervals are duration-based)   |
+| `overridable` | `boolean`   | `false`  | Console may override the schedule in the Store |
+| `perTenant`   | `boolean`   | `false`  | Expand `{name}#{tenantId}` rows                |
+| `description` | `string`    | the name | Console / docs blurb                           |
+
+At least one of `cron` or `every` is required.
+
+</Accordion>
+
+<Accordion title="Leader lock">
+  Dev/prod clock driver is **postgres** (test is **frozen**). A short lease (default **30s**)
+  means only one instance runs each tick.
+
+**Consequence:** three pods calling `runNow` still execute the Flow once. After the lease
+expires, another instance may take the next tick.
+
+`file` (`.oke/crons.json`) elects across processes on one machine. `memory` is single-process.
+
+</Accordion>
+
+<Accordion title="Catch-up policy one">
+  Catch-up is `"one"`: after 5 hours down on an hourly clock, the next tick fires **once**.
+  Missed slots are visible as `missedRuns` — they are not replayed as a burst.
+
+**Consequence:** a digest that missed the night still runs once at boot, not 24 times.
+
+</Accordion>
+
+<Accordion title="DST & overrides">
+  Cron + a DST zone that lands in a spring-forward gap or fall-back overlap attaches a
+  **warning** on the Store row (`gap` / `overlap`). UTC never warns. The scheduler still ticks.
+
+`overridable: true` lets Console edit the effective cron/every. Without it, a Console edit
+fails with `ScheduleNotOverridableError`. Removed declarations become `orphaned` rows and
+do not fire.
+
+</Accordion>
+
+</Accordions>
+
+## CDC
+
+<Callout title="Detailed section">
+  If you only need any-write, jump to the example below. The handle is
+  `db.table(table).changed(column?)` — `db` is a `store.sql` declaration, `table` is a schema
+  handle. `changed("insert")` is **not** an op filter; it stamps a column named `insert`.
+</Callout>
+
+SQL writes through `fx.store` notify CDC after commit. The Flow input is always
+`{ before, after }` (`CdcPayload`).
+
+<Tabs items={["Any write", "Column", "Images"]}>
+
+<Tab value="Any write">
+
+Omit the argument to react to every insert, update, and delete on the table:
+
+```typescript title="src/flows/search/reindex.ts"
+import { on, flow } from "okengine";
+import { db } from "@/core";
+import { notes } from "@/schema";
+
+export const reindexNotes = on(
+  db.table(notes).changed(),
+  flow("search.reindexNotes", {
+    plane: "operator",
+    do: async ({ before, after }, fx) => {
+      const id = String(after?.id ?? before?.id ?? "");
+      if (!id) return;
+      if (!after) {
+        await fx.call(dropNoteIndex, { id });
+        return;
+      }
+      await fx.call(upsertNoteIndex, { id });
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Column">
+
+`changed("status")` stamps `trigger.cdc.column` on the Manifest (Console, extract). Still
+receive `{ before, after }` — filter in `do` when you only care about that field:
+
+```typescript title="src/flows/tasks/on-status.ts"
+import { on, flow } from "okengine";
+import { db } from "@/core";
+import { tasks } from "@/schema";
+
+export const onStatus = on(
+  db.table(tasks).changed("status"),
+  flow("tasks.onStatus", {
+    plane: "operator",
+    do: async ({ before, after }, fx) => {
+      if (before?.status === after?.status) return;
+      const id = String(after?.id ?? before?.id ?? "");
+      if (!id) return;
+      await fx.emit(taskStatusChanged, {
+        id,
+        from: before?.status ?? null,
+        to: after?.status ?? null,
+      });
+    },
+  }),
+);
+```
+
+</Tab>
+
+<Tab value="Images">
+
+Op is inferred from which image is null:
+
+| Write  | `before`     | `after` |
+| ------ | ------------ | ------- |
+| Insert | `null`       | new row |
+| Update | previous row | new row |
+| Delete | previous row | `null`  |
+
+```typescript
+do: async ({ before, after }, fx) => {
+  if (!before && after) {
+    /* insert */
+  } else if (before && after) {
+    /* update */
+  } else if (before && !after) {
+    /* delete */
+  }
+};
+```
+
+</Tab>
+
+</Tabs>
+
+<Accordions>
+
+<Accordion title="CDC payload">
+  Kernel input is `{ before, after }` — not `{ record }`, not `{ op }`. Read the primary key from
+  `after?.id ?? before?.id`.
+
+Writes must go through `fx.store`. A raw SQL client bypasses the sink, so no consumer runs.
+
+</Accordion>
+
+<Accordion title="Outbox">
+  On RLS-capable SQL (`postgres` / `pglite`) the same write is appended to `oke_cdc_outbox`
+  for multi-host delivery. Pending backlog is a doctor finding (`cdc_outbox_backlog`).
+
+Live **queries** (`store.resource({ live: true })` / `http.get(path).live(table)`) share this
+CDC path but classify per subscriber — see [HTTP · Live Streams](/docs/elements/flow/http#live-streams).
+
+</Accordion>
+
+</Accordions>
+
+## Execution
+
+Consumers share the Flow species with HTTP. The differences are the trigger and how failure
+is retried.
+
+| Kind               | Start               | Failure                                | Time             |
+| ------------------ | ------------------- | -------------------------------------- | ---------------- |
+| Signal `once`      | `fx.emit`           | Signal `retries` then DLQ              | `fx.clock.now()` |
+| Signal `broadcast` | `fx.emit`           | Per-subscriber; no `once` DLQ          | `fx.clock.now()` |
+| Clock              | scheduler tick      | Flow `retry` if set; no catch-up burst | `fx.clock.*`     |
+| CDC                | committed SQL write | Flow `retry` if set                    | `fx.clock.now()` |
+
+Mark long work `durable: true` and wrap side effects in `fx.step` — see
+[Workflows](/docs/elements/flow/workflows).
+
+Clock drivers: **postgres** in `dev`/`prod`, **frozen** in `test`. Signal drivers: **redis** in
+`dev`/`prod`, **memory** in `test`.
+
+## Troubleshooting
+
+<Accordions>
+
+<Accordion title='TypeError: clock("name"): require cron or every'>
+  `clock(name)` needs `{cron}` and/or `{every}`. Empty options throw at declaration, before `on()`.
+</Accordion>
+
+<Accordion title="TypeError: on() expected a trigger or signal handle">
+  The first argument must be a Signal handle, a Clock handle, `db.table(…).changed()`, an HTTP
+  trigger, `internal`, or `mcp.tool(…)`. A bare interval string is not a trigger — wrap it in
+  `clock("name", { every: "1h" })`.
+</Accordion>
+
+<Accordion title="OKE1240 — emit with no subscriber">
+  Cause: `Flow "{flow}" emits signal "{resource}" with no subscriber.` Add `on(signal, flow)` or
+  set `{ optional: true }` on the Signal (live firehoses, unused hooks).
+</Accordion>
+
+<Accordion title="OKE1250 — emit failed schema">
+  Cause: `"{resource}": {detail}`. The payload failed the Signal's Standard Schema at emit. Fix the
+  payload; the consumer never ran.
+</Accordion>
+
+<Accordion title='changed("insert") never fires on inserts only'>
+  `changed()` takes an optional **column** name, not an op. `changed("insert")` waits for a column
+  named `insert`. Use `changed()` and branch on `before` / `after` being null.
+</Accordion>
+
+<Accordion title="CDC do never sees record / op">
+  Input is `{ before, after }`. There is no `record` field. Infer insert / update / delete from
+  which image is null.
+</Accordion>
+
+<Accordion title="Cron fired 24 times after overnight downtime">
+  It should not. Catch-up is `"one"` — one fire per overdue clock, then `nextRunAt` advances. If you
+  see a burst, you likely bound several clocks (or `clock.perTenant` expanded many tenants), not a
+  replay of missed hourly slots.
+</Accordion>
+
+<Accordion title="Two pods ran the same job">
+  Clock leader election needs a shared Store (`drivers.clock` postgres, or `file` on one host).
+  `memory` does not coordinate across processes. Check that both instances share `DATABASE_URL`.
+</Accordion>
+
+<Accordion title="ScheduleNotOverridableError from Console">
+  The clock was declared without `overridable: true`. Add it and redeploy, or edit the declaration
+  in source instead of Console.
+</Accordion>
+
+<Accordion title="tz is not a clock option">
+  The field is `timezone` (IANA), default `"UTC"`. `{ tz: "Asia/Riyadh" }` is ignored.
+</Accordion>
+
+</Accordions>
+
+## Learn more
+
+- [Signal](/docs/elements/signal) — `once` / `broadcast` / `live` physics
+- [Signal · Once](/docs/elements/signal/once) — leases, retries, partition keys
+- [Clock](/docs/elements/clock) — schedules, `fx.clock.sleep`
+- [Store · SQL](/docs/elements/store/sql) — tables CDC watches
+- [HTTP · Live Streams](/docs/elements/flow/http#live-streams) — `signal.live` SSE
+- [fx](/docs/reference/fx) — `fx.emit`, `fx.deadLetters`, `fx.clock`
+- [Errors](/docs/reference/errors) — OKE1240 · OKE1250
+- [Workflows](/docs/elements/flow/workflows) — `durable: true` + `fx.step` on a consumer
+
+## Next
+
+<Cards>
+  <Card
+    title="Durable Workflows"
+    description="Step journaling and multi-step distributed execution."
+    href="/docs/elements/flow/workflows"
+  />
+  <Card
+    title="Signal Element"
+    description="Delivery physics — once, broadcast, and live tapes."
+    href="/docs/elements/signal"
+  />
+  <Card
+    title="Clock Element"
+    description="Named schedules, intervals, and durable sleep."
+    href="/docs/elements/clock"
+  />
+  <Card
+    title="HTTP"
+    description="Synchronous REST, QUERY, resources, and live SSE."
+    href="/docs/elements/flow/http"
+  />
+</Cards>