@smthrs/control 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/authority.md

@@ -0,0 +1,109 @@
+---
+title: "Authority, not execution"
+description: "Why the control plane records decisions instead of running work, the three ports that keep that split honest, and what a composition looks like with each of them present or absent."
+sidebar:
+  order: 1
+---
+
+`Control` decides. It does not run anything.
+
+Every operation the service exposes either records an intent or reads back
+evidence. `plan` writes a reviewable card. `approve` resolves a durable token
+and installs a grant. `cancel` writes a request and an attribution. `watch`
+replays a journal it did not write. Nothing in this package executes a flow,
+opens a step, or interprets a graph.
+
+That is not a limitation to work around. It is what lets one control plane
+answer for runs that several processes own, on machines it cannot reach, in a
+database it shares with an engine it never imports.
+
+## The three ports
+
+A host chooses an implementation of each seam, and the seams are what make the
+plane portable.
+
+| Port              | Question it answers                                                     | Implementations here                                    |
+| ----------------- | ----------------------------------------------------------------------- | ------------------------------------------------------- |
+| `ControlRuntime`  | Where do plans, tokens, grants, idempotency records, and run rows live? | `ControlRuntime.layerMemory`, `SqlControlRuntime.layer` |
+| `ControlExecutor` | Who actually runs the work, and what did they do with my request?       | `ControlExecutor.makeNoop`, your own                    |
+| `Journal`         | Where is the evidence of what was decided?                              | [`@smthrs/journal`](/api/journal)                       |
+
+`ControlLive.layer` is the implementation over those three plus the
+[notification queue](/api/notifications) a steer travels through and the
+[registry](/api/registry) a flow listing reads.
+
+Both runtimes are held to one
+[shared contract suite](https://github.com/smithersai/smithers/blob/main/packages/smithers/control/test/ControlContract.ts),
+so a behavior you observe against the memory runtime is a behavior the durable
+one owes you.
+
+## The executor is optional, and the absence is a real composition
+
+`ControlLive` reads `ControlExecutor` through `Effect.serviceOption`. A
+composition with no executor is not broken; it is a plane that starts nothing:
+
+- `run` on an approved plan still mints the run row, still journals
+  `control.run.accepted`, and then releases the row as `control.run.pending`,
+  because nothing here took the launch.
+- `cancel` still writes its attribution and still interrupts a fiber this
+  process is driving, but nothing reaches an engine row in another database.
+- `signal` still records the fact, and no wait point is completed by this call.
+- `resume` and `run` with a Resume input join or claim control-launched runs
+  and journal `control.run.resume`. A caller or journal subscriber must drive
+  the execution. Explicit resume does not offer work through
+  `ControlExecutor.resumeRun`. A run a live host parked is handed to that
+  host as a `requestResume` delegation that carries the operator's consent.
+- A node-approval decision records a durable `requestResume` delegation.
+  Without an executor, it remains available for the owning host's next poll.
+
+That is the shape a monitor, a dashboard, or a read-only operator tool has, and
+it is what [`examples/src/38-monitor-and-alert.ts`](https://github.com/smithersai/smithers/blob/main/examples/src/38-monitor-and-alert.ts)
+builds on purpose.
+
+## Two run tables, one journal
+
+A control run row and an engine run row are different documents about the same
+work, and they do not collide by accident: the plane keeps its own
+`flows_runs`, and the engine keeps its own. The [`smthrs` CLI](/api/cli) runs
+them as two files, `.flows/control.db` and `.flows/engine.db`.
+
+They share the journal, and that is what makes `watch` worth having: one stream
+carries `control.run.accepted` and `flows.engine.attempt-started` in the order
+they happened.
+
+Sharing one database instead is a deployment choice with consequences, because
+`SqlControlRuntime` reads the engine's own columns for several projections:
+
+| Projection                                        | Column or entry it reads                              |
+| ------------------------------------------------- | ----------------------------------------------------- |
+| `RunSummary.waitingReason`                        | `flows_runs.waiting_reason`                           |
+| Engine-created children and forks in `list`       | `flows_run_parents`, `flows.time-travel.fork-created` |
+| `RunSummary.cancellation` with `source: "engine"` | `cancel_requested_at_ms`, `flows.engine.interrupted`  |
+
+Give the control runtime and the engine one `SqlClient` and those projections
+fill in. Keep them apart and the projections are empty, while cancellation
+still converges, because the request travels through the `ControlExecutor` port
+and the owning driver settles from it.
+
+## What the plane owes a caller
+
+Three properties hold across every implementation of every port, and the rest
+of this package exists to keep them:
+
+1. **Every mutation is idempotent under its key.** A retry answers the first
+   call's receipt rather than doing the work twice. See
+   [Receipts and idempotency](./receipts.md).
+2. **Every mutation is attributed.** The runtime stamps a principal, and a
+   server stamps the one it authenticated rather than the one a client claimed.
+3. **Every mutation leaves evidence beside the state it changed.** The journal
+   entry and the state write commit together, so a reader cannot see one
+   without the other.
+
+## Where to go next
+
+- [Receipts and idempotency](./receipts.md): what a receipt means, and what a
+  second ask is worth.
+- [Ownership, fences, and claims](./ownership.md): why a mutation can answer
+  `ClaimLost`, and what a park releases.
+- [Journal projections](./projections.md): how `watch` turns entries into
+  `ControlEvent` values.

package/docs/concepts/cancellation.md

@@ -0,0 +1,129 @@
+---
+title: "Cancellation attribution"
+description: "A durable cancellation is anonymous on its own. How the journal adds back who asked and why, the three sources in the order they rank, and why a cascade inherits its ancestor's principal."
+sidebar:
+  order: 6
+---
+
+A durable cancellation records that somebody asked and when, and nothing else.
+`flows_runs.cancel_requested_at_ms` is one number. It cannot say who, why, or
+whether this run was asked for by name rather than swept up in an ancestor's
+cascade.
+
+`RunSummary.cancellation` is the attribution the journal adds back:
+
+| Field          | Meaning                                                                   |
+| -------------- | ------------------------------------------------------------------------- |
+| `requestedAt`  | When the cancellation was asked for.                                      |
+| `source`       | `control`, `cascade`, or `engine`.                                        |
+| `principal`    | Who asked. Present on a `control` source and on the `cascade` it started. |
+| `reason`       | Why, as the operator stated it.                                           |
+| `cascadedFrom` | The cancelled ancestor this run was swept up with.                        |
+
+`control` is an operator asking through this plane, and it is the only source
+that can name a principal. `cascade` is a run swept up in an ancestor's
+cancellation. `engine` is everything the runtime decided on its own account: a
+lease expiry, a budget, a supervisor.
+
+## The three sources, in order
+
+`Cancellation.attribute` is the fold, and a run's own evidence outranks its
+ancestors':
+
+1. **A `control.run.cancel-requested` entry names this run.** Somebody asked
+   for it by name, and the entry says who and why.
+2. **A cancelled ancestor exists.** The run reports `cascade`, names the
+   nearest cancelled ancestor, and inherits that ancestor's principal and
+   reason. The honest answer to "who cancelled this child" is the operator who
+   cancelled its parent.
+3. **Neither.** The engine cancelled the run on its own account. There is no
+   principal to report, and inventing one would be worse than saying nothing.
+
+A run counts as cancelled when any of three things is true: the run store's
+`cancel_requested_at_ms` is set, the engine journaled
+`flows.engine.interrupted` with outcome `cancelled`, or an attributed request
+names the run. The third matters because a control plane cancelling a run it
+owns interrupts the fiber rather than writing the request column, and its
+journal entry is the whole record.
+
+```ts
+import * as Cancellation from "@smthrs/control/Cancellation"
+
+const attributed = Cancellation.attribute({
+  runs: [
+    { runId: "run-1", cancelRequestedAt: 10 },
+    { runId: "run-2", parentRunId: "run-1", cancelledAt: 12 }
+  ],
+  requests: new Map([[
+    "run-1",
+    { requestedAt: 10, principal: { id: "ada", kind: "user", stampedAt: 10 }, reason: "budget" }
+  ]])
+})
+
+attributed.get("run-1")
+// { requestedAt: 10, source: "control", principal: { id: "ada", ... }, reason: "budget" }
+attributed.get("run-2")
+// { requestedAt: 12, source: "cascade", principal: { id: "ada", ... }, reason: "budget", cascadedFrom: "run-1" }
+```
+
+## Why the fold is pure and scope-independent
+
+`attribute` reads whatever evidence it is handed and never issues a query, so
+the caller chooses how much to read. `SqlControlRuntime` uses two scopes:
+
+- A **listing** folds the whole database, because every row is going to be
+  answered for anyway.
+- **Reading one run** folds that run and its ancestor chain, which is the
+  smallest scope that can still answer the question.
+
+Cascade is a fact about a run's ancestors, so it cannot be decided one row at a
+time: the request that cancelled a child may be several rounds up the chain.
+Reading one run therefore costs one recursive walk over `parent_run_id` plus
+one spawn-edge read per nesting level, and never grows with the size of the
+database. That is what keeps the cost of steering or cancelling a run
+independent of how many runs exist.
+
+The ancestor walk carries a visited set. A cyclic parent chain is not reachable
+through the engine's own cycle detection, but a projection that hung on corrupt
+ancestry would take the control plane down with it.
+
+## Where the attribution is written
+
+`cancel` writes the principal and the reason onto its
+`control.run.cancel-requested` entry, inside the mutation's own transaction, so
+a cancellation cannot commit anonymously. `resume` records the same pair on its
+`control.run.resume` entry.
+
+The request, attribution, and acceptance receipt commit before the local fiber
+is interrupted. Cancellation then awaits its finalizers without holding the
+mutation semaphore or journal transaction, so cleanup can signal another run
+or use the same durable writer. A second transaction rechecks the original
+ownership fence and commits the terminal status and event together. A terminal
+outcome reached during cleanup is preserved. If cleanup loses ownership, the
+durable request remains for the owner or a later cancel attempt.
+
+Attribution is keyed on the request being newly recorded. `cancel` re-executes
+on every ask, so attributing every ask would journal one
+`control.run.cancel-requested` per ask for a single cancellation. The executor
+answers `already-requested` when the engine column was set before this call
+arrived, and that answer suppresses the second record.
+
+A cancel whose executor reports that the engine row has already settled writes
+no attribution, because nobody cancelled anything, and reconciles the control
+row onto the engine's own status instead. Nothing else converges the two rows,
+so a control row left disagreeing with a settled engine row would list the run
+as live forever.
+
+Over RPC the `Cancel` procedure carries the reason and refuses a caller-named
+principal. The server stamps the identity it authenticated, so a remote
+operator states why and never states who. The principal's `stampedAt` records
+when that authentication happened; it is evidence about an external event,
+never a value any decision is replayed from.
+
+## Where to go next
+
+- [Cancel a run, and restart one](../guides/cancel-and-resume.md): the verb,
+  and what each receipt means.
+- [Run lineage](./lineage.md): the ancestor chain a cascade walks.
+- [Store control state in a database](../guides/durable-storage.md): the only
+  runtime that fills `RunSummary.cancellation` in.

package/docs/concepts/lineage.md

@@ -0,0 +1,132 @@
+---
+title: "Run lineage"
+description: "The one vocabulary four ancestry records project onto: child, fork, and continuation, where each is written, which record wins, and how watch derives exactly one lineage delta per edge."
+sidebar:
+  order: 5
+---
+
+A run's ancestry is recorded by whoever created it, in four different places:
+the run row's `parent_run_id`, `lineage_id`, and `round_ordinal` columns; the
+`flows_run_parents` edge a spawn writes; the `created` and `handed-off` run
+decisions the engine journals; and the `fork-created` marker time travel writes
+on a forked child.
+
+`Lineage` owns the one vocabulary all four project onto, and the pure functions
+that do the projecting, so the durable runtime and the watch stream cannot
+disagree about what a run's ancestry means.
+
+## The vocabulary
+
+`Origin` has three values, and a run with no ancestor has no origin at all:
+
+| Origin         | What it means                                  |
+| -------------- | ---------------------------------------------- |
+| `child`        | Another run spawned it.                        |
+| `fork`         | It was branched off a parent frame.            |
+| `continuation` | It is a later round of one trampoline lineage. |
+
+A rewind is deliberately absent. It truncates a run in place and creates none,
+so it is a thing that happened to a run rather than a reason a run exists.
+
+`Lineage.originOf` is the derivation, and it is pure:
+
+```ts
+import * as Lineage from "@smthrs/control/Lineage"
+
+Lineage.originOf({ parentRunId: "run-1" }) // "child"
+Lineage.originOf({ parentRunId: "run-1", forked: true }) // "fork"
+Lineage.originOf({ parentRunId: "run-1", roundOrdinal: 2 }) // "continuation"
+Lineage.originOf({}) // undefined
+```
+
+A fork wins over a plain child because a fork records `parent_run_id` too.
+Without the marker, every fork would be reported as an ordinary child.
+
+## What a run summary reports
+
+`RunSummary` carries all of it under one vocabulary:
+
+| Field          | Source                                                              | Meaning                                                                                                   |
+| -------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- |
+| `parentRunId`  | `flows_runs.parent_run_id`, else the `flows_run_parents` spawn edge | The run this one branched from: its spawner, the run it was forked off, or the previous trampoline round. |
+| `lineageId`    | `flows_runs.lineage_id`                                             | The trampoline lineage this run is a round of.                                                            |
+| `roundOrdinal` | `flows_runs.round_ordinal`                                          | Which round. Absent means a lineage of one, read as round 0 of itself.                                    |
+| `origin`       | derived                                                             | `child`, `fork`, or `continuation`.                                                                       |
+
+The projection reads both recording places because the engine uses both.
+`parent_run_id` is the trampoline chain: the round before this one. A run that
+another run _spawned_ writes nothing in its own row, because the edge lives in
+the `flows_run_parents` graph that cycle detection walks. A projection that
+read the column alone would report every child of every run as an orphan.
+
+The column wins when a row has both. That is round 1 of a run that was itself
+spawned: its nearest ancestor is the round before it.
+
+## The delta `watch` derives
+
+Three journal entries disclose an edge, and each names a different pair:
+
+| Entry                                                                              | Producer                                    | Delta                                                                     |
+| ---------------------------------------------------------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------- |
+| `flows.engine.run-decision` with `decision: "created"` at round 0 or with no round | [`@smthrs/engine-store`](/api/engine-store) | `{ runId, parentRunId, origin: "child" }`                                 |
+| `flows.engine.run-decision` with `decision: "handed-off"`                          | [`@smthrs/engine-store`](/api/engine-store) | `{ runId, parentRunId, lineageId, roundOrdinal, origin: "continuation" }` |
+| `flows.time-travel.fork-created`                                                   | [`@smthrs/time-travel`](/api/time-travel)   | `{ runId, parentRunId, origin: "fork" }`                                  |
+
+The handoff is what carries a trampoline, and a continuation round's own
+`created` decision is deliberately skipped. The engine journals both in one
+transaction: it creates the next round with
+`{decision: "created", lineageId, roundOrdinal, parentExecutionId}` and records
+`{decision: "handed-off", nextExecutionId}` on the round that finished. Both
+name the same pair, so deriving from both would report one run as a `child` of
+its predecessor on one entry and a `continuation` of it on the other.
+
+The handoff is the one kept, because it reaches a consumer watching the run
+that hands off, which is the run an operator is already following when a
+trampoline advances. Exactly one delta therefore names each continuation round,
+whichever round of the lineage the consumer is watching.
+
+```ts
+Lineage.derive({
+  sequence: 12,
+  kind: Lineage.runDecisionEventType,
+  runId: "run-1",
+  occurredAt: 1_700_000_000_000,
+  payload: { decision: "handed-off", nextExecutionId: "run-2", lineageId: "run-1", roundOrdinal: 1 }
+})
+// {
+//   sequence: 12,
+//   kind: "control.run.lineage",
+//   runId: "run-1",
+//   occurredAt: 1700000000000,
+//   payload: { runId: "run-2", parentRunId: "run-1", lineageId: "run-1", roundOrdinal: 1, origin: "continuation" }
+// }
+```
+
+Everything else derives nothing. This is a projection over entries the control
+plane did not write, so an entry it does not recognize is not an error, and a
+`created` decision that names no parent discloses no ancestry.
+
+## Selecting on lineage
+
+`list` filters on the same fields, so an operator can ask both ancestry
+questions:
+
+```ts
+const children = yield * control.list({ _tag: "runs", filters: { parentRunId: "run-17" } })
+const rounds = yield * control.list({ _tag: "runs", filters: { lineageId: "run-17" } })
+```
+
+The durable listing covers every row in `flows_runs`, not only the runs the
+control plane launched itself. A child, a fork, and a later trampoline round
+are all created by the engine straight into the run store, and a plane that
+listed only its own launches could not answer what a run spawned. Runs the
+plane launched keep launch order; the rest follow in creation order. A run
+whose `state_json` is not a control summary is projected from the run row's own
+columns instead, with the engine's `flowName` as its `flowId`.
+
+## Where to go next
+
+- [Find runs and page through them](../guides/list-runs.md): the filters as a
+  task.
+- [Watch a run's events](../guides/watch-a-run.md): where the delta arrives.
+- [Time travel on smithers.sh](/docs/concepts/time-travel/): what makes a fork.

package/docs/concepts/ownership.md

@@ -0,0 +1,139 @@
+---
+title: "Ownership, fences, and claims"
+description: "How a fence makes every owner-sensitive write a compare-and-swap, what a park releases, why a resume can be scoped to runs this plane launched, and how a resume delegation reaches the process that can act on it."
+sidebar:
+  order: 3
+---
+
+Several processes share one control database, and only one of them may be
+driving a given run. Ownership is how the plane decides which, and a **fence**
+is the token that decides it.
+
+A fence is a serialized owner identity: a host id, a process id, and a nonce
+that is regenerated on every claim. Every owner-sensitive write presents its
+fence, and the runtime turns that into a single SQL compare-and-swap, so a
+stale writer loses the `UPDATE` rather than racing a read-then-write. A fence
+taken before a park is not the fence held after the resume that follows it, and
+the stale one is refused.
+
+When a write presents a fence the row has moved past, the plane answers
+[`ClaimLost`](../troubleshooting.md).
+
+## Status is ownership, spelled for an operator
+
+`SqlControlRuntime` maps the control plane's vocabulary onto the run store's:
+
+| Control status                        | Run store status | Ownership            |
+| ------------------------------------- | ---------------- | -------------------- |
+| `accepted`, `running`                 | `running`        | Held by this process |
+| `accepted` after an executor declines | `suspended`      | Released             |
+| `parked`, `waiting-approval`          | `suspended`      | Released             |
+| `cancelled`, `completed`, `failed`    | same             | Released, terminal   |
+
+A claim writes `accepted`. `Control.run` promotes it to `running` when its
+executor takes the launch; an explicit resume leaves it `accepted` until the
+driver writes another status. An `ownerId` distinguishes an owned `accepted`
+run from a released pending launch. Losing a claim to an owned `accepted` or
+`running` row means a live peer holds it.
+
+The authoritative `RunSummary` is written into the row's `state_json` by the
+same fenced `UPDATE` that moves the status, so a projection can never be read
+out of step with the lifecycle.
+
+## A park releases the row, and records who parked it
+
+A parked execution releases its owner columns. That is what makes it resumable
+at all, and it is also why every process sharing the database can see the park.
+
+The fence the park was written under is kept in `RunSummary.parkedBy`, and only
+on a park. It is the one thing left on the row that says which host parked the
+run, so that host recognizes its own park and a short-lived process that would
+drive the run and then exit can tell the execution is not its to take up.
+
+`RunSummary.waitingReason` is the other half of the picture, and the control
+plane only ever reads it. The engine writes it when it parks a run and clears
+it on the wake, so an operator park written through
+`ControlRuntime.writeStatus(runId, fence, "parked")` leaves the column empty.
+An empty column is exactly how an operator park is told apart from an engine
+park, and several behaviors turn on that:
+
+| `waitingReason`  | A steer arriving | A monitor's reading |
+| ---------------- | ---------------- | ------------------- |
+| `event`          | Resumes the run  | Ordinary park       |
+| `released`       | Resumes the run  | Ordinary park       |
+| `approval`       | Leaves it parked | `awaiting-human`    |
+| `timer`, `quota` | Leaves it parked | Ordinary park       |
+| absent           | Leaves it parked | `awaiting-human`    |
+
+## Claim scope: launched, or any
+
+`ControlRuntime.resume` joins a non-terminal run whose fence this process
+still holds, including an `accepted` run. A join preserves the original fence.
+An `accepted` run released by `releasePending` is claimable under a new fence.
+
+`scope: "launched"` restricts claims to runs recorded in `control_runs`, the
+shared index of control-launched runs. Both `Control.resume` and `Control.run`
+with a Resume input, plus every steer wake, pass this scope. An engine-created
+child, fork, or later trampoline round keeps its own continuation and driver.
+
+`scope: "any"`, also the runtime default, is a trusted low-level runtime
+capability for hosts that can drive the claimed execution. It is not the
+public Control resume contract.
+
+## Explicit resume records a journal intent
+
+Both public resume spellings journal `control.run.resume`. A suspended run
+outside the launch index remains unclaimed; the receipt is `Accepted` after
+the journal intent is recorded. A live peer's owned run fails with `ClaimLost`.
+A caller or journal subscriber must drive the execution, including after a
+successful claim. An `Accepted` receipt does not establish that work started.
+
+Explicit resume never calls `ControlExecutor.resumeRun`. A run the caller can
+claim creates no `pendingResumes` entry.
+
+A run a live host parked is the exception. `resume` may take a park only after
+a same-host probe proves the parking process dead. Otherwise the runtime
+refuses with `ClaimLost` naming the host in `parkedBy`, and `resume` hands the
+restart to that host. It journals `control.run.resume` with `handedTo` and
+records a `requestResume` delegation whose `consent` is that entry's journal
+sequence. The receipt is `Accepted` with `handedTo`. The parking host takes the
+delegation up on its next poll, records the per-release retry permission under
+that sequence, and re-drives the run. See
+[Cancel a run, and restart one](../guides/cancel-and-resume.md#a-run-a-live-host-parked).
+
+## Node approval records a durable resume delegation
+
+A decision on an in-run approval restarts the run server-side, and the process
+that decides is usually not the process hosting the execution: an operator's
+`smthrs approvals approve`, a gateway, a second CLI.
+
+So the intent is recorded durably rather than published in process:
+
+1. `requestResume(runId)` writes the delegation and returns its sequence.
+   `RunSummary.pendingResume` reports that sequence while it is outstanding.
+   An approval delegation carries no `consent`: it is background intent, and
+   it keeps an operator's consent that no host has taken up yet.
+2. The plane offers it to its own executor through `ControlExecutor.resumeRun`.
+   An executor that answers `resuming` has claimed the row and is driving, so
+   the delegation is cleared with `clearResume(runId, sequence)`.
+3. An executor that answers `unknown` leaves the delegation standing.
+   `pendingResumes` is what every host polls, and a run parked by a process
+   that has since exited is taken up by whichever host can drive it once the
+   delegation has gone unanswered for the run store's heartbeat staleness
+   window.
+
+The sequence check is what makes the clear safe. A resume requested between the
+read and the clear has a higher sequence and survives, so the host takes it up
+on its next tick instead of losing it.
+
+A settled run's delegation is never reported: no host will ever take it up, and
+reporting it forever would turn a finished run into an unbounded backlog.
+
+## Where to go next
+
+- [Cancel a run, and restart one](../guides/cancel-and-resume.md): the verbs
+  that meet these rules head on.
+- [Connect an execution engine](../guides/implement-an-executor.md): the port
+  that turns a delegation into a running fiber.
+- [Ownership on smithers.sh](/docs/concepts/ownership/): the same fence, from
+  the engine's side.