@smthrs/control 0.0.0-stage → 1.0.0-rc.3

package/docs/guides/list-runs.md

@@ -0,0 +1,160 @@
+---
+title: "Find runs and page through them"
+description: "List the flows a host can plan and the runs it knows about: the run filters, the page bounds a listing enforces, the cursor contract, and the two refusals a listing answers instead of guessing."
+sidebar:
+  order: 2
+---
+
+`list` answers four questions under one verb, selected by the request's tag:
+what can this host plan, what runs exist, which triggers are registered, and
+what became of each trigger occurrence.
+
+## List the flows a host can plan
+
+```ts
+const catalogued = yield * control.list({ _tag: "flows" })
+const flows = catalogued._tag === "flows" ? catalogued.items : []
+// [{ flowId: "ops/Deploy", description: "Deploys one build" }]
+```
+
+The items come from the [registry](/api/registry) when it has discovered
+anything, and fall back to the runtime's own flow catalog when it has not.
+`warnings` carries the registry's discovery diagnostics when a scan produced
+any, so a flow that failed to load is visible rather than silently missing.
+
+## List runs
+
+```ts
+const listed = yield * control.list({ _tag: "runs", filters: { status: "parked" } })
+const runs = listed._tag === "runs" ? listed.items : []
+```
+
+Six filters are supported, and they combine:
+
+| Filter        | Selects                                                      |
+| ------------- | ------------------------------------------------------------ |
+| `runId`       | Exactly one run, read directly rather than scanned.          |
+| `flowId`      | Runs of one flow.                                            |
+| `status`      | Runs in one of the seven `RunStatus` values.                 |
+| `terminal`    | `true`: completed, failed or cancelled; `false`: unfinished. |
+| `parentRunId` | The runs one run spawned, forked, or handed off to.          |
+| `lineageId`   | Every round of one trampoline lineage.                       |
+
+Filtering on `runId` is one read. Other queries filter before decoding the
+selected page. An executor that supplies observed status is post-filtered to
+keep the returned status consistent: a `status` or `terminal` page walks the
+source until it is full, so it can observe more runs than `limit`.
+
+Use `order: "newest"` to order by creation time descending, with stable
+sequence/id tie breakers. Omission preserves the historical order. A cursor
+is bound to both its filters and order. For example, duration history asks:
+
+```ts
+const recent = yield * control.list({
+  _tag: "runs",
+  filters: { flowId: "ops/Deploy", terminal: true },
+  order: "newest",
+  limit: 20
+})
+```
+
+`filters.principalId` selects the runs a principal with that id launched
+(`RunSummary.launchedBy`). It narrows a listing and is not a tenant boundary:
+over RPC the server already restricts a principal that is not an operator to
+its own runs, whatever filter it sends. See
+[serve over RPC](./serve-over-rpc.md#who-reads-which-runs).
+
+## List triggers and their fires
+
+```ts
+const registered = yield * control.list({ _tag: "triggers", filters: { enabled: true } })
+const triggers = registered._tag === "triggers" ? registered.items : []
+// [{ triggerId: "nightly-lint", flowId: "lint", cron: "0 3 * * *", enabled: true, nextOccurrencesMs: [...], ... }]
+
+const ledger = yield * control.list({ _tag: "fires", filters: { runId } })
+const fires = ledger._tag === "fires" ? ledger.items : []
+// [{ triggerId: "nightly-lint", occurrenceAtMs, outcome: "launched", runId }]
+```
+
+`triggers` filters on `triggerId`, `flowId`, and `enabled`; `fires` filters on
+`triggerId`, `runId`, and `outcome`, and the ledger comes back newest first. Both
+are read through the `DispatchReader` port the host composes over its trigger
+store. A host that provides none refuses both variants with `InvalidInput` and
+the issue `this host serves no trigger store`. It never answers an empty page
+for them, because an empty page would claim the host has no triggers when it
+cannot read its store at all.
+
+## Page through the result
+
+A listing is bounded, always:
+
+| Bound             | Value                                |
+| ----------------- | ------------------------------------ |
+| Default page size | `ControlSchema.defaultPageSize`, 100 |
+| Maximum page size | `ControlSchema.maxPageSize`, 500     |
+
+`nextCursor` is present exactly when more rows follow. Pass it back as
+`cursor`:
+
+```ts
+import * as Effect from "effect/Effect"
+
+const everyRun = Effect.gen(function*() {
+  const control = yield* Control
+  const items = []
+  let cursor: string | undefined
+  do {
+    const page = yield* control.list({
+      _tag: "runs",
+      limit: 200,
+      ...(cursor === undefined ? {} : { cursor })
+    })
+    if (page._tag !== "runs") break
+    items.push(...page.items)
+    cursor = page.nextCursor
+  } while (cursor !== undefined)
+  return items
+})
+```
+
+Only a cursor this listing issued is accepted. A `limit` outside 1 to 500, and
+an unparsable cursor, are both refused with `InvalidInput` rather than answered
+with a plausible page:
+
+```text
+InvalidInput: limit: must be an integer between 1 and 500, received 0
+InvalidInput: cursor: must be a cursor this listing returned, received "abc"
+```
+
+## What a summary carries
+
+`RunSummary` is the projection every listing returns. Beyond `runId`,
+`flowId`, `status`, `createdAt`, and `updatedAt`:
+
+| Field                                                | Present when                                                                                          |
+| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------- |
+| `planId`, `planDigest`                               | The run was launched from a plan on this plane.                                                       |
+| `ownerId`                                            | A process holds the row.                                                                              |
+| `parentRunId`, `lineageId`, `roundOrdinal`, `origin` | The run has an ancestor. See [run lineage](../concepts/lineage.md).                                   |
+| `waitingReason`                                      | The engine parked the run and named a reason.                                                         |
+| `parkedBy`                                           | The park was written under a fence.                                                                   |
+| `pendingResume`                                      | A resume has been recorded and no host has taken it up.                                               |
+| `steering`                                           | The notification queue answered how many steers are pending.                                          |
+| `cancellation`                                       | Somebody or something cancelled the run. See [cancellation attribution](../concepts/cancellation.md). |
+
+`steering.pending` is read from the queue rather than from a column, because
+pending is admitted minus promoted and the queue owns both halves. A queue that
+cannot answer leaves the field absent, because "not known" is representable and
+it is the truth.
+
+Several of these fields are filled in only by the durable runtime, and only
+when it shares a database with the engine. See
+[Store control state in a database](./durable-storage.md).
+
+## Where to go next
+
+- [Watch a run's events](./watch-a-run.md): the same runs, as they change.
+- [Run lineage](../concepts/lineage.md): what `parentRunId` and `lineageId`
+  select.
+- [`smthrs runs list`](/cli/runs) and `smthrs runs show`: the operator
+  surface over this verb.

package/docs/guides/monitor-runs.md

@@ -0,0 +1,176 @@
+---
+title: "Monitor a run and heal it"
+description: "Classify a run's health from durable evidence, beat over the control plane, and apply a bounded remedy: the seven healths, the two records a beat writes, and why autoHeal is empty by default."
+sidebar:
+  order: 10
+---
+
+A control plane answers "what is this run doing". `Monitor` answers the
+question after it: is that all right, and if not, what now.
+
+`classify` is pure, so the vocabulary an operator reads on a dashboard is the
+one a heal loop branches on and the one a test can enumerate.
+
+## Classify one observation
+
+```ts
+import * as Monitor from "@smthrs/control/Monitor"
+
+Monitor.classify({
+  summary,
+  events, // the run's journal, oldest first
+  beatsWithoutProgress: 3,
+  stallBeats: 3
+})
+// "stalled"
+```
+
+The rules run in this order, and each earns its place by naming a different
+response:
+
+| Condition                                     | Health           | Because                                                               |
+| --------------------------------------------- | ---------------- | --------------------------------------------------------------------- |
+| No summary                                    | `unknown`        | Nothing to say, and nothing to do.                                    |
+| `failed`                                      | `failing`        | The run itself reported the failure.                                  |
+| `completed`, `cancelled`                      | `healthy`        | A finished run needs nothing.                                         |
+| `waiting-approval`, or parked on `approval`   | `awaiting-human` | A human owes it an answer.                                            |
+| Parked with no waiting reason                 | `awaiting-human` | Only an operator's own park writes no reason, so a person stopped it. |
+| `roundOrdinal` at or past `roundBound`        | `runaway-loop`   | The lineage loops without converging.                                 |
+| The last settled attempt failed               | `failing`        | The run is alive and its work is not landing.                         |
+| No progress for `stallBeats`, an attempt open | `wedged-node`    | One attempt started and never settled.                                |
+| No progress for `stallBeats`                  | `stalled`        | Nothing is happening and nothing is in flight.                        |
+| Anything else                                 | `healthy`        | Entries are still arriving.                                           |
+
+`awaiting-human` outranks `failing` on purpose. A run parked for approval after
+a failed attempt is waiting for a person, and resuming or cancelling it would
+take the decision away from them. A park with no reason is the same case: the
+engine names every park it makes, so an unnamed one was an operator's, and
+undoing a deliberate act is the worst thing an unattended loop can do.
+
+Progress is measured from `flows.engine.attempt-started` and
+`flows.engine.attempt-finished`, which the engine journals as a pair. An excess
+of starts is an attempt still in flight. The classification reads durable
+evidence only, never an in-process fiber, which is what lets a monitor watch a
+run in another process.
+
+`roundBound` defaults to 32 rounds.
+
+## Beat over the control plane
+
+`Monitor.run` requires `Control` and `Journal`, and nothing else. Its first
+beat reads a finite snapshot. Later beats pass the last `event.cursor` as
+`afterCursor` and fold only new events into an open-attempt count and the last
+attempt outcome. Monitor beat and heal records advance the cursor but do not
+count as progress. Providers that omit cursor metadata use `afterSequence`
+after each completed snapshot. The loop retains no historical event array:
+
+```ts
+const report = yield * Monitor.run({
+  runId: "run-17",
+  monitorId: "oncall-supervisor",
+  intervalMs: 5_000,
+  maxChecks: 60,
+  stallBeats: 3,
+  autoHeal: ["stalled", "wedged-node"]
+})
+// { runId: "run-17", beats: [...], health: "healthy" }
+```
+
+| Option       | Default                                                |
+| ------------ | ------------------------------------------------------ |
+| `monitorId`  | `default`                                              |
+| `intervalMs` | 1,000                                                  |
+| `maxChecks`  | 10                                                     |
+| `stallBeats` | 3                                                      |
+| `roundBound` | 32                                                     |
+| `autoHeal`   | none                                                   |
+| `heal`       | `Control.resume` and `Control.cancel`, per `remedyFor` |
+
+Each beat lists the run, replays its journal, classifies, and records
+`control.monitor.beat` carrying the remedy it is about to attempt, _before_
+applying it, so a monitor that crashes mid-heal leaves the evidence of what it
+decided.
+
+The remedy is a second record, `control.monitor.healed`, written only once the
+heal returned an `Accepted` or `AlreadyApplied` receipt. A heal that failed,
+was refused as a `Conflict`, or found the run already `Terminal` must not leave
+a durable record saying the run was healed, and only an applied remedy resets
+the stall evidence. A `Terminal` receipt ends the loop, because there is
+nothing left to remedy.
+
+Both records are excluded from the progress measurement. A monitor that counted
+its own bookkeeping as progress could never observe a stall: the beat it wrote
+at the top of the loop would be the new entry it congratulated the run for at
+the bottom.
+
+## Choose what it may do
+
+`remedyFor` maps a health onto an action, and `autoHeal` decides which of those
+the monitor may actually apply:
+
+| Health                    | Remedy                                                   |
+| ------------------------- | -------------------------------------------------------- |
+| `stalled`, `wedged-node`  | `resume`: nobody is driving the run, so claim it.        |
+| `failing`, `runaway-loop` | `cancel`: it will not get better by being driven harder. |
+| everything else           | `none`                                                   |
+
+`autoHeal` is empty by default, because a monitor that healed by default would
+cancel a run the first time it looked at one.
+
+A remedy resets the stall count, so one stall produces one resume rather than
+one per beat.
+
+## Two monitors on one run
+
+Nothing on the control plane leases a run to one watcher. Two monitors both
+beat and both remedy, so `monitorId` is what makes their evidence tellable
+apart and their remedies distinct:
+
+- Every record this monitor writes has source `/control/monitor/<monitorId>`
+  and carries `monitorId` in its payload.
+- The built-in remedies key on
+  `monitor:<monitorId>:<remedy>:<runId>:<beat>`.
+
+A remedy must be idempotent on the control plane. A custom `heal` owes the same
+property:
+
+```ts
+const report = yield * Monitor.run({
+  runId: "run-17",
+  autoHeal: ["failing"],
+  heal: ({ runId, health, beat }) =>
+    control.cancel({
+      runId,
+      reason: `page-oncall:${health}`,
+      idempotencyKey: `oncall:${runId}:${beat}`
+    })
+})
+```
+
+## Alert on what the monitor wrote
+
+`Monitor.beatEventType` is a journal kind, so an alert policy in
+[`@smthrs/notifications`](/api/notifications) can detect on it directly:
+
+```ts
+const rules = {
+  defaults: { severity: "warning", owner: "oncall" },
+  rules: { "wedged-node": { afterMs: 900_000, runbook: "https://runbook/wedged-runs" } },
+  detectors: {
+    "wedged-node": { field: "health", value: "wedged-node", eventTypes: [Monitor.beatEventType] }
+  }
+}
+```
+
+Narrowing the detector to the beat keeps the heal record, which names the same
+health, from reopening a condition the next beat closed. The complete loop, with
+two real durable runs, is
+[`examples/src/38-monitor-and-alert.ts`](https://github.com/smithersai/smithers/blob/main/examples/src/38-monitor-and-alert.ts).
+
+## Where to go next
+
+- [Watch a run's events](./watch-a-run.md): the stream a beat reads.
+- [Cancel a run, and restart one](./cancel-and-resume.md): the two default
+  remedies.
+- [Test against the control plane](./testing.md): classify is pure, so most of
+  a monitor needs no stack at all.

package/docs/guides/observe-health.md

@@ -0,0 +1,147 @@
+# Configure observational health
+
+`Health` separates authoritative lifecycle from observed activity, derived health,
+human attention, and reading freshness. A live process is not proof of semantic
+work, and an approval wait always remains an approval wait.
+
+Register trusted TypeScript checkers through the native CLI's
+`Application.Config.health`, binding checker IDs to flow IDs. The desktop local
+server accepts the same configuration for role and harness IDs. Pure flow bodies
+and browser requests do not contain executable health functions.
+
+```ts
+import * as Health from "@smthrs/control/Health"
+import { Effect, Schema } from "effect"
+
+const Settings = Schema.Struct({ worker: Schema.String })
+declare const inspectWorker: (id: string) => Effect.Effect<Health.ProbeReport, unknown>
+
+const checker: Health.HealthChecker<typeof Settings.Type> = {
+  id: "worker.activity",
+  configSchema: Settings,
+  probe: (_context, config) => inspectWorker(config.worker)
+}
+
+const health: Health.HealthConfig = {
+  checkers: [checker],
+  bindings: {
+    "build/review": {
+      checkerId: "worker.activity",
+      config: { worker: "reviewer" },
+      policy: { intervalMs: 5_000, timeoutMs: 2_000, ttlMs: 20_000, stallAfterMs: 120_000 }
+    }
+  },
+  limits: { maxSubjects: 128, maxConcurrentProbes: 8 }
+}
+```
+
+Reports contain `activity` (`working`, `idle`, `needs-input`, or `unknown`) and an
+optional closed reason code. Silence and changing terminal output do not establish
+any of these semantics. The lifecycle defaults return unknown activity. Explicit
+`no-progress` or `unreachable` reasons may mark health unhealthy. Known engine
+timer, event, quota, and approval waits retain their authoritative meaning.
+
+A run parked on `released` reports `health: "awaiting-human"`,
+`attention: "needs-resume"`, and reason `released`, whatever a probe says. Its
+owner released executions while still alive, usually because the run lease
+lapsed on a stalled host, and nothing restarts them until an operator runs
+`smthrs runs resume <run>`.
+
+## Detect a session waiting on a person
+
+`jev.session` is registered in every host, so a binding is the whole opt-in. It is
+registration, not a binding: a host that never names it keeps the lifecycle
+default it had.
+
+```ts
+import * as Health from "@smthrs/control/Health"
+
+const health: Health.HealthConfig = {
+  bindings: { terminal: { checkerId: "jev.session", exposeOutput: true } }
+}
+```
+
+The desktop local server writes that binding itself: with a subscription evaluator configured
+in its process it binds `jev.session` with `exposeOutput: true` for every session
+subject it resolves, so a session's output tail is sent to the gateway with zero
+data retention; without the key its configuration is exactly what a caller passed,
+and a subject a caller bound keeps that caller's checker. The native CLI host
+observes runs rather than sessions, so it never binds this checker.
+
+The checker sends the session's `alive`, `exitCode`, and the newest 4 KiB of its
+output to Jev, TypeSafe's decision model, through the Vercel AI Gateway, with zero
+data retention. Jev answers one choice question (`working`, `idle`, `needs-input`)
+and one boolean question about whether the output ends waiting for a person, both
+in one request that returns in about 300 ms. `exposeOutput: true` is required:
+without it the host sends no tail and the probe answers unknown.
+
+An answer becomes a report only at confidence 0.7 or above. TypeSafe claims 76%
+agreement with a human rater, so a Jev that is merely leaning is a coin flip
+dressed as a reading: a false `needs-input` pages a person who is not needed, and
+a false `idle` retires an agent that is still working. `needs-input` carries the
+reason `prompt-detected`, which the rollup turns into `attention: "needs-input"`.
+A reading below the floor is Jev's own answer and stays `unknown`.
+
+## When Jev cannot answer
+
+A host that binds `jev.session` supplies its existing evaluator through
+`Health.makeRegistry(config, kind, evaluator)` or
+`JevSessionChecker.makeJevSessionChecker({ evaluator })`. Native hosts resolve
+that judge from their subscription seats. No separate gateway credential is
+read. Without a judge the probe fails with `unconfigured` and remains
+`probe-error`.
+
+An unavailable subscription, a plan refusal, a rate limit, a dead socket, an unreadable body and
+the call's own 45 s deadline fail the probe the same way, with the reason that
+names the fault: `http` carrying the provider's status, `unreachable`, `timeout`,
+or `malformed`. `Health.evaluate` records any failing probe as `outcome: "error"`
+with reason `probe-error` and no report, so the rollup reads the subject `stale`
+with activity `unknown`, health `unknown`, and reason `probe-error`, and the
+binding's `backoff` spaces the retries from five up to sixty seconds. A monitor
+that cannot see is worse when it looks calm, so it says so.
+
+Two cases are not Jev failing and keep the lifecycle answer: a session that is no
+longer alive, and a session whose output tail the host did not expose. There is
+nothing to ask about either one.
+
+The native host admits at most 128 subjects and eight simultaneous probes by
+default. It scans every five seconds, checks each subject every five seconds,
+times out a probe after two seconds, expires observations after twenty seconds,
+and allows two minutes of missing flow progress before classifying it as stalled.
+Failure backoff starts at five seconds and caps at sixty seconds. Per-binding
+policies and host limits are validated before monitoring starts. Unknown checker
+IDs use the lifecycle fallback; malformed configured policies or checker-specific
+configuration fail startup.
+
+The host stamps ownership, lifecycle, timing, and evidence position. Reports from
+a changed owner are discarded. A successful observation is recorded under
+`control.status.observed` before publication. Equal evidence positions are ordered
+by durable journal sequence, never by comparing opaque owner identities. Unchanged
+observations are coalesced and renewed before expiry. The gateway's existing
+run-summary projection carries `statusRollup`, and clients expire its semantic
+activity independently if the host stops answering.
+
+Native flow monitoring reads events incrementally; the callback receives at most
+256 recent execution events per beat. Observer/notification bookkeeping does not
+count as progress. Derived gateway projections retain bounded recent health
+observations while preserving their original journal cursor, so repeated probes
+do not exhaust the projection's event budget. Raw run-event history keeps its
+existing resource limits and durable journal retention remains the host's policy.
+
+`Health.evaluate` is transport-neutral for other trusted hosts. It returns a
+validated observation as an Effect; the host checks ownership again, commits it,
+and passes `{ observation, sequence }` to `Health.rollup`. `Health.makeRegistry`
+and `registry.resolve(key)` are synchronous constructors. Callback Effects must
+have their service requirements provided before registration.
+
+These callbacks are trusted host code, not sandboxed plugins. Read-only context
+types do not constrain captured ambient authority, and cooperative Effect
+interruption cannot preempt arbitrary synchronous JavaScript. Status output omits
+raw exceptions, terminal contents, and arbitrary metrics. Probe outcomes and
+latency are measured by `smithers.health.probes` and
+`smithers.health.probe_duration_ms`; the probe span is `smithers.health.probe`.
+
+Checker output never authorizes a control mutation. Production observation starts
+with `autoHeal: []`. Existing Alerts policies can consume recorded status fields
+with an explicitly configured real sink; a status observation alone does not
+claim notification delivery.

package/docs/guides/postgres-tests.md

@@ -0,0 +1,5 @@
+# PostgreSQL inventory tests
+
+The default unit suite needs no PostgreSQL server. Set `SMITHERS_TEST_PG_URL` to run `test/SqlControlRuntimePostgres.test.ts` against a dedicated test database.
+
+`//packages/smithers/control:postgresInventory` runs the same assertions on Linux with the `adapterPostgresDatabase` service and loopback networking. It checks filtered inventory pagination, duplicate and absent IDs, empty filters, large ID sets, and time boundaries.