@osolmaz/pi-workflows 0.1.0 → 0.3.0

package/docs/live-replay-protocol.md

@@ -0,0 +1,155 @@
+# Live replay protocol
+
+The Rust viewer (`tui/`) can watch runs in two ways: by reading run bundles
+directly from the filesystem (the default, in-process) or by connecting to a
+`piw serve` WebSocket server. Both paths produce the same semantic state; the
+protocol below is the network form of that state. Protocol id:
+`pi-workflows.replay.v1`.
+
+The server is a reader like any other: it only consumes run bundles (see
+[run-bundles.md](run-bundles.md)) and never writes them. The protocol has no
+authentication, so the server only accepts loopback bind addresses and refuses
+to start on anything else; bundles contain private data, and remote viewing
+goes through an SSH tunnel. Handshakes that
+carry an `Origin` header are rejected: browsers always send one, and a web
+page must not be able to read bundles by opening a WebSocket to localhost.
+
+## Transport and framing
+
+A single WebSocket endpoint (`/ws`). Every message is one JSON object with a
+`type` field. Unknown message types and unknown fields must be ignored by both
+sides. The server sends `hello` on connect; a client that does not recognize
+the protocol id must disconnect.
+
+```json
+{ "type": "hello", "protocol": "pi-workflows.replay.v1" }
+```
+
+## Run views
+
+The unit of synchronization is the **run view**, the semantic state of one
+run:
+
+```json
+{
+  "manifest": { … },
+  "workflow": { … },
+  "state": { … },
+  "events": [ … ],
+  "session": {
+    "binding": { … },
+    "entries": [ … ],
+    "events": [ … ],
+    "eventsMalformed": false,
+    "eventsTornTail": false,
+    "capture": { … }
+  },
+  "live": true,
+  "possiblyInterrupted": false
+}
+```
+
+- `manifest`, `workflow`, `state`, `events`, and every `session` field are the
+  bundle documents verbatim. `workflow` is the definition snapshot,
+  top-level `events` are parsed workflow trace lines, `session.entries` are
+  settled Pi entries, `session.events` are normalized temporal events, and
+  `session.capture` is capture integrity. `session.eventsMalformed` and
+  `session.eventsTornTail` are derived transport diagnostics from the live
+  tailer, not bundle documents. `session` is `null` until a binding exists.
+- `live` is true while the run status is non-terminal and the bundle is still
+  growing. `possiblyInterrupted` is true when the status is `running` but the
+  bundle has not changed for 60 seconds.
+- Artifact references inside the view stay references; contents are fetched
+  on demand.
+
+Because the full trace and session history are part of the view, replay
+scrubbing is a pure client-side operation; rewinding never requires the
+server. Clients order session events by `seq` and use `at` only for playback
+timing.
+
+## Snapshot, then patches
+
+State synchronization follows a snapshot-then-patch model. After a client
+subscribes to a run, the server sends one `run_snapshot`, then a stream of
+`run_patch` messages:
+
+```json
+{ "type": "run_snapshot", "runId": "…", "revision": 3, "view": { … } }
+{ "type": "run_patch", "runId": "…", "revision": 4, "patch": [
+  { "op": "append", "path": "/events", "value": [ { "seq": 18, … } ] },
+  { "op": "replace", "path": "/state", "value": { … } }
+] }
+```
+
+- `revision` increases by exactly 1 per patch. A client that observes a gap
+  must resubscribe and take a fresh snapshot.
+- `patch` is JSON Patch (RFC 6902) plus one extension op: `append`, whose
+  `value` is an array of items appended to the array at `path`. Semantically
+  `append` equals a sequence of `add` ops at `/-`; it exists so that the
+  common case (trace and session growth) stays compact and readable.
+- Session growth uses `append` at `/session/entries` and `/session/events`.
+  Capture changes use `replace` at `/session/capture`. Changes to the derived
+  tail diagnostics use `replace` at `/session/eventsMalformed` and
+  `/session/eventsTornTail`.
+- The server waits 50 ms after a filesystem notification before refreshing,
+  so one token burst normally becomes one revision. Batch boundaries never
+  merge or alter event records.
+- Patches are computed against the previous view revision; applying them in
+  order reproduces the server's view exactly.
+
+## Messages
+
+Client to server:
+
+| type             | fields          | meaning                      |
+| ---------------- | --------------- | ---------------------------- |
+| `watch_runs`     | —               | subscribe to the run listing |
+| `watch_run`      | `runId`         | subscribe to one run's view  |
+| `unwatch_run`    | `runId`         | end a run subscription       |
+| `fetch_artifact` | `runId`, `path` | request artifact contents    |
+
+Server to client:
+
+| type           | fields                       | meaning                                         |
+| -------------- | ---------------------------- | ----------------------------------------------- |
+| `hello`        | `protocol`                   | sent once on connect                            |
+| `runs`         | `runs`                       | full run listing (summaries), re-sent on change |
+| `run_snapshot` | `runId`, `revision`, `view`  | full view after subscribe                       |
+| `run_patch`    | `runId`, `revision`, `patch` | incremental view update                         |
+| `artifact`     | `runId`, `path`, `content`   | artifact contents (UTF-8)                       |
+| `error`        | `message`, `runId?`          | request failed                                  |
+
+Artifact requests are answered only from files below the artifact directory
+declared by `manifest.paths.artifacts`. Paths outside that directory and
+symlinks whose canonical targets leave it are refused. Responses are capped at
+4 MiB of actual file size; anything else produces an `error`.
+
+Run listing summaries are the manifest plus `live` and
+`possiblyInterrupted`:
+
+```json
+{ "type": "runs", "runs": [ { "manifest": { … }, "live": true, "possiblyInterrupted": false } ] }
+```
+
+The run listing is small and changes rarely, so it is always sent whole; only
+run views use patches.
+
+## Reconnection
+
+The native client treats the run listing, selected run, and pending artifact
+reads as desired state rather than one-shot commands. After a connection closes,
+it keeps the cached run visible with a stale/reconnecting label, retries with
+bounded backoff, sends `watch_runs` after the next valid `hello`, and restores
+the current `watch_run`. A reconnect receives a fresh snapshot before later
+patches. Pending artifact reads are resubmitted once per connection.
+
+## Filesystem semantics behind the protocol
+
+The server watches the runs directory (inotify with polling fallback) and
+tails `trace.ndjson`, `session/entries.ndjson`, and `session/events.ndjson`
+incrementally. Torn final NDJSON lines are buffered until complete. The server
+also re-reads atomic `session/capture.json` changes. `state.json` and
+`manifest.json` are
+re-read on change; a `state.json` whose `traceSeq` is older than the last
+tailed trace event is stale and is replaced when the writer catches up. After
+a terminal status, watching stops.

package/docs/plans/2026-08-04-controller-runtime-plan.md

@@ -0,0 +1,169 @@
+---
+title: Add a durable controller runtime
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-04
+updated: 2026-08-04
+status: implemented
+---
+
+# Controller runtime plan
+
+Pi Workflows needs a controller mode for automation that spans repeated events, external state changes, and process restarts. The design in [CONTROLLERS.md](../CONTROLLERS.md) follows the Kubernetes controller pattern. Durable resources hold desired and observed state, events enqueue resource keys, and each reconciliation reads current facts before acting.
+
+The implementation keeps the graph engine focused on finite jobs. Controllers start and observe workflows through a child-run interface. Workflow graphs keep their finite execution model.
+
+## Shipped design
+
+The implementation follows [CONTROLLERS.md](../CONTROLLERS.md) with these resolved choices:
+
+- The local store uses `better-sqlite3` because the Node 22 SQLite module still emits an experimental warning.
+- Controllers are discovered from `.pi/controllers/` and `~/.pi/agent/controllers/`.
+- The existing `pi-workflows` executable provides read-only `controllers` and `controller` commands. Headless workers use the public `ControllerManager` API.
+- The TypeScript CLI lists and inspects resources. The Rust viewer understands interrupted workflow runs. Resource views stay in the CLI for this release.
+- Per-controller concurrency limits are manager configuration, while controller definitions contain reconciliation behavior and timeout only.
+
+The persisted model was reviewed with Schemator before implementation. The review removed generic timestamps from public resources, kept provider details opaque, moved concurrency policy to the manager, and tightened event payloads to recursive JSON values. The run failed aggregate validation on two proposed structural moves, so those moves received a manual review instead of automatic application.
+
+## Requirements
+
+The first release must provide:
+
+- Typed controller definitions and durable resources with `spec`, `status`, generations and conditions, plus compare-and-swap versions.
+- A durable keyed queue with deduplication, delayed wakeups, bounded concurrency, retry backoff, and expiring claims.
+- Level-based reconciliation that receives a resource instead of an event payload.
+- Effect records that recover safely when a process stops around an external mutation.
+- Stable child workflow requests and completion wakeups.
+- Deterministic mutation authority outside agent workflows.
+- Public engine APIs, a Pi extension control surface, and local read-only status views.
+- Tests for retries, duplicate events, conflicts, process interruption, and ambiguous external effects.
+
+## Assumptions
+
+The default controller store will use SQLite and local filesystem permissions that match the private run-bundle directory. The public store and queue contracts will allow another host to provide remote storage later.
+
+The first release will support several workers in one process. Queue claims will survive process failure, but cross-host leader election will wait for a real remote deployment.
+
+Existing workflow definitions and run bundles remain valid. Controller resources use a separate schema and store. Child workflow attempts use the current run-bundle format; their parent links live in the controller store.
+
+## Open questions
+
+Implementation should resolve these points before the public API is frozen:
+
+- Confirm that the built-in Node SQLite API meets package portability and transaction requirements on every supported Node 22 release. Choose one documented dependency if it does not.
+- Decide whether controller discovery belongs in `.pi/controllers/` and `~/.pi/agent/controllers/`, or whether controllers should be registered only from installed packages.
+- Decide whether the existing `pi-workflows` executable should gain controller subcommands or whether the package should expose another binary.
+- Set queue claim and retry defaults from local fault tests. Use the same evidence for retention and payload limits.
+- Define the smallest viewer change that makes blocked and indeterminate resources easy to find.
+
+## Source layout
+
+Add a `src/controllers/` layer with no dependency on Pi:
+
+```text
+src/workflows/                 finite graph execution
+src/controllers/types.ts      resources, conditions, results, definitions
+src/controllers/store.ts      store and queue interfaces
+src/controllers/sqlite.ts     local durable implementation
+src/controllers/manager.ts    claims, workers, retries, cancellation
+src/controllers/effects.ts    effect lifecycle and recovery
+src/controllers/workflows.ts  child workflow scheduler adapter
+src/extension/                 Pi commands and lifecycle hooks, plus UI
+src/viewer/                    read-only local views
+```
+
+`src/workflows` must never import `src/controllers`. The controller layer may depend on exported workflow contracts through `controllers/workflows.ts`. The extension and viewer may use both layers and remain independent from each other. Update `slophammer.yml` before adding cross-layer imports.
+
+## Work stages
+
+### Resource contract
+
+Define the TypeScript API, persisted schemas, condition helpers, generation rules, and compare-and-swap behavior. Review the persisted model with Schemator before implementation. Document the final decisions in [CONTROLLERS.md](../CONTROLLERS.md).
+
+Add tests for spec updates, status-only writes, stale versions, condition transition times, deletion requests, and finalizer removal. A spec update must increase `generation`; a status update must preserve it.
+
+### Store and queue
+
+Implement the SQLite store in WAL mode with restrictive file permissions. Keep schema migration code explicit and fail closed on unknown schema versions.
+
+Implement one queue row per controller and resource key. Enqueue must deduplicate, an expired claim must become available again, and a delayed requeue must survive process restart. Add bounded exponential backoff with jitter and a test clock.
+
+Run concurrent worker tests that force compare-and-swap conflicts and claim expiry. All tests must use temporary directories.
+
+### Controller manager
+
+Implement `defineController`, `ControllerManager`, and the settled, immediate requeue, and delayed requeue results. Reconciliation requests carry a key; the manager reads the resource immediately before calling user code.
+
+Enforce one active reconciliation per key. Apply returned status against the version that was read. A conflict discards the stale status and queues another pass. Add global and per-controller worker limits.
+
+Emit structured lifecycle records with stable reconcile IDs and bounded error messages. Cancellation must stop new claims and give active reconcilers a fixed shutdown period.
+
+### Effect recovery
+
+Implement `ctx.effects.ensure()` around a request fingerprint and stable key. Save the pending record before calling the provider. Record each outcome as applied, rejected, or indeterminate without treating a local timeout as proof that the provider failed.
+
+Require every effect driver to define observation and application behavior. Reusing a key with different input must fail. Add crash tests around the claim and provider request, then after the response and receipt storage.
+
+Use a fake provider to prove that an indeterminate merge-like operation is observed before any retry. Test provider idempotency tokens and conditional request failures separately.
+
+### Child workflows
+
+Implement `ctx.workflows.ensure()` with a stable request key and input fingerprint. A repeated call must find the same active or completed request. A completion or interruption must enqueue the parent resource.
+
+Keep run attempts immutable. Add an explicit interrupted outcome for an abandoned child attempt. Route action nodes through the effect interface when they participate in a controller operation. Document the recovery rule for each node type.
+
+Test duplicate child requests, changed fingerprints, completion races, parent generation changes, and host restart between run creation and parent status update.
+
+### Extension and viewer
+
+Add controller discovery and a `/controller` command for listing resources, inspecting conditions, requesting reconciliation, and cancelling active local work. Use only documented Pi extension APIs.
+
+Start local sources from `session_start` and close them idempotently during `session_shutdown`. Pi exit must leave durable resources and queue rows ready for another host. No background service is installed.
+
+Add a resource list and detail view to the TypeScript viewer first. Extend the Rust viewer only after the text model and fixtures settle. Keep both views read-only.
+
+### Acceptance controller
+
+Build a local pull request controller against a fake GitHub-compatible server. Its spec names a repository, pull request, expected head, requested workflow, and approved mutations. Its status reports the observed head and child run together with check results and readiness conditions.
+
+The controller must re-read the pull request before each effect. A changed head blocks the mutation. Duplicate webhooks and scheduled polls must converge on the same resource and child workflow request. No credential belongs in the child request. Strict credential isolation requires a separate authenticated effect broker because the Pi host and agent tools share a process environment.
+
+Keep this controller as an example or integration package. GitHub-specific policy must stay outside the controller core.
+
+## Acceptance criteria
+
+The work is ready when all of the following hold:
+
+- Ten identical enqueue calls produce one pending key and no duplicate child run.
+- Two workers cannot reconcile the same key concurrently.
+- A stale resource version cannot overwrite newer status.
+- A spec change invalidates readiness from an older generation.
+- Delayed work and expired claims recover after a fresh process opens the store.
+- Each tested process stop around an external effect converges without an unobserved retry.
+- A changed pull request head prevents the acceptance controller from applying its effect.
+- The acceptance controller applies agent output only after deterministic authorization and precondition checks.
+- Pi reload and shutdown leave no in-memory state required for later recovery.
+- Current workflow and viewer tests keep passing, along with run-bundle tests.
+
+## Verification
+
+Run the focused controller tests during each stage, followed by the full repository checks:
+
+```bash
+npm run check
+npm run test:e2e
+npx slophammer-ts@latest dry .
+npx slophammer-ts@latest check . --only ts.dependency-boundaries-required
+npx -y @simpledoc/simpledoc check
+```
+
+Exercise the Pi extension from source with `pi -e src/extension/index.ts`. Start a controller resource, stop Pi while work is pending, reopen the same store, and confirm that reconciliation continues from current state. Repeat the effect crash matrix with the fake provider and preserve the test output as CI evidence.
+
+## Non-goals
+
+The first implementation will not provide Kubernetes API compatibility, a distributed cluster scheduler, automatic service installation, or generic exactly-once execution. It will not place GitHub credentials or mutation policy in agent prompts. It will not use run bundles, comments, or viewer projections as the controller database.
+
+## Documentation updates
+
+Update [CONTROLLERS.md](../CONTROLLERS.md) whenever the public resource, queue, reconciliation, or effect contracts change. Add the shipped authoring surface to [workflows.md](../workflows.md), the source boundaries to [development.md](../development.md), and user-facing installation and commands to the root README after implementation.
+
+Record any meaningful departure from this plan in this document before the implementation is considered complete.

package/docs/plans/2026-08-05-always-on-workflows-plan.md

@@ -0,0 +1,125 @@
+---
+title: Add always-on workflow execution
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-05
+updated: 2026-08-05
+status: implemented
+---
+
+# Always-on workflows plan
+
+Pi Workflows should feel the same whether the user watches a run or walks away from it. In the user's words: "I might start a workflow locally in Pi then I wait for it to complete. All the while I am looking at the screen and I'm not closing the Pi window. When the workflow ends I just want to be able to continue the same Pi session like normal with a session up to date with what happened in the workflow." And: "I just want to interact by starting a workflow, closing it, and then coming back and then still being able to continue it when I open it up. It's syncing continuously or something."
+
+These are not two modes. The user asked for "both in a single unified system." This plan makes the Pi window irrelevant to execution: closing or opening the window is a change in observation, not in the run. The work stays on one machine, uses the merged controller runtime as its foundation, and does not modify Pi core.
+
+This revision incorporates an external design review of the first draft. The review confirmed the three design rules and found five structural gaps in the details. The Decisions section resolves each one before implementation starts.
+
+## Design
+
+Four rules define the system:
+
+1. **Durable state is the only source of truth.** Run bundles, the queue, and the event log live on disk. Nothing important lives in a session's memory.
+2. **Runners are interchangeable, but ownership is exclusive.** Any live process running the engine claims work from the durable queue. A run has exactly one owning runner at a time, proven by a claim token, and only that runner may write to the run.
+3. **The session is always a view.** It attaches to a run's event stream and renders it. An open window sees a live tail; a reopened window catches up from the same stream. Interaction such as approvals uses durable waiting states, never prompts tied to the window's lifetime.
+4. **Claims arbitrate every lifecycle decision.** Only the current claim holder may resume, interrupt, or write a terminal event for a run. Recovery code that finds an abandoned run goes through the queue instead of writing to the bundle directly.
+
+The controller runtime already provides most of the machinery: a deduplicated queue with expiring claims, a structured event table, crash recovery through the trace tail, and guarded effect records. This plan extends that treatment to runs the user starts interactively and adds the view layer.
+
+## Decisions
+
+These points were open in the first draft. External review showed each one is load-bearing, so they are decided here.
+
+1. **Write fencing on bundles.** Every claim records a runner ID, a token, and a lease expiry in the store. Every bundle write verifies the token inside the run lock before appending and fails the runner fast when the token no longer matches. Expiring claims alone do not stop a stalled runner from writing; fencing does.
+2. **Explicit resume protocol.** Resume is a named operation, not a restart. It truncates a torn trace tail to the last complete line, rebuilds run state from the trace, seeds the trace sequence from the tail, accounts for already-executed steps against the step limit, and records a resume boundary event before continuing.
+3. **Resume scope.** User-started runs get node-level resume. Controller child runs keep their current new-attempt semantics, because attempt immutability and parent-side retry already work and are tested. Both behaviors are explicit; nothing mixes silently.
+4. **A dedicated run queue.** Interactive runs do not fit the controller queue, which is keyed to controller resources. A new `workflow_run_queue` table shares the claim, lease, and fencing pattern and adds runner affinity fields. Runs stay out of `/controller list`.
+5. **Origin affinity.** An interactively started run is inserted and claimed in one transaction, so the session that started it owns it from birth. The standalone host takes over only when that claim is released or expires. This guarantees a watched run's conversation happens in the watching session.
+6. **Close-to-park shutdown.** `session_shutdown` stops writing a terminal cancel event for queued runs. It aborts in-flight work without a terminal event and releases the claim, leaving a resumable bundle. A clean close releases claims explicitly; a crash relies on lease expiry.
+7. **Per-attempt capture.** Session capture becomes segmented per attempt, keyed by attempt ID, so a run handed off to the host or a new session starts a fresh capture segment instead of failing integrity checks.
+8. **Continuation runs for waiting.** A run that needs human input ends at a checkpoint with status `waiting`, preserving bundle immutability. An answered checkpoint starts a continuation run with a new run ID chained to its parent, carrying forward prior outputs. The parent link makes the chain inspectable.
+9. **Source pinning.** The manifest stores a content hash of the workflow source at run start. Resume refuses to continue against changed source unless forced, and a forced resume records the mismatch in the trace.
+10. **Snapshot catch-up.** Notifications are idempotent snapshots of run state as of a store sequence number, recomputed from the store, not a stream of one-off messages. The watermark persists per session before sending. A skipped incremental notification is subsumed by the next snapshot.
+11. **A run-level event feed.** Run lifecycle transitions write rows into a store table, so one watermark covers both controller and run events. Tailing `trace.ndjson` is reserved for the single actively watched run.
+12. **Store-error backoff.** Worker loops treat store errors such as `SQLITE_BUSY` as transient and back off instead of letting a worker die silently. The host's advisory lock guards host-versus-host only; the embedded runner does not take it, and a second host refuses to start.
+13. **Orphan reaping.** The host spawns `pi --mode rpc` children in their own process group, records child PIDs in the bundle, and reaps known orphans on startup. Consequential actions stay behind guarded effects regardless.
+
+## Requirements
+
+- Starting `/workflow run` in a Pi session creates a durable queued run claimed by that session, and shows its progress live.
+- Closing Pi mid-run never loses the run and never writes a spurious terminal event. With the standalone host alive, the host reclaims and resumes the run. Without a host, the run waits and resumes when a runner returns.
+- Reopening a session brings it up to date with an idempotent snapshot: what finished, what failed, what waits for input. No state is duplicated and no information is permanently lost.
+- A run that needs human input ends at a checkpoint. The user answers with a command, and a continuation run carries the work forward. The wait survives any process lifetime.
+- Killing any process at any point recovers without duplicate trace sequences, duplicate notifications of record, or duplicate external effects. A stalled runner that loses its claim can never write again.
+- Everything uses documented Pi public APIs. No Pi core changes.
+
+## Work items
+
+1. **Run queue and fencing.** Add the `workflow_run_queue` table with claim tokens, lease expiry, and runner affinity. Gate every bundle write on the token inside the run lock. Insert-and-claim interactive starts in one transaction.
+2. **Node-level resume.** Implement the explicit resume protocol from Decision 2, with source pinning from Decision 9. User-started runs resume at the interrupted node; controller child runs are untouched.
+3. **Close-to-park and capture segments.** Change `session_shutdown` to abort-without-terminal plus claim release for queued runs. Split session capture into per-attempt segments the integrity checker understands.
+4. **Continuation runs.** Chain an answered checkpoint to a new run ID with a parent link and carried-forward outputs. Render the chain as one logical run in views.
+5. **Session sync.** Add the run-level event feed, per-session watermarks, snapshot catch-up on `session_start`, and noteworthy-event messages. Live watching tails `trace.ndjson` from a remembered byte offset with `fs.watch`, reusing the TUI viewer's file-tail path.
+6. **Standalone host.** A `pi-workflows` CLI subcommand loads controller definitions, opens the project store, and runs claiming in a loop with store-error backoff. Conversation child nodes run in spawned headless `pi --mode rpc` sessions with orphan reaping. The host takes an advisory lock against other hosts, drains on SIGTERM, and recovers on restart. It is a foreground process the user runs in a terminal; it is not a service.
+
+## Non-goals
+
+- Pi core changes of any kind. Every integration uses public APIs: commands, session events, widgets, `sendUserMessage`, and `pi --mode rpc`.
+- Multi-machine execution, a remote store, or leader election. SQLite and one machine are in scope; the store contracts leave room for a remote implementation later.
+- Installing or configuring a system or user service. The host is a process the user starts and stops.
+- A push channel from an external process into a live session. Polling the shared store is the mechanism, and it is fast enough.
+- Exactly-once chat notifications. The guarantee is no duplicated state and no permanently lost information, delivered through idempotent snapshots.
+- Exactly-once external side effects beyond the existing guarded effect records.
+
+## Assumptions
+
+- One machine and one user, with the store and run bundles on the local filesystem.
+- Spawning `pi --mode rpc` per conversation child run is acceptable at the expected cadence. If startup cost proves too high, the host keeps a small pool of persistent RPC sessions instead. Both options stay outside Pi core.
+- A polling interval of a few seconds is responsive enough for the session view. File watching covers the live tail of a watched run.
+- The user does not need machine-sleep or power-loss coverage beyond crash recovery. A stopped machine stops work until a runner returns.
+
+## Open questions
+
+- Which events deserve a chat message and which belong only in the widget. The default should be quiet.
+- The exact host command shape, for example `pi-workflows run --project <dir>` versus a subcommand under `controllers`.
+
+## Departures from the decisions
+
+The implementation matches the decisions above with these refinements:
+
+- Graph validation now allows outgoing edges from checkpoint nodes. The old
+  rejection encoded terminal-forever checkpoints; continuations make those
+  edges live. This is a deliberate contract change for workflow authors.
+- The host command is `pi-workflows host`, chosen over `run` because the
+  viewer CLI's vocabulary already uses runs for bundles.
+- The first capture stays flat at `session/`; only binds from the second
+  recorder onward write segments under `session/segments/`. This keeps the
+  layout of every existing single-session bundle and the viewer unchanged.
+- Resume always starts unpaused; the operator can pause again.
+- Headless conversation children keep the exact tool contract: a bridge
+  extension loaded with `-e` registers the `workflow` tool and reports
+  submissions to the host over stderr, so no engine prompt changes were
+  needed.
+- The host accepts explicit `storeFile` and `runsDir` options. Its defaults
+  resolve from the project and environment like every other entry point.
+
+## Acceptance criteria
+
+- Start a run in Pi, then close Pi while a conversation node is mid-response and the host is running. The host reclaims and resumes the run, and reopening Pi shows the catch-up snapshot and allows normal conversation about the result.
+- The same flow without the host: the run resumes at the interrupted node when Pi reopens and completes.
+- A stalled runner that loses its claim cannot write to the bundle afterward; fencing rejects its writes.
+- An open session reflects host-driven progress within a few seconds. Two sessions on one project each get complete catch-up; neither starves the other.
+- A waiting-for-input run survives closing and reopening. Answering it starts a continuation run that carries forward prior outputs.
+- `kill -9` on the host mid-append, followed by a restart, repairs the torn trace tail and recovers without duplicate trace sequence numbers and without repeating an applied external effect. Orphaned RPC children are reaped.
+- No changes to Pi core; the diff touches only this package.
+
+## Verification
+
+- `npm run check` and `npm run test:e2e`, including new real-Pi E2E tests that start a run, kill the host, restart it, and assert continuation.
+- A fencing test with both runners alive and a forced lease loss.
+- A two-session watermark test on one project.
+- A `kill -9` mid-append torn-tail resume test.
+- A checkpoint answer-to-continuation round-trip test.
+- Extension tests with fake timers for the polling loop, watermark, and snapshot catch-up.
+- `npx slophammer-ts@latest dry .` and the dependency-boundary check.
+- `npx -y @simpledoc/simpledoc check` for documentation changes.
+- Manual pass through both usage patterns from the user's request with the host in a terminal.

package/docs/plans/2026-08-10-agent-managed-monitor-workflows-plan.md

@@ -0,0 +1,184 @@
+---
+title: Let agents run and manage monitor workflows
+author: Onur Solmaz <2453968+osolmaz@users.noreply.github.com>
+date: 2026-08-10
+updated: 2026-08-10
+status: implemented
+---
+
+# Agent-managed monitor workflows plan
+
+The user should be able to tell an agent, "Monitor this every 30 minutes," and have the agent start the right workflow. The user must not write controller records or JSON. The existing `workflow` model tool should manage workflows instead of serving only as a step-submission tool.
+
+A monitor is one Pi Workflows graph. It checks the target, reports a meaningful change, sleeps for the requested interval with the existing shell node, and loops. This plan does not use controllers, Unified Exec, a new wait node, or a second scheduler.
+
+## Shipped design
+
+The implementation follows this plan. The built-in monitor defaults to 1,000 checks, caps the interval at 24 hours, and uses a 5,010-step engine limit as a second guard. The normal Pi extension exposes all workflow tool actions. The headless RPC bridge exposes only `submit`. A real-Pi end-to-end test proves that a normal model turn can start the built-in monitor and complete its first check. The automated test inspects the exact 30-minute Node timer command and both timeout margins instead of making the test suite wait for 30 minutes. The Node timer keeps the shell action portable across Pi's supported platforms.
+
+## User experience
+
+A normal request looks like this:
+
+> Monitor PR 123 every 30 minutes. Tell me when checks fail. Stop when it is merged or closed.
+
+The agent recognizes repeated monitoring from the `workflow` tool description and starts the built-in `monitor` workflow. The tool call contains the structured input, but the user does not write it.
+
+The monitor checks immediately, then waits between later checks. It writes a normal assistant message only when the requested report condition is met. It stops when the requested stop condition is met, when it reaches its safety limit, or when the user or agent cancels it.
+
+Existing controls remain available:
+
+```text
+/workflow pause
+/workflow resume
+/workflow cancel
+```
+
+The same operations become available to the model through the `workflow` tool. One active workflow per Pi session remains the rule. A long monitor therefore occupies that session's workflow slot.
+
+## Workflow tool
+
+Replace the current submit-only schema with one discriminated tool schema. Use Pi's documented `registerTool` API and `StringEnum` helper.
+
+The tool supports these actions:
+
+- `list`: list discovered workflows and their source.
+- `start`: start a workflow by name or path with structured input.
+- `status`: return a bounded summary of the active or named run.
+- `pause`: pause the active run at its next node boundary.
+- `resume`: resume the active paused run.
+- `cancel`: cancel the active run.
+- `answer`: answer a waiting checkpoint, with an optional run ID.
+- `submit`: submit the result required by the current workflow step contract.
+
+The `submit` form becomes:
+
+```json
+{
+  "action": "submit",
+  "step": "check",
+  "attempt": "attempt-id",
+  "output": {}
+}
+```
+
+Update workflow step prompts and reminders to include `action: "submit"`. Do not retain the old submit shape as a compatibility alias.
+
+Slash commands and tool actions must call the same internal lifecycle functions. This avoids separate behavior for humans and models. Tool results must be structured and bounded. Errors must state whether the operation failed because no run exists, a run is already active, a checkpoint is not waiting, or the requested workflow cannot be found.
+
+A tool call happens during an agent turn. The `start` action must validate and queue the launch, then deliver the first workflow prompt after the initiating turn settles. This prevents the new run from treating the initiating turn as a failed workflow step. It also prevents a workflow prompt from being injected in the middle of the tool call.
+
+The headless RPC bridge keeps a submit-only version of the tool. Headless workflow children must not start or control other workflows. Its submit schema and parser still change to require `action: "submit"`.
+
+## Agent guidance
+
+The tool description must tell the model when to use the built-in workflow:
+
+- Use `start` with workflow `monitor` when the user asks to watch, monitor, poll, or check something repeatedly.
+- Start directly when the task and interval are clear.
+- Ask one short question when the target or interval is missing.
+- Use observation-only behavior unless the user explicitly authorizes a mutation.
+- Do not create repeated work without a user request or an existing workflow instruction that authorizes it.
+
+The `list` result identifies `monitor` as a built-in workflow and gives a short description. This keeps discovery inside the tool instead of adding a skill or a system-prompt injection.
+
+## Built-in monitor workflow
+
+Ship `monitor` as a built-in workflow in the Pi Workflows package. Built-ins have the lowest discovery precedence:
+
+1. Project workflows under `.pi/workflows/`
+2. Global workflows under `~/.pi/agent/workflows/`
+3. Workflows bundled with Pi Workflows
+
+A project or global `monitor.workflow.ts` can therefore replace the default. The built-in remains a real workflow file so run bundles can record its path and source hash with the existing rules.
+
+The workflow input contains:
+
+- `task`: what to inspect.
+- `everyMinutes`: the interval between checks.
+- `reportWhen`: changes or states that deserve a user message.
+- `stopWhen`: the condition that ends monitoring.
+- `maxChecks`: a hard safety limit.
+
+The model supplies these fields from the user's request. The workflow validates the input and applies documented bounds. The first release supports intervals from 1 minute through 24 hours. It defaults `maxChecks` to a documented finite value when the request gives no end limit.
+
+The graph is:
+
+```text
+prepare -> guard -> check
+                    | continue quietly -> sleep -> guard
+                    | continue and report -> report -> sleep -> guard
+                    | stop quietly -> finish
+                    | stop and report -> report-final -> finish
+```
+
+`prepare` validates and normalizes the input. `guard` enforces `maxChecks`. `check` is an agent node that performs one observation and returns a validated route, a bounded observation, and an optional report. The next `check` can read the previous accepted `check` output, which Pi Workflows already keeps for looped nodes.
+
+The report nodes write a normal assistant message and then submit an acknowledgement. Keeping reporting after accepted check output prevents the agent from showing a report before the structured result passes validation. The final presentation reports why the monitor stopped without repeating a report that the user already saw.
+
+The sleep node uses the existing Pi Workflows shell action to launch the current Node executable with a timer. Set the node timeout above the largest supported interval because the engine default is 15 minutes. Set the shell execution timeout above the requested wait by a small fixed margin. Cancellation aborts the timer process immediately.
+
+If the Pi TUI or standalone workflow host stops during sleep, Pi Workflows parks the run and kills the shell child. Resuming the run starts that sleep node again from the beginning. This is existing workflow behavior and is acceptable for this feature. No special timer persistence is added.
+
+The workflow uses a high but finite `maxSteps` value as a second safety guard. Check and report values have explicit size limits so a long run cannot grow its bundle without bound.
+
+## Repository changes
+
+Make the feature in `osolmaz/pi-workflows`:
+
+- Refactor workflow lifecycle operations out of the slash-command handler.
+- Expand the normal Pi `workflow` tool and update its tests.
+- Update the RPC bridge submit contract.
+- Add built-in workflow discovery with project and global override precedence.
+- Add the built-in monitor workflow and focused tests.
+- Update `README.md` and `docs/workflows.md`.
+
+After the upstream change is complete, update the pinned Pi Workflows commit in OnurPi's thin `packages/workflows` wrapper. Do not add a new OnurPi extension or copy a monitor file into live global state.
+
+## State and API impact
+
+- **Session state:** Normal workflow prompts, model replies, tool calls, and monitor reports are appended through Pi's normal session behavior.
+- **Other persistent data:** No new data model. The feature uses existing run bundles and the existing workflow run queue.
+- **Pi internals:** None.
+- **Pi public API:** `registerTool`, `registerCommand`, `sendUserMessage`, and documented agent and session lifecycle events.
+- **Pi Workflows API:** The workflow definition and run-state models do not change. The model-facing `workflow` tool contract changes, and discovery gains a lowest-priority built-in source.
+
+## Non-goals
+
+This work does not add cron expressions, calendar schedules, a background service, controller resources, OS notifications, concurrent workflows in one session, resumable shell processes, or guaranteed wall-clock wake times across runner shutdowns. It does not let headless child agents recursively start workflows.
+
+## Acceptance criteria
+
+- A user can ask for repeated monitoring in plain language, and the agent starts `monitor` through the existing `workflow` tool.
+- The user does not type JSON or a slash command to start the monitor.
+- The monitor checks immediately, sleeps for the requested interval, and checks again.
+- An unchanged observation produces no normal assistant report.
+- A matching report condition produces one concise assistant report.
+- A matching stop condition reports as requested and ends the run.
+- The safety limit ends a monitor that never reaches its stop condition.
+- The model can list, start, inspect, pause, resume, cancel, answer, and submit through one tool.
+- Slash commands and tool actions use the same lifecycle code.
+- Starting from a model tool call does not cause an early nudge or record the initiating turn as the first workflow attempt.
+- A 30-minute sleep is not stopped by the default 15-minute node timeout.
+- Cancelling during sleep stops the shell child and ends the workflow.
+- Project and global workflows override the built-in `monitor` name.
+- Existing workflow, controller, host, viewer, and run-bundle tests continue to pass.
+- OnurPi loads the updated wrapper and Pi starts successfully.
+
+## Verification
+
+Add unit and integration coverage for tool action validation, lifecycle dispatch, deferred model-start launches, built-in discovery precedence, monitor routing, quiet checks, reports, stop conditions, safety limits, cancellation during sleep, and the headless submit bridge.
+
+Run:
+
+```bash
+npm run check
+npm run test:e2e
+npx slophammer-ts@latest dry .
+npx slophammer-ts@latest check . --only ts.dependency-boundaries-required
+npx -y @simpledoc/simpledoc check
+```
+
+Test the extension from the Pi Workflows checkout with `pi -e src/extension/index.ts`. Use a short test interval in a controlled fixture, then perform one manual 30-minute monitor run to confirm that the configured node timeout does not stop it. Test plain-language startup with the normal model, then test list, status, pause, resume, cancel, and checkpoint answer actions.
+
+After updating OnurPi, run its full checks and start Pi with the installed OnurPi package. Confirm that the model sees one `workflow` tool, discovers `monitor`, and can start it from a plain-language request.