@osolmaz/pi-workflows 0.14.0 → 0.15.0

package/docs/CONTROLLERS.md

@@ -19,7 +19,7 @@ The controller runtime provides:
 - Recoverable records for external effects.
 - Child workflow runs with stable request keys.
 - Conditions and generations, with cleanup and structured events.
-- Local interactive use through the Pi extension and headless use through the engine API.
+- Local resource control through the Pi extension and reconciliation through the global host.
 
 The first production use case is pull request automation. A controller can observe a pull request, start a review or repair workflow, wait for checks, validate the current head, and apply an approved change through deterministic code.
 
@@ -27,7 +27,7 @@ The first production use case is pull request automation. A controller can obser
 
 The graph engine remains the execution layer for finite work. A finite workflow can include another finite workflow in the same run through `includeWorkflow()`. Use a controller child run when work needs an independent retry history, stable request key, parallel lifecycle, or indefinite reconciliation. The graph engine does not import the controller runtime. The controller runtime may start workflows through a narrow scheduler interface.
 
-The Pi extension is a host. It discovers definitions, displays status, and supplies the conversation-backed agent executor while Pi is running. A headless host can use the same controller runtime with another `AgentStepExecutor`.
+The Pi extension is a thin client. It resolves controller initialization in a dedicated child process and sends declarative resource commands to the one global host. The host schedules reconciliation but does not load controller definitions in its event loop. A supervised controller worker loads one definition and proposes bounded state changes through the host.
 
 External events are wake-up hints. An event enqueues a resource key and carries no transition command. The reconciler reloads the resource and the external system before deciding what to do.
 
@@ -155,15 +155,15 @@ export type EffectRecord = {
 
 The key names one intended effect. Reusing the key with another request fingerprint is an error. The next reconciliation observes the external system before retrying an existing pending or indeterminate effect. The effect can be treated as effectively once when the provider offers an idempotency token, a conditional request, or a reliable way to observe the requested result. The runtime does not promise generic exactly-once execution.
 
-Mutation policy stays in deterministic effect drivers. Agent workflows return findings or artifacts for deterministic code to check and apply. The in-process Pi host shares its process environment with agent tools, so it does not provide credential isolation. Deployments that require that boundary should put authenticated effects behind a separate broker or controller host.
+Mutation policy stays in deterministic effect drivers. Agent workflows return findings or artifacts for deterministic code to check and apply. Worker processes run as the same operating-system user and are not credential sandboxes. Deployments that require credential isolation should put authenticated effects behind a separate broker.
 
 ## Child workflows
 
 `ctx.workflows.ensure()` creates or finds a workflow run by a stable request key and input fingerprint. Repeated reconciliations find the same active or completed request. A changed input must use a new key. An asynchronous child completion validates the reserved request and run IDs through a separate scheduler-completion command; it never reuses the controller claim that launched the child. The controller transaction reserves and saves each attempt's run ID before the scheduler starts it, so recovery can find the run row.
 
-A child run is one immutable attempt. Its existing SQLite run state remains the execution record. The parent resource points to the current run, and workflow completion enqueues the parent key. A host restart can record an abandoned attempt and create another attempt for the same stable request.
+A child run has one durable execution record. The parent resource points to the current run, and workflow completion enqueues the parent key. The global host runs it through the same queue and supervised worker protocol as other workflows.
 
-The workflow scheduler records an abandoned running run as `failed` with a final `run_interrupted` event. The controller store treats that child attempt as `interrupted`, so the next reconciliation starts another immutable attempt. Compute work can run again, while consequential external actions belong in the effect API so recovery observes them before retrying.
+A stopped worker does not make the child failed by itself. The host reads the last committed node and effect state. It resumes pure or idempotent work in a new worker epoch. An uncertain manual effect becomes ambiguous and blocks automatic retry. A changed input still requires a new stable request key.
 
 ## Deletion and cleanup
 
@@ -175,9 +175,9 @@ Controllers should add finalizers only when they own something that needs cleanu
 
 A source maps an external event to one or more resource keys. Sources include filesystem watches, webhooks, scheduled polling, and child workflow completion. They share the same enqueue API.
 
-`ControllerManager` sets global and per-controller worker limits. A reconciliation deadline stops lease renewal and requeues the key even when controller code ignores its abort signal. JavaScript cannot stop that non-cooperative promise, so reconciler code must pass the signal to provider calls and keep consequential writes inside guarded effect drivers. The local store supports expiring claims from the start, while the first release can run one process. Leader election belongs in a remote store implementation if several hosts later share the same resources.
+The global host claims controller keys and starts a supervised process group for each active reconciliation. A reconciliation deadline stops and requeues the child even when controller code ignores its abort signal. Reconciler code must still pass the signal to provider calls and keep consequential writes inside guarded effect drivers. One host owns the local database. Distributed leader election remains outside this local runtime.
 
-The Pi extension starts local workers during `session_start` and closes them during `session_shutdown`. It can reconcile only while Pi is running. Another program can host `ControllerManager` through the public engine API. The package does not install a service.
+The extension starts the package host on demand. Reconciliation continues when the Pi session closes. The package installs no operating-system service.
 
 ## Observability
 
@@ -202,13 +202,13 @@ A production controller must follow these rules:
 
 The controller API is exported from `@osolmaz/pi-workflows/controllers`. Controller definitions use a `.controller.ts` suffix. Project definitions live under `.pi/controllers/`; global definitions live under `~/.pi/agent/controllers/`.
 
-The implementation uses documented Pi extension APIs only. Commands and tools use `registerCommand` and `registerTool`. Session lifecycle uses `session_start` and `session_shutdown`. Workflow prompts use `sendUserMessage`, while status uses `setWidget` and `setStatus`.
+The implementation uses documented Pi extension APIs only. `/controller` lists and inspects resources, applies specs, and requests reconciliation or deletion. There are no extension-local worker start or stop controls.
 
-The `/controller` command lists and inspects resources, applies specs, requests reconciliation or deletion, and starts or stops local workers. Stopping workers records an active child as interrupted, so a later worker can create another attempt.
+For `apply`, a source resolver child discovers the named controller, verifies its exported name, computes `initialStatus(spec)`, and hashes the source. The host accepts the proposal only while the path still follows discovery rules and the exact digest still matches. Reconcile code runs only in a supervised controller worker.
 
-Controller resources use the canonical [SQLite state](SQLITE_STATE.md) database. `projects` separates repository-local resources by canonical project path. Controller claims use the shared lease generation, token, expiry, and expected resource revision. Effects and child workflows use the shared transactional outbox and deterministic receipts.
+Controller resources use the canonical [SQLite state](SQLITE_STATE.md) database. `projects` separates repository-local resources by canonical project path. Controller claims use the shared lease generation, token, expiry, and expected resource revision. Effects and child workflows use durable request keys and receipts.
 
-The same database backs the standalone host (`pi-workflows host`). The host reconciles controllers without a Pi session and claims parked interactive runs from `run_queue`. Conversation children execute in headless `pi --mode rpc` sessions. A Pi session and the host can share the database safely because SQLite transactions and durable leases arbitrate ownership, but only one owner can mutate a resource at a time.
+One global host owns that database for all Pi sessions and projects. The extension and mutating CLI paths are local protocol clients; viewers remain read-only. A controller child without an origin session uses a headless `pi --mode rpc` process for structured agent steps.
 
 Normal workflow prompts, tool calls, and replies remain part of the Pi session. No Pi internal type, private API, or persistent Pi schema changes.
 

package/docs/DEFERRED_TURNS.md

@@ -219,10 +219,21 @@ The fallback custom message uses this details object:
   "schema": "pi-workflows.deferred-turn-message.v1",
   "turnIntentId": "deferred-turn:...",
   "runId": "20260821T081731Z-autoimplement-407480dd",
-  "cause": "failed"
+  "cause": "failed",
+  "presentation": {
+    "workflowName": "autoimplement",
+    "state": "failed",
+    "reasonKind": "maxSteps",
+    "restart": {
+      "count": 0,
+      "limit": 3
+    }
+  }
 }
 ```
 
+`presentation` contains small, bounded display fields. `reasonKind` and `restart` are present only when the run records provide them. These fields do not replace or shorten the model prompt.
+
 The message type is `pi-workflows-deferred-turn`. Delivery uses:
 
 ```ts
@@ -232,7 +243,17 @@ The message type is `pi-workflows-deferred-turn`. Delivery uses:
 }
 ```
 
-For a terminal run, the visible content contains the workflow identity and revision, terminal run ID, exact stored input, bounded result, terminal state and reason, restart count, and earlier terminal outcomes in the chain. It tells the model to use the current conversation, prefer a safe restart for an unfinished task after a technical or temporary failure, and stop for completed work, cancellation, missing authority, a required user decision, or a repeated failure. Values from input and result are data, not instructions.
+### Compact TUI card
+
+Interactive Pi registers a custom renderer for `pi-workflows-deferred-turn`. The collapsed card shows the workflow name, state or cause, run identity, and restart count when it is available. It does not show the terminal facts JSON, exact input, result, fingerprint, or model instructions.
+
+The renderer reads only the structured message details for its compact fields. It sanitizes workflow-derived text and uses Pi's standard TUI components, theme colors, and `expanded` state. A message with missing or invalid details renders as a safe generic workflow card.
+
+Expanding the card shows the complete existing message content. The model and session history receive that same content whether the card is collapsed or expanded. Restored messages use the same renderer and do not create another entry or model turn.
+
+This display behavior uses the documented `pi.sendMessage()` and `pi.registerMessageRenderer()` APIs. It does not change headless or RPC delivery. Presentation messages that already use `display: false` stay hidden. It adds no Pi core change, private API, database table, migration, store, or external resource.
+
+For a terminal run, the complete content contains the workflow identity and revision, terminal run ID, exact stored input, bounded result, terminal state and reason, restart count, and earlier terminal outcomes in the chain. It tells the model to use the current conversation, prefer a safe restart for an unfinished task after a technical or temporary failure, and stop for completed work, cancellation, missing authority, a required user decision, or a repeated failure. Values from input and result are data, not instructions.
 
 The content comes only from existing run and queue records. Pi owns conversation history. Pi Workflows does not identify, hash, copy, or store an original user message.
 

package/docs/SQLITE_STATE.md

@@ -8,11 +8,25 @@ Pi Workflows stores all live durable state in one database:
 
 There is one database for the user installation. Project and run IDs separate data inside it. Workflow targets do not read or write this database.
 
+## Viewer projection
+
+The database includes the [incremental and virtualized viewer design](plans/2026-08-28-piw-incremental-viewer-plan.md).
+
+`viewer_runs` stores one presentation revision and retained revision floor for each run. `viewer_deltas` stores ordered target patches by run, presentation revision, and delta index. `viewer_session_checkpoints` stores the bounded active message and tool state at each 256-event boundary. A viewer-visible transaction writes the domain change, advances the presentation revision, and writes its patch blobs before the same commit. Session-event transactions write each reached replay checkpoint in that transaction.
+
+The store retains 256 presentation revisions. A reader with an older cursor must take a bounded snapshot. Patches use `add`, `replace`, `remove`, and `append`. They target small projection documents or pages. Patch creation does not reconstruct and compare complete run views.
+
+`session_entries` and `session_events` have run-wide sequence numbers and indexed `(run_id, run_seq)` ranges. Step, trace, entry, and event reads contain at most 256 rows. Run-list queries read metadata, status, lease facts, and the presentation revision. They do not read payload bodies.
+
+This is an in-place alpha schema change. The schema name and version remain `pi-workflows-state` version 1. The DDL digest and exact shape changed. An older alpha database fails with the standard reset instruction and remains untouched. There is no compatibility reader, migration shim, dual path, feature flag, alias, or `v2` schema.
+
 ## Storage boundary
 
 The database stores:
 
 - workflow definitions, runs, events, node attempts, outputs, and updates
+- global host epochs, command receipts, worker epochs, and worker messages
+- durable origin-session interaction requests and submissions
 - captured Pi session entries and events
 - run and controller queues, claims, retries, and continuations
 - human-decision requests, submissions, resolutions, and cancellations
@@ -98,7 +112,8 @@ The shared records do not replace domain schemas. The following `STRICT` tables
 | ------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
 | Schema and projects | `schema_meta`, `projects`                                                                                                    |
 | Content             | `blobs`                                                                                                                      |
-| Shared lifecycle    | `resources`, `leases`, `events`                                                                                              |
+| Shared lifecycle    | `resources`, `leases`, `events`, `workflow_host_state`                                                                       |
+| Host protocol       | `host_commands`, `run_workers`, `worker_messages`, `interactive_requests`, `interactive_submissions`                         |
 | Workflows           | `workflow_definitions`, `runs`, `run_sources`, `run_steps`, `run_bindings`, `run_queue`, `node_attempts`, `workflow_updates` |
 | Live settings       | `workflow_settings`, `workflow_setting_changes`                                                                              |
 | Post-run follow-ups | `workflow_follow_up_queues`, `workflow_follow_ups`                                                                           |
@@ -111,29 +126,23 @@ The shared records do not replace domain schemas. The following `STRICT` tables
 
 Foreign keys join projects, runs, attempts, decisions, controllers, effects, and channel records. Partial unique indexes enforce one active node attempt per run, one queued or running reservation per Pi session, one decision winner, and one deterministic effect key. A parked waiting parent does not block its continuation. Reserving that continuation settles the parked parent queue in the same transaction, so a failed reservation leaves the parent recoverable.
 
-### Terminal restart state
-
-The [terminal workflow restart contract](plans/2026-08-27-workflow-terminal-restart-plan.md)
-uses the existing tables. It adds no state store or table. A restarted run keeps
-its root run ID, parent run ID, restart number, and parent terminal fingerprint
-in the existing launch-options value. The fingerprint uses the workflow
-identity and revision, exact input, terminal state, canonical result or error,
-and canonical reason. It excludes run IDs, timestamps, and other values that
-change between equivalent attempts.
-
-One terminal turn intent covers result presentation and factual fallback. A
-selected restart, Monitor run, or other workflow start uses the existing
-session reservation, run queue, and effect receipt. The successor run's launch
-options record the source terminal run, intent, model tool call, and canonical
-request fingerprint. A repeated call adopts that reservation or run. Activation
-waits for `agent_settled`, and normal queue recovery activates a surviving
-reservation once.
-
-Restart creates a new run. The terminal run remains unchanged. Restart lineage
-allows three restarts after the original run and rejects a repeated terminal
-fingerprint in the same chain. A Monitor selection records terminal selection
-but no restart lineage. Conversation history remains in Pi. This state does not
-identify, hash, copy, or store an original user message.
+### Hosted commands and interactions
+
+`workflow_host_state` stores the one current host epoch and its live local
+claim. `host_commands` stores each client request fingerprint, operation,
+outcome, revision, and receipt or error. Repeating an exact request adopts the
+stored receipt. Reusing an ID or idempotency key for another request is a
+conflict.
+
+`run_workers` records each worker epoch before spawn and later records its exact
+process identity and terminal outcome. `worker_messages` deduplicates accepted
+state-changing child messages.
+
+`interactive_requests` owns durable origin-session work. It stores the run,
+node attempt, target session, contract, presentation claim, accepted
+submission, and revision. `interactive_submissions` stores the idempotency key,
+payload, outcome, and receipt. Pi reload can adopt the saved presentation entry
+without inserting another visible message.
 
 ## Content-addressed values
 
@@ -149,9 +158,9 @@ Readers derive `steps`, `outputs`, `results`, carried-step count, current-node f
 
 An agent definition records `expectedOutput` as either a submitted-output description or `{ "kind": "assistant-message", "maxChars"?: number }`. Omitted `maxChars` means that Pi Workflows adds no character limit.
 
-A completed interactive assistant-message attempt uses the settled Pi response entry as its output. It does not store a second output blob. Its small receipt keeps the text digest, final Pi session entry ID, optional author-supplied limit, and whether recovery adopted an existing response. A noninteractive attempt with no captured response entry keeps one normal output blob.
+A completed interactive assistant-message attempt stores the accepted visible text as its node output. Its small receipt keeps the text digest, final Pi session entry ID, optional author-supplied limit, and whether recovery adopted an existing response.
 
-An interrupted assistant-message attempt keeps its attempt ID when the origin Pi session resumes it. The executor adopts a matching completed assistant child from the active Pi branch instead of displaying the response twice. Submitted and non-agent attempts keep their normal fresh-attempt resume behavior.
+An interrupted assistant-message attempt keeps its attempt ID. The extension adopts a matching durable request and existing Pi branch entry instead of displaying the prompt or accepting the response twice. Submitted and non-agent attempts use a fresh execution attempt after an uncommitted worker exit.
 
 ## Write contract
 
@@ -163,6 +172,7 @@ verify the exact schema
 verify the actor and operation
 verify the expected resource revision
 verify the claim token, generation, and expiry when ownership is required
+renew that exact still-live token and generation
 verify the domain transition
 write content-addressed values
 increment the resource revision
@@ -189,7 +199,7 @@ Reading or finding a row never gives write authority.
 - Control commands have narrow explicit operations, such as requesting cancellation or deletion.
 - Model-originated workflow answers cannot resolve protected human decisions.
 
-Stores check ownership in the same transaction as the write. Shared scans, status commands, lists, viewers, and the Rust `piw` program are read-only.
+The global host is the normal state writer. Its protected stores check ownership and renew the exact live claim in the same transaction as the write. A stale or expired owner cannot renew itself. Pi extensions and mutating CLI commands use the local host protocol. Shared scans, status commands, lists, viewers, and the Rust `piw` program are read-only.
 
 ## Competing outcomes
 
@@ -209,9 +219,9 @@ Status is a pure projection of domain rows, immutable facts, current leases, and
 
 A settings scope uses its resource revision as its public change number. Each accepted patch, current value, and node binding is saved in one transaction. A checkpoint continuation keeps the same settings resources and transfers them to the continuation run.
 
-A follow-up queue records acceptance order and final-presentation state. Successful terminalization changes queued items to ready or pending presentation in the same transaction as the terminal run fact. Failure, timeout, and cancellation cancel unsent items. Delivery uses item leases and active Pi branch evidence.
+A follow-up queue records acceptance order and settlement state. Failure, timeout, and cancellation cancel unsent items.
 
-- A terminal run fact overrides stale queue presentation.
+- A terminal run fact overrides stale delivery state.
 - An accepted decision is accepted even if its continuation effect is still pending.
 - A cancelled decision is cancelled even if parent cleanup is still pending.
 - A stale owner is not shown as current.
@@ -221,7 +231,7 @@ Read paths do not repair state. Owner reconcilers apply pending effects and writ
 
 ## Projects and concurrency
 
-All projects use the same file. `projects` stores a stable ID and canonical path. Project-scoped controller and run queries use that key. Standalone host lock and child-process registry files use a project hash under the workflow state directory, so different projects can run hosts concurrently while two hosts for one project still conflict.
+All projects use the same file. `projects` stores a stable ID and canonical path. Project-scoped controller and run queries use that key. One global host owns the file for the user installation. Its socket, lock, and exact child-process registry are under `~/.pi/agent/workflows/host/`. A second live host is rejected even when it was started from another project.
 
 SQLite WAL permits concurrent readers while one writer commits. Writers are serialized by SQLite and must keep transactions short. Hashing, model calls, shell work, and external requests happen outside write transactions.
 
@@ -259,4 +269,4 @@ It does not print actor IDs, channel references, payloads, or credentials.
 
 This is a hard cut. Pi Workflows has no normal reader or writer for older live storage. It does not use dual reads, dual writes, aliases, versioned state roots, or automatic import.
 
-Older state remains untouched. If it is present when a new database would be created, Pi Workflows fails with a clear instruction instead of guessing or deleting data.
+Older state remains untouched. Pi Workflows fails before mutation with this instruction: “Pi Workflows durable state is incompatible. Back up and move state.sqlite with its -wal and -shm files, then start Pi Workflows to create a new state.sqlite database. The incompatible state was not changed.”