@llblab/pi-kit 0.5.2 → 0.7.0

package/node_modules/@llblab/pi-state-flow/README.md

@@ -4,194 +4,290 @@
 
 ![pi-state-flow banner](https://raw.githubusercontent.com/llblab/pi-state-flow/main/banner.jpg)
 
-An opt-in Pi extension for explicit, validated state handoffs between complete agent runs.
-
-State Flow preserves Pi's native tool loop inside each user request:
+State Flow exposes a **temporally indexed materialized state** for Pi:
 
 ```text
-user request
-  → model → tool → model → tool → model
-  → one terminal answer (memory patch when needed)
-  → committed state + displayed response
+state[0] = now
+state[1] = one accepted transition ago
+...
+state[7] = seven accepted transitions ago
+
+state.global[n]   state.cwd[n]   state.session[n]
+          projections at the SAME temporal boundary
 ```
 
-The current run keeps its user message, assistant tool calls, tool results, and persistent or current-run context-bearing custom messages from other Pi extensions model-visible. Only after the terminal handoff does the next run replace that trajectory with one materialized state. Pi's complete session trace remains inspectable.
+`patch_state` explicitly advances that stream. Optional Git preserves complete older history for Git-backed transitions; without Git, files retain current state and its proven hot window. State Flow compiles decision-relevant reality; it neither owns Knowledge semantics nor replaces Pi's native tool loop or inspectable session trace.
 
 ## Installation
 
-Requirements: Pi `0.84.4–0.84.x` and Node.js `22.19.0` or newer.
-
-Install from npm:
+Requirements: Pi `0.84.4–0.84.x` and Node.js `22.19.0` or newer. Git is optional; when installed, it needs a configured commit identity. A remote is optional and operator-owned.
 
 ```bash
 pi install npm:@llblab/pi-state-flow
+# Or:
+pi install git:github.com/llblab/pi-state-flow
 ```
 
-Or install from GitHub:
+Pi packages execute with full user permissions; review the source before installing. An installed published version may precede this working-tree architecture.
 
-```bash
-pi install git:github.com/llblab/pi-state-flow
+## Configuration
+
+Optional `~/.pi/agent/state-flow.json` (or `state-flow.json` beneath `PI_CODING_AGENT_DIR`):
+
+```json
+{
+  "directory": "~/.pi/agent/state-flow",
+  "autoStart": false,
+  "remotePublication": "turn-end"
+}
 ```
 
-Pi packages execute with full user permissions; review the source before installing, as with any extension.
+- `directory`: State storage location; omitted defaults to `state-flow/` beneath Pi's agent directory. Absolute paths, `~`/`~/` and relative paths are supported; relative paths resolve from the configuration directory, not the project CWD.
+- `autoStart`: Set to `true` to enable State Flow automatically for genuinely new sessions, including fresh CWDs. Omitted defaults to `false`: manual activation, even when previous CWD materialization exists.
+- `remotePublication`: For new runtimes, `turn-end` (default) queues one newest target after local acceptance and pushes asynchronously; `off` keeps commits local; `transition` preserves legacy synchronous push behavior. A resumed branch keeps its persisted policy.
+
+Configuration is read at extension load; edit it and use `/reload` (or restart Pi) before opening a new session. Existing branches retain their stored enablement on resume/tree navigation. `/state-flow-stop` does not rewrite this file or disable automatic mode for later new sessions. Invalid JSON, unknown keys or invalid values fail extension loading rather than silently choosing another directory. A missing file uses defaults and is not generated automatically.
+
+The configuration file stays in Pi's agent directory when state storage moves. A directory override selects a store; it does not migrate old data or Git objects. SDK embedders can set `StateFlowExtensionOptions.agentDir` for the profile, `repositoryRoot` to override the configured state directory, and `knowledgeRoot` independently for Markdown sources.
 
-## Usage
+## Usage and activation
 
 ```text
-/state-flow-start
+/state-flow-start   # Initialize missing materialization and enable this branch
+/state-flow-status  # Inspect runtime, semantic, freshness, and publication diagnostics
+/state-flow-stop    # Disable this branch without deleting semantic state
 ```
 
-In a fresh session, the next prompt starts the first stateful run. Each later non-retry user prompt becomes the current turn-stable specification while committed state remains intact. When enabled inside an existing session, the first complete run retains the active pre-Flow context and must migrate every future-relevant fact into its terminal handoff.
+A genuinely new session starts in ordinary Pi mode unless `autoStart` is enabled. Manual-mode startup is read-only with respect to storage initialization/migration. Configured automatic start uses the same optional-Git setup as explicit start and can create missing CWD materialization. Each enabled new session gets its own empty session layer and inherits only global/CWD state. It never borrows another session's values. Starting mid-conversation retains the active pre-Flow context for one complete bootstrap run; its terminal handoff must migrate future-relevant context.
 
-The extension registers no model tools. Tool-bearing assistant responses use ordinary Pi tools and contain no State Flow patch. When memory changes, a terminal assistant response contains one transcript-private memory-patch comment, one separating blank line, and the user-facing answer exactly once. When memory is unchanged, an ordinary non-empty answer is sufficient: the runtime treats the missing comment as an empty memory patch, preserving `contract` and `working` while replacing `response`. Transcript-private means removed from the finalized message and later model context, not confidential during streaming:
+Stopping changes the current session/branch configuration, preserves semantic checkpoints and tails, and creates no semantic history step. It does not change the agent-level `autoStart` setting for future new sessions. Resume and `/tree` restore the selected branch's configuration, metadata, and exact temporal lineage, not arbitrary repository `HEAD`. Older revisions are read through Git objects without resetting or checking out the shared worktree.
 
-```html
-<!-- state_flow {"contract":{...},"working":{...}} -->
+The compact status is an accent `state-flow` and dim `#<step>`. Detailed status distinguishes runtime config/meta from semantic materialization, reports scope keys, the selected temporal head and runtime revision, available hot offsets, per-scope retained patch tails, artifact counts, stale Markdown reasons, pending publication, and terminal retries. Retained tails may include inherited pre-origin records, so their counts are not the active history depth. It labels global/CWD/session/effective state without dumping source bodies. Failed inspection means unavailable evidence, never a falsely clean count.
 
-Complete user-facing answer
+## State, scope, and time
+
+Every scope has exactly the same semantic shape:
+
+```json
+{
+  "artifacts": {},
+  "contract": {},
+  "working": {},
+  "response": "Latest complete user-facing answer"
+}
 ```
 
-At `message_end`, an explicit private comment must start the single terminal text block and be followed by exactly one blank line. Without a State Flow comment marker, ordinary text blocks are concatenated without inserted characters and preserved unchanged; no regeneration is triggered merely because the comment is absent. Malformed, incomplete, or embedded State Flow comment markers still fail explicit-envelope validation rather than silently becoming no-op patches. It removes only the comment and separator and stages all three state fields. At `turn_end`, it reconciles `response` with Pi's finalized assistant message after all chained `message_end` handlers, concatenating any text blocks without inserting characters, then commits. If a later handler removes all answer text or adds a tool call after terminal validation, the same hidden regeneration chain runs. Arbitrary response Markdown remains outside the HTML comment, so its own content cannot terminate the private frame. Invalid terminal commits are regenerated through hidden feedback up to three times. If all retries fail, State Flow remains enabled, preserves the last committed snapshot, abandons only the transient validation chain, and lets the next user request start cleanly from that state. Tool-bearing turns do not reset the attempt count. Aborting a regeneration has the same non-destructive behavior.
+- `artifacts`: Source-path-keyed compiled routing metadata and reusable operational knowledge.
+- `contract`: Stable requirements, decisions, rejected approaches, and interface commitments.
+- `working`: Verified observations, validation, failures, unresolved questions, and exact continuation.
+- `response`: The complete answer captured by the runtime, normally in session state; models cannot patch it directly.
+
+Effective state recursively overlays `global → cwd → session`; session wins. Scope is ownership, not instruction authority. Use session for branch/run-local continuation, CWD for project-local reusable state and Skill artifacts, and global for cross-project knowledge. Scope-local deletion reveals lower-scope values, including when reconstructing history.
+
+Each accepted materially effective transition has one opaque identity and explicit parent lineage. Multi-scope patches share that identity. Unchanged scopes have no patch for that boundary and retain their values. Temporal position is not a count of local mutations or a timestamp sort. A branch-local ordinal may order parent-linked identities but cannot replace identity or merge forks.
 
-Commands:
+For example:
 
 ```text
-/state-flow-start   # Start a fresh episode
-/state-flow-status  # Inspect iteration metadata and the complete state JSON
-/state-flow-stop    # Stop and clear the complete episode
+Boundary       T181   T182   T183   T184
+Global         G      G      G'     G'
+CWD            C      C'     C'     C''
+Session        S      S'     S''    S'''
+
+At T184:
+state[1]         = overlay(G', C', S'')
+state.cwd[1]     = C'
+state.cwd[2]     = C'
+state.global[1]  = G'
 ```
 
-The compact Pi status renders an accent `state-flow` label followed by the dimmed committed iteration number, for example `state-flow #7`. `/state-flow-status` reports iteration metadata, then separates the complete pretty-formatted materialized state JSON with a blank line for operator analysis.
+`state.cwd[1]` is not the preceding CWD patch. Every projection first resolves the same target in the active lineage. Offsets zero through seven are the hot range; offsets before a new or migrated lineage's proven origin are explicitly unavailable until enough transitions exist. Offset eight is outside this read contract even when Git contains older history.
 
-`/state-flow-stop` clears the specification, materialized state, validation feedback, and committed-run counter. A later `/state-flow-start` always begins a fresh episode. State follows the active Pi session branch and is restored immediately after `/tree` navigation. If the newest checkpoint is malformed, restoration walks backward to the newest valid checkpoint instead of resetting an otherwise recoverable episode. Entering a branch with no State Flow snapshot leaves the mode disabled there.
+### Lazy historical reads
 
-## Materialized state
+Normal inference receives current effective `state[0]`, useful compact transition context, the current specification, and the complete current-run trajectory. It does not receive eight full snapshots. History is materialized only on a requested effective or scope read.
 
-The model sees one persistent state object before the current run trajectory:
+The pure accessor is `readTemporalState(view, offset, scope?)` in `lib/temporal.ts`, with scope omitted for effective state. The live adapter exposes the same lazy read semantics to embedders through `StateFlowExtensionOptions.onRuntime`; the callback receives a cached-runtime `read(offset?, scope?)` accessor. The model tool `read_state` accepts optional `offset` (integer 0–7, default 0) and `scope` (`effective`, `global`, `cwd`, or `session`, default `effective`). For example:
 
 ```json
-{
-  "contract": {},
-  "working": {},
-  "response": "Latest complete user-facing answer"
-}
+{"offset": 1, "scope": "cwd"}
 ```
 
-Materialized state has exactly three fields:
+It returns one `{offset, scope, boundary, state}` result from cached runtime, with no publication, checkpoint append, or history step. Unavailable pre-origin history is an error, not an empty state. Use it for a concrete historical or scope-specific gap, not routine rereading of current context. Both State Flow tools are active only while the selected branch is enabled; `patch_state` remains the sole mutator and blocks `read_state` siblings at its inference barrier. SDK embedders using an explicit tool allowlist must include `read_state` as well as `patch_state`. The model reasons about state, scope, time, and transitions, not checkpoint folding mechanics.
 
-- `contract` is a flexible object containing durable user requirements, stable decisions, rejected approaches, interface commitments, and compact operational knowledge compiled from relevant Skills or documents.
-- `working` is a flexible object containing verified facts, artifacts, validation, failures, unresolved work, current environment or domain state, and the exact continuation point.
-- `response` is the exact non-empty answer body captured by the runtime. It replaces the previous response on every commit.
+## Temporal storage
 
-The transcript-private wire patch contains only `contract` and `working`; the runtime derives `response` from Pi's finalized answer body. Memory patches merge recursively and materialize immediately. Empty objects preserve existing content; arrays and primitives replace. Nested object-key `null` deletes that key:
+State Flow owns a separate state directory at `~/.pi/agent/state-flow/`, optionally backed by Git. Knowledge remains an independent Markdown source at `~/.pi/agent/knowledge/`. Both defaults follow Pi's configured agent directory (`PI_CODING_AGENT_DIR`); [configuration](#configuration) can select a different state store. Changing the storage root does not redirect Markdown discovery.
 
-```html
-<!-- state_flow {"contract":{},"working":{"move":null,"result":"ok"}} -->
+```text
+state-flow/
+├── .git/                 # When Git-backed
+├── checkpoint.json
+├── patches.jsonl
+└── <cwd-key>/
+    ├── checkpoint.json
+    ├── patches.jsonl
+    └── <session-key>/
+        ├── checkpoint.json
+        ├── patches.jsonl
+        ├── config.json
+        └── meta.json
+```
+
+Directory names deliberately mirror Pi's native session layout. For example, `/home/llb/Repos/deos` is stored as `--home-llb-Repos-deos--`, and `2026-09-07T20-01-08-993Z_<uuid>.jsonl` uses `2026-09-07T20-01-08-993Z_<uuid>` as its session directory. In-memory sessions derive that same `<timestamp>_<uuid>` form from Pi's header. The UUID remains the authoritative runtime identity; session metadata and CWD checkpoint ownership reject foreign identities and CWD-name collisions without hiding another digest in the path.
+
+Explicit `/state-flow-start` creates a missing state directory. With Git available, start initializes an exact-root Git repository when needed, including a populated state directory, without deleting existing files or staging unrelated content. An active file-backed branch must identify its exact current cohort before it can be adopted into Git. If the Git executable is absent, state persists directly to files without repository operations. Git command failures, corruption, or permission errors are not treated as executable absence. Manual-mode startup, status, and branch restoration do not initialize Git; configured automatic start for a new session may do so. The extension creates no GitHub account, external repository, or remote configuration, and does not invent a Git commit identity.
 
-The operation completed.
+For local debugging after Git-backed initialization:
+
+```bash
+git -C ~/.pi/agent/state-flow status --short
+git -C ~/.pi/agent/state-flow log --oneline -10
+git -C ~/.pi/agent/state-flow show <revision>:checkpoint.json
 ```
 
-This removes `working.move`, replaces `working.result`, preserves every other memory key, and stores `The operation completed.` as `response`. Materialized state cannot contain `null`, including inside arrays; represent semantic absence by omitting a key or using an explicit non-null domain value.
+`/state-flow-status` identifies the active scope keys and selected revision. Live files may belong to a later branch; use that selected revision for historical inspection rather than assuming the worktree represents the current Pi branch. State may include private session content; review it before configuring any remote.
 
-State and patch size have no byte, growth, pressure, or project-schema limit. Patch history remains only in the Pi trace, not in model context.
+All three semantic scopes use one storage model:
 
-These three fields are the complete **materialized State Flow state**, but they are not the only state that can shape model behavior. A situational fourth, exogenous state lives outside the handoff: the current project, workspace, tools, processes, and runtime environment. It can change while work is in progress—including through the model's own tool effects—and later observations of those changes can alter subsequent behavior. State Flow neither snapshots nor rolls back this external state; `working` should retain only the decision-relevant facts needed to reconnect the next run to it.
+```text
+current scope = materialize(anchored checkpoint, ordered patch tail)
+current effective = overlay(global current, CWD current, session current)
+```
 
-### Reconnecting to external state
+The checkpoint is an **older** materialized scope snapshot with an unambiguous `through` boundary. Its envelope is runtime-owned; its `state` contains only the four semantic fields. Each `patches.jsonl` holds at most seven scope-local replay records after that anchor. A record carries the shared transition identity/lineage and an effective semantic patch, including runtime-observed artifact freshness when applicable.
 
-Treat `working` as the last observation, not a live workspace. Before consequential actions, revalidate the volatile facts that action depends on: for example, the current revision and dirty files before editing, or the remote publication status before retrying a release. This is targeted inspection, not a requirement to reread stable knowledge or compiled Skills routinely.
+On an eighth scope patch, apply the oldest retained patch into the checkpoint, advance `through`, remove that patch, and append the new one. This deterministically preserves current state and the seven-transition hot window. Sparse scopes may keep older checkpoints; their values are still reconstructed at the same requested effective boundary. Never replay a tail over an already-current snapshot or silently drop an unapplied patch. The implemented `serializeScopeStream`/`parseScopeStream` codec emits deterministic checkpoint/JSONL bytes and validates complete replay input, including semantic results, before acceptance. `publishTemporalStateToGit` writes selected streams from one validated view through exact-output Git publication, rejects omitted changes and stale/wrong-session bases, and leaves unchanged scopes alone. `loadScopeStream` reads the live pair; `loadTemporalRevision` reads regular-file Git objects without moving the worktree. Neither invents missing runtime lineage; callers must supply the lineage belonging to the same revision. Oversized tails, invalid lineage, incomplete pairs, and no-op records are rejected; legacy current-state documents require explicit migration rather than implicit decoding.
 
-After an interruption or session-branch navigation, inspect relevant external effects before repeating operations. A failed terminal commit does not undo file edits, running processes, or remote requests; restoring older memory does not restore the workspace. Missing memory is evidence of neither success nor absence of effects. If the effect cannot be verified, retain that uncertainty and the next discriminating check instead of blindly retrying or claiming completion.
+Accepted replay cohorts contain an identity and exact scope patches, without an independent timestamp clock or current-state publication DTO. Staging validates both semantic hashes and the active causal-boundary identity. Compact transition context derives directly from the selected lineage; `transitionWindow` retains its per-scope projection budget but never changes the shared temporal target of `state[n]`. Legacy explanatory journals and Pi `recentTransitions` payloads are not semantic recovery inputs.
 
-These are protocol obligations, not runtime freshness checks, rollback, or exactly-once execution guarantees. No action ledger or new evidence-retrieval tool is introduced.
+Live files, or their branch-selected cached representation, serve hot materialization. When available, Git serves complete cold history for committed transitions, durable provenance, explicit historical inspection, and branch restoration; it is not queried to rebuild current state on every inference. There is no second unbounded event store, database, `.state-flow`, `scopes`, or parallel history directory.
 
-## Terminal handoff quality
+### Runtime ownership and Git publication
 
-The terminal answer commits a handoff, not a progress phrase or transcript summary. A memory patch is required when future-relevant memory changes; omitting it is only shorthand for preserving existing memory, not a way to infer new memory from prose. The next run may see only this state plus its new user prompt. A fresh model should be able to continue without rereading, rediscovering, re-deriving decisions, or repeating failed approaches.
+Session `config.json` owns runtime behavior such as `enabled` and `transitionWindow`. Session `meta.json` owns temporal head/lineage, counters, durable base, Pi branch correlation, pending publication, and migration/version metadata. They are not model-patchable state and do not overlay into effective state. Pi checkpoints bind the selected branch to the corresponding durable runtime/temporal revision. The implemented runtime codec records version, canonical CWD/session identity, bounded active lineage, counters and run metadata, `revision: "self"`, and an unconfirmed publication intent. The self reference resolves to the last commit owning either member of the config/meta pair, so config-only changes advance runtime revision without semantic history, while unrelated commits in the state repository do not change its meaning. The commit hash is supplied during resolution rather than embedded in its own content; the resulting publication target identifies the existing commit to reconcile or retry. Unconfirmed means no durable confirmation is available, not proof that the push failed.
 
-Useful handoffs capture:
+Git-backed Pi checkpoint data is exactly `{"revision":"<full commit hash>"}`; file-only checkpoints use `{"revision":"file:<64 lowercase hex>"}` for an exact current-cohort reference. `{"disabled":true}` denotes an ordinary disabled branch without durable runtime. Configuration, counters, specification, lineage, and semantic state are not copied into these entries. Older config/meta and state-bearing checkpoints remain read-only compatibility input. Invalid immutable pointer targets fall back through the selected branch; transient publication-lock failure instead retains the selected revision for retry. Explicit start on a pre-runtime branch establishes an empty session origin, even if a later branch already created same-session files. Shared streams remain unchanged and the later session state remains recoverable from its Git revision; it is not imported into the new origin. Legacy checkpoint syntax and immutable targets undergo the same fallback checks, while an unanchored legacy semantic payload cannot be silently reduced to an ordinary-disabled marker. Stop on an unproven branch fails visibly instead of turning failed recovery into permission to replace existing runtime; select a valid checkpoint before retrying.
 
-- Stable requirements, constraints, decisions, rejected approaches, and compiled operational rules in `contract`.
-- Verified facts, changed artifacts, validation evidence, failures, unresolved work, current domain state, and exact continuation in `working`.
+Every materially effective semantic transition, including response-only and session-only changes, immediately publishes affected checkpoint/tail pairs and necessary lineage metadata. In Git mode each accepted cohort gets its own consistent local commit and, when a remote is configured, a push attempt; intermediate patches are not batched until turn end. With no remote configured, Git persistence is intentionally local-only, not a pending publication error. With Git absent, persistence remains file-only and does not report fake commits or pending pushes. Multi-scope changes use one identity and one commit. Config-only changes may persist runtime state but never invent semantic transitions.
 
-Before compressing, preserve active constraints, unresolved questions, consequential negative results, and the next check that would distinguish competing explanations. Keep observations, user requirements, assistant decisions, and hypotheses distinguishable; an assistant conclusion is not a user requirement. Retain decision-relevant hypotheses as uncertain rather than deleting them merely because they are unverified.
+Writes use regular non-symlink owned files, same-directory atomic rename, isolated Git indexes, and compare-and-swap against the reconciled base. Cooperating State Flow Git publishers hold a common-Git-directory publication lock through capture, commit, and rollback. Raw bytes own file identity and recovery; non-UTF-8 explanatory journals are not reconstructed from decoded strings. Per-file publication checks catch changed bases. Both semantic and migration publishers retain exact prepared-output receipts: Git commits those bytes, not later worktree contents, and rollback requires the receipt to preserve detected external changes rather than replacing them. Low-level file helpers require caller exclusion; this is not kernel-atomic multi-file CAS against writers ignoring the protocol. An existing publication lock fails before writes and is not silently stolen; reconcile its active or interrupted owner before retrying. Scope keys mirror Pi's CWD/session directory names and retain separately verified canonical identity provenance inside owned state. Conflicts fail rather than silently selecting another scope or auto-merging semantic state. Unrelated staged and dirty state-repository files remain untouched. With distinct roots, the Knowledge repository's files, index, HEAD, locks, and remote are not used for state publication; keep custom stores separate from sources to retain this isolation.
 
-For consequential facts, include a compact source locator and validity condition when useful (for example, a test command and the revision it checked), not mandatory metadata on every value. Record why an approach was rejected and what would justify reconsidering it. When new information conflicts with an established constraint or observation, reconcile it using evidence or user clarification; neither an unsupported new claim nor fallible old memory wins automatically. If unresolved, preserve the conflict and the next discriminating check.
+A local commit accepts the transition. Push failure is pending publication, not a reason to regenerate the answer or repeat semantic state changes. Pending state survives restart; retries push the existing accepted commit. A restored branch reads its recorded revision without moving the shared worktree and must reconcile safely before publication.
 
-These are model obligations, not semantic validation gates. Protocol tests check that the instructions remain present; only continuation evaluations can establish whether a fresh agent makes the next correct decision.
+### File-only recovery
 
-Do not store raw source, logs, tool output, reasoning traces, or vague values such as `"continue work"` and `"in progress"`.
+File-only mode keeps the same scoped materialization, hot temporal window, and separate config/meta. Its Pi checkpoint identifies the store root and exact current file cohort, including runtime metadata, not a Git commit. Restart can restore that pointer while the complete referenced cohort remains available. Old branches or replaced cohorts can be unavailable; current files must not be passed off as the selected past. Unavailable references and publication locks retain the selected pointer rather than falling through to an older disabled marker. A crash between file publication and Pi checkpoint append can leave the previous pointer unavailable: the files remain, but automatic branch restoration cannot assert they belong to that pointer. There is no second unbounded history store, and Git-linked cold recovery still requires Git and its original objects.
 
-### Skill compilation
+If Git becomes available later, explicit start can adopt the selected current file cohort into Git. Adoption preserves semantic state, step, lineage, hot history and exact scope bytes; it changes runtime provenance, not semantic history. It commits the full current cohort rather than inheriting stale or missing HEAD blobs, preserves unrelated staging, and leaves file state recoverable if Git publication fails. A new session can also initialize Git over inherited file-only global/CWD scopes while retaining its own empty session layer. Git cold history begins at adoption and does not fabricate commits for earlier file-only transitions. An already-anchored identical Git cohort can reuse its owner revision.
 
-A successful `SKILL.md` read is treated as episode-level acquisition. Before terminal commit, the model compiles its future-relevant operational rules, applicability conditions, constraints, syntax, routing decisions, and failure conditions under the exact source path. This compilation lives inside persistent `contract` memory at `contract.compiled_skills`; `compiled_skills` never becomes a fourth top-level state field. The nested representation remains fully flexible:
+### Existing stores and migration
+
+Changing the default directory does not import, move, or delete an old Knowledge-backed store. Existing Pi checkpoints name Git revisions: copying only the current checkpoint/tail files cannot preserve their recovery contract. An old session requires its original revision history in the selected repository; unavailable revisions must not be treated as permission to reset the session. Keep the old store intact until an explicit history-preserving relocation is performed, or use a genuinely new Pi session for an independent debug store. An SDK host can explicitly select the old `repositoryRoot` when accessing that history.
+
+The following is an **in-store format migration**, not a cross-repository relocation. Let State Flow perform it; manually renaming files is not a valid conversion.
+
+An existing current `state.json` becomes the initial checkpoint state, anchored at a proven current transition/base, with an empty tail. Its explanatory old journal is not replayed. Migration must preserve exact semantic state and remove obsolete `state.json` ownership only after successful publication, leaving one authoritative format. It must not fabricate seven historical patches. Git retains older evidence; the new hot window grows from the proven migration origin. The explicit internal migration primitive now plans without writes, publishes all converted scope files in one isolated-index commit, and restores exact old bytes after failure when its published files have not concurrently changed; conflicting external bytes remain preserved with an explicit recovery error. It rejects ambiguous dual snapshots and orphaned tails, does not depend on explanatory legacy journal validity, and retains legacy interpretation only for explicit migration. Production no longer exposes the predecessor current-state/journal writers; legacy fixture writers are test-only. Pi initialization now invokes this migration when establishing the temporal runtime, and revision-linked predecessor snapshots have a one-way restoration path. The untagged pre-0.4 development layout that appended SHA-256 suffixes remains readable at its historical Git revisions; if the selected layout is current `HEAD`, startup adopts its CWD/session files into Pi-native paths with compare-and-swap publication, rollback, and an immediate replacement revision pointer. Historical legacy restoration reads only the authoritative current snapshots, not explanatory journals. Cached view and publication basis are installed together after successful initialization/restoration; a transient restore failure preserves the selected revision for explicit start retry and cannot masquerade as a successful state transition. Independent rereview confirmed these three failure-path fixes. Stop also remains harmless in ordinary disabled sessions and retries a failed branch restoration before durably disabling that branch.
+
+## Intermediate barriers and terminal reconciliation
+
+Use `patch_state` only for established future-relevant state whose delayed persistence risks meaningful loss, never scratchpad, narration, routine progress, or speculative churn:
 
 ```json
-{
-  "contract": {
-    "compiled_skills": {
-      "/exact/path/to/SKILL.md": {
-        "routing": "...",
-        "syntax": { "...": "..." },
-        "constraints": ["..."],
-        "reread_when": ["..."]
-      }
-    }
-  }
-}
+{"scope":"session","patch":{"working":{"verified":"result"}}}
+```
+
+```text
+LLM(state[0] = Sn)
+  → patch_state
+  → validate, publish, materialize Sn+1
+  → next LLM(state[0] = Sn+1, state[1] = Sn)
+  → choose the next action
 ```
 
-No `coverage`/`rules` schema is imposed. The model chooses the smallest structure that faithfully preserves the Skill's useful behavior. Raw Skill text is not copied. The runtime checks only that the exact successfully executed source path has a non-empty compilation and rejects a terminal commit when it is missing. Attribution follows Pi's lifecycle order: it retains the mutable `tool_call` input reference so later interception rewrites resolve to the path actually executed, with `tool_execution_start` arguments as a compatibility fallback.
+The barrier executes alone. State Flow opts into sequential preflight, inspects Pi's synchronized assistant response, requires exactly one `patch_state` call, and blocks every sibling tool before execution. The next inference sees one rebuilt current-state projection; it must not reconstruct the update itself. Tool calls/results remain current-run causality, not duplicate full-state messages.
 
-A complete matching compilation is authoritative episode memory and replaces routine rereading. Rereading is justified only by an explicitly uncovered detail, an incomplete compilation, concrete source-change evidence, a contradiction or execution failure requiring reconciliation, or an explicit user request. The mere possibility that a source changed is not sufficient. A justified reread refreshes the compilation and removes obsolete rules.
+Every successful enabled run ends with exactly one terminal reconciliation after zero or more barriers. The model audits and compacts future-relevant state; the runtime captures the finalized answer. If complete semantic state changed, terminal reconciliation enters the same temporal stream. A changed `response` counts as semantic change. If state including response is identical, runtime finalization still completes without a fake patch, identity, or temporal step.
 
-State is a minimal sufficient memory, not an append-only diary. At every handoff the model audits the complete state and may reorganize inefficient structure, merge fragments, replace verbose history with current conclusions, and delete stale, completed, redundant, or low-value keys with `null`. Active requirements, decisions, interfaces, verified evidence, and unresolved work must survive optimization.
+An ordinary non-empty answer without a State Flow marker preserves the three model-owned fields and replaces session `response`. When a model patch is needed, use one top-level transcript-private comment, exactly one separating blank line, and the answer once:
 
-The model must not invent bookkeeping merely to change `contract` or `working`. When a run creates no future-relevant information and existing memory is already efficient, both patch objects may remain empty, or the comment may be omitted entirely, while `response` still contains the actual answer. Successful Skill reads still require their compilations in materialized memory even when the comment is omitted. Bootstrap migration remains a model obligation: a plain answer cannot migrate pre-Flow context automatically. Structurally invalid handoffs are regenerated through hidden validation feedback; retry diagnostics are not user-facing output.
+```html
+<!-- state_flow {"transitions":[{"scope":"session","patch":{"working":{"verified":true}}}]} -->
 
-## Context lifecycle
+Complete user-facing answer
+```
 
-Within one run, State Flow projects:
+Each transition has exactly one unique known scope and an object patch limited to `artifacts`, `contract`, and `working`. Pre-scoped three-field envelopes remain session compatibility shorthand, not a second storage format. Empty object patches preserve values; nested object-key `null` deletes, arrays/primitives replace, and semantic materializations cannot contain `null`, including in arrays. The null prohibition does not cover runtime envelope fields such as an origin's parent.
 
-```text
-persistent materialized state
-+ current user prompt
-+ current-run assistant tool calls
-+ current-run tool results
-+ persistent and current-run context-bearing custom messages from other extensions
-+ optional terminal validation feedback
+At `message_end`, validate and strip the comment and stage the cohort against one basis. At `turn_end`, capture the finalized answer after chained handlers, concatenating text blocks without inserted characters, then publish. Missing text or an added tool call enters the same bounded hidden regeneration chain. Explicit malformed, embedded, or duplicate markers never silently become ordinary no-op answers. A complete marker inside a fenced or inline example still counts as a duplicate; when explaining the protocol during enabled operation, use plain JSON without HTML comment delimiters.
+
+Validation retries remain hidden from finalized answers, retain cumulative attempt counts across tool-bearing turns, and stop after three retries. Exhaustion or abort preserves enabled mode and the last accepted state, abandons only transient validation, and lets the next user request rotate normally. Transcript-private is not streaming-confidential: generated envelopes may be visible to streaming/RPC observers. Never put secrets in state.
+
+## Artifact compilation and acquisition
+
+Artifacts use exact source paths as keys. Minimum metadata is:
+
+```json
+{"description":"What this source contains and when it is useful","hash":"sha256:<64 lowercase hex characters>","compiler":"artifact-v1"}
 ```
 
-Earlier completed-run trajectories are removed from model context. During a bootstrap run, the active pre-Flow context remains available until the first successful terminal commit.
+Descriptions and compiler revisions are non-empty; hashes are canonical lowercase SHA-256. Optional `kind`, `compiled_at`, `compilation`, and `tags` are validated when present, while arbitrary unknown finite non-null JSON metadata remains forward-compatible. Tags are unique non-empty trimmed strings; `selectArtifactsByTags` provides deterministic all/any candidate filtering, but a tag match never authorizes source acquisition. Ordinary artifacts need no `kind` or tags. Hash plus compiler revision determines freshness, not `compiled_at`.
 
-The normative protocol remains in the system prompt throughout a tool/retry chain. The turn-stable specification remains user-authority input: it stays in the initiating user message and is repeated, together with materialized state and private validation feedback, only in a synthetic user runtime-context message. The protocol explicitly treats persistent state as fallible assistant-produced data whose transport role does not elevate it into user instructions. Empty text specifications used by image-only prompts remain valid. State and the current-run trajectory are intentionally unbounded; State Flow does not claim a hard model-context bound.
+Session initialization generically discovers regular lowercase `*.md` beneath the independent canonical Knowledge source root, normally `~/.pi/agent/knowledge` (or `knowledge/` beneath the agent directory selected by `PI_CODING_AGENT_DIR`). This source directory need not be a Git repository. Discovery hashes opaque bytes and retains byte counts without decoding or storing bodies; it skips symlinks and never escapes the root. A missing root means no candidates. Root and nested Markdown are treated alike, without reserved names, frontmatter parsing, validators, templates, or a `save_knowledge` implementation/call. Arbitrary repository files remain independent from State Flow.
 
-## Boundaries
+New, changed, compiler-stale, malformed, missing-metadata, or explicitly refreshed artifacts enter the runtime-owned `artifact_invalidations` plan. Successful exact-path reads of stale ordinary sources require compact global compiler output at that path; State Flow rejects model-authored hash/compiler fields and attaches the observed source hash plus `artifact-v1`. Removed sources create deterministic global removal transitions without reads or model compilation. Runtime replay records must reproduce accepted compiled state exactly, not store incomplete compiler outputs.
 
-- State commits exactly once per successful complete agent run, at terminal `message_end`/`turn_end`.
-- Tool-bearing responses do not require or commit patches and retain ordinary Pi tool behavior, including multiple sequential calls. The runtime removes an accidental leading terminal envelope only when its JSON, fields, separator, and non-empty response are all structurally valid; malformed or quoted examples remain untouched.
-- Terminal answers cannot contain another complete State Flow comment, even inside a fenced code block or inline code: duplicate detection scans the entire answer body, not Markdown structure. When explaining the protocol while State Flow is enabled, describe the fields or show plain JSON without the HTML comment delimiters. The literal envelope examples in this README are documentation, not valid content to copy into a terminal answer body.
-- A failed tool remains in the current run trajectory for ordinary model reconciliation before terminal commit.
-- If a run never reaches a valid terminal handoff, its external tool effects may exist while persistent State Flow state remains at the previous commit.
-- The terminal envelope exists in ordinary assistant text until `message_end`; `message_update`, RPC, JSON, or other streaming consumers can observe it. Never place secrets in State Flow state.
-- This mode remains a poor fit for auditing or outputs that require complete historical trajectories across user requests.
-- Token, cache, latency, and success-rate advantages still require controlled Pi benchmarks.
+Default compilation is a compact routing description, not raw Markdown or a full-file summary. Preserve uncertainty. Richer `compilation` is for reusable operational semantics. An artifact index is not proof that its body was read or understood.
 
-## Architecture
+### Materialized-first policy
 
-`index.ts` is only the package composition and public-export boundary. Independent runtime domains live under `lib/`: `json` owns lossless JSON and patches, `state` owns the materialized state shape, `episode` owns explicit start/stop and user-run boundaries, `snapshot` owns persistence migration, `session` owns active-branch snapshot discovery and bootstrap detection, `recovery` falls back to the newest valid active-branch checkpoint, `status` owns deterministic operator-facing status rendering, `context` owns trajectory projection, private-feedback filtering, and synthetic runtime-context construction, `skills` owns Skill acquisition rules and mutable lifecycle correlation, `terminal` owns the handoff protocol, `validation` owns bounded retry decisions, `transition` owns atomic staging and compare-and-swap commits, and `extension` coordinates these domains with Pi. Every domain has a same-named test under `tests/`; lifecycle scenarios are colocated with the domain whose contract they exercise, `tests/extension.test.ts` remains focused on composition, shared setup lives in `tests/harness.ts`, and cross-domain constraints live in `tests/invariants.test.ts`.
+Read a source only for a concrete relevant gap not covered by sufficient compilation, an exact-source operation including edits, evidenced invalidation, contradiction/failure reconciliation, explicit request, or bounded maintenance selection. The explicit rehydration planner binds scope, path, candidate hash, and reason, then reuses Pi's visible native `read`; State Flow does not register a duplicate artifact-reading tool. New sessions, routine recall/activation, reassurance, and a description/index alone are not reasons to reread. Changed hashes require reacquisition; prefer the smallest sufficient read. Semantic sufficiency is caller-assessed, not proven by the existence of a compilation field.
 
-## Validation
+Optional `planArtifactMaintenance` selects only otherwise-fresh old artifacts, ranking missing/unparseable timestamps first, then oldest `compiled_at` and path. Defaults admit at most one source and 16 KiB after 30 days. Strict count/byte ceilings can be zero; source bytes conservatively bound tokenizer input. The planner never reads, compiles, or modifies sources. Correctness invalidation takes precedence; explicit full refresh is separate from maintenance budgets and never an automatic startup rebuild.
+
+### Skill compilation
+
+Every successful `SKILL.md` read requires CWD compiler output at the exact executed path at the next barrier or terminal transition, with non-empty `description`, `kind: "skill"`, and a non-empty flexible `compilation` object. Compile applicability, routing, constraints, and failure conditions rather than source text. Runtime attaches the executed byte hash and `skill-artifact-v1`; missing/unhashable/forged freshness or missing compilation fails validation. Matching metadata proves source-version consistency, not semantic fidelity, truth, or higher instruction authority. Refresh replaces the complete old artifact, including obsolete metadata.
+
+Correlation follows finalized mutable `tool_call` arguments with execution-start compatibility fallback. Legacy `contract.compiled_skills` entries migrate into artifacts, preserving behavior and marking unavailable-source fallback hashes unverified. New patches cannot recreate the retired store.
+
+### Optional memory curation
+
+The package includes the progressively disclosed `state-flow-memory` Skill for explicit audits, contradiction cleanup, scope narrowing, stale continuation review, and ownership migration. It is not part of routine retention and never runs as background maintenance. The Skill preserves useful uncertainty, confirmed decisions, bounded results, interaction consequences, and completed prerequisites without fixing provisional methods. Its compilation-first sequence writes and separately verifies a destination before source deletion, then inspects the effective overlay; separate single-scope calls are not atomic migration. External promotion cannot delete the only accepted copy without destination identity, content, and revision evidence. State Flow remains the memory owner and global memory remains available throughout.
+
+External handoffs may use the generic global `working.memory_promotions` map. Each entry requires `status` (`pending`, `accepted`, `failed`, or `unknown`) and a non-empty `owner`; accepted entries additionally require an exact destination `pointer` and `revision`. Other fields remain forward-compatible and destination-specific. This bookkeeping does not replace the retained candidate: failures and uncertainty keep the accepted State Flow body plus exact continuation, while proven acceptance may remove the duplicate body and retain the compact pointer record. `/state-flow-status` reports owner mode, fallback activation, memory-bearing scopes, promotion counts, and bounded pointer/error details; malformed or incompletely evidenced accepted records appear as invalid rather than accepted.
+
+## Handoff quality and external reality
+
+A terminal handoff is decision-relevant memory, not narration. Preserve active commitments, confirmed decisions, relevant interaction consequences, bounded results, unresolved questions, and the exact continuation. Keep user requirements, observations, assistant conclusions, provisional methods, and hypotheses distinct; silence is not acceptance. Retain useful evidence locators, validity conditions, exact rejection reasons, and known reconsideration conditions. A failed mechanism does not disprove every implementation, and one success does not establish unrestricted validity. Reconcile contradictions with evidence or user clarification; preserve decision-relevant conflicts and hypotheses explicitly as uncertain.
+
+At every handoff reconcile information affected by the run and relevant existing commitments, not every repository or scope. Merge fragmented facts and delete stale, redundant, unsupported, or obsolete progress narration without losing completed prerequisites, active commitments, or evidence. Do not invent bookkeeping merely to cause a transition. Structural validators cannot prove semantic usefulness or compilation fidelity; fresh-agent continuation tests provide behavioral evidence.
+
+`working` contains last observations, not live reality. Revalidate volatile facts before consequential actions. After interruption or branch restoration, inspect relevant external effects before repeating operations. A failed commit does not undo tools; restored memory does not restore the workspace. Missing evidence proves neither success nor absence of effects. Preserve uncertainty and the next check rather than adding action ledgers or claiming rollback/exactly-once guarantees.
+
+## Context lifecycle and boundaries
+
+The current run retains its user prompt, assistant/tool trajectory, and persistent/current-run context-bearing custom messages from other extensions. State Flow's separately represented retry feedback is excluded. Completed-run trajectories leave model context only at user-run boundaries; Pi's full trace remains inspectable. Bootstrap retains prior active context for its one complete migration run.
+
+The normative runtime protocol stays turn-stable during tool/retry chains. The current specification rotates on each non-retry user run and stays user-authority text, never interpolated into the system prompt. Synthetic user context repeats it with fallible assistant-produced state; the transport role does not elevate that state into instructions. Image-only specifications may be empty.
+
+State, the active specification, current trajectory, and external full trace have no strict size bound. There are no project schemas, growth-pressure gates, action authorization/observation envelopes, state-byte caps, or total-model-context guarantees. This mode is a poor fit when every new request must reason over complete historical trajectories. Token/cache/latency/success advantages still require controlled benchmarks.
+
+## Architecture and validation
+
+See the [documentation index](docs/README.md), [continuity acceptance map](docs/continuity-acceptance.md), and [twenty temporal acceptance properties](docs/temporal-acceptance.md) for requirement-to-test maps and verification limits.
+
+`index.ts` remains composition/public exports only. Flat independent `lib/` domains own their responsibilities: `temporal` owns anchored streams, folding, active-lineage validation, and lazy historical algebra; `durable` owns exact files; `git` owns revision/CAS publication; `history` owns compact transition records; `state` owns semantic shape/overlay; `transition` coordinates accepted patches; `snapshot`, `session`, `recovery`, and `episode` own runtime/branch lifecycle; `context`, `status`, `terminal`, and `validation` own projection and finalization; artifact domains own discovery/acquisition/compilation/maintenance; `extension` composes Pi integration. `runtime` caches branch-selected streams and their publication basis beneath the Pi adapter; `config` owns read-only agent-level settings independently from branch runtime configuration. Predecessor current-state/journal writers are test-only; production retains explicit migration readers, and Pi checkpoints carry revision pointers.
+
+Each domain has a same-named test, with cross-domain DAG/composition invariants in `tests/invariants.test.ts`. Real Pi SDK scenarios use `tests/pi-harness.ts`; ordinary fixtures use `tests/harness.ts`.
 
 ```bash
 npm install
 npm run validate
 ```
 
-Validation covers TypeScript checking, automated tests, and an extension import smoke check. Lifecycle tests use a mock Pi event harness; they do not establish behavior under the real Pi scheduler, queued prompts, abort/compaction interactions, or combinations of extensions. Those scenarios still need integration checks. Structural patch validation also cannot prove that a handoff preserved every important requirement or compiled a Skill faithfully.
-
-## Project status
+Validation runs TypeScript checks, automated tests, and an import smoke check. Temporal algebra tests compare all available offsets with independently retained test snapshots through dense/sparse repeated folding, shared transitions, deletion overlays, no-ops, and forks. Temporary-repository tests also cover explicit migration, ten committed sparse transitions with folding and all hot offsets, cold Git scope recovery, and historical symlink rejection. Storage-level tests also prove atomic runtime/semantic publication, config-only stop without semantic history, runtime identity and lineage validation, self-reference resolution across unrelated commits, and exact accepted-commit retry. Real Pi tests also exercise configured automatic activation, temporal barriers and next-inference state, lazy runtime offset-one reads, unchanged shared-stream inheritance, and old-branch stop/resume. The complete twenty-case temporal acceptance audit is documented in the property map; the canonical backlog identifies any subsequent open work.
 
-- [Open work](BACKLOG.md)
-- [Release history](CHANGELOG.md)
-- [Agent and contributor constraints](AGENTS.md)
+- [Canonical open work](BACKLOG.md)
+- [Delivery history](CHANGELOG.md)
+- [Agent/contributor protocol](AGENTS.md)

package/node_modules/@llblab/pi-state-flow/docs/README.md

@@ -0,0 +1,4 @@
+# State Flow documentation
+
+- [Architecture](architecture.md): Semantic state, temporal algebra, Pi lifecycle, storage, optional Git, asynchronous publication, artifact routing, durable memory, and session-continuation boundaries.
+- [Temporal acceptance](temporal-acceptance.md): The twenty required temporal properties and their executable witnesses.