@llblab/pi-state-flow 0.9.5 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +13 -10
- package/BACKLOG.md +15 -1
- package/CHANGELOG.md +18 -0
- package/README.md +52 -261
- package/docs/README.md +5 -1
- package/docs/architecture.md +47 -26
- package/docs/compatibility.md +97 -0
- package/docs/fork-contract.md +47 -0
- package/docs/performance.md +459 -0
- package/docs/temporal-acceptance.md +35 -3
- package/docs/usage.md +134 -0
- package/lib/artifact.ts +19 -3
- package/lib/compaction.ts +74 -0
- package/lib/context.ts +12 -12
- package/lib/continuation.ts +4 -2
- package/lib/discovery.ts +21 -5
- package/lib/extension.ts +146 -37
- package/lib/git.ts +142 -42
- package/lib/publication.ts +80 -27
- package/lib/runtime.ts +109 -7
- package/lib/status.ts +7 -12
- package/lib/storage.ts +2 -1
- package/lib/telegram.ts +13 -3
- package/lib/transition.ts +2 -3
- package/package.json +5 -4
package/AGENTS.md
CHANGED
|
@@ -2,16 +2,16 @@
|
|
|
2
2
|
|
|
3
3
|
- Keep independent domain modules under `lib/`, mirror every domain with a same-named file under `tests/`, place cross-domain architecture checks in `tests/invariants.test.ts`, and keep `index.ts` as a minimal composition/public-export boundary.
|
|
4
4
|
- Keep the extension opt-in and preserve clear attribution to SKILL.state wherever the inherited explicit-state approach is described.
|
|
5
|
-
- Preserve Pi's native tool loop and complete inspectable session trace; project completed-run history only at user-run boundaries. Retain persistent and current-run context-bearing custom messages from other extensions, plus the complete current-run trajectory, except State Flow's own validation feedback when it is represented separately.
|
|
5
|
+
- Preserve Pi's native tool loop and complete inspectable session trace; project completed-run history only at user-run boundaries. The runtime-context builder owns model projection of the raw scope overlay; callers must not pre-project it. Retain persistent and current-run context-bearing custom messages from other extensions, plus the complete current-run trajectory, except State Flow's own validation feedback when it is represented separately.
|
|
6
6
|
- Expose one canonical materialized state shape across global, CWD, and session scopes with exactly `artifacts`, `contract`, `working`, and `response`; the first three are flexible semantic objects and the fourth is the latest complete user-facing answer string where applicable.
|
|
7
|
-
- Key artifacts directly by their source path. Model-visible artifact entries require only a non-empty `description` plus optional forward-compatible metadata; runtime-owned freshness evidence (`sourceHash`, `compilerRevision`, `compiledAt`) belongs in the scope `meta.json` provenance registry, never in projected semantic state. Retired embedded `hash`/`compiler`/`compiled_at` fields remain readable compatibility evidence and are stripped from model projection.
|
|
8
|
-
- After local activation and before the next enabled inference, discover regular lowercase `*.md` files recursively beneath the configured Knowledge repository root; use canonical absolute paths, hash opaque source bytes and retain their byte counts without decoding or retaining bodies, skip symlinks, and never traverse outside that canonical root. Treat the current candidate set plus explicit removals as generic global artifact input without invoking Knowledge validators. Materialize removals deterministically as runtime-owned global transitions without rereading missing bodies or requesting model compiler output. Never reserve Knowledge document names, provide built-in Knowledge templates, or implement `save_knowledge`.
|
|
7
|
+
- Key artifacts directly by their source path. Model-visible artifact entries require only a non-empty `description` plus optional forward-compatible metadata; runtime-owned freshness evidence (`sourceHash`, `compilerRevision`, `compiledAt`) belongs in the scope `meta.json` provenance registry, never in projected semantic state. Retired embedded `hash`/`compiler`/`compiled_at` fields remain readable compatibility evidence and are stripped from model projection. Validate provenance restrictions on every authored artifact entry, not on retained legacy state: individual runtime/legacy provenance fields cannot be set or deleted by model patches in any scope.
|
|
8
|
+
- After local activation and before the next enabled inference, discover regular lowercase `*.md` files recursively beneath the configured Knowledge repository root; use canonical absolute paths, hash opaque source bytes and retain their byte counts without decoding or retaining bodies, skip symlinks, and never traverse outside that canonical root. Treat the current candidate set plus explicit owner-confirmed removals as generic global artifact input without invoking Knowledge validators. Partial candidate sets, unavailable roots, and symlink/external/non-Markdown paths never authorize deletion. Re-derive removals from the retained global registry so status inspection and restart cannot consume uncommitted observations; unavailable roots mean unknown freshness. Materialize removals deterministically as runtime-owned global transitions without rereading missing bodies or requesting model compiler output. Never reserve Knowledge document names, provide built-in Knowledge templates, or implement `save_knowledge`.
|
|
9
9
|
- Derive artifact freshness per available evidence: a new source always requires compilation, a matching `sourceHash` or `compilerRevision` detects its own change, malformed present evidence fails closed for the capability that depends on it, and missing evidence degrades to an unknown-but-usable artifact rather than corrupt state or a migration. Never treat `compiled_at` as a correctness signal. Plan acquisition from path/hash identity without source bodies, project stale global candidates as runtime-owned `artifact_invalidations` containing only `{path, reason}`, and publish accepted compilation semantics plus matching provenance in one durable cohort.
|
|
10
10
|
- Correlate successful exact-path reads with the current ordinary global invalidation plan. Require each acquired stale source to have a compact compiler output at the same path in a global transition, reject model-authored provenance fields, and record the runtime-observed source hash plus current ordinary compiler revision in the global provenance registry. Compile routing descriptions by default, never raw Markdown or a full-file summary; preserve uncertainty and reserve richer compilations for reusable operational semantics.
|
|
11
11
|
- Apply one materialized-first acquisition policy to every source: read only for a concrete relevant gap not covered by a sufficient compilation, an exact-source operation including edits, evidenced invalidation, contradiction/failure reconciliation, explicit request, or selection by bounded maintenance. New sessions, routine recall or activation, reassurance, and an index or description alone are not reasons to read. Changed hashes require rereading; prefer the smallest sufficient read.
|
|
12
12
|
- Keep artifact maintenance opt-in, low-frequency, and side-effect free until a selected source is acquired. Select only otherwise-fresh artifacts old enough for maintenance, rank missing/unparseable `compiled_at` first then oldest timestamp and path, and enforce strict per-cycle read-count and source-byte ceilings. Treat source bytes as the conservative source-token upper bound. Never let maintenance displace correctness invalidation; use explicit invalidation refresh for full rebuilds and never mutate source files.
|
|
13
13
|
- Load optional agent-level `state-flow.json` once per extension load/reload. Its `directory` selects the state store, `autoStart` defaults false for genuinely new sessions, and `remotePublication` selects `off`, `turn-end`, or compatibility `transition`. State Flow always owns durable memory while enabled and global semantic memory is always available; these are invariants, not configuration switches. Keep runtime configuration and provenance outside model-patchable semantic state.
|
|
14
|
-
- Keep runtime configuration and provenance separate from model-patchable semantic state. Persist session `config.json` for enablement/projection settings and `meta.json` for lineage, counters, branch identity, durable base, publication, migration metadata, and session-scope artifact provenance; global and CWD scopes own their artifact provenance in scope `meta.json`. Neither participates in scope overlay. New Pi checkpoints contain only a durable revision pointer, or `{disabled:true}` when no durable runtime exists on that branch. Preserve exact Git hashes for Git-backed revisions and give file-only current-cohort references an unambiguous distinct identity; file references use `file:<64 lowercase hex>`, bind the store root and complete scoped/runtime cohort, and must not pretend to provide arbitrary cold or branch history. Operationally unavailable references (including expired file cohorts, missing Git, and publication exclusion) retain selection rather than falling through to older disabled markers; malformed immutable targets remain a distinct recovery case; they are not a parallel authoritative semantic or config/meta store. Emit a disabled marker only from proven pre-runtime branch provenance; missing/invalid revisions and failed restoration do not establish that permission. Predecessor checkpoint formats are read-only migration input. Validate immutable pointer targets before acquiring the live publication basis; transient publication failures preserve the selected retry reference rather than choosing older semantic state. In Git mode, runtime `revision: "self"` resolves to the commit owning the config/meta pair, not an unrelated later repository commit. File publication uses explicit `publication: "files"` provenance and resolves the exact complete live cohort instead of claiming a Git owner. Use a distinct `meta.temporalRevision` when a runtime-only write selects older semantic scopes
|
|
14
|
+
- Keep runtime configuration and provenance separate from model-patchable semantic state. Persist session `config.json` for enablement/projection settings and `meta.json` for lineage, counters, branch identity, durable base, publication, migration metadata, and session-scope artifact provenance; global and CWD scopes own their artifact provenance in scope `meta.json`. Neither participates in scope overlay. New Pi checkpoints contain only a durable revision pointer, or `{disabled:true}` when no durable runtime exists on that branch. Preserve exact Git hashes for Git-backed revisions and give file-only current-cohort references an unambiguous distinct identity; file references use `file:<64 lowercase hex>`, bind the store root and complete scoped/runtime cohort, and must not pretend to provide arbitrary cold or branch history. Operationally unavailable references (including expired file cohorts, missing Git, and publication exclusion) retain selection rather than falling through to older disabled markers; malformed immutable targets remain a distinct recovery case; they are not a parallel authoritative semantic or config/meta store. Emit a disabled marker only from proven pre-runtime branch provenance; missing/invalid revisions and failed restoration do not establish that permission. Predecessor checkpoint formats are read-only migration input. Validate immutable pointer targets before acquiring the live publication basis; transient publication failures preserve the selected retry reference rather than choosing older semantic state. In Git mode, runtime `revision: "self"` resolves to the commit owning the config/meta pair, not an unrelated later repository commit. File publication uses explicit `publication: "files"` provenance and resolves the exact complete live cohort instead of claiming a Git owner. Use a distinct `meta.temporalRevision` when a runtime-only write selects older semantic scopes; restore their artifact provenance from that same revision and preserve live shared checkpoints, tails, and provenance during session config/meta writes. Persist publication intent as unconfirmed before pushing; reconcile that exact existing commit after restart rather than embedding an impossible self-hash or inventing semantic history.
|
|
15
15
|
- Overlay materialized state recursively in `global → cwd → session` order. Scope-local deletion removes only that scope's key so a lower-scope value becomes visible again; scope never changes instruction authority. Route branch/run-local continuation to session, project-local reusable state and Skill artifacts to CWD, and cross-project reusable state to global.
|
|
16
16
|
- Reserve `state` for the runtime materialized semantic view. For each scope, current state is exactly `materialize(checkpoint.json, patches.jsonl)`: an older anchored checkpoint plus its ordered tail of at most seven materially effective semantic patches. On overflow, apply the oldest patch into the checkpoint, advance its `through` boundary, remove it, and append the new patch. Never truncate unapplied replay records or replay patches over an already-current snapshot.
|
|
17
17
|
- Give every accepted semantic transition one opaque identity shared by all affected scope records, with explicit active causal lineage. A branch-local position can order identities but never acts as a global counter, substitutes for identity, or merges forks. Unchanged scopes contribute no record and remain unchanged at that boundary. Adopt revision-proven inherited streams at a new session origin without rewriting their checkpoints/tails; pre-origin coordinates are not one shared clock. A restored branch adopts current proven live shared streams it does not modify at a fresh origin while preserving its selected session layer; a shared scope the accepted transition actually changes must still match its selected basis or fail closed naming that scope. Reconciliation adoption is not a fabricated semantic transition, parent link, or retry loop. True semantic no-ops and origin adoption do not advance semantic history.
|
|
@@ -19,12 +19,13 @@
|
|
|
19
19
|
- Keep historical reads lazy through the smallest runtime/model read interface. Normal inference gets only current effective state and useful bounded compact transition context, never eight full snapshots. `patch_state` is the sole semantic mutation tool. `read_state` exposes one cached effective/global/CWD/session projection at offset 0–7, defaulting to effective/0; it returns the exact boundary with semantic state and never publishes, appends a checkpoint, or advances history. Both tools follow branch enablement and host tool restrictions; the patch barrier blocks reader siblings too. Live/cached checkpoint plus tails own the hot path; use Git for branch restoration and explicit cold inspection, not to rebuild current state on every inference.
|
|
20
20
|
- Store runtime state in its own directory with optional Git durability, defaulting to `state-flow/` beneath Pi's configured agent directory (`~/.pi/agent/state-flow/` normally), independently from the Knowledge Markdown source root. Use exactly these owned paths: `checkpoint.json`, `patches.jsonl`, and `meta.json` for global; `<cwd-key>/checkpoint.json`, `<cwd-key>/patches.jsonl`, and `<cwd-key>/meta.json` for CWD; `<cwd-key>/<session-key>/checkpoint.json`, `patches.jsonl`, `config.json`, and `meta.json` for session. Do not create `.state-flow`, `scopes`, or another storage/history namespace. Mirror Pi's native CWD session-directory encoding and JSONL-basename session key (deriving `<header timestamp>_<UUID>` for in-memory sessions), while retaining separately verifiable canonical identity provenance in owned state. Keep the UUID authoritative, reject unsafe segments and mismatches rather than selecting another scope, and use CWD checkpoint ownership to fail closed on Pi-name collisions.
|
|
21
21
|
- Migrate legacy current `state.json` losslessly into an initial anchored checkpoint with an empty tail; current snapshots are the sole semantic recovery basis, independent of the presence or validity of explanatory journals, including historical revision reads. Preserve only proven history, remove obsolete files within the successful scoped migration, and never retain two authoritative formats. Exercise migrations in temporary repositories rather than modifying the user's active Knowledge data during development.
|
|
22
|
-
- Read and write only regular non-symlink State Flow-owned files at those exact paths, publish each file by same-directory atomic rename, preserve arbitrary repository contents, and classify ownership exactly
|
|
23
|
-
- At instance initialization resolve the repository, CWD key, and session key; load each scope's anchored checkpoint and tail, materialize at the selected temporal boundary, then overlay `global → cwd → session`. Install cached view and publication basis atomically only after successful restoration/initialization; unavailable publication is an error, not a semantic no-op. Preserve the selected revision across transient restore failure so an explicit start can retry it. A genuinely new session gets an empty session layer and inherits only global/CWD values; it must never reuse another same-CWD session layer.
|
|
22
|
+
- Read and write only regular non-symlink State Flow-owned files at those exact paths, publish each file by same-directory atomic rename, preserve arbitrary repository contents, and classify ownership exactly. Git acceptance follows the complete-delta isolated-index contract below. Retain opaque source bytes for file identity and rollback, not decoded-text reconstructions. Use validated structural JSON equality for in-process semantic comparisons; reserve cryptographic hashes for compact identities that cross inference, persistence, process, or source-freshness boundaries. Serialize cooperating Git publications through a common-Git-directory publication lock; low-level file helpers require caller exclusion. Recheck prepared bases before each replacement/deletion and restore only bytes still matching the publisher's own output. Preserve detected concurrent changes and report unresolved rollback conflicts; do not claim kernel-atomic multi-file CAS against nonparticipating writers. Markdown discovery uses its independently configured source root, defaulting to `knowledge/` beneath Pi's agent directory; a storage-root override must not redirect source discovery.
|
|
23
|
+
- At instance initialization resolve the repository, CWD key, and session key; load each scope's anchored checkpoint and tail, materialize at the selected temporal boundary, then overlay `global → cwd → session`. Install cached view and publication basis atomically only after successful restoration/initialization; unavailable publication is an error, not a semantic no-op. Branch recovery may reuse one owner-bound, single-use inspection of the exact selected immutable Git revision, with a detached inspection snapshot and a freshly acquired live publication basis at installation. Revalidate mutable file cohorts, legacy snapshot fallbacks, and redirected runtime owners; never turn inspection into a long-lived revision or publication-basis cache. Preserve the selected revision across transient restore failure so an explicit start can retry it. A genuinely new session gets an empty session layer and inherits only global/CWD values; it must never reuse another same-CWD session layer. Explicit native fork adoption instead follows the [session-copy contract](docs/fork-contract.md): copy the proven source session checkpoint/tail and provenance into a distinct owner over current shared streams, without pruning shared provenance or modifying the parent. Use a fresh origin, preserve copied replay records, reject occupied live/HEAD targets, and publish/checkpoint the child only after CAS acceptance. Inherited parent pointers never authorize an empty reset; fence parent passive Stop projection across child reload.
|
|
24
24
|
- Bind every loaded scope and active Pi branch/checkpoint to the corresponding State Flow Git revision. Resume and tree restoration must recover branch-correct runtime config and semantic layers from that revision without blindly importing Git `HEAD`; reading an older revision must use object-level Git reads and never reset or check out the shared repository worktree.
|
|
25
|
-
- Every accepted semantic change, including session-only and response-only changes, atomically updates affected checkpoint/tail pairs and temporal metadata and creates one immediate local Git commit when Git is available; each commit stages the complete non-ignored worktree delta (`git add -A` semantics, including manual deletions) before overlaying exact prepared State Flow outputs, respects `.gitignore`, keeps State Flow-owned active files under compare-and-swap protection, and synchronizes the caller-visible index to the committed tree. `remotePublication` is branch runtime policy: `turn-end` queues only the newest accepted target for asynchronous non-interactive push, `off` remains local-only, and `transition` preserves synchronous legacy behavior. Queue state is operational metadata beneath the Git common directory, not semantic history;
|
|
25
|
+
- Every accepted semantic change, including session-only and response-only changes, atomically updates affected checkpoint/tail pairs and temporal metadata and creates one immediate local Git commit when Git is available; each commit stages the complete non-ignored worktree delta (`git add -A` semantics, including manual deletions) before overlaying exact prepared State Flow outputs, respects `.gitignore`, keeps State Flow-owned active files under compare-and-swap protection, and synchronizes the caller-visible index to the committed tree. `remotePublication` is branch runtime policy: `turn-end` queues only the newest accepted target for asynchronous non-interactive push, `off` remains local-only, and `transition` preserves synchronous legacy behavior. Queue state is operational metadata beneath the Git common directory, not semantic history; require exact scalar-string commit targets/confirmations without coercion, destination identity, symlink-safe atomic CAS, descendant coalescing with lineage-rewrite recovery that retargets the live commit, cross-process worker leases, restart recovery, and truthful failure diagnostics. Revalidate dead lease ownership under the existing queue writer lock before removal; fresh exclusive creation and live self-PID/token release remain independent of that gate. Preserve malformed and non-regular lease records. Remote failure never rolls back accepted state or regenerates an answer. A repository with no remote is intentionally local-only; only Git executable `ENOENT` authorizes file-only mode.
|
|
26
|
+
- Own asynchronous pushes per extension generation with a 15-second attempt budget and at most two seconds of shutdown waiting. On `session_shutdown`, cancel owned children and permanently fence old-generation queue writes and relaunch. Cancellation or elapsed waiting is not child-exit evidence: keep the lease until exit or proven spawn failure, and warn when cleanup remains unconfirmed. These deadlines govern asynchronous replication, not semantic acceptance or the compatibility synchronous mode.
|
|
26
27
|
- Register `patch_state` as the sole model-authored semantic mutation protocol. It accepts optional fixed `global`, `cwd`, and `session` semantic patches plus optional `final:true`; require at least one scope or `final:true`, reject unknown fields, `final:false`, empty supplied scopes, material no-ops, and the retired `{scope, patch}` / `unchanged` grammar. Validate every supplied scope against one causal basis and publish it all-or-nothing with one identity, temporal boundary, and durable cohort. Never accept model-authored `response`.
|
|
27
|
-
- Treat every semantic `patch_state` as a strict inference barrier. Give it sequential execution mode
|
|
28
|
+
- Treat every semantic `patch_state` as a strict inference barrier. Give it sequential execution mode; find the matching synchronized assistant through public parent traversal from the selected leaf without constructing a full branch during tool preflight; require exactly one `patch_state` call in that response and block every sibling tool call before execution. Every enabled iteration begins terminal-ineligible; only a successful call containing `final:true` latches eligibility for the next accepted `turn_end`, without stopping later reasoning, tools, or patches. If a terminal draft ends before eligibility, preserve it as the runtime-owned response at that `turn_end` and start at most two same-run fallback turns whose only purpose is the `final:true` patch; an eligible draft whose final validation fails after a later acquisition follows the same path. Fallback turns never enter `response`, and their only instruction is to apply `patch_state` with any durable changes and `final:true`, or `{final:true}` alone. A successful fallback closes resolution with the preserved answer intact; two failed fallbacks close the iteration with the preserved answer and current state plus one bounded warning and a finalization diagnostic. Failed patch calls do not consume fallback turns. A subsequent legal patch remains possible, and only an accepted ordinary answer is reconciled.
|
|
28
29
|
- Reconcile `response` only from the actually accepted ordinary assistant answer at `turn_end`; it remains runtime-owned. State Flow has no terminal HTML-comment mutation protocol and does not parse generic service comments. The first terminal draft that cannot be reconciled is preserved as the iteration response rather than discarded; fallback turns during pending resolution never reach `response`. Other extensions retain ownership of their own comments and output handling.
|
|
29
30
|
- Treat terminal state as a decision-relevant handoff, not narration: retain source-addressed reusable operational knowledge in `artifacts`; compile stable requirements, confirmed decisions, rejected approaches, and interface commitments into `contract`; retain observations, validation, failures, current domain state, unresolved work, interaction consequences, and exact continuation in `working`. Preserve relevant completed prerequisites and verified outcomes while removing obsolete progress narration; reconcile only information affected by the run and relevant existing commitments, not every scope or repository surface.
|
|
30
31
|
- Preserve active constraints, unresolved questions, consequential negative results, and the next discriminating check before compression. Distinguish observations, user requirements, assistant decisions, and hypotheses; do not promote assistant conclusions to user requirements. Retain useful source locators and validity conditions for consequential facts without mandatory per-value metadata. Keep rejection reasons and reconsideration conditions. Reconcile contradictions through evidence or user clarification instead of silently overwriting established constraints or observations; retain unresolved conflicts and decision-relevant hypotheses as uncertain. These are protocol obligations, not deterministic semantic validation gates.
|
|
@@ -38,12 +39,14 @@
|
|
|
38
39
|
- Rotate the turn-stable specification on every user-initiated run, while runtime-triggered turn-resolution continuation remains inside that same run. Keep user-controlled specification text at user authority: never interpolate it into the system prompt; repeat it only in synthetic user runtime context. Treat materialized state in that message as fallible assistant-produced data whose transport role does not elevate it into user instructions.
|
|
39
40
|
- When enabled inside an existing session, retain Pi's active context for exactly one complete bootstrap run and require its `patch_state` resolution to migrate all future-relevant context.
|
|
40
41
|
- Restore extension state from the active Pi session branch's checkpoint and recorded State Flow revision, not the full session entry list or current Git `HEAD`, on both startup and successful in-session tree navigation.
|
|
41
|
-
- Render compact status as accent `state-flow` plus dim `#<step>`. `/state-flow-status` must distinguish session config/meta from semantic temporal materialization; show the CWD and session keys, step, active temporal head and State Flow revision, available hot-history depth, per-scope artifact/tail counts, discovered global Markdown stale reasons, and pending publication; and
|
|
42
|
+
- Render compact status as accent `state-flow` plus dim `#<step>`. `/state-flow-status` must distinguish session config/meta from semantic temporal materialization; show the CWD and session keys, step, active temporal head and State Flow revision, available hot-history depth, per-scope artifact/tail counts, discovered global Markdown stale reasons, and pending publication; and render only the effective global → CWD → session materialization as semantic JSON, without reading or dumping source bodies beyond path/hash discovery needed for freshness diagnostics. Distinguish selected retained tail counts from active history depth; inherited tails may predate the origin. When temporal materialization is unavailable, report unknown counts/freshness and unavailable state rather than inventing empty projections.
|
|
42
43
|
- Explicit `/state-flow-start` creates the State Flow directory when missing and returns after locally usable runtime acceptance for normal `turn-end`/`off` policy. If Git is installed, initialize an exact-root Git repository when needed, including a populated file-only store, preserving all existing bytes and unrelated files; an ancestor repository is not a valid substitute. Skip full predecessor-format migration planning only when all exact legacy snapshot names are absent, and defer Markdown freshness discovery until before the next enabled inference. If Git is absent, use file persistence. Manual-mode startup/status/restore do not initialize Git; configured automatic start of a genuinely new session uses the same initialization as explicit start. Never create external accounts, remote repositories, credentials, or remote configuration; those remain operator-owned. Never auto-import, delete, or reset files/history in a previous Knowledge-backed store; old branch revisions require their original Git history to remain available in the selected store. `/state-flow-start` must initialize missing global, CWD, and current-session checkpoint/tail pairs plus session config/meta through compare-and-swap publication when required, enable only the current session branch, and bootstrap prior conversation when needed. Explicit start on a proven pre-runtime branch (no checkpoint or an ordinary-disabled marker) establishes an empty session origin rather than importing a later same-session layer; validate existing runtime identity, retain shared streams unchanged, and preserve later branch data in cold Git history. Ordinary new sessions remain manual unless agent-level `autoStart` is true; CWD materialization alone grants no automatic activation. Configured new sessions may initialize missing CWD state and receive distinct empty session layers while inheriting global/CWD values. Resumed and tree-selected branches restore their own config and temporal lineage, regardless of the global flag.
|
|
43
|
-
- `/state-flow-stop` must persist only the current session/branch's `config.enabled = false` and necessary runtime provenance, preserving all semantic checkpoints/tails and creating no semantic transition.
|
|
44
|
+
- `/state-flow-stop` must persist only the current session/branch's `config.enabled = false` and necessary runtime provenance, preserving all semantic checkpoints/tails and creating no semantic transition. Retain a frozen handoff plus the active user-run trajectory (including later tool results), post-stop conversation, and foreign context-bearing custom messages across same-physical-session reload/resume/tree. Exclude completed pre-run conversation and private State Flow validation feedback; idle Stop does not retain the completed run. Active restart replaces passive mode but uses that bounded boundary for its one migration run; new and forked physical sessions inherit neither projection. It does not rewrite agent-level `autoStart` or change its policy for future new sessions.
|
|
45
|
+
- After a sufficiently large accepted non-bootstrap run settles with no pending input, State Flow may request native manual compaction under a generation-private marker. Keep the final accepted assistant entry, store only revision/step identity in details, and never duplicate state in the summary. Skip prefixes containing foreign custom context. Never customize user manual or native threshold/overflow compaction, discard unfinished pre-patch work, create a State Flow origin, or rewrite Pi's append-only JSONL/tree.
|
|
44
46
|
- Keep the injected runtime protocol compact and normative; put rationale and extended explanation in README rather than the model prompt. Never parse or strip State Flow HTML comments; they are ordinary historical text, while foreign comment handling remains owned by other extensions.
|
|
45
47
|
- Do not claim strict boundedness for state, the current run trajectory, the turn specification, or the external full trace.
|
|
46
48
|
- Remain extension-agnostic in core semantics, storage, and inference: core modules never import, name, special-case, or encode policy for another extension or transport. One optional leaf presentation adapter (`lib/telegram.ts`) may import public `pi-telegram` membranes to mirror the live status on exactly one main-menu section button and expose the same start/stop affordances already owned by `state-flow-start` and `state-flow-stop`; it must fail open when the transport is absent or its registry is unready and must never alter core behavior.
|
|
47
49
|
- Activate State Flow model tools only while enabled on the selected branch; preserve every unrelated active tool when toggling them. Keep mutation confined to `patch_state` and historical observation read-only.
|
|
48
50
|
- Keep `.github/workflows/release.yml` as the sole version-tag release owner: it validates immutable tag identity, publishes through npm Trusted Publisher with provenance, verifies the public package, and only then creates the GitHub Release. Keep package, lockfile, tag, and changelog versions aligned; never add a long-lived npm token fallback.
|
|
51
|
+
- Keep opt-in performance executables and workers in top-level `benchmarks/`; their correctness regressions stay in `tests/`. Benchmark workloads may reuse synthetic test fixtures, remain excluded from runtime packaging, and retain source-bound measurement identities across relocations.
|
|
49
52
|
- Run `npm run validate` after retained code changes.
|
package/BACKLOG.md
CHANGED
|
@@ -1,6 +1,20 @@
|
|
|
1
1
|
# BACKLOG
|
|
2
2
|
|
|
3
|
-
Completed release work belongs in [CHANGELOG.md](CHANGELOG.md).
|
|
3
|
+
Completed release work belongs in [CHANGELOG.md](CHANGELOG.md).
|
|
4
|
+
|
|
5
|
+
## 0.10.0: Session performance and reliability
|
|
6
|
+
|
|
7
|
+
- **Outcome:** Make State Flow complement Pi's native session lifecycle reliably and efficiently after long histories and while independent instances publish to the same store. The operator reports slowdown after resuming a large session with roughly 200 patches; concurrency is a hypothesis, not an established cause.
|
|
8
|
+
- **Evolution:** Treat 0.10.0 as a maturation milestone: deepen reliability, usability, performance, consistency, and explanation of the existing concept. Admit adjacent maintenance only from concrete evidence; do not add features or change the product's core model.
|
|
9
|
+
- **Execution:** Prefer inline implementation and review. If subagent delegation becomes necessary, use only the operator-requested "GPT-5.6 Luna" after verifying its exact available identity; no silent substitution. The coordinator owns review and integration. Existing deterministic benchmark processes may finish without launching more model actors.
|
|
10
|
+
- **Convergence:** Reproduce and measure first, implement only evidenced corrections, validate each owned slice, then separately review the integrated candidate. Stop when the regressions below pass, before/after workload evidence explains the performance changes, and remaining host or environment limits are explicit. Preserve semantic history, source/runtime ownership, native trace, branch selection, and exact publication/CAS guarantees.
|
|
11
|
+
- **Non-goals:** No new semantic mode, state-size cap, database, hidden session replacement, bulk migration, production-store repair, read-only global bootstrap, or Telegram analyzer. Preserve ordinary abort: accepted patches remain available for same-session continuation, with no new immediate remote-enqueue requirement. Test on synthetic sessions and temporary stores; do not inspect or mutate production conversation bodies or state repositories for this work. Publication remains outside preparation scope.
|
|
12
|
+
|
|
13
|
+
### Release gate
|
|
14
|
+
|
|
15
|
+
The [nine-invocation measurement series and interpretation](docs/performance.md#measurement-closure) are complete; no measurement is active. `/tmp/state-flow-final-controls-CzaHqM/final-series-receipts.json` binds the actual exits and complete captures. Preserve this evidence and refresh only claims invalidated by an admitted correction; source/workload changes must not silently inherit old timings. Original production slowdown attribution remains unproven, and production-store inspection is not authorized.
|
|
16
|
+
|
|
17
|
+
The 0.10.0 candidate now includes native completed-history compaction and effective-only status JSON. Full 0.84.4 validation passes 426/426; affected policy/status/native lifecycle tests and typecheck pass on 0.85.1. The operator intentionally aligned package/lock versions to 0.10.0 and explicitly authorized commit, push and release. Release automation/remote CI and published artifacts remain the external acceptance gates.
|
|
4
18
|
|
|
5
19
|
## Candidate evolution
|
|
6
20
|
|
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,24 @@
|
|
|
2
2
|
|
|
3
3
|
> Each release keeps at most 8 outcome records of at most 512 characters.
|
|
4
4
|
|
|
5
|
+
## Unreleased
|
|
6
|
+
|
|
7
|
+
## 0.10.0: Native session memory and reliability
|
|
8
|
+
|
|
9
|
+
- `Git work`: Single-use restoration, exact-path tree reads and NUL-safe index batching cut steady Git calls 120→87 and resume 63→21 without caching live bases or weakening exact-byte/CAS guards. Selected state/runtime blobs now bypass Node's implicit 1 MiB output limit, preserving large-state answers and cold restoration. Legacy/file-cohort, mode/owner and rollback checks remain. Call savings do not establish universal latency gains.
|
|
10
|
+
- `Queue integrity`: Commit targets and confirmations must be scalar strings across queue creation, parsing, serialization, coalescing, confirmation and asynchronous push. Arrays and other coercible values are rejected before ancestry or push effects, while malformed files remain unchanged and valid queue/CAS behavior is preserved.
|
|
11
|
+
- `Publication`: Dead-worker reclamation rechecks validated ownership under the queue writer gate; fresh claims and live PID/token release remain independent, and malformed/foreign records survive. Async pushes have a 15-second budget. Shutdown/native replacement cancel owned children, wait at most two seconds, retain unconfirmed leases, and fence old-generation queue writes/relaunch without losing accepted state or newer queued targets.
|
|
12
|
+
- `Artifacts`: Discovery prunes only owner-confirmed missing Markdown entries, preserving external/non-Markdown artifacts and state under unavailable roots; status reports unavailable freshness and cannot consume pending removals. Every model scope rejects runtime/legacy provenance writes, including field deletion, while legacy reads, semantic edits, whole-artifact deletion, and runtime-owned compilation remain valid.
|
|
13
|
+
- `Context`: Preflight avoids branch rebuilds; projection avoids discarded-history arrays. After eligible accepted runs, native compaction keeps the final answer and full JSONL/tree while shortening active/resumed entries without another model summary; threshold/overflow remains Pi-owned and foreign context blocks shortening. Mid-run Stop retains tools/results. Status JSON renders only effective memory; scoped diagnostics and `read_state` remain.
|
|
14
|
+
- `Integrity`: Runtime-only Stop preserves newer shared provenance; cold reads pair it with the selected semantics. Separate-process witnesses cover running/resumed sessions and fatal Git writers before/after ref advancement across same/different CWDs. Accepted history, other sessions' private files and caller index bytes survive. Interrupted publication locks remain intact; new writes and restore installation fail closed without resetting selected memory.
|
|
15
|
+
- `Validation and guidance`: The final 426-test suite passes on Pi/AI/agent-core 0.84.4; affected compaction/status/native lifecycle checks and typecheck also pass on 0.85.1. Matched long-session controls retain source/resource/exit evidence and adverse timings. Read telemetry separates delivered bytes from truncation; oversized reports reject faux errors. Top-level `benchmarks/` owns opt-in workloads; indexed guides document limits.
|
|
16
|
+
- `Fork`: Native same-CWD fork replacement copies the selected session checkpoint, retained tail and provenance into a distinct owner over unchanged current shared layers. The child starts its own origin/checkpoint; parent data/history and CAS remain intact. Owned reload/resume and independent writes are covered. Inherited parent pointers cannot fall through to a reset marker, and parent Stop projection cannot resurface after child reload. Missing/unsafe source evidence fails closed.
|
|
17
|
+
|
|
18
|
+
## 0.9.6: Monospaced state value in the submenu
|
|
19
|
+
|
|
20
|
+
- `Interop`: The submenu help line now consists solely of the operator-specified sentence "Records the latest accepted state after every turn, so a new session resumes from the last committed point."; the trailing "Start it to keep memory live on this branch, stop it to pause." sentence was removed.
|
|
21
|
+
- `Interop`: The submenu state line renders the live value in monospace — `off` or `#N` — while the section button keeps the same plain-text label.
|
|
22
|
+
|
|
5
23
|
## 0.9.5: Shorter submenu help line
|
|
6
24
|
|
|
7
25
|
- `Interop`: The submenu help line now opens directly with "Records the latest accepted state after every turn…"; the "Durable memory for this conversation: State Flow" prefix was removed at the operator's request.
|
package/README.md
CHANGED
|
@@ -1,303 +1,94 @@
|
|
|
1
1
|
# Pi State Flow
|
|
2
2
|
|
|
3
|
-
> Inspired by [SKILL.state](https://arxiv.org/html/2608.26263v2)
|
|
4
|
-
|
|
5
3
|

|
|
6
4
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
```text
|
|
10
|
-
state[0] = now
|
|
11
|
-
state[1] = one accepted transition ago
|
|
12
|
-
...
|
|
13
|
-
state[7] = seven accepted transitions ago
|
|
14
|
-
|
|
15
|
-
state.global[n] state.cwd[n] state.session[n]
|
|
16
|
-
projections at the SAME temporal boundary
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
`patch_state` explicitly advances that stream. Optional Git preserves complete older history for Git-backed transitions; without Git, files retain current state and its proven hot window. State Flow compiles decision-relevant reality; it neither owns Knowledge semantics nor replaces Pi's native tool loop or inspectable session trace.
|
|
20
|
-
|
|
21
|
-
## Installation
|
|
22
|
-
|
|
23
|
-
Requirements: Pi `0.84.4–0.84.x` and Node.js `22.19.0` or newer. Git is optional; when installed, it needs a configured commit identity. A remote is optional and operator-owned.
|
|
24
|
-
|
|
25
|
-
```bash
|
|
26
|
-
pi install npm:@llblab/pi-state-flow
|
|
27
|
-
# Or:
|
|
28
|
-
pi install git:github.com/llblab/pi-state-flow
|
|
29
|
-
```
|
|
5
|
+
**Working memory for Pi, carried with the session.**
|
|
30
6
|
|
|
31
|
-
|
|
7
|
+
Keep decisions, constraints, verified findings, and next steps without sending every completed tool exchange back to the model. State Flow lets the agent maintain explicit state; Pi still owns the conversation, the native tool loop, and the complete inspectable trace.
|
|
32
8
|
|
|
33
|
-
|
|
9
|
+
> Inspired by [SKILL.state](https://arxiv.org/html/2608.26263v2).
|
|
34
10
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
```json
|
|
38
|
-
{
|
|
39
|
-
"directory": "~/.pi/agent/state-flow",
|
|
40
|
-
"autoStart": false,
|
|
41
|
-
"logging": false,
|
|
42
|
-
"remotePublication": "turn-end"
|
|
43
|
-
}
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
- `directory`: State storage location; omitted defaults to `state-flow/` beneath Pi's agent directory. Absolute paths, `~`/`~/` and relative paths are supported; relative paths resolve from the configuration directory, not the project CWD.
|
|
47
|
-
- `autoStart`: Set to `true` to enable State Flow automatically for genuinely new sessions, including fresh CWDs. Omitted defaults to `false`: manual activation, even when previous CWD materialization exists.
|
|
48
|
-
- `logging`: Set to `true` to append JSONL diagnostics for rejected `patch_state` calls and unresolved terminal drafts to `tmp/state-flow/logs.jsonl` beneath Pi's agent directory; omitted or `false` writes nothing. Rejected calls retain their exact attempted arguments, the precise error, and, when available, the tool identity, call id, resolution attempt, and terminal-eligibility state; successful patches are never logged. Exact text blocks are retained only when useful, while reasoning bodies are excluded. The file stays local unless you move it, deleting or rotating it is operator-owned, and logging never enters semantic state, metadata, checkpoints, or the state repository. If a custom state repository contains the diagnostic path, capture fails closed rather than committing the log. A write failure changes nothing except one local warning.
|
|
49
|
-
- `remotePublication`: For new runtimes, `turn-end` (default) queues one newest target after local acceptance and pushes asynchronously; `off` keeps commits local; `transition` preserves legacy synchronous push behavior. A resumed branch keeps its persisted policy.
|
|
50
|
-
|
|
51
|
-
Configuration is read at extension load; edit it and use `/reload` (or restart Pi) before opening a new session. Existing branches retain their stored enablement on resume/tree navigation. `/state-flow-stop` does not rewrite this file or disable automatic mode for later new sessions. Invalid JSON, unknown keys or invalid values fail extension loading rather than silently choosing another directory. A missing file uses defaults and is not generated automatically.
|
|
52
|
-
|
|
53
|
-
The configuration file stays in Pi's agent directory when state storage moves. A directory override selects a store; it does not migrate old data or Git objects. SDK embedders can set `StateFlowExtensionOptions.agentDir` for the profile, `repositoryRoot` to override the configured state directory, and `knowledgeRoot` independently for Markdown sources.
|
|
54
|
-
|
|
55
|
-
## Usage and activation
|
|
11
|
+
## The idea
|
|
56
12
|
|
|
57
13
|
```text
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
14
|
+
Current state + New request
|
|
15
|
+
↓
|
|
16
|
+
Pi's native tool loop
|
|
17
|
+
↓
|
|
18
|
+
Updated state + Answer
|
|
19
|
+
↓
|
|
20
|
+
Next request
|
|
61
21
|
```
|
|
62
22
|
|
|
63
|
-
|
|
23
|
+
During a request, the model retains the current tool trajectory. Between requests, State Flow replaces completed ordinary conversation history in model context with the current state and recent transitions. It does not delete Pi's session history.
|
|
64
24
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
The compact status is an accent `state-flow` and dim `#<step>`. Detailed status distinguishes runtime config/meta from semantic materialization, reports scope keys, the selected temporal head and runtime revision, available hot offsets, per-scope retained patch tails, artifact counts, stale Markdown reasons, and pending publication. Retained tails may include inherited pre-origin records, so their counts are not active history depth. It labels global/CWD/session/effective state without dumping source bodies. Failed inspection means unavailable evidence, never a falsely clean count.
|
|
68
|
-
|
|
69
|
-
When the optional `pi-telegram` extension is installed, State Flow appears exactly once in Telegram: the inline main menu places a `🌀 State Flow: #N` or `🌀 State Flow: off` section button directly before Settings, and that button's label carries the live status. The button opens a compact submenu whose header repeats the same state line, followed by a short explanation of what State Flow is; the only action is `Start` (or `Stop` while enabled), with the transport-owned `⬆️ Main menu` row above it. Starting while a run is active waits for that run to settle; stopping applies immediately. Registration fails open when the transport is absent, retries when the section registry becomes ready, and disposes on session shutdown; the `/state-flow-*` commands remain the authority and nothing changes for hosts without `pi-telegram`.
|
|
70
|
-
|
|
71
|
-
## State, scope, and time
|
|
72
|
-
|
|
73
|
-
Every scope has exactly the same semantic shape:
|
|
74
|
-
|
|
75
|
-
```json
|
|
76
|
-
{
|
|
77
|
-
"artifacts": {},
|
|
78
|
-
"contract": {},
|
|
79
|
-
"working": {},
|
|
80
|
-
"response": "Latest complete user-facing answer"
|
|
81
|
-
}
|
|
82
|
-
```
|
|
25
|
+
The agent curates what matters; the extension validates, persists, and projects the accepted state. This is working memory, not automatic proof that a remembered fact is true.
|
|
83
26
|
|
|
84
|
-
|
|
85
|
-
- `contract`: Stable requirements, decisions, rejected approaches, and interface commitments.
|
|
86
|
-
- `working`: Verified observations, validation, failures, unresolved questions, and exact continuation.
|
|
87
|
-
- `response`: The complete answer captured by the runtime, normally in session state; models cannot patch it directly.
|
|
27
|
+
## Quick start
|
|
88
28
|
|
|
89
|
-
|
|
29
|
+
Requires Pi `0.84.4–0.84.x` or `0.85.1–0.85.x` and Node.js `22.19.0` or newer. See the [SDK compatibility matrix](docs/compatibility.md) for exact tested stacks. Git is optional; using it requires a configured commit identity.
|
|
90
30
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
For example:
|
|
94
|
-
|
|
95
|
-
```text
|
|
96
|
-
Boundary T181 T182 T183 T184
|
|
97
|
-
Global G G G' G'
|
|
98
|
-
CWD C C' C' C''
|
|
99
|
-
Session S S' S'' S'''
|
|
100
|
-
|
|
101
|
-
At T184:
|
|
102
|
-
state[1] = overlay(G', C', S'')
|
|
103
|
-
state.cwd[1] = C'
|
|
104
|
-
state.cwd[2] = C'
|
|
105
|
-
state.global[1] = G'
|
|
106
|
-
```
|
|
107
|
-
|
|
108
|
-
`state.cwd[1]` is not the preceding CWD patch. Every projection first resolves the same target in the active lineage. Offsets zero through seven are the hot range; offsets before a new or migrated lineage's proven origin are explicitly unavailable until enough transitions exist. Offset eight is outside this read contract even when Git contains older history.
|
|
109
|
-
|
|
110
|
-
### Lazy historical reads
|
|
111
|
-
|
|
112
|
-
Normal inference receives current effective `state[0]`, useful compact transition context, the current specification, and the complete current-run trajectory. It does not receive eight full snapshots. History is materialized only on a requested effective or scope read.
|
|
113
|
-
|
|
114
|
-
The pure accessor is `readTemporalState(view, offset, scope?)` in `lib/temporal.ts`, with scope omitted for effective state. The live adapter exposes the same lazy read semantics to embedders through `StateFlowExtensionOptions.onRuntime`; the callback receives a cached-runtime `read(offset?, scope?)` accessor. The model tool `read_state` accepts optional `offset` (integer 0–7, default 0) and `scope` (`effective`, `global`, `cwd`, or `session`, default `effective`). For example:
|
|
115
|
-
|
|
116
|
-
```json
|
|
117
|
-
{"offset": 1, "scope": "cwd"}
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
It returns one `{offset, scope, boundary, state}` result from cached runtime, with no publication, checkpoint append, or history step. Unavailable pre-origin history is an error, not an empty state. Use it for a concrete historical or scope-specific gap, not routine rereading of current context. Both State Flow tools are active only while the selected branch is enabled; `patch_state` remains the sole mutator and blocks `read_state` siblings at its inference barrier. SDK embedders using an explicit tool allowlist must include `read_state` as well as `patch_state`. The model reasons about state, scope, time, and transitions, not checkpoint folding mechanics.
|
|
121
|
-
|
|
122
|
-
## Temporal storage
|
|
123
|
-
|
|
124
|
-
State Flow owns a separate state directory at `~/.pi/agent/state-flow/`, optionally backed by Git. Knowledge remains an independent Markdown source at `~/.pi/agent/knowledge/`. Both defaults follow Pi's configured agent directory (`PI_CODING_AGENT_DIR`); [configuration](#configuration) can select a different state store. Changing the storage root does not redirect Markdown discovery.
|
|
125
|
-
|
|
126
|
-
```text
|
|
127
|
-
state-flow/
|
|
128
|
-
├── .git/ # When Git-backed
|
|
129
|
-
├── checkpoint.json
|
|
130
|
-
├── patches.jsonl
|
|
131
|
-
├── meta.json
|
|
132
|
-
└── <cwd-key>/
|
|
133
|
-
├── checkpoint.json
|
|
134
|
-
├── patches.jsonl
|
|
135
|
-
├── meta.json
|
|
136
|
-
└── <session-key>/
|
|
137
|
-
├── checkpoint.json
|
|
138
|
-
├── patches.jsonl
|
|
139
|
-
├── config.json
|
|
140
|
-
└── meta.json
|
|
141
|
-
```
|
|
142
|
-
|
|
143
|
-
Directory names deliberately mirror Pi's native session layout. For example, `/home/llb/Repos/deos` is stored as `--home-llb-Repos-deos--`, and `2026-09-07T20-01-08-993Z_<uuid>.jsonl` uses `2026-09-07T20-01-08-993Z_<uuid>` as its session directory. In-memory sessions derive that same `<timestamp>_<uuid>` form from Pi's header. The UUID remains the authoritative runtime identity; session metadata and CWD checkpoint ownership reject foreign identities and CWD-name collisions without hiding another digest in the path.
|
|
144
|
-
|
|
145
|
-
Explicit `/state-flow-start` creates a missing state directory and returns after establishing locally usable state. With Git available, start initializes an exact-root Git repository when needed, including a populated state directory, without deleting existing files; unrelated worktree content follows the complete-delta Git commit rule below. Normal `turn-end` and `off` activation never waits for remote push, canonical stores skip predecessor-format planning, and Markdown freshness discovery is deferred until the next enabled inference. An active file-backed branch must identify its exact current cohort before it can be adopted into Git. If the Git executable is absent, state persists directly to files without repository operations. Git command failures, corruption, or permission errors are not treated as executable absence. Manual-mode startup, status, and branch restoration do not initialize Git; configured automatic start for a new session may do so. The extension creates no GitHub account, external repository, or remote configuration, and does not invent a Git commit identity.
|
|
146
|
-
|
|
147
|
-
For local debugging after Git-backed initialization:
|
|
31
|
+
From npm:
|
|
148
32
|
|
|
149
33
|
```bash
|
|
150
|
-
|
|
151
|
-
git -C ~/.pi/agent/state-flow log --oneline -10
|
|
152
|
-
git -C ~/.pi/agent/state-flow show <revision>:checkpoint.json
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
`/state-flow-status` identifies the active scope keys and selected revision. Live files may belong to a later branch; use that selected revision for historical inspection rather than assuming the worktree represents the current Pi branch. State may include private session content; review it before configuring any remote.
|
|
156
|
-
|
|
157
|
-
All three semantic scopes use one storage model:
|
|
158
|
-
|
|
159
|
-
```text
|
|
160
|
-
current scope = materialize(anchored checkpoint, ordered patch tail)
|
|
161
|
-
current effective = overlay(global current, CWD current, session current)
|
|
34
|
+
pi install npm:@llblab/pi-state-flow
|
|
162
35
|
```
|
|
163
36
|
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
On an eighth scope patch, apply the oldest retained patch into the checkpoint, advance `through`, remove that patch, and append the new one. This deterministically preserves current state and the seven-transition hot window. Sparse scopes may keep older checkpoints; their values are still reconstructed at the same requested effective boundary. Never replay a tail over an already-current snapshot or silently drop an unapplied patch. The implemented `serializeScopeStream`/`parseScopeStream` codec emits deterministic checkpoint/JSONL bytes and validates complete replay input, including semantic results, before acceptance. `publishTemporalStateToGit` writes selected streams from one validated view through exact-output Git publication, rejects omitted changes and stale/wrong-session bases, and leaves unchanged scopes alone. `loadScopeStream` reads the live pair; `loadTemporalRevision` reads regular-file Git objects without moving the worktree. Neither invents missing runtime lineage; callers must supply the lineage belonging to the same revision. Oversized tails, invalid lineage, incomplete pairs, and no-op records are rejected; legacy current-state documents require explicit migration rather than implicit decoding.
|
|
167
|
-
|
|
168
|
-
Accepted replay cohorts contain an identity and exact scope patches, without an independent timestamp clock or current-state publication DTO. Staging validates both semantic hashes and the active causal-boundary identity. Compact transition context derives directly from the selected lineage; the per-scope projection budget is the fixed shared seven-transition window and never changes the shared temporal target of `state[n]`. Legacy explanatory journals and Pi `recentTransitions` payloads are not semantic recovery inputs.
|
|
169
|
-
|
|
170
|
-
Live files, or their branch-selected cached representation, serve hot materialization. When available, Git serves complete cold history for committed transitions, durable provenance, explicit historical inspection, and branch restoration; it is not queried to rebuild current state on every inference. There is no second unbounded event store, database, `.state-flow`, `scopes`, or parallel history directory.
|
|
171
|
-
|
|
172
|
-
### Runtime ownership and Git publication
|
|
173
|
-
|
|
174
|
-
Session `config.json` owns runtime behavior such as `enabled`. Session `meta.json` owns temporal head/lineage, counters, durable base, Pi branch correlation, pending publication, and migration/version metadata. Global, CWD, and session scope `meta.json` files also own runtime artifact provenance (`sourceHash`, `compilerRevision`, `compiledAt`); the session file carries the session scope's provenance beside its runtime record. They are not model-patchable state and do not overlay into effective state. Pi checkpoints bind the selected branch to the corresponding durable runtime/temporal revision. The implemented runtime codec records version, canonical CWD/session identity, bounded active lineage, counters and run metadata, `revision: "self"`, and an unconfirmed publication intent. The self reference resolves to the last commit owning either member of the config/meta pair, so config-only changes advance runtime revision without semantic history, while unrelated commits in the state repository do not change its meaning. The commit hash is supplied during resolution rather than embedded in its own content; the resulting publication target identifies the existing commit to reconcile or retry. Unconfirmed means no durable confirmation is available, not proof that the push failed.
|
|
175
|
-
|
|
176
|
-
Git-backed Pi checkpoint data is exactly `{"revision":"<full commit hash>"}`; file-only checkpoints use `{"revision":"file:<64 lowercase hex>"}` for an exact current-cohort reference. `{"disabled":true}` denotes an ordinary disabled branch without durable runtime. Configuration, counters, specification, lineage, and semantic state are not copied into these entries. Older config/meta and state-bearing checkpoints remain read-only compatibility input. Invalid immutable pointer targets fall back through the selected branch; transient publication-lock failure instead retains the selected revision for retry. Explicit start on a pre-runtime branch establishes an empty session origin, even if a later branch already created same-session files. Shared streams remain unchanged and the later session state remains recoverable from its Git revision; it is not imported into the new origin. Legacy checkpoint syntax and immutable targets undergo the same fallback checks, while an unanchored legacy semantic payload cannot be silently reduced to an ordinary-disabled marker. Stop on an unproven branch fails visibly instead of turning failed recovery into permission to replace existing runtime; select a valid checkpoint before retrying.
|
|
177
|
-
|
|
178
|
-
Every materially effective semantic transition, including response-only and session-only changes, immediately publishes affected checkpoint/tail pairs and necessary lineage metadata. In Git mode each accepted cohort gets its own consistent local commit; `turn-end` queues the newest target for asynchronous replication, `off` stays local, and compatibility `transition` attempts the push synchronously. A reset or re-initialized journal rewrites the lineage; the queue then retargets the live commit and keeps the retired one in local history instead of blocking later turns. Intermediate patches are not batched until turn end. With no remote configured, Git persistence is intentionally local-only, not a pending publication error. With Git absent, persistence remains file-only and does not report fake commits or pending pushes. One model-authored call may update global, CWD, and session together; every supplied scope shares one validation basis, transition identity, temporal boundary, and durable commit. Config-only changes may persist runtime state but never invent semantic transitions.
|
|
179
|
-
|
|
180
|
-
Writes use regular non-symlink owned files, same-directory atomic rename, isolated Git indexes, and compare-and-swap against the reconciled base. Cooperating State Flow Git publishers hold a common-Git-directory publication lock through capture, commit, and rollback. Raw bytes own file identity and recovery; non-UTF-8 explanatory journals are not reconstructed from decoded strings. Per-file publication checks catch changed bases. Both semantic and migration publishers retain exact prepared-output receipts: Git commits those bytes, not later worktree contents, and rollback requires the receipt to preserve detected external changes rather than replacing them. Low-level file helpers require caller exclusion; this is not kernel-atomic multi-file CAS against writers ignoring the protocol. An existing publication lock fails before writes and is not silently stolen; reconcile its active or interrupted owner before retrying. Scope keys mirror Pi's CWD/session directory names and retain separately verified canonical identity provenance inside owned state. Conflicts fail rather than silently selecting another scope or auto-merging semantic state. Every Git commit stages the complete non-ignored worktree delta — tracked edits, new files, and manual deletions, respecting `.gitignore` — then overlays the exact prepared State Flow outputs and synchronizes the caller-visible index to the committed tree, so no artificial staged or unstaged entries remain; unrelated user worktree changes may therefore be committed while ignored files stay untouched. With distinct roots, the Knowledge repository's files, index, HEAD, locks, and remote are not used for state publication; keep custom stores separate from sources to retain this isolation.
|
|
181
|
-
|
|
182
|
-
A local commit accepts the transition. Push failure is pending publication, not a reason to regenerate the answer or repeat semantic state changes. Pending state survives restart; retries push the existing accepted commit. A restored branch reads its recorded revision without moving the shared worktree. On publication it adopts current proven live global/CWD streams it does not modify at a fresh proven origin while preserving its selected session layer, and fails closed with a named target-scope conflict if it would overwrite a shared scope that advanced. Adoption fabricates no parent ancestry and leaves older lineage available through Git.
|
|
183
|
-
|
|
184
|
-
### File-only recovery
|
|
185
|
-
|
|
186
|
-
File-only mode keeps the same scoped materialization, hot temporal window, and separate config/meta. Its Pi checkpoint identifies the store root and exact current file cohort, including runtime metadata, not a Git commit. Restart can restore that pointer while the complete referenced cohort remains available. Old branches or replaced cohorts can be unavailable; current files must not be passed off as the selected past. Unavailable references and publication locks retain the selected pointer rather than falling through to an older disabled marker. A crash between file publication and Pi checkpoint append can leave the previous pointer unavailable: the files remain, but automatic branch restoration cannot assert they belong to that pointer. There is no second unbounded history store, and Git-linked cold recovery still requires Git and its original objects.
|
|
37
|
+
From git:
|
|
187
38
|
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
### Existing stores and migration
|
|
191
|
-
|
|
192
|
-
Changing the default directory does not import, move, or delete an old Knowledge-backed store. Existing Pi checkpoints name Git revisions: copying only the current checkpoint/tail files cannot preserve their recovery contract. An old session requires its original revision history in the selected repository; unavailable revisions must not be treated as permission to reset the session. Keep the old store intact until an explicit history-preserving relocation is performed, or use a genuinely new Pi session for an independent debug store. An SDK host can explicitly select the old `repositoryRoot` when accessing that history.
|
|
193
|
-
|
|
194
|
-
The following is an **in-store format migration**, not a cross-repository relocation. Let State Flow perform it; manually renaming files is not a valid conversion.
|
|
195
|
-
|
|
196
|
-
An existing current `state.json` becomes the initial checkpoint state, anchored at a proven current transition/base, with an empty tail. Its explanatory old journal is not replayed. Migration must preserve exact semantic state and remove obsolete `state.json` ownership only after successful publication, leaving one authoritative format. It must not fabricate seven historical patches. Git retains older evidence; the new hot window grows from the proven migration origin. The explicit internal migration primitive now plans without writes, publishes all converted scope files in one isolated-index commit, and restores exact old bytes after failure when its published files have not concurrently changed; conflicting external bytes remain preserved with an explicit recovery error. It rejects ambiguous dual snapshots and orphaned tails, does not depend on explanatory legacy journal validity, and retains legacy interpretation only for explicit migration. Production no longer exposes the predecessor current-state/journal writers; legacy fixture writers are test-only. Pi initialization now invokes this migration when establishing the temporal runtime, and revision-linked predecessor snapshots have a one-way restoration path. The untagged pre-0.4 development layout that appended SHA-256 suffixes remains readable at its historical Git revisions; if the selected layout is current `HEAD`, startup adopts its CWD/session files into Pi-native paths with compare-and-swap publication, rollback, and an immediate replacement revision pointer. Historical legacy restoration reads only the authoritative current snapshots, not explanatory journals. Cached view and publication basis are installed together after successful initialization/restoration; a transient restore failure preserves the selected revision for explicit start retry and cannot masquerade as a successful state transition. Independent rereview confirmed these three failure-path fixes. Stop also remains harmless in ordinary disabled sessions and retries a failed branch restoration before durably disabling that branch.
|
|
197
|
-
|
|
198
|
-
## Unified semantic mutation and terminal resolution
|
|
199
|
-
|
|
200
|
-
Use `patch_state` only for established future-relevant state whose delayed persistence risks meaningful loss, never scratchpad, narration, routine progress, or speculative churn:
|
|
201
|
-
|
|
202
|
-
```json
|
|
203
|
-
{"session":{"working":{"verified":"result"}},"final":true}
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
```text
|
|
207
|
-
LLM(state[0] = Sn)
|
|
208
|
-
→ patch_state
|
|
209
|
-
→ validate, publish, materialize Sn+1
|
|
210
|
-
→ next LLM(state[0] = Sn+1, state[1] = Sn)
|
|
211
|
-
→ choose the next action
|
|
39
|
+
```bash
|
|
40
|
+
pi install git:github.com/llblab/pi-state-flow
|
|
212
41
|
```
|
|
213
42
|
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
`patch_state` is the only model-authored semantic mutation mechanism. Each invocation may supply any combination of fixed `global`, `cwd`, and `session` patches plus optional `final:true`. Supplied scopes are validated and published atomically; at least one scope or `final:true` is required. The retired `{scope, patch}` and `unchanged` forms are rejected.
|
|
43
|
+
In Pi:
|
|
217
44
|
|
|
218
45
|
```text
|
|
219
|
-
|
|
220
|
-
patch_state({...?, final:true}) → atomic accepted state + terminal eligibility → later turn_end
|
|
221
|
-
patch_state({final:true}) → terminal eligibility only, with no semantic transition
|
|
222
|
-
```
|
|
223
|
-
|
|
224
|
-
Every enabled iteration begins terminal-ineligible. `final:true` latches eligibility for the next accepted `turn_end`; it does not terminate the turn, stop tools or reasoning, freeze state, or prevent later `patch_state` calls. A terminal draft that ends before eligibility, or an eligible draft whose final validation fails after a later acquisition, is preserved: it is reconciled as the runtime-owned `response` at that `turn_end` instead of being discarded, and State Flow starts at most two same-run fallback turns whose only purpose is to obtain the `final:true` patch. Fallback turns are never the response. A successful `final:true` during a fallback commits its patches and closes resolution with the preserved answer intact; when both fallbacks pass without it, the iteration closes with the preserved answer and current state, plus one bounded warning and a finalization diagnostic. Failed patch calls do not consume fallback turns. Only an accepted answer reaches `response`; aborted, length-limited, and provider-error endings never do.
|
|
225
|
-
|
|
226
|
-
A semantic patch remains an inference barrier. The complete scope cohort is validated, durably accepted, and rematerialized before further reasoning. Empty supplied scopes and materially no-op scope cohorts are rejected; omit unchanged scopes and use `{final:true}` alone when no semantic update is needed. `response` is never accepted from the model. The final answer is ordinary assistant output, not a state transport. State Flow has no terminal `state_flow` HTML-comment protocol and does not parse generic service comments. Historical `state_flow` text is ordinary text. Other extensions retain ownership of their own comments such as `telegram_button`. Never put secrets in state.
|
|
227
|
-
|
|
228
|
-
## Artifact compilation and acquisition
|
|
229
|
-
|
|
230
|
-
Artifacts use exact source paths as keys. A model-visible entry requires a non-empty description:
|
|
231
|
-
|
|
232
|
-
```json
|
|
233
|
-
{"description":"What this source contains and when it is useful","compilation":{}}
|
|
234
|
-
```
|
|
235
|
-
|
|
236
|
-
Optional `kind`, `compilation`, and `tags` are validated when present, while arbitrary unknown finite non-null JSON metadata remains forward-compatible. Tags are unique non-empty trimmed strings; `selectArtifactsByTags` provides deterministic all/any candidate filtering, but a tag match never authorizes source acquisition. Ordinary artifacts need no `kind` or tags.
|
|
237
|
-
|
|
238
|
-
Runtime-owned freshness evidence is retained separately per scope in `meta.json`:
|
|
239
|
-
|
|
240
|
-
```json
|
|
241
|
-
{"version":1,"artifacts":{"/path/to/source.md":{"sourceHash":"sha256:<64 lowercase hex characters>","compilerRevision":"artifact-v1","compiledAt":"optional timestamp"}}}
|
|
46
|
+
/state-flow-start
|
|
242
47
|
```
|
|
243
48
|
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
Before the first enabled inference, State Flow generically discovers regular lowercase `*.md` beneath the independent canonical Knowledge source root, normally `~/.pi/agent/knowledge` (or `knowledge/` beneath the agent directory selected by `PI_CODING_AGENT_DIR`). Discovery is deferred from activation's critical path but completes before runtime context is built. This source directory need not be a Git repository. Discovery hashes opaque bytes and retains byte counts without decoding or storing bodies; it skips symlinks and never escapes the root. A missing root means no candidates. Root and nested Markdown are treated alike, without reserved names, frontmatter parsing, validators, templates, or a `save_knowledge` implementation/call. Arbitrary repository files remain independent from State Flow.
|
|
49
|
+
Continue working normally. The agent receives the state protocol and uses `patch_state` to maintain memory.
|
|
247
50
|
|
|
248
|
-
|
|
51
|
+
- `/state-flow-status`: Inspect the selected state, history, artifact freshness, and publication status.
|
|
52
|
+
- `/state-flow-stop`: Disable updates on this branch without deleting retained state.
|
|
249
53
|
|
|
250
|
-
|
|
54
|
+
State Flow is **opt-in**. Starting in an existing conversation keeps its active context for one migration run. To enable genuinely new sessions automatically, set `"autoStart": true` in the optional [configuration](docs/usage.md#configuration).
|
|
251
55
|
|
|
252
|
-
|
|
56
|
+
With `pi-telegram` installed, its main menu also exposes the same Start/Stop controls.
|
|
253
57
|
|
|
254
|
-
|
|
58
|
+
## What carries forward
|
|
255
59
|
|
|
256
|
-
|
|
60
|
+
The same four fields exist at every scope:
|
|
257
61
|
|
|
258
|
-
|
|
62
|
+
- `contract`: Requirements, decisions, constraints, and interface commitments.
|
|
63
|
+
- `working`: Observations, results, unresolved questions, and what to do next.
|
|
64
|
+
- `artifacts`: Source-addressed descriptions and reusable compiled knowledge.
|
|
65
|
+
- `response`: The latest complete answer, captured by the runtime.
|
|
259
66
|
|
|
260
|
-
|
|
67
|
+
Memory overlays **global → project CWD → session**. Put reusable cross-project knowledge in global, project knowledge in CWD, and private task continuation in session.
|
|
261
68
|
|
|
262
|
-
|
|
69
|
+
**Resuming an existing Pi session restores its selected State Flow state and enablement.** A genuinely new session starts with an empty session layer and inherits only shared global/CWD memory; it does not resume another session's private work. Tree navigation follows the selected branch, not whichever state happens to be at Git `HEAD`.
|
|
263
70
|
|
|
264
|
-
|
|
71
|
+
The agent can inspect `state[0]` through `state[7]`: now and up to seven prior accepted transitions. Every scope is read at the same causal boundary. Git-backed stores retain older committed history separately.
|
|
265
72
|
|
|
266
|
-
|
|
73
|
+
## Boundaries worth knowing
|
|
267
74
|
|
|
268
|
-
|
|
75
|
+
- `Not a second agent loop`: State Flow adds memory to Pi; it does not run background reasoning or replace session controls.
|
|
76
|
+
- `Not a transcript archive in the prompt`: Prefer ordinary Pi when each request needs all historical exchanges verbatim.
|
|
77
|
+
- `Not live workspace truth`: Remembered observations can become stale; revalidate consequential facts before acting.
|
|
78
|
+
- `Physical forks copy private memory`: Native fork replacement copies the selected session checkpoint/tail into a new owner while retaining current shared memory. The child starts its own history; older parent checkpoints are not child history. See [fork support and limits](docs/usage.md#fork-support-and-limits).
|
|
79
|
+
- `Not a token or latency guarantee`: State and the current trajectory are not size-capped. Benefits depend on workload and memory quality; see [performance evidence](docs/performance.md).
|
|
80
|
+
- `Use a dedicated store`: By default state lives in `~/.pi/agent/state-flow/`, separately from Knowledge Markdown. Git commits include the store's complete non-ignored worktree delta. Do not point it at an unrelated working repository.
|
|
81
|
+
- `Treat memory as private`: State and diagnostic logs may contain session content. Keep secrets out, and review data before configuring a remote. Removing a value does not erase Git history or other copies.
|
|
269
82
|
|
|
270
|
-
|
|
83
|
+
Pi packages run with your user permissions. Without Git, current state and its proven hot history persist to files, but arbitrary older branches may be unavailable. See [storage and recovery](docs/usage.md#storage-and-recovery) before moving stores.
|
|
271
84
|
|
|
272
|
-
|
|
85
|
+
## Read more
|
|
273
86
|
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
## Context lifecycle and boundaries
|
|
279
|
-
|
|
280
|
-
The current run retains its user prompt, assistant/tool trajectory, and persistent/current-run context-bearing custom messages from other extensions. Private State Flow resolution instructions are excluded from retained projections. Completed-run trajectories leave model context only at user-run boundaries; Pi's full trace remains inspectable. Mid-conversation bootstrap retains prior active context for one migration run; restart after stop instead bootstraps from the frozen exit handoff plus post-stop messages, never the earlier raw transcript.
|
|
281
|
-
|
|
282
|
-
The normative runtime protocol stays stable through the tool loop and runtime-triggered resolution continuation. The current specification rotates on each user-initiated run and stays user-authority text, never interpolated into the system prompt. Synthetic user context repeats it with fallible assistant-produced state; the transport role does not elevate that state into instructions. Image-only specifications may be empty.
|
|
283
|
-
|
|
284
|
-
State, the active specification, current trajectory, and external full trace have no strict size bound. There are no project schemas, growth-pressure gates, action authorization/observation envelopes, state-byte caps, or total-model-context guarantees. This mode is a poor fit when every new request must reason over complete historical trajectories. Token/cache/latency/success advantages still require controlled benchmarks.
|
|
285
|
-
|
|
286
|
-
## Architecture and validation
|
|
287
|
-
|
|
288
|
-
See the [documentation index](docs/README.md) and [twenty temporal acceptance properties](docs/temporal-acceptance.md) for requirement-to-test maps and verification limits.
|
|
289
|
-
|
|
290
|
-
`index.ts` remains composition/public exports only. Flat independent `lib/` domains own their responsibilities: `temporal` owns anchored streams, folding, active-lineage validation, and lazy historical algebra; `durable` owns exact files; `git` owns revision/CAS publication; `history` owns compact transition records; `state` owns semantic shape/overlay; `transition` coordinates accepted patches; `snapshot`, `session`, `recovery`, and `episode` own runtime/branch lifecycle; `context`, `status`, and `terminal` own projection and finalization; artifact domains own discovery/acquisition/compilation/maintenance; `extension` composes Pi integration. `runtime` caches branch-selected streams and their publication basis beneath the Pi adapter; `config` owns read-only agent-level settings independently from branch runtime configuration. Predecessor current-state/journal writers are test-only; production retains explicit migration readers, and Pi checkpoints carry revision pointers.
|
|
291
|
-
|
|
292
|
-
Each domain has a same-named test, with cross-domain DAG/composition invariants in `tests/invariants.test.ts`. Real Pi SDK scenarios use `tests/pi-harness.ts`; ordinary fixtures use `tests/harness.ts`.
|
|
293
|
-
|
|
294
|
-
```bash
|
|
295
|
-
npm install
|
|
296
|
-
npm run validate
|
|
297
|
-
```
|
|
87
|
+
- [Usage and recovery](docs/usage.md): Configuration, session behavior, diagnostics, privacy, and storage recovery.
|
|
88
|
+
- [Architecture](docs/architecture.md): Semantic state, temporal history, barriers, artifacts, and integration contracts.
|
|
89
|
+
- [Performance](docs/performance.md): Reproducible workloads, measured costs, and what the measurements do not prove.
|
|
90
|
+
- [Documentation index](docs/README.md): All maintained guides, including the temporal acceptance map.
|
|
298
91
|
|
|
299
|
-
|
|
92
|
+
For development, run `npm install` and `npm run validate`. `npm run benchmark` runs the opt-in synthetic workload in `benchmarks/`, separate from the normal test suite. In a source checkout, see `benchmarks/README.md` for lifecycle/publisher selection.
|
|
300
93
|
|
|
301
|
-
|
|
302
|
-
- [Delivery history](CHANGELOG.md)
|
|
303
|
-
- [Agent/contributor protocol](AGENTS.md)
|
|
94
|
+
Project context: [AGENTS.md](AGENTS.md) · [BACKLOG.md](BACKLOG.md) · [CHANGELOG.md](CHANGELOG.md).
|