pi-fold 0.1.0 → 1.0.0

package/README.md

@@ -1,19 +1,19 @@
 # pi-fold
 
-pi-fold provides lossless, agent-governed context folding for Pi. It changes only the oldest eligible context while leaving the fresh window untouched.
+pi-fold lets a Pi agent govern its own context window. The registered `active_context` tool gives the session's agent nine actions over its transcript: `status`, `peek`, `fold`, `expand`, `refold`, `protect`, `unprotect`, `rebrief`, and `reboundary`. Folding is lossless: a fold replaces a stale span with a short brief while keeping SHA-256-verified references to every original entry, so any fold can be expanded back to its exact source. Advisory milestones tell the agent what is eligible as measured context pressure rises; the agent decides what stays hot, what folds, and what is pinned.
+
+When the agent does not act, an autonomous ladder manages the window the same lossless way: it folds completed read-only tool batches, folds structurally closed chapters, and consolidates existing folds into a hierarchy only past explicit thresholds. Only the oldest eligible context changes; the fresh window is never touched.
 
 ![pi-fold folding a live session](https://raw.githubusercontent.com/shaneconner/pi-fold/main/docs/demo.gif)
 
-*A live session at 5.5× speed: the autonomous ladder folds a completed read batch mid-conversation — no command issued — and `/context` reports the fold.*
+*A live session at 5.5× speed: the autonomous ladder folds a completed read batch mid-conversation, no command issued, and `/context` reports the fold.*
 
 ## Status
 
-This package is pre-release. Its API may move until 0.1.0. Licensed under [MIT](./LICENSE).
+Stable. The option surface is settled and the append-only projection invariant is enforced by the gate suite. Licensed under [MIT](./LICENSE).
 
 ## Install
 
-After the first 0.1.0 release:
-
 ```sh
 pi install npm:pi-fold
 ```
@@ -22,10 +22,94 @@ pi install npm:pi-fold
 
 pi-fold maps canonical session entries into a fold lattice. A fold replaces an eligible stale span with a short placeholder while retaining ordered references to the exact source entries. Expansion checks the stored SHA-256 identities before restoring that source.
 
-An autonomous ladder first folds completed read-only tool batches, then refolds or consolidates existing folds, and finally folds structurally closed chapters. Cadence and fresh-tail rules prevent repeated or premature folding. Chapter briefs are generated by the session's model by default, using one request per fold with a bounded input span, bounded output, and no tool calls. Every summarizer failure falls back to a deterministic brief; set `summarizer: "deterministic"` to disable model calls entirely. The default spends provider tokens, as native compaction does. Advisory milestones describe eligible actions as measured context pressure rises, and the provider-window fence blocks an unsafe request if no lossless fold can make room.
+Collapsed folds double as a browsable index of earlier work. The briefs sit in the stable prefix of the window, which providers typically serve from cache, so the agent can page through them at little cost, note which spans matter to its current task, expand those, and leave the rest folded.
+
+The briefs are the index, and reading a fold is not the same as committing to it. `peek` is the cheap read: it returns one fold's exact SHA-256-verified source as a tool result, at any depth and with the ancestors still collapsed, without touching the projection; the content lands in the fresh window like any other read and later folds away as a completed read batch. It accepts `offset` and `bytes` for a bounded slice, and any child fold id is peekable, so a large fold has a narrow read. `expand` is the commitment, restoring that source in place until the fold is refolded, and `status` with `detail: "tree"` lists every fold, nested ones included, in transcript order with its depth and parent. Expansion is outside-in, level by level: expanding a parent reveals its children's briefs, each expandable in turn, and collapsing a parent re-collapses its descendants.
+
+The agent's own actions come first. Through the context tool it folds any structurally closed span it judges stale, folds completed tool batches, consolidates two or more adjacent folds into a deeper one at any time, expands any fold whose detail is needed again, and protects entries that must stay raw. None of the ladder's pressure or width thresholds apply to these actions.
+
+Every fold, agent-made or automatic, obeys the same structural rules. Folds form a DAG whose leaves are the exact session entries, and every fold record keeps ordered references down to those leaves, so expansion at any depth restores the originals after SHA-256 verification. Folds nest rather than partially overlap. Chapter folds align to closed user/assistant/tool-batch units, consolidations take two or more adjacent folds, and protected or fresh evidence never folds.
+
+The autonomous ladder is deliberately more conservative than the agent. It folds only completed read-only tool batches and structurally closed chapters, absorbs only the tool-result folds inside a chapter's own span, and creates folds of folds only past explicit thresholds, taking adjacent folds oldest first. Those thresholds are more than ten visible collapsed folds, which keeps the brief index tidy long before pressure matters, or total measured context usage (all of it, not just briefs) reaching 85% of the provider window, at which point consolidation squeezes existing folds to reclaim room. Briefs themselves are bounded at 1,200 characters each, so the visible index normally costs about 1% of the window. Cadence and fresh-tail rules prevent repeated or premature automatic folding.
+
+Chapter briefs are generated by the session's model by default, using one request per fold with a bounded input span, bounded output, and no tool calls. Every summarizer failure falls back to a deterministic brief; set `summarizer: "deterministic"` to disable model calls entirely. The default spends provider tokens, as native compaction does. A cheaper model from the same provider is the recommended summarizer: for example, a session on GPT 5.6 Sol can delegate fold briefs to GPT 5.6 Luna at medium effort with `summarizer: { provider: "openai", model: "gpt-5.6-luna", effort: "medium" }`, using the provider id from your Pi model registry. Advisory milestones describe eligible actions as measured context pressure rises, and the provider-window fence blocks an unsafe request if no lossless fold can make room.
+
+Milestone wording comes in three guidance profiles over the same rungs, thresholds, and per-session budgets. The default `"pressure"` frames each milestone as rising context pressure; `"curation"` frames the same milestones as curating context for the current task and adds one early orientation advisory, at 25% of the window, that introduces the fold index; `"minimal"` delivers only the urgent fence advisory.
 
 By default, evidence ingestion writes read-only artifact files under the session directory's `pi-fold-evidence/`, sets them to mode `0444`, and enforces a 512 MB session cap. These immutable files provide exact-recovery anchors for oversized tool results. Set `evidenceIngestion: false` to disable the hook and all evidence writes.
 
+## Surfacing relevant folds
+
+Paging the fold index is something the agent can do at any moment. Surfacing is that same act performed on its behalf when it did not think to look. As a session grows, the material most worth reopening tends to sit in the oldest and deepest folds, which are exactly the ones the agent stops seeing. pi-fold scores every collapsed fold against what the session is working on right now and hands the best two or three back with the fold id and both affordances, so the agent can peek one ephemerally, expand it in place, or dismiss the whole slate and carry on. The judgment stays with the agent; the mechanism only puts the option where it can be acted on.
+
+The selector is deterministic and model free: no extra provider call, no embeddings, no randomness, no wall clock. A fold scores on the lexical overlap between its brief and the most recent user and assistant text, plus a small recency component read from transcript position and a small bonus for depth (a deeply nested fold is otherwise invisible). Those structural components alone can never clear the threshold, so a suggestion always rests on evidence that the span matches the task in hand, and when nothing clears the threshold, nothing is shown at all. A fold that was suggested and left alone is suppressed for a cooldown; a fold that is currently expanded or protected is never suggested.
+
+Suggestions ride the same ephemeral channel as the advisory milestones: appended at the tail of the window, after the stable prefix, so no cached prefix is invalidated and nothing durable is written to the transcript. They never share an advisory with urgent fence text, where the only useful next action is the fold that keeps the request transmissible.
+
+Every suggestion shown is recorded in durable session state with its source, item id, score and transcript ordinal. The record resolves to `accept` when the agent peeks or expands that item within a short window of turns, and to `reject` otherwise. `active_context {"action":"status"}` returns the live slate and the whole log under `automatic.surfacing`. Set `surfacing: false` to register no sources and emit nothing. Slate size, threshold, cooldown and character budget are internal constants rather than options: they are the knobs the accept and reject record exists to settle.
+
+### Suggestion sources
+
+pi-fold owns the carrier, not the content. Any extension can put its own items into the same slate, with the same ranking, the same budget, and the same accept and reject log. The built-in fold-brief source is registered through that identical interface.
+
+```js
+registerPiFold(pi, {
+  setSuggestionSourceRegistrar: (register) => {
+    const memory = register({
+      id: "my-memory",
+      candidates: ({ state, snapshot, toolName }) => [
+        {
+          id: "note-42",
+          text: "Durable note the current task may want back.",
+          route: "recall {\"address\":\"note-42\"}",
+        },
+      ],
+    });
+    // memory.accepted("note-42") when the agent acts on it; memory.unregister() to withdraw.
+  },
+});
+```
+
+A candidate needs an id, the text that is scored and shown, and a route string naming the action that retrieves it. Optional `position` (a transcript ordinal, where higher is more recent) and `depth` feed the recency and depth components. A source that throws, or an item that is malformed, costs its own items and nothing else: the carrier, the projection, and the ladder are never at risk.
+
+## Epoch fold scheduling
+
+Folding is cheap to decide and expensive to apply, and the expense has nothing to do with how much a fold saves. A provider prefix cache is positional: rewriting the middle of the window invalidates every cached byte after the edit, so the bill is set by how often the projection changes, not by how many tokens each change reclaims. A session that folds once per turn re-sends its prefix once per turn; a session that folds three times in total re-sends it three times. Marking is free because no byte moves: the mark lives in durable session state and the projection stays byte-identical, so the cached prefix survives. A commit is the one moment the projection is rewritten, and it is worth that rewrite because every accumulated decision lands in it at once.
+
+`foldScheduling: "epoch"` splits folding into those two phases. A `fold` action records a pending MARK and returns immediately, having moved nothing. A commit applies every pending mark in one projection rewrite, through the same preparation and commit machinery an immediate fold uses, so a committed mark and an immediate fold produce the identical fold record. Committing is the runtime's call, not a verb the agent can spend: measured 2026-08-07, an agent given the verb called it twice and the runtime correctly held it both times, which is surface without function. `unmark` withdraws a standing decision, and `"immediate"` remains the default, writing no pending-marks key and leaving existing state digests where they are.
+
+The agent is meant to lead here. Marking costs nothing, so the agent can mark as it works, the moment a span stops being useful, without weighing each fold against a cache invalidation; it then commits at a natural seam in the work. The ladder is the fallback, exactly as it is for folding itself. Its own rung decisions become marks instead of folds, and when measured pressure reaches the rung the ladder already uses for refolds and consolidation, it commits the epoch for the agent. No new threshold was introduced for scheduling. Each automatic commit also carries a target: if the agent's marks would free less than a fifth of the window, the ladder tops the epoch up with the stalest unprotected completed tool batches until they do, and the commit result records which marks were the agent's and which were the top-up.
+
+Three refinements keep the two phases from getting in the way. A fold whose span sits within the last few messages of the window applies immediately even in epoch mode, because almost nothing follows it to invalidate. An `expand` issued while marks are pending opens the commit epoch, so a restore plus a batch of folds costs one rewrite between them rather than two; `peek` stays immediate, since it appends a tool result and never edits the prefix. And a peek result is itself a completed read, so at the next commit it folds away automatically unless the agent committed to what it saw by expanding or protecting that fold, which keeps look, decide, discard nearly free.
+
+Protection is unchanged. Top-up never marks protected evidence, and a pending mark that has become protected refuses at commit with a message naming it, leaving the rest of the epoch to apply. `active_context {"action":"status"}` reports the pending marks under `automatic.scheduling` with their origin, the estimated share of the window a commit would free, and the estimated prefix tokens it would rewrite.
+
+## Reclaiming peek output
+
+A `peek` returns a fold's exact stored source, bounded by the chapter cap. That copy then sits in the window as raw evidence while the fold it came from sits beside it as a placeholder: the same bytes, held twice, and the copy is the one the ladder cannot take. Immediate mode does not classify a peek batch as a completed read, so every rung measures the pressure and finds nothing eligible. On a workload that peeks freely this is the whole failure: a measured 64-stage session ended with 14 raw peek results holding 1.9M characters, 82 percent of everything still unfolded, while the ladder had already folded 53 of its 56 stage results and had no supply left.
+
+`foldPeekResults: true` hands that supply back. The peek batch becomes a completed read like any other, eligible for the same cadence rung under the same protections: the freshest turns and the byte tail still stay raw, protected evidence is still refused, and the fold record is the ordinary reversible one, so the source recovers exactly. Nothing about the ladder's thresholds changes; only what counts as foldable does. Epoch scheduling already carries this classification with it, and the option is how immediate mode reaches the same place.
+
+Reach for it when the agent uses `peek` as a browsing verb rather than a rare lookup. Leave it off when a peeked span is normally the span the agent is about to work from, since folding it back means peeking again to read it.
+
+## The append-only projection
+
+A provider prefix cache is positional: it replays the longest byte-identical prefix, so one changed byte at offset K discards everything cached after K. The cost of a mutation has nothing to do with its size. pi-fold therefore holds one invariant: the runtime may move a byte in the window only at a moment it is already rewriting the window, which is a commit or a fold. Everything else is a pure append at the tail that is never later altered, shortened, removed, or repositioned.
+
+That rules out a whole class of otherwise reasonable optimizations. A peek result is a duplicate of bytes the fold store already holds losslessly, and rewriting it in place after a model call has read it looks tail-local; it is not. The rewrite waits for a later assistant message to exist, and by then the window has grown over it, so the edit lands mid-prefix. Measured 2026-08-07: two such reclamations cost 100k fresh tokens. A peek is now append-only, and its duplicate bytes are reclaimed by the tool-fold rung at the next commit, which is the one moment a rewrite is already being paid for.
+
+`active_context {"action":"status"}` reports real projection rewrites separately from observed provider-side cache misses, per-message digests included: a miss on a projection the runtime did not rewrite is provider-side by construction, and conflating the two hides exactly the bug this invariant exists to prevent.
+
+Earlier releases shipped this behavior, along with truthful capacity accounting, admission control on `peek` and `expand`, retained pending marks, the eligible-share commit trigger, stage-identified fold briefs, the current-turn commit guard, the pinned-mass backstop, the status index diet, and delivery-counted advisories, as individually opt-in levers. All of them are now simply how pi-fold works. The flags are gone rather than deprecated; git history is the lineage.
+
+## Relation to prior work
+
+pi-fold combines two recent lines of work in one mechanism. Lossless hierarchical compaction, exemplified by LCM ([arXiv:2605.04050](https://arxiv.org/abs/2605.04050)) with an interactive walkthrough at [losslesscontext.ai](https://www.losslesscontext.ai/), builds a summary DAG over older messages with lossless pointers to every original, but the system alone decides when to compact. Self-GC ([arXiv:2607.00692](https://arxiv.org/abs/2607.00692)) treats context as indexed, recoverable objects with fold, mask, and prune actions, proposed by a side-channel planner under harness enforcement. In pi-fold the session agent itself holds the verbs, in-band: it can fold, expand, refold, protect, and consolidate on its own judgment, and the autonomous ladder is the fallback; a session that never touches the tool degrades gracefully into lossless hierarchical compaction.
+
+Context-Folding ([arXiv:2510.11967](https://arxiv.org/abs/2510.11967), ByteDance Seed) has the agent branch into a sub-trajectory and fold it on completion, with the behavior shaped by reinforcement learning; the fold keeps a summary of the outcome rather than a lossless path back. MemGPT and its successor [Letta](https://github.com/letta-ai/letta) are the canonical agent-held memory-verb systems: the agent edits its own memory blocks and archives while the window pages against external stores. Two things separate pi-fold from that line. First, what is governed: Letta manages traffic between the window and outside storage, and when its message queue overflows the in-window record becomes a lossy recursive summary; pi-fold restructures the transcript in place. Second, recovery: theirs is retrieval, a search that re-ingests whatever comes back as new tokens; ours is expansion, which restores the exact folded entries after SHA-256 verification. Further inspirations include Hindsight and Hermes.
+
+The design also pairs with Pi's session trees: a fold's collapsed-or-expanded state is a natural unit for curating exactly which context a subagent or workflow leg inherits.
+
 ## Configuration
 
 The package entry calls `registerPiFold(pi)` with the defaults below. Hosts that call the named `registerPiFold` export directly may pass an options object. Replacement lists and sets replace the defaults; they are not additive.
@@ -40,8 +124,14 @@ The package entry calls `registerPiFold(pi)` with the defaults below. Hosts that
 | `commandNames` | `undefined` | Optional full-name overrides for the effective defaults `{ status: "context", fold: "fold-context" }`. Both names must be distinct kebab-case strings; supplied values take precedence over `commandPrefix`. |
 | `summarizer` | `"session"` | Brief generator: the live session model, `"deterministic"` for no model calls, or `{ provider, model, effort? }` resolved through Pi's model registry. Cannot be combined with `summarizeContextSpan`. |
 | `summarizeContextSpan` | `undefined` | Optional async brief generator. Results must include a useful bounded brief, provider/model/effort attribution, and `toolCalls: 0`; failure uses the deterministic brief. |
+| `guidance` | `"pressure"` | Advisory wording profile: `"pressure"`, `"curation"` for task-relevance self-curation plus an early orientation advisory, or `"minimal"` for the urgent advisory alone. Rungs, thresholds, and budgets are the same in every profile. |
 | `setProjectionProvider` | `undefined` | Optional host callback that receives the [projection-candidate provider](#projection-candidate-records). Normal Pi context events do not require it. |
-| `toolActions` | all six actions | Replacement allowlist drawn from `status`, `fold`, `expand`, `refold`, `protect`, and `unprotect`. |
+| `surfacing` | `true` | Score collapsed folds against the recent task context and offer the best few back through the ephemeral tail channel. Set `false` to register no suggestion sources and emit nothing. |
+| `setSuggestionSourceRegistrar` | `undefined` | Optional host callback that receives the [suggestion source](#suggestion-sources) registration function. |
+| `foldScheduling` | `"immediate"` | `"immediate"` applies every fold when it is made. `"epoch"` makes a fold a free pending mark and applies all pending marks in one projection rewrite at `commit` or under window pressure. See [epoch fold scheduling](#epoch-fold-scheduling). |
+| `foldPeekResults` | epoch mode `true`, immediate mode `false` | Classify a `peek` result as a completed read, so the tool-fold rung may reclaim it. A peek copies a fold's own stored source back into the window; where a deployment peeks often, leaving that copy raw is the mass the ladder cannot take. See [reclaiming peek output](#reclaiming-peek-output). |
+| `providerTotalWindow` | `400000` | Total serving window. Every ratio, fence and budget is computed against this minus the output reservation actually in force, because the per-request max-input descriptor a provider advertises assumes a full reservation and understates the real ceiling. The descriptor is still read and still reported in status, so the gap stays auditable. |
+| `toolActions` | all nine actions, plus `unmark` in epoch mode | Replacement allowlist drawn from `status`, `peek`, `fold`, `expand`, `refold`, `protect`, `unprotect`, `rebrief`, `reboundary`, and `unmark`. There is deliberately no agent-callable `commit`: marking is the agent's job and folding is the runtime's. |
 | `blockingTools` | `["Agent"]` | Replacement list of tool names that trigger one opportunistic stale-tool fold before the call. Use `[]` to disable it. |
 | `readOnlyTools` | `new Set(["read", "grep", "find", "ls"])` | Replacement set of tool names whose completed batches may fold automatically. Defaults to Pi's built-in read-only tools. Pass a `ReadonlySet<string>`. |
 | `isMcpTool` | `() => false` | Synchronous predicate enabling evidence ingestion for oversized structured MCP results. Names shaped as `mcp__<server>__<tool>` project the actual server; other names use the neutral namespace derived from `entryTypePrefix`. |