pi-fold 0.1.1 → 1.0.0

package/README.md

@@ -1,6 +1,6 @@
 # pi-fold
 
-pi-fold lets a Pi agent govern its own context window. The registered `active_context` tool gives the session's agent seven actions over its transcript: `status`, `peek`, `fold`, `expand`, `refold`, `protect`, and `unprotect`. Folding is lossless: a fold replaces a stale span with a short brief while keeping SHA-256-verified references to every original entry, so any fold can be expanded back to its exact source. Advisory milestones tell the agent what is eligible as measured context pressure rises; the agent decides what stays hot, what folds, and what is pinned.
+pi-fold lets a Pi agent govern its own context window. The registered `active_context` tool gives the session's agent nine actions over its transcript: `status`, `peek`, `fold`, `expand`, `refold`, `protect`, `unprotect`, `rebrief`, and `reboundary`. Folding is lossless: a fold replaces a stale span with a short brief while keeping SHA-256-verified references to every original entry, so any fold can be expanded back to its exact source. Advisory milestones tell the agent what is eligible as measured context pressure rises; the agent decides what stays hot, what folds, and what is pinned.
 
 When the agent does not act, an autonomous ladder manages the window the same lossless way: it folds completed read-only tool batches, folds structurally closed chapters, and consolidates existing folds into a hierarchy only past explicit thresholds. Only the oldest eligible context changes; the fresh window is never touched.
 
@@ -10,12 +10,10 @@ When the agent does not act, an autonomous ladder manages the window the same lo
 
 ## Status
 
-This package is pre-release. Its API may move until 0.1.0. Licensed under [MIT](./LICENSE).
+Stable. The option surface is settled and the append-only projection invariant is enforced by the gate suite. Licensed under [MIT](./LICENSE).
 
 ## Install
 
-After the first 0.1.0 release:
-
 ```sh
 pi install npm:pi-fold
 ```
@@ -26,7 +24,7 @@ pi-fold maps canonical session entries into a fold lattice. A fold replaces an e
 
 Collapsed folds double as a browsable index of earlier work. The briefs sit in the stable prefix of the window, which providers typically serve from cache, so the agent can page through them at little cost, note which spans matter to its current task, expand those, and leave the rest folded.
 
-The briefs are the index, and reading a fold is not the same as committing to it. `peek` is the ephemeral read: it returns one fold's exact SHA-256-verified source as a tool result, at any depth and with the ancestors still collapsed, without touching the projection; the content lands in the fresh window like any other read and can later fold away as a completed read batch. `expand` is the commitment, restoring that source in place until the fold is refolded, and `status` with `detail: "tree"` lists every fold, nested ones included, in transcript order with its depth and parent. Expansion is outside-in, level by level: expanding a parent reveals its children's briefs, each expandable in turn, and collapsing a parent re-collapses its descendants.
+The briefs are the index, and reading a fold is not the same as committing to it. `peek` is the cheap read: it returns one fold's exact SHA-256-verified source as a tool result, at any depth and with the ancestors still collapsed, without touching the projection; the content lands in the fresh window like any other read and later folds away as a completed read batch. It accepts `offset` and `bytes` for a bounded slice, and any child fold id is peekable, so a large fold has a narrow read. `expand` is the commitment, restoring that source in place until the fold is refolded, and `status` with `detail: "tree"` lists every fold, nested ones included, in transcript order with its depth and parent. Expansion is outside-in, level by level: expanding a parent reveals its children's briefs, each expandable in turn, and collapsing a parent re-collapses its descendants.
 
 The agent's own actions come first. Through the context tool it folds any structurally closed span it judges stale, folds completed tool batches, consolidates two or more adjacent folds into a deeper one at any time, expands any fold whose detail is needed again, and protects entries that must stay raw. None of the ladder's pressure or width thresholds apply to these actions.
 
@@ -74,6 +72,36 @@ registerPiFold(pi, {
 
 A candidate needs an id, the text that is scored and shown, and a route string naming the action that retrieves it. Optional `position` (a transcript ordinal, where higher is more recent) and `depth` feed the recency and depth components. A source that throws, or an item that is malformed, costs its own items and nothing else: the carrier, the projection, and the ladder are never at risk.
 
+## Epoch fold scheduling
+
+Folding is cheap to decide and expensive to apply, and the expense has nothing to do with how much a fold saves. A provider prefix cache is positional: rewriting the middle of the window invalidates every cached byte after the edit, so the bill is set by how often the projection changes, not by how many tokens each change reclaims. A session that folds once per turn re-sends its prefix once per turn; a session that folds three times in total re-sends it three times. Marking is free because no byte moves: the mark lives in durable session state and the projection stays byte-identical, so the cached prefix survives. A commit is the one moment the projection is rewritten, and it is worth that rewrite because every accumulated decision lands in it at once.
+
+`foldScheduling: "epoch"` splits folding into those two phases. A `fold` action records a pending MARK and returns immediately, having moved nothing. A commit applies every pending mark in one projection rewrite, through the same preparation and commit machinery an immediate fold uses, so a committed mark and an immediate fold produce the identical fold record. Committing is the runtime's call, not a verb the agent can spend: measured 2026-08-07, an agent given the verb called it twice and the runtime correctly held it both times, which is surface without function. `unmark` withdraws a standing decision, and `"immediate"` remains the default, writing no pending-marks key and leaving existing state digests where they are.
+
+The agent is meant to lead here. Marking costs nothing, so the agent can mark as it works, the moment a span stops being useful, without weighing each fold against a cache invalidation; it then commits at a natural seam in the work. The ladder is the fallback, exactly as it is for folding itself. Its own rung decisions become marks instead of folds, and when measured pressure reaches the rung the ladder already uses for refolds and consolidation, it commits the epoch for the agent. No new threshold was introduced for scheduling. Each automatic commit also carries a target: if the agent's marks would free less than a fifth of the window, the ladder tops the epoch up with the stalest unprotected completed tool batches until they do, and the commit result records which marks were the agent's and which were the top-up.
+
+Three refinements keep the two phases from getting in the way. A fold whose span sits within the last few messages of the window applies immediately even in epoch mode, because almost nothing follows it to invalidate. An `expand` issued while marks are pending opens the commit epoch, so a restore plus a batch of folds costs one rewrite between them rather than two; `peek` stays immediate, since it appends a tool result and never edits the prefix. And a peek result is itself a completed read, so at the next commit it folds away automatically unless the agent committed to what it saw by expanding or protecting that fold, which keeps look, decide, discard nearly free.
+
+Protection is unchanged. Top-up never marks protected evidence, and a pending mark that has become protected refuses at commit with a message naming it, leaving the rest of the epoch to apply. `active_context {"action":"status"}` reports the pending marks under `automatic.scheduling` with their origin, the estimated share of the window a commit would free, and the estimated prefix tokens it would rewrite.
+
+## Reclaiming peek output
+
+A `peek` returns a fold's exact stored source, bounded by the chapter cap. That copy then sits in the window as raw evidence while the fold it came from sits beside it as a placeholder: the same bytes, held twice, and the copy is the one the ladder cannot take. Immediate mode does not classify a peek batch as a completed read, so every rung measures the pressure and finds nothing eligible. On a workload that peeks freely this is the whole failure: a measured 64-stage session ended with 14 raw peek results holding 1.9M characters, 82 percent of everything still unfolded, while the ladder had already folded 53 of its 56 stage results and had no supply left.
+
+`foldPeekResults: true` hands that supply back. The peek batch becomes a completed read like any other, eligible for the same cadence rung under the same protections: the freshest turns and the byte tail still stay raw, protected evidence is still refused, and the fold record is the ordinary reversible one, so the source recovers exactly. Nothing about the ladder's thresholds changes; only what counts as foldable does. Epoch scheduling already carries this classification with it, and the option is how immediate mode reaches the same place.
+
+Reach for it when the agent uses `peek` as a browsing verb rather than a rare lookup. Leave it off when a peeked span is normally the span the agent is about to work from, since folding it back means peeking again to read it.
+
+## The append-only projection
+
+A provider prefix cache is positional: it replays the longest byte-identical prefix, so one changed byte at offset K discards everything cached after K. The cost of a mutation has nothing to do with its size. pi-fold therefore holds one invariant: the runtime may move a byte in the window only at a moment it is already rewriting the window, which is a commit or a fold. Everything else is a pure append at the tail that is never later altered, shortened, removed, or repositioned.
+
+That rules out a whole class of otherwise reasonable optimizations. A peek result is a duplicate of bytes the fold store already holds losslessly, and rewriting it in place after a model call has read it looks tail-local; it is not. The rewrite waits for a later assistant message to exist, and by then the window has grown over it, so the edit lands mid-prefix. Measured 2026-08-07: two such reclamations cost 100k fresh tokens. A peek is now append-only, and its duplicate bytes are reclaimed by the tool-fold rung at the next commit, which is the one moment a rewrite is already being paid for.
+
+`active_context {"action":"status"}` reports real projection rewrites separately from observed provider-side cache misses, per-message digests included: a miss on a projection the runtime did not rewrite is provider-side by construction, and conflating the two hides exactly the bug this invariant exists to prevent.
+
+Earlier releases shipped this behavior, along with truthful capacity accounting, admission control on `peek` and `expand`, retained pending marks, the eligible-share commit trigger, stage-identified fold briefs, the current-turn commit guard, the pinned-mass backstop, the status index diet, and delivery-counted advisories, as individually opt-in levers. All of them are now simply how pi-fold works. The flags are gone rather than deprecated; git history is the lineage.
+
 ## Relation to prior work
 
 pi-fold combines two recent lines of work in one mechanism. Lossless hierarchical compaction, exemplified by LCM ([arXiv:2605.04050](https://arxiv.org/abs/2605.04050)) with an interactive walkthrough at [losslesscontext.ai](https://www.losslesscontext.ai/), builds a summary DAG over older messages with lossless pointers to every original, but the system alone decides when to compact. Self-GC ([arXiv:2607.00692](https://arxiv.org/abs/2607.00692)) treats context as indexed, recoverable objects with fold, mask, and prune actions, proposed by a side-channel planner under harness enforcement. In pi-fold the session agent itself holds the verbs, in-band: it can fold, expand, refold, protect, and consolidate on its own judgment, and the autonomous ladder is the fallback; a session that never touches the tool degrades gracefully into lossless hierarchical compaction.
@@ -100,7 +128,10 @@ The package entry calls `registerPiFold(pi)` with the defaults below. Hosts that
 | `setProjectionProvider` | `undefined` | Optional host callback that receives the [projection-candidate provider](#projection-candidate-records). Normal Pi context events do not require it. |
 | `surfacing` | `true` | Score collapsed folds against the recent task context and offer the best few back through the ephemeral tail channel. Set `false` to register no suggestion sources and emit nothing. |
 | `setSuggestionSourceRegistrar` | `undefined` | Optional host callback that receives the [suggestion source](#suggestion-sources) registration function. |
-| `toolActions` | all seven actions | Replacement allowlist drawn from `status`, `peek`, `fold`, `expand`, `refold`, `protect`, and `unprotect`. |
+| `foldScheduling` | `"immediate"` | `"immediate"` applies every fold when it is made. `"epoch"` makes a fold a free pending mark and applies all pending marks in one projection rewrite at `commit` or under window pressure. See [epoch fold scheduling](#epoch-fold-scheduling). |
+| `foldPeekResults` | epoch mode `true`, immediate mode `false` | Classify a `peek` result as a completed read, so the tool-fold rung may reclaim it. A peek copies a fold's own stored source back into the window; where a deployment peeks often, leaving that copy raw is the mass the ladder cannot take. See [reclaiming peek output](#reclaiming-peek-output). |
+| `providerTotalWindow` | `400000` | Total serving window. Every ratio, fence and budget is computed against this minus the output reservation actually in force, because the per-request max-input descriptor a provider advertises assumes a full reservation and understates the real ceiling. The descriptor is still read and still reported in status, so the gap stays auditable. |
+| `toolActions` | all nine actions, plus `unmark` in epoch mode | Replacement allowlist drawn from `status`, `peek`, `fold`, `expand`, `refold`, `protect`, `unprotect`, `rebrief`, `reboundary`, and `unmark`. There is deliberately no agent-callable `commit`: marking is the agent's job and folding is the runtime's. |
 | `blockingTools` | `["Agent"]` | Replacement list of tool names that trigger one opportunistic stale-tool fold before the call. Use `[]` to disable it. |
 | `readOnlyTools` | `new Set(["read", "grep", "find", "ls"])` | Replacement set of tool names whose completed batches may fold automatically. Defaults to Pi's built-in read-only tools. Pass a `ReadonlySet<string>`. |
 | `isMcpTool` | `() => false` | Synchronous predicate enabling evidence ingestion for oversized structured MCP results. Names shaped as `mcp__<server>__<tool>` project the actual server; other names use the neutral namespace derived from `entryTypePrefix`. |