pi-fold 1.0.4 → 2.0.0

package/README.md

@@ -8,7 +8,7 @@ The full write-up, with interactive versions of every mechanism on this page and
 
 ![pi-fold folding a session in place: the agent marks spans as it works, the marks commit together at a fold event, and a mark inside the protected fresh tail is held until it ages out](https://raw.githubusercontent.com/shaneconner/pi-fold/main/media/fold-demo.gif)
 
-*The mechanism, drawn. One session runs as the transcript it is, user messages in cream, the agent's replies in green, tool results a quieter shade of the same green. The agent calls `pi_fold_context mark` on spans it has finished with: each mark outlines the span it names and moves nothing, and the next fold event commits every standing mark in one rewrite, two outlined spans collapsing to two briefs at once while the ladder takes the unmarked stale end on its own. A dotted rule rides over the newest turns, the protected fresh tail: a mark that lands inside it is held at the fold event, so nothing folds out from under the turn still using it, and it commits a few turns later once it has aged out. The exact messages stay on disk, addressed and verified by SHA-256, and expanding a brief brings them back byte for byte. Illustrative animation, not a screen capture; the measured numbers are further down. A 6 MB mp4 of the same clip is at [media/fold-demo.mp4](https://github.com/shaneconner/pi-fold/blob/main/media/fold-demo.mp4).*
+*The mechanism, drawn. One session runs as the transcript it is, user messages in cream, the agent's replies in green, tool results a quieter shade of the same green. The agent calls `pi_fold_context` with `action: "fold"` on spans it has finished with: each call records a mark that outlines the span it names and moves nothing, and the next fold event commits every standing mark in one rewrite, two outlined spans collapsing to two briefs at once while the ladder takes the unmarked stale end on its own. A dotted rule rides over the newest turns, the protected fresh tail: a mark that lands inside it is held at the fold event, so nothing folds out from under the turn still using it, and it commits a few turns later once it has aged out. The exact messages stay on disk, addressed and verified by SHA-256, and expanding a brief brings them back byte for byte. Illustrative animation, not a screen capture; the measured numbers are further down. A 6 MB mp4 of the same clip is at [media/fold-demo.mp4](https://github.com/shaneconner/pi-fold/blob/main/media/fold-demo.mp4).*
 
 ## Install
 
@@ -48,7 +48,7 @@ Folding keeps the turn. Occupancy crosses a threshold, spans fold in place, and
 
 ## Measured results
 
-The campaign behind these tables is two experiments. The first, here, is pi-fold against Pi's native compaction. The second, further down under [epoch fold scheduling](#epoch-fold-scheduling), is pi-fold against itself with fold scheduling as the only variable.
+The first campaign behind these tables contains two experiments. The first, here, is pi-fold against Pi's native compaction. The second, further down under [epoch fold scheduling](#epoch-fold-scheduling), is pi-fold against itself with fold scheduling as the only variable. A later transcript-only hidden-mass campaign is reported separately below, with its own protocol and claim limits.
 
 One 64-stage staged assignment over the curl C repository: one Pi session per run, one user message, and the agent calls a `repo_stage` tool 64 times inside a single agentic turn. Model gpt-5.6-sol at xhigh effort, provider openai-codex. The 272,000-token figure in these tables is the transport's per-request input descriptor, which is what pi-fold budgeted against; the Codex catalog actually serves gpt-5.6 at a 372,000-token cap, which is where the native arm lived, peaking at 369.0k before each compaction. One arm runs pi-fold, the other runs Pi's native compaction.
 
@@ -108,6 +108,23 @@ The measurement is written up as a paper, with the figure sources, the redacted
 
 Full method, artifact hashes and per-iteration history: [docs/fold_vs_compaction/experiment-log.md](https://github.com/shaneconner/pi-fold/blob/main/docs/fold_vs_compaction/experiment-log.md). Queued work: [docs/next-steps.md](https://github.com/shaneconner/pi-fold/blob/main/docs/next-steps.md).
 
+### Transcript-only hidden mass, protocol v4
+
+The follow-up campaign plants seeded values only in the transcript, never in the repository checkout, and withholds one final 30-cell block until all 64 stages are delivered. The plan, question order and target checkout were frozen before the readout. Both arms ran gpt-5.6-sol at xhigh effort on the same plan hash, `eb488827c46d`.
+
+| Outcome across assigned attempts | pi-fold | native compaction |
+| --- | ---: | ---: |
+| Completed 64 stages | 2 of 2 | 0 of 2 |
+| Join endpoints recorded correctly | 8 of 8 | 0 of 8; all eight recorded `unknown` |
+| Ordinary probes | 40 of 42 | not available from two incomplete runs |
+| Withheld end-block cells | 60 of 60 | not reached |
+
+Native compaction lost a different continuation key in each attempt, after stages 39 and 56. The first run was killed after an unbounded resume defect produced 1,761 provider responses; the second carried the fixed three-resume bound and failed by name within 33 minutes. At that point the campaign stopped native rather than launching until one survived, which would select on the outcome, and ran a second pi-fold attempt to leave two attempted assignments per arm.
+
+Both pi-fold end-block turns included context recovery. The agent made five context calls in repetition 1 and nine in repetition 2 before answering. At the answering requests, the corrected carriage sweep classified 53 of 60 expected end-block values as visible raw and seven as recoverable but not visible, all seven in repetition 1. Those seven answers were correct, but the final projection does not identify how their bytes reached the answer: 60 of 60 is the scored outcome, not proof that recovery made the whole ledger visible. All eight withdrawn correction values were visible raw, and the agent selected the correction every time. The same 102-row certification found one ordinary miss with its expected value visible in a brief and the other recoverable but not visible, while the same matched control in both runs had its value absent from the answering branch.
+
+There is no direct cross-arm end-block score and no cost ratio from this campaign: neither native run completed, and native repetition 1 is excluded from cost comparison because of the harness runaway. Two attempts do not estimate a population failure rate. The portable result contains all 102 rows and the source artifact hashes: [hidden-mass-results.json](https://github.com/shaneconner/pi-fold/blob/main/docs/fold_vs_compaction/hidden-mass-results.json). The full chronology and stop rule are in the [experiment log](https://github.com/shaneconner/pi-fold/blob/main/docs/fold_vs_compaction/experiment-log.md).
+
 ## How it works
 
 pi-fold maps canonical session entries into a fold lattice. Collapsed folds double as a browsable index of earlier work: the briefs sit in the stable prefix of the window, which providers typically serve from cache, so the agent can page through them at little cost, note which spans matter to its current task, expand those, and leave the rest folded. `status` with `detail: "tree"` lists every fold, nested ones included, in transcript order with its depth and parent.
@@ -116,11 +133,11 @@ The agent's own actions come first. Through the context tool it folds any struct
 
 Every fold, agent-made or automatic, obeys the same structural rules. Folds nest rather than partially overlap. Chapter folds align to closed user, assistant and tool-batch units, consolidations take two or more adjacent folds, and protected or fresh evidence never folds.
 
-The automatic law folds only completed tool batches and structurally closed chapters, absorbs only the tool-result folds inside a chapter's own span, and treats fold placeholders as ordinary span material once the window carries `consolidateAfter` unpinned folds, which keeps the brief index tidy long before pressure matters. Briefs are bounded at 1,200 characters each, so the visible index normally costs about 1% of the window. Automatic foldability is membership, not position: a span folds because it is a completed tool batch or older material a chapter can compose, and what holds it back is the agent's pins, the guaranteed-raw fresh tail, the blacklist, and the commit-time guard on the open turn.
+The automatic law folds only completed tool batches and structurally closed chapters, absorbs only the tool-result folds inside a chapter's own span, and treats fold placeholders as ordinary span material once the window carries `consolidateAfter` unpinned folds, which keeps the brief index tidy long before pressure matters. Briefs are bounded at 2,000 characters each, one cap for supplied and runtime-written briefs alike, so the visible index normally costs a low single-digit share of the window. Automatic foldability is membership, not position: a span folds because it is a completed tool batch or older material a chapter can compose, and what holds it back is the agent's pins, the guaranteed-raw fresh tail, the blacklist, and the commit-time guard on the open turn.
 
 Below `maxTarget` occupancy of the serving budget, 80% by default, the runtime is quiet: nothing folds automatically at all, because a commit is an epoch transition that risks the whole prefix cache, and the cadence has to be the fewest commits that still keep the window inside its budget. The remaining fifth of the budget is the runway the commit itself spends. Every occupancy number in this document is a share of that one serving budget: there is a single denominator, `capacityAccounting.budgetTokens`, and no threshold is stated against the raw provider window.
 
-Chapter briefs are generated by the session's model by default, using one request per fold with a bounded input span, bounded output, and no tool calls. Every summarizer failure falls back to a deterministic brief, automatically and with no setting involved. The default spends provider tokens, as native compaction does. A cheaper model from the same provider is the recommended summarizer: a session on GPT 5.6 Sol can delegate fold briefs to GPT 5.6 Luna at medium effort with `summarizer: { provider: "openai-codex", model: "gpt-5.6-luna", effort: "medium" }`, using the provider id from your Pi model registry.
+Every brief is deterministic, written by the runtime from the span itself with no model call: a tool fold carries the call, its arguments, the head and tail of the result, and the agent's own notes from the batch; a chapter carries its opening text and every assistant note in span order; a consolidation indexes each child. A model-written brief generator shipped through 1.0.x and was deleted on its own measurement: across 15 sealed runs it wrote 1,186 upgraded briefs, 75 percent of which were never consulted by the agent or never became visible before the session ended, and a build with the generator removed scored the same recall for less money. The agent that wants better words on a fold writes them itself: `fold` accepts a brief, `rebrief` rewrites one in place, and a supplied brief is kept verbatim. Sessions already holding model-written briefs keep them on load.
 
 The serving budget is what the deployment states it may fill, because the per-request max-input descriptor a provider advertises assumes a full output reservation and understates the real ceiling. With no `providerInputBudget` supplied the runtime falls back to the transport's descriptor and estimates a reservation out of it, which is the conservative default for a model it has no budget fact for; the descriptor is reported in status either way, so the gap stays auditable. A provider-window fence aborts an unsafe request before transmission if no lossless fold can make room, and says so out of band rather than by writing into the window.
 
@@ -174,15 +191,23 @@ The design also pairs with Pi's session trees: a fold's collapsed-or-expanded st
 
 ## Configuration
 
-The package entry calls `registerPiFold(pi)` with the defaults below, and that is the whole surface: five options, and every other name is refused with the replacement named in the message. One VALUE is refused as well, `summarizer: "deterministic"`, because the deterministic brief is the automatic fallback on any summarizer failure rather than a mode to select. Hosts that call the named `registerPiFold` export directly may pass an options object. Replacement sets replace the defaults; they are not additive.
+The package entry calls `registerPiFold(pi)` with the defaults below, and that is the whole surface: four options, and every other name is refused with the replacement named in the message. A host still passing `summarizer` is refused with the deletion message rather than silently ignored. Hosts that call the named `registerPiFold` export directly may pass an options object. Replacement sets replace the defaults; they are not additive.
 
 | Option | Default | Effect |
 | --- | --- | --- |
-| `thresholds` | `{ maxTarget: 0.80, minTarget: 0.35, freshTail: 0.02, consolidateAfter: 10 }` | The thermostat, set whole or not at all. The first three are proportions of the serving budget; `consolidateAfter` is a positive integer count. Validated atomically at registration, and an impossible policy is refused by name rather than clamped. User policy only: no agent action can read it back as a mutable surface, though status reports the values. |
-| `summarizer` | `"session"` | Brief generator: the live session model, or `{ provider, model, effort? }` resolved through Pi's model registry. Those are the only two forms. A cheap model from the same provider is the recommendation, since a brief is a bounded factual summary that does not need the session model's reasoning budget: on a Codex session, `{ provider: "openai-codex", model: "gpt-5.6-luna", effort: "medium" }`. Every summarizer failure falls back to a deterministic brief on its own, so there is no value that selects one. |
+| `thresholds` | `{ maxTarget: 0.80, minTarget: 0.20, freshTail: 0.02, consolidateAfter: 10 }` | The thermostat, set whole or not at all. The first three are proportions of the serving budget; `consolidateAfter` is a positive integer count. Validated atomically at registration, and an impossible policy is refused by name rather than clamped. User policy only: no agent action can read it back as a mutable surface, though status reports the values. |
 | `providerInputBudget` | the transport's descriptor | The tokens this deployment may actually put into a request, already net of whatever output reservation it holds back. Every ratio, fence and budget divides by it directly: pi-fold subtracts nothing from a declared value. Supply it when the deployment knows its real ceiling, because the per-request max-input descriptor a provider advertises assumes a full reservation and understates it. With no value supplied the runtime falls back to that descriptor and estimates the reservation out of it, which is the conservative default for a model it has no budget fact for; the descriptor is read and reported in status either way, so the gap stays auditable. |
 | `blacklistAutoFoldTools` | `new Set()` | The exception list. Every completed tool batch is foldable by the runtime on its own, without an agent mark; name a tool here and its results stay raw instead. Empty by default, because foldability is not the protection: pins, the fresh tail and the commit guard are, and they apply to a blacklisted tool and an ordinary one alike. Pass a `ReadonlySet<string>`. Blacklisting is not free: the mass stays in the window until a chapter fold takes it, and a closed unit that alone exceeds the chapter cap is reclaimed whole rather than in pieces. |
-| `guidance` | `{ thresholdNotices: true, actionResponses: true }` | The two guidance surfaces, set together. `thresholdNotices` is the append-once occupancy waypoint at 25, 50 and 75 percent of the serving budget; `actionResponses` is the persistent acknowledgement a context action returns. Booleans only, and off means the carrier is absent rather than empty. |
+| `guidance` | `{ actionResponses: true }` | The persistent acknowledgement returned by a context action. Boolean only, and off means the carrier is absent rather than empty. |
+
+### Migrating from 1.x
+
+Version 2.0 makes the measured deletions explicit instead of accepting options that no longer control the runtime.
+
+- Remove `summarizer` and `summarizeContextSpan`. The model brief generator is deleted; every new fold carries a deterministic or agent-supplied brief. Passing either name is refused with the deletion reason.
+- Remove `guidance.thresholdNotices`. `guidance` now contains only `actionResponses`.
+- The default `thresholds.minTarget` is now `0.20`. A host supplying the whole threshold object keeps its declared value.
+- Existing sessions remain readable. Fold records already carrying model-authored briefs keep those briefs and their provenance on load.
 
 ### Projection candidate records
 
@@ -221,4 +246,5 @@ Stable. The option surface is settled and the append-only projection invariant i
 npm ci
 npm run lint
 npm test
+node scripts/verify_pi_context_experiment.mjs
 ```