pi-fabric 0.22.2 → 0.22.3

package/docs/certification.md

@@ -0,0 +1,134 @@
+# Context and memory certification
+
+This repository has two separate evaluation paths:
+
+- `pnpm certify:context` is deterministic, offline, and non-billable.
+- `pnpm benchmark:real-resume` is an opt-in, billable Pi RPC benchmark. Its default behavior is a safe skip.
+
+Neither command is part of `pnpm test`. The normal test suite remains offline and fast.
+
+## Deterministic certification
+
+Run on Node 24 or newer:
+
+```sh
+pnpm certify:context
+pnpm certify:context -- --json /tmp/pi-fabric-certification.json
+```
+
+The package command builds `dist/` first, then runs `scripts/certify-context.mjs`. It prints a human summary followed by the complete JSON report and exits nonzero when any threshold fails.
+
+### Compaction endurance
+
+The harness creates a persisted session through Pi's `SessionManager`, appends messages and compactions through its public methods, and reads the active parent-linked branch with `getBranch()`. It performs exactly 100 Fabric compactions under deterministic settings (`contextWindow=64`, `reserveTokens=63`, `keepRecentTokens=1`). Before every hook event it calculates Pi's built context, applies `shouldCompact`, and requires Pi's own `prepareCompaction` to return a preparation. It then invokes the callback registered by `registerCompactionHook` with Pi's event shape, branch, preparation token count, reason, retry state, and signal.
+
+Pi 0.80.6 publicly exports `SessionManager`, `buildContextEntries`, `buildSessionContext`, and `shouldCompact`. It implements and declares `prepareCompaction`, and `AgentSession` uses it, but the package root/export map does not export it. Certification therefore resolves the exact installed 0.80.6 internal module, verifies the installed version and function shape, and reports `prepareCompactionPubliclyExported: false`. There is no public API that supplies a preparation without running an `AgentSession` with a model; claiming otherwise would be inaccurate.
+
+Every persisted summary receives a cycle-unique `PRIOR_SUMMARY_POISON_991_…` suffix in the actual `CompactionEntry`. On the next cycle Pi's preparation must expose that exact stored previous summary. A proxy around the event preparation records whether the registered Fabric callback reads `previousSummary`; the result is derived from those accesses rather than hardcoded. Fabric must not read it or emit its poison. No summary is manually converted to a user message.
+
+Every cycle also checks:
+
+- the original goal, constraint, and pinned Unicode rare fact;
+- cumulative source, file, and unresolved-error addresses;
+- tool-call/result closure at the kept boundary;
+- that every nonempty `firstKeptEntryId` exists on the active branch;
+- exact persisted summary/details round trips;
+- `SessionManager.buildContextEntries()` and public `buildContextEntries()` agreement;
+- after the compaction and after each subsequent append, the built context is exactly the latest `compaction` entry followed by the retained live entries;
+- a valid UTF-8 summary size no larger than 32 KiB.
+
+The last 20 summary sizes must have a range no larger than 512 bytes and an absolute least-squares slope no larger than 16 bytes per cycle. These bounds detect late unbounded growth without requiring every cycle to have the same size.
+
+Six explicit eligible closure fixtures must each execute at least once: normal, compact-all, Pi split-turn preparation, parallel/delayed results, reverse-order call/result, and malformed prior boundary. Every resulting Fabric cut is checked for call/result closure.
+
+A separate approximately 330 KiB maximal source uses multibyte goals, instructions, paths, errors, turns, and typed Fabric activity. It must produce at least 24 KiB of summary output, remain at most 32 KiB, and round-trip through a fatal UTF-8 decoder. This exercises the bound near its reachable projection saturation rather than relying on the endurance fixture's natural approximately 5.8 KiB plateau.
+
+This proves deterministic cumulative projection, actual Pi eligibility/context behavior, closure handling for the named fixtures, and byte-safe saturation for the generated typed event streams. It does not prove semantic quality for arbitrary human conversations or model behavior.
+
+### Cross-layer memory
+
+The same run creates 1,000 additional persisted Pi sessions. The unique rare-fact session receives an old source mtime and must be classified cold while only eight sessions remain hot. Certification calls `MemoryProvider` directly rather than parsing shell output.
+
+The pass conditions are:
+
+- at least 1,000 eligible sessions and complete indexing coverage;
+- exact lexical recall of the cold rare fact;
+- exact source expansion by its stable entry ID;
+- exact expansion of every distinct entry ID emitted by the 100 compaction summaries or their structured details;
+- 100% address expansion agreement with a fresh normalization of the source JSONL;
+- V5 `sourceHash` integrity checks on both cold hydration and context address expansion.
+
+The JSON report includes eligible/indexed/stale counts, emitted/expanded address counts, and cache/source byte sizes.
+
+This proves addressability through the current cache, digest, search, and source-expansion layers. It does not prove fuzzy semantic retrieval, ranking under unrelated corpora, cache performance on all filesystems, or recovery after source deletion.
+
+### Continuation QA
+
+Continuation QA creates two small temporary repositories. Each has exact expected final files, an executable Node oracle, and files that must remain byte-identical. A no-model handoff simulator receives only:
+
+1. the compacted summary and structured compaction details; and
+2. constrained current-session pointer and expansion APIs backed by `MemoryProvider`.
+
+The source phase persists a handoff envelope containing the compacted context and current Pi session ID, not task operations or a captured session path. The resume phase reads that output, constructs a fresh `MemoryProvider`, asks it for a V5 integrity-bound current-session pointer, derives the cumulative source entry ID from compaction details, and expands that address with `expectedSourceHash`. The `addressResolved` score comes from the returned entry, never a constant. No callback closes over `manager.getSessionFile()`.
+
+Only after exact source expansion does the simulator decode `CERT_TASK_V1` and apply its operations. If an exact operation or file payload is unavailable, it throws and fails instead of inventing success. The external oracle then scores exact filesystem state, forbidden-file integrity, and process exit status; it never supplies `task.operations` to the simulator.
+
+This proves that the emitted address, current persisted session identity, and allowed memory operations can carry these mechanically executable tasks across a fresh handoff. Pi's compaction result does not itself expose a session ID or source hash, so those come respectively from persisted current-session context and `MemoryProvider`; the report does not claim they are emitted by Pi. It does not claim that arbitrary prose can be converted into operations, that a model will choose to recall, or that the two fixtures represent all software work.
+
+## Real Pi RPC benchmark
+
+The benchmark compares three arms in deterministic randomized paired order:
+
+- `baseline`: resume the full, uncompacted context;
+- `fabric`: compact with Fabric, terminate that process, then resume in a fresh process;
+- `pi-vcc`: issue `compact` with the exact `__pi_vcc__` sentinel while both Fabric and the configured pi-vcc extension are loaded, terminate that process, then resume in a fresh process.
+
+The resumed process receives exactly:
+
+```text
+Resume and complete the task.
+```
+
+A filesystem/test oracle outside the model scores the result. Reports capture pass/fail diff reasons, tokens, USD cost, tool calls, recall calls, wall time, summary bytes, Wilson 95% pass-rate intervals, and paired win/tie rates. Reports include the credential variable's name but never its value. Session and repository data live in a temporary directory and are removed after the run.
+
+The RPC reader implements strict LF JSONL framing. It splits only on `\n`, strips an optional trailing `\r`, preserves U+2028/U+2029 inside JSON strings, and does not use Node's `readline`.
+
+### Safety gate
+
+Running this command without configuration exits zero and reports `SKIP`:
+
+```sh
+pnpm benchmark:real-resume
+```
+
+A billable run requires every gate below:
+
+```sh
+PI_FABRIC_REAL_RESUME=1 \
+PI_FABRIC_BENCH_PROVIDER=anthropic \
+PI_FABRIC_BENCH_MODEL=claude-sonnet-4-5 \
+PI_FABRIC_BENCH_KEY_ENV=ANTHROPIC_API_KEY \
+PI_FABRIC_BENCH_REPEATS=3 \
+PI_FABRIC_BENCH_MAX_USD=5 \
+PI_VCC_EXTENSION=/absolute/path/to/pi-vcc/extension.ts \
+pnpm benchmark:real-resume
+```
+
+`PI_FABRIC_BENCH_KEY_ENV` names an already-set credential environment variable. The benchmark checks observed session cost before starting each next arm and stops once the configured budget has been reached. A single in-flight request can exceed the remaining budget, so the maximum is a stop boundary, not a provider-side hard spending cap.
+
+The benchmark proves end-to-end behavior only for the selected model, provider, fixture, extension versions, and repeats. Small samples have wide confidence intervals. It does not establish general superiority or isolate every source of provider variance.
+
+## Relationship to pi-vcc stress tooling
+
+The neighboring pi-vcc stress scripts informed the useful ideas of repeated compaction, late-size measurements, paired real-session comparisons, and explicit recall accounting. This harness does not copy their regex-based section scoring, feed the previous rendered summary as the next source, or claim their assumptions. Fabric certification instead uses Pi parent-linked session entries, structured compaction details, direct memory APIs, exact source expansion, and executable continuation oracles.
+
+## Test coverage
+
+`tests/certification/` covers:
+
+- strict LF JSONL parsing, including split UTF-8 and Unicode line separators;
+- the default skip gate and complete opt-in gate;
+- deterministic paired order and benchmark confidence/paired reporting;
+- executable continuation oracle passes and forbidden-change failures;
+- certification rejection when eligibility, poison exclusion, address resolution, or the external oracle is sabotaged;
+- certification report threshold failures.

package/docs/compaction.md

@@ -0,0 +1,177 @@
+# Deterministic compaction
+
+Pi Fabric provides an LLM-free compactor through `session_before_compact`. It is the default engine; set `compaction.engine` to `"pi"` to defer to Pi's compactor.
+
+Fabric targets 65% of the model's advertised context window after compaction by default. This is configurable from `/fabric-settings` or `compaction.targetContextRatio` (bounded to `0.25`–`0.85`):
+
+```json
+{
+  "compaction": {
+    "engine": "fabric",
+    "targetContextRatio": 0.65
+  }
+}
+```
+
+Use `{ "compaction": { "engine": "pi" } }` to disable the Fabric engine.
+
+## Invariants
+
+1. **The session log is ground truth.** The summary is a bounded continuation view with stable entry-id and file addresses.
+2. **Live cut and cumulative truth are separate.** The cut is selected from the window made live by the last compaction. The summary is rebuilt from every raw, typed, content-bearing entry on the supplied active branch prefix before the new kept boundary.
+3. **Rendered summaries are never semantic input.** `compaction`, branch-summary prose, custom summary prose, and unknown roles produce no normalized events. A valid Fabric branch-summary details envelope may contribute its typed facts; its `summary` string never does. Top-level Pi `custom_message` entries are different: Pi puts them in model context, so Fabric preserves their typed `customType`, text content, visibility, and bounded JSON details. Non-context-bearing `custom` state entries remain excluded.
+4. **Structure drives projection.** The core uses entry/message types, roles, content-part types, custom-message fields, tool names, JSON arguments, call ids, `isError`, exit codes, entry ids, ordering, valid Fabric execution traces, and valid Fabric branch-summary facts. It has no semantic regex over prose, code, shell commands, or tool output. Whitespace normalization, bounded truncation, exact identity comparisons, and path segmentation are mechanical operations.
+5. **Serialization is deterministic and bounded.** Identical branch entries and instructions produce byte-identical output. The rendered result is at most 32 KiB in UTF-8.
+6. **The nominal model window is the safety boundary.** Fabric calibrates Pi's structural token estimate against `preparation.tokensBefore`, retains as much recent raw context as fits the configured occupancy target, and reserves both Pi's response budget and an additional estimator-error margin. Undocumented provider headroom is never part of the budget.
+
+This prevents both summary-chain drift and deterministic forgetting. Pi replaces the previous rendered summary, but Fabric re-derives the original goal, cumulative successful file addresses, error state, and user scope changes from raw branch history each time.
+
+## Pipeline
+
+```text
+active branch entries ─┬─► live window ─► calibrated token budget ─► closure-safe cut ─► firstKeptEntryId
+                       └─► raw cumulative prefix ─► normalize ─► project ─► bound/render
+```
+
+- `normalize.ts` converts raw message and top-level `custom_message` entries to typed events. Custom content is selected only from typed string/text parts; JSON details are depth/node/collection/string/byte bounded and malformed details are omitted without dropping otherwise valid content. Tool calls and results are paired only by `toolCallId`. A `fabric_exec` result contributes nested events only through a valid `details.trace` V1 guard, or through the separate strict legacy `details.audits` adapter when no `trace` field exists.
+- `projections.ts` computes goal, file, operation-state, turn, status, and transcript views.
+- `enrichers.ts` permits deterministic optional annotations. Fabric ships no built-in enrichers.
+- `render.ts` independently bounds every rendered block and enforces the global UTF-8 limit.
+- `hook.ts` computes the live cut, selects cumulative source, emits v2 details, and implements Pi/pi-vcc precedence.
+
+## Live cut and closure
+
+The last compaction marker identifies the live window:
+
+- a valid `firstKeptEntryId` starts the window at that entry;
+- a compact-all marker or missing/orphan kept id starts it after the marker;
+- without a marker, the whole supplied active path is live.
+
+When Pi supplies the active model metadata, Fabric chooses the live cut by token budget rather than always preserving or discarding the whole latest user turn:
+
+1. Sum Pi's public structural message estimates for the current context.
+2. Calibrate that estimate with `preparation.tokensBefore`. This compensates for provider tokenization, system prompts, tool schemas, and other fixed context that a character heuristic cannot observe directly.
+3. Reserve the maximum 32 KiB summary, then target `contextWindow × targetContextRatio` while honoring `keepRecentTokens` when it remains safe.
+4. Cap the target below `contextWindow - reserveTokens` with an additional estimator-error margin, and below 95% of `tokensBefore` so a low-usage manual compaction cannot expand context. The advertised window is authoritative even when a provider accepts larger requests.
+5. Select the earliest eligible boundary whose retained suffix fits the budget. User/custom boundaries and assistant boundaries are eligible, so a single enormous autonomous turn can be split instead of surviving compaction intact. On repeated compaction, the kept boundary must follow the previous compaction marker in raw log order; Pi replays entries contiguously from `firstKeptEntryId`, so allowing a boundary before that marker would replay the old rendered summary beside the new one.
+
+Fabric computes structural spans for every call id across the supplied branch and rejects any candidate that separates an actual call/result pair. Therefore both directions are enforced:
+
+- no summarized tool call has a kept result;
+- no kept tool call has a summarized result.
+
+This handles parallel calls, delayed results, reverse/malformed ordering, and malformed prior boundaries. If no non-crossing boundary fits, Fabric uses compact-all (`firstKeptEntryId: ""`), so no kept side remains to orphan either half. If the rendered deterministic summary itself makes the calibrated projection exceed the target, Fabric cancels rather than persisting an expanding or over-budget result. If model metadata is unavailable, the legacy latest-turn closure-safe cut remains as a compatibility fallback.
+
+The live cut determines only what Pi keeps. Summary source is the raw active-branch prefix before that new boundary. Earlier compaction and branch-summary prose within that prefix is skipped by normalization.
+
+## Bounded sections
+
+The original first user goal is emitted first. Later user scope changes and potentially large file, operation-state, and earlier-turn collections use deterministic earliest-plus-latest sampling. Every omission records a count and a source entry-id range. File lines also carry the source call entry id.
+
+Rendered block limits include their headers:
+
+| Block | UTF-8 limit |
+| --- | ---: |
+| `[Session Goal]` | 4096 bytes |
+| `[Compaction Request]` | 3072 bytes |
+| `[Files And Changes]` | 4608 bytes |
+| `[Fabric Activity]` | 2048 bytes |
+| `[Outstanding Context]` | 4608 bytes |
+| `[Earlier Turns]` | 3072 bytes |
+| `[Current Status]` | 2048 bytes |
+| collapsed transcript | 5120 bytes |
+| footer | 1536 bytes |
+
+The limits sum below 32 KiB, leaving room for separators. A final UTF-8 guard enforces the global limit. Projection limits are also finite: 24 later goals, 24 file addresses per operation kind, 32 operation-state records, 32 earlier turns, and 40 transcript events. Omitted source remains executable-addressable through entry-id ranges and the footer recall pointer.
+
+## Sections
+
+- **Session Goal**: up to three bounded lines from the original first user message, followed by sampled later user scope changes.
+- **Compaction Request**: canonicalized, bounded custom instructions; see below.
+- **Files And Changes**: successful typed file-tool addresses grouped as Created, Written, Modified, or Read. `edit` is Modified. `write` is Written unless a typed result explicitly proves creation.
+- **Fabric Activity**: bounded phases and significant non-file nested operations, including bash, agents, workflow, mesh, state, MCP, and extension refs. Every line has a stable `entryId/subordinal` address.
+- **Outstanding Context**: typed tool/bash failures and later exact structural resolutions. File failures require the same action and path, bash failures the same command, and generic failures the same ref and arguments. Explicit error text is quoted and bounded, never parsed or classified. Trace failures use only `operation.outcome` and `operation.error`.
+- **Earlier Turns**: sampled user/custom context one-liners and tool-name counts.
+- **Current Status**: the latest summarized user/custom context, modification address, and assistant line.
+- **Transcript**: the latest 40 typed events, including quoted/bounded custom-message content and bounded structural details, plus an omission range when applicable.
+- **Footer**: deterministic source timestamp, cumulative source range, and session-log recall guidance.
+
+There is intentionally no commit projection. The core does not recognize `git commit` command prefixes and does not parse shell stdout for hashes or summaries. A caller that needs a commit ID across compaction must provide it explicitly through a valid typed `preserve` item or another typed state transition.
+
+## Remaining structural text operations
+
+The clean core retains only these mechanical text operations:
+
+- select text from typed user, assistant, top-level custom-message, tool-result, command-argument, error, phase, ref, and path fields;
+- split user text on literal newlines for bounded goal lines, or select the first line for one-line views;
+- trim/collapse whitespace and truncate by fixed character or UTF-8 byte limits;
+- quote bounded user/custom/assistant/tool/error text without interpreting its content;
+- compare typed action/path, action/command, or ref/JSON-arguments identities exactly for resolution;
+- segment typed paths on `/` or `\\` to compute display roots;
+- split a typed Fabric ref once on `.` to expose provider/action identity;
+- inspect the explicit typed `created: true` result field for write classification;
+- match only the exact `__pi_vcc__` sentinel or exact typed-request prefix, then use a bounded structural JSON parser.
+
+No command prefix, stdout/stderr line format, error wording, path-looking prose, commit-looking prose, source code, or tool-result rendering is recovered into semantic facts.
+
+## Custom instructions
+
+`customInstructions === "__pi_vcc__"` is an exact routing sentinel and is never rendered by Fabric.
+
+Every other plain instruction is explicit user data, not a mini-language. Fabric canonicalizes whitespace, bounds the input, and includes it in `[Compaction Request]` without semantically parsing it.
+
+`compact.request` may add typed `preserve: string[]` values. When present, the controller forwards an exact versioned prefix followed by JSON. The hook accepts only the exact prefix and a strict v1 object. Once that reserved prefix is present, malformed JSON/scalars, duplicate protocol keys (including escaped-key aliases), unknown fields or versions, invalid types, unpaired UTF-16 surrogates, excessive structure, or exceeded bounds produce a structured decode error and cancel the operation; the encoded payload is never reinterpreted or rendered as plain instructions. A UI/RPC context receives a bounded error notification when available.
+
+Typed v1 limits are enforced before value mapping or canonicalization: instructions are at most 8192 characters and 8192 UTF-8 bytes; `preserve` has at most 16 items; each item is at most 2048 characters and 2048 UTF-8 bytes; and the complete prefix-plus-JSON source is at most 16 KiB. The decoder checks the aggregate source limit before invoking its bounded recursive-descent parser, rejects duplicate decoded keys while parsing, validates scalar grammar and surrogate pairing, and checks preserve count before iterating or canonicalizing values. Plain Pi/manual instructions remain explicit bounded text and are not subjected to the typed protocol parser.
+
+## Compaction details v2
+
+New summaries emit `details.compactor: "fabric"` and `details.version: 2` with:
+
+- cumulative source and live-cut ranges;
+- branch, source-entry, event, and live-cut counts;
+- prior recognized Fabric v1/v2 marker counts;
+- per-projection omission counts and the typed preserve count (valid v1 requests cannot exceed the preserve limit);
+- instruction mode, canonicalization, source size, truncation, and preserve counts;
+- stable kept/source entry-id addresses and the source timestamp;
+- when adaptive budgeting is active: advertised window, target ratio/tokens, Pi reserve and recent settings, raw estimate, calibration scale, fixed overhead, retained raw tokens, and Fabric's `projectedTokensAfter`. Pi core independently recomputes its own `estimatedTokensAfter` after persisting the compaction.
+
+Only exact Fabric versions 1 and 2 are recognized. v1 details and rendered prose are not reused as truth. On the next compaction, an old session naturally migrates to v2 because the new result is rebuilt from raw active-branch entries. V2 validation accepts the legacy commit-omission counter for old records, but new summaries do not emit a commit projection or counter.
+
+## Nested Fabric execution traces
+
+For an outer `fabric_exec` tool result, normalization reads only `message.details.trace` through `readFabricExecutionTraceV1`. Operations are emitted in `operation.sequence` order with addresses such as `entry-id/0`; phases use `entry-id/phase:0`. Known `pi.read`, `pi.grep`, `pi.find`, `pi.ls`, `pi.edit`, `pi.write`, and `pi.bash` calls retain exact typed arguments and outcomes. Other refs remain typed Fabric activity.
+
+A present but malformed or unknown trace version is ignored and is not reinterpreted as legacy data. When `trace` is absent, the legacy adapter accepts only an audit array whose records have typed `ref`, JSON `args`, boolean `success`, and optional string `error`; it never reads audit rendering or `result` prose. The outer tool conversation remains in the transcript, but `fabric_exec` source code and outer result prose cannot create file, failure, or activity facts.
+
+## Deterministic branch summaries
+
+When the Fabric engine is active, the same registration also handles `session_before_tree`. It returns nothing when `userWantsSummary` is false and compiles only `preparation.entriesToSummarize` when true. Tree custom instructions use the same plain/typed decoder and fail-closed limits as compaction. The exact `__pi_vcc__` value has routing meaning only for compaction; on the tree path it remains ordinary explicit request text.
+
+`replaceInstructions: true` has Pi replacement-prompt semantics, not append-instructions semantics. A deterministic projection cannot execute an arbitrary replacement summarizer prompt, so Fabric returns `undefined` and defers to Pi or another handler. No Fabric summary or typed Fabric branch details are produced by Fabric in that explicit mode.
+
+Branch details use `kind: "pi-fabric.branch-summary"`, `version: 1`, stable source addresses, and at most 256 bounded typed facts in a 128 KiB envelope. Facts cover source users, top-level custom messages, phases, and operations. Newly generated details record `source.oldLeafId` from `preparation.oldLeafId`; this is the canonical abandoned/from-leaf provenance. Older v1 envelopes without that field remain readable. Pi 0.80.6 writes generic `BranchSummaryEntry.fromId` from the navigation target position rather than the abandoned leaf, and a hook cannot correct that core-generated field, so consumers must use Fabric's typed `source.oldLeafId` when present.
+
+Nested branch summaries re-emit only valid typed facts; branch summary prose is never normalized. Later compaction can therefore resolve abandoned-branch failures against later exact successes and retain custom context, files, and activity through navigation or forks without parsing prose. Since Pi supplies only the active path or the abandoned `entriesToSummarize` path to each compiler, sibling branches do not contaminate one another.
+
+## pi-vcc precedence
+
+Precedence remains:
+
+1. exact `__pi_vcc__` custom-instruction sentinel;
+2. configured Fabric engine;
+3. pi-vcc/default Pi behavior.
+
+Fabric marks claimed events with `_fabricCompaction`. If an earlier pi-vcc handler marked `_piVccOverriding` and Fabric has nothing to compact, Fabric does not return a cancellation that would erase the pi-vcc result. With engine `"pi"`, Fabric neither claims nor cancels the event.
+
+Pi's public extension contract runs `session_before_*` handlers in extension load order and keeps the latest non-cancelling result. Therefore an unrelated handler loaded after Fabric can replace Fabric's compaction or tree result; a later cancellation also terminates dispatch. There is no supported public registration phase that can move one extension behind every subsequently loaded extension. Fabric preserves the explicit pi-vcc sentinel/marker cooperation above, but does not monkeypatch Pi's private runner. Deployments that require Fabric to win over arbitrary hooks must load Fabric after those extensions (while accounting for any intentionally later pi-vcc override).
+
+## Reconstruction QA
+
+`src/compaction/qa.ts` derives probes from normalized source events, never rendered sections. QA probes follow the same bounded sampling policy as projections: directly rendered samples are checked for content, while omitted collections are checked for count/range addressability. Mutation tests remove file, error, turn, and footer information to verify that the report detects loss.
+
+Run:
+
+```sh
+pnpm vitest run tests/compaction-qa.test.ts
+```

package/docs/configuration.md

@@ -0,0 +1,245 @@
+# Configuration
+
+Pi Fabric reads configuration from two JSON files. Project values override global values.
+
+1. `~/.pi/agent/fabric.json` — global defaults.
+2. `<project>/.pi/fabric.json` — project overrides, only for **trusted** projects.
+
+`/fabric settings` writes changes to the same files: trusted projects write to `<project>/.pi/fabric.json`; untrusted sessions write to the global `~/.pi/agent/fabric.json`.
+
+`executor.runtime` selects `"quickjs"` (the default isolated WASM runtime) or `"node-process"` (a disposable native V8 process). QuickJS memory limits are capped at `4294967295` bytes because its WASM32 `size_t` cannot represent 4 GiB; larger values are rejected rather than wrapped. Node process limits may be set as high as detected physical memory and are passed to V8 as `--max-old-space-size`.
+
+`node-process` is an explicit trusted-code escape hatch, not a security sandbox. It preserves Fabric's IPC host bridge, approvals, audit records, timeout, and cancellation, but Node's `vm` API is not a security boundary. Enable it only for workloads and projects whose generated code you are willing to run with the local user account's authority. Each invocation receives a fresh child process and is forcibly terminated when it settles, times out, or is cancelled. Schema enforce mode always forces `quickjs`. Large limits in either runtime can exhaust system memory or destabilize the machine.
+
+## Full reference
+
+```json
+{
+  "fullCodeMode": true,
+  "executor": {
+    "runtime": "quickjs",
+    "timeoutMs": 120000,
+    "memoryLimitBytes": 67108864,
+    "maxOutputChars": 100000,
+    "maxNestedResultChars": 2000000,
+    "resultFormat": "auto"
+  },
+  "approvals": {
+    "read": "allow",
+    "write": "allow",
+    "execute": "allow",
+    "network": "allow",
+    "agent": "allow"
+  },
+  "capture": {
+    "enabled": true,
+    "hideFromModel": true,
+    "keepVisible": ["fabric_exec"],
+    "defaultRisk": "execute",
+    "risks": {
+      "read": "read",
+      "grep": "read",
+      "find": "read",
+      "ls": "read",
+      "edit": "write",
+      "write": "write",
+      "bash": "execute"
+    }
+  },
+  "mcp": {
+    "enabled": true,
+    "disableOAuth": true,
+    "allowDynamicServers": true,
+    "callTimeoutMs": 120000
+  },
+  "subagents": {
+    "enabled": true,
+    "runner": "pi",
+    "transport": "process",
+    "claude": {
+      "binary": "claude"
+    },
+    "thinking": "medium",
+    "maxConcurrent": 4,
+    "maxPerExecution": 100,
+    "maxDepth": 2,
+    "timeoutMs": 3600000,
+    "extensions": true,
+    "defaultTools": ["read", "bash", "edit", "write", "grep", "find", "ls"],
+    "retainRuns": false,
+    "notifyOnComplete": true,
+    "budgetUsd": 0,
+    "maxTokensPerChild": 0
+  },
+  "ui": {
+    "enabled": true,
+    "widget": "auto",
+    "maxRows": 6,
+    "refreshMs": 500,
+    "eventHistory": 80,
+    "haltOnEscape": true,
+    "showNestedToolCalls": true,
+    "nestedToolDebounceMs": 100
+  },
+  "compaction": {
+    "engine": "fabric"
+  },
+  "mesh": {
+    "enabled": true,
+    "actorScope": "project",
+    "maxEventBytes": 262144,
+    "maxReadEvents": 500,
+    "actorPollMs": 250,
+    "actorQueueLimit": 32,
+    "eventContextChars": 40000
+  }
+}
+```
+
+## Result formatting
+
+`executor.resultFormat` sets the default for `fabric_exec` return values and is available under `/fabric settings` → **Executor**. `"auto"` keeps strings as text and renders structured values as syntax-highlighted YAML. `"yaml"`, `"json"`, and `"text"` force the corresponding behavior. A call-level `resultFormat` parameter overrides the configured default.
+
+The compaction engine is available under `/fabric settings` → **Compaction**. Select `"fabric"` for deterministic compaction or `"pi"` to delegate to Pi core.
+
+## Code modes
+
+With the default full code mode, `fabric_exec` exclusively owns Pi core tool execution. The parent model sees one programmable tool instead of direct `read`, `bash`, `edit`, `write`, `grep`, `find`, and `ls` schemas. Fabric programs use those capabilities through `pi.*`:
+
+```ts
+const files = await pi.find({ pattern: "**/*.ts", path: "src" });
+const matches = await pi.grep({ pattern: "TODO", path: "src" });
+return { files, matches };
+```
+
+Independent calls should be parallel:
+
+```ts
+const [packageJson, readme] = await Promise.all([
+  pi.read({ path: "package.json" }),
+  pi.read({ path: "README.md" }),
+]);
+return {
+  package: JSON.parse(packageJson).name,
+  readmeLines: readme.split("\n").length,
+};
+```
+
+### Full code mode (default)
+
+`fullCodeMode: true` is the default. Fabric removes active Pi core tools from the parent model and exposes their implementations only inside `fabric_exec` through `pi.*`. Registered overrides such as security gates and code previews are captured too, so `pi.read()` continues to route through the override rather than bypassing it.
+
+Fabric remembers which native core tools were active before taking ownership. Switching to orchestration-only mode or unloading Fabric restores that selection. Full-mode ownership is applied only when the session initializes or the mode changes. Fabric does not reset an explicitly selected active tool set from input, agent-start, turn-end, or settled lifecycle hooks; the system prompt carries the full-mode execution rule.
+
+Pi core normally includes its model-visible skill catalog only while the native `read` tool is active. Full code mode restores the same catalog from Pi's structured skill registry and adapts only its loader instruction to use `pi.read` inside `fabric_exec`; native core tools remain hidden. When an expanded skill invokes another installed skill, Fabric also adds an exact name-to-path resolution hint for that turn so the delegated `SKILL.md` is loaded before task work.
+
+### Orchestration-only mode
+
+Users who want Fabric for MCP, agents, ambient actors, parallel workflows, councils, and recursive delegation — but want Pi's core tools to remain entirely native — can opt out of full code mode:
+
+```json
+{
+  "fullCodeMode": false
+}
+```
+
+In orchestration-only mode:
+
+- Pi's `read`, `bash`, `edit`, `write`, `grep`, `find`, and `ls` tools stay on Pi's normal model-facing and execution paths.
+- Registered extension tools also remain in Pi's native registry; Fabric does not hide, wrap, or expose them through `extensions.*`.
+- `pi.*`, `extensions.*`, and equivalent `tools.call()` references are unavailable inside `fabric_exec`, including when TypeScript checks are bypassed.
+- MCP and stable Fabric providers remain available through `mcp.*`, `memory.*`, `state.*`, `schema.*`, and `compact.*`; generic discovery and computed refs remain available through `tools.*`. One-shot and recursive agents, persistent ambient actors, dynamic workflows, mesh coordination, councils, explicit Fabric providers, and the Fabric TUI also remain available.
+- Child agents continue using their allowed Pi tools directly, so parallel and ambient setups do not route their coding operations back through Fabric code mode.
+
+### Where to set it
+
+`fullCodeMode` defaults to `true`. A project can set the flag in `.pi/fabric.json`, or a user can set it globally in `~/.pi/agent/fabric.json`. `/fabric settings` toggles it too.
+
+## Captured extension tools
+
+When `fullCodeMode` is enabled, Fabric intercepts Pi's `ExtensionRunner.getAllRegisteredTools()` registry chokepoint. This captures tools registered by other extensions at startup or later through `pi.registerTool()`, regardless of whether those extensions load before or after Fabric.
+
+Captured custom tools are removed from Pi's model-facing registry by default, so their schemas, snippets, and guidelines do not consume the parent model context. The extension itself remains loaded: its commands, event handlers, state, and UI continue to work. Only tool discovery and invocation become lazy.
+
+```ts
+const matches = await tools.search({ query: "deployment status" });
+const schema = await tools.describe({ ref: matches[0].ref });
+const result = await tools.call({
+  ref: schema.ref,
+  args: { environment: "staging" },
+});
+return result;
+```
+
+For tool names valid as JavaScript properties, use the shorter proxy:
+
+```ts
+const result = await extensions.project_status({ verbose: true });
+return result.text;
+```
+
+The result preserves `content`, text content as `text`, `details`, `isError`, `terminate`, and source provenance. Fabric runs the captured definition's `prepareArguments()` and original executor with its owning extension context. Pi's `tool_call`, `tool_result`, and `tool_execution_*` lifecycle handlers are also applied to nested captured calls.
+
+Extension overrides of core tools are captured and hidden with their built-in counterparts in full code mode. Inside Fabric, `pi.read`, `pi.bash`, and the other built-ins automatically route through a captured override when one exists; `extensions.read` exposes the override's full native result shape. `capture.keepVisible` can retain non-core extension tools in Pi's direct registry, but core tool names are always excluded while full code mode owns them.
+
+## Approvals and risk
+
+Fabric risk classes are `read`, `write`, `execute`, `network`, and `agent`; approval policy values are `allow`, `ask`, or `deny`.
+
+- Captured tools default to the conservative `execute` risk because Pi tool definitions do not declare effects. Add exact tool-name overrides under `capture.risks`.
+- Set `capture.hideFromModel` to `false` to index non-core extension tools without hiding them.
+- `capture.keepVisible` names stay in both Fabric and Pi's direct registry, except that Pi core names are always Fabric-owned in full code mode.
+- An `ask` policy is fail-closed in headless modes without interactive UI.
+- Approval is cached by risk class for one `fabric_exec` execution.
+
+## Subagents
+
+`subagents.runner` selects the default harness (`"pi"` or `"claude"`). `subagents.model` is the optional Pi `provider/id` override; `subagents.claude.model` is the optional canonical Claude runtime key. `subagents.claude.binary` defaults to `claude` and can be an absolute path or wrapper; `PI_FABRIC_CLAUDE_BINARY` overrides it for the current process. `/fabric settings` enumerates Claude models from that binary in the background and stores the two runner defaults independently.
+
+Other subagent settings:
+
+- `thinking` — default reasoning effort (`off`, `minimal`, `low`, `medium`, `high`, `xhigh`, `max`), default `medium`.
+- `maxConcurrent` — global child concurrency semaphore.
+- `maxPerExecution` — hard cap on children per `fabric_exec` invocation.
+- `maxDepth` — recursion depth bound for `rlm.query()`.
+- `timeoutMs` — default per-child wall-clock budget and floor for per-call overrides (60 minutes by default). Lower per-call values are ignored; callers should only set `timeoutMs` to request a longer run.
+- `extensions` — whether Claude children keep their normal Claude Code customizations.
+- `defaultTools` — the default tool allowlist for children.
+- `budgetUsd` — shared append-only cost ledger across a recursion tree (0 disables).
+- `maxTokensPerChild` — per-child cumulative token bound (0 disables).
+- `notifyOnComplete` — send a follow-up completion message for a detached `agents.spawn()`.
+
+See [agents, actors & mesh](agents.md) for the runner and transport details.
+
+## MCP
+
+- `mcp.disableOAuth` — when true, MCP calls may use cached credentials but cannot launch a new interactive OAuth flow.
+- `mcp.callTimeoutMs` — per-call timeout bound.
+- `mcp.allowDynamicServers` — permit `mcp.register()` of ephemeral servers.
+- `mcp.enabled` — set to `false` to disable the MCP surface.
+
+See the [`mcp` reference](../skills/fabric-exec/references/mcp.md) for the call surface.
+
+## UI
+
+- `ui.widget` is `auto`, `always`, or `hidden`. `auto` shows active or retained Fabric runs and worker activity. Active one-shot agents and actor workers occupy rows; their recent nested tools appear beneath them when enabled.
+- `ui.showNestedToolCalls` defaults to `true` and controls child-agent/actor tool rows in both the parent `fabric_exec` card and widget.
+- `ui.nestedToolDebounceMs` defaults to `100` and applies one coalescing timer across all regular nested calls in a `fabric_exec` execution. Set it to `0` to emit every update; accepted values are clamped to `0..2000`.
+- The widget renders above the chat (like `pi-supervisor`); set `ui.enabled` to `false` to disable both the widget and dashboard controller.
+
+See the [interface reference](interface.md).
+
+## Mesh
+
+Mesh data defaults to `<project>/.pi/fabric/mesh`. Set `mesh.root` to a relative or absolute path to relocate durable topics, shared state, and actor sessions. Add `.pi/fabric/mesh/` to the project's ignore file unless the coordination log is intentionally versioned. Set `mesh.enabled` to `false` to disable both mesh actions and ambient actor restoration.
+
+`mesh.actorScope` controls where persistent actor definitions, mailboxes, and child sessions are stored and restored from:
+
+- `"project"` (default) keeps a single shared actor registry at `.pi/fabric/mesh/actors/`, so actors survive `/new` and carry over between Pi sessions in the same project without redefinition. One Pi process should own the actor registry at a time — concurrent sessions sharing a registry may race on writes.
+- `"session"` isolates actors per Pi session (under `.pi/fabric/mesh/actors/<sessionId>/`). Use this when you run concurrent Pi sessions in one project and want each to own its own actors.
+
+With project scope, one Pi process should own the actor registry at a time — concurrent sessions sharing a registry may race on writes. Mesh topics and shared state are always project-scoped. Root Pi sessions publish short-lived presence leases under the mesh so `agents.peers()` and the dashboard can show concurrent sessions; the local dashboard owner is Main and other roots are Peers. Actor relay normally follows filesystem change notifications; `mesh.actorPollMs` controls the fallback interval when those notifications are unavailable. A low-frequency reconciliation check protects against coalesced or missed filesystem events.
+
+## Compaction
+
+The deterministic, LLM-free compaction engine is default-on. Set `compaction.engine` to `"pi"` to restore pi-core compaction. When pi-vcc is also installed, Fabric takes precedence for automatic compaction, while an explicit `/pi-vcc` command always uses pi-vcc's engine. See [compaction](compaction.md) for invariants, sections, and limits.