ds4-context-engine 0.3.0-alpha.3 → 0.3.0-alpha.5

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Published prerelease `0.3.0-alpha.2` hardens the confirmation-gated `context_persistence` tool by rejecting its output-only historical egress sentinel on input, without changing canonical Pin/Memory records, SQLite schema 15, or the reference history contract. npm `alpha` points to `0.3.0-alpha.2`, `latest` remains `0.2.0`, and the maintenance line targets Pi `0.84.3`.
+> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Published prerelease `0.3.0-alpha.4` improves `context_persistence` action guidance and privacy-safe validation categories, and moves routine successful compaction lifecycle events to `debug`. Confirmation, canonical records, strict compaction validation, Pi fallback, SQLite schema 15, and runtime contracts remain unchanged. npm `alpha` points to `0.3.0-alpha.4`, `latest` remains `0.2.0`, and the maintenance line targets Pi `0.84.3`.
 
 ## Why DS4
 
@@ -29,7 +29,7 @@ It provides:
 - exact and FTS5 historical retrieval with source provenance;
 - opt-in hybrid semantic retrieval with a deterministic local embedding and lexical fallback;
 - trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
-- hierarchical, validated, non-destructive compaction summaries;
+- hierarchical, validated, non-destructive compaction summaries with overflow-safe multi-request fan-out/fan-in;
 - persistent pins and append-only durable memory stored canonically in Pi JSONL;
 - a bounded, metadata-only `context_persistence` tool with local confirmation for every model-callable write;
 - opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
@@ -92,7 +92,7 @@ The stable `0.2.0` packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-
 To dogfood the published alpha without replacing a global stable installation, pin it in a disposable project:
 
 ```bash
-pi install -l npm:ds4-context-engine@0.3.0-alpha.2
+pi install -l npm:ds4-context-engine@0.3.0-alpha.4
 ```
 
 Follow the [0.3 alpha dogfooding runbook](docs/DOGFOODING_0.3.0_ALPHA.md); use synthetic data and a dedicated session directory.
@@ -416,7 +416,7 @@ npm run schema:context-persistence
 npm run latency:check -- /path/to/exact/ds4-context-core@0.1.2
 npm run pack:check
 # Post-publication, with an exact version rather than a dist-tag:
-npm run registry:check -- 0.3.0-alpha.2
+npm run registry:check -- 0.3.0-alpha.4
 npm pack --dry-run
 npm pack --dry-run --workspace ds4-context-core
 npm pack --dry-run --workspace ds4-context-reference-adapter
@@ -468,6 +468,8 @@ scripts             package and release-readiness checks
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
 - [0.2.0 release notes](docs/releases/0.2.0.md)
 - [0.2.0-rc.1 release notes](docs/releases/0.2.0-rc.1.md)
+- [0.3.0-alpha.4 prerelease notes](docs/releases/0.3.0-alpha.4.md)
+- [0.3.0-alpha.3 prerelease notes](docs/releases/0.3.0-alpha.3.md)
 - [0.3.0-alpha.2 prerelease notes](docs/releases/0.3.0-alpha.2.md)
 - [0.3.0-alpha.1 prerelease notes](docs/releases/0.3.0-alpha.1.md)
 - [Architecture decisions](docs/ADR/README.md)
@@ -477,7 +479,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Prerelease `0.3.0-alpha.2` hardens the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md) delivered in alpha.1 while retaining the stable canonical/configuration/SQLite/runtime contracts. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Prerelease `0.3.0-alpha.4` builds on the prior [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md) and privacy-safe [compaction](docs/COMPACTION.md) hardening with clearer action-specific failures and quieter routine lifecycle logs. Confirmation, exact targeting, strict summary grounding, Pi fallback, and stable canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ARCHITECTURE.md

@@ -101,12 +101,13 @@ agent_settled
 session_before_compact
   -> map Pi preparation messages to exact canonical entry IDs
   -> privacy-sanitize discarded text/instructions/file lists for the active provider
-  -> serialize only the newly discarded segment and preserve Pi's retained boundary
-  -> generate and validate an immutable segment node
+  -> preflight the complete sanitized request against the calibrated model input budget
+  -> when needed, split at contiguous message boundaries while keeping matching tool calls/results atomic
+  -> generate and validate each immutable segment against only its own sanitized evidence
   -> resolve the prior root from the active Pi branch only
-  -> generate and validate a bounded aggregate node when a prior root exists
+  -> recursively generate and validate bounded ordered aggregate layers until one root remains
   -> wrap generated nodes with their highest inherited classification for provider-switch safety
-  -> atomically persist prepared nodes/edges and return only the active root to Pi
+  -> atomically persist the complete prepared graph and return only one active root to Pi
 
 session_compact / session_compact_failed
   -> commit or fail prepared summary lifecycle

package/docs/COMPACTION.md

@@ -7,13 +7,13 @@ DS4 intercepts Pi's `session_before_compact` event but preserves Pi's cut-point
 1. Pi determines `messagesToSummarize`, optional split-turn prefix, and `firstKeptEntryId`.
 2. DS4 maps every source message by exact fingerprint to a canonical branch entry ID.
 3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, custom instructions, and file paths for the active provider.
-4. The active model generates and validates an immutable segment summary against sanitized evidence with cache retention disabled and a fresh routing session ID.
-5. If the active branch has a previous DS4 root, a second bounded call aggregates only that root's content with the new segment content; DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
-6. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as `prepared`; Pi receives only the active segment or aggregate text.
-7. Pi appends its normal `CompactionEntry` with `fromHook: true`.
-8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the prepared batch `failed`.
+4. DS4 estimates the complete sanitized summary request against the calibrated active-model input budget. If the whole request does not fit, it partitions the source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results; an exceptionally large turn may split only between such groups.
+5. The active model generates and validates each immutable segment summary only against that segment's sanitized evidence, with cache retention disabled and a fresh routing session ID per call.
+6. DS4 recursively aggregates ordered child summaries in bounded fan-in calls until one root remains. The ordered roots include the previous branch summary, when present, followed by every new segment. DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
+7. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage is summed across every segment and aggregate request; Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
+8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the complete prepared batch `failed`.
 
-Any mapping, model, output-limit, validation, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
+Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
 
 ## Required summary contract
 
@@ -32,7 +32,7 @@ Any mapping, model, output-limit, validation, or storage error returns `undefine
 ## Critical Exact Values
 ```
 
-Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Segment and aggregate outputs are validated independently; either unrepaired failure prevents the whole graph batch from being installed.
+Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. Any unrepaired failure prevents the whole graph batch from being installed.
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
@@ -76,4 +76,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
 /context summaries
 ```
 
-The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point.
+The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the calibrated input budget, estimated whole-source prompt size, generated segment count, and aggregate-call count without source content. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.

package/docs/CONTEXT_PERSISTENCE_TOOL.md

@@ -19,22 +19,24 @@ Memory is quoted durable factual, decision, or historical data. It supports `ses
 
 ## Actions
 
-| Action | Class | Purpose |
-| --- | --- | --- |
-| `pins_list` | read | List visible Pins with metadata and revisions |
-| `pins_find` | read | Bounded Pin search with sanitized previews |
-| `pin_add` | canonical write | Append a new Pin |
-| `pin_supersede` | canonical write | Replace one exact active Pin immutably |
-| `pin_unpin` | canonical write | Append a deleted lifecycle status |
-| `memory_list` | read | List visible Memory with metadata and revisions |
-| `memory_find` | read | Bounded Memory search with sanitized previews |
-| `memory_add` | canonical write | Append session/project Memory |
-| `memory_supersede` | canonical write | Replace one exact active Memory item immutably |
-| `memory_invalidate` | canonical write | Append an invalid lifecycle status |
-| `memory_expire` | canonical write | Append an expired lifecycle status |
-| `memory_sources` | read | List cross-session sources through volatile references |
-| `memory_source_exclude` | derived write | Exclude one source in local SQLite policy |
-| `memory_source_include` | derived write | Restore one source in local SQLite policy |
+| Action | Class | Parameters | Purpose |
+| --- | --- | --- | --- |
+| `pins_list` | read | optional `activeOnly`, `maxResults` | List visible Pins with metadata and revisions |
+| `pins_find` | read | `query`; optional `activeOnly`, `maxResults` | Bounded Pin search with sanitized previews |
+| `pin_add` | canonical write | `content`; optional `scope`, `classification` | Append a new Pin |
+| `pin_supersede` | canonical write | `id`, `targetRevision`, `content`; optional `classification` | Replace one exact active Pin immutably |
+| `pin_unpin` | canonical write | `id`, `targetRevision`; optional `reason` | Append a deleted lifecycle status |
+| `memory_list` | read | optional `activeOnly`, `maxResults` | List visible Memory with metadata and revisions |
+| `memory_find` | read | `query`; optional `activeOnly`, `maxResults` | Bounded Memory search with sanitized previews |
+| `memory_add` | canonical write | `content`; optional `scope`, `key`, `classification` | Append session/project Memory |
+| `memory_supersede` | canonical write | `id`, `targetRevision`, `content`; optional `classification` | Replace one exact active Memory item immutably |
+| `memory_invalidate` | canonical write | `id`, `targetRevision`; optional `reason` | Append an invalid lifecycle status |
+| `memory_expire` | canonical write | `id`, `targetRevision`; optional `reason` | Append an expired lifecycle status |
+| `memory_sources` | read | optional `maxResults` | List cross-session sources through volatile references |
+| `memory_source_exclude` | derived write | `id`, `targetRevision`; optional `reason` | Exclude one source in local SQLite policy |
+| `memory_source_include` | derived write | `id`, `targetRevision` | Restore one source in local SQLite policy |
+
+The transport schema stays flat for compatibility across supported providers, but DS4 applies the parameter matrix above before runtime access. A known action with an inapplicable field returns `unsupported-parameter`; a known action missing an action-required field returns `missing-required-parameter`. These diagnostics never identify or echo fields or values. Malformed transport input retains the generic `invalid-parameters` category. The output-only egress sentinel is checked before action-specific categorization so it remains `egress-placeholder` even when supplied in an inapplicable field.
 
 Read results are keyset-bounded and metadata-only. List results never include Pin content, Memory claims, keys, paths, source session IDs, reasons, complete errors, or totals requiring an unbounded count. Find results may include a short provider-safe preview in text; `details.items` remains metadata-only.
 

package/docs/DOGFOODING_0.3.0_ALPHA.md

@@ -1,6 +1,6 @@
 # Dogfooding DS4 0.3.0 Alpha
 
-This runbook validates the published `ds4-context-engine@0.3.0-alpha.2` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
+This runbook validates the published `ds4-context-engine@0.3.0-alpha.4` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
 
 The primary target is the model-callable `context_persistence` surface. Pi JSONL must remain canonical and append-only, SQLite must remain rebuildable, and no model-callable write may occur without a fresh positive local UI decision.
 
@@ -9,7 +9,9 @@ The primary target is the model-callable `context_persistence` surface. Pi JSONL
 - Use the exact prerelease version, not the mutable `alpha` dist-tag.
 - Use a disposable trusted project and a dedicated session directory.
 - Use synthetic, non-secret Pin/Memory content. Local TUI dialogs and JSON event streams may display current tool arguments.
-- Published alpha.1 had a retry limitation: copying `[omitted-by-ds4-egress-policy]` from sanitized history could present a confirmation for the literal marker. Alpha.2 reserves that output-only marker and must reject it as `egress-placeholder` before runtime access, confirmation, canonical append, or derived-policy update.
+- Published alpha.1 had a retry limitation: copying `[omitted-by-ds4-egress-policy]` from sanitized history could present a confirmation for the literal marker. Alpha.2 and later reserve that output-only marker and must reject it as `egress-placeholder` before runtime access, confirmation, canonical append, or derived-policy update.
+- Alpha.3 and later keep exact-value summary validation strict. An unrepaired failure may expose only its stage, issue code, repair category, and counts; disputed exact text must not appear in diagnostics.
+- Alpha.4 distinguishes unsupported and missing action parameters without echoing rejected fields or values. Routine successful compaction lifecycle events require `diagnostics.logLevel=debug`; actionable fallback warnings remain visible.
 - Do not use `--no-session` except for the explicit fail-closed test. Without a persistent Pi JSONL destination, both reads and writes return `runtime-unavailable`.
 - Do not retry `committed_projection_pending` or `indeterminate`. Inspect state with a read, `/context health`, or `/context rebuild-index` first.
 - Use `/context` only for local inspection and recovery. Mutations under test must go through `context_persistence` so the confirmation and provider-egress boundaries are exercised.
@@ -26,7 +28,7 @@ mkdir -p /tmp/ds4-alpha-dogfood
 cd /tmp/ds4-alpha-dogfood
 git init
 mkdir -p sessions evidence
-pi install -l npm:ds4-context-engine@0.3.0-alpha.2
+pi install -l npm:ds4-context-engine@0.3.0-alpha.4
 pi list
 pi --version
 ```
@@ -46,19 +48,19 @@ Record the Pi version, DS4 version, provider/model, mode, session name, expected
 Use unique non-sensitive values so duplicates from earlier runs cannot hide a failure. Example run label:
 
 ```text
-alpha2-run-01
+alpha4-run-01
 ```
 
 Example Pin:
 
 ```text
-For alpha2-run-01 verification, use Node.js 22 in this disposable project.
+For alpha4-run-01 verification, use Node.js 22 in this disposable project.
 ```
 
 Example Memory:
 
 ```text
-For alpha2-run-01, the synthetic release channel is amber.
+For alpha4-run-01, the synthetic release channel is amber.
 ```
 
 Never use real credentials, customer data, private paths, or production policy in dogfooding prompts.
@@ -76,7 +78,7 @@ Run these scenarios in order:
 
 1. Inspect `/context health`, `/context pins`, `/context memory`, and `/context privacy`.
 2. Send an ordinary suggestion without asking to persist it, for example: `The synthetic release channel amber seems useful.` No `context_persistence` call or confirmation dialog should appear.
-3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for alpha2-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
+3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for alpha4-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
 4. Ask the model to use `context_persistence` to list active Memory. Verify bounded metadata and no complete claim, key, reason, path, or raw error in the result.
 5. Explicitly request the synthetic Pin, but reject or close the confirmation dialog. Verify with `/context pins` that it was not created.
 6. Ask the model to retry using the sanitized value remaining in history. If it copies `[omitted-by-ds4-egress-policy]`, expect `rejected / egress-placeholder` before any new confirmation, runtime mutation, or append. If it asks for fresh text instead, record that safe routing result and run the exact-marker case from the JSON procedure.
@@ -115,7 +117,7 @@ Enter one JSON object per line on stdin. First request a read:
 Wait for the turn to end before sending the next prompt. For a write:
 
 ```json
-{"id":"write-1","type":"prompt","message":"Persist a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22 for verification."}
+{"id":"write-1","type":"prompt","message":"Persist a session Pin for alpha4-run-01 stating that this disposable project uses Node.js 22 for verification."}
 ```
 
 Pi should emit a request shaped like:
@@ -167,7 +169,7 @@ The read should complete. Then request a write:
 
 ```bash
 pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
-  "Use context_persistence to add a session Memory saying that alpha2-run-01 uses the synthetic channel amber."
+  "Use context_persistence to add a session Memory saying that alpha4-run-01 uses the synthetic channel amber."
 ```
 
 Expected behavior:
@@ -197,7 +199,7 @@ Write case:
 
 ```bash
 pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
-  "Use context_persistence to add a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22." \
+  "Use context_persistence to add a session Pin for alpha4-run-01 stating that this disposable project uses Node.js 22." \
   2>evidence/json-write.stderr.log | tee evidence/json-write.jsonl
 ```
 

package/docs/releases/0.3.0-alpha.3.md

@@ -1,6 +1,6 @@
 # DS4 Context Engine 0.3.0-alpha.3
 
-Status: release candidate on 2026-08-30; publication and tag pending.
+Status: published prerelease on 2026-08-30; tag `v0.3.0-alpha.3`.
 
 This coordinated prerelease hardens proactive compaction after investigation of an intermittent `unsupported-exact-value` fallback. It preserves strict exact-value grounding and Pi fallback behavior while making repair failures diagnosable without exposing disputed source text.
 
@@ -30,7 +30,7 @@ ds4-context-reference-adapter
 ds4-context-engine
 ```
 
-Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.3`. Publication uses the explicit npm `alpha` dist-tag so `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.3`. The packages were published manually under the explicit npm `alpha` dist-tag while `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
 
 ## Validation evidence
 
@@ -48,7 +48,9 @@ Local candidate verification on Node.js `26.5.1`:
 - `git diff --check`: passed.
 - Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, and persistence-tool contract files: unchanged.
 
-Exact registry verification, annotated tag creation, and GitHub prerelease creation remain pending until all three packages are published.
+- `npm run registry:check -- 0.3.0-alpha.3`: passed against all three exact published versions; `alpha` resolves to `0.3.0-alpha.3` and `latest` remains `0.2.0` for every package.
+
+Exact registry verification passed before the annotated tag and GitHub prerelease were created.
 
 ## Documentation
 

package/docs/releases/0.3.0-alpha.4.md

@@ -0,0 +1,61 @@
+# DS4 Context Engine 0.3.0-alpha.4
+
+Status: published prerelease on 2026-09-02; tag `v0.3.0-alpha.4`.
+
+This coordinated prerelease improves model-callable `context_persistence` guidance and categorical validation diagnostics, and removes routine successful compaction lifecycle events from normal `info` output. It preserves strict persistence confirmation, canonical Pi JSONL semantics, privacy-safe egress, compaction validation, and Pi fallback behavior.
+
+## Fixed
+
+- Describes the valid parameter set for every `context_persistence` action in the tool guidance and public action matrix.
+- Distinguishes a known action with an inapplicable field as `unsupported-parameter` and a missing action-required field as `missing-required-parameter`.
+- Keeps validation failures categorical: rejected field names and values are absent from current results, sanitized history, diagnostics, and logs.
+- Continues to reject the output-only egress sentinel before action-specific validation, including when it appears in an otherwise inapplicable field.
+- Emits routine `compaction.summary_graph_prepared` and `compaction.summary_graph_committed` lifecycle events only at `debug` instead of the default `info` level.
+- Preserves actionable compaction fallback, failure, persistence, and reconciliation events at `warn`; the proactive-threshold TUI notice remains user-visible because it explains why automatic compaction started.
+
+## Compatibility and persistence
+
+The flat tool schema, fourteen actions, `ds4-context-persistence-tool-v1`, and `ds4-context-persistence-result-v1` identifiers are unchanged. The new validation categories are metadata-only and do not add a write path or alter confirmation requirements.
+
+Pi JSONL remains canonical and append-only. All model-callable writes still require fresh local UI confirmation and fail closed without UI or canonical persistence. SQLite remains a disposable projection. Schema 15, migrations 1–15, `ds4-context-config-v1`, and `runtime-adapter-v1` are unchanged. Strict compaction validation and fallback to Pi default compaction are unchanged.
+
+## Package/version policy
+
+The coordinated version is `0.3.0-alpha.4` for:
+
+```text
+ds4-context-core
+ds4-context-reference-adapter
+ds4-context-engine
+```
+
+Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.4`. The packages were published manually under the explicit npm `alpha` dist-tag while `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+
+## Validation evidence
+
+Local candidate verification on Node.js `26.5.1`:
+
+- `npm ci`: passed.
+- `npm run check`: 64 files and 290 tests passed.
+- Focused persistence coverage: action-field guidance, unsupported/missing parameter categories, egress-sentinel priority, metadata-only failures, and sanitized historical diagnostics passed.
+- Focused compaction coverage: successful lifecycle events are absent at `info`, present at `debug`, and fallback remains `warn`.
+- `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
+- `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
+- `npm run latency:check -- <exact ds4-context-core@0.1.2>`: passed with ratio `0.923285`, at or below `1.10`.
+- `npm run pack:check`: verified core (203 files), reference adapter (7 files), and Pi adapter (61 files) in a clean consumer.
+- `npm pack --dry-run --json` for all three packages: passed with the same bounded inventories.
+- The committed candidate was replayed from detached clean checkout `6fcf514`; `npm ci`, the 64-file/290-test suite, quality, schema, package verification, tarball review, and `git diff --check` passed.
+- `git diff --check`: passed.
+- Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, and persistence result-contract files: unchanged.
+
+- `npm run registry:check -- 0.3.0-alpha.4`: passed against all three exact published versions; `alpha` resolves to `0.3.0-alpha.4` and `latest` remains `0.2.0` for every package.
+
+Exact registry verification passed before the annotated tag and GitHub prerelease were created.
+
+## Documentation
+
+- [`../CONTEXT_PERSISTENCE_TOOL.md`](../CONTEXT_PERSISTENCE_TOOL.md)
+- [`../COMPACTION.md`](../COMPACTION.md)
+- [`../PRIVACY.md`](../PRIVACY.md)
+- [`../RELEASING.md`](../RELEASING.md)
+- [`0.3.0-alpha.3.md`](0.3.0-alpha.3.md)

package/docs/releases/0.3.0-alpha.5.md

@@ -0,0 +1,58 @@
+# DS4 Context Engine 0.3.0-alpha.5
+
+Status: validated release candidate; not yet published.
+
+This coordinated prerelease adds overflow-safe hierarchical compaction for discarded source that cannot fit in one active-model summary request. It preserves strict summary validation, privacy sanitization, canonical Pi JSONL semantics, one final Pi compaction entry, and fail-closed fallback to Pi default compaction.
+
+## Added
+
+- Preflights the complete sanitized compaction prompt against the calibrated active-model input budget.
+- Partitions oversized discarded source at contiguous message boundaries while keeping matching tool calls and results in the same atomic group.
+- Generates and validates each segment only against that segment's sanitized source and deterministic file inventory.
+- Recursively aggregates ordered child summaries when one fan-in request cannot fit, preserving cumulative provenance and graph topology.
+- Bounds work to 32 segment requests, 64 aggregate requests, and 16 aggregate passes.
+- Exposes metadata-only diagnostics for input budget, whole-source prompt estimate, segment count, and aggregate-call count.
+- Categorizes provider failures as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, `transport`, `aborted`, or `provider-error` without surfacing provider payloads.
+
+## Safety and compatibility
+
+An individual message, atomic tool exchange, aggregate pair, or total operation that cannot fit within the bounded limits fails closed to Pi default compaction. Generated segment and aggregate summaries retain strict structural and exact-value validation; bounded unsupported-exact-value bullet repair remains unchanged. Rejected exact values and raw provider error details are never emitted in diagnostics.
+
+Pi receives one final canonical compaction entry. The complete prepared summary-graph batch is committed only after Pi installs that entry; failed attempts remain non-destructive. Pi JSONL stays canonical and append-only, while SQLite remains a disposable projection. Schema 15, migrations 1–15, `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, and `ds4-context-persistence-result-v1` are unchanged.
+
+## Package/version policy
+
+The coordinated version is `0.3.0-alpha.5` for:
+
+```text
+ds4-context-core
+ds4-context-reference-adapter
+ds4-context-engine
+```
+
+Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.5`. Publication must use the explicit npm `alpha` dist-tag so `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+
+## Validation evidence
+
+Local candidate verification on Node.js `26.5.1`:
+
+- `npm run check`: 65 files and 297 tests passed.
+- Focused compaction coverage includes three-segment fan-out, recursive fan-in, tool-exchange atomicity, exact contiguous source slicing, oversized indivisible-source fallback, and privacy-safe provider input-limit categorization.
+- `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
+- `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
+- `npm run latency:check -- <exact ds4-context-core@0.1.2>`: isolated clean-checkout retry passed with ratio `1.041564`, at or below `1.10`; an immediately preceding noisy sample measured `1.157479`, while the pre-commit local sample measured `0.939056`.
+- `npm run pack:check`: verified core (207 files), reference adapter (7 files), and Pi adapter (62 files) in a clean consumer.
+- `npm pack --dry-run --json` for all three packages passed with the same bounded inventories and no forbidden local/session files.
+- The committed candidate was replayed from detached clean checkout `8b609c0`; `npm ci`, the 65-file/297-test suite, quality, schema, package verification, tarball review, latency retry, and `git diff --check` passed.
+- `git diff --check`: passed.
+- Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, persistence confirmation, and persistence result-contract files are unchanged.
+
+Exact registry verification is required before tag or GitHub prerelease creation.
+
+## Documentation
+
+- [`../COMPACTION.md`](../COMPACTION.md)
+- [`../ARCHITECTURE.md`](../ARCHITECTURE.md)
+- [`../PRIVACY.md`](../PRIVACY.md)
+- [`../RELEASING.md`](../RELEASING.md)
+- [`0.3.0-alpha.4.md`](0.3.0-alpha.4.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.0-alpha.3",
+  "version": "0.3.0-alpha.5",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -58,7 +58,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.0-alpha.3"
+    "ds4-context-core": "0.3.0-alpha.5"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",

package/src/extension/commands.ts

@@ -734,6 +734,10 @@ function formatCompaction(diagnostics: RuntimeDiagnostics, preview: boolean): st
     `Last trigger:            ${compaction.trigger ?? "n/a"}`,
     `Summary ID:              ${compaction.summaryId ?? "n/a"}`,
     `Source entries:          ${count(compaction.sourceEntries)}`,
+    `Input budget:            ${count(compaction.inputBudgetTokens)}`,
+    `Whole-source prompt:     ${count(compaction.sourcePromptTokens)}`,
+    `Generated segments:      ${count(compaction.segmentCount)}`,
+    `Aggregate calls:         ${count(compaction.aggregateCalls)}`,
     `Validation:              ${compaction.validationStatus ?? "n/a"}`,
     `First kept entry:        ${compaction.firstKeptEntryId ?? "n/a"}`,
     `Tokens before:           ${count(compaction.tokensBefore)}`,

package/src/extension/context-persistence-contract.ts

@@ -4,12 +4,13 @@ export const CONTEXT_PERSISTENCE_TOOL_CONTRACT = "ds4-context-persistence-tool-v
 export const CONTEXT_PERSISTENCE_RESULT_CONTRACT = "ds4-context-persistence-result-v1" as const;
 export const CONTEXT_PERSISTENCE_EGRESS_SENTINEL = "[omitted-by-ds4-egress-policy]" as const;
 export const CONTEXT_PERSISTENCE_TOOL_NAME = "context_persistence" as const;
-export const CONTEXT_PERSISTENCE_DESCRIPTION = "Inspect DS4 Pins/Memory. Write only after an explicit user request; writes require local user confirmation." as const;
+export const CONTEXT_PERSISTENCE_DESCRIPTION = "Inspect DS4 Pins/Memory. Use action fields only. Writes require explicit user request and local UI confirmation." as const;
 export const CONTEXT_PERSISTENCE_PROMPT_SNIPPET = "Inspect or manage user-confirmed DS4 pins and durable memory" as const;
 export const CONTEXT_PERSISTENCE_PROMPT_GUIDELINES = [
   "Use context_persistence only to inspect DS4 persistent state or when the user explicitly requests a persistence mutation.",
   "After an explicit persistence request, call the write action directly; context_persistence itself obtains the required local UI confirmation, so do not ask for separate confirmation in chat.",
   "Never reuse an egress omission marker as tool input; use fresh user-provided text or ask the user to restate it.",
+  "Send only fields valid for the selected action: find requires query; add requires content; supersede requires id, targetRevision, and content; lifecycle and source-policy changes require id and targetRevision.",
   "Never create a pin or memory merely because information appears useful.",
   "Use pins for confirmed constraints or instructions that must remain prominent. Use memory for durable facts, decisions, and historical knowledge.",
   "Default new persistence to session scope. Use project or branch scope only when explicitly requested or unambiguous; durable Memory does not support branch scope.",
@@ -58,9 +59,17 @@ export const CONTEXT_PERSISTENCE_PARAMS = Type.Object({
   })),
   id: Type.Optional(Type.String({ minLength: 1, maxLength: 128 })),
   targetRevision: Type.Optional(Type.String({ minLength: 1, maxLength: 64 })),
-  content: Type.Optional(Type.String({ minLength: 1, maxLength: 20_000 })),
+  content: Type.Optional(Type.String({
+    minLength: 1,
+    maxLength: 20_000,
+    description: "Add/replace only",
+  })),
   key: Type.Optional(Type.String({ minLength: 1, maxLength: 256 })),
-  query: Type.Optional(Type.String({ minLength: 2, maxLength: 200 })),
+  query: Type.Optional(Type.String({
+    minLength: 2,
+    maxLength: 200,
+    description: "Find; required",
+  })),
   reason: Type.Optional(Type.String({
     minLength: 1,
     maxLength: 500,
@@ -114,6 +123,8 @@ const REQUIRED_FIELDS = {
 
 export type ContextPersistenceValidationCode =
   | "invalid-parameters"
+  | "missing-required-parameter"
+  | "unsupported-parameter"
   | "invalid-scope"
   | "egress-placeholder";
 
@@ -139,19 +150,19 @@ export function validateContextPersistenceParams(
 ): ContextPersistenceValidation {
   if (!params || !isAction(params.action)) return { ok: false, errorCode: "invalid-parameters" };
   const keys = Object.keys(params) as (keyof ContextPersistenceParams)[];
-  const allowed = new Set<keyof ContextPersistenceParams>(ALLOWED_FIELDS[params.action]);
-  if (keys.some((key) => !allowed.has(key))) {
-    return { ok: false, errorCode: "invalid-parameters" };
-  }
-  if (REQUIRED_FIELDS[params.action].some((key) => params[key] === undefined)) {
-    return { ok: false, errorCode: "invalid-parameters" };
-  }
   if (keys.some((key) => {
     const value = params[key];
     return typeof value === "string" && value.includes(CONTEXT_PERSISTENCE_EGRESS_SENTINEL);
   })) {
     return { ok: false, errorCode: "egress-placeholder" };
   }
+  const allowed = new Set<keyof ContextPersistenceParams>(ALLOWED_FIELDS[params.action]);
+  if (keys.some((key) => !allowed.has(key))) {
+    return { ok: false, errorCode: "unsupported-parameter" };
+  }
+  if (REQUIRED_FIELDS[params.action].some((key) => params[key] === undefined)) {
+    return { ok: false, errorCode: "missing-required-parameter" };
+  }
   if (params.scope === "branch" && params.action === "memory_add") {
     return { ok: false, errorCode: "invalid-scope" };
   }