ds4-context-engine 0.2.0 → 0.3.0-alpha.2

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** Stable `0.2.0` includes M0–M20 and release hardening: quality measurement, structural/hybrid retrieval, cross-session project memory, learned-ranking shadow evaluation, the runtime adapter kit, opt-in local KV reuse, schema-10 upgrades, long-session gates, and frozen 0.2 contracts. The maintenance line targets Pi `0.84.3`.
+> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Prerelease `0.3.0-alpha.2` hardens the confirmation-gated `context_persistence` tool by rejecting its output-only historical egress sentinel on input, without changing canonical Pin/Memory records, SQLite schema 15, or the reference history contract. npm `latest` remains `0.2.0`; the maintenance line targets Pi `0.84.3`.
 
 ## Why DS4
 
@@ -31,6 +31,7 @@ It provides:
 - trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
 - hierarchical, validated, non-destructive compaction summaries;
 - persistent pins and append-only durable memory stored canonically in Pi JSONL;
+- a bounded, metadata-only `context_persistence` tool with local confirmation for every model-callable write;
 - opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
 - content-addressed storage and bounded references for large tool results;
 - privacy classifications, secret redaction and provider-specific allow rules;
@@ -88,6 +89,14 @@ pi install npm:ds4-context-engine
 
 The stable `0.2.0` packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) use the same exact version. Both adapters require the matching core version.
 
+To dogfood the published alpha without replacing a global stable installation, pin it in a disposable project:
+
+```bash
+pi install -l npm:ds4-context-engine@0.3.0-alpha.2
+```
+
+Follow the [0.3 alpha dogfooding runbook](docs/DOGFOODING_0.3.0_ALPHA.md); use synthetic data and a dedicated session directory.
+
 ### Local checkout
 
 ```bash
@@ -198,6 +207,19 @@ Project configuration and project source indexing are disabled when Pi reports t
 
 Valid privacy classifications are `normal`, `internal`, `sensitive` and `local-only`.
 
+### LLM-callable tools
+
+DS4 registers two model-callable tools:
+
+| Tool | Purpose |
+| --- | --- |
+| `context_artifact_search` | Search a known DS4 artifact reference with bounded quoted excerpts |
+| `context_persistence` | Inspect Pins, Memory, and project-memory sources; perform explicitly requested persistence mutations |
+
+`context_persistence` read actions return bounded metadata and sanitized find previews. Every write requires a fresh local `ctx.ui.confirm()` decision. In print/JSON or any other no-UI mode, reads remain available and writes fail closed with `confirmation-required`. Sessions without a persistent Pi JSONL destination (for example `--no-session`) fail closed with `runtime-unavailable` before confirmation. Destructive writes require an exact ID or volatile source reference plus the `targetRevision` returned by a prior read; fuzzy writes are not supported.
+
+Canonical Pin and Memory changes append Pi custom entries and reconcile disposable SQLite projections. Project-memory source include/exclude is derived local SQLite policy and never appends a fake canonical entry. See [`docs/CONTEXT_PERSISTENCE_TOOL.md`](docs/CONTEXT_PERSISTENCE_TOOL.md).
+
 Learned-ranking feedback and local training are explicit:
 
 ```text
@@ -390,9 +412,11 @@ npm run typecheck
 npm test
 npm run check
 npm run quality:compare
+npm run schema:context-persistence
+npm run latency:check -- /path/to/exact/ds4-context-core@0.1.2
 npm run pack:check
 # Post-publication, with an exact version rather than a dist-tag:
-npm run registry:check -- 0.2.0
+npm run registry:check -- 0.3.0-alpha.2
 npm pack --dry-run
 npm pack --dry-run --workspace ds4-context-core
 npm pack --dry-run --workspace ds4-context-reference-adapter
@@ -430,6 +454,8 @@ scripts             package and release-readiness checks
 - [Project knowledge](docs/PROJECT_KNOWLEDGE.md)
 - [Artifacts](docs/ARTIFACTS.md)
 - [Memory and pins](docs/MEMORY_AND_PINS.md)
+- [Context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md)
+- [0.3 alpha dogfooding runbook](docs/DOGFOODING_0.3.0_ALPHA.md)
 - [Privacy](docs/PRIVACY.md)
 - [Model awareness](docs/MODEL_AWARENESS.md)
 - [Native continuation](docs/NATIVE_CONTINUATION.md)
@@ -442,6 +468,8 @@ scripts             package and release-readiness checks
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
 - [0.2.0 release notes](docs/releases/0.2.0.md)
 - [0.2.0-rc.1 release notes](docs/releases/0.2.0-rc.1.md)
+- [0.3.0-alpha.2 prerelease notes](docs/releases/0.3.0-alpha.2.md)
+- [0.3.0-alpha.1 prerelease notes](docs/releases/0.3.0-alpha.1.md)
 - [Architecture decisions](docs/ADR/README.md)
 - [Original development plan](DS4_Context_Engine_Extension_Piano_Sviluppo.md)
 
@@ -449,7 +477,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The [readiness record](docs/RELEASE_READINESS_0.2.0.md) maps every release gate to tests and operator commands. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Prerelease `0.3.0-alpha.2` hardens the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md) delivered in alpha.1 while retaining the stable canonical/configuration/SQLite/runtime contracts. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ADR/README.md

@@ -56,5 +56,10 @@ The initial decisions from the development plan are accepted:
 | 050 | Retry rejected stale continuation state once with the complete managed replay before exposing output | Accepted |
 | 051 | Keep continuation handles volatile and exclude them from manifests, logs, and DS4 persistence | Accepted |
 | 052 | Compile `ds4-context-core` as ESM and keep the Pi adapter dependency one-way | Accepted |
+| 053 | Dispatch `context_persistence` directly through `Ds4ContextRuntime`, never through slash-command parsing | Accepted |
+| 054 | Use fresh local Pi UI confirmation as the V1 authorization boundary for every model-callable write | Accepted |
+| 055 | Enforce a dedicated metadata-only tool egress guard independently of `privacy.enabled` | Accepted |
+| 056 | Keep project-memory source exclusion as disposable derived SQLite policy | Accepted |
+| 057 | Derive mutation provenance from the active branch and exclude model-supplied source IDs from V1 | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/ARCHITECTURE.md

@@ -156,6 +156,14 @@ session_tree / shutdown
   -> transactional reconciliation from canonical JSONL, memory/pin replay, artifact regeneration and forced project rescan
 ```
 
+## Context persistence tool boundary
+
+`context_persistence` is a Pi-adapter capability, not a portable-core or reference-adapter API. Its controller calls `Ds4ContextRuntime` directly rather than relaying slash commands. Read actions use bounded keyset/visible-item APIs and emit only allowlisted metadata plus policy-sanitized find previews. Every LLM-callable write fails closed without Pi UI and requires a fresh local confirmation; model-provided consent fields are not accepted.
+
+Pin and Memory mutations resolve provenance from the active Pi branch, revalidate it after confirmation, append the existing `ds4-context-pin-v1` or `ds4-context-memory-v1` record through `pi.appendEntry()`, and then reconcile the disposable projection. Exact revisions bind target state to active session/project/branch context. Project-source include/exclude resolves a volatile `sourceRef` to an internal session identity and updates only coordinated SQLite policy—never Pi JSONL.
+
+Historical tool arguments and results are a provider-egress surface even when general privacy is disabled. A dedicated guard removes content, query, key, reason, paths and raw errors while preserving only action linkage and safe IDs/revisions. Provenance IDs and source paths are runtime-derived; the model cannot supply them in the V1 schema. Post-append failures distinguish an indeterminate append from a known canonical commit with pending projection, so callers are never encouraged to retry blindly.
+
 ## Boundaries
 
 Dependency direction is one-way:
@@ -198,7 +206,7 @@ Adapters may import core exports. Core source must never import `@earendil-works
 
 ## Canonical and derived state
 
-The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, append-only classified memory/pin mutations and metadata-only learned-ranking feedback/replay labels; live files remain canonical for project knowledge. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, source-hash/model-keyed vectors, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, calibration data, and bounded metadata-only quality samples. The checksummed learned-ranking model is a separate disposable local artifact reconstructed from canonical labels; it contains bounded weights and aggregate gate metadata, never raw text. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
+The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, append-only classified memory/pin mutations and metadata-only learned-ranking feedback/replay labels; live files remain canonical for project knowledge. Volatile `context_persistence` source-reference/revision maps and derived project-source exclusion policy are local process/SQLite state and are never canonical. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, source-hash/model-keyed vectors, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, calibration data, and bounded metadata-only quality samples. The checksummed learned-ranking model is a separate disposable local artifact reconstructed from canonical labels; it contains bounded weights and aggregate gate metadata, never raw text. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
 
 ## Lifecycle
 

package/docs/CONTEXT_PERSISTENCE_TOOL.md

@@ -0,0 +1,131 @@
+# Context Persistence Tool
+
+`context_persistence` is DS4's bounded model-callable interface for inspecting and explicitly updating Persistent Pins, Durable Memory, and cross-session project-memory source policy. It is available in the Pi adapter only; the portable core and reference adapter do not expose this Pi-specific tool.
+
+Contract identifiers:
+
+```text
+tool:   ds4-context-persistence-tool-v1
+result: ds4-context-persistence-result-v1
+```
+
+The tool declares `executionMode: "sequential"`. This orders sibling calls from one model response; SQLite writes still use the shared coordinator and every targeted mutation still checks its revision.
+
+## Pins and Memory
+
+Pins are confirmed constraints or instructions that should remain prominent. They may use `session`, `branch`, or trusted `project` scope. Branch Pins can remain lifecycle-active while not applying to the active branch.
+
+Memory is quoted durable factual, decision, or historical data. It supports `session` and trusted `project` scope, never `branch` scope. DS4 does not automatically extract either concept from ordinary conversation.
+
+## Actions
+
+| Action | Class | Purpose |
+| --- | --- | --- |
+| `pins_list` | read | List visible Pins with metadata and revisions |
+| `pins_find` | read | Bounded Pin search with sanitized previews |
+| `pin_add` | canonical write | Append a new Pin |
+| `pin_supersede` | canonical write | Replace one exact active Pin immutably |
+| `pin_unpin` | canonical write | Append a deleted lifecycle status |
+| `memory_list` | read | List visible Memory with metadata and revisions |
+| `memory_find` | read | Bounded Memory search with sanitized previews |
+| `memory_add` | canonical write | Append session/project Memory |
+| `memory_supersede` | canonical write | Replace one exact active Memory item immutably |
+| `memory_invalidate` | canonical write | Append an invalid lifecycle status |
+| `memory_expire` | canonical write | Append an expired lifecycle status |
+| `memory_sources` | read | List cross-session sources through volatile references |
+| `memory_source_exclude` | derived write | Exclude one source in local SQLite policy |
+| `memory_source_include` | derived write | Restore one source in local SQLite policy |
+
+Read results are keyset-bounded and metadata-only. List results never include Pin content, Memory claims, keys, paths, source session IDs, reasons, complete errors, or totals requiring an unbounded count. Find results may include a short provider-safe preview in text; `details.items` remains metadata-only.
+
+## Read before a targeted write
+
+Supersede, lifecycle, include, and exclude operations require:
+
+```text
+exact id or sourceRef
+targetRevision from a prior read
+```
+
+A revision is a process-local HMAC token bound to the item fingerprint and active session/project/branch context. It expires and is invalid after restart. A changed target, switched context, expired token, unknown reference, or non-active target is rejected. The tool never converts a fuzzy query or a high-scoring match into a write target.
+
+`memory_sources` returns `sourceRef`, not a session ID or file path. The mapping is process-local, TTL/cap-bounded, and never persisted. The opaque reference itself may remain in Pi's normal tool-result history so a subsequent exact call can use it.
+
+## Confirmation and no-UI behavior
+
+Every model-callable write requires a fresh local Pi UI decision:
+
+```ts
+await ctx.ui.confirm("DS4 Context Persistence", message)
+```
+
+The dialog identifies the action and persistence class. Canonical add/supersede dialogs show bounded full content and effective classification; lifecycle dialogs show a bounded local target preview and optional reason. Source-policy dialogs show only the volatile reference and safe status/counters. Confirmation text is never copied into the tool result or logs.
+
+A model cannot provide `confirmed=true`; it is not in the schema. If `ctx.hasUI` is false, all writes return:
+
+```text
+outcome=unavailable
+errorCode=confirmation-required
+```
+
+Reads remain available. Refusal, dialog closure, or abort before dispatch causes no write. Pi `0.84.3` RPC can advertise UI capability even when no client is currently answering UI requests. In that case the confirmation request remains pending in Pi's RPC transport; DS4 does not infer consent or refusal, and no append occurs before a positive response. RPC clients that expose UI capability must answer the request explicitly. If the active Pi session has no persistent JSONL destination (for example `--no-session`), reads and writes both fail closed with `runtime-unavailable`; no confirmation is shown and no canonical commit is claimed.
+
+## Canonical and derived persistence
+
+Pin and Memory mutations call `Ds4ContextRuntime`, append the existing versioned custom entry through `pi.appendEntry()`, and reconcile SQLite:
+
+```text
+ds4-context-pin-v1
+ds4-context-memory-v1
+```
+
+The tool never inserts canonical Pin or Memory state directly into SQLite and never rewrites Pi JSONL. Provenance is derived from the latest preceding user message on the active branch, not from model-supplied source IDs, and is revalidated after confirmation.
+
+Project-memory source exclusion is intentionally different. It updates `project_memory_source_exclusions` through the coordinated runtime repository and reports `persistenceClass=derived-local-policy`. It does not append a fake Pi entry. Deleting `context.db` resets this policy; replay restores source contributions from unchanged sibling JSONL.
+
+## Privacy and provider policy
+
+The tool enforces its own egress policy even when general privacy is disabled:
+
+- current and historical content, query, key, reason, preview, path, source identity, and raw errors are removed unless explicitly allowed;
+- the fixed historical omission sentinel is output-only; any incoming string argument containing it is rejected as `egress-placeholder` before runtime access, confirmation, or persistence, so retries must use fresh user-provided text;
+- results use deterministic metadata-only templates;
+- marker/credential detection may raise a mutation classification;
+- an explicit supersede classification cannot lower the target's effective protection;
+- remote-disallowed or `local-only` writes are rejected before append;
+- provider/trust/provenance/target state is checked again after confirmation.
+
+Selecting `local-only` does not prove that a remote model never saw the original message or tool arguments. Use the direct `/context` command surface or a verified local provider when data must never be disclosed remotely.
+
+## Outcomes and recovery
+
+| Outcome | Meaning |
+| --- | --- |
+| `ok` | Read succeeded, add was duplicate, or source policy already matched |
+| `committed` | Canonical append/materialization or derived policy update succeeded |
+| `rejected` | Validation, policy, trust, provenance, conflict, target, or revision check failed |
+| `cancelled` | User refusal/dialog closure or abort before dispatch |
+| `unavailable` | Runtime/capability/UI unavailable; no commit is claimed |
+| `committed_projection_pending` | Canonical append succeeded but projection/result materialization did not complete safely |
+| `indeterminate` | The append call did not return, so completion cannot be established |
+
+Never automatically retry `committed_projection_pending` or `indeterminate`. Inspect with a read, `/context health`, or `/context rebuild-index` first. SQLite is disposable; canonical Pin/Memory state reappears after replay. Source exclusions intentionally do not.
+
+## `/context` relationship
+
+`/context` remains the direct local administrative and inspection surface. It can show local detail that must not enter a provider-visible tool result. `context_persistence` calls the same runtime mutation primitives but has a narrower schema, automatic branch provenance, exact-revision requirements, confirmation on every write, and a dedicated historical egress guard.
+
+Useful diagnostics:
+
+```text
+/context pins
+/context memory
+/context memory sources
+/context privacy
+/context health
+/context rebuild-index
+```
+
+## Alpha dogfooding
+
+Use the published package with synthetic data and exercise TUI, RPC, print, and JSON behavior through the dedicated [`0.3 alpha dogfooding runbook`](DOGFOODING_0.3.0_ALPHA.md). The runbook distinguishes RPC's UI request/response bridge from genuinely no-UI print/JSON modes and includes a metadata-only canonical append audit.

package/docs/DOGFOODING_0.3.0_ALPHA.md

@@ -0,0 +1,304 @@
+# Dogfooding DS4 0.3.0 Alpha
+
+This runbook validates the published `ds4-context-engine@0.3.0-alpha.2` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
+
+The primary target is the model-callable `context_persistence` surface. Pi JSONL must remain canonical and append-only, SQLite must remain rebuildable, and no model-callable write may occur without a fresh positive local UI decision.
+
+## Safety and scope
+
+- Use the exact prerelease version, not the mutable `alpha` dist-tag.
+- Use a disposable trusted project and a dedicated session directory.
+- Use synthetic, non-secret Pin/Memory content. Local TUI dialogs and JSON event streams may display current tool arguments.
+- Published alpha.1 had a retry limitation: copying `[omitted-by-ds4-egress-policy]` from sanitized history could present a confirmation for the literal marker. Alpha.2 reserves that output-only marker and must reject it as `egress-placeholder` before runtime access, confirmation, canonical append, or derived-policy update.
+- Do not use `--no-session` except for the explicit fail-closed test. Without a persistent Pi JSONL destination, both reads and writes return `runtime-unavailable`.
+- Do not retry `committed_projection_pending` or `indeterminate`. Inspect state with a read, `/context health`, or `/context rebuild-index` first.
+- Use `/context` only for local inspection and recovery. Mutations under test must go through `context_persistence` so the confirmation and provider-egress boundaries are exercised.
+- Never edit a Pi session JSONL file during the run.
+
+Recommended minimum before promoting the alpha: three normal work sessions, two process restarts, one branch change, one projection rebuild, the TUI/RPC/print/JSON matrix, one configured remote provider, and—when available—one verified local provider.
+
+## Isolated setup
+
+Pi packages execute with the user's full permissions. Review the package before installation.
+
+```bash
+mkdir -p /tmp/ds4-alpha-dogfood
+cd /tmp/ds4-alpha-dogfood
+git init
+mkdir -p sessions evidence
+pi install -l npm:ds4-context-engine@0.3.0-alpha.2
+pi list
+pi --version
+```
+
+The `-l` installation is project-local and the exact npm version is pinned. Run Pi from this directory. Use `--approve` only after trusting this disposable project; non-interactive modes cannot show the project-trust dialog.
+
+Use the same dedicated session directory throughout:
+
+```bash
+export DS4_DOGFOOD_SESSIONS="$PWD/sessions"
+```
+
+Record the Pi version, DS4 version, provider/model, mode, session name, expected result, observed result, and pass/fail status for every scenario.
+
+## Synthetic test data
+
+Use unique non-sensitive values so duplicates from earlier runs cannot hide a failure. Example run label:
+
+```text
+alpha2-run-01
+```
+
+Example Pin:
+
+```text
+For alpha2-run-01 verification, use Node.js 22 in this disposable project.
+```
+
+Example Memory:
+
+```text
+For alpha2-run-01, the synthetic release channel is amber.
+```
+
+Never use real credentials, customer data, private paths, or production policy in dogfooding prompts.
+
+## TUI procedure
+
+Start a persistent interactive session:
+
+```bash
+pi --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  --name "ds4-alpha-tui"
+```
+
+Run these scenarios in order:
+
+1. Inspect `/context health`, `/context pins`, `/context memory`, and `/context privacy`.
+2. Send an ordinary suggestion without asking to persist it, for example: `The synthetic release channel amber seems useful.` No `context_persistence` call or confirmation dialog should appear.
+3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for alpha2-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
+4. Ask the model to use `context_persistence` to list active Memory. Verify bounded metadata and no complete claim, key, reason, path, or raw error in the result.
+5. Explicitly request the synthetic Pin, but reject or close the confirmation dialog. Verify with `/context pins` that it was not created.
+6. Ask the model to retry using the sanitized value remaining in history. If it copies `[omitted-by-ds4-egress-policy]`, expect `rejected / egress-placeholder` before any new confirmation, runtime mutation, or append. If it asks for fresh text instead, record that safe routing result and run the exact-marker case from the JSON procedure.
+7. Request the Pin again with fresh synthetic text and accept it. Ask the model to list Pins, then use the exact returned Pin ID and `targetRevision` to unpin it in the same process. Accept the destructive confirmation. Fuzzy targeting must not be used.
+8. With a remote provider, request a `local-only` Pin. Expect provider-policy denial before confirmation and no append.
+9. Restart Pi and continue the session:
+
+   ```bash
+   pi --approve --session-dir "$DS4_DOGFOOD_SESSIONS" -c
+   ```
+
+   Verify the accepted Memory remains visible. Revision handles from the previous process are intentionally invalid; perform a fresh read before any targeted write.
+10. Run `/context rebuild-index`, then verify the same canonical Memory/Pin lifecycle state is reconstructed.
+
+TUI passes when accepted writes append once, refusal/closure appends nothing, destructive writes require an exact fresh revision, ordinary conversation does not persist, and rebuild preserves canonical state.
+
+## RPC procedure
+
+RPC mode exposes extension dialogs through a JSON request/response protocol. It reports `ctx.hasUI=true` because a client can answer those requests; the client is the UI bridge.
+
+Start a persistent RPC process from the disposable project:
+
+```bash
+pi --mode rpc --approve \
+  --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  --name "ds4-alpha-rpc" \
+  2>evidence/rpc.stderr.log
+```
+
+Enter one JSON object per line on stdin. First request a read:
+
+```json
+{"id":"read-1","type":"prompt","message":"Use context_persistence with action pins_list to inspect active Pins. Do not perform a write."}
+```
+
+Wait for the turn to end before sending the next prompt. For a write:
+
+```json
+{"id":"write-1","type":"prompt","message":"Persist a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22 for verification."}
+```
+
+Pi should emit a request shaped like:
+
+```json
+{"type":"extension_ui_request","id":"<dynamic-id>","method":"confirm","title":"DS4 Context Persistence","message":"..."}
+```
+
+After inspecting the request, approve it with the exact dynamic ID:
+
+```json
+{"type":"extension_ui_response","id":"<dynamic-id>","confirmed":true}
+```
+
+Expect the `context_persistence` tool result to report a committed outcome without echoing complete Pin content. Repeat with a different synthetic value and reject it:
+
+```json
+{"type":"extension_ui_response","id":"<dynamic-id>","confirmed":false}
+```
+
+The rejected call must report cancellation and append nothing. `{"cancelled":true}` is also a valid dialog dismissal response.
+
+### RPC without a responding UI client
+
+Start a separate disposable RPC process, request a write, and do not send an `extension_ui_response`. In Pi `0.84.3`, the confirmation remains pending because RPC still advertises UI capability. This is not converted to `confirmation-required`; no append may occur before a positive response. Terminate the disposable process after recording the pending request, then inspect the session from TUI.
+
+### RPC without a persistent session
+
+Start a separate process:
+
+```bash
+pi --mode rpc --approve --no-session
+```
+
+Send a read and a write prompt. Both must return `runtime-unavailable`; no `extension_ui_request` should be emitted and no canonical commit should be claimed.
+
+RPC passes when positive confirmation commits once, negative/cancelled confirmation appends nothing, an unanswered dialog remains pending without append, `--no-session` fails before confirmation, and result content/details remain bounded and metadata-only.
+
+## Print-mode procedure
+
+Print mode has no extension UI. Keep session persistence enabled:
+
+```bash
+pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence with action memory_list to inspect active Memory. Do not write."
+```
+
+The read should complete. Then request a write:
+
+```bash
+pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence to add a session Memory saying that alpha2-run-01 uses the synthetic channel amber."
+```
+
+Expected behavior:
+
+```text
+outcome=unavailable
+errorCode=confirmation-required
+```
+
+No dialog can appear and no canonical custom entry may be appended. The assistant's final wording can vary; use JSON mode or the canonical audit below when the exact tool envelope is needed. If the model does not call the tool, record that separately as a routing observation and repeat with the explicit action name to isolate runtime behavior.
+
+Adding `--no-session` changes the expected error to `runtime-unavailable` for both reads and writes.
+
+## JSON event-stream procedure
+
+JSON mode is also non-interactive, but it exposes authoritative tool lifecycle events. Capture the complete local stream; it may include current non-secret tool arguments.
+
+Read case:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence with action pins_list to inspect active Pins. Do not write." \
+  2>evidence/json-read.stderr.log | tee evidence/json-read.jsonl
+```
+
+Write case:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence to add a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22." \
+  2>evidence/json-write.stderr.log | tee evidence/json-write.jsonl
+```
+
+Reserved historical-placeholder regression:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Call context_persistence exactly once with action pin_add, scope session, classification normal, and content exactly [omitted-by-ds4-egress-policy]." \
+  2>evidence/json-placeholder.stderr.log | tee evidence/json-placeholder.jsonl
+```
+
+Extract tool completions:
+
+```bash
+jq -c '
+  select(.type == "tool_execution_end" and .toolName == "context_persistence")
+  | {isError, result}
+' evidence/json-*.jsonl
+```
+
+The read should succeed. The ordinary write must return `confirmation-required` and append nothing. The placeholder case must return `rejected / egress-placeholder`, not request confirmation, and append nothing. Inspect `result.content` and `result.details` for bounded allowlisted metadata; they must not echo complete content, claims, keys, reasons, paths, source-session identity, confirmation text, or raw errors.
+
+The local `tool_execution_start.args` event can contain the current synthetic arguments supplied to the tool. That local event is not the provider-facing result contract, which is why dogfooding must use non-sensitive data and evidence files must not be published blindly.
+
+## Canonical append audit
+
+List only metadata for canonical Pin/Memory entries in the dedicated sessions:
+
+```bash
+find "$DS4_DOGFOOD_SESSIONS" -name '*.jsonl' -print0 \
+  | xargs -0 -r jq -r '
+      select(
+        .type == "custom"
+        and (
+          .customType == "ds4-context-pin-v1"
+          or .customType == "ds4-context-memory-v1"
+        )
+      )
+      | [.customType, .id, .timestamp, (.data.operation // "unknown")]
+      | @tsv
+    '
+```
+
+Expected invariants:
+
+- each accepted canonical add/supersede/status operation contributes exactly one append-only custom entry;
+- cancelled, rejected, unavailable, and unanswered-confirmation operations contribute none;
+- print/JSON writes contribute none;
+- `--no-session` contributes none;
+- source include/exclude policy contributes no Pin/Memory custom entry because it is derived local SQLite policy;
+- rebuild changes projections, not the JSONL mutation sequence.
+
+Do not publish the session files: they are canonical local history and may contain prompt/tool argument text even when tool results are metadata-only.
+
+## Extended provider and lifecycle matrix
+
+After the basic mode matrix passes, repeat the relevant TUI/RPC cases with:
+
+- a configured remote provider;
+- a verified local provider whose exact provider ID is listed in `privacy.localProviders`;
+- privacy enabled and disabled;
+- a provider switch between read and targeted write;
+- a branch switch between read and targeted write;
+- trusted and untrusted project state;
+- two simultaneous Pi sessions using the shared SQLite database;
+- cross-session project Memory when `memory.crossSession` is explicitly enabled.
+
+Provider, trust, branch, provenance, capability, target state, and classification changes after confirmation must fail safely. A model-supplied `local-only` classification is never evidence that earlier input stayed local.
+
+## Result record
+
+Use one record per scenario:
+
+```text
+Run ID:
+Date:
+Pi version:
+DS4 exact version:
+Provider/model:
+Mode and session persistence:
+Scenario:
+Expected outcome:
+Observed outcome:
+Confirmation shown/answered:
+Canonical entries before/after:
+Projection/rebuild observation:
+Result leak check:
+Pass/fail:
+Issue/reference:
+```
+
+## Promotion criteria
+
+Do not promote the alpha if any run shows:
+
+- a write without a fresh positive local UI decision;
+- canonical JSONL rewrite, loss, duplication, or a false commit claim;
+- complete persistent content or prohibited metadata in provider-facing results/history;
+- fuzzy destructive targeting or acceptance of a stale revision;
+- failure to reconstruct canonical state from JSONL;
+- an actionable warning hidden as routine debug output;
+- a reproducible regression above the release latency/quality/schema gates.
+
+A missing live local-provider run should remain explicitly recorded rather than inferred from remote-provider or automated-test results.

package/docs/MEMORY_AND_PINS.md

@@ -1,6 +1,6 @@
 # Memory and Persistent Pins
 
-M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical. M17 optionally reconstructs explicit project-scoped mutations from sibling Pi sessions for the same trusted canonical project.
+DS4 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical. Cross-session replay optionally reconstructs explicit project-scoped mutations from sibling Pi sessions for the same trusted canonical project.
 
 ## Authority and scope
 
@@ -37,7 +37,7 @@ Pins remain subordinate to system/developer instructions. Memory tells the model
 
 Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust. Source exclusion affects only project-scoped contributions and persists until `include`; the active session cannot be newly excluded but can restore a prior exclusion.
 
-Mutations are manual-first. Repeating the same normalized pin/claim returns its existing ID without appending another entry.
+Mutations are manual-first: persistence requires an explicit user request, and every LLM-tool write additionally requires local UI confirmation. DS4 never harvests ordinary conversation into Pins or Memory automatically. `/context` remains the direct local command surface; `context_persistence` is the bounded model-callable surface and requires a prior exact read plus revision for destructive operations. Repeating the same normalized pin/claim returns its existing ID without appending another entry.
 
 ## Canonical append-only mutations
 
@@ -63,7 +63,7 @@ No row is silently overwritten. SQLite mutation and materialized tables are disp
 3. replays all known mutations in timestamp + canonical entry order;
 4. rebuilds memory, pin, source, lifecycle, and FTS rows transactionally.
 
-Deleting `context.db` and reopening the canonical source session reconstructs its state. With `memory.crossSession: true`, DS4 discovers bounded sibling `.jsonl` files, accepts only headers whose `cwd` resolves to the exact trusted canonical project identity, incrementally indexes each source, and reconstructs project items without opening every session manually. No claim is extracted from ordinary conversation text.
+Deleting `context.db` and reopening the canonical source session reconstructs its state. With `memory.crossSession: true`, DS4 discovers bounded sibling `.jsonl` files, accepts only headers whose `cwd` resolves to the exact trusted canonical project identity, incrementally indexes each source, and reconstructs project items without opening every session manually. No claim is extracted from ordinary conversation text. Source exclusions are deliberately derived local policy: deleting the database removes them, replay restores the source from unchanged canonical JSONL, and no source-policy operation rewrites a session file.
 
 ## Supersession and contradiction handling
 

package/docs/PRIVACY.md

@@ -34,7 +34,7 @@ Persistent pin and memory commands also accept an explicit classification:
 
 The classification is part of the canonical Pi custom-entry mutation and survives SQLite deletion, session resume, branch changes, and compaction. Existing unclassified mutations use `privacy.defaultClassification` at selection time.
 
-Automatic sensitivity classification remains disabled. M10 uses explicit markers, explicit pin/memory metadata, and the configured default.
+Automatic harvesting/classification of ordinary conversation remains disabled. Persistent commands use explicit markers, explicit Pin/Memory metadata, and the configured default. For a confirmed `context_persistence` content-bearing mutation, marker and credential-like detection may only raise the stored classification floor; it can never lower an existing target classification.
 
 ## Provider destination and allow rules
 
@@ -91,6 +91,14 @@ Pi's fallback compactor is still covered by the final provider-payload hook.
 
 The local content-addressed object may retain exact restricted bytes because Pi JSONL is canonical and object hashes require exact recovery. Artifact references persist the derived classification in `metadata_json`. Context selection hides prohibited tool results before offload/reference injection. `context_artifact_search` applies the stored artifact classification to every returned excerpt; a remote request receives no matches/content for a prohibited artifact.
 
+### Context persistence tool egress
+
+`context_persistence` treats both its current result and historical Pi tool-call/result records as provider-egress surfaces independently of `privacy.enabled`. List/source results contain only bounded IDs or volatile references, scope/lifecycle/classification/timestamps, revisions and safe counters. Find previews are sanitized across the complete bounded source before Unicode-scalar truncation; prohibited previews become metadata-only omissions. Mutation results never echo content, claim, key or reason.
+
+A dedicated historical guard preserves provider-specific tool-call/result linkage while replacing `content`, `query`, `key`, `reason`, unknown fields, raw previews and malformed payloads with the fixed omission sentinel `[omitted-by-ds4-egress-policy]`. That sentinel is output-only and reserved: any incoming `context_persistence` string argument containing it is rejected with `egress-placeholder` before runtime lookup, UI confirmation, canonical append, or derived-policy update. A follow-up call must use fresh user-provided text rather than copying the historical placeholder. IDs and revisions must match their opaque grammars; paths, source session identity, complete errors and UI confirmation text are never copied. Local `sourceRef` mappings, revision HMAC secrets and target fingerprints are volatile and contain no content.
+
+Every write is policy-checked before local confirmation and checked again immediately before dispatch, covering provider or trust changes while the dialog is open. `local-only` supplied by a model is not proof that earlier input stayed local; remote-denied writes return only `provider-policy-denied` and do not append.
+
 ### Final provider payload
 
 `before_provider_request` runs after provider-specific serialization. DS4 recursively checks known provider content containers (`system`, `messages`, `input`, `contents`, `context`, tool descriptions/arguments, and related text fields), strips classification markers, removes prohibited blocks, and redacts credential-like values. Structural provider fields remain unchanged.