ds4-context-engine 0.3.0-alpha.5 → 0.3.0-beta.1

package/docs/DOGFOODING_0.3.0_BETA.md

@@ -0,0 +1,331 @@
+# Dogfooding DS4 0.3.0 Beta
+
+This runbook validates the published `ds4-context-engine@0.3.0-beta.1` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
+
+The primary target is the model-callable `context_persistence` surface. Pi JSONL must remain canonical and append-only, SQLite must remain rebuildable, and no model-callable write may occur without a fresh positive local UI decision.
+
+## Safety and scope
+
+- Use the exact prerelease version, not the mutable `beta` dist-tag.
+- Use a disposable trusted project and a dedicated session directory.
+- Use synthetic, non-secret Pin/Memory content. Local TUI dialogs and JSON event streams may display current tool arguments.
+- Published alpha.1 had a retry limitation: copying `[omitted-by-ds4-egress-policy]` from sanitized history could present a confirmation for the literal marker. Alpha.2 and later reserve that output-only marker and must reject it as `egress-placeholder` before runtime access, confirmation, canonical append, or derived-policy update.
+- Alpha.3 and later keep exact-value summary validation strict. An unrepaired failure may expose only its stage, issue code, repair category, and counts; disputed exact text must not appear in diagnostics.
+- Alpha.4 distinguishes unsupported and missing action parameters without echoing rejected fields or values. Routine successful compaction lifecycle events require `diagnostics.logLevel=debug`; actionable fallback warnings remain visible.
+- Alpha.5 partitions an oversized compaction source into bounded atomic segments and recursively aggregates validated child summaries. An indivisible message or tool exchange still fails closed to Pi default compaction, and provider failures expose only a metadata category.
+- Do not use `--no-session` except for the explicit fail-closed test. Without a persistent Pi JSONL destination, both reads and writes return `runtime-unavailable`.
+- Do not retry `committed_projection_pending` or `indeterminate`. Inspect state with a read, `/context health`, or `/context rebuild-index` first.
+- Use `/context` only for local inspection and recovery. Mutations under test must go through `context_persistence` so the confirmation and provider-egress boundaries are exercised.
+- Never edit a Pi session JSONL file during the run.
+
+Recommended minimum before promoting the beta: three normal work sessions, two process restarts, one branch change, one projection rebuild, the TUI/RPC/print/JSON matrix, one configured remote provider, and—when available—one verified local provider.
+
+## Isolated setup
+
+Pi packages execute with the user's full permissions. Review the package before installation.
+
+```bash
+mkdir -p /tmp/ds4-beta-dogfood
+cd /tmp/ds4-beta-dogfood
+git init
+mkdir -p sessions evidence
+pi install -l npm:ds4-context-engine@0.3.0-beta.1
+pi list
+pi --version
+```
+
+The `-l` installation is project-local and the exact npm version is pinned. Run Pi from this directory. Use `--approve` only after trusting this disposable project; non-interactive modes cannot show the project-trust dialog.
+
+Use the same dedicated session directory throughout:
+
+```bash
+export DS4_DOGFOOD_SESSIONS="$PWD/sessions"
+```
+
+Record the Pi version, DS4 version, provider/model, mode, session name, expected result, observed result, and pass/fail status for every scenario.
+
+## Synthetic test data
+
+Use unique non-sensitive values so duplicates from earlier runs cannot hide a failure. Example run label:
+
+```text
+beta1-run-01
+```
+
+Example Pin:
+
+```text
+For beta1-run-01 verification, use Node.js 22 in this disposable project.
+```
+
+Example Memory:
+
+```text
+For beta1-run-01, the synthetic release channel is amber.
+```
+
+Never use real credentials, customer data, private paths, or production policy in dogfooding prompts.
+
+## TUI procedure
+
+Start a persistent interactive session:
+
+```bash
+pi --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  --name "ds4-beta-tui"
+```
+
+Run these scenarios in order:
+
+1. Inspect `/context health`, `/context pins`, `/context memory`, and `/context privacy`.
+2. Send an ordinary suggestion without asking to persist it, for example: `The synthetic release channel amber seems useful.` No `context_persistence` call or confirmation dialog should appear.
+3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for beta1-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
+4. Ask the model to use `context_persistence` to list active Memory. Verify bounded metadata and no complete claim, key, reason, path, or raw error in the result.
+5. Explicitly request the synthetic Pin, but reject or close the confirmation dialog. Verify with `/context pins` that it was not created.
+6. Ask the model to retry using the sanitized value remaining in history. If it copies `[omitted-by-ds4-egress-policy]`, expect `rejected / egress-placeholder` before any new confirmation, runtime mutation, or append. If it asks for fresh text instead, record that safe routing result and run the exact-marker case from the JSON procedure.
+7. Request the Pin again with fresh synthetic text and accept it. Ask the model to list Pins, then use the exact returned Pin ID and `targetRevision` to unpin it in the same process. Accept the destructive confirmation. Fuzzy targeting must not be used.
+8. With a remote provider, request a `local-only` Pin. Expect provider-policy denial before confirmation and no append.
+9. Restart Pi and continue the session:
+
+   ```bash
+   pi --approve --session-dir "$DS4_DOGFOOD_SESSIONS" -c
+   ```
+
+   Verify the accepted Memory remains visible. Revision handles from the previous process are intentionally invalid; perform a fresh read before any targeted write.
+10. Run `/context rebuild-index`, then verify the same canonical Memory/Pin lifecycle state is reconstructed.
+
+TUI passes when accepted writes append once, refusal/closure appends nothing, destructive writes require an exact fresh revision, ordinary conversation does not persist, and rebuild preserves canonical state.
+
+## RPC procedure
+
+RPC mode exposes extension dialogs through a JSON request/response protocol. It reports `ctx.hasUI=true` because a client can answer those requests; the client is the UI bridge.
+
+Start a persistent RPC process from the disposable project:
+
+```bash
+pi --mode rpc --approve \
+  --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  --name "ds4-beta-rpc" \
+  2>evidence/rpc.stderr.log
+```
+
+Enter one JSON object per line on stdin. First request a read:
+
+```json
+{"id":"read-1","type":"prompt","message":"Use context_persistence with action pins_list to inspect active Pins. Do not perform a write."}
+```
+
+Wait for the turn to end before sending the next prompt. For a write:
+
+```json
+{"id":"write-1","type":"prompt","message":"Persist a session Pin for beta1-run-01 stating that this disposable project uses Node.js 22 for verification."}
+```
+
+Pi should emit a request shaped like:
+
+```json
+{"type":"extension_ui_request","id":"<dynamic-id>","method":"confirm","title":"DS4 Context Persistence","message":"..."}
+```
+
+After inspecting the request, approve it with the exact dynamic ID:
+
+```json
+{"type":"extension_ui_response","id":"<dynamic-id>","confirmed":true}
+```
+
+Expect the `context_persistence` tool result to report a committed outcome without echoing complete Pin content. Repeat with a different synthetic value and reject it:
+
+```json
+{"type":"extension_ui_response","id":"<dynamic-id>","confirmed":false}
+```
+
+The rejected call must report cancellation and append nothing. `{"cancelled":true}` is also a valid dialog dismissal response.
+
+### RPC without a responding UI client
+
+Start a separate disposable RPC process, request a write, and do not send an `extension_ui_response`. In Pi `0.84.3`, the confirmation remains pending because RPC still advertises UI capability. This is not converted to `confirmation-required`; no append may occur before a positive response. Terminate the disposable process after recording the pending request, then inspect the session from TUI.
+
+### RPC without a persistent session
+
+Start a separate process:
+
+```bash
+pi --mode rpc --approve --no-session
+```
+
+Send a read and a write prompt. Both must return `runtime-unavailable`; no `extension_ui_request` should be emitted and no canonical commit should be claimed.
+
+RPC passes when positive confirmation commits once, negative/cancelled confirmation appends nothing, an unanswered dialog remains pending without append, `--no-session` fails before confirmation, and result content/details remain bounded and metadata-only.
+
+## Print-mode procedure
+
+Print mode has no extension UI. Keep session persistence enabled:
+
+```bash
+pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence with action memory_list to inspect active Memory. Do not write."
+```
+
+The read should complete. Then request a write:
+
+```bash
+pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence to add a session Memory saying that beta1-run-01 uses the synthetic channel amber."
+```
+
+Expected behavior:
+
+```text
+outcome=unavailable
+errorCode=confirmation-required
+```
+
+No dialog can appear and no canonical custom entry may be appended. The assistant's final wording can vary; use JSON mode or the canonical audit below when the exact tool envelope is needed. If the model does not call the tool, record that separately as a routing observation and repeat with the explicit action name to isolate runtime behavior.
+
+Adding `--no-session` changes the expected error to `runtime-unavailable` for both reads and writes.
+
+## JSON event-stream procedure
+
+JSON mode is also non-interactive, but it exposes authoritative tool lifecycle events. Capture the complete local stream; it may include current non-secret tool arguments.
+
+Read case:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence with action pins_list to inspect active Pins. Do not write." \
+  2>evidence/json-read.stderr.log | tee evidence/json-read.jsonl
+```
+
+Write case:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Use context_persistence to add a session Pin for beta1-run-01 stating that this disposable project uses Node.js 22." \
+  2>evidence/json-write.stderr.log | tee evidence/json-write.jsonl
+```
+
+Reserved historical-placeholder regression:
+
+```bash
+pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
+  "Call context_persistence exactly once with action pin_add, scope session, classification normal, and content exactly [omitted-by-ds4-egress-policy]." \
+  2>evidence/json-placeholder.stderr.log | tee evidence/json-placeholder.jsonl
+```
+
+Extract tool completions:
+
+```bash
+jq -c '
+  select(.type == "tool_execution_end" and .toolName == "context_persistence")
+  | {isError, result}
+' evidence/json-*.jsonl
+```
+
+The read should succeed. The ordinary write must return `confirmation-required` and append nothing. The placeholder case must return `rejected / egress-placeholder`, not request confirmation, and append nothing. Inspect `result.content` and `result.details` for bounded allowlisted metadata; they must not echo complete content, claims, keys, reasons, paths, source-session identity, confirmation text, or raw errors.
+
+The local `tool_execution_start.args` event can contain the current synthetic arguments supplied to the tool. That local event is not the provider-facing result contract, which is why dogfooding must use non-sensitive data and evidence files must not be published blindly.
+
+## Canonical append audit
+
+List only metadata for canonical Pin/Memory entries in the dedicated sessions:
+
+```bash
+find "$DS4_DOGFOOD_SESSIONS" -name '*.jsonl' -print0 \
+  | xargs -0 -r jq -r '
+      select(
+        .type == "custom"
+        and (
+          .customType == "ds4-context-pin-v1"
+          or .customType == "ds4-context-memory-v1"
+        )
+      )
+      | [.customType, .id, .timestamp, (.data.operation // "unknown")]
+      | @tsv
+    '
+```
+
+Expected invariants:
+
+- each accepted canonical add/supersede/status operation contributes exactly one append-only custom entry;
+- cancelled, rejected, unavailable, and unanswered-confirmation operations contribute none;
+- print/JSON writes contribute none;
+- `--no-session` contributes none;
+- source include/exclude policy contributes no Pin/Memory custom entry because it is derived local SQLite policy;
+- rebuild changes projections, not the JSONL mutation sequence.
+
+Do not publish the session files: they are canonical local history and may contain prompt/tool argument text even when tool results are metadata-only.
+
+## Extended provider and lifecycle matrix
+
+After the basic mode matrix passes, repeat the relevant TUI/RPC cases with:
+
+- a configured remote provider;
+- a verified local provider whose exact provider ID is listed in `privacy.localProviders`;
+- privacy enabled and disabled;
+- a provider switch between read and targeted write;
+- a branch switch between read and targeted write;
+- trusted and untrusted project state;
+- two simultaneous Pi sessions using the shared SQLite database;
+- cross-session project Memory when `memory.crossSession` is explicitly enabled.
+
+Provider, trust, branch, provenance, capability, target state, and classification changes after confirmation must fail safely. A model-supplied `local-only` classification is never evidence that earlier input stayed local.
+
+## Storage containment and offline maintenance candidate
+
+For a build that includes bounded storage support, keep normal online validation separate from physical maintenance.
+
+While Pi is running:
+
+1. Run `/context storage`; verify metadata-only output and no manifest/message/Pin/Memory/snippet/tool content.
+2. Run `/context health`; a storage high-water warning must produce `WARN` while SQLite quick/FK/schema checks remain independently visible.
+3. Perform multiple normal model calls and verify manifest count converges toward `128` without startup pause or provider-request changes.
+4. Verify provider usage remains visible after close/reopen and that `manifest_json` is not rewritten at `message_end` in a disposable test database.
+5. Start a second current-version Pi process against the same disposable database; both client leases must exist, and closing each process must remove only its own lease.
+
+Do not compact a live or production database as part of ordinary dogfooding. On an explicitly disposable copy only:
+
+1. close every Pi process;
+2. run `ds4-context-storage inspect --database <exact-copy-path>`;
+3. preserve the metadata output;
+4. run `compact` and complete the local TTY confirmation;
+5. verify the fixed backup exists, manifest/calibration limits converge, source exclusions remain, and quick/FK/schema checks pass;
+6. reopen the copy through Pi, run `/context health` and `/context storage`, then verify Pin/Memory/source-policy projections;
+7. retain the backup during observation and remove it manually afterward.
+
+Also verify refusal for an active client, an existing backup/stage, insufficient simulated disk, a corrupt source, and ambiguous recovery state. Maintenance of the actual user database requires a separate explicit authorization after online dogfooding succeeds. See [`STORAGE_MAINTENANCE.md`](STORAGE_MAINTENANCE.md).
+
+## Result record
+
+Use one record per scenario:
+
+```text
+Run ID:
+Date:
+Pi version:
+DS4 exact version:
+Provider/model:
+Mode and session persistence:
+Scenario:
+Expected outcome:
+Observed outcome:
+Confirmation shown/answered:
+Canonical entries before/after:
+Projection/rebuild observation:
+Result leak check:
+Pass/fail:
+Issue/reference:
+```
+
+## Promotion criteria
+
+Do not promote the beta if any run shows:
+
+- a write without a fresh positive local UI decision;
+- canonical JSONL rewrite, loss, duplication, or a false commit claim;
+- complete persistent content or prohibited metadata in provider-facing results/history;
+- fuzzy destructive targeting or acceptance of a stale revision;
+- failure to reconstruct canonical state from JSONL;
+- an actionable warning hidden as routine debug output;
+- a reproducible regression above the release latency/quality/schema gates.
+
+A missing live local-provider run should remain explicitly recorded rather than inferred from remote-provider or automated-test results.

package/docs/RELEASE_READINESS_0.2.0.md

@@ -69,7 +69,7 @@ Pi JSONL, reference-adapter JSONL, and live project files are never deleted by a
 | minimum Node and current LTS | CI matrix: Node `22.19.0` and `24.x` |
 | migration/privacy/limitations/rollback docs | this document, `STORAGE.md`, `PRIVACY.md`, adapter/KV documentation |
 
-The latency check loads exact `ds4-context-core@0.1.2` and the local 0.2 build in one process, runs the same deterministic feature-disabled 401-message fixture, alternates samples to reduce host drift, and rejects a p95 ratio above `1.10`. The comparison contains only timings and package versions.
+The latency check loads exact `ds4-context-core@0.1.2` and the local candidate build in one process, runs the same deterministic feature-disabled 401-message fixture, alternates samples to reduce host drift, amortizes 50 planner calls per reported sample to reduce sub-millisecond noise, and rejects a p95 ratio above `1.10`. The comparison contains only timings and package versions.
 
 ## Candidate validation
 

package/docs/RELEASING.md

@@ -41,7 +41,7 @@ npm run latency:check -- "$BASELINE_DIR/node_modules/ds4-context-core"
 rm -rf "$BASELINE_DIR"
 ```
 
-The check rejects a candidate p95 above 110% of the exact 0.1.2 baseline. Run latency measurements on an otherwise idle host and repeat an anomalous run before drawing a release conclusion. See [`RELEASE_READINESS_0.2.0.md`](RELEASE_READINESS_0.2.0.md) for the stable-line gate matrix, the versioned notes under [`releases/`](releases/) for prerelease evidence, and [`DOGFOODING_0.3.0_ALPHA.md`](DOGFOODING_0.3.0_ALPHA.md) for the post-publication operating matrix.
+The check rejects a candidate p95 above 110% of the exact 0.1.2 baseline. Each reported sample amortizes 50 identical planner calls to reduce sub-millisecond timer and scheduler noise without weakening the threshold. Run latency measurements on an otherwise idle host and repeat an anomalous run before drawing a release conclusion. See [`RELEASE_READINESS_0.2.0.md`](RELEASE_READINESS_0.2.0.md) for the stable-line gate matrix, the versioned notes under [`releases/`](releases/) for prerelease evidence, and [`DOGFOODING_0.3.0_BETA.md`](DOGFOODING_0.3.0_BETA.md) for the current post-publication operating matrix.
 
 CI runs the same checks on the minimum supported Node.js version and the current Node.js LTS line. `npm run pack:check` uses a temporary directory and removes it when complete. Set `DS4_KEEP_PACK_TMP=1` only when diagnosing a failed package check.
 
@@ -82,12 +82,12 @@ npm publish --workspace ds4-context-reference-adapter --access public
 npm publish --access public
 ```
 
-Prereleases must pass the same explicit channel tag to all three commands so they cannot move `latest`. For a 0.3 alpha:
+Prereleases must pass the same explicit channel tag to all three commands so they cannot move `latest`. For a 0.3 beta:
 
 ```bash
-npm publish --workspace ds4-context-core --access public --tag alpha
-npm publish --workspace ds4-context-reference-adapter --access public --tag alpha
-npm publish --access public --tag alpha
+npm publish --workspace ds4-context-core --access public --tag beta
+npm publish --workspace ds4-context-reference-adapter --access public --tag beta
+npm publish --access public --tag beta
 ```
 
 After prerelease publication, verify both the exact artifacts and that `latest` still resolves to the intended stable version. If core succeeds but an adapter publication fails, fix that adapter release and retry it with the same version and channel tag. Do not rewrite or unpublish a valid core release merely to make the commands appear atomic.

package/docs/STORAGE.md

@@ -115,7 +115,19 @@ A full index rebuild replays all message entries, recreates missing qualifying o
 
 For persisted sessions, each `context` hook stores a metadata-only manifest containing token counts, session/project/pin/memory source and atomic-group IDs, inclusion/exclusion reasons, classifications and scores, original/selected counts, exact-model override/calibration/adaptive budgets, aggregate learned-ranking status/disagreement, model-switch/cache disposition, provider destination/allow names, privacy counters, optional continuation mode/item counts/retry reasons, project revision/hash/line references, tool names, a SHA-256 prompt hash, and planner/policy versions. Prompt text, message text, pin content, memory claims, project snippet text, tool arguments, image data, rendered provider payloads, and provider response/conversation IDs are not stored in the manifest.
 
-`before_provider_request` updates the pending manifest with final-check/redaction counters but never the provider payload. The following finalized assistant response updates it with uncached input, cache-read, cache-write and total provider input usage, then adds at most one exact-model calibration sample. Ephemeral sessions retain this information only in memory.
+`before_provider_request` updates the pending in-memory manifest with final-check/redaction counters but never the provider payload. The following finalized assistant response updates only the existing scalar usage columns (`actual_tokens`, `input_tokens`, `cache_read_tokens`, and `cache_write_tokens`) and adds at most one exact-model calibration sample. It does not read or rewrite `manifest_json`. Repository reads hydrate authoritative usage from those columns. Ephemeral, oversize-skipped, concurrently pruned, and otherwise uncorrelated manifests retain bounded calibration only in memory.
+
+Retention is bounded without a schema change: SQLite keeps the latest 128 manifests globally and at most 200 calibration samples for each provider/model/estimator profile. A manifest prune first detaches its small calibration row, then removes the large diagnostic JSON; calibration has its own per-profile retention. Save and prune are one transaction. Existing oversized stores are reduced incrementally by at most 32 rows and 8 MiB of serialized manifest payload per subsequent manifest write; one individually oversized oldest row may be removed to guarantee progress. There is no startup purge.
+
+New manifest persistence is byte-bounded. Payloads up to 256 KiB remain complete. Larger payloads preserve all `included` provenance and replace only the `excluded` inventory with a deterministic first/last sample of at most 256 details plus explicit `ds4-context-manifest-inventory-v1` counts, token/classification/kind rollups, and digests. The wrapper returned by `getStored()` declares `complete` or `excluded-rollup`; the live runtime manifest remains complete. A projected payload over 1 MiB is skipped without affecting the model request. Deleted pages become reusable by SQLite but do not promise an immediate reduction in filesystem size. Manifests and calibration remain disposable; Pi JSONL and project files are untouched.
+
+## Storage diagnostics and offline maintenance
+
+`/context storage` is read-only and metadata-only. It reports schema/journal, database and sidecar sizes, page/free-page counts, bounded manifest/calibration aggregates, active-project aggregate counts, artifact totals, convergence, and categorical maintenance reasons. `/context health` reports storage warning state separately from SQLite integrity: high-water or retention warnings produce an overall `WARN` without changing `DatabaseHealth.ok`.
+
+File-backed runtime clients use a private two-phase lease next to the database. An explicit maintenance lock prevents new clients from opening SQLite, and active or ambiguous client leases block compaction. Database close precedes lease release. Pi remains on its existing fallback path if it starts while maintenance is active.
+
+Physical size recovery uses the separately packaged, interactive `ds4-context-storage` CLI. It creates and validates a standalone backup, rewrites only an offline working copy, runs `VACUUM INTO` a candidate, checks schema/migrations/foreign keys, protected-table counts and private full-row digests, and source exclusions, then performs a persisted recoverable swap that retires WAL/SHM sidecars before installing the candidate. It never runs automatically or through an LLM-callable tool, never overwrites its fixed backup, and never edits Pi JSONL or project files. See [`STORAGE_MAINTENANCE.md`](STORAGE_MAINTENANCE.md).
 
 ## Compaction summaries
 
@@ -137,4 +149,6 @@ Rollback is also projection-based. A 0.1 binary refuses schema 15 by design. Sto
 
 A full rebuild does not blindly delete unchanged entries. It upserts all observed entries, marks them in a temporary seen-set, and removes only stale rows. This preserves foreign-key provenance for unchanged source entries. FTS rows and checkpoint state update in the same transaction.
 
-Session reconciliation is transactional. Memory/pin mutation replacement, checkpoint update, source exclusion and full materialization each occur under the shared write coordinator. Each quality upsert and bounded-retention prune share one transaction; quality failures do not affect manifests or planning. Each changed project file is also replaced transactionally with its snippets and FTS rows; embedding upserts and canonical-source pruning are transactional; artifact object/reference metadata and project deletion batches are atomic. A filesystem artifact write precedes its metadata transaction, so an interrupted metadata write may leave only an unreferenced content-addressed cache file; canonical JSONL remains sufficient for recovery. If parsing, validation, or SQLite writing fails, the prior derived state remains available. Artifact/project failures contribute no replacement/snippets; planner failures discard all synthetic evidence; Pi continues with its native context.
+Session reconciliation is transactional. Memory/pin mutation replacement, checkpoint update, source exclusion and full materialization each occur under the shared write coordinator. Each manifest upsert and dual-bound incremental retention prune share one transaction; each scalar usage/calibration update and its independent per-profile prune do the same. Each quality upsert and bounded-retention prune also share one transaction; quality failures do not affect manifests or planning. Each changed project file is replaced transactionally with its snippets and FTS rows; embedding upserts and canonical-source pruning are transactional; artifact object/reference metadata and project deletion batches are atomic. A filesystem artifact write precedes its metadata transaction, so an interrupted metadata write may leave only an unreferenced content-addressed cache file; canonical JSONL remains sufficient for recovery. If manifest serialization, projection, retention, or SQLite writing fails, the complete current manifest remains in memory and the provider request is unchanged. Other artifact/project failures contribute no replacement/snippets; planner failures discard all synthetic evidence; Pi continues with its native context.
+
+After bounded busy-aware replay is exhausted, DS4 emits `database.write_lock_timeout` with only the coordinator operation name, attempt count, elapsed/configured waits, and SQLite primary code. The thrown error repeats the operation and categorical lock status but never includes SQL, bound values, provider content, or the raw SQLite message. Retry and rollback diagnostics follow the same metadata-only rule.

package/docs/STORAGE_MAINTENANCE.md

@@ -0,0 +1,121 @@
+# Offline SQLite Storage Maintenance
+
+DS4 keeps `context.db` as disposable derived state, while Pi session JSONL and live project files remain canonical. Normal runtime retention stops unbounded manifest growth and makes deleted pages reusable. It does not promise that an existing high-water SQLite file shrinks physically.
+
+Physical compaction is therefore an explicit offline operation. It is never model-callable, never runs at startup, and never edits Pi JSONL or project files.
+
+## Commands
+
+The root package installs:
+
+```text
+ds4-context-storage inspect --database <exact-path>
+ds4-context-storage compact --database <exact-path>
+ds4-context-storage recover --database <exact-path>
+```
+
+`inspect` is metadata-only and read-only. `compact` and `recover` require an interactive local TTY; there is no V1 `--yes` option. The database path is mandatory and is normalized before use.
+
+Typical default path:
+
+```text
+~/.pi/agent/ds4-context/context.db
+```
+
+Do not run `compact` until every Pi process that may use that database is closed. On Linux, an additional read-only holder check can be made with:
+
+```text
+fuser -v context.db context.db-wal context.db-shm
+```
+
+The utility never terminates a process.
+
+## Online diagnostics first
+
+With Pi running:
+
+```text
+/context health
+/context storage
+```
+
+`/context storage` reports only schema, journal mode, file/page sizes, manifest and calibration counts, current-project aggregate counts, artifact totals, convergence state, and categorical maintenance reasons. It does not expose manifest rows, messages, claims, snippets, tool payloads, SQL, or raw SQLite errors.
+
+A storage warning makes overall health display `WARN`, but it does not make SQLite corruption checks fail and does not alter the provider request.
+
+## Cooperative exclusion protocol
+
+Each new file-backed `ContextDatabase` client creates a private lease under:
+
+```text
+context.db.clients/<client-id>.json
+```
+
+The lease contains only protocol version, client ID, PID, creation time, and a database-path fingerprint. The database is opened only after a two-phase check for:
+
+```text
+context.db.maintenance.lock
+```
+
+The maintenance utility creates that lock exclusively and refuses compaction when a verified client is alive or a lease/PID is ambiguous. Verified dead-client leases may be removed. A process started during maintenance sees the lock, does not open SQLite, and leaves Pi on its existing fallback path. Database close occurs before client-lease removal.
+
+Older DS4 versions do not create leases, so the interactive assertion that every Pi instance is closed remains mandatory.
+
+## Compact protocol
+
+`compact` performs these phases:
+
+1. validate exact path, regular-file type, stage-file absence, schema 15, migration checksums, `quick_check`, and foreign keys;
+2. acquire the maintenance lock and refuse active or ambiguous clients;
+3. verify conservative free space for backup, working copy, candidate, and safety margin;
+4. create a standalone SQLite backup using `node:sqlite.backup()` from a read-only source connection;
+5. validate that backup;
+6. create an exclusive working copy from the backup;
+7. on the working copy only, retain 128 newest manifests, retain 200 calibration samples per exact profile, detach calibration from deleted manifests, and rewrite retained manifests through the bounded serializer;
+8. checkpoint and close the working copy;
+9. create `context.db.compact-ready` with `VACUUM INTO`;
+10. validate quick/FK/schema checks, hard manifest bounds, protected-table counts and private full-row digests, and source-exclusion keys;
+11. persist swap state, retire the source and its WAL/SHM sidecars, install the candidate, fsync, and validate again;
+12. remove temporary working/retired files, release the lock, and retain exactly one fixed backup.
+
+The fixed files are:
+
+```text
+context.db.maintenance-work
+context.db.compact-ready
+context.db.precompact.bak
+context.db.swap-old
+context.db.maintenance-state.json
+```
+
+Existing stage or backup files are never overwritten. A failure before swap leaves the source in place. A failure during swap attempts immediate restoration of the retired source and sidecars. Persisted state supports deterministic recovery after process interruption.
+
+## Recover
+
+Use `recover` only when a prior operation reports recovery state or leaves `context.db.maintenance-state.json`.
+
+The command fails closed on ambiguous combinations. Depending on the persisted phase it will:
+
+- remove uninstalled staging while retaining the source;
+- restore the retired source after an interrupted first rename;
+- keep a valid installed candidate and clean the retired source;
+- restore the retired source when the installed candidate fails validation.
+
+Every database selected for use is checked again. Persisted state is accepted only when every fixed path matches the selected database. Recovery does not guess from timestamps or file sizes and never overwrites the fixed backup.
+
+## Post-maintenance checks
+
+After successful compaction:
+
+1. retain `context.db.precompact.bak`;
+2. reopen Pi;
+3. run `/context health` and `/context storage`;
+4. verify expected Pin, Memory, and local project-source exclusion behavior;
+5. dogfood normal model calls;
+6. remove the backup manually only after the observation period succeeds.
+
+If rollback is needed, close Pi before invoking recovery or manually restoring the validated standalone backup. No storage rollback requires a Pi JSONL change.
+
+## Packaging and privacy
+
+Package verification rejects database, WAL, SHM, backup, candidate, work, retired swap, lock, state, client-lease, JSONL, `.pi`, and `.serena` paths. Protected-table digests are compared only in process and are never printed. CLI output is bounded to phases, counts, sizes, schema/check status, exact user-selected local paths, and categorical recovery guidance.

package/docs/releases/0.3.0-alpha.5.md

@@ -1,6 +1,6 @@
 # DS4 Context Engine 0.3.0-alpha.5
 
-Status: validated release candidate; not yet published.
+Status: published prerelease on 2026-09-02; tag `v0.3.0-alpha.5`.
 
 This coordinated prerelease adds overflow-safe hierarchical compaction for discarded source that cannot fit in one active-model summary request. It preserves strict summary validation, privacy sanitization, canonical Pi JSONL semantics, one final Pi compaction entry, and fail-closed fallback to Pi default compaction.
 
@@ -30,7 +30,7 @@ ds4-context-reference-adapter
 ds4-context-engine
 ```
 
-Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.5`. Publication must use the explicit npm `alpha` dist-tag so `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+Both adapters depend exactly on `ds4-context-core@0.3.0-alpha.5`. The packages were published manually under the explicit npm `alpha` dist-tag while `latest` remains `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
 
 ## Validation evidence
 
@@ -45,9 +45,11 @@ Local candidate verification on Node.js `26.5.1`:
 - `npm pack --dry-run --json` for all three packages passed with the same bounded inventories and no forbidden local/session files.
 - The committed candidate was replayed from detached clean checkout `8b609c0`; `npm ci`, the 65-file/297-test suite, quality, schema, package verification, tarball review, latency retry, and `git diff --check` passed.
 - `git diff --check`: passed.
+- Validation-only CI passed on Node.js `22.19.0` and `24.x`; an initial unrelated schema-v10 upgrade-test timeout on Node 22 passed three isolated local reruns and the failed-job retry.
 - Protected CI, compatibility golden, Pi fixture, migration, canonical Pin/Memory, persistence confirmation, and persistence result-contract files are unchanged.
+- `npm run registry:check -- 0.3.0-alpha.5`: passed against all three exact published versions; `alpha` resolves to `0.3.0-alpha.5` and `latest` remains `0.2.0` for every package.
 
-Exact registry verification is required before tag or GitHub prerelease creation.
+Exact registry verification passed before the annotated tag and GitHub prerelease were created.
 
 ## Documentation
 

package/docs/releases/0.3.0-beta.1.md

@@ -0,0 +1,81 @@
+# DS4 Context Engine 0.3.0-beta.1
+
+Status: release candidate; publication and exact registry verification pending.
+
+This coordinated beta contains SQLite growth caused by persisted Context Manifests and adds explicit, recoverable offline maintenance for already-large derived databases. It also hardens compaction provider retries while preserving strict grounding, privacy filtering, canonical Pi JSONL semantics, atomic tool exchanges, one final Pi compaction entry, and fail-open delegation to Pi.
+
+## Added
+
+- Retains at most 128 Context Manifests globally and at most 200 calibration samples per exact provider/model/estimator profile.
+- Bounds each online manifest prune to 32 rows and 8 MiB, while allowing one individually oversized oldest row to be removed so convergence cannot stall.
+- Persists complete manifests up to 256 KiB; larger manifests retain complete included provenance and a deterministic excluded-only rollup with complete counts, aggregates, and digests plus at most 256 sampled excluded details.
+- Skips manifest persistence above the 1 MiB hard bound without altering the active in-memory manifest or provider request.
+- Exposes persisted inventory completeness through `getStored()` while legacy complete rows remain readable through `get()`.
+- Adds metadata-only `/context storage` diagnostics and storage high-water integration in `/context health`.
+- Adds cooperative database client leases and a two-phase maintenance lock; active or ambiguous clients block physical maintenance.
+- Adds the `ds4-context-storage inspect|compact|recover --database <exact-path>` CLI. Mutating commands require a local TTY and have no non-interactive bypass.
+- Adds standalone backup, working-copy transformation, validated `VACUUM INTO` candidate, protected-table full-row digests, recoverable swap state, and deterministic crash recovery.
+
+## Changed
+
+- Provider usage is authoritative in scalar Context Manifest columns; `message_end` no longer rewrites `manifest_json`.
+- Only compaction failures categorized as `transport` are retried, with at most three attempts, abort-aware 200 ms and 500 ms delays, fresh routing session IDs, and cumulative usage accounting.
+- Runtime startup during maintenance degrades safely to Pi fallback with categorical diagnostics rather than exposing raw SQLite messages.
+- SQLite lock timeout and rollback diagnostics expose bounded categories, operation names, attempt counts, and numeric SQLite codes rather than raw database errors.
+- Package verification rejects database, WAL, SHM, backup, staging, retired-swap, maintenance-state, client-lease, JSONL, `.pi`, and `.serena` paths.
+
+## Safety and compatibility
+
+Pi JSONL remains canonical and append-only. Physical maintenance never edits session history or project files and is unavailable to model-callable tools. Every `context_persistence` write still requires a fresh positive local UI decision, exact destructive targeting remains revision-bound, and canonical Pin/Memory mutations still append through Pi before SQLite reconciliation.
+
+An individually oversized compaction group, unsupported exact value, non-transport provider failure, abort, or bounded-operation failure still delegates to Pi default compaction. Rejected values and provider payloads are not emitted in diagnostics.
+
+SQLite schema remains 15. Migrations 1–15, `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, and `ds4-context-persistence-result-v1` are unchanged. Earlier schema-15 readers can parse additive rollup metadata but may present sampled excluded details as complete, so historical excluded-inventory rendering is unsupported after downgrade once rollups have been written.
+
+## Package/version policy
+
+The coordinated version is `0.3.0-beta.1` for:
+
+```text
+ds4-context-core
+ds4-context-reference-adapter
+ds4-context-engine
+```
+
+Both adapters depend exactly on `ds4-context-core@0.3.0-beta.1`. Publication uses the explicit npm `beta` dist-tag for all three packages while `latest` must remain stable `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+
+## Candidate validation evidence
+
+Local candidate verification on Node.js `26.5.1`:
+
+- `npm run check`: 69 files and 341 tests passed.
+- Three consecutive focused multi-process SQLite concurrency runs passed.
+- `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
+- `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
+- `npm run latency:check -- <exact ds4-context-core@0.1.2>`: passed after clean install with ratio `0.93982`, at or below `1.10`, using 200 samples and 50 calls per sample.
+- `npm run pack:check`: verified core (223 files), reference adapter (7 files), and Pi adapter (67 files) in a clean consumer.
+- `npm pack --dry-run --json` for all three packages passed with the same inventories and no forbidden local/session/storage files.
+- `git diff --check`: passed.
+- Version, exact core dependencies, package-lock entries, extension constant, and reference-adapter constant are synchronized to `0.3.0-beta.1`.
+- Protected compatibility golden, Pi JSONL fixture, migration definitions, canonical persistence contracts, and CI publication permissions are unchanged.
+
+Large-database validation used a temporary private copy created from a read-only source connection; the live database was not compacted, vacuumed, rebuilt, replaced, or maintained:
+
+- before: schema 15, `quick_check=ok`, 3,691 manifests, 2,101,666,663 manifest bytes, 2,489,868,288 database bytes;
+- after: 128 manifests, 15,626,529 manifest bytes, 414 calibration samples across three profiles, 389,746,688 database bytes;
+- all 128 retained oversized manifests were rewritten as explicit `excluded-rollup` projections with no irreducible oversize;
+- the final candidate passed quick check and foreign-key validation while the standalone backup remained present;
+- temporary copy, backup, and stage files were removed afterward, with no maintenance artifact created next to the live database or in the repository.
+
+Publication, exact registry verification, annotated tag, and GitHub prerelease are performed only after the committed candidate passes validation-only CI.
+
+## Documentation
+
+- [`../CONTEXT_MANIFEST.md`](../CONTEXT_MANIFEST.md)
+- [`../STORAGE.md`](../STORAGE.md)
+- [`../STORAGE_MAINTENANCE.md`](../STORAGE_MAINTENANCE.md)
+- [`../COMPACTION.md`](../COMPACTION.md)
+- [`../ARCHITECTURE.md`](../ARCHITECTURE.md)
+- [`../DOGFOODING_0.3.0_BETA.md`](../DOGFOODING_0.3.0_BETA.md)
+- [`../RELEASING.md`](../RELEASING.md)
+- [`0.3.0-alpha.5.md`](0.3.0-alpha.5.md)