ds4-context-engine 0.3.0-alpha.4 → 0.3.0-beta.1

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Published prerelease `0.3.0-alpha.3` adds privacy-safe exact-value repair diagnostics and tighter summary prompting while preserving strict compaction validation, Pi fallback, canonical records, SQLite schema 15, and runtime contracts. npm `alpha` points to `0.3.0-alpha.3`, `latest` remains `0.2.0`, and the maintenance line targets Pi `0.84.3`.
+> **Project status:** Stable `0.2.0` includes M0–M20 and frozen 0.2 contracts. Candidate prerelease `0.3.0-beta.1` adds bounded Context Manifest storage, cooperative database leases, metadata-only storage diagnostics, and explicit offline inspect/compact/recover maintenance on top of alpha.5's overflow-safe hierarchical compaction. Canonical records, SQLite schema 15, and runtime contracts remain unchanged. The beta will use npm's explicit `beta` tag while `latest` remains `0.2.0`, and the maintenance line targets Pi `0.84.3`.
 
 ## Why DS4
 
@@ -29,7 +29,7 @@ It provides:
 - exact and FTS5 historical retrieval with source provenance;
 - opt-in hybrid semantic retrieval with a deterministic local embedding and lexical fallback;
 - trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
-- hierarchical, validated, non-destructive compaction summaries;
+- hierarchical, validated, non-destructive compaction summaries with overflow-safe multi-request fan-out/fan-in;
 - persistent pins and append-only durable memory stored canonically in Pi JSONL;
 - a bounded, metadata-only `context_persistence` tool with local confirmation for every model-callable write;
 - opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
@@ -89,13 +89,13 @@ pi install npm:ds4-context-engine
 
 The stable `0.2.0` packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) use the same exact version. Both adapters require the matching core version.
 
-To dogfood the published alpha without replacing a global stable installation, pin it in a disposable project:
+To dogfood the beta without replacing a global stable installation, pin the exact version in a disposable project:
 
 ```bash
-pi install -l npm:ds4-context-engine@0.3.0-alpha.3
+pi install -l npm:ds4-context-engine@0.3.0-beta.1
 ```
 
-Follow the [0.3 alpha dogfooding runbook](docs/DOGFOODING_0.3.0_ALPHA.md); use synthetic data and a dedicated session directory.
+Follow the [0.3 beta dogfooding runbook](docs/DOGFOODING_0.3.0_BETA.md); use synthetic data and a dedicated session directory.
 
 ### Local checkout
 
@@ -391,16 +391,25 @@ By default, derived state is stored below Pi's agent directory:
 └── artifacts/
 ```
 
-The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. The optional checksummed learned-ranking model is also derived local state; its classified metadata-only labels remain canonical Pi custom entries. All Pi sessions share this WAL database: writes use bounded busy-aware transaction replay, and a renewable project lease prevents multiple Pi processes from indexing the same project concurrently. `busyTimeoutMs` controls each SQLite lock wait, while `writeRetryTimeoutMs` bounds the total replay window. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
+The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. The optional checksummed learned-ranking model is also derived local state; its classified metadata-only labels remain canonical Pi custom entries. All Pi sessions share this WAL database: writes use bounded busy-aware transaction replay, and a renewable project lease prevents multiple Pi processes from indexing the same project concurrently. `busyTimeoutMs` controls each SQLite lock wait, while `writeRetryTimeoutMs` bounds the total replay window. Exhausted lock retries identify only the coordinator operation and categorical SQLite metadata. Diagnostic storage keeps the latest 128 manifests globally and 200 calibration samples per exact profile. Online manifest pruning is bounded to 32 rows and 8 MiB per related write. Manifests above the 256 KiB preferred bound retain complete included provenance and use an explicit deterministic excluded-only rollup; projected payloads above 1 MiB are skipped. Current readers label rollups explicitly; earlier schema-15 readers may parse them but mislabel sampled excluded details, so that historical rendering is not downgrade-supported after rollups are written. Provider usage updates existing scalar columns without rewriting the JSON payload. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
 
-To validate or rebuild derived state:
+To inspect, validate, or rebuild derived state:
 
 ```text
 /context health
+/context storage
 /context rebuild-index
 ```
 
-Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated. When `memory.crossSession` is enabled for a trusted project, DS4 discovers bounded sibling Pi JSONL files by exact canonical header identity, incrementally replays their explicit project mutations, and excludes missing or unverifiable sources.
+Physical size recovery is deliberately offline and interactive:
+
+```text
+ds4-context-storage inspect --database <exact-path>
+ds4-context-storage compact --database <exact-path>
+ds4-context-storage recover --database <exact-path>
+```
+
+Close every Pi process before `compact` or `recover`. New runtimes create cooperative client leases and refuse to open SQLite while the maintenance lock exists; the CLI also refuses active or ambiguous clients, validates a standalone backup and candidate, and keeps one fixed pre-compaction backup. See [`docs/STORAGE_MAINTENANCE.md`](docs/STORAGE_MAINTENANCE.md). Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated. When `memory.crossSession` is enabled for a trusted project, DS4 discovers bounded sibling Pi JSONL files by exact canonical header identity, incrementally replays their explicit project mutations, and excludes missing or unverifiable sources.
 
 ## Development
 
@@ -416,13 +425,13 @@ npm run schema:context-persistence
 npm run latency:check -- /path/to/exact/ds4-context-core@0.1.2
 npm run pack:check
 # Post-publication, with an exact version rather than a dist-tag:
-npm run registry:check -- 0.3.0-alpha.3
+npm run registry:check -- 0.3.0-beta.1
 npm pack --dry-run
 npm pack --dry-run --workspace ds4-context-core
 npm pack --dry-run --workspace ds4-context-reference-adapter
 ```
 
-The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, local-KV eligibility/replay, runtime-adapter conformance, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check builds all three tarballs, installs them in a clean temporary consumer, reruns compiled reference-adapter conformance and starts the packaged Pi extension with isolated RPC state.
+The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, local-KV eligibility/replay, runtime-adapter conformance, the portable-core dependency boundary and Pi extension lifecycle behavior. The latency comparison times 50 planner calls per sample to reduce sub-millisecond timer and scheduler noise while preserving the 1.10 p95 rejection threshold. The package check builds all three tarballs, installs them in a clean temporary consumer, reruns compiled reference-adapter conformance and starts the packaged Pi extension with isolated RPC state.
 
 ### Portable core
 
@@ -455,6 +464,7 @@ scripts             package and release-readiness checks
 - [Artifacts](docs/ARTIFACTS.md)
 - [Memory and pins](docs/MEMORY_AND_PINS.md)
 - [Context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md)
+- [0.3 beta dogfooding runbook](docs/DOGFOODING_0.3.0_BETA.md)
 - [0.3 alpha dogfooding runbook](docs/DOGFOODING_0.3.0_ALPHA.md)
 - [Privacy](docs/PRIVACY.md)
 - [Model awareness](docs/MODEL_AWARENESS.md)
@@ -463,11 +473,15 @@ scripts             package and release-readiness checks
 - [Runtime adapter kit](docs/RUNTIME_ADAPTER_KIT.md)
 - [Local KV reuse](docs/LOCAL_KV_REUSE.md)
 - [Storage](docs/STORAGE.md)
+- [Offline storage maintenance](docs/STORAGE_MAINTENANCE.md)
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
 - [0.2.0 release notes](docs/releases/0.2.0.md)
 - [0.2.0-rc.1 release notes](docs/releases/0.2.0-rc.1.md)
+- [0.3.0-beta.1 prerelease notes](docs/releases/0.3.0-beta.1.md)
+- [0.3.0-alpha.5 prerelease notes](docs/releases/0.3.0-alpha.5.md)
+- [0.3.0-alpha.4 prerelease notes](docs/releases/0.3.0-alpha.4.md)
 - [0.3.0-alpha.3 prerelease notes](docs/releases/0.3.0-alpha.3.md)
 - [0.3.0-alpha.2 prerelease notes](docs/releases/0.3.0-alpha.2.md)
 - [0.3.0-alpha.1 prerelease notes](docs/releases/0.3.0-alpha.1.md)
@@ -478,7 +492,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Prerelease `0.3.0-alpha.3` retains the confirmed [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md) hardening from alpha.2 and adds privacy-safe [compaction](docs/COMPACTION.md) repair diagnostics without weakening exact-value grounding or Pi fallback. Stable canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. Candidate `0.3.0-beta.1` builds on the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md) and privacy-safe [compaction](docs/COMPACTION.md) hardening with bounded persisted manifests, per-profile calibration retention, cooperative client leases, storage diagnostics, and recoverable offline maintenance. Confirmation, exact targeting, strict summary grounding, Pi fallback, and stable canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ADR/058-bounded-manifest-storage.md

@@ -0,0 +1,31 @@
+# ADR 058 — Bound persisted Context Manifests before considering compression
+
+- Status: Accepted
+- Scope: SQLite schema 15 and the first storage-containment rollout
+
+## Context
+
+`context_manifests` is diagnostic derived state, but an observed database contained 3,358 rows and about 1.8 GB of serialized manifest JSON. Roughly 95% of a representative large manifest was the inventory of context items that were excluded from the provider request. Large inserts, later JSON rewrites for provider usage, and a global backlog can increase writer-lock duration.
+
+Compression reduces bytes effectively, but adopting compressed payloads under schema 15 would make the JSON column misleading and would add codec, downgrade, migration, corruption, and decompression-limit concerns.
+
+## Decision
+
+1. Keep the latest 128 manifests globally.
+2. Limit one online prune to 32 rows and 8 MiB of serialized payload, except that one individually oversized oldest row may be removed to guarantee progress.
+3. Keep the runtime manifest complete.
+4. Persist complete manifests up to 256 KiB.
+5. Above 256 KiB, retain every included item and deterministically sample at most 256 excluded details (first 128 and last 128). Persist complete excluded counts, token rollups, classification/kind aggregates, and stable digests.
+6. Skip persistence when the projected payload still exceeds 1 MiB.
+7. Keep provider usage authoritative in existing scalar columns and do not rewrite `manifest_json` at `message_end`.
+8. Keep up to 200 calibration samples per exact provider/model/estimator profile, independently of manifest retention.
+9. Keep SQLite schema version 15 and configuration contract `ds4-context-config-v1` unchanged.
+10. Defer payload compression to a separate ADR and append-only migration.
+
+## Consequences
+
+- Included provenance remains complete; excluded historical detail is explicitly marked `excluded-rollup` and is never presented as a complete inventory by the current runtime.
+- Earlier schema-15 readers can parse the additive JSON but may mislabel sampled `excluded` details; downgrade support therefore excludes historical excluded-inventory rendering after a rollup has been written.
+- Existing databases converge incrementally without a long startup purge.
+- Deleted pages are reusable, but reclaiming physical file size requires explicit offline copy–validate–swap maintenance.
+- A future compressed representation must define a versioned codec, raw/stored byte limits, payload hash, decompression-bomb protection, downgrade behavior, rebuild behavior, and a new migration. Base64-compressed JSON in `manifest_json` is not permitted under schema 15.

package/docs/ADR/README.md

@@ -61,5 +61,6 @@ The initial decisions from the development plan are accepted:
 | 055 | Enforce a dedicated metadata-only tool egress guard independently of `privacy.enabled` | Accepted |
 | 056 | Keep project-memory source exclusion as disposable derived SQLite policy | Accepted |
 | 057 | Derive mutation provenance from the active branch and exclude model-supplied source IDs from V1 | Accepted |
+| [058](058-bounded-manifest-storage.md) | Bound persisted manifests and defer compression to a versioned migration | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/ARCHITECTURE.md

@@ -101,12 +101,13 @@ agent_settled
 session_before_compact
   -> map Pi preparation messages to exact canonical entry IDs
   -> privacy-sanitize discarded text/instructions/file lists for the active provider
-  -> serialize only the newly discarded segment and preserve Pi's retained boundary
-  -> generate and validate an immutable segment node
+  -> preflight the complete sanitized request against the calibrated model input budget
+  -> when needed, split at contiguous message boundaries while keeping matching tool calls/results atomic
+  -> generate and validate each immutable segment against only its own sanitized evidence
   -> resolve the prior root from the active Pi branch only
-  -> generate and validate a bounded aggregate node when a prior root exists
+  -> recursively generate and validate bounded ordered aggregate layers until one root remains
   -> wrap generated nodes with their highest inherited classification for provider-switch safety
-  -> atomically persist prepared nodes/edges and return only the active root to Pi
+  -> atomically persist the complete prepared graph and return only one active root to Pi
 
 session_compact / session_compact_failed
   -> commit or fail prepared summary lifecycle
@@ -192,8 +193,8 @@ ds4-context-engine                        ds4-context-reference-adapter
 - `packages/core/src/project`: trust-gated file discovery, hashing, Git state, symbol/chunk extraction, invalidation, retrieval and source quoting;
 - `packages/core/src/quality`: versioned replay fixtures/contracts, deterministic metrics, static/candidate comparison and metadata-only aggregation;
 - `packages/core/src/ranking`: bounded metadata-only features, classified label contracts, deterministic local training, checksummed model artifacts, aggregate shadow comparison and promotion-gated inference;
-- `packages/core/src/persistence`: rebuildable session/project/vector/memory/pin/quality SQLite state, repositories, FTS5, event replay and transactional migrations;
-- `packages/core/src/manifest` and `packages/core/src/shared`: runtime-neutral projections, provenance, hashing, stable serialization and logging.
+- `packages/core/src/persistence`: rebuildable session/project/vector/memory/pin/quality SQLite state, repositories, FTS5, event replay, transactional migrations, storage diagnostics, cooperative database-client leases and offline copy–validate–swap maintenance;
+- `packages/core/src/manifest` and `packages/core/src/shared`: runtime-neutral projections, provenance, bounded persisted-manifest serialization, hashing, stable serialization and logging.
 
 The `packages/reference-adapter` workspace is the non-Pi reference adapter: it reads bounded append-only canonical JSONL, injects completion through a host callback, enforces privacy at that callback boundary, rebuilds disposable snapshots and explicitly disables unsupported native features. A local host can inject a handle-free `LocalKvRuntimePort`; the port alone retains native handles and transport while core receives only exact prefix bytes transiently for hashing.
 
@@ -210,7 +211,7 @@ The Pi session JSONL remains canonical for conversation/tool state, inline class
 
 ## Lifecycle
 
-Database resources are opened during `session_start`, not from the extension factory. Eligible provider wrappers are registered after trusted configuration loads, and their volatile continuation manager is reset for every session lifecycle. Resources are closed idempotently during `session_shutdown`. Reload and session replacement therefore cannot reuse stale `SessionManager` instances, database handles, or provider continuation state.
+Database resources are opened during `session_start`, not from the extension factory. Before a file-backed database is opened, the runtime performs a two-phase maintenance-lock check around creation of a private process client lease. A maintenance lock prevents the open and leaves Pi on the existing degraded fallback; a maintenance utility refuses active or ambiguous leases. Eligible provider wrappers are registered after trusted configuration loads, and their volatile continuation manager is reset for every session lifecycle. Resources are closed idempotently during `session_shutdown`, with SQLite closed before the client lease is released. Reload and session replacement therefore cannot reuse stale `SessionManager` instances, database handles, or provider continuation state.
 
 ## SQLite choice
 
@@ -225,10 +226,14 @@ Database settings:
 - 5-second busy timeout;
 - transactional, checksummed migrations;
 - FTS5 tables created by schema migration;
-- containing directory mode `0700` and database file mode `0600` where supported.
+- containing/client-lease directories mode `0700` and database/protocol files mode `0600` where supported;
+- global manifest retention of 128 with online prune bounded to 32 rows and 8 MiB;
+- persisted manifest projection preferred at 256 KiB and hard-skipped above 1 MiB after deterministic excluded-only rollup;
+- provider usage updated in scalar columns without rewriting the manifest JSON;
+- physical compaction only through the explicit offline maintenance state machine.
 
 ## Failure policy
 
-Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, quality measurement, and diagnostics failures are caught at the extension boundary. Session index failures retain the previous transactional snapshot. Cross-session source failures exclude only the unverifiable source and retain explicit diagnostics; they do not disable current-session memory. Historical and project FTS errors degrade to exact matches; embedding consent/privacy/model/timeout/corruption/provider failures degrade to lexical results; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
+Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, quality measurement, and diagnostics failures are caught at the extension boundary. Manifest serialization/persistence/retention failure never changes a planned provider request; the complete manifest remains in memory and usage calibration falls back to a bounded volatile sample when correlation is unavailable. A maintenance lock prevents SQLite startup and emits only categorical local diagnostics while Pi continues on fallback. Session index failures retain the previous transactional snapshot. Cross-session source failures exclude only the unverifiable source and retain explicit diagnostics; they do not disable current-session memory. Historical and project FTS errors degrade to exact matches; embedding consent/privacy/model/timeout/corruption/provider failures degrade to lexical results; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
 
 Privacy is the exception to ordinary fail-open behavior. Once enabled, planner failures return the sanitized native array, preparation failures replace message content with structural placeholders, and provider-payload sanitizer failures return an empty object so the remote request fails rather than receiving unchecked content. Pi 0.84.3 runs provider-payload handlers in extension load order, so DS4 should be loaded last when other extensions can rewrite provider payloads.

package/docs/COMPACTION.md

@@ -7,13 +7,13 @@ DS4 intercepts Pi's `session_before_compact` event but preserves Pi's cut-point
 1. Pi determines `messagesToSummarize`, optional split-turn prefix, and `firstKeptEntryId`.
 2. DS4 maps every source message by exact fingerprint to a canonical branch entry ID.
 3. Pi's serializer converts the newly discarded span to bounded conversation text; enabled privacy policy sanitizes conversation, custom instructions, and file paths for the active provider.
-4. The active model generates and validates an immutable segment summary against sanitized evidence with cache retention disabled and a fresh routing session ID.
-5. If the active branch has a previous DS4 root, a second bounded call aggregates only that root's content with the new segment content; DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
-6. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as `prepared`; Pi receives only the active segment or aggregate text.
-7. Pi appends its normal `CompactionEntry` with `fromHook: true`.
-8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the prepared batch `failed`.
+4. DS4 estimates the complete sanitized summary request against the calibrated active-model input budget. If the whole request does not fit, it partitions the source into ordered contiguous segments. Individual messages are indivisible, and every tool call remains in the same atomic group as all matching results; an exceptionally large turn may split only between such groups.
+5. The active model generates and validates each immutable segment summary only against that segment's sanitized evidence, with cache retention disabled and a fresh routing session ID per call. Only failures categorized as `transport` are retried: at most two retries follow bounded 200 ms and 500 ms delays, each with another fresh routing session ID. The delay and the next call both honor Pi's abort signal; all other failure categories fall back immediately.
+6. DS4 recursively aggregates ordered child summaries in bounded fan-in calls until one root remains. The ordered roots include the previous branch summary, when present, followed by every new segment. DS4-generated IDs, hashes, kinds, and graph levels remain outside model-visible evidence. A Pi-native predecessor is imported as an explicitly unverified branch node.
+7. The highest input classification is wrapped around each generated node, then all nodes are persisted atomically as one `prepared` graph batch. Usage is summed across every segment and aggregate request; Pi receives only the final root text and still appends exactly one canonical `CompactionEntry` with `fromHook: true`.
+8. `session_compact` commits all nodes and associates the active root with the Pi entry; failure marks the complete prepared batch `failed`.
 
-Any mapping, model, output-limit, validation, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
+Fan-out and fan-in are bounded to 32 segment requests, 64 aggregate requests, and 16 aggregate passes. Transport replay is independently bounded to three attempts per segment or aggregate call and does not retry input, usage, rate, authentication, validation, or output-limit failures. A base prompt, individual message, atomic tool exchange, pair of child summaries, or total operation that cannot fit within those limits fails closed. Any mapping, budget, model, output-limit, validation, abort, or storage error returns `undefined` from the hook, allowing Pi's default compaction to run.
 
 ## Required summary contract
 
@@ -32,7 +32,7 @@ Any mapping, model, output-limit, validation, or storage error returns `undefine
 ## Critical Exact Values
 ```
 
-Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Segment and aggregate outputs are validated independently; either unrepaired failure prevents the whole graph batch from being installed.
+Every section must occur once, in order, and contain content or `- None`. DS4 replaces each unique `Files Read` and `Files Modified` section with one exact path per bullet from Pi's sanitized file-operation inventory before validation; missing or duplicate sections still fail. Backticked exact values must occur in the serialized segment source, ordered child-summary content, or those known file-operation paths. A bounded unsupported exact-value bullet is removed as a whole and recorded as a validation warning; unsupported prose, more than eight affected bullets, or removal above 25% still fails closed to Pi. The prompt explicitly asks the model to omit a bullet whose complete backticked span cannot be copied verbatim. Every segment is validated independently against only its own sanitized source and deterministic file inventory. Every aggregate is validated independently against only the sanitized content of its ordered children and cumulative deterministic file inventory. Any unrepaired failure prevents the whole graph batch from being installed.
 
 An unrepaired exact-value failure reports only the stage, issue code, categorical repair status, unsupported-span count, and affected-bullet count. Repair statuses distinguish an unsupported location, more than eight bullets, removal above 25%, and an unexpected invalid second validation. The disputed text is intentionally absent from logs, UI notifications, and diagnostics because it may contain sensitive source material.
 
@@ -76,4 +76,4 @@ It requests compaction at most once per session leaf. Pi's native threshold and
 /context summaries
 ```
 
-The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.
+The preview reports thresholds and eligibility; Pi remains authoritative for the exact cut point. Runtime diagnostics additionally report the calibrated input budget, estimated whole-source prompt size, generated segment count, aggregate-call count, and completed transport-retry count without source content. Each retry emits only stage, failed/next attempt, maximum attempts, and delay at `debug`. Routine `summary_graph_prepared` and `summary_graph_committed` lifecycle events are also emitted only at `debug`; fallback, failure, persistence, and reconciliation problems remain actionable warnings. Oversized-group and bounded-operation errors expose only numeric budgets/counts, never rejected source text. Provider failures are reduced to metadata-only categories such as `input-limit`, `usage-limit`, `rate-limit`, `authentication`, or `transport`; raw provider error details are not logged or shown. The proactive-threshold TUI notification remains a user-visible `info` notice because it explains why an automatic compaction started.

package/docs/CONTEXT_MANIFEST.md

@@ -44,7 +44,15 @@ actualInputTokens = input + cacheRead + cacheWrite
 rawCalibrationRatio = actualInputTokens / estimatedInputTokens
 ```
 
-The raw estimate is retained even after calibration so ratios cannot recursively calibrate already-corrected values. The next call reads only the exact provider/model window and applies bounded median/MAD outlier rejection. Error, aborted, missing, zero-usage, duplicate, or uncorrelated responses do not create calibration samples. Every manifest is correlated with at most one assistant response. See [`MODEL_AWARENESS.md`](MODEL_AWARENESS.md).
+The existing scalar SQLite columns are authoritative for persisted usage. `message_end` updates those columns and the calibration sample without reading or rewriting `manifest_json`; repository reads hydrate the usage fields from the scalars. The raw estimate is retained even after calibration so ratios cannot recursively calibrate already-corrected values. The next call reads only the exact provider/model window and applies bounded median/MAD outlier rejection. Error, aborted, missing, zero-usage, duplicate, or uncorrelated responses do not create calibration samples. Every manifest is correlated with at most one assistant response. If a concurrently pruned or oversize-skipped manifest cannot be correlated, DS4 keeps only bounded volatile calibration and does not retry against another row. See [`MODEL_AWARENESS.md`](MODEL_AWARENESS.md).
+
+## Retention
+
+Persisted diagnostic history keeps the latest 128 manifests globally. Calibration is retained independently at up to 200 samples per exact provider/model/estimator profile, so deleting an old large manifest does not prematurely discard a still-useful small calibration sample. Both limits are projection-only and require no schema migration.
+
+A manifest at or below 256 KiB is stored unchanged. Above that preferred bound, DS4 preserves `included` and all selected provenance, samples at most 256 `excluded` details deterministically (first 128 and last 128), and adds `ds4-context-manifest-inventory-v1` metadata with `excluded-rollup` completeness, complete counts/token aggregates, classification/kind rollups, and stable digests. Repository `getStored()` exposes the completeness wrapper; legacy rows without inventory metadata are treated as complete. The active in-memory manifest is never replaced by the sampled projection. If selected provenance plus rollup still exceeds 1 MiB, persistence is skipped without changing the model request. Schema-15 readers from an earlier release can still parse the additive JSON, but may present sampled `excluded` details as complete; downgrade support therefore excludes historical excluded-inventory rendering after rolled-up rows have been written.
+
+Each manifest transaction prunes at most 32 excess rows and 8 MiB of serialized payload; one individually oversized oldest row may exceed the byte limit to guarantee progress. Calibration pruning is independently limited to 32 rows per related profile write. This incrementally repairs an existing oversized database without adding a long startup write or extending SQLite lock duration with an unbounded purge. Deleted pages become reusable by SQLite; the database file may remain at its previous high-water size until explicit [offline maintenance](STORAGE_MAINTENANCE.md). No retention action edits canonical Pi JSONL or project files.
 
 ## Reproducibility
 
@@ -66,4 +74,5 @@ Use:
 /context ranking
 /context continuation
 /context artifacts
+/context storage
 ```

package/docs/CONTEXT_PERSISTENCE_TOOL.md

@@ -128,6 +128,6 @@ Useful diagnostics:
 /context rebuild-index
 ```
 
-## Alpha dogfooding
+## Beta dogfooding
 
-Use the published package with synthetic data and exercise TUI, RPC, print, and JSON behavior through the dedicated [`0.3 alpha dogfooding runbook`](DOGFOODING_0.3.0_ALPHA.md). The runbook distinguishes RPC's UI request/response bridge from genuinely no-UI print/JSON modes and includes a metadata-only canonical append audit.
+Use the published package with synthetic data and exercise TUI, RPC, print, and JSON behavior through the dedicated [`0.3 beta dogfooding runbook`](DOGFOODING_0.3.0_BETA.md). The runbook distinguishes RPC's UI request/response bridge from genuinely no-UI print/JSON modes and includes a metadata-only canonical append audit. The earlier [`0.3 alpha runbook`](DOGFOODING_0.3.0_ALPHA.md) remains available as historical release evidence.

package/docs/DOGFOODING_0.3.0_ALPHA.md

@@ -1,6 +1,6 @@
 # Dogfooding DS4 0.3.0 Alpha
 
-This runbook validates the published `ds4-context-engine@0.3.0-alpha.3` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
+This runbook validates the published `ds4-context-engine@0.3.0-alpha.5` package through sustained real Pi use. It complements automated tests and release smoke checks; it does not replace them.
 
 The primary target is the model-callable `context_persistence` surface. Pi JSONL must remain canonical and append-only, SQLite must remain rebuildable, and no model-callable write may occur without a fresh positive local UI decision.
 
@@ -10,7 +10,9 @@ The primary target is the model-callable `context_persistence` surface. Pi JSONL
 - Use a disposable trusted project and a dedicated session directory.
 - Use synthetic, non-secret Pin/Memory content. Local TUI dialogs and JSON event streams may display current tool arguments.
 - Published alpha.1 had a retry limitation: copying `[omitted-by-ds4-egress-policy]` from sanitized history could present a confirmation for the literal marker. Alpha.2 and later reserve that output-only marker and must reject it as `egress-placeholder` before runtime access, confirmation, canonical append, or derived-policy update.
-- Alpha.3 keeps exact-value summary validation strict. An unrepaired failure may expose only its stage, issue code, repair category, and counts; disputed exact text must not appear in diagnostics.
+- Alpha.3 and later keep exact-value summary validation strict. An unrepaired failure may expose only its stage, issue code, repair category, and counts; disputed exact text must not appear in diagnostics.
+- Alpha.4 distinguishes unsupported and missing action parameters without echoing rejected fields or values. Routine successful compaction lifecycle events require `diagnostics.logLevel=debug`; actionable fallback warnings remain visible.
+- Alpha.5 partitions an oversized compaction source into bounded atomic segments and recursively aggregates validated child summaries. An indivisible message or tool exchange still fails closed to Pi default compaction, and provider failures expose only a metadata category.
 - Do not use `--no-session` except for the explicit fail-closed test. Without a persistent Pi JSONL destination, both reads and writes return `runtime-unavailable`.
 - Do not retry `committed_projection_pending` or `indeterminate`. Inspect state with a read, `/context health`, or `/context rebuild-index` first.
 - Use `/context` only for local inspection and recovery. Mutations under test must go through `context_persistence` so the confirmation and provider-egress boundaries are exercised.
@@ -27,7 +29,7 @@ mkdir -p /tmp/ds4-alpha-dogfood
 cd /tmp/ds4-alpha-dogfood
 git init
 mkdir -p sessions evidence
-pi install -l npm:ds4-context-engine@0.3.0-alpha.3
+pi install -l npm:ds4-context-engine@0.3.0-alpha.5
 pi list
 pi --version
 ```
@@ -47,19 +49,19 @@ Record the Pi version, DS4 version, provider/model, mode, session name, expected
 Use unique non-sensitive values so duplicates from earlier runs cannot hide a failure. Example run label:
 
 ```text
-alpha2-run-01
+alpha5-run-01
 ```
 
 Example Pin:
 
 ```text
-For alpha2-run-01 verification, use Node.js 22 in this disposable project.
+For alpha5-run-01 verification, use Node.js 22 in this disposable project.
 ```
 
 Example Memory:
 
 ```text
-For alpha2-run-01, the synthetic release channel is amber.
+For alpha5-run-01, the synthetic release channel is amber.
 ```
 
 Never use real credentials, customer data, private paths, or production policy in dogfooding prompts.
@@ -77,7 +79,7 @@ Run these scenarios in order:
 
 1. Inspect `/context health`, `/context pins`, `/context memory`, and `/context privacy`.
 2. Send an ordinary suggestion without asking to persist it, for example: `The synthetic release channel amber seems useful.` No `context_persistence` call or confirmation dialog should appear.
-3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for alpha2-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
+3. Explicitly request the synthetic Memory: `Remember for this session that the synthetic release channel for alpha5-run-01 is amber.` Verify that the dialog identifies the action and canonical persistence class. Accept it. Expect one committed Memory mutation.
 4. Ask the model to use `context_persistence` to list active Memory. Verify bounded metadata and no complete claim, key, reason, path, or raw error in the result.
 5. Explicitly request the synthetic Pin, but reject or close the confirmation dialog. Verify with `/context pins` that it was not created.
 6. Ask the model to retry using the sanitized value remaining in history. If it copies `[omitted-by-ds4-egress-policy]`, expect `rejected / egress-placeholder` before any new confirmation, runtime mutation, or append. If it asks for fresh text instead, record that safe routing result and run the exact-marker case from the JSON procedure.
@@ -116,7 +118,7 @@ Enter one JSON object per line on stdin. First request a read:
 Wait for the turn to end before sending the next prompt. For a write:
 
 ```json
-{"id":"write-1","type":"prompt","message":"Persist a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22 for verification."}
+{"id":"write-1","type":"prompt","message":"Persist a session Pin for alpha5-run-01 stating that this disposable project uses Node.js 22 for verification."}
 ```
 
 Pi should emit a request shaped like:
@@ -168,7 +170,7 @@ The read should complete. Then request a write:
 
 ```bash
 pi -p --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
-  "Use context_persistence to add a session Memory saying that alpha2-run-01 uses the synthetic channel amber."
+  "Use context_persistence to add a session Memory saying that alpha5-run-01 uses the synthetic channel amber."
 ```
 
 Expected behavior:
@@ -198,7 +200,7 @@ Write case:
 
 ```bash
 pi --mode json --approve --session-dir "$DS4_DOGFOOD_SESSIONS" \
-  "Use context_persistence to add a session Pin for alpha2-run-01 stating that this disposable project uses Node.js 22." \
+  "Use context_persistence to add a session Pin for alpha5-run-01 stating that this disposable project uses Node.js 22." \
   2>evidence/json-write.stderr.log | tee evidence/json-write.jsonl
 ```
 
@@ -268,6 +270,30 @@ After the basic mode matrix passes, repeat the relevant TUI/RPC cases with:
 
 Provider, trust, branch, provenance, capability, target state, and classification changes after confirmation must fail safely. A model-supplied `local-only` classification is never evidence that earlier input stayed local.
 
+## Storage containment and offline maintenance candidate
+
+For a build that includes bounded storage support, keep normal online validation separate from physical maintenance.
+
+While Pi is running:
+
+1. Run `/context storage`; verify metadata-only output and no manifest/message/Pin/Memory/snippet/tool content.
+2. Run `/context health`; a storage high-water warning must produce `WARN` while SQLite quick/FK/schema checks remain independently visible.
+3. Perform multiple normal model calls and verify manifest count converges toward `128` without startup pause or provider-request changes.
+4. Verify provider usage remains visible after close/reopen and that `manifest_json` is not rewritten at `message_end` in a disposable test database.
+5. Start a second current-version Pi process against the same disposable database; both client leases must exist, and closing each process must remove only its own lease.
+
+Do not compact a live or production database as part of ordinary dogfooding. On an explicitly disposable copy only:
+
+1. close every Pi process;
+2. run `ds4-context-storage inspect --database <exact-copy-path>`;
+3. preserve the metadata output;
+4. run `compact` and complete the local TTY confirmation;
+5. verify the fixed backup exists, manifest/calibration limits converge, source exclusions remain, and quick/FK/schema checks pass;
+6. reopen the copy through Pi, run `/context health` and `/context storage`, then verify Pin/Memory/source-policy projections;
+7. retain the backup during observation and remove it manually afterward.
+
+Also verify refusal for an active client, an existing backup/stage, insufficient simulated disk, a corrupt source, and ambiguous recovery state. Maintenance of the actual user database requires a separate explicit authorization after online dogfooding succeeds. See [`STORAGE_MAINTENANCE.md`](STORAGE_MAINTENANCE.md).
+
 ## Result record
 
 Use one record per scenario: