ds4-context-engine 0.3.0-alpha.4 → 0.3.0-beta.1

package/docs/releases/0.3.0-beta.1.md

@@ -0,0 +1,81 @@
+# DS4 Context Engine 0.3.0-beta.1
+
+Status: release candidate; publication and exact registry verification pending.
+
+This coordinated beta contains SQLite growth caused by persisted Context Manifests and adds explicit, recoverable offline maintenance for already-large derived databases. It also hardens compaction provider retries while preserving strict grounding, privacy filtering, canonical Pi JSONL semantics, atomic tool exchanges, one final Pi compaction entry, and fail-open delegation to Pi.
+
+## Added
+
+- Retains at most 128 Context Manifests globally and at most 200 calibration samples per exact provider/model/estimator profile.
+- Bounds each online manifest prune to 32 rows and 8 MiB, while allowing one individually oversized oldest row to be removed so convergence cannot stall.
+- Persists complete manifests up to 256 KiB; larger manifests retain complete included provenance and a deterministic excluded-only rollup with complete counts, aggregates, and digests plus at most 256 sampled excluded details.
+- Skips manifest persistence above the 1 MiB hard bound without altering the active in-memory manifest or provider request.
+- Exposes persisted inventory completeness through `getStored()` while legacy complete rows remain readable through `get()`.
+- Adds metadata-only `/context storage` diagnostics and storage high-water integration in `/context health`.
+- Adds cooperative database client leases and a two-phase maintenance lock; active or ambiguous clients block physical maintenance.
+- Adds the `ds4-context-storage inspect|compact|recover --database <exact-path>` CLI. Mutating commands require a local TTY and have no non-interactive bypass.
+- Adds standalone backup, working-copy transformation, validated `VACUUM INTO` candidate, protected-table full-row digests, recoverable swap state, and deterministic crash recovery.
+
+## Changed
+
+- Provider usage is authoritative in scalar Context Manifest columns; `message_end` no longer rewrites `manifest_json`.
+- Only compaction failures categorized as `transport` are retried, with at most three attempts, abort-aware 200 ms and 500 ms delays, fresh routing session IDs, and cumulative usage accounting.
+- Runtime startup during maintenance degrades safely to Pi fallback with categorical diagnostics rather than exposing raw SQLite messages.
+- SQLite lock timeout and rollback diagnostics expose bounded categories, operation names, attempt counts, and numeric SQLite codes rather than raw database errors.
+- Package verification rejects database, WAL, SHM, backup, staging, retired-swap, maintenance-state, client-lease, JSONL, `.pi`, and `.serena` paths.
+
+## Safety and compatibility
+
+Pi JSONL remains canonical and append-only. Physical maintenance never edits session history or project files and is unavailable to model-callable tools. Every `context_persistence` write still requires a fresh positive local UI decision, exact destructive targeting remains revision-bound, and canonical Pin/Memory mutations still append through Pi before SQLite reconciliation.
+
+An individually oversized compaction group, unsupported exact value, non-transport provider failure, abort, or bounded-operation failure still delegates to Pi default compaction. Rejected values and provider payloads are not emitted in diagnostics.
+
+SQLite schema remains 15. Migrations 1–15, `ds4-context-config-v1`, `runtime-adapter-v1`, `ds4-context-persistence-tool-v1`, and `ds4-context-persistence-result-v1` are unchanged. Earlier schema-15 readers can parse additive rollup metadata but may present sampled excluded details as complete, so historical excluded-inventory rendering is unsupported after downgrade once rollups have been written.
+
+## Package/version policy
+
+The coordinated version is `0.3.0-beta.1` for:
+
+```text
+ds4-context-core
+ds4-context-reference-adapter
+ds4-context-engine
+```
+
+Both adapters depend exactly on `ds4-context-core@0.3.0-beta.1`. Publication uses the explicit npm `beta` dist-tag for all three packages while `latest` must remain stable `0.2.0`. GitHub Actions remains validation-only with OIDC and package-write permissions denied.
+
+## Candidate validation evidence
+
+Local candidate verification on Node.js `26.5.1`:
+
+- `npm run check`: 69 files and 341 tests passed.
+- Three consecutive focused multi-process SQLite concurrency runs passed.
+- `npm run quality:compare`: candidate quality `0.9875` versus baseline `0.808156`.
+- `npm run schema:context-persistence`: 1,266 bytes and 317 estimated tokens; below the 1,500 absolute and 320 relative limits.
+- `npm run latency:check -- <exact ds4-context-core@0.1.2>`: passed after clean install with ratio `0.93982`, at or below `1.10`, using 200 samples and 50 calls per sample.
+- `npm run pack:check`: verified core (223 files), reference adapter (7 files), and Pi adapter (67 files) in a clean consumer.
+- `npm pack --dry-run --json` for all three packages passed with the same inventories and no forbidden local/session/storage files.
+- `git diff --check`: passed.
+- Version, exact core dependencies, package-lock entries, extension constant, and reference-adapter constant are synchronized to `0.3.0-beta.1`.
+- Protected compatibility golden, Pi JSONL fixture, migration definitions, canonical persistence contracts, and CI publication permissions are unchanged.
+
+Large-database validation used a temporary private copy created from a read-only source connection; the live database was not compacted, vacuumed, rebuilt, replaced, or maintained:
+
+- before: schema 15, `quick_check=ok`, 3,691 manifests, 2,101,666,663 manifest bytes, 2,489,868,288 database bytes;
+- after: 128 manifests, 15,626,529 manifest bytes, 414 calibration samples across three profiles, 389,746,688 database bytes;
+- all 128 retained oversized manifests were rewritten as explicit `excluded-rollup` projections with no irreducible oversize;
+- the final candidate passed quick check and foreign-key validation while the standalone backup remained present;
+- temporary copy, backup, and stage files were removed afterward, with no maintenance artifact created next to the live database or in the repository.
+
+Publication, exact registry verification, annotated tag, and GitHub prerelease are performed only after the committed candidate passes validation-only CI.
+
+## Documentation
+
+- [`../CONTEXT_MANIFEST.md`](../CONTEXT_MANIFEST.md)
+- [`../STORAGE.md`](../STORAGE.md)
+- [`../STORAGE_MAINTENANCE.md`](../STORAGE_MAINTENANCE.md)
+- [`../COMPACTION.md`](../COMPACTION.md)
+- [`../ARCHITECTURE.md`](../ARCHITECTURE.md)
+- [`../DOGFOODING_0.3.0_BETA.md`](../DOGFOODING_0.3.0_BETA.md)
+- [`../RELEASING.md`](../RELEASING.md)
+- [`0.3.0-alpha.5.md`](0.3.0-alpha.5.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.0-alpha.4",
+  "version": "0.3.0-beta.1",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -31,12 +31,16 @@
     "docs",
     "quality",
     "scripts/compare-context-quality.mjs",
+    "scripts/ds4-context-storage.mjs",
     "README.md",
     "LICENSE"
   ],
   "exports": {
     ".": "./src/extension/index.ts"
   },
+  "bin": {
+    "ds4-context-storage": "./scripts/ds4-context-storage.mjs"
+  },
   "scripts": {
     "build:core": "npm run build --workspace ds4-context-core",
     "build:reference-adapter": "npm run build --workspace ds4-context-reference-adapter",
@@ -58,7 +62,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.0-alpha.4"
+    "ds4-context-core": "0.3.0-beta.1"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",

package/scripts/ds4-context-storage.mjs

@@ -0,0 +1,148 @@
+#!/usr/bin/env node
+import { createInterface } from "node:readline/promises";
+import { stdin, stdout } from "node:process";
+import { DatabaseProtocolError } from "ds4-context-core/persistence/database-client-lease";
+import {
+  StorageMaintenanceError,
+  compactStorage,
+  inspectStorage,
+  recoverStorage,
+  storageMaintenancePaths,
+} from "ds4-context-core/persistence/storage-maintenance";
+
+function usage() {
+  return [
+    "Usage:",
+    "  ds4-context-storage inspect --database <exact-path>",
+    "  ds4-context-storage compact --database <exact-path>",
+    "  ds4-context-storage recover --database <exact-path>",
+    "",
+    "compact and recover require an interactive local TTY confirmation.",
+  ].join("\n");
+}
+
+function parseArguments(argv) {
+  const command = argv[0];
+  if (command !== "inspect" && command !== "compact" && command !== "recover") {
+    throw new Error("usage");
+  }
+  let database;
+  for (let index = 1; index < argv.length; index++) {
+    const argument = argv[index];
+    if (argument !== "--database" || database !== undefined || !argv[index + 1]) {
+      throw new Error("usage");
+    }
+    database = argv[++index];
+  }
+  if (!database) throw new Error("usage");
+  return { command, database };
+}
+
+function formatBytes(value) {
+  const units = ["B", "KiB", "MiB", "GiB", "TiB"];
+  let scaled = value;
+  let unit = 0;
+  while (scaled >= 1024 && unit < units.length - 1) {
+    scaled /= 1024;
+    unit++;
+  }
+  const digits = unit === 0 ? 0 : scaled >= 100 ? 0 : scaled >= 10 ? 1 : 2;
+  return `${scaled.toFixed(digits)} ${units[unit]}`;
+}
+
+function printInspection(inspection) {
+  const storage = inspection.diagnostics;
+  console.log([
+    "DS4 Context Storage Inspection",
+    "",
+    `Database:                    ${inspection.databasePath}`,
+    `Schema / journal:            ${inspection.schemaVersion} / ${storage.journalMode ?? "n/a"}`,
+    `quick_check / foreign keys:  ${inspection.quickCheck} / ${inspection.foreignKeyViolations}`,
+    `Database / WAL:              ${formatBytes(storage.databaseBytes ?? 0)} / ${formatBytes(storage.walBytes ?? 0)}`,
+    `Allocated / reusable:        ${formatBytes(storage.allocatedBytes ?? 0)} / ${formatBytes(storage.reusableBytes ?? 0)}`,
+    `Manifests / target:          ${storage.manifests.rows} / ${storage.manifests.retainedLimit}`,
+    `Manifest payload:            ${formatBytes(storage.manifests.serializedBytes)}`,
+    `Manifests to prune:          ${inspection.manifestsToPrune}`,
+    `Calibration to prune:        ${inspection.calibrationToPrune}`,
+    `Available / required space:  ${formatBytes(inspection.availableBytes)} / ${formatBytes(inspection.requiredBytes)}`,
+    `Backup:                      ${inspection.paths.backup}`,
+    `Maintenance recommended:     ${storage.maintenance.recommended ? "yes" : "no"}`,
+    ...storage.maintenance.reasons.map((reason) => `Reason:                      ${reason}`),
+  ].join("\n"));
+}
+
+async function confirm(action, database) {
+  if (!stdin.isTTY || !stdout.isTTY) {
+    throw new StorageMaintenanceError("interactive-confirmation-required", "confirmation");
+  }
+  console.log([
+    "",
+    `Exact database path: ${database}`,
+    "All Pi instances must be closed before this operation.",
+    "The source database will not be overwritten in place.",
+  ].join("\n"));
+  const expected = action.toUpperCase();
+  const prompt = createInterface({ input: stdin, output: stdout });
+  try {
+    const answer = await prompt.question(`Type ${expected} to continue: `);
+    if (answer !== expected) throw new StorageMaintenanceError("confirmation-declined", "confirmation");
+  } finally {
+    prompt.close();
+  }
+}
+
+async function main() {
+  let parsed;
+  try {
+    parsed = parseArguments(process.argv.slice(2));
+  } catch {
+    console.error(usage());
+    process.exitCode = 2;
+    return;
+  }
+
+  if (parsed.command === "inspect") {
+    printInspection(inspectStorage(parsed.database));
+    return;
+  }
+
+  if (parsed.command === "compact") {
+    const inspection = inspectStorage(parsed.database);
+    printInspection(inspection);
+    await confirm("compact", inspection.databasePath);
+    const result = await compactStorage(inspection.databasePath);
+    console.log([
+      "",
+      "DS4 Context Storage Compaction Completed",
+      `Database:              ${result.databasePath}`,
+      `Backup retained:       ${result.backupPath}`,
+      `Before / after:        ${formatBytes(result.beforeBytes)} / ${formatBytes(result.afterBytes)}`,
+      `Manifests pruned:      ${result.maintenance.prunedManifests}`,
+      `Manifests rolled up:   ${result.maintenance.rolledUpManifests}`,
+      `Oversize skipped:      ${result.maintenance.skippedOversizeManifests}`,
+      `Calibration pruned:    ${result.maintenance.prunedCalibrationSamples}`,
+      "Run /context health and /context storage after reopening Pi.",
+    ].join("\n"));
+    return;
+  }
+
+  const paths = storageMaintenancePaths(parsed.database);
+  await confirm("recover", paths.source);
+  const result = recoverStorage(paths.source);
+  console.log([
+    "DS4 Context Storage Recovery Completed",
+    `Database: ${result.databasePath}`,
+    `Action:   ${result.action}`,
+  ].join("\n"));
+}
+
+try {
+  await main();
+} catch (error) {
+  if (error instanceof StorageMaintenanceError || error instanceof DatabaseProtocolError) {
+    console.error(error.message);
+  } else {
+    console.error("Storage maintenance failed (category=unexpected-local-failure)");
+  }
+  process.exitCode = 1;
+}

package/src/extension/commands.ts

@@ -3,6 +3,7 @@ import {
   isPrivacyClassification,
   type PrivacyClassification,
 } from "ds4-context-core/privacy/privacy-policy";
+import type { StorageDiagnostics } from "ds4-context-core/persistence/storage-diagnostics";
 import type {
   Ds4ContextRuntime,
   RuntimeDiagnostics,
@@ -33,6 +34,7 @@ const SUBCOMMANDS = [
   "artifacts",
   "compaction",
   "compact-preview",
+  "storage",
   "health",
   "rebuild-index",
 ] as const;
@@ -50,6 +52,19 @@ function count(value: number | undefined): string {
   return value === undefined ? "n/a" : NUMBER_FORMAT.format(value);
 }
 
+function bytes(value: number | undefined): string {
+  if (value === undefined) return "n/a";
+  const units = ["B", "KiB", "MiB", "GiB", "TiB"];
+  let scaled = value;
+  let unit = 0;
+  while (scaled >= 1024 && unit < units.length - 1) {
+    scaled /= 1024;
+    unit++;
+  }
+  const digits = unit === 0 ? 0 : scaled >= 100 ? 0 : scaled >= 10 ? 1 : 2;
+  return `${scaled.toFixed(digits)} ${units[unit]}`;
+}
+
 function present(ctx: ExtensionCommandContext, message: string, level: "info" | "warning" | "error" = "info"): void {
   if (ctx.hasUI) ctx.ui.notify(message, level);
   else console.log(message);
@@ -251,6 +266,7 @@ function formatManifest(diagnostics: RuntimeDiagnostics): string {
   const manifest = diagnostics.lastManifest;
   if (!manifest) return "No Context Manifest has been built for this session yet.";
 
+  const inventory = manifest.persistedInventory;
   const kinds = new Map<string, { items: number; tokens: number }>();
   for (const item of manifest.included) {
     const aggregate = kinds.get(item.kind) ?? { items: 0, tokens: 0 };
@@ -273,7 +289,11 @@ function formatManifest(diagnostics: RuntimeDiagnostics): string {
     `Actual input:       ${count(manifest.actualInputTokens)}`,
     `Usage input/cache:  ${count(manifest.providerUsage?.inputTokens)} / ${count(manifest.providerUsage?.cacheReadTokens)} read / ${count(manifest.providerUsage?.cacheWriteTokens)} write`,
     `Target / hard:      ${count(manifest.targetInputTokens)} / ${count(manifest.hardInputLimit)}`,
-    `Included / excluded:${count(manifest.included.length)} / ${count(manifest.excluded.length)}`,
+    `Included / excluded:${count(manifest.included.length)} / ${count(inventory?.excluded.total ?? manifest.excluded.length)}`,
+    `Persisted inventory: ${inventory?.completeness ?? "complete"}`,
+    ...(inventory?.completeness === "excluded-rollup" ? [
+      `Excluded details:  ${count(inventory.excluded.retained)} / ${count(inventory.excluded.total)} retained in persisted projection`,
+    ] : []),
     `Planning mode:      ${manifest.planning?.mode ?? "observer"}`,
     `Original messages:  ${count(manifest.planning?.originalMessageCount ?? manifest.composition.messageCount)}`,
     `Selected messages:  ${count(manifest.composition.messageCount)}`,
@@ -299,9 +319,13 @@ function formatManifestItems(diagnostics: RuntimeDiagnostics, type: "included" |
   const manifest = diagnostics.lastManifest;
   if (!manifest) return "No Context Manifest has been built for this session yet.";
   const items = manifest[type];
+  const inventory = manifest.persistedInventory;
   return [
     `DS4 Context ${type === "included" ? "Included" : "Excluded"} Items`,
     "",
+    ...(type === "excluded" && inventory?.completeness === "excluded-rollup"
+      ? [`Persisted projection: ${count(inventory.excluded.retained)} / ${count(inventory.excluded.total)} excluded details retained; this is not the complete historical inventory.`, ""]
+      : []),
     ...(items.length === 0
       ? ["none"]
       : items.map((item, index) => {
@@ -717,6 +741,43 @@ function formatAdapter(diagnostics: RuntimeDiagnostics): string {
   ].join("\n");
 }
 
+function formatStorage(storage: StorageDiagnostics, databasePath?: string): string {
+  if (storage.status === "unavailable") {
+    return [
+      "DS4 Storage",
+      "",
+      "Status:                     unavailable",
+      `Database:                   ${databasePath ?? "unavailable"}`,
+      "Pi fallback remains active; no storage mutation was attempted.",
+    ].join("\n");
+  }
+  return [
+    "DS4 Storage",
+    "",
+    `Status:                     ${storage.status}`,
+    `Database:                   ${databasePath ?? "unavailable"}`,
+    `Schema / journal:           ${storage.schemaVersion ?? "n/a"} / ${storage.journalMode ?? "n/a"}`,
+    `Database / WAL / SHM:       ${bytes(storage.databaseBytes)} / ${bytes(storage.walBytes)} / ${bytes(storage.shmBytes)}`,
+    `Allocated / reusable:       ${bytes(storage.allocatedBytes)} / ${bytes(storage.reusableBytes)}`,
+    `Pages total / reusable:     ${count(storage.pageCount)} / ${count(storage.freePages)}`,
+    `Manifests:                  ${count(storage.manifests.rows)} / target ${count(storage.manifests.retainedLimit)}`,
+    `Manifest payload:           ${bytes(storage.manifests.serializedBytes)}`,
+    `Rolled-up manifests:        ${count(storage.manifests.rolledUpRows)}`,
+    `Calibration samples/profiles: ${count(storage.calibration.rows)} / ${count(storage.calibration.profiles)} (limit ${count(storage.calibration.retainedPerProfile)}/profile)`,
+    `Sessions:                   ${count(storage.sessions)}`,
+    `Artifact objects / refs:    ${count(storage.artifacts.objects)} / ${count(storage.artifacts.references)}`,
+    `Artifact bytes:             ${bytes(storage.artifacts.bytes)}`,
+    ...(storage.activeProject ? [
+      `Active project files:       ${count(storage.activeProject.files)}`,
+      `Project snippets/stale:     ${count(storage.activeProject.snippets)} / ${count(storage.activeProject.staleSnippets)}`,
+      `Project indexed tokens:     ${count(storage.activeProject.indexedTokens)}`,
+    ] : []),
+    `Retention converged:        ${storage.retention.converged ? "yes" : "no"}`,
+    `Offline maintenance:       ${storage.maintenance.recommended ? "recommended" : "not required"}`,
+    ...storage.maintenance.reasons.map((reason) => `Reason:                     ${reason}`),
+  ].join("\n");
+}
+
 function formatCompaction(diagnostics: RuntimeDiagnostics, preview: boolean): string {
   const compaction = diagnostics.compaction;
   return [
@@ -734,6 +795,11 @@ function formatCompaction(diagnostics: RuntimeDiagnostics, preview: boolean): st
     `Last trigger:            ${compaction.trigger ?? "n/a"}`,
     `Summary ID:              ${compaction.summaryId ?? "n/a"}`,
     `Source entries:          ${count(compaction.sourceEntries)}`,
+    `Input budget:            ${count(compaction.inputBudgetTokens)}`,
+    `Whole-source prompt:     ${count(compaction.sourcePromptTokens)}`,
+    `Generated segments:      ${count(compaction.segmentCount)}`,
+    `Aggregate calls:         ${count(compaction.aggregateCalls)}`,
+    `Transport retries:       ${count(compaction.transportRetries)}`,
     `Validation:              ${compaction.validationStatus ?? "n/a"}`,
     `First kept entry:        ${compaction.firstKeptEntryId ?? "n/a"}`,
     `Tokens before:           ${count(compaction.tokensBefore)}`,
@@ -1003,6 +1069,17 @@ export function registerContextCommand(pi: ExtensionAPI, runtime: Ds4ContextRunt
           return;
         }
 
+        if (subcommand === "storage") {
+          const diagnostics = runtime.diagnostics(ctx);
+          const storage = runtime.storageDiagnostics();
+          present(
+            ctx,
+            formatStorage(storage, diagnostics.databasePath),
+            storage.status === "ok" ? "info" : "warning",
+          );
+          return;
+        }
+
         if (subcommand === "rebuild-index") {
           await ctx.waitForIdle();
           const result = runtime.rebuildIndex(ctx);
@@ -1033,6 +1110,7 @@ export function registerContextCommand(pi: ExtensionAPI, runtime: Ds4ContextRunt
           }
           runtime.verifyArtifactHealth(ctx);
           const diagnostics = runtime.diagnostics(ctx);
+          const storage = runtime.storageDiagnostics();
           const staleProjectSnippets = diagnostics.project.stats?.staleSnippets ?? 0;
           const artifactIntegrityIssues = diagnostics.artifacts.stats.missing + diagnostics.artifacts.stats.corrupt;
           const healthy = health.ok
@@ -1046,7 +1124,8 @@ export function registerContextCommand(pi: ExtensionAPI, runtime: Ds4ContextRunt
             && diagnostics.quality.lastError === undefined
             && diagnostics.ranking.warnings.length === 0
             && artifactIntegrityIssues === 0
-            && diagnostics.artifacts.warnings.length === 0;
+            && diagnostics.artifacts.warnings.length === 0
+            && storage.status === "ok";
           present(
             ctx,
             [
@@ -1058,6 +1137,8 @@ export function registerContextCommand(pi: ExtensionAPI, runtime: Ds4ContextRunt
               `Foreign keys:        ${health.foreignKeys ? "enabled" : "disabled"}`,
               `Schema version:      ${health.schemaVersion}`,
               `Applied migrations:  ${health.appliedMigrations}`,
+              `Storage status:      ${storage.status}`,
+              `Maintenance recommended: ${storage.maintenance.recommended ? "yes" : "no"}`,
               `Project stale snippets: ${count(staleProjectSnippets)}`,
               `Memory/pin warnings: ${count(diagnostics.memory.warnings.length)}`,
               `Privacy enforcement:  ${diagnostics.privacy.enforcement}`,

package/src/extension/runtime.ts

@@ -66,6 +66,11 @@ import type {
   PrivacyManifest,
   ProviderUsageManifest,
 } from "ds4-context-core/manifest/context-manifest";
+import { HARD_PERSISTED_MANIFEST_BYTES } from "ds4-context-core/manifest/context-manifest-storage";
+import {
+  unavailableStorageDiagnostics,
+  type StorageDiagnostics,
+} from "ds4-context-core/persistence/storage-diagnostics";
 import type { ExcludedContextSource, ObservedTool } from "ds4-context-core/manifest/observer";
 import {
   disabledCrossSessionMemoryDiagnostics,
@@ -133,6 +138,7 @@ import {
 } from "ds4-context-core/retrieval/retrieval-engine";
 import { currentRequestText } from "ds4-context-core/retrieval/task-descriptor";
 import { LocalFeatureHashEmbedding } from "../pi-adapter/local-embedding.ts";
+import { DatabaseProtocolError } from "ds4-context-core/persistence/database-client-lease";
 import { ContextDatabase, type DatabaseHealth, type SessionIndexStats } from "ds4-context-core/persistence/sqlite";
 import {
   activeTools,
@@ -371,6 +377,7 @@ export class Ds4ContextRuntime {
   private observation?: ContextObservation;
   private lastManifest?: ContextManifest;
   private pendingManifestId?: string;
+  private pendingManifestPersisted = false;
   private retrievalEngine?: HistoricalRetrievalEngine;
   private lastRetrieval: RetrievalDiagnostics = emptyRetrievalDiagnostics();
   private semanticIndex?: SemanticEmbeddingIndex;
@@ -440,6 +447,7 @@ export class Ds4ContextRuntime {
     this.lastIndexResult = undefined;
     this.lastManifest = undefined;
     this.pendingManifestId = undefined;
+    this.pendingManifestPersisted = false;
     this.retrievalEngine = undefined;
     this.semanticIndex = undefined;
     this.semanticFallbackReason = undefined;
@@ -1228,9 +1236,7 @@ export class Ds4ContextRuntime {
       }
       this.lastManifest = manifest;
       this.pendingManifestId = manifest.id;
-      if (this.config.diagnostics.storeContextManifest && this.session?.sessionFile) {
-        this.database?.manifests.save(manifest);
-      }
+      this.pendingManifestPersisted = this.persistManifest(manifest);
 
       this.logger.trace("context.planned", {
         sessionId: this.session?.sessionId,
@@ -1311,6 +1317,47 @@ export class Ds4ContextRuntime {
     }
   }
 
+  private persistManifest(manifest: ContextManifest): boolean {
+    if (!this.config.diagnostics.storeContextManifest || !this.session?.sessionFile || !this.database) {
+      return false;
+    }
+    try {
+      const result = this.database.manifests.save(manifest);
+      if (result.status === "skipped-oversize") {
+        this.logger.warn("context.manifest_persistence_skipped", {
+          category: "oversize",
+          sourceBytes: result.sourceBytes,
+          projectedBytes: result.projectedBytes,
+          includedCount: manifest.included.length,
+          excludedCount: manifest.excluded.length,
+          hardLimitBytes: HARD_PERSISTED_MANIFEST_BYTES,
+        });
+        return false;
+      }
+      this.logger.debug("context.manifest_persisted", {
+        manifestId: manifest.id,
+        completeness: result.completeness,
+        sourceBytes: result.sourceBytes,
+        storedBytes: result.storedBytes,
+      });
+      if (result.prunedManifests > 0) {
+        this.logger.debug("database.retention_progress", {
+          category: "context-manifest",
+          prunedRows: result.prunedManifests,
+          prunedBytes: result.prunedBytes,
+        });
+      }
+      return true;
+    } catch (error) {
+      this.logger.warn("context.manifest_persistence_skipped", {
+        category: "storage-write",
+        manifestId: manifest.id,
+        errorType: error instanceof Error ? error.name : "UnknownError",
+      });
+      return false;
+    }
+  }
+
   private updateManifestPrivacy(): void {
     if (!this.lastManifest?.privacy) return;
     const updatedPrivacy = this.privacyManifest(
@@ -1322,8 +1369,9 @@ export class Ds4ContextRuntime {
     );
     if (!updatedPrivacy) return;
     this.lastManifest = { ...this.lastManifest, privacy: updatedPrivacy };
-    if (this.config.diagnostics.storeContextManifest && this.session?.sessionFile) {
-      this.database?.manifests.save(this.lastManifest);
+    const persisted = this.persistManifest(this.lastManifest);
+    if (this.pendingManifestId === this.lastManifest.id) {
+      this.pendingManifestPersisted = this.pendingManifestPersisted || persisted;
     }
   }
 
@@ -1453,14 +1501,9 @@ export class Ds4ContextRuntime {
       ...this.lastManifest,
       nativeContinuation: { ...continuation },
     };
-    if (!this.config.diagnostics.storeContextManifest || !this.session?.sessionFile) return;
-    try {
-      this.database?.manifests.save(this.lastManifest);
-    } catch (error) {
-      this.logger.warn("native_continuation.manifest_persistence_failed", {
-        manifestId,
-        errorType: error instanceof Error ? error.name : "UnknownError",
-      });
+    const persisted = this.persistManifest(this.lastManifest);
+    if (this.pendingManifestId === manifestId) {
+      this.pendingManifestPersisted = this.pendingManifestPersisted || persisted;
     }
   }
 
@@ -1470,7 +1513,9 @@ export class Ds4ContextRuntime {
     if (record.role !== "assistant") return;
 
     const manifestId = this.pendingManifestId;
+    const manifestPersisted = this.pendingManifestPersisted;
     this.pendingManifestId = undefined;
+    this.pendingManifestPersisted = false;
     if (record.stopReason === "error" || record.stopReason === "aborted") return;
     const usage = record.usage;
     if (!usage || typeof usage !== "object") return;
@@ -1491,19 +1536,17 @@ export class Ds4ContextRuntime {
       };
     }
     const createdAt = this.now();
-    const persisted = Boolean(
-      this.config.diagnostics.storeContextManifest
-      && this.session?.sessionFile
-      && this.database,
-    );
     try {
-      if (persisted) {
-        const updated = this.database?.manifests.recordProviderUsage(
+      if (manifestPersisted && this.database) {
+        const updated = this.database.manifests.recordProviderUsage(
           manifestId,
           providerUsage,
           createdAt,
+          manifest?.modelAwareness?.calibration.estimator ?? "chars-v1",
         );
-        if (updated && this.lastManifest?.id === manifestId) this.lastManifest = updated;
+        if (!updated && manifest?.estimatedInputTokens) {
+          this.rememberVolatileCalibration(manifest, providerUsage, createdAt);
+        }
       } else if (manifest?.estimatedInputTokens) {
         this.rememberVolatileCalibration(manifest, providerUsage, createdAt);
       }
@@ -1513,7 +1556,8 @@ export class Ds4ContextRuntime {
       }
       this.logger.warn("context.usage_persistence_failed", {
         manifestId,
-        error: error instanceof Error ? error.message : String(error),
+        category: "storage-write",
+        errorType: error instanceof Error ? error.name : "UnknownError",
       });
     }
     this.logger.debug("context.actual_usage_recorded", {
@@ -2913,6 +2957,11 @@ export class Ds4ContextRuntime {
     return this.database.health();
   }
 
+  storageDiagnostics(): StorageDiagnostics {
+    return this.database?.storageDiagnostics(this.session?.projectPath)
+      ?? unavailableStorageDiagnostics();
+  }
+
   shutdown(ctx?: ExtensionContext): void {
     if (ctx) this.syncSessionIndex(ctx);
     this.flushContextQuality();
@@ -2931,8 +2980,16 @@ export class Ds4ContextRuntime {
     this.nativeContinuation.invalidate("runtime-degraded");
     this.closeDatabase();
     this.phase = "degraded";
-    this.lastError = error instanceof Error ? error.message : String(error);
-    this.logger.error("runtime.degraded", { stage, error: this.lastError });
+    const maintenanceActive = error instanceof DatabaseProtocolError && error.category === "maintenance-active";
+    const errorType = error instanceof Error ? error.name : "UnknownError";
+    this.lastError = maintenanceActive
+      ? "Database maintenance is active (category=maintenance-active)"
+      : `Database startup unavailable (category=storage-unavailable; errorType=${errorType})`;
+    if (maintenanceActive) {
+      this.logger.warn("database.maintenance_active", { category: "maintenance-active" });
+    } else {
+      this.logger.error("runtime.degraded", { stage, category: "storage-unavailable", errorType });
+    }
     this.setStatus(ctx, "DS4 ctx: Pi fallback");
     if (ctx.hasUI) {
       ctx.ui.notify(`DS4 Context Engine unavailable; using Pi context. ${this.lastError}`, "warning");