ds4-context-engine 0.3.10 → 0.4.0

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** The coordinated `0.3.10` release adds opt-in BPE token estimation, estimator-specific provider calibration and bounded auto-tuning of context category ceilings. `chars-v1` and disabled auto-tuning remain the defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. See the [0.3.10 release record](docs/releases/0.3.10.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
+> **Project status:** The coordinated `0.4.0` release adds `storage.scope: "agent" | "project"`, now defaulting to per-project SQLite projections with shared token calibration in the agent database (opt out with `storage.scope: "agent"`). The opt-in BPE estimation and bounded auto-tuning from `0.3.10` keep `chars-v1` and disabled auto-tuning as their defaults. The bounded compaction controls from `0.3.9` remain in place; canonical history, SQLite schema 16 and runtime contracts are unchanged. Pi remains pinned to `0.84.3`. See the [0.4.0 release record](docs/releases/0.4.0.md), [ADR 064](docs/ADR/064-per-project-databases-with-shared-calibration.md) and [model-awareness validation](docs/MODEL_AWARENESS.md).
 
 **Current compaction defaults:** `compaction.directUpdate=true`, `compaction.inputBudget="context"`, `compaction.segmentTargetTokens=30000`, `compaction.maxRequestInputTokens=64000`, `compaction.maxOperationInputTokens=2000000`, `compaction.maxConcurrentSegments=2`. Every DS4 provider attempt is bounded by the effective request limit, and the operation limit includes retries; `inputBudget="summary"` remains an explicit throughput-oriented opt-in. Existing compaction/master switches still apply. See [latency controls and compatibility](docs/COMPACTION.md#latency-controls). No real-provider speedup is claimed from mock tests. The five optional editing/reading/artifact/job features introduced in `0.3.4` remain default-off.
 
@@ -385,6 +385,7 @@ The following example shows the main configuration groups. Omitted values use th
     "logLevel": "info"
   },
   "storage": {
+    "scope": "project",
     "databasePath": "ds4-context/context.db",
     "busyTimeoutMs": 5000,
     "writeRetryTimeoutMs": 30000,
@@ -514,6 +515,7 @@ scripts             package and release-readiness checks
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
 - [0.2.0 release readiness](docs/RELEASE_READINESS_0.2.0.md)
+- [0.4.0 release notes](docs/releases/0.4.0.md)
 - [0.3.10 release notes](docs/releases/0.3.10.md)
 - [0.3.9 release notes](docs/releases/0.3.9.md)
 - [0.3.8 release notes](docs/releases/0.3.8.md)
@@ -542,7 +544,7 @@ scripts             package and release-readiness checks
 
 The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) is complete. The stable 0.3 line carries forward the [context persistence tool](docs/CONTEXT_PERSISTENCE_TOOL.md), privacy-safe [compaction](docs/COMPACTION.md), bounded persisted manifests, cooperative client leases and recoverable offline maintenance. Version 0.4.0 introduces `storage.scope` (`"agent" | "project"`, default `"project"`): one rebuildable SQLite projection per trusted canonical project root, with token calibration shared in the agent database; `storage.scope: "agent"` restores the previous single-file layout. Version 0.3.10 adds opt-in BPE estimation and bounded model-budget auto-tuning while retaining the `chars-v1` default. Version 0.3.9 extends the bounded compaction updates, summary input headroom, concurrent segments and phase timings introduced in 0.3.5 with per-request and cumulative operation input limits; 0.3.8 adds indexed FTS key deletion without changing search results. The opt-in [anchored editing](docs/ANCHORED_EDITING.md) and [portable agent tools](docs/PORTABLE_AGENT_TOOLS.md) from 0.3.4 remain default-off, without backend rewind, forced sampling or operational KV integration. Confirmation, provenance, Pi fallback and canonical/configuration/SQLite/runtime contracts remain unchanged. The [0.2 readiness record](docs/RELEASE_READINESS_0.2.0.md) remains the compatibility baseline; the lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ADR/064-per-project-databases-with-shared-calibration.md

@@ -0,0 +1,88 @@
+# 064 — Per-project databases with shared token calibration
+
+**Date:** 2026-09-25
+**Status:** Accepted
+**Related:** [002](002-pi-jsonl-canonical-sqlite-rebuildable.md), [058](058-bounded-manifest-storage.md), [063](063-fts-rowid-key-mappings.md)
+
+## Context
+
+The storage plan decided **D1**: keep one shared SQLite projection at
+`~/.pi/agent/ds4-context/context.db` for every Pi session and explicitly avoid
+per-session *or per-project* databases. That projection is deliberately
+derived, rebuildable and disposable.
+
+The session index (`entries`, `entries_fts`) is the table that actually grows
+with total indexed history, and automatic eviction remains deferred until an
+on-demand rehydration path is verified. A single file therefore also means a
+single growth boundary, a single write lock and a single point of physical
+reset shared by unrelated projects. Content is already partitioned logically
+by `sessions.project_path`, `project_states`, `project_files` and
+`project_memory_sessions`; the file boundary was the only thing missing.
+
+The one piece of genuinely global learning is token calibration. Schema v10's
+`token_calibration` is keyed by exact `provider + model + estimator_version`
+and has **no project column**: a naive per-project split would restart
+calibration for every project (minimum 3 samples, 8 accepted for the opt-in
+`autoTune` expansion), weakening exactly the BPE/auto-tuning path it should
+protect.
+
+## Decision
+
+Add `storage.scope: "agent" | "project"` (default `"project"`) to the storage
+configuration:
+
+- `agent` keeps the previous shared-database behavior and remains selectable.
+- `project` derives one database per trusted canonical project root:
+  `projects/<sha256(canonicalRoot)[0..32]>.db`, next to the configured agent
+  database. Untrusted projects, broad roots (home directory, filesystem root)
+  and any resolution failure fall back to the agent database.
+
+Both files receive the **same schema and the same migrations**; there is no
+schema fork and migrations 1–15 are untouched. The split changes only which
+repository each handle is used for:
+
+- the agent database keeps `token_calibration` (shared learning);
+- the project database keeps the session index, project index, context
+  manifests, summary graph, memory/pin projections, embeddings, quality
+  samples and artifact metadata; artifact object bytes move to
+  `projects/artifacts/<project-digest>/` so the orphan garbage collector,
+  which only sees references in the current database, can never delete
+  another project's objects;
+- `resource_leases` and the client lease stay per file, protecting each
+  database independently.
+
+With `project`, a calibration sample is derived from the project manifest and
+inserted into the agent database with `manifest_id = NULL` (the column and its
+partial unique index already allow this). The two writes are intentionally
+**not** one cross-database transaction: losing one calibration sample is
+harmless, whereas losing manifest/usage consistency is not. In `agent` scope
+the previous single transaction is unchanged. Manifest pruning only detaches
+calibration rows in `agent` scope; the project database's calibration table
+stays empty.
+
+Every project database starts empty and is rebuilt from canonical Pi JSONL,
+project files and memory/pin `CustomEntry` records. The previous shared
+database is left untouched: with the default change, existing users cold-start
+their per-project indexes while their existing calibration remains available
+in the agent database.
+
+## Consequences
+
+- Physical isolation per project: separate growth, separate write lock,
+  "reset project state" = remove one file, and no eviction needed to bound a
+  single project's index.
+- Calibration stays global: a sample learned in project A is immediately
+  visible in project B for the same provider/model/estimator.
+- Default behavior changes. A pre-existing shared database becomes the agent
+  database (calibration and old manifests) and is no longer the active
+  projection for new sessions. This requires a minor release and release
+  notes; `storage.scope: "agent"` restores the old layout.
+- Maintenance and diagnostics become per file: `/context storage` reports the
+  active project database and, when split, the shared agent database;
+  `ds4-context-storage inspect|compact|recover --database <path>` must be
+  pointed at each file.
+- `storage.databasePath` now names the agent database; project databases
+  derive from its directory. A manually configured per-project path keeps
+  working, but the derived `projects/` directory is the supported layout.
+- D1 is superseded by this ADR; the development plan keeps the original text
+  with an explicit amendment pointer.

package/docs/ADR/README.md

@@ -67,5 +67,6 @@ The initial decisions from the development plan are accepted:
 | [061](061-compaction-latency.md) | Bound compaction update calls, input budgets, concurrent segments and phase timings | Accepted |
 | [062](062-cache-aware-context-planning.md) | Opt-in cache-aware tail planning using model pricing and observed cache shares | Accepted |
 | [063](063-fts-rowid-key-mappings.md) | Resolve FTS key deletes through rowid mapping tables | Accepted |
+| [064](064-per-project-databases-with-shared-calibration.md) | Split project state into per-project databases and keep token calibration shared | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/MODEL_AWARENESS.md

@@ -86,6 +86,8 @@ Until enough samples exist, the multiplier remains `1.0`. The default window is
 
 Provider-token capacities are computed first from context window, output reserve, safety margin, and policy ratios. Global hard/soft/preferred input limits are converted into local-estimator units using **at least** a multiplier of 1: observed underestimation may reduce them, but apparent overestimation on short prompts never raises them above nominal model limits. Raw manifest estimates remain uncalibrated so future samples do not feed a corrected estimate back into itself. Adaptive tail/history/project budgets use the accepted multiplier but remain capped by the configured `context.*` maxima even after conversion.
 
+Calibration samples are global learning, not project data: with the default `storage.scope: "project"` the manifests live in the per-project database while `token_calibration` stays in the shared agent database (see [ADR 064](ADR/064-per-project-databases-with-shared-calibration.md)). A sample learned in one project therefore applies to every other project for the same provider/model/estimator.
+
 ### Optional BPE estimator and measured budget tuning
 
 The default remains `chars-v1`. To opt a profile into local OpenAI `o200k_base` BPE text counting (without fetching vocabulary over the network):

package/docs/STORAGE.md

@@ -10,6 +10,19 @@ Pi's session JSONL is canonical for conversations and live project files are can
 
 The extension never edits or rewrites Pi JSONL or project source files. Manual memory/pin commands, confirmed `context_persistence` canonical writes, and learned-ranking feedback append versioned classified Pi `CustomEntry` records through Pi's official `appendEntry()` API. The tool does not write SQLite as a substitute for a canonical Pin or Memory append.
 
+## Storage scope
+
+`storage.scope` selects where the disposable projection lives (see [ADR 064](ADR/064-per-project-databases-with-shared-calibration.md)):
+
+- `agent` — the previous behavior: one shared `context.db` for every session and project.
+- `project` (**default**) — one database per trusted canonical project root, derived as `projects/<sha256(root)[0..32]>.db` next to the agent database. Untrusted projects and broad roots (home directory, filesystem root) fall back to the agent database.
+
+Both files receive the same schema and migrations. The agent database keeps only `token_calibration`, so a sample learned in one project is visible to every other project for the same provider/model/estimator; project databases keep the session index, project index, manifests, summary graph, memory/pin projections, embeddings, quality samples and artifact metadata. Project artifact object bytes move under `projects/artifacts/<project-digest>/` so garbage collection stays scoped to one project. A project database starts empty and is rebuilt from canonical JSONL and project files; the split never rewrites the previous shared database.
+
+Calibration and manifest writes are intentionally not one cross-database transaction: a failed calibration insert loses at most one sample, while manifest/usage consistency stays inside the project database. Manifest pruning detaches calibration rows only in `agent` scope; in `project` scope the project database's calibration table stays empty. `storage.databasePath` names the agent database; the derived `projects/` directory is the supported layout.
+
+Maintenance and diagnostics are per file: `/context storage` reports the active project database and, when split, the shared agent database; `ds4-context-storage inspect|compact|recover --database <path>` must be pointed at each file.
+
 The M19 non-Pi reference adapter owns a separate `ds4-runtime-session-v1` JSONL source selected by its host runtime. Its header binds runtime/session identity and the exact canonical project root; following records contain provenance-checked canonical messages. DS4 snapshots and capability diagnostics are disposable. `createReferenceHistory()` refuses overwrite, append uses a dedicated provenance-checked operation, files are mode `0600` where supported, and rebuild never edits this runtime-owned canonical file. Reference JSONL is not imported into Pi or `context.db`.
 
 M20 local KV state is entirely runtime-owned and volatile. Core returns only an in-memory eligibility fingerprint to the runtime port; it has no cache-handle field or serialization API. Prefixes, fingerprints, handles and provider outputs are absent from Pi/reference JSONL, Context Manifests, ranking artifacts and every SQLite table. Aggregate hit/miss/prefill counters live only on the adapter controller, and a restart safely resets them with the runtime cache. M20 adds no database migration.
@@ -83,7 +96,7 @@ Custom entries have empty lexical search text and never enter Pi context directl
 
 Schema v10 extends `context_manifests` and `token_calibration` with separate uncached-input, cache-read, and cache-write token columns. New calibration rows also carry the correlated manifest ID and explicit estimator version. Legacy pre-v10 samples migrate as `chars-v1` with their prior total stored as uncached input and zero cache fields; this preserves historical ratio behavior without inventing cache hits.
 
-Calibration rows are derived telemetry, isolated by exact provider/model and bounded to the latest configured window at read time. The runtime recomputes median/MAD outlier filtering deterministically; no learned model or mutable provider state is stored. Deleting the database loses calibration and cache history but never session content. Ephemeral sessions and configurations that disable manifest persistence keep only a bounded in-memory window.
+Calibration rows are derived telemetry, isolated by exact provider/model and bounded to the latest configured window at read time. The runtime recomputes median/MAD outlier filtering deterministically; no learned model or mutable provider state is stored. Deleting the database loses calibration and cache history but never session content. Ephemeral sessions and configurations that disable manifest persistence keep only a bounded in-memory window. With `storage.scope: "project"`, calibration lives in the agent database while the manifest that produced the sample lives in the project database; the sample therefore carries no `manifest_id` and the two writes are separate transactions.
 
 ## Context quality samples
 
@@ -107,7 +120,7 @@ The volatile state is cleared on lifecycle/model/branch/compaction boundaries an
 
 Schema v8 splits content objects from source references. `artifact_objects` is keyed by SHA-256 and stores the private file path, MIME, byte size, verification timestamps, and integrity status. `artifacts` is keyed by a deterministic source-specific ID and references session/entry/tool identity plus original/condensed token estimates and an optional derived privacy classification in `metadata_json`. Equal bytes across calls or sessions deduplicate to one object while retaining independent provenance.
 
-Objects live under `ds4-context/artifacts/<sha-prefix>/<sha256>` with private permissions and atomic writes. Pi's full JSONL tool result remains canonical; the object file is a rebuildable local cache. No artifact content is stored in Context Manifests. Search recomputes SHA-256 and returns only bounded, redacted, JSON-quoted literal-match windows for a current-branch reference. The runtime reapplies the stored artifact classification before returning excerpts to the active provider; prohibited remote searches return no content.
+Objects live under `ds4-context/artifacts/<sha-prefix>/<sha256>` with private permissions and atomic writes. With `storage.scope: "project"` the store root becomes `projects/artifacts/<project-digest>/...` so artifact bytes and their metadata share one boundary; the orphan garbage collector only sees references in the current database and must never delete another project's objects. Pi's full JSONL tool result remains canonical; the object file is a rebuildable local cache. No artifact content is stored in Context Manifests. Search recomputes SHA-256 and returns only bounded, redacted, JSON-quoted literal-match windows for a current-branch reference. The runtime reapplies the stored artifact classification before returning excerpts to the active provider; prohibited remote searches return no content.
 
 A full index rebuild replays all message entries, recreates missing qualifying objects, removes stale session references, and garbage-collects object rows/files with no references. Missing/corrupt states are reported by `/context health` without blocking Pi.
 
@@ -117,7 +130,7 @@ For persisted sessions, each `context` hook stores a metadata-only manifest cont
 
 `before_provider_request` updates the pending in-memory manifest with final-check/redaction counters but never the provider payload. The following finalized assistant response updates only the existing scalar usage columns (`actual_tokens`, `input_tokens`, `cache_read_tokens`, and `cache_write_tokens`) and adds at most one exact-model calibration sample. It does not read or rewrite `manifest_json`. Repository reads hydrate authoritative usage from those columns. Ephemeral, oversize-skipped, concurrently pruned, and otherwise uncorrelated manifests retain bounded calibration only in memory.
 
-Retention is bounded without a schema change: SQLite keeps the latest 128 manifests globally and at most 200 calibration samples for each provider/model/estimator profile. A manifest prune first detaches its small calibration row, then removes the large diagnostic JSON; calibration has its own per-profile retention. Save and prune are one transaction. Existing oversized stores are reduced incrementally by at most 32 rows and 8 MiB of serialized manifest payload per subsequent manifest write; one individually oversized oldest row may be removed to guarantee progress. There is no startup purge.
+Retention is bounded without a schema change: SQLite keeps the latest 128 manifests globally and at most 200 calibration samples for each provider/model/estimator profile. A manifest prune first detaches its small calibration row, then removes the large diagnostic JSON; calibration has its own per-profile retention. Save and prune are one transaction. In `project` scope the manifest transaction runs in the project database while the calibration sample is inserted separately into the agent database. Existing oversized stores are reduced incrementally by at most 32 rows and 8 MiB of serialized manifest payload per subsequent manifest write; one individually oversized oldest row may be removed to guarantee progress. There is no startup purge.
 
 New manifest persistence is byte-bounded. Payloads up to 256 KiB remain complete. Larger payloads preserve all `included` provenance and replace only the `excluded` inventory with a deterministic first/last sample of at most 256 details plus explicit `ds4-context-manifest-inventory-v1` counts, token/classification/kind rollups, and digests. The wrapper returned by `getStored()` declares `complete` or `excluded-rollup`; the live runtime manifest remains complete. A projected payload over 1 MiB is skipped without affecting the model request. Deleted pages become reusable by SQLite but do not promise an immediate reduction in filesystem size. Manifests and calibration remain disposable; Pi JSONL and project files are untouched.
 
@@ -152,3 +165,25 @@ A full rebuild does not blindly delete unchanged entries. It upserts all observe
 Session reconciliation is transactional. Memory/pin mutation replacement, checkpoint update, source exclusion and full materialization each occur under the shared write coordinator. Each manifest upsert and dual-bound incremental retention prune share one transaction; each scalar usage/calibration update and its independent per-profile prune do the same. Each quality upsert and bounded-retention prune also share one transaction; quality failures do not affect manifests or planning. Each changed project file is replaced transactionally with its snippets and FTS rows; embedding upserts and canonical-source pruning are transactional; artifact object/reference metadata and project deletion batches are atomic. A filesystem artifact write precedes its metadata transaction, so an interrupted metadata write may leave only an unreferenced content-addressed cache file; canonical JSONL remains sufficient for recovery. If manifest serialization, projection, retention, or SQLite writing fails, the complete current manifest remains in memory and the provider request is unchanged. Other artifact/project failures contribute no replacement/snippets; planner failures discard all synthetic evidence; Pi continues with its native context.
 
 After bounded busy-aware replay is exhausted, DS4 emits `database.write_lock_timeout` with only the coordinator operation name, attempt count, elapsed/configured waits, and SQLite primary code. The thrown error repeats the operation and categorical lock status but never includes SQL, bound values, provider content, or the raw SQLite message. Retry and rollback diagnostics follow the same metadata-only rule.
+
+## Growth measurements
+
+`tests/benchmarks/storage-scale.bench.ts` seeds session indexes of 5,000 / 50,000 / 200,000 entries and measures the paths a session actually pays. Run it on demand:
+
+```bash
+npx vitest bench tests/benchmarks/storage-scale.bench.ts
+```
+
+Measured means on this development machine (Node.js 26.5.1, `node:sqlite`, one session per database):
+
+| Path | 5k entries | 50k entries | 200k entries |
+| --- | ---: | ---: | ---: |
+| Exact identifier scan (`instr` over one session) | 0.97 ms | 12.5 ms | 52.7 ms |
+| Exact phrase scan | 0.98 ms | 12.8 ms | 53.2 ms |
+| FTS retrieval, common token | 0.14 ms | 2.3 ms | 9.0 ms |
+| FTS retrieval, rare token | 0.06 ms | 0.15 ms | 0.94 ms |
+| Per-session stats (`COUNT`/`SUM` for one session) | 0.33 ms | 4.8 ms | 21.8 ms |
+| Storage diagnostics | 0.10 ms | 0.10 ms | 0.10 ms |
+| Append-only unchanged re-check (1,000 entries) | 1.4 ms | 2.9 ms | 7.2 ms |
+
+Growth is **real but bounded**: exact identifier and phrase scans are literal `instr()` scans over the session's rows and grow roughly linearly, passing the 50 ms typical-operation target only around 200k indexed entries in a single session. FTS retrieval and per-session aggregate SQL grow sublinearly and stay in single-digit milliseconds; bounded manifest/calibration diagnostics are flat. The dominant cost tracks a single session's size, not the file size, so `storage.scope: "project"` bounds physical growth, lock scope and reset per project but does not by itself change this per-session scan profile. Long single sessions near or above the 200k-entry range are the case where exact-identifier retrieval latency becomes measurable.

package/docs/STORAGE_MAINTENANCE.md

@@ -1,6 +1,6 @@
 # Offline SQLite Storage Maintenance
 
-DS4 keeps `context.db` as disposable derived state, while Pi session JSONL and live project files remain canonical. Normal runtime retention stops unbounded manifest growth and makes deleted pages reusable. It does not promise that an existing high-water SQLite file shrinks physically.
+DS4 keeps `context.db` as disposable derived state, while Pi session JSONL and live project files remain canonical. Normal runtime retention stops unbounded manifest growth and makes deleted pages reusable. It does not promise that an existing high-water SQLite file shrinks physically. With the default `storage.scope: "project"` each trusted project has its own database under `ds4-context/projects/` and the agent database keeps shared token calibration: run the maintenance commands per file, not only on the agent database (see [`STORAGE.md`](STORAGE.md)).
 
 Physical compaction is therefore an explicit offline operation. It is never model-callable, never runs at startup, and never edits Pi JSONL or project files.
 

package/docs/releases/0.3.10.md

@@ -26,4 +26,4 @@ No new defaults are enabled: `chars-v1`, disabled `autoTune`, and disabled DS4 n
 
 ## Validation and publication
 
-On Node 26.5.1, the coordinated release passed a clean `npm ci`, `npm run check` (97 Vitest files, 607 tests, TypeScript builds and root typecheck), deterministic `npm run quality:compare`, the `npm run schema:context-persistence` size bound, and `npm run pack:check` in a clean consumer after synchronizing both exported runtime version constants. All typecheck/tests were also rerun after that correction. Dry-run tarball review contained 243 core files, 7 reference-adapter files, and 97 extension files; none included untracked local state. Registry publication and exact-artifact verification are still pending.
+On Node 26.5.1, the coordinated release passed a clean `npm ci`, `npm run check` (97 Vitest files, 607 tests, TypeScript builds and root typecheck), deterministic `npm run quality:compare`, the `npm run schema:context-persistence` size bound, and `npm run pack:check` in a clean consumer after synchronizing both exported runtime version constants. All typecheck/tests were also rerun after that correction. Dry-run tarball review contained 243 core files, 7 reference-adapter files, and 97 extension files; none included untracked local state. All three packages were then published to npm at 0.3.10 in dependency order. After registry propagation, `npm run registry:check -- 0.3.10` verified all three exact-version artifacts in a fresh consumer.

package/docs/releases/0.4.0.md

@@ -0,0 +1,86 @@
+# Release 0.4.0 — Per-project databases with shared token calibration
+
+**Coordinated packages:** `ds4-context-core`, `ds4-context-reference-adapter`, and `ds4-context-engine` 0.4.0.
+**Implementation commit:** `2c2e248`.
+**Decision record:** [ADR 064](../ADR/064-per-project-databases-with-shared-calibration.md).
+
+## Summary
+
+Adds the opt-out `storage.scope: "agent" | "project"` setting and defaults it to
+`project`. Each trusted canonical project root now gets its own SQLite
+projection next to the configured agent database, while token calibration stays
+in the agent database and remains shared across projects. `storage.scope:
+"agent"` restores the previous single-file layout. The reference adapter and Pi
+extension continue to depend exactly on the matching core version.
+
+## Changes
+
+- `storage.scope: "project"` (new default) derives one database per trusted
+  canonical project root at `projects/<sha256(root)[0..32]>.db`, next to
+  `storage.databasePath`. Untrusted projects, broad roots (home directory,
+  filesystem root) and any resolution failure fall back to the agent database.
+- Both files receive the same schema and the same migrations; migrations 1–15
+  are untouched and there is no schema fork. The project database holds the
+  session index, project index, context manifests, summary graph, memory/pin
+  projections, embeddings, quality samples and artifact metadata; the agent
+  database holds `token_calibration`.
+- With the split, a calibration sample is written to the agent database with
+  `manifest_id = NULL`, exactly once per manifest. The two writes are
+  intentionally not one cross-database transaction: a lost calibration sample is
+  harmless, a lost manifest/usage consistency is not. In `agent` scope the
+  previous single transaction is unchanged.
+- Artifact object bytes move to `projects/artifacts/<project-digest>/` under
+  project scope so the orphan garbage collector, which only sees references in
+  the current database, can never delete another project's objects.
+- `/context storage` and `/context diagnostics` report the active project
+  database and, when split, the shared agent database. Storage maintenance
+  stays per file: `ds4-context-storage inspect|compact|recover --database <path>`
+  must be pointed at each database.
+- The previous shared database is left untouched. With the new default, existing
+  users cold-start per-project indexes while existing calibration remains
+  available in the agent database; project indexes rebuild from canonical Pi
+  JSONL, project files and memory/pin `CustomEntry` records.
+
+## Breaking behavior
+
+The default storage layout changes. A pre-existing
+`~/.pi/agent/ds4-context/context.db` becomes the agent database and is no longer
+the active projection for new sessions. `storage.scope: "agent"` restores the
+old single-database behavior. Rows are not migrated between files; the project
+databases are derived and rebuildable. Decision D1 of the storage plan ("no
+per-session or per-project databases") is superseded by ADR 064 and the plan
+keeps the original text with an explicit amendment pointer.
+
+## Measured scope and limitations
+
+`tests/benchmarks/storage-scale.bench.ts` seeds one session with 5,000 /
+50,000 / 200,000 entries and measures the paths a session pays. Measured means
+on the development machine (Node.js 26.5.1, `node:sqlite`): exact identifier
+scan 0.97 / 12.5 / 52.7 ms, exact phrase scan 0.98 / 12.8 / 53.2 ms, FTS
+common token 0.14 / 2.3 / 9.0 ms, per-session stats 0.33 / 4.8 / 21.8 ms,
+storage diagnostics flat at 0.10 ms, unchanged re-check 1.4 / 2.9 / 7.2 ms.
+Growth is real but bounded and tracks single-session size, not file size:
+project scope bounds physical growth, lock scope and reset per project, but does
+not by itself change the per-session exact-scan profile. See
+[storage growth measurements](../STORAGE.md#growth-measurements).
+
+These are local measurements on one host, not portable guarantees. No provider
+calls are involved in this release procedure.
+
+## Compatibility
+
+No provider-facing default changes: `chars-v1` remains the default estimator,
+`modelAwareness.autoTune` and DS4 native continuation stay disabled. No SQLite
+migration, no change to canonical Pi JSONL, privacy consent, native continuation
+or the portable runtime-adapter contract. Restart Pi after upgrading so the new
+compiled core and session configuration are loaded.
+
+## Validation and publication
+
+On Node 26.5.1, the coordinated release passed `npm run check` (99 Vitest files,
+615 tests, TypeScript builds and root typecheck), deterministic
+`npm run quality:compare`, the `npm run schema:context-persistence` size bound,
+and `npm run pack:check` in a clean consumer after synchronizing both exported
+runtime version constants. `git diff --check` was clean and dry-run tarball
+review contained 247 core files, 7 reference-adapter files and 98 extension
+files, none including untracked local state.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.10",
+  "version": "0.4.0",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -52,6 +52,7 @@
     "quality:compare": "node scripts/compare-context-quality.mjs",
     "schema:context-persistence": "node scripts/measure-context-persistence-schema.mjs",
     "latency:check": "npm run build:core && node scripts/compare-disabled-planning-latency.mjs",
+    "storage:scale": "npm run build:core && vitest bench tests/benchmarks/storage-scale.bench.ts",
     "pack:check": "node scripts/verify-packages.mjs",
     "registry:check": "node scripts/verify-registry-packages.mjs",
     "prepare": "npm run build:core && npm run build:adapters"
@@ -62,7 +63,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.10",
+    "ds4-context-core": "0.4.0",
     "js-tiktoken": "1.0.21"
   },
   "peerDependencies": {

package/src/extension/commands.ts

@@ -835,13 +835,20 @@ function formatAdapter(diagnostics: RuntimeDiagnostics): string {
   ].join("\n");
 }
 
-function formatStorage(storage: StorageDiagnostics, databasePath?: string): string {
+function formatStorage(
+  storage: StorageDiagnostics,
+  databasePath?: string,
+  agentDatabasePath?: string,
+): string {
+  const agentLine = agentDatabasePath && agentDatabasePath !== databasePath
+    ? [`Agent database (shared):    ${agentDatabasePath}`] : [];
   if (storage.status === "unavailable") {
     return [
       "DS4 Storage",
       "",
       "Status:                     unavailable",
       `Database:                   ${databasePath ?? "unavailable"}`,
+      ...agentLine,
       "Pi fallback remains active; no storage mutation was attempted.",
     ].join("\n");
   }
@@ -850,6 +857,7 @@ function formatStorage(storage: StorageDiagnostics, databasePath?: string): stri
     "",
     `Status:                     ${storage.status}`,
     `Database:                   ${databasePath ?? "unavailable"}`,
+    ...agentLine,
     `Schema / journal:           ${storage.schemaVersion ?? "n/a"} / ${storage.journalMode ?? "n/a"}`,
     `Database / WAL / SHM:       ${bytes(storage.databaseBytes)} / ${bytes(storage.walBytes)} / ${bytes(storage.shmBytes)}`,
     `Allocated / reusable:       ${bytes(storage.allocatedBytes)} / ${bytes(storage.reusableBytes)}`,
@@ -1234,7 +1242,7 @@ export function registerContextCommand(pi: ExtensionAPI, runtime: Ds4ContextRunt
           const storage = runtime.storageDiagnostics();
           present(
             ctx,
-            formatStorage(storage, diagnostics.databasePath),
+            formatStorage(storage, diagnostics.databasePath, diagnostics.agentDatabasePath),
             storage.status === "ok" ? "info" : "warning",
           );
           return;

package/src/extension/runtime.ts

@@ -1,7 +1,7 @@
 import { randomUUID } from "node:crypto";
 import { existsSync, mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs";
 import { homedir } from "node:os";
-import { dirname, join, parse, resolve } from "node:path";
+import { basename, dirname, join, parse, resolve } from "node:path";
 import type {
   Api,
   AssistantMessage,
@@ -46,11 +46,12 @@ export type {
 import {
   loadConfig,
   resolveDatabasePath,
+  resolveProjectDatabasePath as deriveProjectDatabasePath,
   resolveRankingModelPath,
   validateConfigFile,
   type LoadedConfig,
 } from "ds4-context-core/config/config-loader";
-import { CONFIG_SCHEMA_VERSION, createDefaultConfig, type Ds4ContextConfig } from "ds4-context-core/config/config";
+import { CONFIG_SCHEMA_VERSION, createDefaultConfig, type Ds4ContextConfig, type StorageScope } from "ds4-context-core/config/config";
 import {
   applyConfigValue,
   findConfigField,
@@ -387,6 +388,8 @@ export interface RuntimeDiagnostics {
   session?: PiSessionSnapshot;
   model?: { provider: string; id: string };
   databasePath?: string;
+  /** Present only when storage.scope splits calibration from project state. */
+  agentDatabasePath?: string;
   databaseSchemaVersion?: number;
   indexed?: SessionIndexStats;
   observation?: ContextObservation;
@@ -426,8 +429,10 @@ export class Ds4ContextRuntime {
   private loadedConfig?: LoadedConfig;
   private session?: PiSessionSnapshot;
   private database?: ContextDatabase;
+  private agentDatabase?: ContextDatabase;
   private indexer?: PiSessionIndexer;
   private databasePath?: string;
+  private agentDatabasePath?: string;
   private observation?: ContextObservation;
   private lastManifest?: ContextManifest;
   private lastPersistedInventory?: PersistedManifestInventory;
@@ -579,17 +584,33 @@ export class Ds4ContextRuntime {
         return;
       }
 
-      this.databasePath = resolveDatabasePath(
+      const agentDatabasePath = resolveDatabasePath(
         this.config.storage.databasePath,
         this.dependencies.agentDir,
         this.dependencies.homeDir,
       );
+      this.agentDatabasePath = agentDatabasePath;
+      const projectDatabasePath = this.resolveProjectDatabasePath(
+        this.config.storage.scope,
+        ctx,
+        agentDatabasePath,
+      );
+      this.databasePath = projectDatabasePath ?? agentDatabasePath;
+      if (projectDatabasePath) {
+        this.agentDatabase = ContextDatabase.open(agentDatabasePath, {
+          logger: this.logger,
+          now: this.now(),
+          busyTimeoutMs: this.config.storage.busyTimeoutMs,
+          writeRetryTimeoutMs: this.config.storage.writeRetryTimeoutMs,
+        });
+      }
       this.database = ContextDatabase.open(this.databasePath, {
         logger: this.logger,
         now: this.now(),
         busyTimeoutMs: this.config.storage.busyTimeoutMs,
         writeRetryTimeoutMs: this.config.storage.writeRetryTimeoutMs,
       });
+      this.agentDatabase ??= this.database;
       this.initializeRanking();
 
       this.indexer = new PiSessionIndexer(this.database.sessionIndex, {
@@ -765,10 +786,25 @@ export class Ds4ContextRuntime {
       ? O200K_ESTIMATOR : CHARS_ESTIMATOR;
   }
 
+  private resolveProjectDatabasePath(
+    scope: StorageScope,
+    ctx: ExtensionContext,
+    agentDatabasePath: string,
+  ): string | undefined {
+    if (scope !== "project") return undefined;
+    // Untrusted projects ignore project configuration; they also fall back to
+    // the shared agent database instead of creating a new physical boundary.
+    if (!ctx.isProjectTrusted()) return undefined;
+    const projectRoot = canonicalProjectPath(ctx.cwd);
+    if (isBroadProjectRoot(projectRoot, this.dependencies.homeDir ?? homedir())) return undefined;
+    const projectDatabasePath = deriveProjectDatabasePath(agentDatabasePath, projectRoot);
+    return projectDatabasePath === agentDatabasePath ? undefined : projectDatabasePath;
+  }
+
   private calibrationSamples(model: ModelDescriptor): TokenCalibrationSample[] {
     const version = this.estimatorForModel(model).version;
     if (this.database && this.session?.sessionFile && this.config.diagnostics.storeContextManifest) {
-      return this.database.manifests.listCalibrationSamples(
+      return (this.agentDatabase ?? this.database).calibrations.list(
         model.provider,
         model.id,
         this.config.modelAwareness.calibrationWindow,
@@ -1884,13 +1920,39 @@ export class Ds4ContextRuntime {
     const createdAt = this.now();
     try {
       if (manifestPersisted && this.database) {
-        const updated = this.database.manifests.recordProviderUsage(
+        const estimatorVersion = manifest?.modelAwareness?.calibration.estimator ?? "chars-v1";
+        // With storage.scope=project the manifest lives in the project database
+        // while calibration stays shared in the agent database. The two writes
+        // are intentionally not one transaction: losing one sample is harmless,
+        // losing manifest/usage consistency is not.
+        const calibrationDatabase = this.agentDatabase !== this.database
+          ? this.agentDatabase : undefined;
+        const updated = this.database.manifests.recordProviderUsageOutcome(
           manifestId,
           providerUsage,
           createdAt,
-          manifest?.modelAwareness?.calibration.estimator ?? "chars-v1",
+          estimatorVersion,
+          { writeCalibration: calibrationDatabase === undefined },
         );
-        if (!updated && manifest?.estimatedInputTokens) {
+        if (updated.outcome === "recorded" && calibrationDatabase) {
+          const source = this.database.manifests.calibrationSource(manifestId);
+          const recorded = source
+            ? calibrationDatabase.calibrations.record({
+              provider: source.provider,
+              model: source.model,
+              estimatedTokens: source.estimatedTokens,
+              actualInputTokens: providerUsage.totalInputTokens,
+              inputTokens: providerUsage.inputTokens,
+              cacheReadTokens: providerUsage.cacheReadTokens,
+              cacheWriteTokens: providerUsage.cacheWriteTokens,
+              createdAt,
+              estimatorVersion,
+            })
+            : false;
+          if (!recorded && manifest?.estimatedInputTokens) {
+            this.rememberVolatileCalibration(manifest, providerUsage, createdAt);
+          }
+        } else if (!updated.manifest && manifest?.estimatedInputTokens) {
           this.rememberVolatileCalibration(manifest, providerUsage, createdAt);
         }
       } else if (manifest?.estimatedInputTokens) {
@@ -2707,10 +2769,13 @@ export class Ds4ContextRuntime {
       return;
     }
     try {
-      const store = new FileArtifactStore(
-        join(this.dependencies.agentDir, "ds4-context", "artifacts"),
-        this.now,
-      );
+      // Artifact bytes and their metadata must share one boundary: the orphan
+      // garbage collector only sees references in the current database, so a
+      // shared store would let one project delete another project's objects.
+      const storeRoot = this.agentDatabase !== this.database && this.databasePath
+        ? join(dirname(this.databasePath), "artifacts", basename(this.databasePath, ".db"))
+        : join(this.dependencies.agentDir, "ds4-context", "artifacts");
+      const store = new FileArtifactStore(storeRoot, this.now);
       this.artifactManager = new ArtifactManager(
         store,
         this.database.artifacts,
@@ -3295,6 +3360,8 @@ export class Ds4ContextRuntime {
       session: currentSession,
       ...(ctx.model ? { model: { provider: ctx.model.provider, id: ctx.model.id } } : {}),
       ...(this.databasePath ? { databasePath: this.databasePath } : {}),
+      ...(this.agentDatabasePath && this.agentDatabasePath !== this.databasePath
+        ? { agentDatabasePath: this.agentDatabasePath } : {}),
       ...(this.database ? { databaseSchemaVersion: this.database.schemaVersion } : {}),
       ...(indexed ? { indexed } : {}),
       ...(this.observation ? { observation: this.observation } : {}),
@@ -3332,7 +3399,9 @@ export class Ds4ContextRuntime {
   }
 
   storageDiagnostics(): StorageDiagnostics {
-    return this.database?.storageDiagnostics(this.session?.projectPath)
+    const calibrationDatabase = this.agentDatabase && this.agentDatabase !== this.database
+      ? this.agentDatabase : undefined;
+    return this.database?.storageDiagnostics(this.session?.projectPath, calibrationDatabase)
       ?? unavailableStorageDiagnostics();
   }
 
@@ -3459,17 +3528,25 @@ export class Ds4ContextRuntime {
     try {
       this.database?.close();
     } finally {
-      this.compaction = undefined;
-      this.retrievalEngine = undefined;
-      this.projectKnowledge = undefined;
-      this.projectRefreshPending = false;
-      this.memoryManager = undefined;
-      this.projectMemorySynchronizer = undefined;
-      this.lastCrossSessionMemory = disabledCrossSessionMemoryDiagnostics();
-      this.lastMemoryMutationSignature = undefined;
-      this.artifactManager = undefined;
-      this.indexer = undefined;
-      this.database = undefined;
+      try {
+        if (this.agentDatabase && this.agentDatabase !== this.database) {
+          this.agentDatabase.close();
+        }
+      } finally {
+        this.agentDatabase = undefined;
+        this.agentDatabasePath = undefined;
+        this.compaction = undefined;
+        this.retrievalEngine = undefined;
+        this.projectKnowledge = undefined;
+        this.projectRefreshPending = false;
+        this.memoryManager = undefined;
+        this.projectMemorySynchronizer = undefined;
+        this.lastCrossSessionMemory = disabledCrossSessionMemoryDiagnostics();
+        this.lastMemoryMutationSignature = undefined;
+        this.artifactManager = undefined;
+        this.indexer = undefined;
+        this.database = undefined;
+      }
     }
   }
 

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.3.10";
+export const EXTENSION_VERSION = "0.4.0";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";