ds4-context-engine 0.1.2 → 0.2.0-beta.2

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** M0–M13 are implemented. The Pi adapter and standalone `ds4-context-core` package are version `0.1.2`; the adapter targets Pi `0.84.3`.
+> **Project status:** M0–M20 are implemented on `main`. Prerelease `0.2.0-beta.2` includes M14–M20: quality measurement, structural/hybrid retrieval, cross-session project memory, learned-ranking shadow evaluation, the runtime adapter kit, and opt-in local KV reuse. Stable `0.1.2` remains available; all lines target Pi `0.84.3`.
 
 ## Why DS4
 
@@ -27,13 +27,19 @@ It provides:
 - deterministic token budgeting with soft and hard input limits;
 - preservation of the current request, recent turns and atomic tool call/result groups;
 - exact and FTS5 historical retrieval with source provenance;
-- trust-gated project indexing, Git-aware invalidation and bounded source snippets;
+- opt-in hybrid semantic retrieval with a deterministic local embedding and lexical fallback;
+- trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
 - hierarchical, validated, non-destructive compaction summaries;
 - persistent pins and append-only durable memory stored canonically in Pi JSONL;
+- opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
 - content-addressed storage and bounded references for large tool results;
 - privacy classifications, secret redaction and provider-specific allow rules;
 - model-specific calibration and adaptive context allocation;
 - optional verified continuation for eligible OpenAI Responses profiles;
+- opt-in metadata-only context-quality metrics and deterministic replay comparisons;
+- checksummed metadata-only learned ranking with shadow mode, canonical classified feedback and static fallback;
+- a versioned runtime adapter contract, reusable conformance kit and non-Pi callback/JSONL reference adapter;
+- opt-in exact-prefix local KV reuse for capable local runtime adapters, with volatile handles and full-replay fallback;
 - an inspectable Context Manifest explaining included and excluded material;
 - fail-open recovery to Pi's native context path for operational failures.
 
@@ -74,12 +80,20 @@ pi -e git:github.com/Alucard24/ds4-context-engine
 
 ### From npm
 
-Install the public npm package with:
+Install the latest stable public npm package with:
 
 ```bash
 pi install npm:ds4-context-engine
 ```
 
+Install the opt-in 0.2 prerelease from the `beta` dist-tag with:
+
+```bash
+pi install npm:ds4-context-engine@beta
+```
+
+All prerelease packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) use the same exact version.
+
 ### Local checkout
 
 ```bash
@@ -106,6 +120,7 @@ After loading the extension, inspect its state:
 
 ```text
 /context status
+/context adapter
 /context tokens
 /context health
 ```
@@ -149,6 +164,7 @@ Project configuration and project source indexing are disabled when Pi reports t
 | Command | Purpose |
 | --- | --- |
 | `/context` or `/context status` | Runtime, session, planner and subsystem status |
+| `/context adapter` | Runtime contract and per-capability negotiation diagnostics |
 | `/context tokens` | Token budget and active-context composition |
 | `/context manifest` | Latest Context Manifest |
 | `/context explain` | Human-readable planning explanation |
@@ -159,6 +175,8 @@ Project configuration and project source indexing are disabled when Pi reports t
 | `/context project` | Project index and retrieval status |
 | `/context privacy` | Classification and provider-policy status |
 | `/context model` | Active model profile and calibration |
+| `/context quality` | Metadata-only context-quality scores and sample counts |
+| `/context ranking` | Learned model, promotion gate, aggregate shadow comparison and feedback counts |
 | `/context continuation` | Native continuation decisions and counters |
 | `/context artifacts` | Artifact storage and integrity status |
 | `/context compaction` | Last compaction status |
@@ -179,10 +197,22 @@ Project configuration and project source indexing are disabled when Pi reports t
 /context memory supersede MEMORY_ID [--source ID,ID] <new claim>
 /context memory invalidate MEMORY_ID [reason]
 /context memory expire MEMORY_ID [reason]
+/context memory sources
+/context memory exclude SESSION_ID [reason]
+/context memory include SESSION_ID
 ```
 
 Valid privacy classifications are `normal`, `internal`, `sensitive` and `local-only`.
 
+Learned-ranking feedback and local training are explicit:
+
+```text
+/context ranking feedback useful|irrelevant CANDIDATE_ID [--classification LEVEL]
+/context ranking train
+```
+
+See [`docs/LEARNED_RANKING.md`](docs/LEARNED_RANKING.md).
+
 ## Configuration reference
 
 The following example shows the main configuration groups. Omitted values use the defaults in [`packages/core/src/config/config.ts`](packages/core/src/config/config.ts).
@@ -208,7 +238,19 @@ The following example shows the main configuration groups. Omitted values use th
     "exact": true,
     "fts": true,
     "semantic": false,
-    "maxResults": 12
+    "maxResults": 12,
+    "embedding": {
+      "mode": "local",
+      "provider": "ds4-local",
+      "model": "feature-hash-v1",
+      "dimensions": 256,
+      "remoteProfiles": [],
+      "maxSources": 50000,
+      "candidatePool": 80,
+      "batchSize": 64,
+      "queryCacheSize": 64,
+      "timeoutMs": 2000
+    }
   },
   "project": {
     "enabled": true,
@@ -221,6 +263,8 @@ The following example shows the main configuration groups. Omitted values use th
   },
   "memory": {
     "enabled": true,
+    "crossSession": false,
+    "maxProjectSessions": 250,
     "maxPinChars": 4000,
     "maxClaimChars": 2000,
     "maxResults": 12
@@ -271,13 +315,27 @@ The following example shows the main configuration groups. Omitted values use th
     "maxStateAgeMs": 1800000,
     "retryManagedReplay": true
   },
+  "quality": {
+    "enabled": false,
+    "maxSamples": 1000
+  },
+  "ranking": {
+    "mode": "off",
+    "modelPath": "ds4-context/ranking-model.json",
+    "minimumTrainingSamples": 20,
+    "maxTrainingSamples": 10000,
+    "maxLatencyMs": 10
+  },
   "diagnostics": {
     "storeContextManifest": true,
     "storeFullRenderedContext": false,
     "logLevel": "info"
   },
   "storage": {
-    "databasePath": "ds4-context/context.db"
+    "databasePath": "ds4-context/context.db",
+    "busyTimeoutMs": 5000,
+    "writeRetryTimeoutMs": 30000,
+    "projectIndexLeaseMs": 120000
   }
 }
 ```
@@ -302,7 +360,9 @@ Native continuation is also disabled by default. Enabling it requires both expli
 
 Eligible OpenAI Responses requests then set `store: true`. Review the provider's retention policy before enabling this option. DS4 keeps response handles only in volatile memory, verifies exact managed prefixes before reuse and retries once with a full managed replay when recognized continuation state is stale.
 
-See [`docs/PRIVACY.md`](docs/PRIVACY.md) and [`docs/NATIVE_CONTINUATION.md`](docs/NATIVE_CONTINUATION.md).
+Local KV reuse is separately disabled by default through `localKvReuse.enabled`. It also requires a local runtime adapter with a versioned `local-kv-reuse` capability and a volatile runtime port. Pi exposes no such handles and remains unsupported even if configuration is enabled.
+
+See [`docs/PRIVACY.md`](docs/PRIVACY.md), [`docs/NATIVE_CONTINUATION.md`](docs/NATIVE_CONTINUATION.md), and [`docs/LOCAL_KV_REUSE.md`](docs/LOCAL_KV_REUSE.md).
 
 ## Storage and recovery
 
@@ -311,10 +371,11 @@ By default, derived state is stored below Pi's agent directory:
 ```text
 ~/.pi/agent/ds4-context/
 ├── context.db
+├── ranking-model.json
 └── artifacts/
 ```
 
-The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
+The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. The optional checksummed learned-ranking model is also derived local state; its classified metadata-only labels remain canonical Pi custom entries. All Pi sessions share this WAL database: writes use bounded busy-aware transaction replay, and a renewable project lease prevents multiple Pi processes from indexing the same project concurrently. `busyTimeoutMs` controls each SQLite lock wait, while `writeRetryTimeoutMs` bounds the total replay window. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
 
 To validate or rebuild derived state:
 
@@ -323,32 +384,36 @@ To validate or rebuild derived state:
 /context rebuild-index
 ```
 
-Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated.
+Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated. When `memory.crossSession` is enabled for a trusted project, DS4 discovers bounded sibling Pi JSONL files by exact canonical header identity, incrementally replays their explicit project mutations, and excludes missing or unverifiable sources.
 
 ## Development
 
 ```bash
 npm ci
 npm run build:core
+npm run build:adapters
 npm run typecheck
 npm test
 npm run check
+npm run quality:compare
 npm run pack:check
 npm pack --dry-run
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 ```
 
-The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check also builds both tarballs, installs them in a clean temporary consumer and starts the packaged extension with isolated Pi RPC state.
+The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, local-KV eligibility/replay, runtime-adapter conformance, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check builds all three tarballs, installs them in a clean temporary consumer, reruns compiled reference-adapter conformance and starts the packaged Pi extension with isolated RPC state.
 
 ### Portable core
 
-`ds4-context-core` is a compiled ESM package with no Pi dependency. It owns runtime-neutral policy, storage and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root `ds4-context-engine` package is the Pi adapter and depends one-way on the core workspace.
+`ds4-context-core` is a compiled ESM package with no runtime SDK dependency. It owns runtime-neutral policy, storage, adapter contracts and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root `ds4-context-engine` package is the Pi adapter. `ds4-context-reference-adapter` is a separately compiled non-Pi callback/JSONL implementation. Both depend one-way and exactly on matching core.
 
 ### Repository layout
 
 ```text
-packages/core/src   portable policy, planning, compaction, retrieval and storage
-src/pi-adapter      Pi JSONL projection, summary completion and provider integration
+packages/core/src                 portable policy, adapter kit, planning, retrieval and storage
+packages/reference-adapter/src    non-Pi callback/JSONL reference runtime boundary
+src/pi-adapter                    Pi JSONL projection, summary completion and provider integration
 src/extension       Pi hooks, commands and fail-open orchestration
 tests               core contract, unit, integration, golden and benchmark coverage
 scripts             package and release-readiness checks
@@ -359,10 +424,13 @@ scripts             package and release-readiness checks
 
 - [Architecture](docs/ARCHITECTURE.md)
 - [Context planner](docs/CONTEXT_PLANNER.md)
+- [Context quality](docs/CONTEXT_QUALITY.md)
+- [Learned ranking](docs/LEARNED_RANKING.md)
 - [Context Manifest](docs/CONTEXT_MANIFEST.md)
 - [Compaction](docs/COMPACTION.md)
 - [Summary graph](docs/SUMMARY_GRAPH.md)
 - [Historical retrieval](docs/RETRIEVAL.md)
+- [Hybrid semantic retrieval](docs/HYBRID_RETRIEVAL.md)
 - [Project knowledge](docs/PROJECT_KNOWLEDGE.md)
 - [Artifacts](docs/ARTIFACTS.md)
 - [Memory and pins](docs/MEMORY_AND_PINS.md)
@@ -370,6 +438,8 @@ scripts             package and release-readiness checks
 - [Model awareness](docs/MODEL_AWARENESS.md)
 - [Native continuation](docs/NATIVE_CONTINUATION.md)
 - [Portable core](docs/PORTABLE_CORE.md)
+- [Runtime adapter kit](docs/RUNTIME_ADAPTER_KIT.md)
+- [Local KV reuse](docs/LOCAL_KV_REUSE.md)
 - [Storage](docs/STORAGE.md)
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
@@ -378,9 +448,9 @@ scripts             package and release-readiness checks
 
 ## Roadmap
 
-The original M0–M13 roadmap is complete. `ds4-context-core` now contains the compiled Pi-independent implementation, while runtime-specific behavior remains in the Pi adapter.
+The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation, M19's runtime adapter/conformance kit, and M20 opt-in local KV eligibility/replay are implemented on `main`. Learned active ranking remains promotion-gated, Pi reports local KV as unsupported, and static ranking/native completion stay authoritative on every failure.
 
-The planned [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) covers context-quality metrics, richer symbol indexing, hybrid semantic retrieval, cross-session project memory, optional learned ranking, a runtime adapter kit with one reference adapter, and optional local KV reuse. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) now proceeds to release-candidate hardening. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ARCHITECTURE.md

@@ -9,7 +9,12 @@ Pi session_start
   -> validate Pi JSONL v3 header
   -> full index or checkpointed append sync
   -> replay versioned memory/pin custom-entry mutations into transactional projections
+  -> when opted in and trusted, checkpoint/replay explicit project mutations from exact-identity sibling Pi sessions
   -> if trusted, canonicalize project root and incrementally index bounded text files
+  -> parse TypeScript/JavaScript/Python/Go declaration boundaries through the runtime-neutral parser interface
+  -> fall back deterministically to bounded text windows when adapters, syntax, or language support are unavailable
+  -> persist only rebuildable symbol/signature/parent/import/reference projections
+  -> when semantic retrieval is opted in, refresh source-hash/model-keyed vectors through the runtime embedding port
   -> snapshot Git root/branch/HEAD/dirty paths
 
 Pi context hook
@@ -30,16 +35,22 @@ Pi context hook
   -> select a contiguous, model-adaptive recent tail
   -> derive current-task identifiers, files, errors, phrases and keywords
   -> query exact matches and FTS5 over canonical indexed entries
+  -> optionally add bounded cosine-ranked history candidates and deterministic lexical/vector rank fusion
   -> reject active-context duplicates and all alternate-branch candidates
   -> rank, deduplicate, quote and budget historical evidence groups
-  -> query exact path/symbol/phrase and project FTS5 candidates
-  -> live-validate candidate SHA-256 and reindex changed files
+  -> query exact literal path and qualified/simple declaration indexes ahead of phrase and project FTS5 candidates
+  -> optionally add bounded project vectors while retaining exact path/symbol priority
+  -> live-validate candidate SHA-256 and reindex only changed-file projections
   -> rank, overlap-deduplicate, quote and budget project source groups
   -> omit prohibited history/project/pin/memory supplements with metadata-only privacy reasons
   -> fit active Pi summaries in the remaining budget
   -> validate hard limit, current request, and tool call/results
   -> return selected messages or fail open to the already privacy-sanitized native context
   -> persist metadata-only Context Manifest, privacy counters, and prompt hash
+  -> when opted in, queue the completed metadata-only manifest for deferred quality measurement
+
+Pi agent_settled / shutdown
+  -> materialize and persist bounded content-free quality counts without changing the plan
 
 before_provider_request
   -> recheck provider-specific serialized system/messages/tools/content
@@ -55,6 +66,15 @@ optional OpenAI Responses provider wrapper
   -> retry a rejected stale handle once through the complete managed replay before exposing stream events
   -> record metadata-only mode/item counts/retry/invalidation diagnostics; never record the provider handle
 
+optional local runtime KV port (non-Pi adapters)
+  -> require opt-in configuration plus a negotiated versioned local-KV capability
+  -> apply current privacy policy before runtime-specific prefix/options extraction
+  -> hash exact prefix bytes, provider/model/revision, options, privacy policy, runtime revision and capability version
+  -> let the runtime port map only the fingerprint to its volatile native handle
+  -> replay the complete sanitized payload after miss, stale rejection, unavailable state or runtime restart
+  -> report aggregate hits/misses/saved-prefill/replay latency separately from context occupancy
+  -> never place handles, prefixes or payloads in canonical history, manifests, SQLite or diagnostics
+
 assistant message_end
   -> attach uncached input plus cache read/write usage to the pending manifest
   -> append one exact provider/model calibration sample
@@ -109,10 +129,10 @@ session_tree / shutdown
   -> immutable graph nodes, ordered edges, roots, levels and current-branch active path
 
 /context retrieved
-  -> query terms, candidate counts, branch blocks, budget decisions and injected excerpts
+  -> query terms, lexical/vector/fused counts, embedding profile/freshness/fallback, branch blocks, budgets and excerpts
 
 /context project
-  -> trust, Git revision, file/snippet/stale counts, retrieval decisions and local excerpts
+  -> trust, Git revision, file/snippet/stale/vector counts, retrieval decisions and local excerpts
 
 /context pins | pin | unpin
   -> inspect or append immutable session/branch/project pin mutations
@@ -141,16 +161,18 @@ session_tree / shutdown
 Dependency direction is one-way:
 
 ```text
-Pi native types and lifecycle
-        ↓
-ds4-context-engine (src/pi-adapter + src/extension)
-        ↓
-ds4-context-core (packages/core)
+Pi native types and lifecycle                 callback/JSONL runtime
+        ↓                                              ↓
+ds4-context-engine                        ds4-context-reference-adapter
+        └──────────────────────┬───────────────────────┘
+                               ↓
+                  ds4-context-core (packages/core)
 ```
 
 `ds4-context-core` is compiled ESM and has no dependency on Pi. Its workspace contains:
 
-- `packages/core/src/core`: portable model profiles, robust calibration, adaptive category limits, budgets and token-estimation policy;
+- `packages/core/src/adapter`: versioned runtime contract, canonical tool-group validation, isolated capability negotiation, exact local-KV eligibility/replay orchestration and framework-neutral conformance runner;
+- `packages/core/src/core`: portable canonical messages, model profiles, robust calibration, adaptive category limits, budgets and token-estimation policy;
 - `packages/core/src/continuation`: hashed-prefix continuation decisions without provider transport or response APIs;
 - `packages/core/src/config`: runtime-neutral configuration model and filesystem loader;
 - `packages/core/src/planner`: atomic grouping, deterministic ranking, fitting, validation and privacy-aware plans;
@@ -158,21 +180,25 @@ ds4-context-core (packages/core)
 - `packages/core/src/memory`: mutation projections, conservative contradiction/key detection, scope selection, prompt boundaries, ranking and diagnostics;
 - `packages/core/src/artifacts`: atomic content-addressed files, deterministic condensation, redaction, branch-safe literal search, reconciliation and garbage collection;
 - `packages/core/src/compaction`: structured summary contract, hierarchical graph model, validation, lifecycle metadata and source hashing;
-- `packages/core/src/retrieval`: task descriptors, safe FTS queries, deterministic ranking, evidence quoting, deduplication and token fitting;
+- `packages/core/src/retrieval`: task descriptors, safe FTS queries, runtime-neutral embedding port, semantic index orchestration, deterministic rank fusion, evidence quoting, quality comparison, deduplication and token fitting;
 - `packages/core/src/project`: trust-gated file discovery, hashing, Git state, symbol/chunk extraction, invalidation, retrieval and source quoting;
-- `packages/core/src/persistence`: rebuildable session/project/memory/pin SQLite state, repositories, FTS5, event replay and transactional migrations;
+- `packages/core/src/quality`: versioned replay fixtures/contracts, deterministic metrics, static/candidate comparison and metadata-only aggregation;
+- `packages/core/src/ranking`: bounded metadata-only features, classified label contracts, deterministic local training, checksummed model artifacts, aggregate shadow comparison and promotion-gated inference;
+- `packages/core/src/persistence`: rebuildable session/project/vector/memory/pin/quality SQLite state, repositories, FTS5, event replay and transactional migrations;
 - `packages/core/src/manifest` and `packages/core/src/shared`: runtime-neutral projections, provenance, hashing, stable serialization and logging.
 
+The `packages/reference-adapter` workspace is the non-Pi reference adapter: it reads bounded append-only canonical JSONL, injects completion through a host callback, enforces privacy at that callback boundary, rebuilds disposable snapshots and explicitly disables unsupported native features. A local host can inject a handle-free `LocalKvRuntimePort`; the port alone retains native handles and transport while core receives only exact prefix bytes transiently for hashing.
+
 The root `ds4-context-engine` package is the Pi adapter:
 
-- `src/pi-adapter`: byte-safe Pi JSONL reading, provenance mapping, custom mutation projection, active label discovery, checkpoints, runtime snapshots, Pi model completion for summaries and the narrow Pi-AI OpenAI Responses transport wrapper;
+- `src/pi-adapter`: byte-safe Pi JSONL reading, provenance mapping, memory/pin and learned-ranking label projection, active label discovery, checkpoints, runtime snapshots, Pi model completion for summaries and the narrow Pi-AI OpenAI Responses transport wrapper;
 - `src/extension`: Pi hooks, lifecycle, command presentation and fail-open/fail-closed orchestration.
 
-The adapter may import core exports. Core source must never import `@earendil-works/pi-ai`, `@earendil-works/pi-coding-agent`, `src/pi-adapter` or `src/extension`; an automated boundary test enforces this rule.
+Adapters may import core exports. Core source must never import `@earendil-works/pi-ai`, `@earendil-works/pi-coding-agent`, `src/pi-adapter`, `src/extension` or a reference-adapter source path; an automated boundary test enforces this rule. Runtime SDK dependencies belong only to their adapter package.
 
 ## Canonical and derived state
 
-The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, and append-only classified memory/pin custom mutations; live files remain canonical for project knowledge. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, and calibration data. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
+The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, append-only classified memory/pin mutations and metadata-only learned-ranking feedback/replay labels; live files remain canonical for project knowledge. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, source-hash/model-keyed vectors, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, calibration data, and bounded metadata-only quality samples. The checksummed learned-ranking model is a separate disposable local artifact reconstructed from canonical labels; it contains bounded weights and aggregate gate metadata, never raw text. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
 
 ## Lifecycle
 
@@ -195,6 +221,6 @@ Database settings:
 
 ## Failure policy
 
-Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, and diagnostics failures are caught at the extension boundary. Session index failures retain the previous transactional snapshot. Historical and project FTS errors degrade to exact matches; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
+Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, quality measurement, and diagnostics failures are caught at the extension boundary. Session index failures retain the previous transactional snapshot. Cross-session source failures exclude only the unverifiable source and retain explicit diagnostics; they do not disable current-session memory. Historical and project FTS errors degrade to exact matches; embedding consent/privacy/model/timeout/corruption/provider failures degrade to lexical results; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
 
 Privacy is the exception to ordinary fail-open behavior. Once enabled, planner failures return the sanitized native array, preparation failures replace message content with structural placeholders, and provider-payload sanitizer failures return an empty object so the remote request fails rather than receiving unchecked content. Pi 0.84.3 runs provider-payload handlers in extension load order, so DS4 should be loaded last when other extensions can rewrite provider payloads.

package/docs/CONTEXT_MANIFEST.md

@@ -20,13 +20,14 @@ A Context Manifest explains the context visible at DS4's Pi `context` hook witho
 - artifact IDs, SHA-256, bytes, MIME, classification, exact source entry/tool IDs, error state, and before/after token estimates;
 - provider destination and allow-set names, selected classification counts, blocked/excluded/redacted counts, final provider-check count, and enforcement stage;
 - planner mode/version, original and selected counts, group counts, internal budgets, duration, and fallback reason;
+- learned-ranking mode/status, feature/model versions, candidate count, aggregate disagreement/rank shift, duration, and generic static-fallback reason;
 - planner and policy versions;
 - deterministic SHA-256 over system prompt, active tools, and messages;
 - Pi's reported context usage when available;
 - finalized uncached input, cache-read, cache-write, total provider input, and cache shares when available;
 - optional native-continuation eligibility, storage-consent state, request mode, full/sent/omitted input-item counts, state age, generic fallback/invalidation reason, and managed-replay retry outcome.
 
-The manifest does **not** contain system instructions, message text, classified spans, pin content, memory claims, project snippets, artifact content/excerpts, tool arguments/results, image data, provider payloads, provider response/conversation IDs, API keys, or headers.
+The manifest does **not** contain system instructions, message text, classified spans, pin content, memory claims, project snippets, artifact content/excerpts, learned-ranking feature vectors/labels/candidate IDs/model weights, local-KV prefixes/fingerprints/handles, tool arguments/results, image data, provider payloads, provider response/conversation IDs, API keys, or headers. Local-KV hit/miss/prefill counters are volatile adapter diagnostics and are not copied into Pi manifests.
 
 ## Provenance mapping
 
@@ -62,6 +63,7 @@ Use:
 /context pins
 /context memory
 /context privacy
+/context ranking
 /context continuation
 /context artifacts
 ```

package/docs/CONTEXT_PLANNER.md

@@ -76,6 +76,14 @@ With privacy disabled, DS4 returns Pi's original `AgentMessage[]` when:
 
 Expected fallbacks are recorded in the Context Manifest. With privacy enabled, the fallback baseline is the sanitized native array—not raw Pi messages—and an unexpected privacy failure replaces content/payload fields instead of sending unchecked data. Observer mode disables planning but still enforces enabled privacy policy and records manifests/usage calibration.
 
+## Quality measurement
+
+M14 can queue the finalized manifest after planning when `quality.enabled` is true; materialization runs after `agent_settled`, outside provider planning latency. It records only counts, ratios, normalized reason codes, budget utilization and separate timing; it does not inspect or persist message/evidence text and cannot alter the active selection. Live samples remain unlabeled for evidence recall. The versioned synthetic replay corpus supplies expected source IDs for deterministic baseline/candidate comparisons. Any quality failure is isolated and the 0.1 plan remains active. See [`CONTEXT_QUALITY.md`](CONTEXT_QUALITY.md).
+
+## Learned ranking
+
+M18 can evaluate bounded metadata-only features after privacy exclusion and before supplemental candidates enter category fitting. `shadow` keeps every static score/order authoritative and records aggregate disagreement only. `active` is accepted only for a compatible checksummed model carrying an eligible held-out promotion report. Privacy exclusions, mandatory pins/current turns, atomic groups and hard budgets cannot be overridden. See [`LEARNED_RANKING.md`](LEARNED_RANKING.md).
+
 ## Current limits
 
-The planner does not call a model inside the `context` hook. Model calibration uses only finalized provider usage and deterministic local statistics. Historical/project retrieval and memory ranking are lexical; semantic reranking is intentionally disabled even if configured. Project symbol extraction is heuristic, artifact search is literal, and memory/pin creation is manual-first. Automatic memory extraction remains disabled; M10 supplies policy enforcement but not an automatic classifier or confirmation workflow. Provider-payload coverage targets Pi 0.84.3's supported serializers, and DS4 must load after any extension allowed to replace payloads when strict final ordering is required.
+The planner does not call a model inside the `context` hook. Model calibration uses only finalized provider usage and deterministic local statistics. Historical/project retrieval can opt into derived semantic candidates; learned supplemental reranking remains off by default and active mode is promotion-gated. Project symbol extraction is heuristic, artifact search is literal, and memory/pin creation is manual-first. Automatic memory extraction remains disabled; M10 supplies policy enforcement but not an automatic classifier or confirmation workflow. Provider-payload coverage targets Pi 0.84.3's supported serializers, and DS4 must load after any extension allowed to replace payloads when strict final ordering is required.

package/docs/CONTEXT_QUALITY.md

@@ -0,0 +1,91 @@
+# Context Quality Metrics
+
+M14 adds an opt-in, metadata-only quality layer around the deterministic 0.1 planner. The context hook only queues the completed metadata-only manifest; metric materialization and storage run after `agent_settled` (or during shutdown flush), outside provider planning latency. Measurement never changes selection, ranking, privacy enforcement, provider payloads, or Pi JSONL.
+
+## Configuration
+
+Quality sampling is disabled by default:
+
+```json
+{
+  "quality": {
+    "enabled": true,
+    "maxSamples": 1000
+  }
+}
+```
+
+`maxSamples` must be between 1 and 100,000. Retention is bounded in SQLite. Disabling the feature returns before sample scheduling. Enabling it adds only a bounded metadata-reference queue operation to the context path; the more expensive metric pass is deferred.
+
+## Metrics
+
+`context-quality-v1` records deterministic counts and ratios for:
+
+- expected evidence recall;
+- irrelevant selected-token ratio;
+- duplicate evidence references;
+- provenance coverage;
+- current-request retention;
+- atomic-group validity;
+- overflow and planner-fallback rates;
+- selected/dropped source-kind counts;
+- category budget utilization;
+- normalized selection/drop reason counts.
+
+The primary score is versioned with the metric contract:
+
+```text
+35% evidence recall
+20% relevant-token share
+15% provenance coverage
+15% current-request retention
+10% atomic-group validity
+ 5% no-overflow/no-fallback reliability
+```
+
+A ratio with no applicable denominator is reported as `null` and contributes a neutral value once an aggregate contains labeled evidence. An aggregate with no labeled evidence reports a zero primary score rather than claiming success. Live requests do not have human expected-evidence labels, so their evidence recall remains explicitly unlabeled rather than being assigned a tautological success. Versioned replay fixtures supply expected source IDs and produce labeled recall.
+
+Planning duration is stored and reported separately. It is excluded from deterministic aggregates and golden output because wall-clock time is not byte-stable.
+
+## Privacy and storage
+
+`context_quality_samples` is a disposable SQLite v11 projection. A stored sample contains only:
+
+- schema, metric, corpus, planner and profile versions;
+- aggregate source-kind, token, budget and decision counts;
+- outcome labels;
+- normalized timing values.
+
+It does **not** contain prompts, messages, summaries, memory claims, artifact text, project paths, evidence text, provider payloads, provider response IDs, or raw evidence source IDs. Live source IDs are hashed only while constructing the volatile metric input; only resulting counts are persisted.
+
+Malformed, incomplete, unknown-version, or structurally inconsistent rows are ignored during aggregation. Quality write/read failures are caught independently and cannot replace or block the 0.1 context plan.
+
+Deleting SQLite discards samples without affecting canonical state. Replaying the same versioned local corpus reconstructs byte-identical non-timing aggregates.
+
+## Replay corpus and comparisons
+
+[`quality/corpus-v1.json`](../quality/corpus-v1.json) contains synthetic, sanitized metadata fixtures with task descriptors, expected evidence source IDs, atomic groups, token costs, and planner budgets. It contains no captured user or project text.
+
+Run the 0.1 static-ranking baseline against the task-weighted 0.2 candidate interface:
+
+```bash
+npm run quality:compare
+```
+
+The final stdout line is stable JSON. Add `-- --timing` to report wall-clock duration separately on stderr. The task-weighted candidate remains evaluation-only. M18 adds a separate sanitized learned-ranking promotion fixture contract that enforces quality, exact-recall, privacy, atomicity, overflow, latency and determinism gates before active ordering is eligible; see [`LEARNED_RANKING.md`](LEARNED_RANKING.md).
+
+## Diagnostics
+
+```text
+/context quality
+```
+
+The command reports sample counts, labeled coverage, aggregate scores, rates, category utilization, normalized reasons, and separate mean/p95 planning duration. It never renders source content.
+
+## Verification
+
+- `tests/unit/context-quality.test.ts` covers every metric, weighted aggregation, deterministic replay, and live-sample redaction.
+- `tests/integration/context-quality-repository.test.ts` covers delete/rebuild equivalence, bounded retention, and corrupt-row isolation.
+- `tests/golden/context-quality-comparison.test.ts` locks byte-stable non-timing comparison output.
+- `tests/integration/extension.test.ts` covers opt-in runtime recording and `/context quality`.
+- `tests/benchmarks/context-quality.bench.ts` measures disabled/enabled context-path scheduling separately from deferred 1,000-item materialization. On the development host, a 1,000-message planner measured `2.5680 ms` p99 with metrics disabled and `2.5929 ms` with enabled scheduling (about `0.97%` overhead); the deferred 1,000-item pass measured `11.6510 ms` p99. These are observational, not portable guarantees.