ds4-context-engine 0.1.2 → 0.2.0-beta.1

package/README.md

@@ -16,7 +16,7 @@ bounded active context with provenance
 Pi provider
 ```
 
-> **Project status:** M0–M13 are implemented. The Pi adapter and standalone `ds4-context-core` package are version `0.1.2`; the adapter targets Pi `0.84.3`.
+> **Project status:** M0–M19 are implemented. Prerelease `0.2.0-beta.1` adds quality measurement, structural/hybrid retrieval, cross-session project memory, learned-ranking shadow evaluation, and the runtime adapter kit. Stable `0.1.2` remains available; both lines target Pi `0.84.3`. M20 local KV reuse is not included in this prerelease.
 
 ## Why DS4
 
@@ -27,13 +27,18 @@ It provides:
 - deterministic token budgeting with soft and hard input limits;
 - preservation of the current request, recent turns and atomic tool call/result groups;
 - exact and FTS5 historical retrieval with source provenance;
-- trust-gated project indexing, Git-aware invalidation and bounded source snippets;
+- opt-in hybrid semantic retrieval with a deterministic local embedding and lexical fallback;
+- trust-gated structural project indexing, Git-aware invalidation and bounded source snippets;
 - hierarchical, validated, non-destructive compaction summaries;
 - persistent pins and append-only durable memory stored canonically in Pi JSONL;
+- opt-in checkpointed project-memory replay across exact trusted Pi project sessions;
 - content-addressed storage and bounded references for large tool results;
 - privacy classifications, secret redaction and provider-specific allow rules;
 - model-specific calibration and adaptive context allocation;
 - optional verified continuation for eligible OpenAI Responses profiles;
+- opt-in metadata-only context-quality metrics and deterministic replay comparisons;
+- checksummed metadata-only learned ranking with shadow mode, canonical classified feedback and static fallback;
+- a versioned runtime adapter contract, reusable conformance kit and non-Pi callback/JSONL reference adapter;
 - an inspectable Context Manifest explaining included and excluded material;
 - fail-open recovery to Pi's native context path for operational failures.
 
@@ -74,12 +79,20 @@ pi -e git:github.com/Alucard24/ds4-context-engine
 
 ### From npm
 
-Install the public npm package with:
+Install the latest stable public npm package with:
 
 ```bash
 pi install npm:ds4-context-engine
 ```
 
+Install the opt-in 0.2 prerelease from the `beta` dist-tag with:
+
+```bash
+pi install npm:ds4-context-engine@beta
+```
+
+All prerelease packages (`ds4-context-engine`, `ds4-context-core`, and `ds4-context-reference-adapter`) use the same exact version.
+
 ### Local checkout
 
 ```bash
@@ -106,6 +119,7 @@ After loading the extension, inspect its state:
 
 ```text
 /context status
+/context adapter
 /context tokens
 /context health
 ```
@@ -149,6 +163,7 @@ Project configuration and project source indexing are disabled when Pi reports t
 | Command | Purpose |
 | --- | --- |
 | `/context` or `/context status` | Runtime, session, planner and subsystem status |
+| `/context adapter` | Runtime contract and per-capability negotiation diagnostics |
 | `/context tokens` | Token budget and active-context composition |
 | `/context manifest` | Latest Context Manifest |
 | `/context explain` | Human-readable planning explanation |
@@ -159,6 +174,8 @@ Project configuration and project source indexing are disabled when Pi reports t
 | `/context project` | Project index and retrieval status |
 | `/context privacy` | Classification and provider-policy status |
 | `/context model` | Active model profile and calibration |
+| `/context quality` | Metadata-only context-quality scores and sample counts |
+| `/context ranking` | Learned model, promotion gate, aggregate shadow comparison and feedback counts |
 | `/context continuation` | Native continuation decisions and counters |
 | `/context artifacts` | Artifact storage and integrity status |
 | `/context compaction` | Last compaction status |
@@ -179,10 +196,22 @@ Project configuration and project source indexing are disabled when Pi reports t
 /context memory supersede MEMORY_ID [--source ID,ID] <new claim>
 /context memory invalidate MEMORY_ID [reason]
 /context memory expire MEMORY_ID [reason]
+/context memory sources
+/context memory exclude SESSION_ID [reason]
+/context memory include SESSION_ID
 ```
 
 Valid privacy classifications are `normal`, `internal`, `sensitive` and `local-only`.
 
+Learned-ranking feedback and local training are explicit:
+
+```text
+/context ranking feedback useful|irrelevant CANDIDATE_ID [--classification LEVEL]
+/context ranking train
+```
+
+See [`docs/LEARNED_RANKING.md`](docs/LEARNED_RANKING.md).
+
 ## Configuration reference
 
 The following example shows the main configuration groups. Omitted values use the defaults in [`packages/core/src/config/config.ts`](packages/core/src/config/config.ts).
@@ -208,7 +237,19 @@ The following example shows the main configuration groups. Omitted values use th
     "exact": true,
     "fts": true,
     "semantic": false,
-    "maxResults": 12
+    "maxResults": 12,
+    "embedding": {
+      "mode": "local",
+      "provider": "ds4-local",
+      "model": "feature-hash-v1",
+      "dimensions": 256,
+      "remoteProfiles": [],
+      "maxSources": 50000,
+      "candidatePool": 80,
+      "batchSize": 64,
+      "queryCacheSize": 64,
+      "timeoutMs": 2000
+    }
   },
   "project": {
     "enabled": true,
@@ -221,6 +262,8 @@ The following example shows the main configuration groups. Omitted values use th
   },
   "memory": {
     "enabled": true,
+    "crossSession": false,
+    "maxProjectSessions": 250,
     "maxPinChars": 4000,
     "maxClaimChars": 2000,
     "maxResults": 12
@@ -271,13 +314,27 @@ The following example shows the main configuration groups. Omitted values use th
     "maxStateAgeMs": 1800000,
     "retryManagedReplay": true
   },
+  "quality": {
+    "enabled": false,
+    "maxSamples": 1000
+  },
+  "ranking": {
+    "mode": "off",
+    "modelPath": "ds4-context/ranking-model.json",
+    "minimumTrainingSamples": 20,
+    "maxTrainingSamples": 10000,
+    "maxLatencyMs": 10
+  },
   "diagnostics": {
     "storeContextManifest": true,
     "storeFullRenderedContext": false,
     "logLevel": "info"
   },
   "storage": {
-    "databasePath": "ds4-context/context.db"
+    "databasePath": "ds4-context/context.db",
+    "busyTimeoutMs": 5000,
+    "writeRetryTimeoutMs": 30000,
+    "projectIndexLeaseMs": 120000
   }
 }
 ```
@@ -311,10 +368,11 @@ By default, derived state is stored below Pi's agent directory:
 ```text
 ~/.pi/agent/ds4-context/
 ├── context.db
+├── ranking-model.json
 └── artifacts/
 ```
 
-The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
+The database contains rebuildable indexes, summary metadata, manifests, project projections and calibration data. The optional checksummed learned-ranking model is also derived local state; its classified metadata-only labels remain canonical Pi custom entries. All Pi sessions share this WAL database: writes use bounded busy-aware transaction replay, and a renewable project lease prevents multiple Pi processes from indexing the same project concurrently. `busyTimeoutMs` controls each SQLite lock wait, while `writeRetryTimeoutMs` bounds the total replay window. Canonical memory and pin mutations remain append-only entries in Pi JSONL. Project files remain canonical for project knowledge. Complete tool results remain in Pi JSONL while the artifact store keeps verified, content-addressed copies for bounded retrieval.
 
 To validate or rebuild derived state:
 
@@ -323,32 +381,36 @@ To validate or rebuild derived state:
 /context rebuild-index
 ```
 
-Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated.
+Deleting DS4's database must not alter a Pi session or project, although derived indexes and calibration data will be regenerated. When `memory.crossSession` is enabled for a trusted project, DS4 discovers bounded sibling Pi JSONL files by exact canonical header identity, incrementally replays their explicit project mutations, and excludes missing or unverifiable sources.
 
 ## Development
 
 ```bash
 npm ci
 npm run build:core
+npm run build:adapters
 npm run typecheck
 npm test
 npm run check
+npm run quality:compare
 npm run pack:check
 npm pack --dry-run
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 ```
 
-The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check also builds both tarballs, installs them in a clean temporary consumer and starts the packaged extension with isolated Pi RPC state.
+The test suite covers configuration, migrations, canonical JSONL projection, planning, atomic tool groups, retrieval, compaction, project knowledge, artifacts, memory, privacy, model awareness, continuation, runtime-adapter conformance, the portable-core dependency boundary and Pi extension lifecycle behavior. The package check builds all three tarballs, installs them in a clean temporary consumer, reruns compiled reference-adapter conformance and starts the packaged Pi extension with isolated RPC state.
 
 ### Portable core
 
-`ds4-context-core` is a compiled ESM package with no Pi dependency. It owns runtime-neutral policy, storage and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root `ds4-context-engine` package is the Pi adapter and depends one-way on the core workspace.
+`ds4-context-core` is a compiled ESM package with no runtime SDK dependency. It owns runtime-neutral policy, storage, adapter contracts and projections; agent adapters translate native sessions and lifecycle hooks at the boundary. The root `ds4-context-engine` package is the Pi adapter. `ds4-context-reference-adapter` is a separately compiled non-Pi callback/JSONL implementation. Both depend one-way and exactly on matching core.
 
 ### Repository layout
 
 ```text
-packages/core/src   portable policy, planning, compaction, retrieval and storage
-src/pi-adapter      Pi JSONL projection, summary completion and provider integration
+packages/core/src                 portable policy, adapter kit, planning, retrieval and storage
+packages/reference-adapter/src    non-Pi callback/JSONL reference runtime boundary
+src/pi-adapter                    Pi JSONL projection, summary completion and provider integration
 src/extension       Pi hooks, commands and fail-open orchestration
 tests               core contract, unit, integration, golden and benchmark coverage
 scripts             package and release-readiness checks
@@ -359,10 +421,13 @@ scripts             package and release-readiness checks
 
 - [Architecture](docs/ARCHITECTURE.md)
 - [Context planner](docs/CONTEXT_PLANNER.md)
+- [Context quality](docs/CONTEXT_QUALITY.md)
+- [Learned ranking](docs/LEARNED_RANKING.md)
 - [Context Manifest](docs/CONTEXT_MANIFEST.md)
 - [Compaction](docs/COMPACTION.md)
 - [Summary graph](docs/SUMMARY_GRAPH.md)
 - [Historical retrieval](docs/RETRIEVAL.md)
+- [Hybrid semantic retrieval](docs/HYBRID_RETRIEVAL.md)
 - [Project knowledge](docs/PROJECT_KNOWLEDGE.md)
 - [Artifacts](docs/ARTIFACTS.md)
 - [Memory and pins](docs/MEMORY_AND_PINS.md)
@@ -370,6 +435,7 @@ scripts             package and release-readiness checks
 - [Model awareness](docs/MODEL_AWARENESS.md)
 - [Native continuation](docs/NATIVE_CONTINUATION.md)
 - [Portable core](docs/PORTABLE_CORE.md)
+- [Runtime adapter kit](docs/RUNTIME_ADAPTER_KIT.md)
 - [Storage](docs/STORAGE.md)
 - [Roadmap 0.2.0](docs/ROADMAP_0.2.0.md)
 - [Release process](docs/RELEASING.md)
@@ -378,9 +444,9 @@ scripts             package and release-readiness checks
 
 ## Roadmap
 
-The original M0–M13 roadmap is complete. `ds4-context-core` now contains the compiled Pi-independent implementation, while runtime-specific behavior remains in the Pi adapter.
+The original M0–M13 roadmap is complete. `ds4-context-core` contains the compiled runtime-neutral implementation. M14 context-quality metrics, M15 rich symbol indexing, M16 hybrid semantic retrieval, M17 cross-session project memory, M18 learned-ranking shadow evaluation and M19's runtime adapter/conformance kit plus non-Pi reference adapter are implemented on `main`. Learned active ranking remains promotion-gated and static ranking stays authoritative on every failure.
 
-The planned [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) covers context-quality metrics, richer symbol indexing, hybrid semantic retrieval, cross-session project memory, optional learned ranking, a runtime adapter kit with one reference adapter, and optional local KV reuse. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
+The remaining [0.2.0 roadmap](docs/ROADMAP_0.2.0.md) covers optional local KV reuse. Sensitive or transport-specific behavior remains opt-in, and the 0.1 lexical planner stays available as the deterministic fallback.
 
 ## Contributing
 

package/docs/ARCHITECTURE.md

@@ -9,7 +9,12 @@ Pi session_start
   -> validate Pi JSONL v3 header
   -> full index or checkpointed append sync
   -> replay versioned memory/pin custom-entry mutations into transactional projections
+  -> when opted in and trusted, checkpoint/replay explicit project mutations from exact-identity sibling Pi sessions
   -> if trusted, canonicalize project root and incrementally index bounded text files
+  -> parse TypeScript/JavaScript/Python/Go declaration boundaries through the runtime-neutral parser interface
+  -> fall back deterministically to bounded text windows when adapters, syntax, or language support are unavailable
+  -> persist only rebuildable symbol/signature/parent/import/reference projections
+  -> when semantic retrieval is opted in, refresh source-hash/model-keyed vectors through the runtime embedding port
   -> snapshot Git root/branch/HEAD/dirty paths
 
 Pi context hook
@@ -30,16 +35,22 @@ Pi context hook
   -> select a contiguous, model-adaptive recent tail
   -> derive current-task identifiers, files, errors, phrases and keywords
   -> query exact matches and FTS5 over canonical indexed entries
+  -> optionally add bounded cosine-ranked history candidates and deterministic lexical/vector rank fusion
   -> reject active-context duplicates and all alternate-branch candidates
   -> rank, deduplicate, quote and budget historical evidence groups
-  -> query exact path/symbol/phrase and project FTS5 candidates
-  -> live-validate candidate SHA-256 and reindex changed files
+  -> query exact literal path and qualified/simple declaration indexes ahead of phrase and project FTS5 candidates
+  -> optionally add bounded project vectors while retaining exact path/symbol priority
+  -> live-validate candidate SHA-256 and reindex only changed-file projections
   -> rank, overlap-deduplicate, quote and budget project source groups
   -> omit prohibited history/project/pin/memory supplements with metadata-only privacy reasons
   -> fit active Pi summaries in the remaining budget
   -> validate hard limit, current request, and tool call/results
   -> return selected messages or fail open to the already privacy-sanitized native context
   -> persist metadata-only Context Manifest, privacy counters, and prompt hash
+  -> when opted in, queue the completed metadata-only manifest for deferred quality measurement
+
+Pi agent_settled / shutdown
+  -> materialize and persist bounded content-free quality counts without changing the plan
 
 before_provider_request
   -> recheck provider-specific serialized system/messages/tools/content
@@ -109,10 +120,10 @@ session_tree / shutdown
   -> immutable graph nodes, ordered edges, roots, levels and current-branch active path
 
 /context retrieved
-  -> query terms, candidate counts, branch blocks, budget decisions and injected excerpts
+  -> query terms, lexical/vector/fused counts, embedding profile/freshness/fallback, branch blocks, budgets and excerpts
 
 /context project
-  -> trust, Git revision, file/snippet/stale counts, retrieval decisions and local excerpts
+  -> trust, Git revision, file/snippet/stale/vector counts, retrieval decisions and local excerpts
 
 /context pins | pin | unpin
   -> inspect or append immutable session/branch/project pin mutations
@@ -141,16 +152,18 @@ session_tree / shutdown
 Dependency direction is one-way:
 
 ```text
-Pi native types and lifecycle
-        ↓
-ds4-context-engine (src/pi-adapter + src/extension)
-        ↓
-ds4-context-core (packages/core)
+Pi native types and lifecycle                 callback/JSONL runtime
+        ↓                                              ↓
+ds4-context-engine                        ds4-context-reference-adapter
+        └──────────────────────┬───────────────────────┘
+                               ↓
+                  ds4-context-core (packages/core)
 ```
 
 `ds4-context-core` is compiled ESM and has no dependency on Pi. Its workspace contains:
 
-- `packages/core/src/core`: portable model profiles, robust calibration, adaptive category limits, budgets and token-estimation policy;
+- `packages/core/src/adapter`: versioned runtime contract, canonical tool-group validation, isolated capability negotiation and framework-neutral conformance runner;
+- `packages/core/src/core`: portable canonical messages, model profiles, robust calibration, adaptive category limits, budgets and token-estimation policy;
 - `packages/core/src/continuation`: hashed-prefix continuation decisions without provider transport or response APIs;
 - `packages/core/src/config`: runtime-neutral configuration model and filesystem loader;
 - `packages/core/src/planner`: atomic grouping, deterministic ranking, fitting, validation and privacy-aware plans;
@@ -158,21 +171,25 @@ ds4-context-core (packages/core)
 - `packages/core/src/memory`: mutation projections, conservative contradiction/key detection, scope selection, prompt boundaries, ranking and diagnostics;
 - `packages/core/src/artifacts`: atomic content-addressed files, deterministic condensation, redaction, branch-safe literal search, reconciliation and garbage collection;
 - `packages/core/src/compaction`: structured summary contract, hierarchical graph model, validation, lifecycle metadata and source hashing;
-- `packages/core/src/retrieval`: task descriptors, safe FTS queries, deterministic ranking, evidence quoting, deduplication and token fitting;
+- `packages/core/src/retrieval`: task descriptors, safe FTS queries, runtime-neutral embedding port, semantic index orchestration, deterministic rank fusion, evidence quoting, quality comparison, deduplication and token fitting;
 - `packages/core/src/project`: trust-gated file discovery, hashing, Git state, symbol/chunk extraction, invalidation, retrieval and source quoting;
-- `packages/core/src/persistence`: rebuildable session/project/memory/pin SQLite state, repositories, FTS5, event replay and transactional migrations;
+- `packages/core/src/quality`: versioned replay fixtures/contracts, deterministic metrics, static/candidate comparison and metadata-only aggregation;
+- `packages/core/src/ranking`: bounded metadata-only features, classified label contracts, deterministic local training, checksummed model artifacts, aggregate shadow comparison and promotion-gated inference;
+- `packages/core/src/persistence`: rebuildable session/project/vector/memory/pin/quality SQLite state, repositories, FTS5, event replay and transactional migrations;
 - `packages/core/src/manifest` and `packages/core/src/shared`: runtime-neutral projections, provenance, hashing, stable serialization and logging.
 
+The `packages/reference-adapter` workspace is the non-Pi reference adapter: it reads bounded append-only canonical JSONL, injects completion through a host callback, enforces privacy at that callback boundary, rebuilds disposable snapshots and explicitly disables unsupported native features.
+
 The root `ds4-context-engine` package is the Pi adapter:
 
-- `src/pi-adapter`: byte-safe Pi JSONL reading, provenance mapping, custom mutation projection, active label discovery, checkpoints, runtime snapshots, Pi model completion for summaries and the narrow Pi-AI OpenAI Responses transport wrapper;
+- `src/pi-adapter`: byte-safe Pi JSONL reading, provenance mapping, memory/pin and learned-ranking label projection, active label discovery, checkpoints, runtime snapshots, Pi model completion for summaries and the narrow Pi-AI OpenAI Responses transport wrapper;
 - `src/extension`: Pi hooks, lifecycle, command presentation and fail-open/fail-closed orchestration.
 
-The adapter may import core exports. Core source must never import `@earendil-works/pi-ai`, `@earendil-works/pi-coding-agent`, `src/pi-adapter` or `src/extension`; an automated boundary test enforces this rule.
+Adapters may import core exports. Core source must never import `@earendil-works/pi-ai`, `@earendil-works/pi-coding-agent`, `src/pi-adapter`, `src/extension` or a reference-adapter source path; an automated boundary test enforces this rule. Runtime SDK dependencies belong only to their adapter package.
 
 ## Canonical and derived state
 
-The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, and append-only classified memory/pin custom mutations; live files remain canonical for project knowledge. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, and calibration data. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
+The Pi session JSONL remains canonical for conversation/tool state, inline classification markers, append-only classified memory/pin mutations and metadata-only learned-ranking feedback/replay labels; live files remain canonical for project knowledge. Native continuation keeps only volatile request/response-item hashes plus the minimum response handle and creates no continuation table or custom entry. SQLite and content-addressed object files store only rebuildable indexes, source-hash/model-keyed vectors, summary nodes/edges, metadata-only manifests, project file/snippet projections, artifact copies/references, materialized memory/pins, calibration data, and bounded metadata-only quality samples. The checksummed learned-ranking model is a separate disposable local artifact reconstructed from canonical labels; it contains bounded weights and aggregate gate metadata, never raw text. Each aggregate's active text is the Pi compaction summary; non-active nodes created by the same operation are embedded in its details, while older ancestors remain in earlier entries. Deleting the database must never damage or alter a Pi session or project. Reopening a source session replays its memory/pin mutations. Ephemeral sessions keep manifests and graph nodes in memory, disable durable memory/pins/artifacts, and may share the project index because files—not session JSONL—are its durable source.
 
 ## Lifecycle
 
@@ -195,6 +212,6 @@ Database settings:
 
 ## Failure policy
 
-Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, and diagnostics failures are caught at the extension boundary. Session index failures retain the previous transactional snapshot. Historical and project FTS errors degrade to exact matches; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
+Configuration, database, session/project indexing, memory/pin replay, artifact offload/search, retrieval, planning, observer, native continuation, quality measurement, and diagnostics failures are caught at the extension boundary. Session index failures retain the previous transactional snapshot. Cross-session source failures exclude only the unverifiable source and retain explicit diagnostics; they do not disable current-session memory. Historical and project FTS errors degrade to exact matches; embedding consent/privacy/model/timeout/corruption/provider failures degrade to lexical results; project subsystem failure contributes no snippets without disabling session management. Expected planning hazards produce an explicit fallback manifest and discard synthetic evidence.
 
 Privacy is the exception to ordinary fail-open behavior. Once enabled, planner failures return the sanitized native array, preparation failures replace message content with structural placeholders, and provider-payload sanitizer failures return an empty object so the remote request fails rather than receiving unchecked content. Pi 0.84.3 runs provider-payload handlers in extension load order, so DS4 should be loaded last when other extensions can rewrite provider payloads.

package/docs/CONTEXT_MANIFEST.md

@@ -20,13 +20,14 @@ A Context Manifest explains the context visible at DS4's Pi `context` hook witho
 - artifact IDs, SHA-256, bytes, MIME, classification, exact source entry/tool IDs, error state, and before/after token estimates;
 - provider destination and allow-set names, selected classification counts, blocked/excluded/redacted counts, final provider-check count, and enforcement stage;
 - planner mode/version, original and selected counts, group counts, internal budgets, duration, and fallback reason;
+- learned-ranking mode/status, feature/model versions, candidate count, aggregate disagreement/rank shift, duration, and generic static-fallback reason;
 - planner and policy versions;
 - deterministic SHA-256 over system prompt, active tools, and messages;
 - Pi's reported context usage when available;
 - finalized uncached input, cache-read, cache-write, total provider input, and cache shares when available;
 - optional native-continuation eligibility, storage-consent state, request mode, full/sent/omitted input-item counts, state age, generic fallback/invalidation reason, and managed-replay retry outcome.
 
-The manifest does **not** contain system instructions, message text, classified spans, pin content, memory claims, project snippets, artifact content/excerpts, tool arguments/results, image data, provider payloads, provider response/conversation IDs, API keys, or headers.
+The manifest does **not** contain system instructions, message text, classified spans, pin content, memory claims, project snippets, artifact content/excerpts, learned-ranking feature vectors/labels/candidate IDs/model weights, tool arguments/results, image data, provider payloads, provider response/conversation IDs, API keys, or headers.
 
 ## Provenance mapping
 
@@ -62,6 +63,7 @@ Use:
 /context pins
 /context memory
 /context privacy
+/context ranking
 /context continuation
 /context artifacts
 ```

package/docs/CONTEXT_PLANNER.md

@@ -76,6 +76,14 @@ With privacy disabled, DS4 returns Pi's original `AgentMessage[]` when:
 
 Expected fallbacks are recorded in the Context Manifest. With privacy enabled, the fallback baseline is the sanitized native array—not raw Pi messages—and an unexpected privacy failure replaces content/payload fields instead of sending unchecked data. Observer mode disables planning but still enforces enabled privacy policy and records manifests/usage calibration.
 
+## Quality measurement
+
+M14 can queue the finalized manifest after planning when `quality.enabled` is true; materialization runs after `agent_settled`, outside provider planning latency. It records only counts, ratios, normalized reason codes, budget utilization and separate timing; it does not inspect or persist message/evidence text and cannot alter the active selection. Live samples remain unlabeled for evidence recall. The versioned synthetic replay corpus supplies expected source IDs for deterministic baseline/candidate comparisons. Any quality failure is isolated and the 0.1 plan remains active. See [`CONTEXT_QUALITY.md`](CONTEXT_QUALITY.md).
+
+## Learned ranking
+
+M18 can evaluate bounded metadata-only features after privacy exclusion and before supplemental candidates enter category fitting. `shadow` keeps every static score/order authoritative and records aggregate disagreement only. `active` is accepted only for a compatible checksummed model carrying an eligible held-out promotion report. Privacy exclusions, mandatory pins/current turns, atomic groups and hard budgets cannot be overridden. See [`LEARNED_RANKING.md`](LEARNED_RANKING.md).
+
 ## Current limits
 
-The planner does not call a model inside the `context` hook. Model calibration uses only finalized provider usage and deterministic local statistics. Historical/project retrieval and memory ranking are lexical; semantic reranking is intentionally disabled even if configured. Project symbol extraction is heuristic, artifact search is literal, and memory/pin creation is manual-first. Automatic memory extraction remains disabled; M10 supplies policy enforcement but not an automatic classifier or confirmation workflow. Provider-payload coverage targets Pi 0.84.3's supported serializers, and DS4 must load after any extension allowed to replace payloads when strict final ordering is required.
+The planner does not call a model inside the `context` hook. Model calibration uses only finalized provider usage and deterministic local statistics. Historical/project retrieval can opt into derived semantic candidates; learned supplemental reranking remains off by default and active mode is promotion-gated. Project symbol extraction is heuristic, artifact search is literal, and memory/pin creation is manual-first. Automatic memory extraction remains disabled; M10 supplies policy enforcement but not an automatic classifier or confirmation workflow. Provider-payload coverage targets Pi 0.84.3's supported serializers, and DS4 must load after any extension allowed to replace payloads when strict final ordering is required.

package/docs/CONTEXT_QUALITY.md

@@ -0,0 +1,91 @@
+# Context Quality Metrics
+
+M14 adds an opt-in, metadata-only quality layer around the deterministic 0.1 planner. The context hook only queues the completed metadata-only manifest; metric materialization and storage run after `agent_settled` (or during shutdown flush), outside provider planning latency. Measurement never changes selection, ranking, privacy enforcement, provider payloads, or Pi JSONL.
+
+## Configuration
+
+Quality sampling is disabled by default:
+
+```json
+{
+  "quality": {
+    "enabled": true,
+    "maxSamples": 1000
+  }
+}
+```
+
+`maxSamples` must be between 1 and 100,000. Retention is bounded in SQLite. Disabling the feature returns before sample scheduling. Enabling it adds only a bounded metadata-reference queue operation to the context path; the more expensive metric pass is deferred.
+
+## Metrics
+
+`context-quality-v1` records deterministic counts and ratios for:
+
+- expected evidence recall;
+- irrelevant selected-token ratio;
+- duplicate evidence references;
+- provenance coverage;
+- current-request retention;
+- atomic-group validity;
+- overflow and planner-fallback rates;
+- selected/dropped source-kind counts;
+- category budget utilization;
+- normalized selection/drop reason counts.
+
+The primary score is versioned with the metric contract:
+
+```text
+35% evidence recall
+20% relevant-token share
+15% provenance coverage
+15% current-request retention
+10% atomic-group validity
+ 5% no-overflow/no-fallback reliability
+```
+
+A ratio with no applicable denominator is reported as `null` and contributes a neutral value once an aggregate contains labeled evidence. An aggregate with no labeled evidence reports a zero primary score rather than claiming success. Live requests do not have human expected-evidence labels, so their evidence recall remains explicitly unlabeled rather than being assigned a tautological success. Versioned replay fixtures supply expected source IDs and produce labeled recall.
+
+Planning duration is stored and reported separately. It is excluded from deterministic aggregates and golden output because wall-clock time is not byte-stable.
+
+## Privacy and storage
+
+`context_quality_samples` is a disposable SQLite v11 projection. A stored sample contains only:
+
+- schema, metric, corpus, planner and profile versions;
+- aggregate source-kind, token, budget and decision counts;
+- outcome labels;
+- normalized timing values.
+
+It does **not** contain prompts, messages, summaries, memory claims, artifact text, project paths, evidence text, provider payloads, provider response IDs, or raw evidence source IDs. Live source IDs are hashed only while constructing the volatile metric input; only resulting counts are persisted.
+
+Malformed, incomplete, unknown-version, or structurally inconsistent rows are ignored during aggregation. Quality write/read failures are caught independently and cannot replace or block the 0.1 context plan.
+
+Deleting SQLite discards samples without affecting canonical state. Replaying the same versioned local corpus reconstructs byte-identical non-timing aggregates.
+
+## Replay corpus and comparisons
+
+[`quality/corpus-v1.json`](../quality/corpus-v1.json) contains synthetic, sanitized metadata fixtures with task descriptors, expected evidence source IDs, atomic groups, token costs, and planner budgets. It contains no captured user or project text.
+
+Run the 0.1 static-ranking baseline against the task-weighted 0.2 candidate interface:
+
+```bash
+npm run quality:compare
+```
+
+The final stdout line is stable JSON. Add `-- --timing` to report wall-clock duration separately on stderr. The task-weighted candidate remains evaluation-only. M18 adds a separate sanitized learned-ranking promotion fixture contract that enforces quality, exact-recall, privacy, atomicity, overflow, latency and determinism gates before active ordering is eligible; see [`LEARNED_RANKING.md`](LEARNED_RANKING.md).
+
+## Diagnostics
+
+```text
+/context quality
+```
+
+The command reports sample counts, labeled coverage, aggregate scores, rates, category utilization, normalized reasons, and separate mean/p95 planning duration. It never renders source content.
+
+## Verification
+
+- `tests/unit/context-quality.test.ts` covers every metric, weighted aggregation, deterministic replay, and live-sample redaction.
+- `tests/integration/context-quality-repository.test.ts` covers delete/rebuild equivalence, bounded retention, and corrupt-row isolation.
+- `tests/golden/context-quality-comparison.test.ts` locks byte-stable non-timing comparison output.
+- `tests/integration/extension.test.ts` covers opt-in runtime recording and `/context quality`.
+- `tests/benchmarks/context-quality.bench.ts` measures disabled/enabled context-path scheduling separately from deferred 1,000-item materialization. On the development host, a 1,000-message planner measured `2.5680 ms` p99 with metrics disabled and `2.5929 ms` with enabled scheduling (about `0.97%` overhead); the deferred 1,000-item pass measured `11.6510 ms` p99. These are observational, not portable guarantees.

package/docs/HYBRID_RETRIEVAL.md

@@ -0,0 +1,117 @@
+# Hybrid Semantic Retrieval
+
+M16 adds opt-in vector candidate generation to historical and trusted-project retrieval. Exact identifiers and FTS5 remain active and authoritative; semantic candidates are fused into the same bounded, deterministic ranking and never replace source provenance or live-hash checks.
+
+## Configuration
+
+Semantic retrieval remains disabled for upgrades from 0.1:
+
+```json
+{
+  "retrieval": {
+    "semantic": true
+  }
+}
+```
+
+The supported default is the runtime-owned local feature-hash embedding:
+
+```json
+{
+  "retrieval": {
+    "semantic": true,
+    "embedding": {
+      "mode": "local",
+      "provider": "ds4-local",
+      "model": "feature-hash-v1",
+      "dimensions": 256,
+      "maxSources": 50000,
+      "candidatePool": 80,
+      "batchSize": 64,
+      "queryCacheSize": 64,
+      "timeoutMs": 2000
+    }
+  }
+}
+```
+
+It is deterministic, has no native dependency and performs no network access. Core defines `EmbeddingPort`; the Pi adapter supplies the implementation. Other runtimes can inject a compatible local, WASM or remote port without adding model invocation to core.
+
+Remote embedding requires all of the following:
+
+- `mode: "remote"`;
+- an exact `provider/model` entry in `remoteProfiles` (wildcards are rejected);
+- `privacy.enabled: true`;
+- provider-specific privacy allow rules;
+- a runtime-injected `EmbeddingPort` whose provider, model, dimensions and remote destination exactly match configuration.
+
+```json
+{
+  "retrieval": {
+    "semantic": true,
+    "embedding": {
+      "mode": "remote",
+      "provider": "embedding.example",
+      "model": "semantic-v2",
+      "dimensions": 768,
+      "remoteProfiles": ["embedding.example/semantic-v2"]
+    }
+  },
+  "privacy": {
+    "enabled": true,
+    "remoteProviders": {
+      "embedding.example": ["normal", "internal"]
+    }
+  }
+}
+```
+
+The packaged Pi adapter intentionally includes no default remote embedding client. Missing or mismatched ports produce lexical-only results.
+
+## Privacy
+
+Every remote source and query passes through `PrivacyPolicyEngine` before `EmbeddingPort.embed`. A value whose effective classification is `local-only`, or which contains a provider-blocked span, is excluded as a whole and never reaches the remote port. Allowed text is secret-redacted before invocation. Local mode does not cross a provider boundary.
+
+Embedding diagnostics contain only counts, model identity, dimensions, destination, freshness, cache status, timings and normalized fallback reasons. They never contain query text, source text, vectors, provider response IDs or remote handles.
+
+## Derived storage
+
+SQLite schema v13 adds `derived_embeddings`. Each row is keyed by:
+
+```text
+source kind + scope + source key + source hash + chunking version
++ embedding provider + embedding model + dimensions
+```
+
+Rows contain the source group, numeric vector JSON and indexing time, but no copied query or evidence text. Session entries use `pi-session-entry-v1`; project chunks use their parser version or `text-window-v1`.
+
+A source edit prunes only vectors whose key/hash/chunk version is no longer current. Unchanged source rows and unrelated model/dimension profiles remain intact. A model or dimension switch selects a separate profile instead of rewriting compatible rows. Deleting SQLite discards the whole vector projection; canonical Pi JSONL and live project files rebuild it.
+
+## Candidate generation and rank fusion
+
+Candidate pools are bounded. Generation order is:
+
+1. exact project path, qualified symbol, simple symbol, quoted phrase or historical identifier;
+2. escaped FTS5 candidates;
+3. cosine-ranked vector candidates.
+
+The lexical and vector ranks are combined with deterministic reciprocal-rank fusion, semantic similarity and stable source-ID tie-breaking. Exact matches retain a score tier that vectors cannot displace. Historical candidates still require active-branch membership and exclusion from the active native context. Project candidates still require project trust, sensitive-file exclusion and a live SHA-256 match before injection.
+
+Source vectors are generated during session/project index sync. Query vectors use a bounded volatile hash-keyed cache. Repeating a request with current source vectors performs no embedding call. Query hashes and vectors are not persisted as conversation state.
+
+## Failure behavior
+
+Missing models, consent failure, privacy exclusion, invalid dimensions, corrupt vectors, synchronous-port timeout, provider exceptions, vector storage errors and vector search errors all return the available exact/FTS result. They cannot fail context planning. Diagnostics expose the fallback reason without source content.
+
+## Quality gate
+
+[`quality/semantic-corpus-v1.json`](../quality/semantic-corpus-v1.json) is a synthetic extension of the M14 replay methodology. It measures the same evidence-recall and irrelevant-token signals for semantic synonyms and exact-match preservation. The byte-stable golden report records:
+
+```text
+lexical evidence recall  0.25
+hybrid evidence recall   1.00
+recall delta             +0.75
+irrelevant-token delta    0.00
+```
+
+Verification covers local query reuse, source-vector caching, profile isolation, changed-source pruning, remote `local-only` exclusion, provider failure, exact priority, historical/project fusion and schema v13 rebuild behavior.