ds4-context-engine 0.1.2 → 0.2.0-beta.2

package/docs/RELEASING.md

@@ -1,21 +1,22 @@
 # Releasing DS4
 
-DS4 publishes two packages with the same version:
+DS4 publishes three packages with the same version:
 
-1. `ds4-context-core`, the runtime-neutral compiled package;
-2. `ds4-context-engine`, the Pi adapter.
+1. `ds4-context-core`, the runtime-neutral compiled package and adapter kit;
+2. `ds4-context-reference-adapter`, the non-Pi callback/JSONL reference implementation;
+3. `ds4-context-engine`, the Pi adapter.
 
-The adapter has an exact dependency on the matching core version, so the core package must always be published first.
+Both adapters have an exact dependency on the matching core version, so core must be published first.
 
 ## Prerequisites
 
 - use a clean `main` checkout synchronized with `origin/main`;
-- use a supported Node.js release and npm account authorized for both package names;
-- verify that both `package.json` files use the intended version;
-- verify that `ds4-context-engine` depends exactly on that version of `ds4-context-core`;
+- use a supported Node.js release and npm account authorized for all three package names;
+- verify that all three `package.json` files use the intended version;
+- verify that both adapters depend exactly on that version of `ds4-context-core`;
 - do not include session data, `.pi` state, databases, credentials, or provider payloads.
 
-The automated package check enforces matching versions, the exact core dependency, bounded tarball inventories, clean consumer installation, core ESM exports, and packaged Pi extension startup through isolated RPC state.
+The automated package check enforces matching versions, exact core dependencies, runtime-SDK isolation, bounded tarball inventories, clean consumer installation, core ESM exports, compiled reference-adapter conformance, and packaged Pi extension startup through isolated RPC state.
 
 ## Validate
 
@@ -29,26 +30,29 @@ git status --short
 
 CI runs the same checks on the minimum supported Node.js version and the current Node.js LTS line. `npm run pack:check` uses a temporary directory and removes it when complete. Set `DS4_KEEP_PACK_TMP=1` only when diagnosing a failed package check.
 
-Review both public tarballs before publishing:
+Review all public tarballs before publishing:
 
 ```bash
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 npm pack --dry-run
 ```
 
 ## Version
 
-Keep the root package, core workspace, and exact adapter dependency synchronized. For a future version stored in `$VERSION`:
+Keep the root package, both workspaces, and exact adapter dependencies synchronized. For a future version stored in `$VERSION`:
 
 ```bash
 npm pkg set version="$VERSION"
 npm pkg set version="$VERSION" --workspace ds4-context-core
+npm pkg set version="$VERSION" --workspace ds4-context-reference-adapter
 npm pkg set dependencies.ds4-context-core="$VERSION"
+npm pkg set dependencies.ds4-context-core="$VERSION" --workspace ds4-context-reference-adapter
 npm install --package-lock-only
 npm run pack:check
 ```
 
-Review `package.json`, `packages/core/package.json`, and `package-lock.json` before committing the release change.
+Review `package.json`, both workspace package manifests, and `package-lock.json` before committing the release change.
 
 ## Publish
 
@@ -57,12 +61,13 @@ Authenticate with npm, verify the active account, and publish in dependency orde
 ```bash
 npm whoami
 npm publish --workspace ds4-context-core --access public
+npm publish --workspace ds4-context-reference-adapter --access public
 npm publish --access public
 ```
 
-If core succeeds but adapter publication fails, fix the adapter release and retry it with the same version. Do not rewrite or unpublish a valid core release merely to make the two commands appear atomic.
+If core succeeds but an adapter publication fails, fix that adapter release and retry it with the same version. Do not rewrite or unpublish a valid core release merely to make the commands appear atomic.
 
-After both registry packages are available:
+After all registry packages are available:
 
 1. install them in a fresh temporary project and rerun the package smoke check if registry propagation was delayed;
 2. create and push the signed or annotated `v$VERSION` tag;

package/docs/RETRIEVAL.md

@@ -15,7 +15,7 @@ M6 recovers original session evidence that Pi compaction removed from the active
 9. Deduplicate normalized identical text, preferring the higher-ranked/newer source.
 10. Build individually bounded evidence messages, enforce the active provider privacy policy, and let the managed planner fit allowed groups after recent turns but before summaries.
 
-No LLM is called during retrieval. `retrieval.semantic: true` produces a diagnostic warning but does not enable embeddings in M6.
+No chat LLM is called during retrieval. Since M16, `retrieval.semantic: true` adds opt-in vectors through the runtime-neutral `EmbeddingPort`; exact and FTS retrieval remain authoritative and any embedding failure falls back to lexical results. See [Hybrid Semantic Retrieval](HYBRID_RETRIEVAL.md).
 
 ## Ranking
 

package/docs/ROADMAP_0.2.0.md

@@ -49,6 +49,8 @@ M14 is first because adaptive ranking must be measured against a stable baseline
 
 ## M14 — Context Quality Metrics
 
+Status: **implemented on `main` for `0.2.0-alpha.1`**. The active planner remains the 0.1 deterministic baseline; quality collection is opt-in and candidate strategies are replay-only.
+
 ### Deliverables
 
 - A sanitized replay corpus with expected evidence source IDs and task descriptors.
@@ -70,6 +72,8 @@ Quality samples store planner/profile versions, source-kind counts, token totals
 
 ## M15 — Rich Symbol Indexing
 
+Status: **implemented on `main` for `0.2.0-alpha.1`**. Structural parsing remains a disposable local projection; unsupported or invalid source retains deterministic text-window behavior.
+
 ### Deliverables
 
 - A parser interface in `ds4-context-core` with deterministic regex fallback.
@@ -90,6 +94,8 @@ The parser implementation must not introduce a mandatory native build dependency
 
 ## M16 — Hybrid Semantic Retrieval
 
+Status: **implemented on `main` for `0.2.0-alpha.2`**. Semantic retrieval remains opt-in; exact/FTS ranking and lexical-only failure behavior remain available.
+
 ### Deliverables
 
 - A runtime-neutral embedding port; model invocation remains outside core.
@@ -111,6 +117,8 @@ Semantic retrieval does not replace exact matching. Exact paths, symbols, quoted
 
 ## M17 — Cross-Session Project Memory
 
+Status: **implemented on `main` behind `memory.crossSession` for `0.2.0-alpha.3`**. Discovery remains opt-in and exact trusted project identity is mandatory.
+
 ### Deliverables
 
 - Discovery of canonical Pi session JSONL files associated with the same trusted canonical project identity.
@@ -131,6 +139,8 @@ Version 0.2.0 does not silently extract new memories from conversation text. A d
 
 ## M18 — Learned Ranking
 
+Status: **implemented on `main` for `0.2.0-beta.1` with `off`, aggregate-only `shadow`, and promotion-gated `active` modes**. Static ranking remains authoritative for missing, corrupt, incompatible, unpromoted or regressing artifacts.
+
 ### Deliverables
 
 - A bounded feature schema based on metadata such as source kind, exact/FTS/vector scores, recency, branch relation, symbol relation, classification eligibility, token cost and prior selection outcome.
@@ -153,6 +163,8 @@ If the gate is not met, 0.2.0 ships shadow mode but keeps static ranking active.
 
 ## M19 — Runtime Adapter Kit
 
+Status: **implemented on `main` and released in `0.2.0-beta.1`**. Core ships `runtime-adapter-v1` plus a framework-neutral conformance runner; Pi exposes contract capability diagnostics, and the separately packaged callback/JSONL reference adapter passes all seven conformance cases.
+
 ### Deliverables
 
 - A documented adapter contract for canonical history snapshots, model limits, tool atomicity, trusted project roots, completion, privacy enforcement and lifecycle shutdown.
@@ -170,6 +182,8 @@ If the gate is not met, 0.2.0 ships shadow mode but keeps static ranking active.
 
 ## M20 — Local KV Capability
 
+Status: **implemented on `main` and released in `0.2.0-beta.2` behind `localKvReuse.enabled`**. Core derives exact metadata-only eligibility fingerprints and guarantees full replay on every non-hit; injected runtime ports retain all volatile handles and transport. Pi remains explicitly unsupported.
+
 ### Deliverables
 
 - An optional adapter capability for local inference runtimes that expose reusable prefix/KV state.
@@ -219,10 +233,10 @@ SQLite schema changes use forward migrations plus complete rebuild tests from ca
 
 - M18 learned ranker in shadow mode.
 - Privacy, rebuild, corruption and performance hardening for M14–M18.
+- M19 adapter contract, conformance kit and reference adapter.
 
 ### `0.2.0-beta.2`
 
-- M19 adapter contract, conformance kit and reference adapter.
 - M20 local KV capability for runtimes that support it.
 
 ### `0.2.0-rc.1`

package/docs/RUNTIME_ADAPTER_KIT.md

@@ -0,0 +1,145 @@
+# Runtime Adapter Kit
+
+M19 defines `runtime-adapter-v1`, a runtime-neutral boundary between DS4 policy and an agent host. The kit lives in `ds4-context-core/adapter/*`; the non-Pi implementation is published separately as `ds4-context-reference-adapter`. Core imports no runtime SDK.
+
+## Dependency and state boundary
+
+```text
+native agent runtime / provider SDK
+                ↓
+runtime adapter package
+                ↓
+ds4-context-core/adapter/runtime-adapter
+```
+
+An adapter owns native history conversion, trusted-root discovery, provider completion, final privacy enforcement, optional runtime capabilities, and shutdown. Core owns contract types, deterministic capability negotiation, canonical tool-group validation, and the framework-neutral conformance runner.
+
+The runtime's native history remains canonical. A `RuntimeHistorySnapshot` is a read-only projection with:
+
+- `runtimeId`, `sessionId`, schema version, and deterministic source revision;
+- one exact trusted canonical project root;
+- ordered `CanonicalMessage` values with stable source-entry provenance;
+- complete tool call/result atomic groups derived from canonical blocks.
+
+Applying a DS4 context plan must not write provider-facing synthetic messages back into canonical history. Rebuild deletes or discards only adapter/DS4 projections and reproduces the snapshot from the native source.
+
+## Contract
+
+`RuntimeAdapter` requires:
+
+| Method | Responsibility | Failure rule |
+|---|---|---|
+| `snapshotHistory()` | Project the active canonical history and tool relations. | Reject an invalid, corrupt, identity-mismatched, or untrusted source. Never invent history. |
+| `rebuildDerivedState()` | Discard adapter caches/projections and replay canonical state. | Preserve canonical files and the previous valid source. |
+| `currentModel()` | Return provider/model limits through `ModelDescriptor`. | Return `undefined` when no model is selected. |
+| `trustedProjectRoot()` | Return the runtime-approved canonical root. | Reject untrusted roots; do not broaden them. |
+| `enforcePrivacy()` | Sanitize a provider payload immediately before transport. | Fail closed and do not invoke transport on error. |
+| `complete()` | Invoke the runtime/provider boundary with the sanitized payload. | Return an explicit fallback result for invalid input, privacy failure, transport failure, or closed lifecycle. |
+| `capabilityDeclarations()` / `negotiateCapabilities()` | Version and independently enable optional host features. | Missing or malformed declarations are unsupported, not fatal to other features. |
+| `diagnostics()` | Return bounded metadata-only state. | Never include prompts, history, credentials, payloads, cache handles, or provider output. |
+| `shutdown()` | Release adapter resources idempotently. | Further writes/completions fall back; history reads are rejected. |
+
+Completion output remains runtime-owned and is not automatically canonical. An adapter must append a final runtime-native message through its ordinary canonical history API if the host chooses to persist it.
+
+M20-capable adapters may additionally expose `localKvPort` and aggregate `localKvDiagnostics()`. Both are optional because most runtimes, including Pi, expose no native KV state. The port interface carries eligibility hashes and complete sanitized replay payloads but never returns or accepts a cache handle.
+
+## Tool atomicity
+
+`buildCanonicalToolAtomicGroups()` connects every canonical `toolCall` block to all matching `toolResult` blocks. Calls sharing one assistant message form one component, so parallel tool batches cannot be split. `validateRuntimeHistorySnapshot()` rejects missing, overlapping, unknown, or mismatched groups. An incomplete live call is represented by a group with `complete: false`; it cannot be treated as a complete selectable exchange.
+
+## Capability negotiation
+
+The v1 registry is fixed and additive contracts must use a later contract version:
+
+- `compaction`;
+- `provider-continuation`;
+- `embeddings`;
+- `local-kv-reuse`.
+
+A supported capability requires a non-empty implementation version. Unsupported capabilities require a bounded reason. `negotiateRuntimeCapabilities()` evaluates each request independently:
+
+- a supported requested capability is enabled;
+- an unsupported optional capability is disabled with an informational diagnostic;
+- an unsupported required capability is disabled with a warning;
+- a duplicate, missing, or unversioned declaration is malformed and disabled;
+- no failure in one capability disables canonical history, privacy, completion fallback, or another valid capability.
+
+M19 introduced negotiation for `local-kv-reuse`; M20 adds `local-kv-eligibility-v1`, `LocalKvReuseController`, and the handle-free `LocalKvRuntimePort`. KV handles are never part of canonical history, core state, manifests, or diagnostics. See [Local KV Reuse](LOCAL_KV_REUSE.md).
+
+Pi advertises versioned compaction, OpenAI Responses continuation, and embedding-port support. It explicitly reports local KV reuse as unavailable. `/context adapter` shows the complete negotiation; `/context status` shows aggregate enabled/disabled counts.
+
+## Reusable conformance kit
+
+The conformance runner has no Vitest/Jest dependency. Adapter packages provide a factory that maps a versioned fixture and injected completion transport into their native runtime:
+
+```ts
+import {
+  assertRuntimeAdapterConformance,
+  runRuntimeAdapterConformance,
+} from "ds4-context-core/adapter/conformance";
+
+const report = await runRuntimeAdapterConformance({
+  name: "my-runtime",
+  async create(fixture, transport) {
+    // Create native canonical history from fixture.messages.
+    // Inject transport at the final provider boundary.
+    return { adapter, expectedProjectRoot, cleanup };
+  },
+});
+assertRuntimeAdapterConformance(report);
+```
+
+The seven checks cover:
+
+1. adapter/contract identity, model limits, and trusted root;
+2. complete capability declarations and isolated unsupported diagnostics;
+3. ordered canonical history and tool atomicity;
+4. deterministic rebuild from canonical state;
+5. remote privacy filtering plus synthetic sanitizer failure before observed transport;
+6. safe transport failure with native fallback still available;
+7. idempotent shutdown and closed-state behavior.
+
+Reports contain case IDs, booleans, and fixed failure codes only. The private marker and credential probes never appear in a report. Package smoke tests install core, Pi, and reference tarballs in a clean consumer and rerun reference conformance from compiled exports.
+
+## Reference-adapter compatibility spike
+
+M19 compared three reference targets:
+
+| Candidate | Strength | M19 risk |
+|---|---|---|
+| another full agent SDK | realistic lifecycle | adds fast-moving SDK/version coupling and obscures which behavior belongs to DS4 versus the SDK |
+| provider-client-only adapter | realistic completion | has no canonical branch, tool-group, or lifecycle contract to validate |
+| callback runtime with append-only JSONL | exercises every adapter responsibility with no vendor SDK | intentionally minimal; not a production orchestration framework |
+
+The callback JSONL target was selected because it covers the full contract while keeping the example inspectable and dependency-neutral. `ds4-context-reference-adapter` is therefore a non-Pi executable boundary example, not a claim of production support for a specific third-party agent framework.
+
+Its `ds4-runtime-session-v1` file is canonical and append-only. The header binds runtime ID, session ID, and canonical project root. Message records contain DS4 canonical messages. `createReferenceHistory()` refuses overwrite, `appendReferenceHistoryMessage()` checks provenance before append, and `rebuildDerivedState()` discards only the in-memory snapshot. File size and message count are bounded. Files are mode `0600` where supported.
+
+The reference adapter provides a callback completion transport, enabled fail-closed privacy by default, and optional `EmbeddingPort`. Native compaction and provider continuation remain explicitly unsupported. Local KV reuse is unsupported by default but becomes a versioned supported capability when the host injects a `LocalKvRuntimePort`; configuration remains disabled until explicitly enabled.
+
+## Packaging rules
+
+All release packages use the same version.
+
+- `ds4-context-core` has no runtime SDK dependency.
+- `ds4-context-engine` contains the Pi SDK boundary and depends exactly on matching core.
+- `ds4-context-reference-adapter` contains no Pi SDK and depends exactly on matching core.
+- Adapter source is not bundled into the core tarball; core source is not bundled into adapter tarballs.
+- Core and reference TypeScript compile to ESM JavaScript and declarations before packing.
+- Publish core first, then reference, then Pi.
+
+Use:
+
+```bash
+npm run check
+npm run pack:check
+npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
+npm pack --dry-run
+```
+
+## Limitations and rollback
+
+The reference adapter is intentionally synchronous-history/callback-completion infrastructure: it does not implement streaming, native compaction, branch editing, provider continuation, or a built-in provider-specific KV cache. Its optional KV path requires a host-owned port that retains handles and transport. Unsupported features remain disabled and visible rather than emulated.
+
+Removing the reference package does not affect Pi or core. A runtime can roll back its adapter by stopping it and returning to native history/context behavior; no canonical migration or SQLite downgrade is required. Deleting adapter caches is safe and produces a transparent full replay. Never delete the runtime-owned JSONL file as part of DS4 rollback.

package/docs/STORAGE.md

@@ -8,7 +8,11 @@ Pi's session JSONL is canonical for conversations and live project files are can
 /context rebuild-index
 ```
 
-The extension never edits or rewrites Pi JSONL or project source files. Manual memory/pin commands append versioned Pi `CustomEntry` records through Pi's official `appendEntry()` API.
+The extension never edits or rewrites Pi JSONL or project source files. Manual memory/pin commands and learned-ranking feedback append versioned classified Pi `CustomEntry` records through Pi's official `appendEntry()` API.
+
+The M19 non-Pi reference adapter owns a separate `ds4-runtime-session-v1` JSONL source selected by its host runtime. Its header binds runtime/session identity and the exact canonical project root; following records contain provenance-checked canonical messages. DS4 snapshots and capability diagnostics are disposable. `createReferenceHistory()` refuses overwrite, append uses a dedicated provenance-checked operation, files are mode `0600` where supported, and rebuild never edits this runtime-owned canonical file. Reference JSONL is not imported into Pi or `context.db`.
+
+M20 local KV state is entirely runtime-owned and volatile. Core returns only an in-memory eligibility fingerprint to the runtime port; it has no cache-handle field or serialization API. Prefixes, fingerprints, handles and provider outputs are absent from Pi/reference JSONL, Context Manifests, ranking artifacts and every SQLite table. Aggregate hit/miss/prefill counters live only on the adapter controller, and a restart safely resets them with the runtime cache. M20 adds no database migration.
 
 ## Session index
 
@@ -45,19 +49,33 @@ Malformed newline-terminated records are skipped consistently and counted. A val
 
 ## Project knowledge index
 
-Schema v7 adds `project_states`, `project_files`, `project_snippets`, and `project_snippets_fts`. The state row records canonical root plus Git branch/HEAD/dirty paths. Current file rows store SHA-256, size, mtime, language, indexed Git HEAD, tracked/modified state, and lifecycle. Snippet rows store the immutable file hash, line range, source text, heuristic symbols, estimate, and stale bit.
+Schema v7 adds `project_states`, `project_files`, `project_snippets`, and `project_snippets_fts`. The state row records canonical root plus Git branch/HEAD/dirty paths. Current file rows store SHA-256, size, mtime, language, indexed Git HEAD, tracked/modified state and lifecycle. Snippet rows store the immutable file hash, line range, source text, estimate and stale bit.
 
-Changed hashes never overwrite old snippet rows silently: prior rows become stale and new hash-derived snippet IDs become current. Deleted, renamed, oversized, binary, symlinked, or newly sensitive files similarly invalidate prior snippets. Exact and FTS queries join current files and require `stale = 0`; stale FTS rows may remain locally for derived-history diagnostics but cannot enter context.
+Schema v12 extends snippet rows with text/symbol chunk kind, parser ID, stable symbol ID, simple and qualified names, kind, signature, parent, imports and references. The exact-name indexes and all parser output are disposable. Built-in TypeScript/JavaScript/Python/Go structural parsing has no native dependency; unsupported, invalid or declaration-free files retain bounded v7 text windows.
+
+Changed hashes never overwrite old snippet rows silently: only the changed path's prior rows become stale and new hash-derived snippet/symbol IDs become current. Unrelated files preserve their rows and IDs. Deleted, renamed, oversized, binary, symlinked or newly sensitive files similarly invalidate prior snippets. Exact path/symbol and FTS queries join current files and require `stale = 0`; stale FTS rows may remain locally for derived-history diagnostics but cannot enter context.
 
 Project source text is duplicated in SQLite only to provide local FTS and bounded snippet injection. Deleting the database loses no source truth. `/context rebuild-index` clears/rebuilds current projections from trusted live files. No project table is read or written while Pi reports the project untrusted.
 
+## Derived embedding index
+
+Schema v13 adds `derived_embeddings` for opt-in M16 historical/project vectors. The compound key contains source kind/scope/key/hash, chunking version, embedding provider/model and dimensions. Rows store only source grouping metadata, numeric vector JSON and indexing time; no query text, copied evidence text, provider response ID or remote handle is added.
+
+Current source keys and hashes prune obsolete vectors without touching unrelated sources. Provider/model/dimension profiles coexist, so changing one profile does not rewrite another. Query vectors live only in a bounded volatile cache. The table is ignored when semantic retrieval is disabled and can always be recreated from Pi JSONL plus trusted live project files and the configured runtime embedding port.
+
+Remote embedding text is privacy-filtered before the port is called. `local-only` values are omitted entirely and allowed values are secret-redacted. These rules govern transport; SQLite remains a local disposable projection.
+
 ## Memory and pin event projection
 
 Schema v9 adds append-only `memory_mutations` and `pin_mutations`, each keyed to the canonical scoped `entries.entry_key` for its Pi custom entry. Mutation payloads describe immutable add, explicit supersede, or lifecycle status operations. `entry_order` preserves causal order when several Pi entries share one millisecond timestamp.
 
 `memory_items`, `memory_sources`, `memory_fts`, and `pins` are materialized transactionally by replaying every known mutation. Materialized memory records retain normalized keys, origin session, source entries, optional privacy classification, active/superseded/invalid/expired status and immutable replacement links. Pins retain session/branch/project scope, creation leaf, optional classification, source entry/file and active/superseded/deleted lifecycle. Classification lives in canonical mutation JSON and derived `metadata_json`; M10 required no schema migration beyond v9.
 
-Before replay, the current session's mutation rows are replaced from its complete Pi entry tree. Other indexed sessions remain available, enabling trusted project-scope state across sessions. Deleting the database loses no canonical mutation; reopening each source session recreates its projection. Unbacked legacy pre-v9 materialized rows are inspectable immediately after migration but are not treated as canonical during a later full replay.
+Before replay, the current session's mutation rows are replaced from its complete Pi entry tree. Other indexed sessions remain available, preserving the 0.1 project-scope behavior.
+
+Schema v15 adds `project_memory_sessions`, `project_memory_source_exclusions`, and mutation creation-parent columns. When `memory.crossSession` is opted in, DS4 enumerates at most `memory.maxProjectSessions` sibling Pi JSONL files, validates each header against the exact trusted canonical project path, indexes only changed suffixes, and materializes their explicit mutations. Source rows retain header/checkpoint hashes, offsets, record/mutation counts, malformed-line counts, status and bounded error text. Exclusions are local derived policy; mutation content remains only in canonical JSONL and existing local projection tables.
+
+A missing, moved, truncated, identity-mismatched or corrupt sibling source stops contributing unverifiable project-scoped mutations without discarding its isolated session-scoped projection. The active session retains its last transactional projection if an auxiliary cross-session refresh fails. Restored files rebuild deterministically. Deleting the database loses no canonical mutation; source discovery recreates the complete project projection without opening every source session. Unbacked legacy pre-v9 materialized rows are inspectable immediately after migration but are not treated as canonical during a later full replay.
 
 Custom entries have empty lexical search text and never enter Pi context directly. The managed planner creates bounded, source-labelled synthetic pin/memory messages. Context Manifests store only metadata and hashes, never content/claims.
 
@@ -67,9 +85,21 @@ Schema v10 extends `context_manifests` and `token_calibration` with separate unc
 
 Calibration rows are derived telemetry, isolated by exact provider/model and bounded to the latest configured window at read time. The runtime recomputes median/MAD outlier filtering deterministically; no learned model or mutable provider state is stored. Deleting the database loses calibration and cache history but never session content. Ephemeral sessions and configurations that disable manifest persistence keep only a bounded in-memory window.
 
+## Context quality samples
+
+Schema v11 adds bounded `context_quality_samples` for opt-in M14 measurement. Rows contain only metric/corpus/planner/profile versions, source-kind/token/budget/decision counts, normalized outcome labels and separate planning duration. Prompt text, evidence text, paths, memory claims, provider payloads, response IDs and raw source IDs are never stored. Invalid or corrupt rows are skipped during aggregation.
+
+The table is a disposable replay projection. Deleting it loses no canonical state; replaying the versioned sanitized corpus recreates byte-stable non-timing aggregates. Runtime samples without expected-evidence labels remain explicitly unlabeled. See [`CONTEXT_QUALITY.md`](CONTEXT_QUALITY.md).
+
+## Learned-ranking labels and model artifact
+
+M18 adds no SQLite table. Classified `ds4-context-ranking-feedback-v1` Pi custom entries are canonical and contain only version/hash/label metadata plus ten bounded numeric features. Stable repository and candidate identities are SHA-256 hashes; raw prompt, evidence, claim, path and symbol text are absent. Sanitized replay labels use the same entry schema and explicitly identify their label source.
+
+The trained `ds4-context/ranking-model.json` file is a disposable private artifact containing schema/algorithm versions, bounded weights, aggregate training counts, optional promotion-gate metrics and a stable-payload checksum. Deleting or corrupting it restores static ranking; canonical labels remain available for local retraining. Shadow comparison stores only aggregate disagreement in Context Manifests. No label, feature vector or candidate ID is copied into a manifest. See [`LEARNED_RANKING.md`](LEARNED_RANKING.md).
+
 ## Native continuation state
 
-M12 adds no SQLite migration or continuation table; schema remains v10. The active process keeps only deterministic SHA-256 hashes of the previous full request items and serialized response items, a hash of non-input request options, completion time, and the minimum provider response handle needed for `previous_response_id`.
+M12 adds no continuation table. It was introduced at schema v10; M14 adds quality samples in v11, M15 adds project-symbol columns/indexes in v12, and M16 adds only disposable vectors in v13. The active process keeps only deterministic SHA-256 hashes of the previous full request items and serialized response items, a hash of non-input request options, completion time, and the minimum provider response handle needed for `previous_response_id`.
 
 The volatile state is cleared on lifecycle/model/branch/compaction boundaries and is not reconstructed on resume. The first request after a cold start is therefore always the complete managed replay. Pi may persist its normal `AssistantMessage.responseId` in canonical JSONL, but DS4 does not create a custom entry, copy that ID into SQLite/manifest/logs, or depend on it for recovery.
 
@@ -83,7 +113,7 @@ A full index rebuild replays all message entries, recreates missing qualifying o
 
 ## Context manifests
 
-For persisted sessions, each `context` hook stores a metadata-only manifest containing token counts, session/project/pin/memory source and atomic-group IDs, inclusion/exclusion reasons, classifications and scores, original/selected counts, exact-model override/calibration/adaptive budgets, model-switch/cache disposition, provider destination/allow names, privacy counters, optional continuation mode/item counts/retry reasons, project revision/hash/line references, tool names, a SHA-256 prompt hash, and planner/policy versions. Prompt text, message text, pin content, memory claims, project snippet text, tool arguments, image data, rendered provider payloads, and provider response/conversation IDs are not stored in the manifest.
+For persisted sessions, each `context` hook stores a metadata-only manifest containing token counts, session/project/pin/memory source and atomic-group IDs, inclusion/exclusion reasons, classifications and scores, original/selected counts, exact-model override/calibration/adaptive budgets, aggregate learned-ranking status/disagreement, model-switch/cache disposition, provider destination/allow names, privacy counters, optional continuation mode/item counts/retry reasons, project revision/hash/line references, tool names, a SHA-256 prompt hash, and planner/policy versions. Prompt text, message text, pin content, memory claims, project snippet text, tool arguments, image data, rendered provider payloads, and provider response/conversation IDs are not stored in the manifest.
 
 `before_provider_request` updates the pending manifest with final-check/redaction counters but never the provider payload. The following finalized assistant response updates it with uncached input, cache-read, cache-write and total provider input usage, then adds at most one exact-model calibration sample. Ephemeral sessions retain this information only in memory.
 
@@ -99,4 +129,4 @@ Schema-v2 `CompactionEntry.details.ds4ContextEngine` records the active/segment
 
 A full rebuild does not blindly delete unchanged entries. It upserts all observed entries, marks them in a temporary seen-set, and removes only stale rows. This preserves foreign-key provenance for unchanged source entries. FTS rows and checkpoint state update in the same transaction.
 
-Session reconciliation is transactional. Memory/pin mutation replacement and full materialization are one transaction. Each changed project file is also replaced transactionally with its snippets and FTS rows; artifact object/reference metadata and project deletion batches are atomic. A filesystem artifact write precedes its metadata transaction, so an interrupted metadata write may leave only an unreferenced content-addressed cache file; canonical JSONL remains sufficient for recovery. If parsing, validation, or SQLite writing fails, the prior derived state remains available. Artifact/project failures contribute no replacement/snippets; planner failures discard all synthetic evidence; Pi continues with its native context.
+Session reconciliation is transactional. Memory/pin mutation replacement, checkpoint update, source exclusion and full materialization each occur under the shared write coordinator. Each quality upsert and bounded-retention prune share one transaction; quality failures do not affect manifests or planning. Each changed project file is also replaced transactionally with its snippets and FTS rows; embedding upserts and canonical-source pruning are transactional; artifact object/reference metadata and project deletion batches are atomic. A filesystem artifact write precedes its metadata transaction, so an interrupted metadata write may leave only an unreferenced content-addressed cache file; canonical JSONL remains sufficient for recovery. If parsing, validation, or SQLite writing fails, the prior derived state remains available. Artifact/project failures contribute no replacement/snippets; planner failures discard all synthetic evidence; Pi continues with its native context.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.1.2",
+  "version": "0.2.0-beta.2",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -29,6 +29,8 @@
   "files": [
     "src",
     "docs",
+    "quality",
+    "scripts/compare-context-quality.mjs",
     "README.md",
     "LICENSE"
   ],
@@ -37,12 +39,15 @@
   },
   "scripts": {
     "build:core": "npm run build --workspace ds4-context-core",
-    "typecheck": "npm run build:core && tsc --noEmit",
-    "test": "npm run build:core && vitest run",
-    "test:watch": "npm run build:core && vitest",
-    "check": "npm run build:core && tsc --noEmit && vitest run",
+    "build:reference-adapter": "npm run build --workspace ds4-context-reference-adapter",
+    "build:adapters": "npm run build:reference-adapter",
+    "typecheck": "npm run build:core && npm run build:adapters && tsc --noEmit",
+    "test": "npm run build:core && npm run build:adapters && vitest run",
+    "test:watch": "npm run build:core && npm run build:adapters && vitest",
+    "check": "npm run build:core && npm run build:adapters && tsc --noEmit && vitest run",
+    "quality:compare": "node scripts/compare-context-quality.mjs",
     "pack:check": "node scripts/verify-packages.mjs",
-    "prepare": "npm run build:core"
+    "prepare": "npm run build:core && npm run build:adapters"
   },
   "pi": {
     "extensions": [
@@ -50,7 +55,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.1.2"
+    "ds4-context-core": "0.2.0-beta.2"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",