ds4-context-engine 0.1.1 → 0.2.0-beta.1

package/docs/LEARNED_RANKING.md

@@ -0,0 +1,90 @@
+# Learned Ranking
+
+M18 adds an optional metadata-only linear ranker while preserving the deterministic static ranker as the universal fallback.
+
+## Modes
+
+```json
+{
+  "ranking": {
+    "mode": "shadow",
+    "modelPath": "ds4-context/ranking-model.json",
+    "minimumTrainingSamples": 20,
+    "maxTrainingSamples": 10000,
+    "maxLatencyMs": 10
+  }
+}
+```
+
+- `off` does not load or evaluate a learned model.
+- `shadow` evaluates a valid model, records only aggregate rank disagreement, and keeps static selection unchanged.
+- `active` changes supplemental-candidate ordering only when the checksummed artifact contains a successful promotion report. Missing, incompatible, corrupt, unpromoted, or regressing artifacts fall back to static ranking.
+
+`off` is the upgrade default. Privacy filtering, mandatory pins, current-request retention, category budgets, atomic groups, and hard input limits remain authoritative in every mode.
+
+## Bounded feature schema
+
+`ranking-features-v1` contains only a source-kind enum and ten finite normalized numbers:
+
+- static score;
+- exact-match score;
+- FTS score;
+- vector score;
+- recency;
+- active-branch relation;
+- symbol relation;
+- classification eligibility;
+- token cost;
+- prior selection outcome.
+
+All numbers are bounded to `[0, 1]`. Candidate text, excerpts, prompts, file paths, symbols, claims, provider payloads, and credentials are not model features. Stable candidate and repository identities are SHA-256 hashes in canonical labels.
+
+## Canonical labels and local training
+
+Explicit feedback is appended through Pi as a classified `ds4-context-ranking-feedback-v1` custom entry:
+
+```text
+/context ranking feedback useful CANDIDATE_ID --classification internal
+/context ranking feedback irrelevant CANDIDATE_ID --classification local-only
+```
+
+Only candidates from the latest managed context are accepted. A feedback entry contains schema/version, feedback ID, timestamp, classification, label source, label, two hashes, and the bounded feature vector. It contains no candidate text. Sanitized replay labels use the same schema with `labelSource: "replay"`.
+
+Train a local shadow artifact from the active canonical Pi branch:
+
+```text
+/context ranking train
+```
+
+Training is deterministic for the same unique labels and timestamp, requires both useful and irrelevant examples, and is bounded by `minimumTrainingSamples` and `maxTrainingSamples`. The artifact is written with mode `0600` where supported. Training never calls a provider and does not make the resulting artifact active.
+
+## Artifact integrity and deterministic inference
+
+`ranking-model.json` uses schema version `1`, feature version `ranking-features-v1`, algorithm `bounded-linear-centroid-v1`, bounded weights, training counts, a deterministic model ID, and a SHA-256 checksum over its stable JSON payload. Inference uses rounded finite arithmetic and resolves equal scores by static score and then candidate ID.
+
+The model artifact is local derived state. Deleting it preserves Pi JSONL labels and restores static ranking. A newly trained artifact can be reconstructed from canonical labels.
+
+## Promotion gate
+
+`evaluateRankingPromotion()` in `ds4-context-core/ranking/learned-ranker` evaluates sanitized held-out fixtures. `withRankingPromotion()` seals the resulting report into a new checksummed artifact. Promotion succeeds only when all of these conditions hold:
+
+- primary quality score improves;
+- exact-identifier recall does not regress;
+- privacy violations do not increase;
+- atomicity failures do not increase;
+- overflow does not increase;
+- measured p95 latency stays within budget;
+- repeated inference yields identical ordering;
+- the configured minimum number of held-out repositories is represented.
+
+An artifact without an eligible report can run in shadow mode but cannot alter context in active mode.
+
+## Diagnostics
+
+```text
+/context ranking
+/context health
+/context manifest
+```
+
+Ranking diagnostics include mode/status, versions, model identity, promotion state, label counts, malformed/duplicate counts, candidate count, top-rank disagreement, pairwise disagreements, mean rank shift, inference duration, and fallback reason. Shadow diagnostics and Context Manifests contain aggregate comparison only—never candidate IDs, labels, features, or text.

package/docs/MEMORY_AND_PINS.md

@@ -1,6 +1,6 @@
 # Memory and Persistent Pins
 
-M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical.
+M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical. M17 optionally reconstructs explicit project-scoped mutations from sibling Pi sessions for the same trusted canonical project.
 
 ## Authority and scope
 
@@ -30,9 +30,12 @@ Pins remain subordinate to system/developer instructions. Memory tells the model
 /context memory supersede MEMORY_ID [--classification LEVEL] [--source ID,ID] <new claim>
 /context memory invalidate MEMORY_ID [reason]
 /context memory expire MEMORY_ID [reason]
+/context memory sources
+/context memory exclude SESSION_ID [reason]
+/context memory include SESSION_ID
 ```
 
-Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust.
+Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust. Source exclusion affects only project-scoped contributions and persists until `include`; the active session cannot be newly excluded but can restore a prior exclusion.
 
 Mutations are manual-first. Repeating the same normalized pin/claim returns its existing ID without appending another entry.
 
@@ -60,7 +63,7 @@ No row is silently overwritten. SQLite mutation and materialized tables are disp
 3. replays all known mutations in timestamp + canonical entry order;
 4. rebuilds memory, pin, source, lifecycle, and FTS rows transactionally.
 
-Deleting `context.db` and reopening the canonical source session reconstructs its state. Project items from other sessions reappear when those canonical sessions are replayed.
+Deleting `context.db` and reopening the canonical source session reconstructs its state. With `memory.crossSession: true`, DS4 discovers bounded sibling `.jsonl` files, accepts only headers whose `cwd` resolves to the exact trusted canonical project identity, incrementally indexes each source, and reconstructs project items without opening every session manually. No claim is extracted from ordinary conversation text.
 
 ## Supersession and contradiction handling
 
@@ -80,7 +83,7 @@ A conflicting `add` is rejected with the IDs involved. The user must issue expli
   "Package export mode defaults to SingleFile."
 ```
 
-The old item remains stored as `superseded` and points to the new active item. During replay, concurrent active records with the same key are both preserved; the deterministic later record is marked `invalid` with a conflict reason rather than replacing the earlier one.
+The old item remains stored as `superseded` and points to the new active item. During replay, records are ordered by timestamp, source session ID, source entry order, mutation ID and mutation entry key. Concurrent active records with the same key are both preserved; the deterministic later record is marked `invalid` with a conflict reason rather than replacing the earlier one. Selected items retain source-session file/ID, mutation entry, creation branch leaf, evidence entries, classification, supersession and contradiction IDs.
 
 Pins use the same immutable replacement pattern through `--supersedes`. `/context unpin` records a soft `deleted` lifecycle mutation.
 
@@ -126,7 +129,7 @@ User text is JSON-quoted. Context Manifests contain IDs, scope, classification,
 
 M10 stores an optional classification in the canonical mutation. Before a remote call, a prohibited pin/memory is omitted as a whole and recorded only by ID/classification/reason. Explicit classification cannot be downgraded by markers inside its content. See [`PRIVACY.md`](PRIVACY.md).
 
-## SQLite schema v9
+## SQLite schema v9 and v15
 
 Schema v9 extends materialized `memory_items` and `pins` with:
 
@@ -136,6 +139,8 @@ Schema v9 extends materialized `memory_items` and `pins` with:
 
 `memory_mutations` and `pin_mutations` reference canonical indexed Pi custom entries. `memory_sources` retains exact scoped entry provenance. `memory_fts` is rebuilt transactionally.
 
+Schema v15 adds source-branch provenance, per-session project-memory checkpoints, source status and explicit source exclusions. These rows are derived and disposable. Missing, truncated, moved, identity-mismatched or corrupt sibling files are marked unavailable and their unverifiable project-scoped mutations stop contributing; session-scoped state remains isolated and the active session keeps its last transactional projection on refresh failure. Restoration or database deletion causes deterministic replay from JSONL.
+
 Migration preserves legacy materialized rows for inspection. Because they have no canonical mutation entry, a later full replay may discard them; new operations are always event-sourced.
 
 ## Diagnostics
@@ -148,6 +153,7 @@ Migration preserves legacy materialized rows for inspection. Because they have n
 /context excluded
 /context pins
 /context memory
+/context memory sources
 /context health
 /context rebuild-index
 ```

package/docs/PORTABLE_CORE.md

@@ -1,6 +1,6 @@
 # Portable Core
 
-M13 extracts the runtime-neutral implementation into the independently buildable `ds4-context-core` workspace package. The root `ds4-context-engine` package remains the Pi integration.
+M13 extracts the runtime-neutral implementation into the independently buildable `ds4-context-core` workspace package. The root `ds4-context-engine` package remains the Pi integration. M19 adds the versioned adapter contract/conformance kit in core and a separately compiled non-Pi callback/JSONL reference adapter.
 
 ## Dependency rule
 
@@ -14,7 +14,7 @@ ds4-context-core
 
 Core never imports an adapter or runtime SDK. The Pi adapter imports core through its public ESM exports.
 
-The core may use Node.js standard-library facilities such as `node:sqlite`, filesystem APIs and cryptographic hashing. Portable means independent of Pi and reusable by another Node-based agent runtime; it does not mean browser-compatible.
+The core may use Node.js standard-library facilities such as `node:sqlite`, filesystem APIs and cryptographic hashing. Portable means independent of every agent runtime SDK and reusable by another Node-based agent runtime; it does not mean browser-compatible.
 
 ## Package contents
 
@@ -44,7 +44,7 @@ The root adapter owns all Pi-specific behavior:
 
 ## Adapter responsibilities
 
-A future runtime adapter must:
+Every `runtime-adapter-v1` implementation must:
 
 1. preserve its runtime's canonical history and expose stable source identifiers;
 2. convert native messages to DS4 canonical messages without losing tool-call/result atomicity;
@@ -57,14 +57,18 @@ A future runtime adapter must:
 9. discard or rebuild SQLite and artifact projections safely;
 10. fall back to native runtime behavior when operational integration fails.
 
+The executable contract, independent capability negotiation and framework-neutral seven-case conformance runner are exported from `ds4-context-core/adapter/runtime-adapter` and `ds4-context-core/adapter/conformance`. See [Runtime Adapter Kit](RUNTIME_ADAPTER_KIT.md).
+
 ## Build and exports
 
 ```bash
 npm run build:core
+npm run build:adapters
 npm run typecheck
 npm test
 npm run pack:check
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 ```
 
 TypeScript sources compile to `packages/core/dist` as ESM JavaScript, source maps and declaration files. The npm package exports a top-level API and fine-grained subpaths such as:
@@ -74,11 +78,11 @@ import { calculateContextBudget } from "ds4-context-core";
 import { planManagedContext } from "ds4-context-core/planner/context-planner";
 ```
 
-The Pi package declares an exact same-release dependency on `ds4-context-core`. Release order is therefore core first, adapter second. `npm run pack:check` verifies both tarball inventories, installs them together in a clean temporary consumer, probes core ESM exports and starts the packaged adapter with isolated Pi RPC state. See [Releasing DS4](RELEASING.md) for the publication checklist.
+The Pi and reference packages declare exact same-release dependencies on `ds4-context-core`. Release order is core, reference adapter, then Pi adapter. `npm run pack:check` verifies all three tarball inventories, installs them together in a clean temporary consumer, probes core ESM exports, runs compiled reference conformance and starts the packaged Pi adapter with isolated RPC state. See [Releasing DS4](RELEASING.md) for the publication checklist.
 
 ## Enforcement
 
-`tests/unit/portable-core-boundary.test.ts` recursively rejects Pi SDK and adapter imports from core source, then imports the compiled package and exercises portable model/budget policy. Existing integration tests consume core through package exports, so the Pi adapter is tested across the actual package boundary.
+`tests/unit/portable-core-boundary.test.ts` recursively rejects Pi SDK and adapter imports from core source, then imports the compiled package and exercises portable model/budget policy. Runtime-adapter unit tests validate capability isolation and canonical tool groups; the reference package passes the shared conformance runner. Existing integration tests consume core through package exports, so the Pi adapter is tested across the actual package boundary.
 
 ## State guarantees
 

package/docs/PROJECT_KNOWLEDGE.md

@@ -1,6 +1,6 @@
 # Project Knowledge
 
-M7 indexes trusted project files as a disposable SQLite projection and injects only task-relevant, hash-current snippets. Live files remain canonical.
+M7 indexes trusted project files as a disposable SQLite projection and injects only task-relevant, hash-current snippets. M15 adds runtime-neutral structural symbol parsing and exact symbol lookup. M16 optionally adds vectors over those structural/text chunks while preserving exact and FTS retrieval. Live files remain canonical.
 
 ## Trust boundary
 
@@ -16,7 +16,9 @@ This is independent of global `project.enabled`: a global setting cannot overrid
 
 ## Discovery and exclusions
 
-When Git is available, discovery combines tracked and non-ignored untracked files under `ctx.cwd`. Outside Git, DS4 walks the project tree deterministically. It never follows symbolic links and rejects paths outside the canonical project root.
+When Git is available, discovery combines tracked and non-ignored untracked files under `ctx.cwd`. Outside Git, DS4 walks the project tree deterministically. Discovery stops as soon as the configured file bound is reached and also bounds visited directories. It never follows symbolic links and rejects paths outside the canonical project root.
+
+DS4 skips project indexing when `ctx.cwd` is the filesystem root or the user's home directory. This prevents a normal Pi launch from recursively scanning an entire drive or user profile; start Pi inside the intended project directory to enable project knowledge.
 
 Default bounds:
 
@@ -34,29 +36,37 @@ DS4 excludes VCS metadata, `.pi`, dependencies, build outputs, caches, virtual e
 
 ## File and snippet index
 
-SQLite schema v7 stores:
+SQLite schema v7 introduced:
 
 - `project_states`: canonical project path, Git root/branch/HEAD, dirty flag, changed paths, index time;
 - `project_files`: path, SHA-256, bytes, mtime, language, indexed Git HEAD, tracked/modified state, lifecycle;
-- `project_snippets`: immutable file hash, line range, source, heuristic declarations, token estimate, stale flag;
+- `project_snippets`: immutable file hash, line range, source, token estimate and stale flag;
 - `project_snippets_fts`: FTS5 content/path/symbol index.
 
-Files are split into overlapping line windows. Heuristic symbols cover class, interface, enum, namespace, record, struct, trait, type, function, common language declarations, and SQL objects. No parser or LLM is needed.
+Schema v12 adds derived structural metadata to each snippet: chunk kind, parser version, symbol ID/name/kind, qualified name, signature, parent symbol, imports and references. Exact-name indexes remain disposable and rebuild from live project files. Schema v13 stores separately keyed derived vectors; a file-hash change prunes only vectors for no-longer-current chunks, while unrelated source and embedding-profile rows remain intact.
+
+`ds4-context-core` exposes a `SymbolParser` interface. The built-in `regex-structural-v1` parser has deterministic coverage for TypeScript, JavaScript, Python and Go without native dependencies. Optional parser adapters run first through `SymbolParserChain`; an unavailable or throwing adapter falls through to the built-in parser. Unsupported languages, malformed delimiter structure, supported files with no declarations and files carrying explicit DS4 classification spans retain the M7 overlapping text-window fallback. Keeping marked spans intact ensures M10 privacy enforcement sees the same boundaries as the 0.1 index.
+
+Structural chunks follow declaration boundaries and retain signatures, parent relationships, imports/references and exact line ranges. Large declarations are split into bounded overlapping subchunks while sharing one symbol identity. Symbol IDs are SHA-256 values derived from canonical project identity, relative path, file hash, structural range, symbol kind and qualified name. Text fallback IDs retain their deterministic project/path/hash/range derivation.
 
-A full or incremental sync compares size, mtime, Git revision, and modified state. Source is re-read and SHA-256 hashed whenever metadata changes. Old snippets are marked `stale`; they remain inspectable but all exact and FTS queries require `stale = 0` and a current file row.
+A full or incremental sync compares size, mtime, Git revision and modified state. Source is re-read and SHA-256 hashed whenever metadata changes. Replacing one file marks only that path's prior snippets stale, then transactionally inserts the new file-hash rows and FTS entries; unrelated file rows keep their IDs. Stale rows remain inspectable, but every exact and FTS query requires `stale = 0` and a current file row.
 
 ## Retrieval and ranking
 
-The same current-request `TaskDescriptor` used for historical retrieval supplies file paths, symbols, identifiers, errors, quoted phrases, technologies, and keywords. Candidate generation combines case-sensitive literal `instr()` search with escaped FTS5 terms.
+The same current-request `TaskDescriptor` used for historical retrieval supplies file paths, symbols, identifiers, errors, quoted phrases, technologies and keywords. Candidate generation queries literal exact path/basename and exact qualified/simple symbol indexes before generic literal content and escaped FTS5 candidates. Exact lookup does not treat comment, string or reference text as a declaration.
+
+When `retrieval.semantic` is enabled, the current embedding profile also contributes a bounded cosine-ranked pool. Deterministic reciprocal-rank fusion combines lexical and vector ranks; exact paths and symbols retain higher score tiers. Source vectors are populated during index sync and query vectors use a bounded volatile cache. Vector/model/privacy failures return the exact/FTS candidates unchanged. See [`HYBRID_RETRIEVAL.md`](HYBRID_RETRIEVAL.md).
 
 Deterministic ranking prioritizes:
 
 ```text
-exact project-relative path  140
-exact basename               125
-declared symbol              115
+exact qualified symbol       190
+exact project-relative path  180
+exact simple symbol          170
+exact basename               150
+declared fallback symbol     115
 exact phrase                  90
-symbol text                   85
+symbol text                   65
 FTS match                     60..20
 working-tree change           10
 tracked source                 3
@@ -121,17 +131,19 @@ Source text is not copied into the Context Manifest or structured logs. It remai
 
 ## Fail-open behavior
 
-Project discovery, Git, indexing, FTS, validation, and retrieval errors are isolated from session indexing and context planning. A project failure records local diagnostics and contributes no snippets; Pi and historical retrieval continue. SQLite startup failure retains the existing runtime-wide Pi fallback.
+Project discovery, Git, optional parser adapters, indexing, FTS, validation and retrieval errors are isolated from session indexing and context planning. Parser adapter failure uses deterministic regex fallback; unsupported or invalid source uses bounded text chunks. A project subsystem failure records local diagnostics and contributes no snippets; Pi and historical retrieval continue. SQLite startup failure retains the existing runtime-wide Pi fallback.
+
+## Verification and benchmark
 
-## Benchmark
+`quality/symbol-corpus-v1.json` is a synthetic, versioned TypeScript/JavaScript/Python/Go corpus with expected and forbidden declarations. Unit coverage verifies that structural parsing produces fewer false symbol matches than the preserved 0.1 regex extractor. Integration coverage verifies exact qualified/path lookup, metadata fields, stable IDs, unsupported/invalid fallback, changed-file invalidation, unrelated-file stability, trust/exclusion behavior and live-hash validation.
 
 `tests/benchmarks/project-knowledge.bench.ts` creates and indexes 5,000 source files, then executes exact path/symbol plus FTS retrieval with live hash validation:
 
 ```text
-mean  12.28 ms
-p75   13.12 ms
-p99   21.04 ms
-max   21.04 ms
+mean  29.99 ms
+p75   31.71 ms
+p99   37.21 ms
+max   37.21 ms
 ```
 
 Command:

package/docs/RELEASING.md

@@ -1,21 +1,22 @@
 # Releasing DS4
 
-DS4 publishes two packages with the same version:
+DS4 publishes three packages with the same version:
 
-1. `ds4-context-core`, the runtime-neutral compiled package;
-2. `ds4-context-engine`, the Pi adapter.
+1. `ds4-context-core`, the runtime-neutral compiled package and adapter kit;
+2. `ds4-context-reference-adapter`, the non-Pi callback/JSONL reference implementation;
+3. `ds4-context-engine`, the Pi adapter.
 
-The adapter has an exact dependency on the matching core version, so the core package must always be published first.
+Both adapters have an exact dependency on the matching core version, so core must be published first.
 
 ## Prerequisites
 
 - use a clean `main` checkout synchronized with `origin/main`;
-- use a supported Node.js release and npm account authorized for both package names;
-- verify that both `package.json` files use the intended version;
-- verify that `ds4-context-engine` depends exactly on that version of `ds4-context-core`;
+- use a supported Node.js release and npm account authorized for all three package names;
+- verify that all three `package.json` files use the intended version;
+- verify that both adapters depend exactly on that version of `ds4-context-core`;
 - do not include session data, `.pi` state, databases, credentials, or provider payloads.
 
-The automated package check enforces matching versions, the exact core dependency, bounded tarball inventories, clean consumer installation, core ESM exports, and packaged Pi extension startup through isolated RPC state.
+The automated package check enforces matching versions, exact core dependencies, runtime-SDK isolation, bounded tarball inventories, clean consumer installation, core ESM exports, compiled reference-adapter conformance, and packaged Pi extension startup through isolated RPC state.
 
 ## Validate
 
@@ -29,26 +30,29 @@ git status --short
 
 CI runs the same checks on the minimum supported Node.js version and the current Node.js LTS line. `npm run pack:check` uses a temporary directory and removes it when complete. Set `DS4_KEEP_PACK_TMP=1` only when diagnosing a failed package check.
 
-Review both public tarballs before publishing:
+Review all public tarballs before publishing:
 
 ```bash
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 npm pack --dry-run
 ```
 
 ## Version
 
-Keep the root package, core workspace, and exact adapter dependency synchronized. For a future version stored in `$VERSION`:
+Keep the root package, both workspaces, and exact adapter dependencies synchronized. For a future version stored in `$VERSION`:
 
 ```bash
 npm pkg set version="$VERSION"
 npm pkg set version="$VERSION" --workspace ds4-context-core
+npm pkg set version="$VERSION" --workspace ds4-context-reference-adapter
 npm pkg set dependencies.ds4-context-core="$VERSION"
+npm pkg set dependencies.ds4-context-core="$VERSION" --workspace ds4-context-reference-adapter
 npm install --package-lock-only
 npm run pack:check
 ```
 
-Review `package.json`, `packages/core/package.json`, and `package-lock.json` before committing the release change.
+Review `package.json`, both workspace package manifests, and `package-lock.json` before committing the release change.
 
 ## Publish
 
@@ -57,12 +61,13 @@ Authenticate with npm, verify the active account, and publish in dependency orde
 ```bash
 npm whoami
 npm publish --workspace ds4-context-core --access public
+npm publish --workspace ds4-context-reference-adapter --access public
 npm publish --access public
 ```
 
-If core succeeds but adapter publication fails, fix the adapter release and retry it with the same version. Do not rewrite or unpublish a valid core release merely to make the two commands appear atomic.
+If core succeeds but an adapter publication fails, fix that adapter release and retry it with the same version. Do not rewrite or unpublish a valid core release merely to make the commands appear atomic.
 
-After both registry packages are available:
+After all registry packages are available:
 
 1. install them in a fresh temporary project and rerun the package smoke check if registry propagation was delayed;
 2. create and push the signed or annotated `v$VERSION` tag;

package/docs/RETRIEVAL.md

@@ -15,7 +15,7 @@ M6 recovers original session evidence that Pi compaction removed from the active
 9. Deduplicate normalized identical text, preferring the higher-ranked/newer source.
 10. Build individually bounded evidence messages, enforce the active provider privacy policy, and let the managed planner fit allowed groups after recent turns but before summaries.
 
-No LLM is called during retrieval. `retrieval.semantic: true` produces a diagnostic warning but does not enable embeddings in M6.
+No chat LLM is called during retrieval. Since M16, `retrieval.semantic: true` adds opt-in vectors through the runtime-neutral `EmbeddingPort`; exact and FTS retrieval remain authoritative and any embedding failure falls back to lexical results. See [Hybrid Semantic Retrieval](HYBRID_RETRIEVAL.md).
 
 ## Ranking
 

package/docs/ROADMAP_0.2.0.md

@@ -49,6 +49,8 @@ M14 is first because adaptive ranking must be measured against a stable baseline
 
 ## M14 — Context Quality Metrics
 
+Status: **implemented on `main` for `0.2.0-alpha.1`**. The active planner remains the 0.1 deterministic baseline; quality collection is opt-in and candidate strategies are replay-only.
+
 ### Deliverables
 
 - A sanitized replay corpus with expected evidence source IDs and task descriptors.
@@ -70,6 +72,8 @@ Quality samples store planner/profile versions, source-kind counts, token totals
 
 ## M15 — Rich Symbol Indexing
 
+Status: **implemented on `main` for `0.2.0-alpha.1`**. Structural parsing remains a disposable local projection; unsupported or invalid source retains deterministic text-window behavior.
+
 ### Deliverables
 
 - A parser interface in `ds4-context-core` with deterministic regex fallback.
@@ -90,6 +94,8 @@ The parser implementation must not introduce a mandatory native build dependency
 
 ## M16 — Hybrid Semantic Retrieval
 
+Status: **implemented on `main` for `0.2.0-alpha.2`**. Semantic retrieval remains opt-in; exact/FTS ranking and lexical-only failure behavior remain available.
+
 ### Deliverables
 
 - A runtime-neutral embedding port; model invocation remains outside core.
@@ -111,6 +117,8 @@ Semantic retrieval does not replace exact matching. Exact paths, symbols, quoted
 
 ## M17 — Cross-Session Project Memory
 
+Status: **implemented on `main` behind `memory.crossSession` for `0.2.0-alpha.3`**. Discovery remains opt-in and exact trusted project identity is mandatory.
+
 ### Deliverables
 
 - Discovery of canonical Pi session JSONL files associated with the same trusted canonical project identity.
@@ -131,6 +139,8 @@ Version 0.2.0 does not silently extract new memories from conversation text. A d
 
 ## M18 — Learned Ranking
 
+Status: **implemented on `main` for `0.2.0-beta.1` with `off`, aggregate-only `shadow`, and promotion-gated `active` modes**. Static ranking remains authoritative for missing, corrupt, incompatible, unpromoted or regressing artifacts.
+
 ### Deliverables
 
 - A bounded feature schema based on metadata such as source kind, exact/FTS/vector scores, recency, branch relation, symbol relation, classification eligibility, token cost and prior selection outcome.
@@ -153,6 +163,8 @@ If the gate is not met, 0.2.0 ships shadow mode but keeps static ranking active.
 
 ## M19 — Runtime Adapter Kit
 
+Status: **implemented on `main` for `0.2.0-beta.2`**. Core ships `runtime-adapter-v1` plus a framework-neutral conformance runner; Pi exposes contract capability diagnostics, and the separately packaged callback/JSONL reference adapter passes all seven conformance cases.
+
 ### Deliverables
 
 - A documented adapter contract for canonical history snapshots, model limits, tool atomicity, trusted project roots, completion, privacy enforcement and lifecycle shutdown.

package/docs/RUNTIME_ADAPTER_KIT.md

@@ -0,0 +1,143 @@
+# Runtime Adapter Kit
+
+M19 defines `runtime-adapter-v1`, a runtime-neutral boundary between DS4 policy and an agent host. The kit lives in `ds4-context-core/adapter/*`; the non-Pi implementation is published separately as `ds4-context-reference-adapter`. Core imports no runtime SDK.
+
+## Dependency and state boundary
+
+```text
+native agent runtime / provider SDK
+                ↓
+runtime adapter package
+                ↓
+ds4-context-core/adapter/runtime-adapter
+```
+
+An adapter owns native history conversion, trusted-root discovery, provider completion, final privacy enforcement, optional runtime capabilities, and shutdown. Core owns contract types, deterministic capability negotiation, canonical tool-group validation, and the framework-neutral conformance runner.
+
+The runtime's native history remains canonical. A `RuntimeHistorySnapshot` is a read-only projection with:
+
+- `runtimeId`, `sessionId`, schema version, and deterministic source revision;
+- one exact trusted canonical project root;
+- ordered `CanonicalMessage` values with stable source-entry provenance;
+- complete tool call/result atomic groups derived from canonical blocks.
+
+Applying a DS4 context plan must not write provider-facing synthetic messages back into canonical history. Rebuild deletes or discards only adapter/DS4 projections and reproduces the snapshot from the native source.
+
+## Contract
+
+`RuntimeAdapter` requires:
+
+| Method | Responsibility | Failure rule |
+|---|---|---|
+| `snapshotHistory()` | Project the active canonical history and tool relations. | Reject an invalid, corrupt, identity-mismatched, or untrusted source. Never invent history. |
+| `rebuildDerivedState()` | Discard adapter caches/projections and replay canonical state. | Preserve canonical files and the previous valid source. |
+| `currentModel()` | Return provider/model limits through `ModelDescriptor`. | Return `undefined` when no model is selected. |
+| `trustedProjectRoot()` | Return the runtime-approved canonical root. | Reject untrusted roots; do not broaden them. |
+| `enforcePrivacy()` | Sanitize a provider payload immediately before transport. | Fail closed and do not invoke transport on error. |
+| `complete()` | Invoke the runtime/provider boundary with the sanitized payload. | Return an explicit fallback result for invalid input, privacy failure, transport failure, or closed lifecycle. |
+| `capabilityDeclarations()` / `negotiateCapabilities()` | Version and independently enable optional host features. | Missing or malformed declarations are unsupported, not fatal to other features. |
+| `diagnostics()` | Return bounded metadata-only state. | Never include prompts, history, credentials, payloads, cache handles, or provider output. |
+| `shutdown()` | Release adapter resources idempotently. | Further writes/completions fall back; history reads are rejected. |
+
+Completion output remains runtime-owned and is not automatically canonical. An adapter must append a final runtime-native message through its ordinary canonical history API if the host chooses to persist it.
+
+## Tool atomicity
+
+`buildCanonicalToolAtomicGroups()` connects every canonical `toolCall` block to all matching `toolResult` blocks. Calls sharing one assistant message form one component, so parallel tool batches cannot be split. `validateRuntimeHistorySnapshot()` rejects missing, overlapping, unknown, or mismatched groups. An incomplete live call is represented by a group with `complete: false`; it cannot be treated as a complete selectable exchange.
+
+## Capability negotiation
+
+The v1 registry is fixed and additive contracts must use a later contract version:
+
+- `compaction`;
+- `provider-continuation`;
+- `embeddings`;
+- `local-kv-reuse`.
+
+A supported capability requires a non-empty implementation version. Unsupported capabilities require a bounded reason. `negotiateRuntimeCapabilities()` evaluates each request independently:
+
+- a supported requested capability is enabled;
+- an unsupported optional capability is disabled with an informational diagnostic;
+- an unsupported required capability is disabled with a warning;
+- a duplicate, missing, or unversioned declaration is malformed and disabled;
+- no failure in one capability disables canonical history, privacy, completion fallback, or another valid capability.
+
+M19 only negotiates `local-kv-reuse`; M20 defines its eligibility and handle lifecycle. KV handles are never part of canonical history or these diagnostics.
+
+Pi advertises versioned compaction, OpenAI Responses continuation, and embedding-port support. It explicitly reports local KV reuse as unavailable. `/context adapter` shows the complete negotiation; `/context status` shows aggregate enabled/disabled counts.
+
+## Reusable conformance kit
+
+The conformance runner has no Vitest/Jest dependency. Adapter packages provide a factory that maps a versioned fixture and injected completion transport into their native runtime:
+
+```ts
+import {
+  assertRuntimeAdapterConformance,
+  runRuntimeAdapterConformance,
+} from "ds4-context-core/adapter/conformance";
+
+const report = await runRuntimeAdapterConformance({
+  name: "my-runtime",
+  async create(fixture, transport) {
+    // Create native canonical history from fixture.messages.
+    // Inject transport at the final provider boundary.
+    return { adapter, expectedProjectRoot, cleanup };
+  },
+});
+assertRuntimeAdapterConformance(report);
+```
+
+The seven checks cover:
+
+1. adapter/contract identity, model limits, and trusted root;
+2. complete capability declarations and isolated unsupported diagnostics;
+3. ordered canonical history and tool atomicity;
+4. deterministic rebuild from canonical state;
+5. remote privacy filtering plus synthetic sanitizer failure before observed transport;
+6. safe transport failure with native fallback still available;
+7. idempotent shutdown and closed-state behavior.
+
+Reports contain case IDs, booleans, and fixed failure codes only. The private marker and credential probes never appear in a report. Package smoke tests install core, Pi, and reference tarballs in a clean consumer and rerun reference conformance from compiled exports.
+
+## Reference-adapter compatibility spike
+
+M19 compared three reference targets:
+
+| Candidate | Strength | M19 risk |
+|---|---|---|
+| another full agent SDK | realistic lifecycle | adds fast-moving SDK/version coupling and obscures which behavior belongs to DS4 versus the SDK |
+| provider-client-only adapter | realistic completion | has no canonical branch, tool-group, or lifecycle contract to validate |
+| callback runtime with append-only JSONL | exercises every adapter responsibility with no vendor SDK | intentionally minimal; not a production orchestration framework |
+
+The callback JSONL target was selected because it covers the full contract while keeping the example inspectable and dependency-neutral. `ds4-context-reference-adapter` is therefore a non-Pi executable boundary example, not a claim of production support for a specific third-party agent framework.
+
+Its `ds4-runtime-session-v1` file is canonical and append-only. The header binds runtime ID, session ID, and canonical project root. Message records contain DS4 canonical messages. `createReferenceHistory()` refuses overwrite, `appendReferenceHistoryMessage()` checks provenance before append, and `rebuildDerivedState()` discards only the in-memory snapshot. File size and message count are bounded. Files are mode `0600` where supported.
+
+The reference adapter provides a callback completion transport, enabled fail-closed privacy by default, optional `EmbeddingPort`, and explicit unsupported declarations for native compaction, provider continuation, and local KV state.
+
+## Packaging rules
+
+All release packages use the same version.
+
+- `ds4-context-core` has no runtime SDK dependency.
+- `ds4-context-engine` contains the Pi SDK boundary and depends exactly on matching core.
+- `ds4-context-reference-adapter` contains no Pi SDK and depends exactly on matching core.
+- Adapter source is not bundled into the core tarball; core source is not bundled into adapter tarballs.
+- Core and reference TypeScript compile to ESM JavaScript and declarations before packing.
+- Publish core first, then reference, then Pi.
+
+Use:
+
+```bash
+npm run check
+npm run pack:check
+npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
+npm pack --dry-run
+```
+
+## Limitations and rollback
+
+The reference adapter is intentionally synchronous-history/callback-completion infrastructure: it does not implement streaming, native compaction, branch editing, provider continuation, or KV reuse. Unsupported features remain disabled and visible rather than emulated.
+
+Removing the reference package does not affect Pi or core. A runtime can roll back its adapter by stopping it and returning to native history/context behavior; no canonical migration or SQLite downgrade is required. Deleting adapter caches is safe. Never delete the runtime-owned JSONL file as part of DS4 rollback.