ds4-context-engine 0.1.2 → 0.2.0-beta.2

package/docs/HYBRID_RETRIEVAL.md

@@ -0,0 +1,117 @@
+# Hybrid Semantic Retrieval
+
+M16 adds opt-in vector candidate generation to historical and trusted-project retrieval. Exact identifiers and FTS5 remain active and authoritative; semantic candidates are fused into the same bounded, deterministic ranking and never replace source provenance or live-hash checks.
+
+## Configuration
+
+Semantic retrieval remains disabled for upgrades from 0.1:
+
+```json
+{
+  "retrieval": {
+    "semantic": true
+  }
+}
+```
+
+The supported default is the runtime-owned local feature-hash embedding:
+
+```json
+{
+  "retrieval": {
+    "semantic": true,
+    "embedding": {
+      "mode": "local",
+      "provider": "ds4-local",
+      "model": "feature-hash-v1",
+      "dimensions": 256,
+      "maxSources": 50000,
+      "candidatePool": 80,
+      "batchSize": 64,
+      "queryCacheSize": 64,
+      "timeoutMs": 2000
+    }
+  }
+}
+```
+
+It is deterministic, has no native dependency and performs no network access. Core defines `EmbeddingPort`; the Pi adapter supplies the implementation. Other runtimes can inject a compatible local, WASM or remote port without adding model invocation to core.
+
+Remote embedding requires all of the following:
+
+- `mode: "remote"`;
+- an exact `provider/model` entry in `remoteProfiles` (wildcards are rejected);
+- `privacy.enabled: true`;
+- provider-specific privacy allow rules;
+- a runtime-injected `EmbeddingPort` whose provider, model, dimensions and remote destination exactly match configuration.
+
+```json
+{
+  "retrieval": {
+    "semantic": true,
+    "embedding": {
+      "mode": "remote",
+      "provider": "embedding.example",
+      "model": "semantic-v2",
+      "dimensions": 768,
+      "remoteProfiles": ["embedding.example/semantic-v2"]
+    }
+  },
+  "privacy": {
+    "enabled": true,
+    "remoteProviders": {
+      "embedding.example": ["normal", "internal"]
+    }
+  }
+}
+```
+
+The packaged Pi adapter intentionally includes no default remote embedding client. Missing or mismatched ports produce lexical-only results.
+
+## Privacy
+
+Every remote source and query passes through `PrivacyPolicyEngine` before `EmbeddingPort.embed`. A value whose effective classification is `local-only`, or which contains a provider-blocked span, is excluded as a whole and never reaches the remote port. Allowed text is secret-redacted before invocation. Local mode does not cross a provider boundary.
+
+Embedding diagnostics contain only counts, model identity, dimensions, destination, freshness, cache status, timings and normalized fallback reasons. They never contain query text, source text, vectors, provider response IDs or remote handles.
+
+## Derived storage
+
+SQLite schema v13 adds `derived_embeddings`. Each row is keyed by:
+
+```text
+source kind + scope + source key + source hash + chunking version
++ embedding provider + embedding model + dimensions
+```
+
+Rows contain the source group, numeric vector JSON and indexing time, but no copied query or evidence text. Session entries use `pi-session-entry-v1`; project chunks use their parser version or `text-window-v1`.
+
+A source edit prunes only vectors whose key/hash/chunk version is no longer current. Unchanged source rows and unrelated model/dimension profiles remain intact. A model or dimension switch selects a separate profile instead of rewriting compatible rows. Deleting SQLite discards the whole vector projection; canonical Pi JSONL and live project files rebuild it.
+
+## Candidate generation and rank fusion
+
+Candidate pools are bounded. Generation order is:
+
+1. exact project path, qualified symbol, simple symbol, quoted phrase or historical identifier;
+2. escaped FTS5 candidates;
+3. cosine-ranked vector candidates.
+
+The lexical and vector ranks are combined with deterministic reciprocal-rank fusion, semantic similarity and stable source-ID tie-breaking. Exact matches retain a score tier that vectors cannot displace. Historical candidates still require active-branch membership and exclusion from the active native context. Project candidates still require project trust, sensitive-file exclusion and a live SHA-256 match before injection.
+
+Source vectors are generated during session/project index sync. Query vectors use a bounded volatile hash-keyed cache. Repeating a request with current source vectors performs no embedding call. Query hashes and vectors are not persisted as conversation state.
+
+## Failure behavior
+
+Missing models, consent failure, privacy exclusion, invalid dimensions, corrupt vectors, synchronous-port timeout, provider exceptions, vector storage errors and vector search errors all return the available exact/FTS result. They cannot fail context planning. Diagnostics expose the fallback reason without source content.
+
+## Quality gate
+
+[`quality/semantic-corpus-v1.json`](../quality/semantic-corpus-v1.json) is a synthetic extension of the M14 replay methodology. It measures the same evidence-recall and irrelevant-token signals for semantic synonyms and exact-match preservation. The byte-stable golden report records:
+
+```text
+lexical evidence recall  0.25
+hybrid evidence recall   1.00
+recall delta             +0.75
+irrelevant-token delta    0.00
+```
+
+Verification covers local query reuse, source-vector caching, profile isolation, changed-source pruning, remote `local-only` exclusion, provider failure, exact priority, historical/project fusion and schema v13 rebuild behavior.

package/docs/LEARNED_RANKING.md

@@ -0,0 +1,90 @@
+# Learned Ranking
+
+M18 adds an optional metadata-only linear ranker while preserving the deterministic static ranker as the universal fallback.
+
+## Modes
+
+```json
+{
+  "ranking": {
+    "mode": "shadow",
+    "modelPath": "ds4-context/ranking-model.json",
+    "minimumTrainingSamples": 20,
+    "maxTrainingSamples": 10000,
+    "maxLatencyMs": 10
+  }
+}
+```
+
+- `off` does not load or evaluate a learned model.
+- `shadow` evaluates a valid model, records only aggregate rank disagreement, and keeps static selection unchanged.
+- `active` changes supplemental-candidate ordering only when the checksummed artifact contains a successful promotion report. Missing, incompatible, corrupt, unpromoted, or regressing artifacts fall back to static ranking.
+
+`off` is the upgrade default. Privacy filtering, mandatory pins, current-request retention, category budgets, atomic groups, and hard input limits remain authoritative in every mode.
+
+## Bounded feature schema
+
+`ranking-features-v1` contains only a source-kind enum and ten finite normalized numbers:
+
+- static score;
+- exact-match score;
+- FTS score;
+- vector score;
+- recency;
+- active-branch relation;
+- symbol relation;
+- classification eligibility;
+- token cost;
+- prior selection outcome.
+
+All numbers are bounded to `[0, 1]`. Candidate text, excerpts, prompts, file paths, symbols, claims, provider payloads, and credentials are not model features. Stable candidate and repository identities are SHA-256 hashes in canonical labels.
+
+## Canonical labels and local training
+
+Explicit feedback is appended through Pi as a classified `ds4-context-ranking-feedback-v1` custom entry:
+
+```text
+/context ranking feedback useful CANDIDATE_ID --classification internal
+/context ranking feedback irrelevant CANDIDATE_ID --classification local-only
+```
+
+Only candidates from the latest managed context are accepted. A feedback entry contains schema/version, feedback ID, timestamp, classification, label source, label, two hashes, and the bounded feature vector. It contains no candidate text. Sanitized replay labels use the same schema with `labelSource: "replay"`.
+
+Train a local shadow artifact from the active canonical Pi branch:
+
+```text
+/context ranking train
+```
+
+Training is deterministic for the same unique labels and timestamp, requires both useful and irrelevant examples, and is bounded by `minimumTrainingSamples` and `maxTrainingSamples`. The artifact is written with mode `0600` where supported. Training never calls a provider and does not make the resulting artifact active.
+
+## Artifact integrity and deterministic inference
+
+`ranking-model.json` uses schema version `1`, feature version `ranking-features-v1`, algorithm `bounded-linear-centroid-v1`, bounded weights, training counts, a deterministic model ID, and a SHA-256 checksum over its stable JSON payload. Inference uses rounded finite arithmetic and resolves equal scores by static score and then candidate ID.
+
+The model artifact is local derived state. Deleting it preserves Pi JSONL labels and restores static ranking. A newly trained artifact can be reconstructed from canonical labels.
+
+## Promotion gate
+
+`evaluateRankingPromotion()` in `ds4-context-core/ranking/learned-ranker` evaluates sanitized held-out fixtures. `withRankingPromotion()` seals the resulting report into a new checksummed artifact. Promotion succeeds only when all of these conditions hold:
+
+- primary quality score improves;
+- exact-identifier recall does not regress;
+- privacy violations do not increase;
+- atomicity failures do not increase;
+- overflow does not increase;
+- measured p95 latency stays within budget;
+- repeated inference yields identical ordering;
+- the configured minimum number of held-out repositories is represented.
+
+An artifact without an eligible report can run in shadow mode but cannot alter context in active mode.
+
+## Diagnostics
+
+```text
+/context ranking
+/context health
+/context manifest
+```
+
+Ranking diagnostics include mode/status, versions, model identity, promotion state, label counts, malformed/duplicate counts, candidate count, top-rank disagreement, pairwise disagreements, mean rank shift, inference duration, and fallback reason. Shadow diagnostics and Context Manifests contain aggregate comparison only—never candidate IDs, labels, features, or text.

package/docs/LOCAL_KV_REUSE.md

@@ -0,0 +1,131 @@
+# Local KV Reuse
+
+M20 adds an opt-in, runtime-neutral prefix/KV capability for local inference runtimes. KV state is only an inference optimization: it is not canonical history, memory, retrieval evidence, a Context Manifest source, or a SQLite projection.
+
+Pi does not expose native KV handles and continues to report `local-kv-reuse` as unsupported. The callback/JSONL reference adapter can exercise the capability when its host injects a `LocalKvRuntimePort`; no provider SDK is added to core or the reference package.
+
+## Boundary
+
+```text
+runtime/provider payload
+        ↓
+adapter privacy enforcement
+        ↓
+runtime-specific prefix/options extraction from sanitized payload
+        ↓
+core exact-byte eligibility hashes
+        ↓
+volatile LocalKvRuntimePort lookup + transport
+        ├─ exact hit ───────────────> cached completion
+        └─ miss/rejected/unavailable -> full prompt replay
+```
+
+Core receives the provider-ready prefix only long enough to hash it. It returns metadata-only component hashes and an aggregate fingerprint. A runtime port maps that fingerprint to its own volatile handle. Handles never cross the port, and core has no API for serializing them.
+
+## Configuration and negotiation
+
+Configuration is disabled by default:
+
+```json
+{
+  "localKvReuse": {
+    "enabled": false
+  }
+}
+```
+
+Enabling configuration is not sufficient. The active adapter must independently declare a versioned `local-kv-reuse` capability and provide a `LocalKvRuntimePort`. Unsupported or malformed declarations disable KV reuse without affecting history, privacy, completion fallback, or any other capability.
+
+A reference-adapter host opts in explicitly:
+
+```ts
+const adapter = new JsonlReferenceRuntimeAdapter({
+  // canonical history, model, privacy, and ordinary transport options...
+  localKv: {
+    enabled: true,
+    port: myVolatileKvPort,
+    runtimeRevision: "llama-runtime-build-42",
+    modelRevision: "model-checksum-abc",
+    prepare(sanitizedPayload) {
+      return {
+        promptPrefix: exactProviderPrefixBytes(sanitizedPayload),
+        systemOptions: providerSystemOptions(sanitizedPayload),
+        toolOptions: providerToolOptions(sanitizedPayload),
+        prefixTokenCount: 32_000,
+        contextTokenCount: 40_000,
+      };
+    },
+  },
+});
+```
+
+`prepare()` receives only the payload returned by adapter privacy enforcement. Remote destinations never enter the local KV path.
+
+## Exact eligibility
+
+`deriveLocalKvEligibility()` uses `local-kv-eligibility-v1`. An eligible fingerprint includes independent SHA-256 components for:
+
+- exact provider, model, and model revision;
+- exact UTF-8/string or byte prefix;
+- deterministic JSON-compatible system and tool options;
+- privacy policy version and local destination;
+- runtime identity, runtime revision, and capability implementation version.
+
+Identifiers are not case-folded or wildcard-matched. Prefix bytes are not normalized. Option object keys are ordered only for deterministic serialization; values, array order, missing/undefined values, and system/tool separation remain significant. Functions, symbols, binary option values, accessors, custom prototypes, circular values, and non-finite numbers are ineligible instead of being hashed ambiguously.
+
+Any changed prefix byte, provider/model identity, model revision, system/tool option, privacy version, runtime revision, or capability version produces a different fingerprint. Disabled configuration, unsupported capability, remote destination, invalid identifiers, an empty prefix, or invalid token counts bypass reuse.
+
+## Runtime port and replay
+
+`LocalKvRuntimePort` deliberately exposes no handle:
+
+```ts
+interface LocalKvRuntimePort {
+  tryReuse(request): Promise<
+    | { status: "hit"; output: unknown; savedPrefillTokens: number; prefillLatencyMs: number }
+    | { status: "miss" | "rejected" | "unavailable" }
+  >;
+  fullReplay(request): Promise<{
+    output: unknown;
+    prefillTokens: number;
+    prefillLatencyMs: number;
+  }>;
+  shutdown?(): Promise<void>;
+}
+```
+
+The runtime owns fingerprint-to-handle lookup, eviction, provider transport, and restart cleanup. `LocalKvReuseController` treats thrown lookup errors and malformed lookup results as unavailable state. Every miss, stale rejection, unavailable state, or runtime restart invokes `fullReplay()` with the complete sanitized payload. If the injected KV full replay itself fails, the reference adapter retries the complete sanitized payload through its ordinary native transport.
+
+Adapter shutdown calls optional port shutdown once. Cache deletion, eviction, process loss, or runtime restart therefore changes performance only; it cannot change canonical state or prevent a full replay.
+
+## Aggregate diagnostics
+
+`local-kv-diagnostics-v1` contains numbers only:
+
+- requests, eligible requests, and bypasses;
+- hits, misses, rejected states, unavailable states, full replays, and transport failures;
+- reusable prefix tokens and total context-occupancy tokens;
+- saved prefill tokens;
+- full-replay prefill tokens and runtime-reported prefill latency.
+
+Diagnostics never include provider/model text, prefix or payload content, component fingerprints, runtime handles, credentials, provider response IDs, or output. Per-request completion metadata keeps `contextTokens`, `savedPrefillTokens`, `replayPrefillTokens`, and `prefillLatencyMs` as separate fields; context occupancy is never reported as cache savings.
+
+## Privacy and persistence
+
+Privacy enforcement precedes runtime-specific prefix extraction and hash verification. A remote payload is sanitized and sent through ordinary transport without consulting local KV state. An enabled local policy is still applied before extraction. A prior cached state cannot authorize material excluded by the current privacy policy because the policy version and sanitized exact prefix are both part of eligibility.
+
+No KV handle or prefix copy is written to:
+
+- Pi JSONL or reference canonical JSONL;
+- Context Manifests;
+- DS4 SQLite;
+- learned-ranking artifacts;
+- adapter/core diagnostics.
+
+There is no M20 database migration. Deleting all volatile runtime KV state requires no DS4 rebuild and no canonical-history migration.
+
+## Verification and benchmark
+
+Unit and reference-adapter integration tests cover exact hits, changed bytes/options, provider/model/privacy/runtime invalidation, cache loss, stale rejection, unavailable state, full replay, privacy-before-verification, aggregate-only diagnostics, idempotent shutdown, and unchanged canonical JSONL.
+
+`tests/benchmarks/local-kv.bench.ts` hashes a synthetic 32,000-token reusable prefix inside a 40,000-token context and separately benchmarks a warm completion whose simulated runtime metadata reports 32,000 saved prefill tokens and `0.5 ms` prefill latency. On the development host, exact eligibility averaged about `0.259 ms` and the warm controller path about `0.248 ms`. These are observational, not portable provider-latency guarantees; the benchmark intentionally reports runtime prefill latency/token savings separately from context occupancy.

package/docs/MEMORY_AND_PINS.md

@@ -1,6 +1,6 @@
 # Memory and Persistent Pins
 
-M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical.
+M9 separates durable, user-curated state from conversation history while keeping Pi JSONL canonical. M17 optionally reconstructs explicit project-scoped mutations from sibling Pi sessions for the same trusted canonical project.
 
 ## Authority and scope
 
@@ -30,9 +30,12 @@ Pins remain subordinate to system/developer instructions. Memory tells the model
 /context memory supersede MEMORY_ID [--classification LEVEL] [--source ID,ID] <new claim>
 /context memory invalidate MEMORY_ID [reason]
 /context memory expire MEMORY_ID [reason]
+/context memory sources
+/context memory exclude SESSION_ID [reason]
+/context memory include SESSION_ID
 ```
 
-Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust.
+Arguments support single/double quotes and backslash escaping. `--` ends option parsing. Source entry IDs must be on Pi's active branch. Project scope requires Pi project trust. Source exclusion affects only project-scoped contributions and persists until `include`; the active session cannot be newly excluded but can restore a prior exclusion.
 
 Mutations are manual-first. Repeating the same normalized pin/claim returns its existing ID without appending another entry.
 
@@ -60,7 +63,7 @@ No row is silently overwritten. SQLite mutation and materialized tables are disp
 3. replays all known mutations in timestamp + canonical entry order;
 4. rebuilds memory, pin, source, lifecycle, and FTS rows transactionally.
 
-Deleting `context.db` and reopening the canonical source session reconstructs its state. Project items from other sessions reappear when those canonical sessions are replayed.
+Deleting `context.db` and reopening the canonical source session reconstructs its state. With `memory.crossSession: true`, DS4 discovers bounded sibling `.jsonl` files, accepts only headers whose `cwd` resolves to the exact trusted canonical project identity, incrementally indexes each source, and reconstructs project items without opening every session manually. No claim is extracted from ordinary conversation text.
 
 ## Supersession and contradiction handling
 
@@ -80,7 +83,7 @@ A conflicting `add` is rejected with the IDs involved. The user must issue expli
   "Package export mode defaults to SingleFile."
 ```
 
-The old item remains stored as `superseded` and points to the new active item. During replay, concurrent active records with the same key are both preserved; the deterministic later record is marked `invalid` with a conflict reason rather than replacing the earlier one.
+The old item remains stored as `superseded` and points to the new active item. During replay, records are ordered by timestamp, source session ID, source entry order, mutation ID and mutation entry key. Concurrent active records with the same key are both preserved; the deterministic later record is marked `invalid` with a conflict reason rather than replacing the earlier one. Selected items retain source-session file/ID, mutation entry, creation branch leaf, evidence entries, classification, supersession and contradiction IDs.
 
 Pins use the same immutable replacement pattern through `--supersedes`. `/context unpin` records a soft `deleted` lifecycle mutation.
 
@@ -126,7 +129,7 @@ User text is JSON-quoted. Context Manifests contain IDs, scope, classification,
 
 M10 stores an optional classification in the canonical mutation. Before a remote call, a prohibited pin/memory is omitted as a whole and recorded only by ID/classification/reason. Explicit classification cannot be downgraded by markers inside its content. See [`PRIVACY.md`](PRIVACY.md).
 
-## SQLite schema v9
+## SQLite schema v9 and v15
 
 Schema v9 extends materialized `memory_items` and `pins` with:
 
@@ -136,6 +139,8 @@ Schema v9 extends materialized `memory_items` and `pins` with:
 
 `memory_mutations` and `pin_mutations` reference canonical indexed Pi custom entries. `memory_sources` retains exact scoped entry provenance. `memory_fts` is rebuilt transactionally.
 
+Schema v15 adds source-branch provenance, per-session project-memory checkpoints, source status and explicit source exclusions. These rows are derived and disposable. Missing, truncated, moved, identity-mismatched or corrupt sibling files are marked unavailable and their unverifiable project-scoped mutations stop contributing; session-scoped state remains isolated and the active session keeps its last transactional projection on refresh failure. Restoration or database deletion causes deterministic replay from JSONL.
+
 Migration preserves legacy materialized rows for inspection. Because they have no canonical mutation entry, a later full replay may discard them; new operations are always event-sourced.
 
 ## Diagnostics
@@ -148,6 +153,7 @@ Migration preserves legacy materialized rows for inspection. Because they have n
 /context excluded
 /context pins
 /context memory
+/context memory sources
 /context health
 /context rebuild-index
 ```

package/docs/PORTABLE_CORE.md

@@ -1,6 +1,6 @@
 # Portable Core
 
-M13 extracts the runtime-neutral implementation into the independently buildable `ds4-context-core` workspace package. The root `ds4-context-engine` package remains the Pi integration.
+M13 extracts the runtime-neutral implementation into the independently buildable `ds4-context-core` workspace package. The root `ds4-context-engine` package remains the Pi integration. M19 adds the versioned adapter contract/conformance kit in core and a separately compiled non-Pi callback/JSONL reference adapter. M20 adds exact local-KV eligibility and replay orchestration while leaving every provider SDK and native cache handle in the runtime adapter.
 
 ## Dependency rule
 
@@ -14,7 +14,7 @@ ds4-context-core
 
 Core never imports an adapter or runtime SDK. The Pi adapter imports core through its public ESM exports.
 
-The core may use Node.js standard-library facilities such as `node:sqlite`, filesystem APIs and cryptographic hashing. Portable means independent of Pi and reusable by another Node-based agent runtime; it does not mean browser-compatible.
+The core may use Node.js standard-library facilities such as `node:sqlite`, filesystem APIs and cryptographic hashing. Portable means independent of every agent runtime SDK and reusable by another Node-based agent runtime; it does not mean browser-compatible.
 
 ## Package contents
 
@@ -30,6 +30,7 @@ The core may use Node.js standard-library facilities such as `node:sqlite`, file
 - memory/pin materialization;
 - privacy classification and provider policy;
 - native-continuation eligibility and hash state;
+- exact local-KV eligibility, handle-free runtime port types, replay policy and aggregate diagnostics;
 - rebuildable SQLite repositories;
 - stable serialization, hashing and logging.
 
@@ -44,7 +45,7 @@ The root adapter owns all Pi-specific behavior:
 
 ## Adapter responsibilities
 
-A future runtime adapter must:
+Every `runtime-adapter-v1` implementation must:
 
 1. preserve its runtime's canonical history and expose stable source identifiers;
 2. convert native messages to DS4 canonical messages without losing tool-call/result atomicity;
@@ -54,17 +55,22 @@ A future runtime adapter must:
 6. append memory/pin mutations to canonical runtime history before materializing derived state;
 7. invoke model completion at the adapter boundary for generated summaries;
 8. enforce privacy immediately before provider transport;
-9. discard or rebuild SQLite and artifact projections safely;
-10. fall back to native runtime behavior when operational integration fails.
+9. retain optional local-KV handles inside its volatile runtime port and replay the full sanitized payload after every non-hit;
+10. discard or rebuild SQLite and artifact projections safely;
+11. fall back to native runtime behavior when operational integration fails.
+
+The executable contract, independent capability negotiation and framework-neutral seven-case conformance runner are exported from `ds4-context-core/adapter/runtime-adapter` and `ds4-context-core/adapter/conformance`. M20's handle-free port and controller are exported from `ds4-context-core/adapter/local-kv`. See [Runtime Adapter Kit](RUNTIME_ADAPTER_KIT.md) and [Local KV Reuse](LOCAL_KV_REUSE.md).
 
 ## Build and exports
 
 ```bash
 npm run build:core
+npm run build:adapters
 npm run typecheck
 npm test
 npm run pack:check
 npm pack --dry-run --workspace ds4-context-core
+npm pack --dry-run --workspace ds4-context-reference-adapter
 ```
 
 TypeScript sources compile to `packages/core/dist` as ESM JavaScript, source maps and declaration files. The npm package exports a top-level API and fine-grained subpaths such as:
@@ -74,11 +80,11 @@ import { calculateContextBudget } from "ds4-context-core";
 import { planManagedContext } from "ds4-context-core/planner/context-planner";
 ```
 
-The Pi package declares an exact same-release dependency on `ds4-context-core`. Release order is therefore core first, adapter second. `npm run pack:check` verifies both tarball inventories, installs them together in a clean temporary consumer, probes core ESM exports and starts the packaged adapter with isolated Pi RPC state. See [Releasing DS4](RELEASING.md) for the publication checklist.
+The Pi and reference packages declare exact same-release dependencies on `ds4-context-core`. Release order is core, reference adapter, then Pi adapter. `npm run pack:check` verifies all three tarball inventories, installs them together in a clean temporary consumer, probes core ESM exports, runs compiled reference conformance and starts the packaged Pi adapter with isolated RPC state. See [Releasing DS4](RELEASING.md) for the publication checklist.
 
 ## Enforcement
 
-`tests/unit/portable-core-boundary.test.ts` recursively rejects Pi SDK and adapter imports from core source, then imports the compiled package and exercises portable model/budget policy. Existing integration tests consume core through package exports, so the Pi adapter is tested across the actual package boundary.
+`tests/unit/portable-core-boundary.test.ts` recursively rejects Pi SDK and adapter imports from core source, then imports the compiled package and exercises portable model/budget policy. Runtime-adapter unit tests validate capability isolation and canonical tool groups; the reference package passes the shared conformance runner. Existing integration tests consume core through package exports, so the Pi adapter is tested across the actual package boundary.
 
 ## State guarantees
 
@@ -87,6 +93,7 @@ Extraction does not change DS4 state semantics:
 - Pi JSONL remains canonical for Pi sessions;
 - SQLite remains disposable and rebuildable;
 - compaction remains non-destructive and strictly validated;
-- provider continuation handles remain volatile;
+- provider continuation and local-KV handles remain volatile and adapter-owned;
+- local-KV cache loss changes only prefill performance and transparently replays the full sanitized payload;
 - planner, retrieval, compaction and persistence failures still fail open at the adapter boundary;
 - enabled privacy enforcement still fails closed before remote transport.

package/docs/PROJECT_KNOWLEDGE.md

@@ -1,6 +1,6 @@
 # Project Knowledge
 
-M7 indexes trusted project files as a disposable SQLite projection and injects only task-relevant, hash-current snippets. Live files remain canonical.
+M7 indexes trusted project files as a disposable SQLite projection and injects only task-relevant, hash-current snippets. M15 adds runtime-neutral structural symbol parsing and exact symbol lookup. M16 optionally adds vectors over those structural/text chunks while preserving exact and FTS retrieval. Live files remain canonical.
 
 ## Trust boundary
 
@@ -36,29 +36,37 @@ DS4 excludes VCS metadata, `.pi`, dependencies, build outputs, caches, virtual e
 
 ## File and snippet index
 
-SQLite schema v7 stores:
+SQLite schema v7 introduced:
 
 - `project_states`: canonical project path, Git root/branch/HEAD, dirty flag, changed paths, index time;
 - `project_files`: path, SHA-256, bytes, mtime, language, indexed Git HEAD, tracked/modified state, lifecycle;
-- `project_snippets`: immutable file hash, line range, source, heuristic declarations, token estimate, stale flag;
+- `project_snippets`: immutable file hash, line range, source, token estimate and stale flag;
 - `project_snippets_fts`: FTS5 content/path/symbol index.
 
-Files are split into overlapping line windows. Heuristic symbols cover class, interface, enum, namespace, record, struct, trait, type, function, common language declarations, and SQL objects. No parser or LLM is needed.
+Schema v12 adds derived structural metadata to each snippet: chunk kind, parser version, symbol ID/name/kind, qualified name, signature, parent symbol, imports and references. Exact-name indexes remain disposable and rebuild from live project files. Schema v13 stores separately keyed derived vectors; a file-hash change prunes only vectors for no-longer-current chunks, while unrelated source and embedding-profile rows remain intact.
 
-A full or incremental sync compares size, mtime, Git revision, and modified state. Source is re-read and SHA-256 hashed whenever metadata changes. Old snippets are marked `stale`; they remain inspectable but all exact and FTS queries require `stale = 0` and a current file row.
+`ds4-context-core` exposes a `SymbolParser` interface. The built-in `regex-structural-v1` parser has deterministic coverage for TypeScript, JavaScript, Python and Go without native dependencies. Optional parser adapters run first through `SymbolParserChain`; an unavailable or throwing adapter falls through to the built-in parser. Unsupported languages, malformed delimiter structure, supported files with no declarations and files carrying explicit DS4 classification spans retain the M7 overlapping text-window fallback. Keeping marked spans intact ensures M10 privacy enforcement sees the same boundaries as the 0.1 index.
+
+Structural chunks follow declaration boundaries and retain signatures, parent relationships, imports/references and exact line ranges. Large declarations are split into bounded overlapping subchunks while sharing one symbol identity. Symbol IDs are SHA-256 values derived from canonical project identity, relative path, file hash, structural range, symbol kind and qualified name. Text fallback IDs retain their deterministic project/path/hash/range derivation.
+
+A full or incremental sync compares size, mtime, Git revision and modified state. Source is re-read and SHA-256 hashed whenever metadata changes. Replacing one file marks only that path's prior snippets stale, then transactionally inserts the new file-hash rows and FTS entries; unrelated file rows keep their IDs. Stale rows remain inspectable, but every exact and FTS query requires `stale = 0` and a current file row.
 
 ## Retrieval and ranking
 
-The same current-request `TaskDescriptor` used for historical retrieval supplies file paths, symbols, identifiers, errors, quoted phrases, technologies, and keywords. Candidate generation combines case-sensitive literal `instr()` search with escaped FTS5 terms.
+The same current-request `TaskDescriptor` used for historical retrieval supplies file paths, symbols, identifiers, errors, quoted phrases, technologies and keywords. Candidate generation queries literal exact path/basename and exact qualified/simple symbol indexes before generic literal content and escaped FTS5 candidates. Exact lookup does not treat comment, string or reference text as a declaration.
+
+When `retrieval.semantic` is enabled, the current embedding profile also contributes a bounded cosine-ranked pool. Deterministic reciprocal-rank fusion combines lexical and vector ranks; exact paths and symbols retain higher score tiers. Source vectors are populated during index sync and query vectors use a bounded volatile cache. Vector/model/privacy failures return the exact/FTS candidates unchanged. See [`HYBRID_RETRIEVAL.md`](HYBRID_RETRIEVAL.md).
 
 Deterministic ranking prioritizes:
 
 ```text
-exact project-relative path  140
-exact basename               125
-declared symbol              115
+exact qualified symbol       190
+exact project-relative path  180
+exact simple symbol          170
+exact basename               150
+declared fallback symbol     115
 exact phrase                  90
-symbol text                   85
+symbol text                   65
 FTS match                     60..20
 working-tree change           10
 tracked source                 3
@@ -123,17 +131,19 @@ Source text is not copied into the Context Manifest or structured logs. It remai
 
 ## Fail-open behavior
 
-Project discovery, Git, indexing, FTS, validation, and retrieval errors are isolated from session indexing and context planning. A project failure records local diagnostics and contributes no snippets; Pi and historical retrieval continue. SQLite startup failure retains the existing runtime-wide Pi fallback.
+Project discovery, Git, optional parser adapters, indexing, FTS, validation and retrieval errors are isolated from session indexing and context planning. Parser adapter failure uses deterministic regex fallback; unsupported or invalid source uses bounded text chunks. A project subsystem failure records local diagnostics and contributes no snippets; Pi and historical retrieval continue. SQLite startup failure retains the existing runtime-wide Pi fallback.
+
+## Verification and benchmark
 
-## Benchmark
+`quality/symbol-corpus-v1.json` is a synthetic, versioned TypeScript/JavaScript/Python/Go corpus with expected and forbidden declarations. Unit coverage verifies that structural parsing produces fewer false symbol matches than the preserved 0.1 regex extractor. Integration coverage verifies exact qualified/path lookup, metadata fields, stable IDs, unsupported/invalid fallback, changed-file invalidation, unrelated-file stability, trust/exclusion behavior and live-hash validation.
 
 `tests/benchmarks/project-knowledge.bench.ts` creates and indexes 5,000 source files, then executes exact path/symbol plus FTS retrieval with live hash validation:
 
 ```text
-mean  12.28 ms
-p75   13.12 ms
-p99   21.04 ms
-max   21.04 ms
+mean  29.99 ms
+p75   31.71 ms
+p99   37.21 ms
+max   37.21 ms
 ```
 
 Command: