ds4-context-engine 0.3.7 → 0.3.8

package/docs/ADR/063-fts-rowid-key-mappings.md

@@ -0,0 +1,53 @@
+# 063 — Resolve FTS key deletes through rowid mapping tables
+
+**Date:** 2026-09-06
+**Status:** Accepted
+**Related:** [002](002-pi-jsonl-canonical-sqlite-rebuildable.md), [058](058-bounded-manifest-storage.md)
+
+## Context
+
+The FTS5 shadow tables used by the session index and the project knowledge
+index declare `entry_key` / `snippet_id` as `UNINDEXED` columns. FTS5 cannot
+use the token vocabulary to locate a row by an `UNINDEXED` column, so every
+`DELETE FROM …_fts WHERE entry_key = ?` is a full scan of the whole virtual
+table, whose cost is proportional to the size of the entire FTS index.
+
+For incremental appends the cost is small, but a session-index **rebuild**
+(fork, path change, truncation, missing checkpoint) re-writes every entry:
+a large session (11k entries, ~600MB database) paid thousands of full scans
+and stalled for hours, surfacing as an indefinite hang of the agent session.
+
+A first candidate — making `entry_key` an indexed FTS5 column and recreating
+the tables in a migration — was benchmarked and rejected: FTS5 then treats the
+key as searchable vocabulary, changing the `MATCH` surface (a term present
+only in an entry key becomes a false-positive hit) and the `bm25` column
+weight mapping of existing queries.
+
+## Decision
+
+Keep the FTS table definitions byte-identical and add dedicated mapping
+tables that derive the SQLite rowid of each FTS row:
+
+- `entries_fts_keys(entry_key PRIMARY KEY, fts_rowid)`
+- `project_snippets_fts_keys(snippet_id, project_path, fts_rowid)`
+
+Deletes become `DELETE FROM …_fts WHERE rowid = <mapped>` (O(log n)) and
+rebuild cleanups resolve stale rows through the same mapping joined on the
+base tables. Insert paths upsert the mapping from the FTS `last_insert_rowid`
+inside the same write transaction as the FTS insert, so mapping and index
+cannot diverge (a crash rolls back both).
+
+The mapping tables are purely derived, rebuildable state: migration 16
+backfills them from the live FTS rows with one scan, in a single
+transaction — a failure rolls back and the previous schema keeps working.
+
+## Consequences
+
+- Per-row FTS deletes: ~0.23 ms instead of ~40 ms–2 s (cold) per row;
+  a 11k-row rebuild drops from hours to ~3 s.
+- The FTS schema, the `MATCH` surface and the `bm25` weight mapping are
+  unchanged (verified by the migration and repository tests).
+- Existing databases upgrade non-destructively on first open after the
+  extension update; no user data is stored in the mapping tables.
+- Future migrations that drop/recreate FTS tables must rebuild the mapping
+  tables in the same migration.

package/docs/ADR/README.md

@@ -66,5 +66,6 @@ The initial decisions from the development plan are accepted:
 | [060](060-optional-portable-agent-tools.md) | Opt in to edit reports, adaptive reads/results and session-owned local jobs | Accepted |
 | [061](061-compaction-latency.md) | Bound compaction update calls, input budgets, concurrent segments and phase timings | Accepted |
 | [062](062-cache-aware-context-planning.md) | Opt-in cache-aware tail planning using model pricing and observed cache shares | Accepted |
+| [063](063-fts-rowid-key-mappings.md) | Resolve FTS key deletes through rowid mapping tables | Accepted |
 
 Each decision will receive a dedicated record when implementation pressure introduces alternatives or consequences not already covered by the development plan.

package/docs/releases/0.3.7.md

@@ -70,6 +70,15 @@ same tail caps as 0.3.6.
 - Prefix-cache simulator: 7 passed.
 - Cache-aware runtime integration: 4 passed.
 - Config catalog/loader validation: passed.
+- `pack:check` from a clean HEAD snapshot: core 239 files,
+  reference-adapter 7 files, engine 91 files; consumer install clean.
+- Published and registry-verified: `ds4-context-core` 0.3.7,
+  `ds4-context-reference-adapter` 0.3.7, `ds4-context-engine` 0.3.7
+  (`registry:check` PASS at exact version; one transient registry replica
+  miss on first attempt, confirmed present via `npm view`).
+- Release fix `7404a9a`: root `package.json` must depend exactly on
+  `ds4-context-core@0.3.7` (caught by the `pack:check` gate on the
+  coordinated-version requirement).
 
 ## Known limits
 

package/docs/releases/0.3.8.md

@@ -0,0 +1,64 @@
+# Release 0.3.8 — Indexed FTS key deletes via rowid mappings (schema 16)
+
+**Version analyzed:** DS4 Context Engine `0.3.8`
+**Commit:** (recorded after pack verification)
+**Coordinated packages:** `ds4-context-core` 0.3.8, `ds4-context-reference-adapter` 0.3.8, `ds4-context-engine` 0.3.8
+
+## Summary
+
+Bugfix release: FTS deletes were full virtual-table scans (FTS5 `UNINDEXED`
+key columns), making session-index rebuilds on large sessions stall for
+hours. The FTS tables are now rebuilt over rowid mapping tables, turning
+every keyed delete into an indexed lookup. No behavior, configuration or
+search-surface change: the fix is transparent and addresses the
+fork/resume stall on large sessions.
+
+## Root cause
+
+`DELETE FROM entries_fts WHERE entry_key = ?` cannot use the FTS5
+vocabulary because `entry_key` (and `project_snippets_fts.snippet_id`) are
+declared `UNINDEXED`. Every row removal scanned the entire FTS index.
+A rebuild of a session with ~11k entries performed thousands of those
+scans: measured 65k `pread64`/s against a 615MB database and ~100 full
+index passes per ~140s, with an estimated **hours** to completion —
+surfacing as an indefinite hang of the agent session after a session
+fork/resume. The behavior class pre-dates 0.3.7; the 0.3.7 work sessions
+made it visible by pushing the database into the hundreds of MB.
+
+Indexing the keys directly inside FTS5 was benchmarked and **rejected**:
+it would make key text part of the `MATCH` vocabulary (false positives)
+and shift the `bm25` column weights. See ADR 063.
+
+## Changes
+
+- Migration **16** `fts-rowid-key-mappings`: two derived mapping tables
+  (`entries_fts_keys`, `project_snippets_fts_keys`) backfilled from the
+  live FTS rows in one scan, inside a single transaction.
+- `SessionIndexRepository`: per-row deletes go through the mapping
+  (`DELETE … WHERE rowid = ?`); the rebuild stale-row cleanup joins the
+  mapping with `entries` and uses the session index; the mapping is
+  upserted from the FTS `last_insert_rowid` in the same transaction.
+- `ProjectKnowledgeRepository`: same pattern for per-snippet deletes
+  (`replaceFile`) and per-project cleanup (`clearProject`).
+- The FTS schemas, the `MATCH` queries and the `bm25` weights are
+  byte-identical to 0.3.7; search results do not change.
+- Tests: `tests/integration/fts-rowid-keys.test.ts` covering the
+  v15 → v16 upgrade (data preservation, key ↔ rowid correctness, search
+  surface unchanged), rebuild/append/stale-cleanup consistency and the
+  snippet `replaceFile`/`clearProject` paths; version-pinned tests moved
+  to `CURRENT_SCHEMA_VERSION`.
+
+## Compatibility and migration
+
+- **Non-destructive by design:** the FTS indexes are derived state; the
+  migration is transactional, so on any failure the previous schema
+  remains intact and the extension keeps working with the old behavior.
+- **No data stored in the mapping tables:** the base tables remain the
+  single source of truth.
+- Upgrade runs automatically on first open after the extension update
+  (measured 467 ms on a 615MB database with 31k entries / 62k snippets;
+  no user-visible step required).
+- The mapping backfill is one scan; a very large database adds only
+  seconds to the first open after update.
+- Existing guarantees (fail-open, atomicity, privacy, pins, compaction,
+  hard limits) are unchanged.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "ds4-context-engine",
-  "version": "0.3.7",
+  "version": "0.3.8",
   "description": "Non-destructive, provider-independent context management for Pi.",
   "type": "module",
   "license": "MIT",
@@ -62,7 +62,7 @@
     ]
   },
   "dependencies": {
-    "ds4-context-core": "0.3.7"
+    "ds4-context-core": "0.3.8"
   },
   "peerDependencies": {
     "@earendil-works/pi-ai": "0.84.3",
@@ -77,5 +77,9 @@
   },
   "engines": {
     "node": ">=22.19.0"
+  },
+  "allowScripts": {
+    "@google/genai@1.52.0": true,
+    "protobufjs@7.6.5": true
   }
 }

package/src/pi-adapter/version.ts

@@ -1,4 +1,4 @@
-export const EXTENSION_VERSION = "0.3.7";
+export const EXTENSION_VERSION = "0.3.8";
 export const SUPPORTED_PI_VERSION = "0.84.3";
 export const OBSERVER_PLANNER_VERSION = "observer-model-aware-v1";
 export const PLANNER_VERSION = "managed-learned-ranking-v1";