@fgv/ts-agent-memory-sqlite-vec 5.1.0-54 → 5.1.0-56

package/CAPABILITIES.md

@@ -0,0 +1,41 @@
+# `@fgv/ts-agent-memory-sqlite-vec` — persistent `IVectorIndex` + `IFragmentVectorIndex` (Node)
+
+> **This file is authoritative for what ``@fgv/ts-agent-memory-sqlite-vec`` provides and what not to hand-roll.**
+> `README.md`, where present, is getting-started material. The always-loaded index at
+> [`.ai/instructions/LIBRARY_CAPABILITIES.md`](../../.ai/instructions/LIBRARY_CAPABILITIES.md)
+> routes here; it never duplicates this content.
+
+
+---
+
+[libraries/ts-agent-memory-sqlite-vec](https://github.com/ErikFortune/fgv/tree/release/libraries/ts-agent-memory-sqlite-vec)
+
+`SqliteVecVectorIndex` — the **durable** counterpart to `InMemoryCosineIndex`: a Result-integration boundary over `better-sqlite3` + `sqlite-vec` that implements `@fgv/ts-agent-memory`'s `IVectorIndex` seam against a SQLite `vec0` file. Embeddings **persist across process restarts**, so a reopened vault answers queries with **no re-embedding** — closing the "re-embed the whole vault on every open" cost of the in-memory index at v2 scale. **Two factories, differing only in who owns the connection.** `SqliteVecVectorIndex.create({ database })` takes a consumer-owned `better-sqlite3` `Database` (BYO; the index never opens/closes it) — this is the seam for backing a record index *and* a fragment index with one connection. `SqliteVecVectorIndex.open({ path })` opens the file itself and returns an `ISqliteVecVectorIndexHandle` (`{ index, close() }`), so the single-index case needs **no consumer value-import of `better-sqlite3` and no hand-rolled `captureResult` around a constructor that throws** — the one place this wrapper leaked its own dependency into consumer source. The disposer travels on the handle rather than on the class precisely because a `create()`-made index holds a connection it does not own and must stay incapable of closing it. `close()` is idempotent, a failed `open` closes what it opened rather than leaking it, and **two `open` calls on one path give two independent connections, not a shared one** (use `create` for the shared case). **An index caches prepared `Statement`s, which reference the connection, so closing a connection while an index over it is still reachable leaves native destructors to run at process teardown — `release()` drops those statements and marks the index unusable, without touching the connection** (which is why it is safe on a `create()`-made index that owns none). `open()`'s handle calls it before closing; with `create()` the ordering is yours, and it is `release()` then `close()`. A released index **fails** — or, for the synchronous `size` / `recordCount` / `fragmentCount`, **throws** — rather than answering `0`, which would be indistinguishable from an index that has simply never had an `add`. The limit is worth stating: `better-sqlite3` exposes no public `finalize()`, so dropping the last reference makes a statement collectable *earlier*, while the environment is alive, rather than finalizing it — that narrows the teardown window, it does not close it. **And `release()` can only reach statements the index retains**, which is why the rebuild-path table clear now runs through `Database.exec` rather than a prepared statement: the statement it used to prepare was referenced by nothing the moment it returned, so it was never in `_stmts` and `release()` never had it. `exec` creates no `Statement` at all. One `prepare` outside `_stmts` remains — the schema probe each `create` / `open` runs against `sqlite_master`, which needs a bound parameter and a returned row and so cannot become `exec`. `SqliteVecFragmentIndex` carries the identical pair. Either factory loads the `sqlite-vec` extension and recovers the established dimension from the existing table on reopen. Keyed on `edgeTargetKey` (`(scope, id)` — collision-proof across scopes), cosine similarity (`score = 1 − cosineDistance`, **byte-identical scoring to `InMemoryCosineIndex`**), replace via delete-then-insert (vec0 rejects `INSERT OR REPLACE`). Its `rebuild` reports the same per-kind `IVectorRebuildReport` on the same `DetailedResult` as the in-memory index, failure detail included — the two are kept observably identical so a swap changes durability and nothing else. **Drop-in swap with zero core change** — wire it as the store's `vectorIndex` and skip the consumer-opt-in `rebuild(asRecordSource())` on open. `better-sqlite3` / `sqlite-vec` are **peer dependencies**. **NOT in scope:** ANN/large-N indexing (brute-force `vec0` KNN, same regime as the in-memory index), connection *pooling* / multi-process coordination (plain open+close is supported via `open`), embedding, a browser sibling. **Use this to back `FileTreeMemoryStore` when embeddings must survive a restart; use `InMemoryCosineIndex` for ephemeral / small vaults.**
+
+`SqliteVecFragmentIndex` — the **durable** counterpart to `InMemoryFragmentCosineIndex`, and the fragment-granular sibling of `SqliteVecVectorIndex` in the same package (N-Ask5, #562). Implements `@fgv/ts-agent-memory`'s `IFragmentVectorIndex` seam against a `vec0` table keyed on `target_key` as a **`PARTITION KEY`** (many rows per record) with each fragment's identity in `+start_off` / `+end_off` / `+fragment_id` **auxiliary columns** (read back on query, never filtered — `fragment_id` in particular is stored and returned verbatim and is never parsed). `addFragments` is whole-record-replace (atomic delete-then-insert transaction), `remove` drops every fragment of a target, and `query(vector, topK, options?)` applies both the per-record cap and the `scope` / `id` record narrowing during selection, before the topK cut — a `scope` + `id` narrowing becomes a **partition-restricted** KNN (`AND target_key = ?`), while a scope-only one (the versioned-kind case) cannot, since `target_key` equality cannot express a prefix, so it prefix-filters the full ranked set instead; correct either way, and the cost asymmetry is recorded in `docs/FUTURE.md` — **byte-identical semantics + cosine scoring to `InMemoryFragmentCosineIndex`**, with the same reopen dimension recovery so a persistent fragment index answers with **no re-embedding on open**. `SqliteVecFragmentIndex.create({ database, tableName? })` (default table `'memory_fragments'`, distinct from the record index's `'memory_vectors'`; both may share one file). Locator offsets are validated as safe integers on write and coerced on read (safe-integer-mode-tolerant); an absent locator persists as a NULL offset pair, and a half-NULL pair (or a row with neither identity) fails the query loudly as corrupt persisted data. As of the derived-state stream it also implements **`has`**, **`recordCount`** / **`fragmentCount`** and a real **`rebuild`** — before this it had none of them, so a persistent fragment index could not be backfilled at all and the store's own reconciliation promise was undeliverable for it. Its `rebuild` reports the same `IFragmentVectorRebuildReport` on the same `DetailedResult` as `InMemoryFragmentCosineIndex`, failure detail included, so a swap changes durability and nothing else — the property already held for the record pair. **Use this to back `FileTreeMemoryStore`'s `fragmentIndex` when sub-document fragment embeddings must survive a restart.**
+
+**`vec0` schema changes require a drop-and-re-index — there are no in-place migrations.** `CREATE VIRTUAL TABLE IF NOT EXISTS` is a no-op against an existing table (SQLite never compares schemas) and `vec0` has no `ALTER TABLE ADD COLUMN`, so a table written by an earlier release of this package survives an upgrade untouched. When a release changes a table's columns, consumers must `DROP TABLE` (or point the index at a fresh `tableName`) and re-index. `SqliteVecFragmentIndex.create` detects the mismatch up front by parsing the stored `CREATE VIRTUAL TABLE` SQL and fails with an actionable message naming expected-vs-found columns, rather than letting an opaque `no such column` surface at statement-prepare time. **The cost is embedding time, never data** — vectors are derived artifacts; the vault records remain the source of truth. Known instance: the release that added `IEmbeddedFragment.fragmentId` added the `+fragment_id` column to the fragment table (the record table is unchanged).
+
+---
+
+---
+
+## Decision shortcuts
+
+- **Vector/fragment embeddings must survive a process restart (no re-embed on open)?** → back the store with `SqliteVecVectorIndex` / `SqliteVecFragmentIndex` from `@fgv/ts-agent-memory-sqlite-vec` instead of the in-memory indexes; skip the `rebuild(asRecordSource())` on open. Node-only. **Pick the factory by connection ownership:** `open({ path })` when this index is the only thing on the file — it opens the driver for you (no consumer `better-sqlite3` import) and hands back `{ index, close() }`; `create({ database })` when one connection must back both the record and fragment indexes. Don't call `open` twice on one path expecting a shared connection — that is two connections.
+
+---
+
+## Recent additions
+
+*Newest first. **Generated** — see the repo index; do not hand-edit inside the markers.*
+
+<!-- BEGIN GENERATED: recent-additions -->
+
+- **2026-08-22** — **Shipped:** the rebuild-path table clear runs through `exec`, so the one statement `release()` could never reach no longer exists. ([#654](https://github.com/ErikFortune/fgv/pull/654))
+- **2026-08-21** — **Shipped:** `release()` on both index classes — it drops the index's prepared statements and marks it unusable, and **never touches the connection**. A released index fails, or for the synchronous counts throws, rather than answering as an empty one. ([#651](https://github.com/ErikFortune/fgv/pull/651))
+- **2026-08-15** — **Outcome:** delivered. Additive; `create()` is untouched on both classes.
+- **2026-08-15** — **Shipped:** every count in `IVectorRebuildReport` is now resolved by `Kind`, the exclusion count originates in the layer that decides it, and a failed rebuild carries its partial report — one change rather than the staged pair the consumer left open. ([#633](https://github.com/ErikFortune/fgv/pull/633))
+- **2026-07** — Branch `agent-memory-fragment-id` (from `origin/release`) → **PR #585** against `release` (not merged). ([#585](https://github.com/ErikFortune/fgv/pull/585))
+
+<!-- END GENERATED: recent-additions -->

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@fgv/ts-agent-memory-sqlite-vec",
-  "version": "5.1.0-54",
+  "version": "5.1.0-56",
   "description": "Result-integration boundary providing a persistent, sqlite-vec-backed IVectorIndex for @fgv/ts-agent-memory (survives restarts — no re-embed on open)",
   "main": "lib/index.js",
   "types": "dist/ts-agent-memory-sqlite-vec.d.ts",
@@ -22,6 +22,7 @@
     "dist",
     "CHANGELOG.json",
     "README.md",
+    "CAPABILITIES.md",
     "LICENSE",
     "!lib/test",
     "!dist/test",
@@ -53,11 +54,11 @@
     "@types/better-sqlite3": "^7.6.11",
     "@microsoft/api-extractor": "^7.55.2",
     "@rushstack/eslint-config": "4.6.4",
-    "@rushstack/heft": "1.2.7",
-    "@rushstack/heft-jest-plugin": "1.2.6",
-    "@rushstack/heft-node-rig": "2.11.27",
+    "@rushstack/heft": "1.3.0",
+    "@rushstack/heft-jest-plugin": "2.0.17",
+    "@rushstack/heft-node-rig": "2.11.50",
     "@types/heft-jest": "1.0.6",
-    "@types/jest": "^29.5.14",
+    "@types/jest": "^30.0.0",
     "@types/node": "^20.14.9",
     "@typescript-eslint/eslint-plugin": "^8.52.0",
     "@typescript-eslint/parser": "^8.52.0",
@@ -67,21 +68,21 @@
     "eslint-plugin-node": "^11.1.0",
     "eslint-plugin-promise": "^7.2.1",
     "eslint-plugin-tsdoc": "~0.5.2",
-    "jest": "^29.7.0",
+    "jest": "^30.5.2",
     "rimraf": "^6.1.2",
-    "ts-jest": "^29.4.6",
+    "ts-jest": "^29.4.12",
     "ts-node": "^10.9.2",
     "typescript": "5.9.3",
-    "@fgv/ts-utils": "5.1.0-54",
-    "@fgv/ts-agent-memory": "5.1.0-54",
-    "@fgv/ts-utils-jest": "5.1.0-54",
-    "@fgv/heft-dual-rig": "5.1.0-54"
+    "@fgv/heft-dual-rig": "5.1.0-56",
+    "@fgv/ts-utils": "5.1.0-56",
+    "@fgv/ts-agent-memory": "5.1.0-56",
+    "@fgv/ts-utils-jest": "5.1.0-56"
   },
   "peerDependencies": {
     "better-sqlite3": "^12.0.0",
     "sqlite-vec": "^0.1.9",
-    "@fgv/ts-utils": "5.1.0-54",
-    "@fgv/ts-agent-memory": "5.1.0-54"
+    "@fgv/ts-utils": "5.1.0-56",
+    "@fgv/ts-agent-memory": "5.1.0-56"
   },
   "scripts": {
     "build": "heft build --clean",