@fgv/ts-agent-memory-sqlite-vec 5.1.0-49 → 5.1.0-51

package/README.md

@@ -25,6 +25,31 @@ rush add -p @fgv/ts-agent-memory-sqlite-vec   # or npm/pnpm add
 
 ## Quick start
 
+Two factories, differing only in **who owns the connection**. If this index is the only
+thing on the file, `open` is the shorter path and needs no `better-sqlite3` import of
+your own:
+
+```ts
+import { SqliteVecVectorIndex } from '@fgv/ts-agent-memory-sqlite-vec';
+import { FileTreeMemoryStore } from '@fgv/ts-agent-memory';
+
+// We open the file and hand back a disposer for the connection we created.
+const handle = (await SqliteVecVectorIndex.open({ path: '/path/to/vault/vectors.db' })).orThrow();
+
+const store = (
+  await FileTreeMemoryStore.create({ root, registry, vectorIndex: handle.index, embed })
+).orThrow();
+
+// ...use the store; embeddings are written to vectors.db on every put.
+handle.close(); // closes the connection THIS open() created; idempotent.
+```
+
+`SqliteVecFragmentIndex.open({ path })` is the identical shape for the fragment lane.
+
+**Two `open` calls on one path give two independent connections, not a shared one.** To
+back a record index *and* a fragment index with a single connection, own it yourself and
+pass it to both `create` methods:
+
 ```ts
 import Database from 'better-sqlite3';
 import { SqliteVecVectorIndex } from '@fgv/ts-agent-memory-sqlite-vec';
@@ -108,7 +133,7 @@ Known instance: the release that added `IEmbeddedFragment.fragmentId` added a `+
 Deliberately excluded — reach for the upstream libraries (or a different backend) directly if you need these:
 
 - **ANN / large-N indexing.** Query is a brute-force `vec0` KNN scan — correct and durable for the same "thousands of records" regime the in-memory index targets. An approximate-nearest-neighbor structure for very large N is a different backend behind the same `IVectorIndex` / `IFragmentVectorIndex` seam. (This applies to both indexes, including `SqliteVecFragmentIndex`, whose capped query fetches the full ranked set.)
-- **Connection lifecycle.** You open and close the `better-sqlite3` `Database`; this index never does. Pooling, WAL/pragma tuning, backups, and multi-process coordination are yours.
+- **Connection lifecycle beyond plain open/close.** With `create({ database })` you open and close the `better-sqlite3` `Database` and this index never does; with `open({ path })` this package opens the file and the returned handle's `close()` disposes of exactly what it opened. Either way, pooling, WAL/pragma tuning, backups, and multi-process coordination are yours.
 - **Embedding.** This is a vector *index*, not an embedder — the store's consumer-wired `MemoryEmbedder` produces the vectors (`@fgv/ts-extras/ai-assist` `callProviderEmbedding`, `@fgv/ts-extras-transformers`, etc.).
 - **A browser sibling.** `better-sqlite3` is Node-only. A WASM-SQLite browser variant, if ever needed, is a separate package.
 - **Schema migration of any kind.** Re-embedding with a different-dimension model against an existing table fails loudly, and a package release that changes a `vec0` table's columns requires a drop-and-re-index — see [Upgrading](#upgrading-vec0-schema-changes-require-a-drop-and-re-index). Drop the table (or use a new `tableName`) to re-index; `vec0` cannot be altered in place.

package/dist/packlets/sqlite-vec-index/connection.js

@@ -0,0 +1,54 @@
+/*
+ * Copyright (c) 2026 Erik Fortune
+ * SPDX-License-Identifier: MIT
+ */
+import { captureAsyncResult, captureResult } from '@fgv/ts-utils';
+/**
+ * Opens a `better-sqlite3` connection this package owns.
+ *
+ * @remarks
+ * **The value import of `better-sqlite3` lives here, and it is deliberately
+ * lazy.** Every other module in this package imports the driver as `import type`
+ * only, so merely importing `@fgv/ts-agent-memory-sqlite-vec` has never loaded the
+ * native binding — a property the path-based factories must not cost consumers who
+ * only ever call `create()`. A static import would move the native load (and its
+ * failure mode) to package-load time for everyone. The dynamic `import()` keeps it
+ * at the one call that actually needs a connection.
+ *
+ * Taking this import is the entire point of the path-based factories: it is the
+ * only place this wrapper leaked its own dependency into consumer source.
+ *
+ * @param path - Filesystem path to the database file. `':memory:'` is accepted and
+ * yields an owned ephemeral connection.
+ * @param label - Package-facing prefix for the failure message.
+ * @returns `Success` with a connection this package owns, or `Failure` if the
+ * driver could not be loaded or the file could not be opened.
+ * @internal
+ */
+export async function openOwnedConnection(path, label) {
+    return (await captureAsyncResult(async () => (await import('better-sqlite3')).default))
+        .withErrorFormat((m) => `${label}: failed to load the 'better-sqlite3' driver: ${m}`)
+        .onSuccess((driver) => captureResult(() => new driver(path)).withErrorFormat((m) => `${label}: failed to open '${path}': ${m}`));
+}
+/**
+ * Closes a connection this package opened.
+ *
+ * @remarks
+ * Only ever called on a connection produced by {@link openOwnedConnection}. A
+ * connection supplied through a `create()` param is the consumer's and is never
+ * closed here — that asymmetry is the reason `close` travels on the handle
+ * returned by `open` rather than sitting on the index class.
+ *
+ * `better-sqlite3`'s own `close()` is safe to call on an already-closed
+ * connection, so a second `close()` on the same handle succeeds rather than
+ * failing.
+ *
+ * @internal
+ */
+export function closeOwnedConnection(db, label) {
+    return captureResult(() => {
+        db.close();
+        return true;
+    }).withErrorFormat((m) => `${label}: failed to close the connection: ${m}`);
+}
+//# sourceMappingURL=connection.js.map

package/dist/packlets/sqlite-vec-index/connection.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"connection.js","sourceRoot":"","sources":["../../../src/packlets/sqlite-vec-index/connection.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAGH,OAAO,EAAU,kBAAkB,EAAE,aAAa,EAAE,MAAM,eAAe,CAAC;AAE1E;;;;;;;;;;;;;;;;;;;;;GAqBG;AACH,MAAM,CAAC,KAAK,UAAU,mBAAmB,CACvC,IAAY,EACZ,KAAa;IAEb,OAAO,CAAC,MAAM,kBAAkB,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,MAAM,MAAM,CAAC,gBAAgB,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;SACpF,eAAe,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,KAAK,iDAAiD,CAAC,EAAE,CAAC;SACpF,SAAS,CAAC,CAAC,MAAM,EAAE,EAAE,CACpB,aAAa,CAAC,GAAG,EAAE,CAAC,IAAI,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,KAAK,qBAAqB,IAAI,MAAM,CAAC,EAAE,CAAC,CACzG,CAAC;AACN,CAAC;AAED;;;;;;;;;;;;;;GAcG;AACH,MAAM,UAAU,oBAAoB,CAAC,EAA0B,EAAE,KAAa;IAC5E,OAAO,aAAa,CAAO,GAAG,EAAE;QAC9B,EAAE,CAAC,KAAK,EAAE,CAAC;QACX,OAAO,IAAI,CAAC;IACd,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,KAAK,qCAAqC,CAAC,EAAE,CAAC,CAAC;AAC9E,CAAC","sourcesContent":["/*\n * Copyright (c) 2026 Erik Fortune\n * SPDX-License-Identifier: MIT\n */\n\nimport type BetterSqlite3 from 'better-sqlite3';\nimport { Result, captureAsyncResult, captureResult } from '@fgv/ts-utils';\n\n/**\n * Opens a `better-sqlite3` connection this package owns.\n *\n * @remarks\n * **The value import of `better-sqlite3` lives here, and it is deliberately\n * lazy.** Every other module in this package imports the driver as `import type`\n * only, so merely importing `@fgv/ts-agent-memory-sqlite-vec` has never loaded the\n * native binding — a property the path-based factories must not cost consumers who\n * only ever call `create()`. A static import would move the native load (and its\n * failure mode) to package-load time for everyone. The dynamic `import()` keeps it\n * at the one call that actually needs a connection.\n *\n * Taking this import is the entire point of the path-based factories: it is the\n * only place this wrapper leaked its own dependency into consumer source.\n *\n * @param path - Filesystem path to the database file. `':memory:'` is accepted and\n * yields an owned ephemeral connection.\n * @param label - Package-facing prefix for the failure message.\n * @returns `Success` with a connection this package owns, or `Failure` if the\n * driver could not be loaded or the file could not be opened.\n * @internal\n */\nexport async function openOwnedConnection(\n  path: string,\n  label: string\n): Promise<Result<BetterSqlite3.Database>> {\n  return (await captureAsyncResult(async () => (await import('better-sqlite3')).default))\n    .withErrorFormat((m) => `${label}: failed to load the 'better-sqlite3' driver: ${m}`)\n    .onSuccess((driver) =>\n      captureResult(() => new driver(path)).withErrorFormat((m) => `${label}: failed to open '${path}': ${m}`)\n    );\n}\n\n/**\n * Closes a connection this package opened.\n *\n * @remarks\n * Only ever called on a connection produced by {@link openOwnedConnection}. A\n * connection supplied through a `create()` param is the consumer's and is never\n * closed here — that asymmetry is the reason `close` travels on the handle\n * returned by `open` rather than sitting on the index class.\n *\n * `better-sqlite3`'s own `close()` is safe to call on an already-closed\n * connection, so a second `close()` on the same handle succeeds rather than\n * failing.\n *\n * @internal\n */\nexport function closeOwnedConnection(db: BetterSqlite3.Database, label: string): Result<true> {\n  return captureResult<true>(() => {\n    db.close();\n    return true;\n  }).withErrorFormat((m) => `${label}: failed to close the connection: ${m}`);\n}\n"]}

package/dist/packlets/sqlite-vec-index/model.js.map

@@ -1 +1 @@
-{"version":3,"file":"model.js","sourceRoot":"","sources":["../../../src/packlets/sqlite-vec-index/model.ts"],"names":[],"mappings":"AAAA;;;GAGG","sourcesContent":["/*\n * Copyright (c) 2026 Erik Fortune\n * SPDX-License-Identifier: MIT\n */\n\nimport type BetterSqlite3 from 'better-sqlite3';\n\n/**\n * Parameters for {@link SqliteVecVectorIndex.create}.\n * @public\n */\nexport interface ISqliteVecVectorIndexCreateParams {\n  /**\n   * A `better-sqlite3` `Database` the consumer owns (bring-your-own, mirroring\n   * the boundary-package convention). The consumer opens it (`new Database(path)`\n   * for a persistent file, or `new Database(':memory:')` for an ephemeral index)\n   * and owns its lifecycle — this index never closes it. `create` loads the\n   * `sqlite-vec` extension onto the connection and, if the vector table already\n   * exists (a reopened persistent file), recovers its established dimension so no\n   * re-embedding is required on open.\n   */\n  readonly database: BetterSqlite3.Database;\n\n  /**\n   * Name of the `vec0` virtual table that holds the embeddings. Must be a simple\n   * SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to `'memory_vectors'`.\n   * Supply a distinct name to hold more than one independent index in a single\n   * database file.\n   */\n  readonly tableName?: string;\n}\n\n/**\n * Parameters for {@link SqliteVecFragmentIndex.create}.\n * @public\n */\nexport interface ISqliteVecFragmentIndexCreateParams {\n  /**\n   * A `better-sqlite3` `Database` the consumer owns (bring-your-own, mirroring\n   * {@link ISqliteVecVectorIndexCreateParams.database}). The consumer opens it and\n   * owns its lifecycle — this index never closes it. `create` loads the\n   * `sqlite-vec` extension onto the connection and, if the fragment table already\n   * exists (a reopened persistent file), recovers its established dimension so no\n   * re-embedding is required on open.\n   */\n  readonly database: BetterSqlite3.Database;\n\n  /**\n   * Name of the `vec0` virtual table that holds the fragment embeddings. Must be a\n   * simple SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to\n   * `'memory_fragments'`. Supply a distinct name (distinct from any record-level\n   * {@link SqliteVecVectorIndex} table) to hold more than one independent index in\n   * a single database file.\n   */\n  readonly tableName?: string;\n}\n"]}
+{"version":3,"file":"model.js","sourceRoot":"","sources":["../../../src/packlets/sqlite-vec-index/model.ts"],"names":[],"mappings":"AAAA;;;GAGG","sourcesContent":["/*\n * Copyright (c) 2026 Erik Fortune\n * SPDX-License-Identifier: MIT\n */\n\nimport type BetterSqlite3 from 'better-sqlite3';\nimport type { Result } from '@fgv/ts-utils';\nimport type { SqliteVecFragmentIndex } from './sqliteVecFragmentIndex';\nimport type { SqliteVecVectorIndex } from './sqliteVecVectorIndex';\n\n/**\n * Parameters for {@link SqliteVecVectorIndex.create}.\n * @public\n */\nexport interface ISqliteVecVectorIndexCreateParams {\n  /**\n   * A `better-sqlite3` `Database` the consumer owns (bring-your-own, mirroring\n   * the boundary-package convention). The consumer opens it (`new Database(path)`\n   * for a persistent file, or `new Database(':memory:')` for an ephemeral index)\n   * and owns its lifecycle — this index never closes it. `create` loads the\n   * `sqlite-vec` extension onto the connection and, if the vector table already\n   * exists (a reopened persistent file), recovers its established dimension so no\n   * re-embedding is required on open.\n   */\n  readonly database: BetterSqlite3.Database;\n\n  /**\n   * Name of the `vec0` virtual table that holds the embeddings. Must be a simple\n   * SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to `'memory_vectors'`.\n   * Supply a distinct name to hold more than one independent index in a single\n   * database file.\n   */\n  readonly tableName?: string;\n}\n\n/**\n * Parameters for {@link SqliteVecVectorIndex.open}.\n * @public\n */\nexport interface ISqliteVecVectorIndexOpenParams {\n  /**\n   * Filesystem path to the database file, opened by this package rather than by\n   * the consumer. Created if it does not exist, exactly as `better-sqlite3` would.\n   * `':memory:'` is accepted and yields an owned ephemeral connection.\n   *\n   * **Two `open` calls on one path produce two independent connections, not a\n   * shared one.** That is legal in SQLite and has a different locking story than\n   * the single-connection case — writes contend, and a reader can see a\n   * `SQLITE_BUSY`. To put a record index and a fragment index on one connection\n   * (the intended shared-handle case), open the connection yourself and pass it to\n   * both `create` methods.\n   */\n  readonly path: string;\n\n  /**\n   * Name of the `vec0` virtual table that holds the embeddings. Must be a simple\n   * SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to `'memory_vectors'`.\n   * Supply a distinct name to hold more than one independent index in a single\n   * database file.\n   */\n  readonly tableName?: string;\n}\n\n/**\n * An index plus the connection {@link SqliteVecVectorIndex.open} opened for it.\n *\n * @public\n */\nexport interface ISqliteVecVectorIndexHandle {\n  /** The index, ready to use. */\n  readonly index: SqliteVecVectorIndex;\n\n  /**\n   * Closes the connection **this `open` call created**. Idempotent — a second\n   * `close` succeeds rather than failing.\n   *\n   * @remarks\n   * The disposer travels on this handle rather than on the index class because an\n   * index built by `create` holds a connection the **consumer** owns, and must stay\n   * incapable of closing it. A `close()` method meaningful on some instances and\n   * forbidden on others would be a lie in the type; here, only the caller that\n   * caused the connection to exist is handed the means to end it.\n   *\n   * The index is unusable afterwards — every operation on it will fail against a\n   * closed connection.\n   */\n  close(): Result<true>;\n}\n\n/**\n * Parameters for {@link SqliteVecFragmentIndex.open}.\n * @public\n */\nexport interface ISqliteVecFragmentIndexOpenParams {\n  /**\n   * Filesystem path to the database file, opened by this package rather than by\n   * the consumer. Created if it does not exist, exactly as `better-sqlite3` would.\n   * `':memory:'` is accepted and yields an owned ephemeral connection.\n   *\n   * **Two `open` calls on one path produce two independent connections, not a\n   * shared one** — see {@link ISqliteVecVectorIndexOpenParams.path}. To put a\n   * fragment index and a record index on one connection, open it yourself and pass\n   * it to both `create` methods.\n   */\n  readonly path: string;\n\n  /**\n   * Name of the `vec0` virtual table that holds the fragment embeddings. Must be a\n   * simple SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to\n   * `'memory_fragments'`.\n   */\n  readonly tableName?: string;\n}\n\n/**\n * An index plus the connection {@link SqliteVecFragmentIndex.open} opened for it.\n *\n * @public\n */\nexport interface ISqliteVecFragmentIndexHandle {\n  /** The index, ready to use. */\n  readonly index: SqliteVecFragmentIndex;\n\n  /**\n   * Closes the connection **this `open` call created**. Idempotent — a second\n   * `close` succeeds rather than failing. See\n   * {@link ISqliteVecVectorIndexHandle.close} for why the disposer lives here\n   * rather than on the index class.\n   */\n  close(): Result<true>;\n}\n\n/**\n * Parameters for {@link SqliteVecFragmentIndex.create}.\n * @public\n */\nexport interface ISqliteVecFragmentIndexCreateParams {\n  /**\n   * A `better-sqlite3` `Database` the consumer owns (bring-your-own, mirroring\n   * {@link ISqliteVecVectorIndexCreateParams.database}). The consumer opens it and\n   * owns its lifecycle — this index never closes it. `create` loads the\n   * `sqlite-vec` extension onto the connection and, if the fragment table already\n   * exists (a reopened persistent file), recovers its established dimension so no\n   * re-embedding is required on open.\n   */\n  readonly database: BetterSqlite3.Database;\n\n  /**\n   * Name of the `vec0` virtual table that holds the fragment embeddings. Must be a\n   * simple SQL identifier (`[A-Za-z_][A-Za-z0-9_]*`). Defaults to\n   * `'memory_fragments'`. Supply a distinct name (distinct from any record-level\n   * {@link SqliteVecVectorIndex} table) to hold more than one independent index in\n   * a single database file.\n   */\n  readonly tableName?: string;\n}\n"]}

package/dist/packlets/sqlite-vec-index/rebuildHelpers.js

@@ -0,0 +1,48 @@
+/*
+ * Copyright (c) 2026 Erik Fortune
+ * SPDX-License-Identifier: MIT
+ */
+import { captureAsyncResult } from '@fgv/ts-utils';
+/**
+ * Invoke a consumer-supplied hook that already returns a `Result`, converting a
+ * synchronous throw or a promise rejection into a `Failure` rather than letting
+ * it escape.
+ *
+ * @remarks
+ * Package-internal. `@fgv/ts-agent-memory` carries an identical private copy for
+ * its in-memory indexes. Exporting a single `AsyncDeferredResult`-invoking
+ * primitive from `ts-utils` is the right home and is recorded in
+ * `docs/TECH_DEBT.md`; this module exists because *both* index classes in *this*
+ * package now need it, which is the point at which a second in-package copy stops
+ * being the cheaper thing.
+ */
+export async function invokeHook(hook) {
+    return (await captureAsyncResult(hook)).onSuccess((inner) => inner);
+}
+/**
+ * Compose the failure that aborted a rebuild with the outcome of the rollback
+ * that followed it.
+ *
+ * @remarks
+ * A rollback that ALSO fails is worth saying out loud: the `'fail'` path promises
+ * an empty index, and a caller that retries against a table which is neither the
+ * old index nor empty is working from a state the contract never described. This
+ * matters more here than in the in-memory package — these tables are **durable**,
+ * so a botched rollback survives the process.
+ */
+export function withRollbackNote(error, rollback) {
+    return rollback.isFailure() ? `${error} (rollback also failed: ${rollback.message})` : error;
+}
+/**
+ * Increment `kind`'s tally by `by` (default one).
+ *
+ * @remarks
+ * The `by` parameter exists for the fragment lane, whose `fragments` count
+ * accumulates a fan-out rather than a record count — the one place a rebuild adds
+ * more than one per record.
+ */
+export function tally(counts, kind, by = 1) {
+    var _a;
+    counts.set(kind, ((_a = counts.get(kind)) !== null && _a !== void 0 ? _a : 0) + by);
+}
+//# sourceMappingURL=rebuildHelpers.js.map

package/dist/packlets/sqlite-vec-index/rebuildHelpers.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"rebuildHelpers.js","sourceRoot":"","sources":["../../../src/packlets/sqlite-vec-index/rebuildHelpers.ts"],"names":[],"mappings":"AAAA;;;GAGG;AAEH,OAAO,EAAU,kBAAkB,EAAE,MAAM,eAAe,CAAC;AAG3D;;;;;;;;;;;;GAYG;AACH,MAAM,CAAC,KAAK,UAAU,UAAU,CAAI,IAA8B;IAChE,OAAO,CAAC,MAAM,kBAAkB,CAAC,IAAI,CAAC,CAAC,CAAC,SAAS,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,KAAK,CAAC,CAAC;AACtE,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,gBAAgB,CAAC,KAAa,EAAE,QAAsB;IACpE,OAAO,QAAQ,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC,GAAG,KAAK,2BAA2B,QAAQ,CAAC,OAAO,GAAG,CAAC,CAAC,CAAC,KAAK,CAAC;AAC/F,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,KAAK,CAAC,MAAyB,EAAE,IAAU,EAAE,KAAa,CAAC;;IACzE,MAAM,CAAC,GAAG,CAAC,IAAI,EAAE,CAAC,MAAA,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,mCAAI,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;AACjD,CAAC","sourcesContent":["/*\n * Copyright (c) 2026 Erik Fortune\n * SPDX-License-Identifier: MIT\n */\n\nimport { Result, captureAsyncResult } from '@fgv/ts-utils';\nimport { Kind } from '@fgv/ts-agent-memory';\n\n/**\n * Invoke a consumer-supplied hook that already returns a `Result`, converting a\n * synchronous throw or a promise rejection into a `Failure` rather than letting\n * it escape.\n *\n * @remarks\n * Package-internal. `@fgv/ts-agent-memory` carries an identical private copy for\n * its in-memory indexes. Exporting a single `AsyncDeferredResult`-invoking\n * primitive from `ts-utils` is the right home and is recorded in\n * `docs/TECH_DEBT.md`; this module exists because *both* index classes in *this*\n * package now need it, which is the point at which a second in-package copy stops\n * being the cheaper thing.\n */\nexport async function invokeHook<T>(hook: () => Promise<Result<T>>): Promise<Result<T>> {\n  return (await captureAsyncResult(hook)).onSuccess((inner) => inner);\n}\n\n/**\n * Compose the failure that aborted a rebuild with the outcome of the rollback\n * that followed it.\n *\n * @remarks\n * A rollback that ALSO fails is worth saying out loud: the `'fail'` path promises\n * an empty index, and a caller that retries against a table which is neither the\n * old index nor empty is working from a state the contract never described. This\n * matters more here than in the in-memory package — these tables are **durable**,\n * so a botched rollback survives the process.\n */\nexport function withRollbackNote(error: string, rollback: Result<true>): string {\n  return rollback.isFailure() ? `${error} (rollback also failed: ${rollback.message})` : error;\n}\n\n/**\n * Increment `kind`'s tally by `by` (default one).\n *\n * @remarks\n * The `by` parameter exists for the fragment lane, whose `fragments` count\n * accumulates a fan-out rather than a record count — the one place a rebuild adds\n * more than one per record.\n */\nexport function tally(counts: Map<Kind, number>, kind: Kind, by: number = 1): void {\n  counts.set(kind, (counts.get(kind) ?? 0) + by);\n}\n"]}

package/dist/packlets/sqlite-vec-index/sqliteVecFragmentIndex.js

@@ -3,10 +3,14 @@
  * SPDX-License-Identifier: MIT
  */
 import { load as loadSqliteVec } from 'sqlite-vec';
-import { captureResult, fail, succeed } from '@fgv/ts-utils';
+import { captureResult, fail, failWithDetail, succeed, succeedWithDetail } from '@fgv/ts-utils';
 import { edgeTargetKey } from '@fgv/ts-agent-memory';
+import { invokeHook, tally, withRollbackNote } from './rebuildHelpers';
+import { closeOwnedConnection, openOwnedConnection } from './connection';
 /** Default name for the fragment `vec0` virtual table. */
 const DEFAULT_TABLE_NAME = 'memory_fragments';
+/** Package-facing prefix for this class's failure messages. */
+const LABEL = 'sqlite-vec fragment index';
 /** A simple SQL identifier — the only shape allowed for the table name (it is interpolated into DDL). */
 const IDENTIFIER_RE = /^[A-Za-z_][A-Za-z0-9_]*$/;
 /**
@@ -61,9 +65,13 @@ const AUXILIARY_COLUMN_RE = /\+\s*([A-Za-z_][A-Za-z0-9_]*)/g;
  * cosine (`score = 1 - cosineDistance`), byte-identical to the in-memory index.
  * Large-N ANN indexing is explicitly out of scope, same regime as the record index.
  *
- * The `better-sqlite3` `Database` is consumer-owned (bring-your-own): this index
- * loads the `sqlite-vec` extension onto it and reads/writes the table, but never
- * opens or closes the connection.
+ * **Connection ownership depends on which factory you use.** With
+ * {@link SqliteVecFragmentIndex.create} the `Database` is consumer-owned
+ * (bring-your-own): this index loads the `sqlite-vec` extension onto it and
+ * reads/writes the table, but never opens or closes the connection — and that is
+ * the seam for backing this index and a record index with one connection. With
+ * {@link SqliteVecFragmentIndex.open} this package opens the file itself and hands
+ * back a handle carrying the disposer for the connection it created.
  * @public
  */
 export class SqliteVecFragmentIndex {
@@ -114,6 +122,48 @@ export class SqliteVecFragmentIndex {
             return new SqliteVecFragmentIndex(params.database, table, dimension);
         }).withErrorFormat((e) => `sqlite-vec fragment index: failed to initialize: ${e}`));
     }
+    /**
+     * Path-based factory. Opens the database file itself and returns the index
+     * together with a disposer for the connection it created.
+     *
+     * @remarks
+     * The fragment-granular sibling of {@link SqliteVecVectorIndex.open}, and present
+     * for the same reason: a consumer doing sub-document retrieval only would
+     * otherwise still value-import `better-sqlite3` and hand-roll a `captureResult`
+     * around a constructor that throws.
+     *
+     * **Use `create` instead when one connection must back both a fragment index and
+     * a record index** — the intended shared-handle case. Two `open` calls on one path
+     * give two independent connections, not a shared one.
+     *
+     * If initialization fails after the file is opened, the connection is closed
+     * before returning, so a failed `open` does not leak the descriptor it created.
+     * Should that close *itself* fail — the connection is then genuinely leaked — the
+     * returned message says so rather than hiding it. That includes the
+     * auxiliary-column mismatch failure, which is reported by `create` only after the
+     * file is open.
+     *
+     * @param params - See {@link ISqliteVecFragmentIndexOpenParams}.
+     * @returns `Success` with a {@link ISqliteVecFragmentIndexHandle}, or `Failure` if
+     * the driver could not be loaded, the file could not be opened, the table name is
+     * not a simple identifier, the extension fails to load, or the existing table was
+     * written by a version with a different auxiliary-column set.
+     */
+    static async open(params) {
+        return (await openOwnedConnection(params.path, LABEL)).thenOnSuccess(async (database) => (await SqliteVecFragmentIndex.create({ database, tableName: params.tableName }))
+            .onFailure((message) => 
+        // This call opened the connection, so a failure to initialize on top of it
+        // must not leave the file handle behind. A close that ALSO fails is said out
+        // loud rather than swallowed — the same reasoning, and the same helper, as
+        // `withRollbackNote`: silently discarding it would make the "a failed open
+        // leaks nothing" guarantee untrue exactly when it stopped holding, with no
+        // way for a caller to detect it.
+        fail(withRollbackNote(message, closeOwnedConnection(database, LABEL))))
+            .onSuccess((index) => succeed({
+            index,
+            close: () => closeOwnedConnection(database, LABEL)
+        })));
+    }
     /** {@inheritDoc IFragmentVectorIndex.addFragments} */
     addFragments(target, fragments) {
         const key = edgeTargetKey(target);
@@ -181,8 +231,106 @@ export class SqliteVecFragmentIndex {
             return target;
         }).withErrorFormat((e) => `fragment index: cannot remove '${edgeTargetKey(target)}': ${e}`));
     }
+    /** {@inheritDoc IFragmentVectorIndex.has} */
+    has(target) {
+        return Promise.resolve(captureResult(() => {
+            // Before any add has created the table there is nothing held — a truthful
+            // `false`, matching `remove`'s idempotence and the zero counts.
+            if (this._stmts === undefined) {
+                return false;
+            }
+            return this._stmts.has.get(edgeTargetKey(target)) !== undefined;
+        }).withErrorFormat((e) => `fragment index: cannot check '${edgeTargetKey(target)}': ${e}`));
+    }
+    /** {@inheritDoc IFragmentVectorIndex.rebuild} */
+    async rebuild(source, embed, options) {
+        var _a;
+        const lenient = ((_a = options === null || options === void 0 ? void 0 : options.onRecordError) !== null && _a !== void 0 ? _a : 'fail') === 'skip';
+        // `source` is consumer-supplied, so a throw or rejection becomes a `Failure`
+        // here rather than escaping as an exception.
+        const listed = await invokeHook(() => source.list());
+        if (listed.isFailure()) {
+            // Deliberately BEFORE any clear, matching both siblings: a failed list is no
+            // evidence about the fragments already held, and clearing here would destroy
+            // a healthy PERSISTED index over a transient read error. No detail — there is
+            // nothing this call disturbed to describe.
+            return failWithDetail(`fragment index rebuild: failed to list records: ${listed.message}`);
+        }
+        const cleared = this._clear();
+        if (cleared.isFailure()) {
+            // Also nothing established: the table still holds whatever it held.
+            return failWithDetail(`fragment index rebuild: failed to clear the index: ${cleared.message}`);
+        }
+        const indexed = new Map();
+        const fragments = new Map();
+        const declined = new Map();
+        const skipped = [];
+        // Absent stays absent — only the source knows whether it filtered anything.
+        const report = () => ({
+            indexed,
+            fragments,
+            declined,
+            excluded: listed.value.excluded,
+            skipped
+        });
+        for (const scoped of listed.value.records) {
+            const kind = scoped.record.envelope.kind;
+            // Capture-wrapped: an embedder that throws mid-loop would otherwise escape
+            // past the `'fail'` rollback below, leaving this DURABLE table holding a
+            // partial index that survives the process.
+            const embedded = await invokeHook(() => embed(scoped.record));
+            if (embedded.isFailure()) {
+                const error = `fragment index rebuild: embedding '${edgeTargetKey(scoped.target)}' failed: ${embedded.message}`;
+                if (!lenient) {
+                    // A rollback that also fails is said out loud: the `'fail'` path
+                    // promises an empty index, and on a DURABLE table a botched rollback
+                    // survives the process.
+                    return failWithDetail(withRollbackNote(error, this._clear()), report());
+                }
+                skipped.push({ target: scoped.target, error });
+                continue;
+            }
+            // An empty array is this lane's decline, and it is still WRITTEN — the
+            // whole-record-replace is what clears any stale fragments.
+            const added = await this.addFragments(scoped.target, embedded.value);
+            if (added.isFailure()) {
+                const error = `fragment index rebuild: ${added.message}`;
+                if (!lenient) {
+                    return failWithDetail(withRollbackNote(error, this._clear()), report());
+                }
+                skipped.push({ target: scoped.target, error });
+                continue;
+            }
+            if (added.value === 0) {
+                tally(declined, kind);
+                continue;
+            }
+            tally(indexed, kind);
+            tally(fragments, kind, added.value);
+        }
+        return succeedWithDetail(report());
+    }
+    /**
+     * **Empties the rows; does NOT release the table's declared dimension.** That
+     * is a `vec0` constraint rather than a choice — the dimension is schema, and
+     * there is no `ALTER TABLE` for it — so a rebuild at a new dimension fails
+     * here where it would succeed on the in-memory sibling, which forgets its
+     * dimension on reset. Changing dimension needs a drop-and-re-index; see the
+     * note on `IVectorIndex.rebuild`. Tolerates a table that does not exist yet.
+     */
+    _clear() {
+        if (this._stmts === undefined) {
+            return succeed(true);
+        }
+        // Capture-wrapped like every other statement path: a closed connection or an
+        // I/O error is a `Failure`, not an exception out of a `Result`-returning method.
+        return captureResult(() => this._db.prepare(`DELETE FROM "${this._table}"`).run()).onSuccess(() => succeed(true));
+    }
     /** {@inheritDoc IFragmentVectorIndex.query} */
-    query(vector, topK, maxPerRecord) {
+    query(vector, topK, options) {
+        const maxPerRecord = options === null || options === void 0 ? void 0 : options.maxPerRecord;
+        const scope = options === null || options === void 0 ? void 0 : options.scope;
+        const id = options === null || options === void 0 ? void 0 : options.id;
         if (topK <= 0 || this._stmts === undefined) {
             return Promise.resolve(succeed([]));
         }
@@ -196,11 +344,33 @@ export class SqliteVecFragmentIndex {
             // capped record's later fragments are skipped), so fetch the full ranked set
             // and apply the cap + topK cut here — exactly as the in-memory index does.
             // Uncapped, KNN's own `k = topK` is already the answer.
-            const fetchK = maxPerRecord === undefined ? topK : Number(stmts.fragmentCount.get().c);
+            // A scope-only narrowing (a versioned kind's per-entity subtree) spans several
+            // records, and `target_key` equality cannot express a prefix, so it is applied
+            // over the full ranked set below. Correct either way — the caller's `topK` is
+            // applied to the NARROWED set, which is the property that matters — but only
+            // the single-record case gets the partition push-down.
+            const recordKey = scope !== undefined && id !== undefined ? edgeTargetKey({ scope, id }) : undefined;
+            // The cap forces the full ranked set ONLY when other records can fill from
+            // behind a capped one. Under a single-record narrowing every row belongs to
+            // that record, so the result is exactly `min(topK, maxPerRecord, fragments)`
+            // and those are the first rows KNN returns — `k = topK` suffices, and
+            // expanding to the table-wide `fragmentCount` would ask an
+            // already-partition-restricted query for far more rows than it can use.
+            const wholeSet = recordKey === undefined && (maxPerRecord !== undefined || scope !== undefined);
+            const fetchK = wholeSet
+                ? Number(stmts.fragmentCount.get().c)
+                : topK;
             if (fetchK <= 0) {
                 return [];
             }
-            const rows = stmts.query.all(SqliteVecFragmentIndex._toBlob(vector), fetchK);
+            const blob = SqliteVecFragmentIndex._toBlob(vector);
+            const rows = (recordKey !== undefined
+                ? stmts.queryScopedToRecord.all(blob, fetchK, recordKey)
+                : stmts.query.all(blob, fetchK));
+            // The scope prefix every record in `scope` shares. `edgeTargetKey` joins with
+            // a NUL, so this cannot collide with a longer scope that merely starts the
+            // same way.
+            const scopePrefix = scope !== undefined && recordKey === undefined ? `${scope}\0` : undefined;
             // sqlite-vec returns rows ascending by distance (nearest first); score is
             // `1 - cosineDistance`, so this order is already descending score.
             const hits = [];
@@ -209,6 +379,9 @@ export class SqliteVecFragmentIndex {
                 if (hits.length >= topK) {
                     break;
                 }
+                if (scopePrefix !== undefined && !row.target_key.startsWith(scopePrefix)) {
+                    continue;
+                }
                 if (maxPerRecord !== undefined) {
                     const used = (_a = perRecord.get(row.target_key)) !== null && _a !== void 0 ? _a : 0;
                     if (used >= maxPerRecord) {
@@ -259,8 +432,16 @@ export class SqliteVecFragmentIndex {
             },
             query: this._db.prepare(`SELECT target_key, start_off, end_off, fragment_id, distance FROM "${this._table}" ` +
                 `WHERE embedding MATCH ? AND k = ?`),
+            // The single-record narrowing constrains `target_key`, which is the table's
+            // PARTITION KEY — so this is a partition-restricted KNN rather than a scan
+            // plus a filter. That is the performance reason this narrowing belongs in the
+            // library instead of in a bigger over-fetch on the caller's side.
+            queryScopedToRecord: this._db.prepare(`SELECT target_key, start_off, end_off, fragment_id, distance FROM "${this._table}" ` +
+                `WHERE embedding MATCH ? AND k = ? AND target_key = ?`),
             fragmentCount: this._db.prepare(`SELECT count(*) AS c FROM "${this._table}"`),
-            recordCount: this._db.prepare(`SELECT count(DISTINCT target_key) AS c FROM "${this._table}"`)
+            recordCount: this._db.prepare(`SELECT count(DISTINCT target_key) AS c FROM "${this._table}"`),
+            // `LIMIT 1`: membership needs existence, not cardinality.
+            has: this._db.prepare(`SELECT 1 FROM "${this._table}" WHERE target_key = ? LIMIT 1`)
         };
     }
     /**