libsql-search 0.9.0 → 0.10.0

package/docs/API.md

@@ -52,6 +52,25 @@ It also exports these types:
 - [Migration and reindexing guide](./MIGRATIONS.md)
 - [Indexing and operational behavior](./INDEXING.md)
 - [Testing guidance](./TESTING.md)
+- [Turso Database backend](./TURSO.md)
+
+## The `client` argument
+
+Every function that talks to a database takes a `client`. It accepts either:
+
+- a `@libsql/client` `Client` — the default, and the only backend the main entry point references, or
+- an adapter from a backend entry point, currently `tursoAdapter()` from `libsql-search/turso`
+
+`IndexerOptions["client"]` and `SearchOptions["client"]` are typed as
+`Client | DatabaseAdapter`, so existing `Client`-typed code needs no change. The
+main entry point exports no new symbol for this and never imports a backend
+package other than `@libsql/client`.
+
+The adapter carries a capability flag that changes two behaviors on a backend
+without an approximate-nearest-neighbor vector index: `createTable()` skips the
+vector index, and `search()` uses the exact path automatically. Both are
+described in the [Turso Database backend guide](./TURSO.md), which is currently
+the only backend where they apply.
 
 ## `createTable(client, tableName?, dimensions?)`
 
@@ -61,6 +80,10 @@ Creates the search table and supporting indexes.
 await createTable(client);
 ```
 
+On a backend with no vector index support, the `<tableName>_embedding_idx`
+vector index is skipped; the table, the folder index, and the slug index are
+still created. On `@libsql/client` it is never skipped.
+
 Defaults:
 
 - `tableName`: `"articles"`
@@ -88,7 +111,7 @@ Indexes Markdown files from a directory on disk.
 
 ```ts
 interface IndexerOptions {
-  client: Client;
+  client: Client | DatabaseAdapter;
   contentPath: string;
   embeddingOptions?: EmbeddingOptions;
   fileExtensions?: string[];
@@ -185,7 +208,7 @@ Generates a query embedding and performs vector similarity search.
 
 ```ts
 interface SearchOptions {
-  client: Client;
+  client: Client | DatabaseAdapter;
   query: string;
   limit?: number;
   tableName?: string;
@@ -244,6 +267,8 @@ const results = await search({ client, query, exact: true });
 
 This is the guaranteed-exact path: it computes `vector_distance_cos` for every row with a non-`NULL` embedding, sorts by `(distance, id)`, and trims to `limit`. Cost grows linearly with table size, so it is intended for small corpora, correctness checks against the index path, and tables that have no vector index.
 
+A backend with no vector index at all takes this path whether or not `exact` is set, because there is no index path for it to fall back from. That is the case for `@tursodatabase/database`; see the [Turso Database backend guide](./TURSO.md).
+
 ### Missing Vector Index
 
 If the target table has no `<tableName>_embedding_idx`, the default path throws an error naming the missing index and pointing at `createTable()` and `exact: true`. libSQL's own message for this case ("failed to parse vector index parameters") says nothing about a missing index, so it is preserved as the thrown error's `cause` rather than surfaced directly.
@@ -256,7 +281,9 @@ Search never falls back to the exact scan on its own. A silent fallback would tu
 
 ### Requirements
 
-`vector_top_k()` and `libsql_vector_idx()` require a libSQL build with native vector support. The peer dependency is `@libsql/client ^0.15.0`; this behavior is verified against `@libsql/client` `0.15.15` with a local `:memory:` database. Remote Turso/libSQL servers must also support vector indexes — that is a property of the server, not the client, and no minimum server version is claimed here beyond that requirement. A deployment without it fails the default path with an error saying so and naming `exact: true` as the remedy. Use `exact: true` against any deployment where vector index support is unavailable or unverified.
+`vector_top_k()` and `libsql_vector_idx()` require a libSQL build with native vector support. The peer dependency is `@libsql/client ^0.15.0 || ^0.17.0`; this behavior is verified against `@libsql/client` `0.15.15` and `0.17.4` with a local `:memory:` database. On both, `vector_top_k()` returns the matched rowids in an `id` column, libSQL's missing-index wording ("failed to parse vector index parameters") is byte-identical, and so is the dimension-mismatch wording that is passed through unchanged — so the index path and the diagnostics above behave the same on either line. The two lines share one embedded engine (`libsql` `0.5.29`), which is why the on-disk format and index semantics do not differ between them.
+
+Coverage differs by layer, and is worth stating plainly: the packaged build is smoke-tested against **both** peer arms on every release, which covers table and vector-index creation. The full test suite — including the byte-exact assertions on the messages above — runs against the dev-pinned client, currently `0.17.4`. The no-vector-support case is the one message not reproducible against a local build on either version; it is asserted from a synthesized error rather than a measured one. Remote Turso/libSQL servers must also support vector indexes — that is a property of the server, not the client, and no minimum server version is claimed here beyond that requirement. A deployment without it fails the default path with an error saying so and naming `exact: true` as the remedy. Use `exact: true` against any deployment where vector index support is unavailable or unverified.
 
 Result shape:
 
@@ -295,7 +322,7 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
-All retrieval helpers validate `tableName` before executing SQL.
+All retrieval helpers validate `tableName` before executing SQL, and all of them accept either client kind.
 
 ## Embedding Helpers
 

package/docs/INDEXING.md

@@ -174,6 +174,6 @@ Many projects wire indexing into a dedicated script and call it before their sit
 ## Runtime Notes
 
 - local embeddings may download and cache a model on the first run
-- Node users need `@libsql/client` installed alongside the package
+- Node users need `@libsql/client` installed alongside the package, at `^0.15.0 || ^0.17.0`; the packaged build is smoke-tested against both arms (`0.15.15` and `0.17.4`), which covers table and vector-index creation. `batch()` rollback behaves identically on both at the contract level, though its error text differs — see [Version differences](./TROUBLESHOOTING.md#libsqlclient-version-differences). Upgrading the client is not a prerequisite for upgrading this package. Deno/JSR users are not covered by that range and should pin the client themselves — see [Install](../README.md#install)
 - hosted providers send indexed or queried text to external services
 - the repository validates package build and `deno check`, but indexing still depends on filesystem access

package/docs/MIGRATIONS.md

@@ -23,6 +23,7 @@ References:
 - if dimensions stay the same but provider, model, endpoint, model revision, pooling, normalization, or input formatting changes, fully reindex anyway
 - never mix two embedding spaces in one table
 - prefer a parallel table migration because `indexContent()` replaces the whole target table, so an in-place rebuild leaves no way back to the old vectors
+- upgrading `@libsql/client` is not one of these migrations. The client moves bytes; it does not define the embedding space. Moving between the supported `^0.15.0` and `^0.17.0` lines leaves stored vectors, table widths, and the embedding index untouched and needs no reindex. This is checkable rather than merely inferred: `libsql`, the embedded native engine that owns the on-disk `F32_BLOB` format and the vector index, resolves to `0.5.29` under both client lines — the client bump does not move it. See [`@libsql/client` version differences](./TROUBLESHOOTING.md#libsqlclient-version-differences) for the two client-side behaviors that do change.
 
 In practice, this means:
 

package/docs/README.md

@@ -8,6 +8,7 @@ This directory holds the longer-form reference material for `libsql-search`.
 - [API reference](./API.md): exported functions, option shapes, and result data
 - [Indexing and operations](./INDEXING.md): content layout, rebuild behavior, and search quality notes
 - [Testing guidance](./TESTING.md): CI-safe mocks and no-live-call policy
+- [Turso Database backend](./TURSO.md): experimental exact-search-only support for the in-process `@tursodatabase/database` client
 - [Troubleshooting](./TROUBLESHOOTING.md): known install and runtime issues
 - [Releasing](./RELEASING.md): maintainer release workflow
 

package/docs/TROUBLESHOOTING.md

@@ -38,3 +38,41 @@ Common operational checks:
   `createTable()`, `indexContent()`, and `search()`, and re-index if the stored
   vectors are in the wrong space; see the
   [Migration and reindexing guide](./MIGRATIONS.md)
+
+## `@libsql/client` Version Differences
+
+The peer range is `^0.15.0 || ^0.17.0`, verified against `0.15.15` and `0.17.4`.
+There is no `0.16.x` line upstream. Everything this package depends on behaves
+the same on both — `vector_top_k()` returns rowids in an `id` column, the
+missing-index message this library rewrites ("failed to parse vector index
+parameters") and the dimension-mismatch message it passes through unchanged are
+both byte-identical, and `batch(..., "write")` still rolls the whole rebuild back
+on failure — so upgrading the client is optional and neither direction requires
+re-indexing. (The third message, the no-vector-support case, is not reproducible
+against a local build on either version; see
+[Requirements](./API.md#requirements).)
+
+Two client-side differences are visible to callers on `0.17.x`:
+
+- the client no longer exports `./package.json`. Reading it by specifier, as in
+  `require("@libsql/client/package.json")`, throws
+  `ERR_PACKAGE_PATH_NOT_EXPORTED`. Nothing in this package does that; if your own
+  tooling did, read the file through a direct `node_modules` path instead
+- **constraint error codes lost the `_UNIQUE` suffix.** A duplicate slug that
+  reported `SQLITE_CONSTRAINT_UNIQUE` on `0.15.x` reports the broader
+  `SQLITE_CONSTRAINT` on `0.17.x`. This affects every caller on both query paths:
+
+  | | `0.15.15` | `0.17.4` |
+  | --- | --- | --- |
+  | `execute()` | `SQLITE_CONSTRAINT_UNIQUE: UNIQUE constraint failed: …` | `SQLITE_CONSTRAINT: UNIQUE constraint failed: …` |
+  | `batch(…, "write")` | `SQLITE_CONSTRAINT_UNIQUE: UNIQUE constraint failed: …` | `SQLITE_CONSTRAINT: SQLITE_CONSTRAINT: UNIQUE constraint failed: …` |
+
+  Note that the prefix is additionally **doubled on the `batch()` path only** —
+  that is the path `indexContent()` uses, so it is what surfaces as the `cause`
+  of an `IndexingError` with `phase: "replace"`. Your own `execute()` calls show
+  the single prefix. Both are cosmetic: the rollback and the `IndexingError`
+  contract are unchanged.
+
+  A log matcher or alert rule keyed on `SQLITE_CONSTRAINT_UNIQUE` will stop
+  matching after the upgrade, on either path. Match on `UNIQUE constraint failed`
+  instead — it is the one substring stable across all four cells above.

package/docs/TURSO.md

@@ -0,0 +1,230 @@
+# Turso Database backend (experimental)
+
+`libsql-search` can run against the in-process [`@tursodatabase/database`](https://www.npmjs.com/package/@tursodatabase/database)
+client in addition to `@libsql/client`.
+
+**This backend is experimental, and search on it is exact-only.** Read
+[What is different on Turso](#what-is-different-on-turso) before adopting it —
+the difference is architectural, not a rough edge that will be smoothed out by
+an upgrade.
+
+`@libsql/client` remains the default and the only backend the main entry point
+knows about. Nothing on this page affects you if you do not import
+`libsql-search/turso`.
+
+## Install
+
+`@tursodatabase/database` is an **optional** peer dependency, declared as
+`^0.7.0 || ^0.8.0`. It is not installed, resolved, or imported unless you ask
+for it. Both lines behave identically on every dimension this adapter depends
+on; the disjunction is a caret-per-minor range for the same reason
+`@libsql/client`'s is.
+
+```bash
+pnpm add libsql-search @tursodatabase/database
+```
+
+```bash
+npm install libsql-search @tursodatabase/database
+```
+
+## Usage
+
+Wrap the connected handle with `tursoAdapter()` and pass the result as `client`
+everywhere you would otherwise pass a libSQL client.
+
+```ts
+import { connect } from "@tursodatabase/database";
+import { tursoAdapter } from "libsql-search/turso";
+import { createTable, indexContent, search } from "libsql-search";
+
+// connect() is async. Awaiting it is not optional — tursoAdapter() rejects a
+// pending Promise with a message saying so, because it is the most common
+// first-use mistake.
+const database = await connect("./local.db");
+const client = tursoAdapter(database);
+
+await createTable(client, "articles", 384);
+
+await indexContent({
+  client,
+  contentPath: "./content",
+  embeddingOptions: { provider: "local" },
+});
+
+const results = await search({
+  client,
+  query: "how do I deploy my docs site",
+  limit: 5,
+  embeddingOptions: { provider: "local" },
+});
+```
+
+Use `":memory:"` instead of a file path for an ephemeral database.
+
+`createTable`, `indexContent`, `search`, `getAllArticles`, `getArticleBySlug`,
+`getArticlesByFolder`, and `getFolders` all accept the adapter. The option and
+result shapes — `SearchOptions`, `SearchResult`, `IndexerOptions`,
+`IndexResult` — are identical on both backends.
+
+## What is different on Turso
+
+### There is no ANN vector index, so search is a full scan
+
+Turso Database implements the vector *functions* — `vector()`, `vector32()`,
+`vector_distance_cos()` — but not the approximate-nearest-neighbor index that
+libSQL exposes through `libsql_vector_idx()` and `vector_top_k()`. Attempting
+either produces a parse error and a "no such table: vector_top_k" respectively.
+
+Two consequences, both handled for you:
+
+- `createTable()` creates the table, the folder index, and the slug index, and
+  **skips** the vector index. It does not skip it on `@libsql/client`.
+- `search()` runs the exact full-scan path automatically. You do not pass
+  `exact: true`, and passing it changes nothing.
+
+**Search cost is therefore linear in the number of indexed rows.** For a
+personal site or a docs set this is generally fine and often faster than an
+index probe. For a large corpus, use `@libsql/client`, which is not affected by
+any of this.
+
+This is a design direction rather than an unshipped feature:
+
+- [`tursodatabase/turso` #832](https://github.com/tursodatabase/turso/issues/832),
+  the DiskANN port, is open, unassigned, and in the Backlog milestone.
+- [`tursodatabase/turso` #3778](https://github.com/tursodatabase/turso/issues/3778)
+  was closed as *completed* by shipping SIMD-accelerated **exact** vector search,
+  with the maintainer noting that "fast exact search is what many use cases
+  actually want".
+- The project's `COMPAT.md` lists the vector functions as supported and has no
+  row for `libsql_vector_idx` or `vector_top_k` at all.
+
+One thing worth knowing, because it costs people an afternoon: Turso's own
+**AI & Embeddings** documentation page markets `vector_top_k` as a "Turso and
+libSQL" feature without distinguishing the two engines. If you arrived expecting
+it to work on `@tursodatabase/database`, that page is why. It does not.
+
+### Index replacement uses a transaction, not a batch
+
+`indexContent()` builds every document in memory and then replaces the table
+contents in one atomic write, so a failed rebuild leaves the previous index
+intact. That guarantee holds identically on both backends, but the primitive
+behind it is inverted between them:
+
+| | `@libsql/client` | `@tursodatabase/database` |
+| --- | --- | --- |
+| `batch(statements, "write")` | atomic | **not atomic** |
+| `transaction()` | breaks in-memory clients | atomic, but **deferred** — see below |
+
+Turso's `batch()` is not transactional: a batch that fails part way through
+leaves the statements before the failure committed. On this backend the
+replacement therefore runs inside an explicit `BEGIN IMMEDIATE` / `COMMIT`, with
+an explicit `ROLLBACK` on failure. If you are extending this library, do not
+"simplify" the Turso path to use `batch()` — it would silently convert a failed
+rebuild into a half-destroyed index.
+
+**Nor is `transaction()` the shortcut it looks like.** Turso's `transaction()`
+is better-sqlite3-style: it *returns a wrapped function* rather than executing.
+`await db.transaction(fn)` resolves to that function and never runs `fn` — no
+error, no rows written. That is why this adapter issues `BEGIN IMMEDIATE`
+itself.
+
+This one is genuinely dangerous to get wrong, because it fails green. Rewriting
+`executeAtomicWrite()` as `await database.transaction(async () => { ... })`
+throws nothing and **still passes every test in the suite**, including the one
+asserting that a failed rebuild leaves the previous index intact — an index that
+was never touched trivially satisfies "unchanged". The reindex becomes a silent
+no-op in production while CI stays green. `tests/turso-database.test.ts` pins
+the deferred-execution behavior directly for this reason.
+
+## Platform support
+
+`@tursodatabase/database` ships prebuilt native binaries for these targets only:
+
+- `darwin-arm64` (Apple Silicon)
+- `linux-x64-gnu`
+- `linux-arm64-gnu`
+- `win32-x64-msvc`
+
+Two gaps follow from that list, and Turso's documentation does not address
+either one, so this is an observation about the published binaries rather than a
+statement of their support policy:
+
+- **No musl target.** Alpine-based images are not covered by a prebuilt binary.
+- **No `darwin-x64`.** Intel Macs are not covered either.
+
+`@libsql/client` has no such constraint, which is another reason it stays the
+default.
+
+The integration suite in `tests/turso-database.test.ts` skips itself — with a
+warning naming the platform and the reason — anywhere the native module fails to
+load, rather than failing the run. The capability-driven behavior it covers
+(skipping the vector index, forcing the exact search path, routing the
+replacement through the atomic write) is additionally covered on every platform
+by `tests/database.test.ts`, which uses a stub adapter and no native code.
+
+Tests never require Turso Cloud credentials, a URL, or a token. Everything runs
+against an in-process `:memory:` database.
+
+## Deno and JSR
+
+The JSR package exports only the main entry point. `libsql-search/turso` is not
+available there.
+
+That is a scope decision, not a technical limit. `src/turso.ts` imports nothing
+from `@tursodatabase/database` — it accepts the handle through a structural
+`TursoDatabase` interface declared in this package — so exposing it on JSR would
+not force Deno to resolve a Node-native package. It is left off because the
+backend is experimental and the native binaries target Node platforms. The entry
+point is still type-checked by `deno task check` so the claim above stays true.
+
+## Type compatibility
+
+`IndexerOptions["client"]` and `SearchOptions["client"]` accept either a
+`@libsql/client` `Client` or an adapter. Existing `Client`-typed code compiles
+unchanged, and the main entry point exports no new symbol and references no
+Turso type.
+
+`tursoAdapter()` returns a `DatabaseAdapter`, and that type is exported from
+`libsql-search/turso` so you can name it:
+
+```ts
+import { tursoAdapter, type DatabaseAdapter } from "libsql-search/turso";
+
+let client: DatabaseAdapter;
+```
+
+It is exported from the subpath only. The main entry point does not export it,
+so `libsql-search`'s public surface is unchanged for everyone else. An export
+modifier is not a structural member, so the asymmetry does not affect
+assignability between the two entry points.
+
+`libsql-search` and `libsql-search/turso` are bundled separately, so each ships
+its own structural copy of the adapter declaration. `pnpm check:dist-types`
+compiles an adapter from the subpath against the client type from the main entry
+to prove the two stay interchangeable, and runs as part of
+`pnpm validate:package`.
+
+## If you are extending this adapter
+
+Three things in `src/turso.ts` look like noise and are not. Each has a
+regression test; none of them fails loudly at runtime if removed.
+
+**Prepared statements must be closed.** A statement holds native memory that the
+garbage collector cannot reclaim, because it is not JavaScript heap. Every
+`prepare()` is released with `close()` in a `finally`. Measured through the
+built bundle, 60 000 `executeQuery()` calls on one handle grow RSS by ~570 MB
+without the close and ~150 MB with it. `search()` issues exactly one query, so
+an SSR site calling it per request is the case that turns this from untidy into
+an OOM. Inside `executeAtomicWrite()` the cached statements are released only
+*after* `COMMIT` or `ROLLBACK` — a statement stays bound to the transaction
+while it is open.
+
+**`BEGIN IMMEDIATE` sits outside the `try`.** If it were inside, a `BEGIN` that
+fails because another rebuild already holds the write lock would fall into the
+`catch` and issue a `ROLLBACK`, ending the *other* call's in-flight transaction
+and destroying a good rebuild that was about to commit. Failing to start a
+transaction must never end one.
+
+**Do not reach for `batch()` or `transaction()`.** Both are covered above; both
+fail silently rather than loudly.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.9.0",
+  "version": "0.10.0",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",
@@ -11,6 +11,11 @@
       "types": "./dist/index.d.ts",
       "import": "./dist/index.esm.js",
       "require": "./dist/index.cjs"
+    },
+    "./turso": {
+      "types": "./dist/turso.d.ts",
+      "import": "./dist/turso.esm.js",
+      "require": "./dist/turso.cjs"
     }
   },
   "files": [
@@ -29,9 +34,10 @@
     "test:coverage": "vitest run --coverage",
     "test:release-plan": "node --test ./scripts/plan-release.node-test.mjs",
     "check:deno": "deno task check",
+    "check:dist-types": "node ./scripts/check-dist-types.mjs",
     "smoke:package": "node ./scripts/smoke-package.mjs",
     "validate:release-version": "node ./scripts/validate-release-version.mjs",
-    "validate:package": "tsc --noEmit && rollup -c && rollup -c rollup.dts.config.js && node ./scripts/smoke-package.mjs",
+    "validate:package": "tsc --noEmit && rollup -c && rollup -c rollup.dts.config.js && node ./scripts/check-dist-types.mjs && node ./scripts/smoke-package.mjs",
     "validate:release": "pnpm audit --audit-level moderate && pnpm test:release-plan && pnpm test:coverage && pnpm validate:package && pnpm check:deno",
     "prepublishOnly": "pnpm validate:release"
   },
@@ -60,16 +66,23 @@
     "node": ">=22.12.0"
   },
   "peerDependencies": {
-    "@libsql/client": "^0.15.0"
+    "@libsql/client": "^0.15.0 || ^0.17.0",
+    "@tursodatabase/database": "^0.7.0 || ^0.8.0"
+  },
+  "peerDependenciesMeta": {
+    "@tursodatabase/database": {
+      "optional": true
+    }
   },
   "dependencies": {
     "@huggingface/transformers": "4.2.0",
     "gray-matter": "^4.0.3"
   },
   "devDependencies": {
-    "@libsql/client": "^0.15.15",
+    "@libsql/client": "^0.17.4",
     "@rollup/plugin-commonjs": "^29.0.3",
     "@rollup/plugin-node-resolve": "^16.0.3",
+    "@tursodatabase/database": "^0.7.2",
     "@types/node": "^24.13.3",
     "@vitest/coverage-v8": "4.1.11",
     "@vitest/ui": "^4.1.11",