libsql-search 0.9.1 → 0.10.0

package/docs/API.md

@@ -52,6 +52,25 @@ It also exports these types:
 - [Migration and reindexing guide](./MIGRATIONS.md)
 - [Indexing and operational behavior](./INDEXING.md)
 - [Testing guidance](./TESTING.md)
+- [Turso Database backend](./TURSO.md)
+
+## The `client` argument
+
+Every function that talks to a database takes a `client`. It accepts either:
+
+- a `@libsql/client` `Client` — the default, and the only backend the main entry point references, or
+- an adapter from a backend entry point, currently `tursoAdapter()` from `libsql-search/turso`
+
+`IndexerOptions["client"]` and `SearchOptions["client"]` are typed as
+`Client | DatabaseAdapter`, so existing `Client`-typed code needs no change. The
+main entry point exports no new symbol for this and never imports a backend
+package other than `@libsql/client`.
+
+The adapter carries a capability flag that changes two behaviors on a backend
+without an approximate-nearest-neighbor vector index: `createTable()` skips the
+vector index, and `search()` uses the exact path automatically. Both are
+described in the [Turso Database backend guide](./TURSO.md), which is currently
+the only backend where they apply.
 
 ## `createTable(client, tableName?, dimensions?)`
 
@@ -61,6 +80,10 @@ Creates the search table and supporting indexes.
 await createTable(client);
 ```
 
+On a backend with no vector index support, the `<tableName>_embedding_idx`
+vector index is skipped; the table, the folder index, and the slug index are
+still created. On `@libsql/client` it is never skipped.
+
 Defaults:
 
 - `tableName`: `"articles"`
@@ -88,7 +111,7 @@ Indexes Markdown files from a directory on disk.
 
 ```ts
 interface IndexerOptions {
-  client: Client;
+  client: Client | DatabaseAdapter;
   contentPath: string;
   embeddingOptions?: EmbeddingOptions;
   fileExtensions?: string[];
@@ -185,7 +208,7 @@ Generates a query embedding and performs vector similarity search.
 
 ```ts
 interface SearchOptions {
-  client: Client;
+  client: Client | DatabaseAdapter;
   query: string;
   limit?: number;
   tableName?: string;
@@ -244,6 +267,8 @@ const results = await search({ client, query, exact: true });
 
 This is the guaranteed-exact path: it computes `vector_distance_cos` for every row with a non-`NULL` embedding, sorts by `(distance, id)`, and trims to `limit`. Cost grows linearly with table size, so it is intended for small corpora, correctness checks against the index path, and tables that have no vector index.
 
+A backend with no vector index at all takes this path whether or not `exact` is set, because there is no index path for it to fall back from. That is the case for `@tursodatabase/database`; see the [Turso Database backend guide](./TURSO.md).
+
 ### Missing Vector Index
 
 If the target table has no `<tableName>_embedding_idx`, the default path throws an error naming the missing index and pointing at `createTable()` and `exact: true`. libSQL's own message for this case ("failed to parse vector index parameters") says nothing about a missing index, so it is preserved as the thrown error's `cause` rather than surfaced directly.
@@ -297,7 +322,7 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
-All retrieval helpers validate `tableName` before executing SQL.
+All retrieval helpers validate `tableName` before executing SQL, and all of them accept either client kind.
 
 ## Embedding Helpers
 

package/docs/README.md

@@ -8,6 +8,7 @@ This directory holds the longer-form reference material for `libsql-search`.
 - [API reference](./API.md): exported functions, option shapes, and result data
 - [Indexing and operations](./INDEXING.md): content layout, rebuild behavior, and search quality notes
 - [Testing guidance](./TESTING.md): CI-safe mocks and no-live-call policy
+- [Turso Database backend](./TURSO.md): experimental exact-search-only support for the in-process `@tursodatabase/database` client
 - [Troubleshooting](./TROUBLESHOOTING.md): known install and runtime issues
 - [Releasing](./RELEASING.md): maintainer release workflow
 

package/docs/TURSO.md

@@ -0,0 +1,230 @@
+# Turso Database backend (experimental)
+
+`libsql-search` can run against the in-process [`@tursodatabase/database`](https://www.npmjs.com/package/@tursodatabase/database)
+client in addition to `@libsql/client`.
+
+**This backend is experimental, and search on it is exact-only.** Read
+[What is different on Turso](#what-is-different-on-turso) before adopting it —
+the difference is architectural, not a rough edge that will be smoothed out by
+an upgrade.
+
+`@libsql/client` remains the default and the only backend the main entry point
+knows about. Nothing on this page affects you if you do not import
+`libsql-search/turso`.
+
+## Install
+
+`@tursodatabase/database` is an **optional** peer dependency, declared as
+`^0.7.0 || ^0.8.0`. It is not installed, resolved, or imported unless you ask
+for it. Both lines behave identically on every dimension this adapter depends
+on; the disjunction is a caret-per-minor range for the same reason
+`@libsql/client`'s is.
+
+```bash
+pnpm add libsql-search @tursodatabase/database
+```
+
+```bash
+npm install libsql-search @tursodatabase/database
+```
+
+## Usage
+
+Wrap the connected handle with `tursoAdapter()` and pass the result as `client`
+everywhere you would otherwise pass a libSQL client.
+
+```ts
+import { connect } from "@tursodatabase/database";
+import { tursoAdapter } from "libsql-search/turso";
+import { createTable, indexContent, search } from "libsql-search";
+
+// connect() is async. Awaiting it is not optional — tursoAdapter() rejects a
+// pending Promise with a message saying so, because it is the most common
+// first-use mistake.
+const database = await connect("./local.db");
+const client = tursoAdapter(database);
+
+await createTable(client, "articles", 384);
+
+await indexContent({
+  client,
+  contentPath: "./content",
+  embeddingOptions: { provider: "local" },
+});
+
+const results = await search({
+  client,
+  query: "how do I deploy my docs site",
+  limit: 5,
+  embeddingOptions: { provider: "local" },
+});
+```
+
+Use `":memory:"` instead of a file path for an ephemeral database.
+
+`createTable`, `indexContent`, `search`, `getAllArticles`, `getArticleBySlug`,
+`getArticlesByFolder`, and `getFolders` all accept the adapter. The option and
+result shapes — `SearchOptions`, `SearchResult`, `IndexerOptions`,
+`IndexResult` — are identical on both backends.
+
+## What is different on Turso
+
+### There is no ANN vector index, so search is a full scan
+
+Turso Database implements the vector *functions* — `vector()`, `vector32()`,
+`vector_distance_cos()` — but not the approximate-nearest-neighbor index that
+libSQL exposes through `libsql_vector_idx()` and `vector_top_k()`. Attempting
+either produces a parse error and a "no such table: vector_top_k" respectively.
+
+Two consequences, both handled for you:
+
+- `createTable()` creates the table, the folder index, and the slug index, and
+  **skips** the vector index. It does not skip it on `@libsql/client`.
+- `search()` runs the exact full-scan path automatically. You do not pass
+  `exact: true`, and passing it changes nothing.
+
+**Search cost is therefore linear in the number of indexed rows.** For a
+personal site or a docs set this is generally fine and often faster than an
+index probe. For a large corpus, use `@libsql/client`, which is not affected by
+any of this.
+
+This is a design direction rather than an unshipped feature:
+
+- [`tursodatabase/turso` #832](https://github.com/tursodatabase/turso/issues/832),
+  the DiskANN port, is open, unassigned, and in the Backlog milestone.
+- [`tursodatabase/turso` #3778](https://github.com/tursodatabase/turso/issues/3778)
+  was closed as *completed* by shipping SIMD-accelerated **exact** vector search,
+  with the maintainer noting that "fast exact search is what many use cases
+  actually want".
+- The project's `COMPAT.md` lists the vector functions as supported and has no
+  row for `libsql_vector_idx` or `vector_top_k` at all.
+
+One thing worth knowing, because it costs people an afternoon: Turso's own
+**AI & Embeddings** documentation page markets `vector_top_k` as a "Turso and
+libSQL" feature without distinguishing the two engines. If you arrived expecting
+it to work on `@tursodatabase/database`, that page is why. It does not.
+
+### Index replacement uses a transaction, not a batch
+
+`indexContent()` builds every document in memory and then replaces the table
+contents in one atomic write, so a failed rebuild leaves the previous index
+intact. That guarantee holds identically on both backends, but the primitive
+behind it is inverted between them:
+
+| | `@libsql/client` | `@tursodatabase/database` |
+| --- | --- | --- |
+| `batch(statements, "write")` | atomic | **not atomic** |
+| `transaction()` | breaks in-memory clients | atomic, but **deferred** — see below |
+
+Turso's `batch()` is not transactional: a batch that fails part way through
+leaves the statements before the failure committed. On this backend the
+replacement therefore runs inside an explicit `BEGIN IMMEDIATE` / `COMMIT`, with
+an explicit `ROLLBACK` on failure. If you are extending this library, do not
+"simplify" the Turso path to use `batch()` — it would silently convert a failed
+rebuild into a half-destroyed index.
+
+**Nor is `transaction()` the shortcut it looks like.** Turso's `transaction()`
+is better-sqlite3-style: it *returns a wrapped function* rather than executing.
+`await db.transaction(fn)` resolves to that function and never runs `fn` — no
+error, no rows written. That is why this adapter issues `BEGIN IMMEDIATE`
+itself.
+
+This one is genuinely dangerous to get wrong, because it fails green. Rewriting
+`executeAtomicWrite()` as `await database.transaction(async () => { ... })`
+throws nothing and **still passes every test in the suite**, including the one
+asserting that a failed rebuild leaves the previous index intact — an index that
+was never touched trivially satisfies "unchanged". The reindex becomes a silent
+no-op in production while CI stays green. `tests/turso-database.test.ts` pins
+the deferred-execution behavior directly for this reason.
+
+## Platform support
+
+`@tursodatabase/database` ships prebuilt native binaries for these targets only:
+
+- `darwin-arm64` (Apple Silicon)
+- `linux-x64-gnu`
+- `linux-arm64-gnu`
+- `win32-x64-msvc`
+
+Two gaps follow from that list, and Turso's documentation does not address
+either one, so this is an observation about the published binaries rather than a
+statement of their support policy:
+
+- **No musl target.** Alpine-based images are not covered by a prebuilt binary.
+- **No `darwin-x64`.** Intel Macs are not covered either.
+
+`@libsql/client` has no such constraint, which is another reason it stays the
+default.
+
+The integration suite in `tests/turso-database.test.ts` skips itself — with a
+warning naming the platform and the reason — anywhere the native module fails to
+load, rather than failing the run. The capability-driven behavior it covers
+(skipping the vector index, forcing the exact search path, routing the
+replacement through the atomic write) is additionally covered on every platform
+by `tests/database.test.ts`, which uses a stub adapter and no native code.
+
+Tests never require Turso Cloud credentials, a URL, or a token. Everything runs
+against an in-process `:memory:` database.
+
+## Deno and JSR
+
+The JSR package exports only the main entry point. `libsql-search/turso` is not
+available there.
+
+That is a scope decision, not a technical limit. `src/turso.ts` imports nothing
+from `@tursodatabase/database` — it accepts the handle through a structural
+`TursoDatabase` interface declared in this package — so exposing it on JSR would
+not force Deno to resolve a Node-native package. It is left off because the
+backend is experimental and the native binaries target Node platforms. The entry
+point is still type-checked by `deno task check` so the claim above stays true.
+
+## Type compatibility
+
+`IndexerOptions["client"]` and `SearchOptions["client"]` accept either a
+`@libsql/client` `Client` or an adapter. Existing `Client`-typed code compiles
+unchanged, and the main entry point exports no new symbol and references no
+Turso type.
+
+`tursoAdapter()` returns a `DatabaseAdapter`, and that type is exported from
+`libsql-search/turso` so you can name it:
+
+```ts
+import { tursoAdapter, type DatabaseAdapter } from "libsql-search/turso";
+
+let client: DatabaseAdapter;
+```
+
+It is exported from the subpath only. The main entry point does not export it,
+so `libsql-search`'s public surface is unchanged for everyone else. An export
+modifier is not a structural member, so the asymmetry does not affect
+assignability between the two entry points.
+
+`libsql-search` and `libsql-search/turso` are bundled separately, so each ships
+its own structural copy of the adapter declaration. `pnpm check:dist-types`
+compiles an adapter from the subpath against the client type from the main entry
+to prove the two stay interchangeable, and runs as part of
+`pnpm validate:package`.
+
+## If you are extending this adapter
+
+Three things in `src/turso.ts` look like noise and are not. Each has a
+regression test; none of them fails loudly at runtime if removed.
+
+**Prepared statements must be closed.** A statement holds native memory that the
+garbage collector cannot reclaim, because it is not JavaScript heap. Every
+`prepare()` is released with `close()` in a `finally`. Measured through the
+built bundle, 60 000 `executeQuery()` calls on one handle grow RSS by ~570 MB
+without the close and ~150 MB with it. `search()` issues exactly one query, so
+an SSR site calling it per request is the case that turns this from untidy into
+an OOM. Inside `executeAtomicWrite()` the cached statements are released only
+*after* `COMMIT` or `ROLLBACK` — a statement stays bound to the transaction
+while it is open.
+
+**`BEGIN IMMEDIATE` sits outside the `try`.** If it were inside, a `BEGIN` that
+fails because another rebuild already holds the write lock would fall into the
+`catch` and issue a `ROLLBACK`, ending the *other* call's in-flight transaction
+and destroying a good rebuild that was about to commit. Failing to start a
+transaction must never end one.
+
+**Do not reach for `batch()` or `transaction()`.** Both are covered above; both
+fail silently rather than loudly.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.9.1",
+  "version": "0.10.0",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",
@@ -11,6 +11,11 @@
       "types": "./dist/index.d.ts",
       "import": "./dist/index.esm.js",
       "require": "./dist/index.cjs"
+    },
+    "./turso": {
+      "types": "./dist/turso.d.ts",
+      "import": "./dist/turso.esm.js",
+      "require": "./dist/turso.cjs"
     }
   },
   "files": [
@@ -29,9 +34,10 @@
     "test:coverage": "vitest run --coverage",
     "test:release-plan": "node --test ./scripts/plan-release.node-test.mjs",
     "check:deno": "deno task check",
+    "check:dist-types": "node ./scripts/check-dist-types.mjs",
     "smoke:package": "node ./scripts/smoke-package.mjs",
     "validate:release-version": "node ./scripts/validate-release-version.mjs",
-    "validate:package": "tsc --noEmit && rollup -c && rollup -c rollup.dts.config.js && node ./scripts/smoke-package.mjs",
+    "validate:package": "tsc --noEmit && rollup -c && rollup -c rollup.dts.config.js && node ./scripts/check-dist-types.mjs && node ./scripts/smoke-package.mjs",
     "validate:release": "pnpm audit --audit-level moderate && pnpm test:release-plan && pnpm test:coverage && pnpm validate:package && pnpm check:deno",
     "prepublishOnly": "pnpm validate:release"
   },
@@ -60,7 +66,13 @@
     "node": ">=22.12.0"
   },
   "peerDependencies": {
-    "@libsql/client": "^0.15.0 || ^0.17.0"
+    "@libsql/client": "^0.15.0 || ^0.17.0",
+    "@tursodatabase/database": "^0.7.0 || ^0.8.0"
+  },
+  "peerDependenciesMeta": {
+    "@tursodatabase/database": {
+      "optional": true
+    }
   },
   "dependencies": {
     "@huggingface/transformers": "4.2.0",
@@ -70,6 +82,7 @@
     "@libsql/client": "^0.17.4",
     "@rollup/plugin-commonjs": "^29.0.3",
     "@rollup/plugin-node-resolve": "^16.0.3",
+    "@tursodatabase/database": "^0.7.2",
     "@types/node": "^24.13.3",
     "@vitest/coverage-v8": "4.1.11",
     "@vitest/ui": "^4.1.11",