libsql-search 0.10.1 → 0.11.1

package/docs/MIGRATIONS.md

@@ -27,7 +27,7 @@ References:
 
 In practice, this means:
 
-- `384 local` and `1024 Mistral` can never share a table
+- a legacy `384` embedding space and `1024 Mistral` can never share a table
 - `1024 Cloudflare` and `1024 Mistral` still need separate rebuilds because equal width does not make the vectors compatible
 - a custom endpoint change at the same width still needs a new table because the model or serving stack may have changed
 
@@ -84,14 +84,14 @@ Re-running `createTable()` with the table's existing width is the fix. It is ide
 
 ```ts
 // Same name and same width as the existing table
-await createTable(client, "articles_local_384", 384);
+await createTable(client, "articles_legacy_384", 384);
 ```
 
 Equivalently, in SQL:
 
 ```sql
-CREATE INDEX IF NOT EXISTS "articles_local_384_embedding_idx"
-ON "articles_local_384"(libsql_vector_idx(embedding));
+CREATE INDEX IF NOT EXISTS "articles_legacy_384_embedding_idx"
+ON "articles_legacy_384"(libsql_vector_idx(embedding));
 ```
 
 Reindexing does not create the index; `indexContent()` only replaces rows. Any new table created by `createTable()` as part of a migration already has it, so this applies only to pre-existing tables you are carrying forward. Until the index exists, `search({ ..., exact: true })` keeps queries working on the exact full-scan path.
@@ -100,7 +100,7 @@ Reindexing does not create the index; `indexContent()` only replaces rows. Any n
 
 | From | To | Why a rebuild is required | Recommended table move |
 | --- | --- | --- | --- |
-| Legacy padded local `768` | Native local `384` | Old tables stored `384` model values plus zero padding; current local provider is a native `384`-dimension space | Build into `articles_local_384`, validate, then retire the legacy table |
+| Legacy in-process Transformers.js index | Any external provider | The runtime and provider were removed; every replacement service has its own embedding space | Build a parallel table named for the external provider/model, validate, then retire the legacy table |
 | Any `768` space | Any `1024` space | Width changes from `F32_BLOB(768)` to `F32_BLOB(1024)` | Create a new `1024` table and reindex |
 | Cloudflare `1024` | Mistral `1024` | Width stays the same, but provider/model space changes | Use a parallel `1024` table such as `articles_mistral_1024` |
 | Mistral `1024` | Cloudflare `1024` | Same reason in reverse | Use a parallel `1024` table such as `articles_cf_bgem3_1024` |
@@ -111,21 +111,21 @@ Reindexing does not create the index; `indexContent()` only replaces rows. Any n
 
 ## Scenario Notes
 
-### Legacy Local `768` To Native Local `384`
+### Legacy In-Process Embeddings To An External Service
 
-Earlier local migrations sometimes relied on zero padding to fit a `768`-wide table. The current local adapter emits the model's native `384` dimensions and rejects any other local dimension count.
+Versions that exposed the in-process Transformers.js provider produced a separate embedding space that this release can no longer query. Choose an external provider or separately deployed OpenAI-compatible service and rebuild every vector into a new table.
 
 Safe path:
 
 ```ts
-await createTable(client, "articles_local_384", 384);
+await createTable(client, "articles_bge_1024", 1024);
 ```
 
-Reindex into `articles_local_384`; do not keep writing new local vectors into the legacy padded table.
+Reindex into `articles_bge_1024` with the external service configuration; do not mix its vectors with the legacy table.
 
 ### `768` To `1024`
 
-Any move from `768` dimensions to `1024` dimensions changes the schema width. Examples include a legacy local table moving to Cloudflare or Mistral.
+Any move from `768` dimensions to `1024` dimensions changes the schema width. Examples include a legacy table moving to Cloudflare, Mistral, or a 1024-dimensional OpenAI-compatible service.
 
 ```ts
 await createTable(client, "articles_mistral_1024", 1024);

package/docs/PROVIDERS.md

@@ -2,9 +2,8 @@
 
 Use this page to choose an embedding provider, confirm the table width it needs, and understand what crosses a network boundary.
 
-`libsql-search` supports these provider values:
+`libsql-search` only talks to external embedding services; it never loads or hosts an embedding model in-process. It supports these provider values:
 
-- `local`
 - `cloudflare`
 - `mistral`
 - `gemini`
@@ -17,8 +16,7 @@ All providers share the same `EmbeddingOptions` surface:
 
 ```ts
 interface EmbeddingOptions {
-  provider?:
-    | "local"
+  provider:
     | "cloudflare"
     | "mistral"
     | "gemini"
@@ -40,7 +38,7 @@ interface EmbeddingOptions {
 
 Shared defaults and rules:
 
-- `provider` defaults to `local`
+- `provider` is required; there is no implicit embedding runtime or service
 - `maxLength` defaults to `8000` code units
 - `timeoutMs` defaults to `30000`
 - `indexContent()` defaults to `intent: "document"`
@@ -59,7 +57,6 @@ The `model` option is only used by `openai-compatible`.
 
 | Provider | Literal | Upstream model used by this adapter | Dimensions | Credentials | Batching | Network and privacy boundary | Cost and table planning |
 | --- | --- | --- | --- | --- | --- | --- | --- |
-| Local | `local` | `Xenova/all-MiniLM-L6-v2` | Fixed `384` | None | Sequential in-process | No hosted API call. First use may download model artifacts and cache them locally. | No hosted API bill. Table must be `F32_BLOB(384)`. |
 | Cloudflare Workers AI | `cloudflare` | `@cf/baai/bge-m3` | Fixed `1024` | `accountId` and `apiToken`, or `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` | Native batch in one request | Indexed and queried text is sent to Cloudflare. | Check Cloudflare pricing before large rebuilds. Table must be `F32_BLOB(1024)`. |
 | Mistral | `mistral` | `mistral-embed` | Fixed `1024` | `apiKey`, or `MISTRAL_API_KEY` | Native batch in one request | Indexed and queried text is sent to Mistral. | Check Mistral pricing before rebuilds. Table must be `F32_BLOB(1024)`. |
 | Gemini | `gemini` | `gemini-embedding-2` | Default `3072`; allowed integers `128-3072` | `apiKey`, or `GEMINI_API_KEY` | Sequential SDK request per input | Indexed and queried text is sent to Google. The adapter currently rewrites payload text by intent. | Check Gemini pricing before rebuilds. Table width must match the chosen dimension count exactly. |
@@ -68,23 +65,6 @@ The `model` option is only used by `openai-compatible`.
 
 ## Provider Notes
 
-### Local
-
-```ts
-embeddingOptions: {
-  provider: "local",
-}
-```
-
-- fixed at `384` dimensions
-- rejects any other `dimensions` value before loading the runtime
-- uses `@huggingface/transformers` lazily and caches the local pipeline by model name
-
-References:
-
-- [Transformers.js in Node.js](https://huggingface.co/docs/transformers.js/en/tutorials/node)
-- [Transformers.js environment and cache controls](https://huggingface.co/docs/transformers.js/en/api/env)
-
 ### Cloudflare Workers AI
 
 ```ts

package/docs/README.md

@@ -2,7 +2,7 @@
 
 This directory holds the longer-form reference material for `libsql-search`.
 
-- [Provider selection and configuration](./PROVIDERS.md): compare local, hosted, and custom embedding providers before you build an index
+- [Provider selection and configuration](./PROVIDERS.md): compare external and custom embedding providers before you build an index
 - [Integration examples](./INTEGRATIONS.md): reusable provider flow plus Astro and Next.js examples
 - [Migration and reindexing guide](./MIGRATIONS.md): table-width changes, provider/model swaps, and safe cutovers
 - [API reference](./API.md): exported functions, option shapes, and result data

package/docs/TESTING.md

@@ -9,30 +9,31 @@ Routine unit tests and CI should not make live embedding-provider calls and shou
 - validate option handling and response parsing with mocks first
 - assert failures happen before network calls when configuration is invalid
 
-The current test suite follows that pattern in `tests/embeddings.test.ts` and [`tests/huggingface-transformers.mock.ts`](../tests/huggingface-transformers.mock.ts).
+The current test suite follows that pattern in `tests/embeddings.test.ts` and [`tests/embedding-service.mock.ts`](../tests/embedding-service.mock.ts).
 
-## Local Provider Mocks
+## Shared Embedding Service Mock
 
-The local provider should use a lightweight Transformers.js mock instead of downloading the real model during routine tests.
+Indexer, search, and database tests use a deterministic OpenAI-compatible service mock. This keeps the tests on the same external-service boundary as production without making network calls.
 
 ```ts
 import {
-  huggingFaceTransformersMock,
-  resetHuggingFaceTransformersMock,
-} from "./huggingface-transformers.mock.js";
+  embeddingServiceMock,
+  resetEmbeddingServiceMock,
+  TEST_EMBEDDING_OPTIONS,
+} from "./embedding-service.mock.js";
 
 beforeEach(() => {
-  resetHuggingFaceTransformersMock();
+  resetEmbeddingServiceMock();
 });
 ```
 
-The repository source file is `huggingface-transformers.mock.ts`. The example keeps the `.js` import suffix because this repo's ESM TypeScript source uses explicit `.js` relative imports that resolve after compilation.
+The example keeps the `.js` import suffix because this repo's ESM TypeScript source uses explicit `.js` relative imports that resolve after compilation.
 
 Test the contract you care about:
 
-- the library requests `Xenova/all-MiniLM-L6-v2`
-- the call uses `pooling: "mean"` and `normalize: true`
-- non-`384` local dimensions fail before runtime loading
+- indexing and querying use the same endpoint, model, and dimensions
+- provider failures remain classified as build-stage failures
+- queued deterministic vectors exercise exact ranking and tie behavior
 
 ## HTTP Provider Mocks
 
@@ -116,9 +117,9 @@ Apply the same pattern to `GEMINI_API_KEY`, `MISTRAL_API_KEY`, `CLOUDFLARE_ACCOU
 Prefer tests that prove bad inputs fail locally:
 
 - unknown provider
+- missing provider
 - missing provider credentials
 - blank credentials where trimming is expected
-- invalid local dimensions
 - invalid Gemini dimensions
 - invalid `openai-compatible` `baseUrl`
 - invalid `openai-compatible` `batchSize`

package/docs/TROUBLESHOOTING.md

@@ -1,9 +1,5 @@
 # Troubleshooting
 
-Use the page that matches the failure mode:
-
-- [Sharp native module issues](./TROUBLESHOOTING-SHARP.md)
-
 Common operational checks:
 
 - verify you called `createTable()` before indexing or searching
@@ -15,8 +11,6 @@ Common operational checks:
 - after upgrading an existing Gemini index, fully re-embed with
   `gemini-embedding-2`; for 3072-dimensional Gemini indexes, recreate the table
   or use a new table name before rebuilding
-- after upgrading an existing local 768-dimensional padded index, create or
-  recreate a 384-dimensional table and fully re-index before querying it
 - if `search()` reports that the `<tableName>_embedding_idx` vector index could
   not be used, the table has no embedding vector index: re-run `createTable()`
   with the table's existing name and width to add it without touching rows, or

package/docs/TURSO.md

@@ -44,20 +44,32 @@ import { createTable, indexContent, search } from "libsql-search";
 const database = await connect("./local.db");
 const client = tursoAdapter(database);
 
-await createTable(client, "articles", 384);
+const embeddingOptions = {
+  provider: "openai-compatible" as const,
+  baseUrl: process.env.EMBEDDING_BASE_URL!,
+  model: "bge-large-en-v1.5",
+  dimensions: 1024,
+};
+
+await createTable(client, "articles", 1024);
 
 await indexContent({
   client,
   contentPath: "./content",
-  embeddingOptions: { provider: "local" },
+  embeddingOptions,
 });
 
 const results = await search({
   client,
   query: "how do I deploy my docs site",
   limit: 5,
-  embeddingOptions: { provider: "local" },
+  embeddingOptions,
 });
+
+// When this adapter's lifetime ends, drain and close its cached statements
+// before closing the caller-owned database handle.
+await client.dispose();
+await database.close();
 ```
 
 Use `":memory:"` instead of a file path for an ephemeral database.
@@ -67,6 +79,29 @@ Use `":memory:"` instead of a file path for an ephemeral database.
 result shapes — `SearchOptions`, `SearchResult`, `IndexerOptions`,
 `IndexResult` — are identical on both backends.
 
+### Adapter lifetime and disposal
+
+One adapter caches up to **32 query statements**, keyed by SQL and evicted in
+least-recently-used order. The bound matters because public `tableName` options
+are embedded in SQL: an unbounded cache would let a long-lived process retain a
+new native statement for every valid table name it sees. Statements that are
+currently running or queued are closed after they finish rather than being
+evicted out from under a call.
+
+Call `await client.dispose()` after all work using that adapter has settled.
+Disposal drains queued query calls and closes every cached statement, but does
+**not** close the database handle you supplied. It is terminal: later calls on
+that adapter reject. Close the database separately, after disposal. If you omit
+disposal, the reusable cache is still bounded at 32 entries and lives until the
+underlying handle or process exits; statements evicted while in flight live
+only until their queued calls finish.
+
+The adapter serializes concurrent calls that share one cached statement. This
+is required for correctness, not just memory use: the native statement mutates
+its current bindings, so overlapping `all()` calls with different arguments can
+otherwise return another caller's rows. Different cached SQL statements remain
+independent.
+
 ## What is different on Turso
 
 ### There is no ANN vector index, so search is a full scan
@@ -185,13 +220,14 @@ point is still type-checked by `deno task check` so the claim above stays true.
 unchanged, and the main entry point exports no new symbol and references no
 Turso type.
 
-`tursoAdapter()` returns a `DatabaseAdapter`, and that type is exported from
-`libsql-search/turso` so you can name it:
+`tursoAdapter()` returns a `TursoAdapter`, which extends `DatabaseAdapter` with
+the disposal hook. Both types are exported from `libsql-search/turso` so you can
+name them:
 
 ```ts
-import { tursoAdapter, type DatabaseAdapter } from "libsql-search/turso";
+import { tursoAdapter, type TursoAdapter } from "libsql-search/turso";
 
-let client: DatabaseAdapter;
+let client: TursoAdapter;
 ```
 
 It is exported from the subpath only. The main entry point does not export it,
@@ -210,15 +246,18 @@ to prove the two stay interchangeable, and runs as part of
 Three things in `src/turso.ts` look like noise and are not. Each has a
 regression test; none of them fails loudly at runtime if removed.
 
-**Prepared statements must be closed.** A statement holds native memory that the
-garbage collector cannot reclaim, because it is not JavaScript heap. Every
-`prepare()` is released with `close()` in a `finally`. Measured through the
-built bundle, 60 000 `executeQuery()` calls on one handle grow RSS by ~570 MB
-without the close and ~150 MB with it. `search()` issues exactly one query, so
-an SSR site calling it per request is the case that turns this from untidy into
-an OOM. Inside `executeAtomicWrite()` the cached statements are released only
-*after* `COMMIT` or `ROLLBACK` — a statement stays bound to the transaction
-while it is open.
+**Prepared statements are bounded, serialized, and explicitly disposed.** A
+statement holds native memory that the garbage collector cannot reclaim,
+because it is not JavaScript heap. The query path therefore reuses a 32-entry
+LRU instead of preparing on every request, serializes rebinds per entry, and
+closes evictions only after their queued calls finish. `TursoAdapter.dispose()`
+drains and closes the remaining cache. Measured through the built bundle,
+60 000 `executeQuery()` calls grew RSS by ~570 MB when statements were never
+closed, ~150 MB when each call prepared and closed, and ~7 MB when one statement
+was reused. `search()` issues exactly one query, so the SSR-per-request path is
+where reuse matters most. Inside `executeAtomicWrite()` the transaction-local
+statements are still released only *after* `COMMIT` or `ROLLBACK` — a statement
+stays bound to the transaction while it is open.
 
 **`BEGIN IMMEDIATE` sits outside the `try`.** If it were inside, a `BEGIN` that
 fails because another rebuild already holds the write lock would fall into the

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.10.1",
+  "version": "0.11.1",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",
@@ -75,7 +75,6 @@
     }
   },
   "dependencies": {
-    "@huggingface/transformers": "4.2.0",
     "gray-matter": "^4.0.3"
   },
   "devDependencies": {

package/docs/TROUBLESHOOTING-SHARP.md

@@ -1,65 +0,0 @@
-# Troubleshooting: Transitive `sharp` Install Errors
-
-`libsql-search` does not directly import `sharp`, but local embeddings use
-`@huggingface/transformers`, which currently brings in `sharp` as a transitive
-runtime dependency. If you see an install error mentioning `sharp`, it is
-usually a native-package install or approval issue.
-
-This page exists because the error can show up before your application reaches
-any `libsql-search` code.
-
-## Typical Error
-
-```text
-Cannot find module '../build/Release/sharp-*.node'
-```
-
-Or:
-
-```text
-Error: Something went wrong installing the "sharp" module
-```
-
-## Why It Happens
-
-With pnpm, native packages may need explicit build-script approval. If the
-relevant install script is blocked, the native binary is never downloaded or
-built.
-
-## What To Do
-
-First inspect which build scripts pnpm blocked:
-
-```bash
-pnpm ignored-builds
-```
-
-Then approve the package that is actually failing and reinstall:
-
-```bash
-pnpm approve-builds
-pnpm install
-```
-
-In the interactive `pnpm approve-builds` prompt, select `sharp` if that is the
-package reporting the native-module failure.
-
-For a committed repository-level fix, you can also allow the package explicitly
-in `pnpm-workspace.yaml` with `onlyBuiltDependencies`.
-
-## Relation To `libsql-search`
-
-- local embeddings use `@huggingface/transformers`
-- the first local embedding run may download a model at runtime
-- that runtime model download is separate from a pnpm native-module install
-  failure
-
-## Verification
-
-After reinstalling, rerun the command that originally failed. If your app uses
-`sharp` directly, verify that import in your own project context.
-
-## Additional Resources
-
-- [pnpm approve-builds](https://pnpm.io/10.x/cli/approve-builds)
-- [Sharp installation docs](https://sharp.pixelplumbing.com/install)