libsql-search 0.6.0 → 0.7.1

package/docs/PROVIDERS.md

@@ -1,25 +1,35 @@
 # Embedding Providers
 
-`libsql-search` currently supports five embedding providers:
+Use this page to choose an embedding provider, confirm the table width it needs, and understand what crosses a network boundary.
+
+`libsql-search` supports these provider values:
 
 - `local`
 - `cloudflare`
 - `mistral`
 - `gemini`
 - `openai`
+- `openai-compatible`
 
-Use the same provider and dimensions for both indexing and querying. A mismatch
-between stored vectors and query vectors will break search quality or fail at
-query time.
+## Shared Behavior
 
-## Shared Options
+All providers share the same `EmbeddingOptions` surface:
 
 ```ts
 interface EmbeddingOptions {
-  provider?: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
+  provider?:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
   apiKey?: string;
   accountId?: string;
   apiToken?: string;
+  baseUrl?: string;
+  model?: string;
+  batchSize?: number;
   dimensions?: number;
   maxLength?: number;
   intent?: "document" | "query";
@@ -28,79 +38,37 @@ interface EmbeddingOptions {
 }
 ```
 
-- `provider` defaults to `"local"`
-- `dimensions` defaults to `384` for the library's default local provider.
-  Provider-specific defaults can differ; Cloudflare and Mistral use `1024`,
-  and Gemini defaults to `3072`.
-- `maxLength` defaults to `8000`
-- `intent` can be `"document"` or `"query"`; indexing defaults to
-  `"document"` and search defaults to `"query"` unless explicitly set
-- `timeoutMs` defaults to `30000`
-- `apiKey` is used by Mistral, Gemini, and OpenAI and falls back to
-  `MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY`
-- `accountId` and `apiToken` are used by Cloudflare and fall back to
-  `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`
-
-## Provider Contract
-
-Each provider exposes immutable metadata:
-
-```ts
-interface EmbeddingProviderMetadata {
-  name: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
-  model: string;
-  dimensions: number;
-  batch: {
-    mode: "native" | "sequential";
-    maxSize?: number;
-  };
-}
-```
+Shared defaults and rules:
 
-Use `getEmbeddingProviderMetadata(options)` or
-`createEmbeddingProvider(options).metadata` to inspect the effective model,
-dimensions, and batch behavior. Metadata inspection does not require hosted
-provider credentials.
-
-Batch modes:
-
-- `"native"` means the upstream provider accepts the batch in one request
-- `"sequential"` means the library accepts a batch and processes items one at a
-  time
-- when `maxSize` is present, it is a hard maximum enforced before provider or
-  network work
-
-`generateEmbeddings(texts, options)` returns vectors in the same order as the
-input texts. Provider responses are validated before database writes:
-
-- result count must match input count
-- each vector must match the provider's effective dimensions
-- every vector value must be a finite number
-- indexed batch responses must contain unique contiguous indices and are
-  reordered before being returned
+- `provider` defaults to `local`
+- `maxLength` defaults to `8000` code units
+- `timeoutMs` defaults to `30000`
+- `indexContent()` defaults to `intent: "document"`
+- `search()` defaults to `intent: "query"`
+- `getEmbeddingProviderMetadata()` reports the effective provider, model, dimensions, and batch mode without making a hosted call
+- use one embedding space per table: if provider, dimensions, model selection, endpoint, or formatting contract changes, build a new table and reindex
 
-Empty batches return `[]` without loading a local model, creating hosted clients,
-or making network calls.
+Intent behavior is intentionally narrow today:
 
-Lower-level provider clients return an `EmbeddingBatchResult` with the validated
-vectors plus provider, model, dimensions, and intent. The compatibility helpers
-`generateEmbedding()` and `generateEmbeddings()` return only arrays.
+- only the current Gemini adapter changes the formatted payload for `"document"` versus `"query"`
+- all other providers still carry `intent` metadata through the API, but they embed the same text string either way
 
-Cloudflare, Mistral, Gemini, and OpenAI clients are scoped to their current
-options. They are not cached globally across different credentials or
-configurations. The local Hugging Face Transformers pipeline is loaded lazily
-and cached by model name.
+The `model` option is only used by `openai-compatible`.
 
-Hosted provider failures are reported with bounded provider/status/request-id
-context and without raw upstream bodies, credentials, Authorization headers, or
-full URLs with query strings.
+## Provider Matrix
 
-## Local
+| Provider | Literal | Upstream model used by this adapter | Dimensions | Credentials | Batching | Network and privacy boundary | Cost and table planning |
+| --- | --- | --- | --- | --- | --- | --- | --- |
+| Local | `local` | `Xenova/all-MiniLM-L6-v2` | Fixed `384` | None | Sequential in-process | No hosted API call. First use may download model artifacts and cache them locally. | No hosted API bill. Table must be `F32_BLOB(384)`. |
+| Cloudflare Workers AI | `cloudflare` | `@cf/baai/bge-m3` | Fixed `1024` | `accountId` and `apiToken`, or `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` | Native batch in one request | Indexed and queried text is sent to Cloudflare. | Check Cloudflare pricing before large rebuilds. Table must be `F32_BLOB(1024)`. |
+| Mistral | `mistral` | `mistral-embed` | Fixed `1024` | `apiKey`, or `MISTRAL_API_KEY` | Native batch in one request | Indexed and queried text is sent to Mistral. | Check Mistral pricing before rebuilds. Table must be `F32_BLOB(1024)`. |
+| Gemini | `gemini` | `gemini-embedding-2` | Default `3072`; allowed integers `128-3072` | `apiKey`, or `GEMINI_API_KEY` | Sequential SDK request per input | Indexed and queried text is sent to Google. The adapter currently rewrites payload text by intent. | Check Gemini pricing before rebuilds. Table width must match the chosen dimension count exactly. |
+| OpenAI | `openai` | `text-embedding-3-small` when `dimensions <= 1536`, otherwise `text-embedding-3-large` | Default `768`; any positive integer accepted locally and forwarded as `dimensions` | `apiKey`, or `OPENAI_API_KEY` | Native batch in one request, max `2048` inputs | Indexed and queried text is sent to OpenAI. | Check OpenAI pricing before rebuilds. Table width must match the chosen dimension count exactly. |
+| OpenAI-compatible | `openai-compatible` | Your configured `model` | Required positive integer; no default | `baseUrl`, `model`, and `dimensions` are required. `apiKey` is optional and never falls back to env. | Metadata reports `native`; outbound requests are chunked sequentially at `batchSize`, default `32` | Boundary depends on the operator behind `baseUrl`. Treat `baseUrl` as a trusted server-side setting and review HTTPS, SSRF, logging, and retention controls yourself. | Table width must match the configured `dimensions`. Any endpoint or model change should use a new table plus full reindex. |
 
-Provider value: `local`
+## Provider Notes
 
-The local provider loads `Xenova/all-MiniLM-L6-v2` through
-`@huggingface/transformers`.
+### Local
 
 ```ts
 embeddingOptions: {
@@ -108,26 +76,16 @@ embeddingOptions: {
 }
 ```
 
-Notes:
-
-- the model emits 384 dimensions, and local vectors are validated at exactly
-  384 finite numbers
-- `dimensions: 384` is accepted explicitly; any other local dimension is
-  rejected before the runtime is imported or loaded
-- metadata reports 384 dimensions
-- batch metadata is `{ mode: "sequential" }`
-- the first run downloads and caches the model and can take longer on a fresh
-  machine
-- no API key is required
-- this remains the default provider for offline use
+- fixed at `384` dimensions
+- rejects any other `dimensions` value before loading the runtime
+- uses `@huggingface/transformers` lazily and caches the local pipeline by model name
 
-## Cloudflare Workers AI
+References:
 
-Provider value: `cloudflare`
+- [Transformers.js in Node.js](https://huggingface.co/docs/transformers.js/en/tutorials/node)
+- [Transformers.js environment and cache controls](https://huggingface.co/docs/transformers.js/en/api/env)
 
-Cloudflare is the recommended hosted provider for low-cost Markdown search.
-It uses Workers AI `@cf/baai/bge-m3` through Cloudflare's OpenAI-compatible
-embeddings endpoint.
+### Cloudflare Workers AI
 
 ```ts
 embeddingOptions: {
@@ -137,25 +95,16 @@ embeddingOptions: {
 }
 ```
 
-Behavior:
-
-- if `accountId` is omitted, the library reads `CLOUDFLARE_ACCOUNT_ID`
-- if `apiToken` is omitted, the library reads `CLOUDFLARE_API_TOKEN`
-- blank Cloudflare credentials are treated as missing
-- `@cf/baai/bge-m3` returns 1024 dimensions
-- metadata reports `@cf/baai/bge-m3` and 1024 dimensions without requiring
-  credentials
-- batch metadata is `{ mode: "native" }`
-- response items are reordered by provider-supplied index before being returned
-- Cloudflare does not accept custom dimensions in this provider; use
-  `createTable(client, "articles", 1024)` for Cloudflare-backed indexes
+- fixed at `1024` dimensions
+- uses the account-scoped Workers AI embeddings endpoint
+- blank credentials are treated as missing
 
-## Mistral
+References:
 
-Provider value: `mistral`
+- [Cloudflare `@cf/baai/bge-m3`](https://developers.cloudflare.com/workers-ai/models/bge-m3/)
+- [Cloudflare Workers AI pricing](https://developers.cloudflare.com/workers-ai/platform/pricing/)
 
-Mistral uses the hosted `mistral-embed` model through
-`https://api.mistral.ai/v1/embeddings`.
+### Mistral
 
 ```ts
 embeddingOptions: {
@@ -164,24 +113,17 @@ embeddingOptions: {
 }
 ```
 
-Behavior:
+- fixed at `1024` dimensions
+- sends `encoding_format: "float"`
+- expects indexed upstream responses
 
-- if `apiKey` is omitted, the library reads `MISTRAL_API_KEY`
-- blank Mistral credentials are treated as missing
-- `mistral-embed` returns 1024 dimensions
-- metadata reports `mistral-embed` and 1024 dimensions without requiring
-  credentials
-- batch metadata is `{ mode: "native" }`
-- request bodies send `encoding_format: "float"`
-- response items are reordered by provider-supplied index before being returned
-- Mistral does not accept custom dimensions in this provider; use
-  `createTable(client, "articles", 1024)` for Mistral-backed indexes
+References:
 
-## Gemini
+- [Mistral embeddings guide](https://docs.mistral.ai/studio/knowledge-rag/embeddings/text_embeddings)
+- [Mistral embeddings API reference](https://docs.mistral.ai/api/endpoint/embeddings)
+- [Mistral model overview for `mistral-embed`](https://docs.mistral.ai/models/mistral-embed-23-12)
 
-Provider value: `gemini`
-
-Gemini uses Google `gemini-embedding-2` through `@google/genai`.
+### Gemini
 
 ```ts
 embeddingOptions: {
@@ -191,26 +133,18 @@ embeddingOptions: {
 }
 ```
 
-Behavior:
-
-- if `apiKey` is omitted, the library reads `GEMINI_API_KEY`
-- blank Gemini credentials are treated as missing
-- Gemini defaults to 3072 dimensions
-- explicit Gemini dimensions must be integers from 128 through 3072
-- 768, 1536, and 3072 are recommended practical sizes
-- metadata reports `gemini-embedding-2` and the effective dimensions
-- batch metadata is `{ mode: "sequential" }`
-- the library sends one SDK request per input and verifies one vector per input
-- document inputs are formatted as `title: none | text: ...`
-- query inputs are formatted as `task: search result | query: ...`
-- the current implementation does not expose custom model selection
+- defaults to `3072` dimensions
+- accepts only integer dimensions from `128` through `3072`
+- currently formats document inputs as `title: none | text: ...`
+- currently formats query inputs as `task: search result | query: ...`
+- sends one SDK request per input, even when you call `generateEmbeddings()`
 
-## OpenAI
+References:
 
-Provider value: `openai`
+- [Gemini embeddings guide](https://ai.google.dev/gemini-api/docs/embeddings)
+- [Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing)
 
-OpenAI uses `text-embedding-3-small` when `dimensions <= 1536` and
-`text-embedding-3-large` when `dimensions > 1536`.
+### OpenAI
 
 ```ts
 embeddingOptions: {
@@ -220,42 +154,45 @@ embeddingOptions: {
 }
 ```
 
-Behavior:
-
-- if `apiKey` is omitted, the library reads `OPENAI_API_KEY`
-- the request sends the `dimensions` value to the OpenAI embeddings API
-- metadata reports `text-embedding-3-small` when `dimensions <= 1536` and
-  `text-embedding-3-large` when `dimensions > 1536`
-- batch metadata is `{ mode: "native", maxSize: 2048 }`
-- use the same dimension count in `createTable()`
-
-## Dimension Guidelines
-
-- `local` is fixed at `384`
-- `cloudflare` is fixed at `1024`
-- `mistral` is fixed at `1024`
-- Gemini defaults to `3072` and accepts explicit dimensions from `128` through
-  `3072`; use `768`, `1536`, or `3072` unless you have a specific reason
-- OpenAI can be used at 1536 or 3072, or another supported OpenAI dimension
-  value you explicitly set
-
-If you switch provider or dimensions for an existing table, recreate the table
-or rebuild the index into a separate table so stored vectors stay consistent.
-
-Existing Gemini indexes created with `text-embedding-004` must be fully
-re-embedded for `gemini-embedding-2`, even if you keep `dimensions: 768`,
-because both the model and query/document input formatting changed. If you move
-to the new 3072-dimensional default, create a new table or recreate the vector
-table first; `indexContent()` clears rows but does not change the `F32_BLOB`
-width. A separate table is safer because rebuilds are not transactional.
-
-Existing local indexes created with the older padded-local behavior usually have
-`F32_BLOB(768)` rows containing the 384 model values followed by zero padding.
-The current local contract stores the native 384-dimensional model output. To
-migrate, create or recreate a `F32_BLOB(384)` table and run a full re-index
-before querying it. Using the same model ID avoids an intentional model-space
-change, but bit-identical vectors are not promised across runtime, model
-revision, dtype, pooling, or normalization changes; validate search quality and
-re-index when those details change.
-
-Routine unit tests mock the local runtime and do not download the model.
+- defaults to `768` dimensions
+- uses `text-embedding-3-small` through `1536`
+- uses `text-embedding-3-large` above `1536`
+- rejects batches above `2048` inputs before any network request
+
+References:
+
+- [OpenAI embeddings guide](https://developers.openai.com/api/docs/guides/embeddings)
+- [OpenAI embeddings API reference](https://developers.openai.com/api/reference/resources/embeddings/methods/create/)
+- [OpenAI data controls](https://developers.openai.com/api/docs/guides/your-data#default-usage-policies-by-endpoint)
+- [OpenAI models overview](https://developers.openai.com/api/docs/models)
+- [OpenAI API pricing](https://openai.com/api/pricing/)
+
+### OpenAI-compatible
+
+```ts
+embeddingOptions: {
+  provider: "openai-compatible",
+  baseUrl: process.env.EMBEDDING_BASE_URL,
+  model: process.env.EMBEDDING_MODEL,
+  dimensions: Number(process.env.EMBEDDING_DIMENSIONS),
+  apiKey: process.env.EMBEDDING_API_KEY,
+  batchSize: 32,
+}
+```
+
+- `baseUrl`, `model`, and `dimensions` are required
+- `baseUrl` must be an absolute `http` or `https` URL without URL credentials, query strings, or fragments
+- the library normalizes `baseUrl` to an `/embeddings` endpoint
+- `batchSize` defaults to `32` and controls outbound chunking
+- `apiKey` is optional and never falls back to `OPENAI_API_KEY`
+
+References:
+
+- [Text Embeddings Inference quick tour](https://huggingface.co/docs/text-embeddings-inference/quick_tour)
+- [Text Embeddings Inference CLI arguments](https://huggingface.co/docs/text-embeddings-inference/cli_arguments)
+
+## Next Steps
+
+- [Integration examples](./INTEGRATIONS.md) for end-to-end configuration flows
+- [Migration guide](./MIGRATIONS.md) before changing widths, models, or endpoints
+- [Testing guide](./TESTING.md) for CI-safe provider coverage

package/docs/README.md

@@ -1,16 +1,14 @@
 # Documentation
 
 This directory holds the longer-form reference material for `libsql-search`.
-Start with the page that matches the job you are doing:
 
-- [Provider guide](./PROVIDERS.md): local, Cloudflare, Mistral, Gemini, and
-  OpenAI embedding options, dimensions, and API key behavior
+- [Provider selection and configuration](./PROVIDERS.md): compare local, hosted, and custom embedding providers before you build an index
+- [Integration examples](./INTEGRATIONS.md): reusable provider flow plus Astro and Next.js examples
+- [Migration and reindexing guide](./MIGRATIONS.md): table-width changes, provider/model swaps, and safe cutovers
 - [API reference](./API.md): exported functions, option shapes, and result data
-- [Integration examples](./INTEGRATIONS.md): Astro and Next.js server-side usage
-- [Indexing and operations](./INDEXING.md): content layout, rebuild scripts,
-  search quality tips, and indexing gotchas
-- [Troubleshooting](./TROUBLESHOOTING.md): known install/runtime issues
+- [Indexing and operations](./INDEXING.md): content layout, rebuild behavior, and search quality notes
+- [Testing guidance](./TESTING.md): CI-safe mocks and no-live-call policy
+- [Troubleshooting](./TROUBLESHOOTING.md): known install and runtime issues
 - [Releasing](./RELEASING.md): maintainer release workflow
 
-For the shortest first-use path, go back to the repository
-[README](../README.md).
+For the shortest first-use path, go back to the repository [README](../README.md).

package/docs/TESTING.md

@@ -0,0 +1,138 @@
+# Testing Guidance
+
+Routine unit tests and CI should not make live embedding-provider calls and should not require real provider credentials.
+
+## Repository Policy
+
+- no live provider keys in unit tests or CI
+- no routine network calls to hosted embedding providers in CI
+- validate option handling and response parsing with mocks first
+- assert failures happen before network calls when configuration is invalid
+
+The current test suite follows that pattern in `tests/embeddings.test.ts` and [`tests/huggingface-transformers.mock.ts`](../tests/huggingface-transformers.mock.ts).
+
+## Local Provider Mocks
+
+The local provider should use a lightweight Transformers.js mock instead of downloading the real model during routine tests.
+
+```ts
+import {
+  huggingFaceTransformersMock,
+  resetHuggingFaceTransformersMock,
+} from "./huggingface-transformers.mock.js";
+
+beforeEach(() => {
+  resetHuggingFaceTransformersMock();
+});
+```
+
+The repository source file is `huggingface-transformers.mock.ts`. The example keeps the `.js` import suffix because this repo's ESM TypeScript source uses explicit `.js` relative imports that resolve after compilation.
+
+Test the contract you care about:
+
+- the library requests `Xenova/all-MiniLM-L6-v2`
+- the call uses `pooling: "mean"` and `normalize: true`
+- non-`384` local dimensions fail before runtime loading
+
+## HTTP Provider Mocks
+
+Cloudflare, Mistral, OpenAI, and `openai-compatible` should use `fetch` mocks.
+
+```ts
+import { vi } from "vitest";
+
+const fetchMock = vi.fn().mockResolvedValue({
+  ok: true,
+  headers: new Headers(),
+  json: async () => ({
+    data: [
+      { index: 0, embedding: [1, 2] },
+      { index: 1, embedding: [3, 4] },
+    ],
+  }),
+});
+
+vi.stubGlobal("fetch", fetchMock);
+```
+
+Useful assertions:
+
+- request body includes the expected `model`
+- OpenAI includes `dimensions`
+- Mistral and `openai-compatible` include `encoding_format: "float"`
+- `openai-compatible` chunking honors `batchSize`
+- Cloudflare and Mistral reorder indexed responses correctly
+- invalid config fails before `fetch` is called
+
+## Gemini SDK Mocks
+
+Gemini uses the `@google/genai` SDK, so routine tests should mock the SDK client rather than calling Google.
+
+```ts
+vi.mock("@google/genai", () => ({
+  GoogleGenAI: class {
+    readonly models = {
+      embedContent: vi.fn(async () => ({
+        embeddings: [{ values: [1, 2, 3] }],
+      })),
+    };
+  },
+}));
+```
+
+Useful assertions:
+
+- the adapter sends `gemini-embedding-2`
+- document intent formats text as `title: none | text: ...`
+- query intent formats text as `task: search result | query: ...`
+- explicit dimensions are forwarded as `outputDimensionality`
+- invalid dimensions fail before SDK work
+
+## Environment Cleanup
+
+Tests that touch provider credentials should save and restore environment variables so one case cannot leak into another:
+
+```ts
+let originalOpenAIKey: string | undefined;
+
+beforeEach(() => {
+  originalOpenAIKey = process.env.OPENAI_API_KEY;
+  delete process.env.OPENAI_API_KEY;
+});
+
+afterEach(() => {
+  if (originalOpenAIKey === undefined) {
+    delete process.env.OPENAI_API_KEY;
+  } else {
+    process.env.OPENAI_API_KEY = originalOpenAIKey;
+  }
+});
+```
+
+Apply the same pattern to `GEMINI_API_KEY`, `MISTRAL_API_KEY`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_API_TOKEN`.
+
+## Validation-before-network Coverage
+
+Prefer tests that prove bad inputs fail locally:
+
+- unknown provider
+- missing provider credentials
+- blank credentials where trimming is expected
+- invalid local dimensions
+- invalid Gemini dimensions
+- invalid `openai-compatible` `baseUrl`
+- invalid `openai-compatible` `batchSize`
+- OpenAI batches above `2048`
+
+These checks keep CI fast and prove the library rejects bad inputs before it ships them to a provider.
+
+## Optional Live Smoke Tests
+
+If you want live provider smoke coverage, keep it outside routine CI:
+
+- run it only when a developer explicitly opts in
+- use dedicated throwaway credentials and test content
+- isolate it from unit-test jobs
+- expect provider cost and external data transfer
+
+This repository does not require or expect live provider smoke tests for normal pull-request validation.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.6.0",
+  "version": "0.7.1",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",