libsql-search 0.5.0 → 0.7.0

package/docs/PROVIDERS.md

@@ -1,12 +1,13 @@
 # Embedding Providers
 
-`libsql-search` currently supports five embedding providers:
+`libsql-search` currently supports six embedding providers:
 
 - `local`
 - `cloudflare`
 - `mistral`
 - `gemini`
 - `openai`
+- `openai-compatible`
 
 Use the same provider and dimensions for both indexing and querying. A mismatch
 between stored vectors and query vectors will break search quality or fail at
@@ -16,10 +17,19 @@ query time.
 
 ```ts
 interface EmbeddingOptions {
-  provider?: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
+  provider?:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
   apiKey?: string;
   accountId?: string;
   apiToken?: string;
+  baseUrl?: string;
+  model?: string;
+  batchSize?: number;
   dimensions?: number;
   maxLength?: number;
   intent?: "document" | "query";
@@ -29,16 +39,20 @@ interface EmbeddingOptions {
 ```
 
 - `provider` defaults to `"local"`
-- `dimensions` defaults to `768` for the library's default local provider.
-  Provider-specific defaults can differ; Gemini defaults to `3072`.
+- `dimensions` defaults to `384` for the library's default local provider.
+  Provider-specific defaults can differ; Cloudflare and Mistral use `1024`,
+  and Gemini defaults to `3072`.
 - `maxLength` defaults to `8000`
 - `intent` can be `"document"` or `"query"`; indexing defaults to
   `"document"` and search defaults to `"query"` unless explicitly set
 - `timeoutMs` defaults to `30000`
 - `apiKey` is used by Mistral, Gemini, and OpenAI and falls back to
-  `MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY`
+  `MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY`. For
+  `openai-compatible`, `apiKey` is optional and never falls back to
+  `OPENAI_API_KEY`.
 - `accountId` and `apiToken` are used by Cloudflare and fall back to
   `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`
+- `baseUrl`, `model`, and `batchSize` are used by `openai-compatible`
 
 ## Provider Contract
 
@@ -46,7 +60,13 @@ Each provider exposes immutable metadata:
 
 ```ts
 interface EmbeddingProviderMetadata {
-  name: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
+  name:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
   model: string;
   dimensions: number;
   batch: {
@@ -87,7 +107,8 @@ vectors plus provider, model, dimensions, and intent. The compatibility helpers
 
 Cloudflare, Mistral, Gemini, and OpenAI clients are scoped to their current
 options. They are not cached globally across different credentials or
-configurations. The local Xenova model can be cached by model name.
+configurations. The local Hugging Face Transformers pipeline is loaded lazily
+and cached by model name.
 
 Hosted provider failures are reported with bounded provider/status/request-id
 context and without raw upstream bodies, credentials, Authorization headers, or
@@ -98,24 +119,26 @@ full URLs with query strings.
 Provider value: `local`
 
 The local provider loads `Xenova/all-MiniLM-L6-v2` through
-`@xenova/transformers`.
+`@huggingface/transformers`.
 
 ```ts
 embeddingOptions: {
   provider: "local",
-  dimensions: 768,
 }
 ```
 
 Notes:
 
-- the model emits 384 dimensions and `libsql-search` pads or truncates to your
-  requested size
-- metadata reports the requested output dimensions
+- the model emits 384 dimensions, and local vectors are validated at exactly
+  384 finite numbers
+- `dimensions: 384` is accepted explicitly; any other local dimension is
+  rejected before the runtime is imported or loaded
+- metadata reports 384 dimensions
 - batch metadata is `{ mode: "sequential" }`
-- the first run downloads the model and can take longer on a fresh machine
+- the first run downloads and caches the model and can take longer on a fresh
+  machine
 - no API key is required
-- this remains the default provider for backward compatibility and offline use
+- this remains the default provider for offline use
 
 ## Cloudflare Workers AI
 
@@ -225,19 +248,95 @@ Behavior:
 - batch metadata is `{ mode: "native", maxSize: 2048 }`
 - use the same dimension count in `createTable()`
 
+## OpenAI-Compatible Endpoints
+
+Provider value: `openai-compatible`
+
+Use this provider for trusted OpenAI-compatible embedding services such as
+Hugging Face Text Embeddings Inference (TEI) or an internal gateway. This is an
+optional escape hatch; `local` remains the default offline provider, and the
+named hosted providers above are still preferred when their fixed adapters fit.
+
+```ts
+embeddingOptions: {
+  provider: "openai-compatible",
+  baseUrl: "http://localhost:8080/v1",
+  model: "BAAI/bge-large-en-v1.5",
+  dimensions: 1024,
+  batchSize: 32,
+}
+```
+
+If your endpoint requires bearer auth, pass `apiKey` explicitly:
+
+```ts
+embeddingOptions: {
+  provider: "openai-compatible",
+  baseUrl: "https://embeddings.example.com/v1",
+  apiKey: process.env.EMBEDDINGS_API_KEY,
+  model: "BAAI/bge-large-en-v1.5",
+  dimensions: 1024,
+}
+```
+
+Behavior:
+
+- `baseUrl`, `model`, and `dimensions` are required
+- `baseUrl` is the API base, such as `http://localhost:8080/v1`; the library
+  sends requests to `/embeddings` below that base
+- a base URL that already ends in `/embeddings` is used as-is
+- only absolute `http` and `https` URLs are accepted
+- URL usernames, passwords, query strings, and fragments are rejected
+- `apiKey` is optional; blank keys are ignored and no `Authorization` header is
+  sent
+- `OPENAI_API_KEY` is never read for this provider
+- `batchSize` defaults to `32`, matching TEI's conservative client batch size;
+  larger input arrays are split into sequential outbound requests and returned
+  in the original input order
+- requests send `{ input, model, dimensions, encoding_format: "float" }`
+- responses must use the standard OpenAI embeddings shape with indexed
+  `data[]` items; each response chunk must include unique contiguous indices
+- batch metadata is `{ mode: "native" }`; the internal outbound chunk size is
+  not reported as `batch.maxSize`
+
+Security boundary:
+
+- treat `baseUrl` as trusted server-side configuration only
+- never pass user-supplied request values directly into `baseUrl`
+- use HTTPS for remote endpoints
+- credentials are not sent across redirects
+- non-2xx response bodies are not read, and errors avoid echoing API keys or
+  configured endpoint URLs
+
+TEI exposes an OpenAI-compatible base at `/v1`, so a local TEI server usually
+uses:
+
+```ts
+embeddingOptions: {
+  provider: "openai-compatible",
+  baseUrl: "http://localhost:8080/v1",
+  model: "BAAI/bge-large-en-v1.5",
+  dimensions: 1024,
+}
+```
+
+This library does not call TEI's native `/embed` endpoint.
+
 ## Dimension Guidelines
 
-- `local` defaults to `768`
+- `local` is fixed at `384`
 - `cloudflare` is fixed at `1024`
 - `mistral` is fixed at `1024`
-- local embeddings are padded from 384 to your target size
 - Gemini defaults to `3072` and accepts explicit dimensions from `128` through
   `3072`; use `768`, `1536`, or `3072` unless you have a specific reason
 - OpenAI can be used at 1536 or 3072, or another supported OpenAI dimension
   value you explicitly set
+- `openai-compatible` requires you to set the dimension count that your
+  endpoint/model actually returns
 
-If you switch provider or dimensions for an existing table, recreate the table
-or rebuild the index into a separate table so stored vectors stay consistent.
+If you switch provider, endpoint, model, or dimensions for an existing table,
+recreate the table or rebuild the index into a separate table so stored vectors
+stay consistent.
 
 Existing Gemini indexes created with `text-embedding-004` must be fully
 re-embedded for `gemini-embedding-2`, even if you keep `dimensions: 768`,
@@ -245,3 +344,14 @@ because both the model and query/document input formatting changed. If you move
 to the new 3072-dimensional default, create a new table or recreate the vector
 table first; `indexContent()` clears rows but does not change the `F32_BLOB`
 width. A separate table is safer because rebuilds are not transactional.
+
+Existing local indexes created with the older padded-local behavior usually have
+`F32_BLOB(768)` rows containing the 384 model values followed by zero padding.
+The current local contract stores the native 384-dimensional model output. To
+migrate, create or recreate a `F32_BLOB(384)` table and run a full re-index
+before querying it. Using the same model ID avoids an intentional model-space
+change, but bit-identical vectors are not promised across runtime, model
+revision, dtype, pooling, or normalization changes; validate search quality and
+re-index when those details change.
+
+Routine unit tests mock the local runtime and do not download the model.

package/docs/README.md

@@ -3,8 +3,8 @@
 This directory holds the longer-form reference material for `libsql-search`.
 Start with the page that matches the job you are doing:
 
-- [Provider guide](./PROVIDERS.md): local, Cloudflare, Mistral, Gemini, and
-  OpenAI embedding options, dimensions, and API key behavior
+- [Provider guide](./PROVIDERS.md): local, hosted, and OpenAI-compatible
+  embedding options, dimensions, and API key behavior
 - [API reference](./API.md): exported functions, option shapes, and result data
 - [Integration examples](./INTEGRATIONS.md): Astro and Next.js server-side usage
 - [Indexing and operations](./INDEXING.md): content layout, rebuild scripts,

package/docs/TROUBLESHOOTING-SHARP.md

@@ -1,11 +1,12 @@
 # Troubleshooting: Transitive `sharp` Install Errors
 
-`libsql-search` does not directly depend on `sharp`. If you see an install error
-mentioning `sharp`, it is coming from another dependency in your application or
-toolchain.
+`libsql-search` does not directly import `sharp`, but local embeddings use
+`@huggingface/transformers`, which currently brings in `sharp` as a transitive
+runtime dependency. If you see an install error mentioning `sharp`, it is
+usually a native-package install or approval issue.
 
-This page exists because the error can show up in environments that also use
-`libsql-search`, and it is easy to misattribute the failure to this package.
+This page exists because the error can show up before your application reaches
+any `libsql-search` code.
 
 ## Typical Error
 
@@ -48,7 +49,7 @@ in `pnpm-workspace.yaml` with `onlyBuiltDependencies`.
 
 ## Relation To `libsql-search`
 
-- local embeddings use `@xenova/transformers`
+- local embeddings use `@huggingface/transformers`
 - the first local embedding run may download a model at runtime
 - that runtime model download is separate from a pnpm native-module install
   failure

package/docs/TROUBLESHOOTING.md

@@ -15,3 +15,5 @@ Common operational checks:
 - after upgrading an existing Gemini index, fully re-embed with
   `gemini-embedding-2`; for 3072-dimensional Gemini indexes, recreate the table
   or use a new table name before rebuilding
+- after upgrading an existing local 768-dimensional padded index, create or
+  recreate a 384-dimensional table and fully re-index before querying it

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.5.0",
+  "version": "0.7.0",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",
@@ -63,7 +63,7 @@
     "@libsql/client": "^0.15.0"
   },
   "dependencies": {
-    "@xenova/transformers": "^2.17.2",
+    "@huggingface/transformers": "4.2.0",
     "gray-matter": "^4.0.3"
   },
   "devDependencies": {