libsql-search 0.6.0 → 0.7.1

package/docs/API.md

@@ -37,9 +37,16 @@ It also exports these types:
 - `SearchOptions`
 - `SearchResult`
 
+## Navigation
+
+- [Provider matrix and credential rules](./PROVIDERS.md)
+- [Migration and reindexing guide](./MIGRATIONS.md)
+- [Indexing and operational behavior](./INDEXING.md)
+- [Testing guidance](./TESTING.md)
+
 ## `createTable(client, tableName?, dimensions?)`
 
-Creates the table and supporting indexes used by search.
+Creates the search table and supporting indexes.
 
 ```ts
 await createTable(client);
@@ -50,10 +57,7 @@ Defaults:
 - `tableName`: `"articles"`
 - `dimensions`: `384`
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved
-words such as `"select"` are safe to use. `dimensions` must be a positive
-integer.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved words such as `"select"` are safe to use. `dimensions` must be a positive integer.
 
 The created schema includes:
 
@@ -63,10 +67,12 @@ The created schema includes:
 - `content`
 - `folder`
 - `tags`
-- `embedding`
+- `embedding F32_BLOB(dimensions)`
 - `created_at`
 - `updated_at`
 
+`createTable()` uses `CREATE TABLE IF NOT EXISTS`, so it does not resize an existing vector column. See [Migration and reindexing guide](./MIGRATIONS.md) before changing widths or providers.
+
 ## `indexContent(options)`
 
 Indexes Markdown files from a directory on disk.
@@ -89,8 +95,6 @@ Defaults:
 - `exclude`: ["node_modules", ".git", "dist", "build"]
 - `tableName`: `"articles"`
 
-`tableName` follows the same identifier policy as `createTable()`.
-
 Return shape:
 
 ```ts
@@ -104,10 +108,9 @@ Return shape:
 Behavior notes:
 
 - `indexContent()` deletes existing rows in the target table before rebuilding
-- frontmatter `title`, `description`, and `tags` are folded into the embedding
-  text
-- embeddings default to `intent: "document"` unless `embeddingOptions.intent`
-  is set explicitly
+- rebuilds are not transactional
+- frontmatter `title`, `description`, and `tags` are folded into the embedding text
+- embeddings default to `intent: "document"` unless `embeddingOptions.intent` is set explicitly
 - if a file has no frontmatter title, the filename becomes the title
 
 ## `search(options)`
@@ -129,9 +132,7 @@ Defaults:
 - `limit`: `10`
 - `tableName`: `"articles"`
 
-`limit` must be an integer from `1` through `100`; invalid values are rejected
-before query embedding generation. `tableName` follows the same identifier
-policy as `createTable()`.
+`limit` must be an integer from `1` through `100`; invalid values are rejected before query embedding generation.
 
 Result shape:
 
@@ -150,8 +151,7 @@ interface SearchResult {
 
 Lower `distance` values are better matches.
 
-Search embeddings default to `intent: "query"` unless
-`embeddingOptions.intent` is set explicitly.
+Search embeddings default to `intent: "query"` unless `embeddingOptions.intent` is set explicitly.
 
 ## Article Retrieval Helpers
 
@@ -171,82 +171,141 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
-All article retrieval helpers validate `tableName` before executing SQL.
+All retrieval helpers validate `tableName` before executing SQL.
 
 ## Embedding Helpers
 
-### `generateEmbedding(text, options?)`
+### `EmbeddingOptions`
 
-Generates an embedding for arbitrary text using the selected provider.
+```ts
+interface EmbeddingOptions {
+  provider?:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
+  apiKey?: string;
+  accountId?: string;
+  apiToken?: string;
+  baseUrl?: string;
+  model?: string;
+  batchSize?: number;
+  dimensions?: number;
+  maxLength?: number;
+  intent?: "document" | "query";
+  timeoutMs?: number;
+  signal?: AbortSignal;
+}
+```
 
-### `generateEmbeddings(texts, options?)`
+Important option rules:
 
-Generates an ordered batch of embeddings. Empty batches return `[]` without
-creating a hosted provider client or making a network request.
+- `provider` defaults to `local`
+- `maxLength` defaults to `8000`
+- `timeoutMs` defaults to `30000`
+- `model` is only used by `openai-compatible`
+- `baseUrl`, `model`, and `dimensions` are required for `openai-compatible`
+- `batchSize` only applies to `openai-compatible` and defaults to `32`
+- `openai-compatible` never reads `OPENAI_API_KEY`
+- only the Gemini adapter currently changes payload formatting by `intent`
 
-### `createEmbeddingProvider(options?)`
+Dimension rules:
+
+- local: fixed `384`
+- Cloudflare: fixed `1024`
+- Mistral: fixed `1024`
+- Gemini: default `3072`, allowed integer range `128-3072`
+- OpenAI: default `768`; `text-embedding-3-small` through `1536`, `text-embedding-3-large` above `1536`
+- OpenAI-compatible: required positive integer, no default
 
-Creates a provider client with immutable metadata and an `embed(texts, options?)`
-method. Provider clients return a rich `EmbeddingBatchResult`; the compatibility
-helpers `generateEmbedding()` and `generateEmbeddings()` continue returning only
-vectors.
+See [Provider matrix and credential rules](./PROVIDERS.md) for the canonical provider table.
+
+### `generateEmbedding(text, options?)`
+
+Generates one embedding vector.
 
 ```ts
-const provider = createEmbeddingProvider({
+const embedding = await generateEmbedding("deploy docs", {
   provider: "openai",
   apiKey: process.env.OPENAI_API_KEY,
   dimensions: 1536,
 });
-
-console.log(provider.metadata);
 ```
 
-Provider metadata includes:
+### `generateEmbeddings(texts, options?)`
 
-- `name`
-- `model`
-- `dimensions`
-- `batch.mode`
-- `batch.maxSize`, when the provider has a hard maximum
+Generates an ordered batch of embeddings.
 
-Hosted provider clients are scoped to their options. The library does not reuse
-a Cloudflare, Mistral, Gemini, or OpenAI client created with different
-credentials or configuration.
+- empty batches return `[]` without loading the local model or making a hosted call
+- OpenAI batches above `2048` inputs are rejected before network work
+- `openai-compatible` batches are chunked sequentially according to `batchSize`
 
-Gemini uses `gemini-embedding-2`. Its default is 3072 dimensions, and explicit
-Gemini dimensions must be an integer from 128 through 3072.
+### `createEmbeddingProvider(options?)`
 
-Local embeddings use `Xenova/all-MiniLM-L6-v2` through
-`@huggingface/transformers` and are fixed at the model's native 384 dimensions.
-Passing any other local dimension is rejected before the runtime is loaded.
+Creates a provider client with immutable metadata and an `embed(texts, options?)` method.
+
+```ts
+const provider = createEmbeddingProvider({
+  provider: "openai-compatible",
+  baseUrl: "https://tei.example.internal/v1",
+  model: "bge-large-en-v1.5",
+  dimensions: 1024,
+  batchSize: 32,
+});
+
+console.log(provider.metadata);
+```
+
+Provider clients return a rich `EmbeddingBatchResult`; the compatibility helpers `generateEmbedding()` and `generateEmbeddings()` return only vectors.
+
+Hosted provider clients are scoped to their current options. The library does not reuse a Cloudflare, Mistral, Gemini, or OpenAI client across different credential sets or configurations.
 
 ### `getEmbeddingProviderMetadata(options?)`
 
-Returns the same metadata exposed by `createEmbeddingProvider(options).metadata`
-without resolving hosted-provider credentials.
+Returns the same metadata exposed by `createEmbeddingProvider(options).metadata` without resolving hosted-provider credentials.
 
-Provider batch metadata uses:
+Metadata shape:
 
 ```ts
-type EmbeddingBatchMode = "native" | "sequential";
-
-interface EmbeddingBatchBehavior {
-  mode: EmbeddingBatchMode;
-  maxSize?: number;
+interface EmbeddingProviderMetadata {
+  name:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
+  model: string;
+  dimensions: number;
+  batch: {
+    mode: "native" | "sequential";
+    maxSize?: number;
+  };
 }
 ```
 
-`"native"` means the upstream provider accepts the batch in one request.
-`"sequential"` means the library accepts an input batch but processes items one
-at a time. If `maxSize` is present, the library enforces it before provider or
-network work.
+Batch interpretation:
+
+- `"native"` means the provider accepts a batch request upstream
+- `"sequential"` means the library accepts a batch but processes items one-by-one
+- `batch.maxSize` is a hard client-side limit when present
 
-Provider clients return:
+`openai-compatible` metadata reports `batch.mode: "native"` because the remote endpoint is expected to accept batch inputs, even though the library may split large arrays into sequential outbound chunks at `batchSize`.
+
+### `EmbeddingBatchResult`
 
 ```ts
 interface EmbeddingBatchResult {
   embeddings: number[][];
-  provider: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
+  provider:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
   model: string;
   dimensions: number;
   intent: "document" | "query";
@@ -255,47 +314,17 @@ interface EmbeddingBatchResult {
 
 ### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
 
-Validates provider results before they are written to the database. It checks
-cardinality, dimensions, finite numeric values, and indexed batch ordering.
-
-`EmbeddingOptions` supports:
-
-```ts
-interface EmbeddingOptions {
-  provider?: "local" | "cloudflare" | "mistral" | "gemini" | "openai";
-  apiKey?: string;
-  accountId?: string;
-  apiToken?: string;
-  dimensions?: number;
-  maxLength?: number;
-  intent?: "document" | "query";
-  timeoutMs?: number;
-  signal?: AbortSignal;
-}
-```
+Validates provider responses before they reach the database:
 
-`apiKey` is used by Mistral, Gemini, and OpenAI. It falls back to
-`MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY` for those providers.
-Cloudflare uses `accountId` and `apiToken`, which fall back to
-`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`.
+- result count must match the requested input count
+- vectors must match the effective dimensions
+- values must be finite numbers
+- indexed provider responses are reordered and checked for contiguous indices
 
 ### `padEmbedding(embedding, targetDimensions)`
 
-Pads or truncates an embedding array to the requested length.
-
-This helper remains exported for callers that used it directly. The local
-provider does not use it; local vectors are validated at native 384 dimensions.
+Pads or truncates a vector to the target width. This is exported for compatibility and migration workflows, but the current local provider uses its native `384` dimensions rather than padding by default.
 
 ### `prepareTextForEmbedding(fields)`
 
-Combines title, description, tags, and content into the text sent to the
-embedding model.
-
-```ts
-const text = prepareTextForEmbedding({
-  title: "My Article",
-  description: "How semantic search works",
-  tags: ["search", "turso"],
-  content: "# Content",
-});
-```
+Builds the text that is embedded from title, description, content, and tags.

package/docs/INDEXING.md

@@ -2,8 +2,7 @@
 
 ## Content Shape
 
-`indexContent()` walks a directory tree, reads Markdown files, parses
-frontmatter with `gray-matter`, and stores:
+`indexContent()` walks a directory tree, reads Markdown files, parses frontmatter with `gray-matter`, and stores:
 
 - `slug`
 - `title`
@@ -22,41 +21,32 @@ The slug is derived from the file path relative to `contentPath`.
 await indexContent({
   client,
   contentPath: "./content",
-  tableName: "articles",
+  tableName: "articles_local_384",
   embeddingOptions: {
     provider: "local",
   },
 });
 ```
 
-That keeps the implementation simple, but it also means a failed rebuild can
-leave the index partially repopulated.
+That keeps the implementation simple, but it also means:
 
-Changing an embedding provider or dimension count requires a full re-embed.
-For existing local indexes created with the older padded-local behavior, create
-or recreate a 384-dimensional table before rebuilding. Those older local
-768-dimensional tables stored 384 model values followed by zero padding;
-`indexContent()` clears rows but does not change the table's `F32_BLOB` width.
+- failed rebuilds can leave the table partially repopulated
+- provider or dimension changes should use a parallel table migration
+- `createTable()` does not resize an existing vector column
 
-For Gemini specifically, indexes created with the retired `text-embedding-004`
-model must be rebuilt for `gemini-embedding-2` even when staying at 768
-dimensions, because the model and query/document formatting both changed. If
-you adopt Gemini's 3072-dimensional default, recreate the vector table or build
-into a separate table first; clearing rows with `indexContent()` does not change
-the table's `F32_BLOB` width.
+If provider, dimensions, model, endpoint, or embedding-space assumptions change, fully reindex into a new table. See the canonical [Migration and reindexing guide](./MIGRATIONS.md).
 
 ## Quality Guidelines
 
 - include descriptive frontmatter titles
 - add meaningful `tags` when they help retrieval
-- use the same embedding provider and dimensions at index and query time
+- use the same provider and dimensions at index and query time
 - keep `maxLength` intentional if your content is very large
 - start with a smaller search `limit` and tune from real query behavior
 
 ## Build Integration
 
-Many projects wire indexing into a dedicated script and call it before their
-site build:
+Many projects wire indexing into a dedicated script and call it before their site build:
 
 ```json
 {
@@ -69,14 +59,11 @@ site build:
 
 ## Table Names
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index
-SQL, so reserved words such as `"select"` work safely. Invalid names fail before
-database calls or embedding generation.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index SQL, so reserved words such as `"select"` work safely. Invalid names fail before database calls or embedding generation.
 
 ## Runtime Notes
 
 - local embeddings may download and cache a model on the first run
 - Node users need `@libsql/client` installed alongside the package
-- the repository validates both the npm package build and `deno check`, but the
-  indexing flow itself still depends on filesystem access
+- hosted providers send indexed or queried text to external services
+- the repository validates package build and `deno check`, but indexing still depends on filesystem access