libsql-search 0.7.0 → 0.7.1

package/README.md

@@ -5,27 +5,13 @@
 [![CI](https://github.com/llbbl/libsql-search/actions/workflows/ci.yml/badge.svg)](https://github.com/llbbl/libsql-search/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-`libsql-search` adds semantic search to Markdown-backed sites using libSQL/Turso.
-It indexes frontmatter and content from files on disk, stores vectors in your
-database, and lets you query by meaning instead of exact keywords.
+`libsql-search` adds semantic search to Markdown-backed sites with a small TypeScript API. It indexes frontmatter and content from files on disk, stores vectors in libSQL/Turso, and lets you query by meaning instead of exact keywords.
 
 Use it when you want:
 
-- a small TypeScript library instead of a hosted search product
-- one search index shared across static-site builds and app routes
-- local or hosted embeddings behind the same indexing/search API
-- direct control over table names, dimensions, content shape, and deployment
-
-## What It Supports
-
-- Markdown indexing from local directories with frontmatter via `gray-matter`
-- libSQL/Turso storage and vector search
-- Embedding providers: local Hugging Face `Xenova/all-MiniLM-L6-v2`,
-  Cloudflare Workers AI `@cf/baai/bge-m3`, Mistral `mistral-embed`, Google Gemini
-  `gemini-embedding-2`, and OpenAI `text-embedding-3-small` /
-  `text-embedding-3-large`; self-hosted OpenAI-compatible endpoints are also
-  available for TEI and similar trusted deployments
-- npm distribution plus JSR publishing
+- one indexing/search API across local and hosted embedding providers
+- direct control over vector dimensions, table names, and deployment shape
+- a lightweight library instead of a hosted search product
 
 ## Install
 
@@ -44,18 +30,10 @@ deno add jsr:@logan/libsql-search npm:@libsql/client
 ```
 
 For npm usage, the package requires Node `>=22.12.0`.
-Node examples in this README import from `libsql-search` and `@libsql/client`.
-In Deno, after `deno add`, import from `@logan/libsql-search` and
-`@libsql/client`.
 
 ## Quick Start
 
-The shortest working flow is:
-
-1. create a libSQL client
-2. create the search table
-3. index a Markdown directory
-4. query it with the same embedding provider and dimensions
+This example uses the default local provider. Local embeddings run in-process after the initial model download and cache warmup; they are not automatically air-gapped.
 
 ```ts
 import { createClient } from "@libsql/client";
@@ -66,11 +44,12 @@ const client = createClient({
   authToken: "your-auth-token",
 });
 
-await createTable(client);
+await createTable(client, "articles_local_384", 384);
 
 await indexContent({
   client,
   contentPath: "./content",
+  tableName: "articles_local_384",
   embeddingOptions: {
     provider: "local",
   },
@@ -79,6 +58,7 @@ await indexContent({
 const results = await search({
   client,
   query: "how do I deploy my docs site",
+  tableName: "articles_local_384",
   limit: 5,
   embeddingOptions: {
     provider: "local",
@@ -95,33 +75,31 @@ console.log(results.map((result) => ({
 Important behavior:
 
 - Call `createTable()` before indexing or searching.
-- Keep dimensions aligned across table creation, indexing, and search queries.
-- `indexContent()` clears existing rows before rebuilding the index.
-- `local` is the default offline provider and uses 384 dimensions. Cloudflare is
-  the recommended hosted option. Cloudflare and Mistral use 1024 dimensions.
-  Gemini defaults to 3072 dimensions and supports 128-3072.
-
-## Core API
-
-- `createTable(client, tableName?, dimensions?)`
-- `indexContent(options)`
-- `search(options)`
-- `getAllArticles(client, tableName?)`
-- `getArticleBySlug(client, slug, tableName?)`
-- `getArticlesByFolder(client, folder, tableName?)`
-- `getFolders(client, tableName?)`
-- `generateEmbedding(text, options?)`
-- `prepareTextForEmbedding(fields)`
+- Keep table width, provider, and dimensions aligned across create/index/query.
+- `indexContent()` clears existing rows before rebuilding and is not transactional.
+- Hosted providers send indexed and queried text to external services and may incur provider charges.
+
+## Providers
+
+Built-in providers:
+
+- `local` with `Xenova/all-MiniLM-L6-v2` at 384 dimensions
+- `cloudflare` with `@cf/baai/bge-m3` at 1024 dimensions
+- `mistral` with `mistral-embed` at 1024 dimensions
+- `gemini` with `gemini-embedding-2` at 128-3072 dimensions, default 3072
+- `openai` with `text-embedding-3-small` or `text-embedding-3-large`, default 768
+- `openai-compatible` for trusted OpenAI-compatible endpoints such as TEI
 
 ## Docs
 
-- [Docs index](./docs/README.md)
-- [Provider guide](./docs/PROVIDERS.md)
+- [Documentation index](./docs/README.md)
+- [Provider selection and configuration](./docs/PROVIDERS.md)
 - [API reference](./docs/API.md)
 - [Integration examples](./docs/INTEGRATIONS.md)
+- [Migration and reindexing guide](./docs/MIGRATIONS.md)
+- [Testing guidance](./docs/TESTING.md)
 - [Indexing and operations](./docs/INDEXING.md)
 - [Troubleshooting](./docs/TROUBLESHOOTING.md)
-- [Release workflow](./docs/RELEASING.md)
 
 ## License
 

package/docs/API.md

@@ -37,9 +37,16 @@ It also exports these types:
 - `SearchOptions`
 - `SearchResult`
 
+## Navigation
+
+- [Provider matrix and credential rules](./PROVIDERS.md)
+- [Migration and reindexing guide](./MIGRATIONS.md)
+- [Indexing and operational behavior](./INDEXING.md)
+- [Testing guidance](./TESTING.md)
+
 ## `createTable(client, tableName?, dimensions?)`
 
-Creates the table and supporting indexes used by search.
+Creates the search table and supporting indexes.
 
 ```ts
 await createTable(client);
@@ -50,10 +57,7 @@ Defaults:
 - `tableName`: `"articles"`
 - `dimensions`: `384`
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved
-words such as `"select"` are safe to use. `dimensions` must be a positive
-integer.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved words such as `"select"` are safe to use. `dimensions` must be a positive integer.
 
 The created schema includes:
 
@@ -63,10 +67,12 @@ The created schema includes:
 - `content`
 - `folder`
 - `tags`
-- `embedding`
+- `embedding F32_BLOB(dimensions)`
 - `created_at`
 - `updated_at`
 
+`createTable()` uses `CREATE TABLE IF NOT EXISTS`, so it does not resize an existing vector column. See [Migration and reindexing guide](./MIGRATIONS.md) before changing widths or providers.
+
 ## `indexContent(options)`
 
 Indexes Markdown files from a directory on disk.
@@ -89,8 +95,6 @@ Defaults:
 - `exclude`: ["node_modules", ".git", "dist", "build"]
 - `tableName`: `"articles"`
 
-`tableName` follows the same identifier policy as `createTable()`.
-
 Return shape:
 
 ```ts
@@ -104,10 +108,9 @@ Return shape:
 Behavior notes:
 
 - `indexContent()` deletes existing rows in the target table before rebuilding
-- frontmatter `title`, `description`, and `tags` are folded into the embedding
-  text
-- embeddings default to `intent: "document"` unless `embeddingOptions.intent`
-  is set explicitly
+- rebuilds are not transactional
+- frontmatter `title`, `description`, and `tags` are folded into the embedding text
+- embeddings default to `intent: "document"` unless `embeddingOptions.intent` is set explicitly
 - if a file has no frontmatter title, the filename becomes the title
 
 ## `search(options)`
@@ -129,9 +132,7 @@ Defaults:
 - `limit`: `10`
 - `tableName`: `"articles"`
 
-`limit` must be an integer from `1` through `100`; invalid values are rejected
-before query embedding generation. `tableName` follows the same identifier
-policy as `createTable()`.
+`limit` must be an integer from `1` through `100`; invalid values are rejected before query embedding generation.
 
 Result shape:
 
@@ -150,8 +151,7 @@ interface SearchResult {
 
 Lower `distance` values are better matches.
 
-Search embeddings default to `intent: "query"` unless
-`embeddingOptions.intent` is set explicitly.
+Search embeddings default to `intent: "query"` unless `embeddingOptions.intent` is set explicitly.
 
 ## Article Retrieval Helpers
 
@@ -171,82 +171,106 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
-All article retrieval helpers validate `tableName` before executing SQL.
+All retrieval helpers validate `tableName` before executing SQL.
 
 ## Embedding Helpers
 
-### `generateEmbedding(text, options?)`
+### `EmbeddingOptions`
+
+```ts
+interface EmbeddingOptions {
+  provider?:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
+  apiKey?: string;
+  accountId?: string;
+  apiToken?: string;
+  baseUrl?: string;
+  model?: string;
+  batchSize?: number;
+  dimensions?: number;
+  maxLength?: number;
+  intent?: "document" | "query";
+  timeoutMs?: number;
+  signal?: AbortSignal;
+}
+```
 
-Generates an embedding for arbitrary text using the selected provider.
+Important option rules:
 
-### `generateEmbeddings(texts, options?)`
+- `provider` defaults to `local`
+- `maxLength` defaults to `8000`
+- `timeoutMs` defaults to `30000`
+- `model` is only used by `openai-compatible`
+- `baseUrl`, `model`, and `dimensions` are required for `openai-compatible`
+- `batchSize` only applies to `openai-compatible` and defaults to `32`
+- `openai-compatible` never reads `OPENAI_API_KEY`
+- only the Gemini adapter currently changes payload formatting by `intent`
 
-Generates an ordered batch of embeddings. Empty batches return `[]` without
-creating a hosted provider client or making a network request.
+Dimension rules:
 
-### `createEmbeddingProvider(options?)`
+- local: fixed `384`
+- Cloudflare: fixed `1024`
+- Mistral: fixed `1024`
+- Gemini: default `3072`, allowed integer range `128-3072`
+- OpenAI: default `768`; `text-embedding-3-small` through `1536`, `text-embedding-3-large` above `1536`
+- OpenAI-compatible: required positive integer, no default
 
-Creates a provider client with immutable metadata and an `embed(texts, options?)`
-method. Provider clients return a rich `EmbeddingBatchResult`; the compatibility
-helpers `generateEmbedding()` and `generateEmbeddings()` continue returning only
-vectors.
+See [Provider matrix and credential rules](./PROVIDERS.md) for the canonical provider table.
+
+### `generateEmbedding(text, options?)`
+
+Generates one embedding vector.
 
 ```ts
-const provider = createEmbeddingProvider({
+const embedding = await generateEmbedding("deploy docs", {
   provider: "openai",
   apiKey: process.env.OPENAI_API_KEY,
   dimensions: 1536,
 });
-
-console.log(provider.metadata);
 ```
 
-Provider metadata includes:
+### `generateEmbeddings(texts, options?)`
 
-- `name`
-- `model`
-- `dimensions`
-- `batch.mode`
-- `batch.maxSize`, when the provider has a hard maximum
+Generates an ordered batch of embeddings.
 
-Hosted provider clients are scoped to their options. The library does not reuse
-a Cloudflare, Mistral, Gemini, or OpenAI client created with different
-credentials or configuration.
+- empty batches return `[]` without loading the local model or making a hosted call
+- OpenAI batches above `2048` inputs are rejected before network work
+- `openai-compatible` batches are chunked sequentially according to `batchSize`
 
-Gemini uses `gemini-embedding-2`. Its default is 3072 dimensions, and explicit
-Gemini dimensions must be an integer from 128 through 3072.
+### `createEmbeddingProvider(options?)`
 
-Local embeddings use `Xenova/all-MiniLM-L6-v2` through
-`@huggingface/transformers` and are fixed at the model's native 384 dimensions.
-Passing any other local dimension is rejected before the runtime is loaded.
+Creates a provider client with immutable metadata and an `embed(texts, options?)` method.
 
-### `getEmbeddingProviderMetadata(options?)`
+```ts
+const provider = createEmbeddingProvider({
+  provider: "openai-compatible",
+  baseUrl: "https://tei.example.internal/v1",
+  model: "bge-large-en-v1.5",
+  dimensions: 1024,
+  batchSize: 32,
+});
 
-Returns the same metadata exposed by `createEmbeddingProvider(options).metadata`
-without resolving hosted-provider credentials.
+console.log(provider.metadata);
+```
 
-Provider batch metadata uses:
+Provider clients return a rich `EmbeddingBatchResult`; the compatibility helpers `generateEmbedding()` and `generateEmbeddings()` return only vectors.
 
-```ts
-type EmbeddingBatchMode = "native" | "sequential";
+Hosted provider clients are scoped to their current options. The library does not reuse a Cloudflare, Mistral, Gemini, or OpenAI client across different credential sets or configurations.
 
-interface EmbeddingBatchBehavior {
-  mode: EmbeddingBatchMode;
-  maxSize?: number;
-}
-```
+### `getEmbeddingProviderMetadata(options?)`
 
-`"native"` means the upstream provider accepts the batch in one request.
-`"sequential"` means the library accepts an input batch but processes items one
-at a time. If `maxSize` is present, the library enforces it before provider or
-network work.
+Returns the same metadata exposed by `createEmbeddingProvider(options).metadata` without resolving hosted-provider credentials.
 
-Provider clients return:
+Metadata shape:
 
 ```ts
-interface EmbeddingBatchResult {
-  embeddings: number[][];
-  provider:
+interface EmbeddingProviderMetadata {
+  name:
     | "local"
     | "cloudflare"
     | "mistral"
@@ -255,69 +279,52 @@ interface EmbeddingBatchResult {
     | "openai-compatible";
   model: string;
   dimensions: number;
-  intent: "document" | "query";
+  batch: {
+    mode: "native" | "sequential";
+    maxSize?: number;
+  };
 }
 ```
 
-### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
+Batch interpretation:
 
-Validates provider results before they are written to the database. It checks
-cardinality, dimensions, finite numeric values, and indexed batch ordering.
+- `"native"` means the provider accepts a batch request upstream
+- `"sequential"` means the library accepts a batch but processes items one-by-one
+- `batch.maxSize` is a hard client-side limit when present
 
-`EmbeddingOptions` supports:
+`openai-compatible` metadata reports `batch.mode: "native"` because the remote endpoint is expected to accept batch inputs, even though the library may split large arrays into sequential outbound chunks at `batchSize`.
+
+### `EmbeddingBatchResult`
 
 ```ts
-interface EmbeddingOptions {
-  provider?:
+interface EmbeddingBatchResult {
+  embeddings: number[][];
+  provider:
     | "local"
     | "cloudflare"
     | "mistral"
     | "gemini"
     | "openai"
     | "openai-compatible";
-  apiKey?: string;
-  accountId?: string;
-  apiToken?: string;
-  baseUrl?: string;
-  model?: string;
-  batchSize?: number;
-  dimensions?: number;
-  maxLength?: number;
-  intent?: "document" | "query";
-  timeoutMs?: number;
-  signal?: AbortSignal;
+  model: string;
+  dimensions: number;
+  intent: "document" | "query";
 }
 ```
 
-`apiKey` is used by Mistral, Gemini, and OpenAI. It falls back to
-`MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY` for those providers.
-Cloudflare uses `accountId` and `apiToken`, which fall back to
-`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`.
+### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
 
-For `provider: "openai-compatible"`, callers must pass `baseUrl`, `model`, and
-`dimensions`. `apiKey` is optional and is used only when explicitly provided;
-the provider never reads `OPENAI_API_KEY`. `batchSize` controls the maximum
-items per outbound request and defaults to `32`. The endpoint is treated as
-trusted server-side configuration and should not be derived from untrusted
-request input.
+Validates provider responses before they reach the database:
 
-### `padEmbedding(embedding, targetDimensions)`
+- result count must match the requested input count
+- vectors must match the effective dimensions
+- values must be finite numbers
+- indexed provider responses are reordered and checked for contiguous indices
 
-Pads or truncates an embedding array to the requested length.
+### `padEmbedding(embedding, targetDimensions)`
 
-This helper remains exported for callers that used it directly. The local
-provider does not use it; local vectors are validated at native 384 dimensions.
+Pads or truncates a vector to the target width. This is exported for compatibility and migration workflows, but the current local provider uses its native `384` dimensions rather than padding by default.
 
 ### `prepareTextForEmbedding(fields)`
 
-Combines title, description, tags, and content into the text sent to the
-embedding model.
-
-```ts
-const text = prepareTextForEmbedding({
-  title: "My Article",
-  description: "How semantic search works",
-  tags: ["search", "turso"],
-  content: "# Content",
-});
-```
+Builds the text that is embedded from title, description, content, and tags.

package/docs/INDEXING.md

@@ -2,8 +2,7 @@
 
 ## Content Shape
 
-`indexContent()` walks a directory tree, reads Markdown files, parses
-frontmatter with `gray-matter`, and stores:
+`indexContent()` walks a directory tree, reads Markdown files, parses frontmatter with `gray-matter`, and stores:
 
 - `slug`
 - `title`
@@ -22,41 +21,32 @@ The slug is derived from the file path relative to `contentPath`.
 await indexContent({
   client,
   contentPath: "./content",
-  tableName: "articles",
+  tableName: "articles_local_384",
   embeddingOptions: {
     provider: "local",
   },
 });
 ```
 
-That keeps the implementation simple, but it also means a failed rebuild can
-leave the index partially repopulated.
+That keeps the implementation simple, but it also means:
 
-Changing an embedding provider or dimension count requires a full re-embed.
-For existing local indexes created with the older padded-local behavior, create
-or recreate a 384-dimensional table before rebuilding. Those older local
-768-dimensional tables stored 384 model values followed by zero padding;
-`indexContent()` clears rows but does not change the table's `F32_BLOB` width.
+- failed rebuilds can leave the table partially repopulated
+- provider or dimension changes should use a parallel table migration
+- `createTable()` does not resize an existing vector column
 
-For Gemini specifically, indexes created with the retired `text-embedding-004`
-model must be rebuilt for `gemini-embedding-2` even when staying at 768
-dimensions, because the model and query/document formatting both changed. If
-you adopt Gemini's 3072-dimensional default, recreate the vector table or build
-into a separate table first; clearing rows with `indexContent()` does not change
-the table's `F32_BLOB` width.
+If provider, dimensions, model, endpoint, or embedding-space assumptions change, fully reindex into a new table. See the canonical [Migration and reindexing guide](./MIGRATIONS.md).
 
 ## Quality Guidelines
 
 - include descriptive frontmatter titles
 - add meaningful `tags` when they help retrieval
-- use the same embedding provider and dimensions at index and query time
+- use the same provider and dimensions at index and query time
 - keep `maxLength` intentional if your content is very large
 - start with a smaller search `limit` and tune from real query behavior
 
 ## Build Integration
 
-Many projects wire indexing into a dedicated script and call it before their
-site build:
+Many projects wire indexing into a dedicated script and call it before their site build:
 
 ```json
 {
@@ -69,14 +59,11 @@ site build:
 
 ## Table Names
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index
-SQL, so reserved words such as `"select"` work safely. Invalid names fail before
-database calls or embedding generation.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index SQL, so reserved words such as `"select"` work safely. Invalid names fail before database calls or embedding generation.
 
 ## Runtime Notes
 
 - local embeddings may download and cache a model on the first run
 - Node users need `@libsql/client` installed alongside the package
-- the repository validates both the npm package build and `deno check`, but the
-  indexing flow itself still depends on filesystem access
+- hosted providers send indexed or queried text to external services
+- the repository validates package build and `deno check`, but indexing still depends on filesystem access