libsql-search 0.7.0 → 0.8.0

package/docs/API.md

@@ -18,6 +18,7 @@
 - `validateEmbeddingBatch`
 - `padEmbedding`
 - `prepareTextForEmbedding`
+- `IndexingError`
 
 It also exports these types:
 
@@ -34,12 +35,24 @@ It also exports these types:
 - `EmbeddingOptions`
 - `IndexerOptions`
 - `IndexedDocument`
+- `IndexResult`
+- `IndexFailure`
+- `IndexFailurePolicy`
+- `IndexFailureStage`
+- `IndexingErrorPhase`
 - `SearchOptions`
 - `SearchResult`
 
+## Navigation
+
+- [Provider matrix and credential rules](./PROVIDERS.md)
+- [Migration and reindexing guide](./MIGRATIONS.md)
+- [Indexing and operational behavior](./INDEXING.md)
+- [Testing guidance](./TESTING.md)
+
 ## `createTable(client, tableName?, dimensions?)`
 
-Creates the table and supporting indexes used by search.
+Creates the search table and supporting indexes.
 
 ```ts
 await createTable(client);
@@ -50,10 +63,7 @@ Defaults:
 - `tableName`: `"articles"`
 - `dimensions`: `384`
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved
-words such as `"select"` are safe to use. `dimensions` must be a positive
-integer.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved words such as `"select"` are safe to use. `dimensions` must be a positive integer.
 
 The created schema includes:
 
@@ -63,10 +73,12 @@ The created schema includes:
 - `content`
 - `folder`
 - `tags`
-- `embedding`
+- `embedding F32_BLOB(dimensions)`
 - `created_at`
 - `updated_at`
 
+`createTable()` uses `CREATE TABLE IF NOT EXISTS`, so it does not resize an existing vector column. See [Migration and reindexing guide](./MIGRATIONS.md) before changing widths or providers.
+
 ## `indexContent(options)`
 
 Indexes Markdown files from a directory on disk.
@@ -80,6 +92,8 @@ interface IndexerOptions {
   exclude?: string[];
   tableName?: string;
   onProgress?: (current: number, total: number, file: string) => void;
+  failurePolicy?: "abort" | "skip";
+  allowEmptyIndex?: boolean;
 }
 ```
 
@@ -88,28 +102,80 @@ Defaults:
 - `fileExtensions`: [".md", ".markdown"]
 - `exclude`: ["node_modules", ".git", "dist", "build"]
 - `tableName`: `"articles"`
-
-`tableName` follows the same identifier policy as `createTable()`.
+- `failurePolicy`: `"abort"`
+- `allowEmptyIndex`: `false`
 
 Return shape:
 
 ```ts
-{
-  success: number;
-  failed: number;
-  total: number;
+interface IndexResult {
+  success: number;   // documents written
+  failed: number;    // files that could not be indexed
+  total: number;     // files discovered on disk
+  replaced: boolean; // whether table contents were replaced by this call
+  partial: boolean;  // replaced, but some files were skipped
+  failures: IndexFailure[];
+}
+
+interface IndexFailure {
+  file: string;                        // path relative to contentPath
+  stage: "read" | "parse" | "embed";
+  error: Error;
 }
 ```
 
 Behavior notes:
 
-- `indexContent()` deletes existing rows in the target table before rebuilding
-- frontmatter `title`, `description`, and `tags` are folded into the embedding
-  text
-- embeddings default to `intent: "document"` unless `embeddingOptions.intent`
-  is set explicitly
+- every file is read, parsed, and embedded in memory before any database state changes
+- the target table is then replaced in a single write transaction, so a failed rebuild leaves the previous index exactly as it was
+- that costs peak memory proportional to the whole corpus, and against remote clients the replacement travels as a single un-chunked batch request; see [Costs of the two-phase rebuild](./INDEXING.md#costs-of-the-two-phase-rebuild) before rebuilding a very large corpus in place
+- files are discovered and indexed in sorted path order
+- frontmatter `title` must be a scalar; a structured title such as a YAML list fails the file at the `parse` stage
+- two files that reduce to the same slug (`foo.md` and `foo.markdown`) collide: the first in sorted path order keeps the slug and the later file is reported as a `parse` failure
+- `failurePolicy: "abort"` throws `IndexingError` on the first file that fails
+- `failurePolicy: "skip"` drops the failing file, records it in `failures`, and rebuilds from the survivors, returning `partial: true`
+- under `"skip"`, if every discovered file fails, the rebuild throws instead of replacing a valid index with an empty one
+- an empty source directory throws unless `allowEmptyIndex: true`, which intentionally empties the index
+- `onProgress` is called once per file during the build phase
+- frontmatter `title`, `description`, and `tags` are folded into the embedding text
+- embeddings default to `intent: "document"` unless `embeddingOptions.intent` is set explicitly
 - if a file has no frontmatter title, the filename becomes the title
 
+### `IndexingError`
+
+Thrown when a rebuild cannot complete. The previously indexed rows are always left unchanged.
+
+```ts
+class IndexingError extends Error {
+  readonly phase: "build" | "replace";
+  readonly failures: IndexFailure[];
+}
+```
+
+- `phase: "build"` means the failure happened before any database work: a file failed, every file failed, the source directory was empty, or it could not be scanned
+- `phase: "replace"` means the replacement transaction failed and was rolled back
+- `cause` carries the underlying error
+- on a `phase: "replace"` error, `failures` lists files skipped during the build phase. They are not the cause of the rollback, which is carried by `cause`
+
+```ts
+import { indexContent, IndexingError } from "libsql-search";
+
+try {
+  await indexContent({ client, contentPath: "./content" });
+} catch (error) {
+  if (error instanceof IndexingError) {
+    console.error(error.phase, error.failures);
+  }
+
+  throw error;
+}
+```
+
+Breaking changes in this behavior:
+
+- partial failures previously counted into `failed` and still replaced the table; they now throw. Pass `failurePolicy: "skip"` for the previous lenient behavior.
+- an empty source directory previously returned zeros and left stale rows in place; it now throws. Pass `allowEmptyIndex: true` to intentionally empty the index.
+
 ## `search(options)`
 
 Generates a query embedding and performs vector similarity search.
@@ -129,9 +195,7 @@ Defaults:
 - `limit`: `10`
 - `tableName`: `"articles"`
 
-`limit` must be an integer from `1` through `100`; invalid values are rejected
-before query embedding generation. `tableName` follows the same identifier
-policy as `createTable()`.
+`limit` must be an integer from `1` through `100`; invalid values are rejected before query embedding generation.
 
 Result shape:
 
@@ -150,8 +214,7 @@ interface SearchResult {
 
 Lower `distance` values are better matches.
 
-Search embeddings default to `intent: "query"` unless
-`embeddingOptions.intent` is set explicitly.
+Search embeddings default to `intent: "query"` unless `embeddingOptions.intent` is set explicitly.
 
 ## Article Retrieval Helpers
 
@@ -171,82 +234,106 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
-All article retrieval helpers validate `tableName` before executing SQL.
+All retrieval helpers validate `tableName` before executing SQL.
 
 ## Embedding Helpers
 
-### `generateEmbedding(text, options?)`
+### `EmbeddingOptions`
 
-Generates an embedding for arbitrary text using the selected provider.
+```ts
+interface EmbeddingOptions {
+  provider?:
+    | "local"
+    | "cloudflare"
+    | "mistral"
+    | "gemini"
+    | "openai"
+    | "openai-compatible";
+  apiKey?: string;
+  accountId?: string;
+  apiToken?: string;
+  baseUrl?: string;
+  model?: string;
+  batchSize?: number;
+  dimensions?: number;
+  maxLength?: number;
+  intent?: "document" | "query";
+  timeoutMs?: number;
+  signal?: AbortSignal;
+}
+```
 
-### `generateEmbeddings(texts, options?)`
+Important option rules:
 
-Generates an ordered batch of embeddings. Empty batches return `[]` without
-creating a hosted provider client or making a network request.
+- `provider` defaults to `local`
+- `maxLength` defaults to `8000`
+- `timeoutMs` defaults to `30000`
+- `model` is only used by `openai-compatible`
+- `baseUrl`, `model`, and `dimensions` are required for `openai-compatible`
+- `batchSize` only applies to `openai-compatible` and defaults to `32`
+- `openai-compatible` never reads `OPENAI_API_KEY`
+- only the Gemini adapter currently changes payload formatting by `intent`
 
-### `createEmbeddingProvider(options?)`
+Dimension rules:
 
-Creates a provider client with immutable metadata and an `embed(texts, options?)`
-method. Provider clients return a rich `EmbeddingBatchResult`; the compatibility
-helpers `generateEmbedding()` and `generateEmbeddings()` continue returning only
-vectors.
+- local: fixed `384`
+- Cloudflare: fixed `1024`
+- Mistral: fixed `1024`
+- Gemini: default `3072`, allowed integer range `128-3072`
+- OpenAI: default `768`; `text-embedding-3-small` through `1536`, `text-embedding-3-large` above `1536`
+- OpenAI-compatible: required positive integer, no default
+
+See [Provider matrix and credential rules](./PROVIDERS.md) for the canonical provider table.
+
+### `generateEmbedding(text, options?)`
+
+Generates one embedding vector.
 
 ```ts
-const provider = createEmbeddingProvider({
+const embedding = await generateEmbedding("deploy docs", {
   provider: "openai",
   apiKey: process.env.OPENAI_API_KEY,
   dimensions: 1536,
 });
-
-console.log(provider.metadata);
 ```
 
-Provider metadata includes:
+### `generateEmbeddings(texts, options?)`
 
-- `name`
-- `model`
-- `dimensions`
-- `batch.mode`
-- `batch.maxSize`, when the provider has a hard maximum
+Generates an ordered batch of embeddings.
 
-Hosted provider clients are scoped to their options. The library does not reuse
-a Cloudflare, Mistral, Gemini, or OpenAI client created with different
-credentials or configuration.
+- empty batches return `[]` without loading the local model or making a hosted call
+- OpenAI batches above `2048` inputs are rejected before network work
+- `openai-compatible` batches are chunked sequentially according to `batchSize`
 
-Gemini uses `gemini-embedding-2`. Its default is 3072 dimensions, and explicit
-Gemini dimensions must be an integer from 128 through 3072.
+### `createEmbeddingProvider(options?)`
 
-Local embeddings use `Xenova/all-MiniLM-L6-v2` through
-`@huggingface/transformers` and are fixed at the model's native 384 dimensions.
-Passing any other local dimension is rejected before the runtime is loaded.
+Creates a provider client with immutable metadata and an `embed(texts, options?)` method.
 
-### `getEmbeddingProviderMetadata(options?)`
+```ts
+const provider = createEmbeddingProvider({
+  provider: "openai-compatible",
+  baseUrl: "https://tei.example.internal/v1",
+  model: "bge-large-en-v1.5",
+  dimensions: 1024,
+  batchSize: 32,
+});
 
-Returns the same metadata exposed by `createEmbeddingProvider(options).metadata`
-without resolving hosted-provider credentials.
+console.log(provider.metadata);
+```
 
-Provider batch metadata uses:
+Provider clients return a rich `EmbeddingBatchResult`; the compatibility helpers `generateEmbedding()` and `generateEmbeddings()` return only vectors.
 
-```ts
-type EmbeddingBatchMode = "native" | "sequential";
+Hosted provider clients are scoped to their current options. The library does not reuse a Cloudflare, Mistral, Gemini, or OpenAI client across different credential sets or configurations.
 
-interface EmbeddingBatchBehavior {
-  mode: EmbeddingBatchMode;
-  maxSize?: number;
-}
-```
+### `getEmbeddingProviderMetadata(options?)`
 
-`"native"` means the upstream provider accepts the batch in one request.
-`"sequential"` means the library accepts an input batch but processes items one
-at a time. If `maxSize` is present, the library enforces it before provider or
-network work.
+Returns the same metadata exposed by `createEmbeddingProvider(options).metadata` without resolving hosted-provider credentials.
 
-Provider clients return:
+Metadata shape:
 
 ```ts
-interface EmbeddingBatchResult {
-  embeddings: number[][];
-  provider:
+interface EmbeddingProviderMetadata {
+  name:
     | "local"
     | "cloudflare"
     | "mistral"
@@ -255,69 +342,52 @@ interface EmbeddingBatchResult {
     | "openai-compatible";
   model: string;
   dimensions: number;
-  intent: "document" | "query";
+  batch: {
+    mode: "native" | "sequential";
+    maxSize?: number;
+  };
 }
 ```
 
-### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
+Batch interpretation:
 
-Validates provider results before they are written to the database. It checks
-cardinality, dimensions, finite numeric values, and indexed batch ordering.
+- `"native"` means the provider accepts a batch request upstream
+- `"sequential"` means the library accepts a batch but processes items one-by-one
+- `batch.maxSize` is a hard client-side limit when present
 
-`EmbeddingOptions` supports:
+`openai-compatible` metadata reports `batch.mode: "native"` because the remote endpoint is expected to accept batch inputs, even though the library may split large arrays into sequential outbound chunks at `batchSize`.
+
+### `EmbeddingBatchResult`
 
 ```ts
-interface EmbeddingOptions {
-  provider?:
+interface EmbeddingBatchResult {
+  embeddings: number[][];
+  provider:
     | "local"
     | "cloudflare"
     | "mistral"
     | "gemini"
     | "openai"
     | "openai-compatible";
-  apiKey?: string;
-  accountId?: string;
-  apiToken?: string;
-  baseUrl?: string;
-  model?: string;
-  batchSize?: number;
-  dimensions?: number;
-  maxLength?: number;
-  intent?: "document" | "query";
-  timeoutMs?: number;
-  signal?: AbortSignal;
+  model: string;
+  dimensions: number;
+  intent: "document" | "query";
 }
 ```
 
-`apiKey` is used by Mistral, Gemini, and OpenAI. It falls back to
-`MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY` for those providers.
-Cloudflare uses `accountId` and `apiToken`, which fall back to
-`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`.
+### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
+
+Validates provider responses before they reach the database:
 
-For `provider: "openai-compatible"`, callers must pass `baseUrl`, `model`, and
-`dimensions`. `apiKey` is optional and is used only when explicitly provided;
-the provider never reads `OPENAI_API_KEY`. `batchSize` controls the maximum
-items per outbound request and defaults to `32`. The endpoint is treated as
-trusted server-side configuration and should not be derived from untrusted
-request input.
+- result count must match the requested input count
+- vectors must match the effective dimensions
+- values must be finite numbers
+- indexed provider responses are reordered and checked for contiguous indices
 
 ### `padEmbedding(embedding, targetDimensions)`
 
-Pads or truncates an embedding array to the requested length.
-
-This helper remains exported for callers that used it directly. The local
-provider does not use it; local vectors are validated at native 384 dimensions.
+Pads or truncates a vector to the target width. This is exported for compatibility and migration workflows, but the current local provider uses its native `384` dimensions rather than padding by default.
 
 ### `prepareTextForEmbedding(fields)`
 
-Combines title, description, tags, and content into the text sent to the
-embedding model.
-
-```ts
-const text = prepareTextForEmbedding({
-  title: "My Article",
-  description: "How semantic search works",
-  tags: ["search", "turso"],
-  content: "# Content",
-});
-```
+Builds the text that is embedded from title, description, content, and tags.

package/docs/INDEXING.md

@@ -2,8 +2,7 @@
 
 ## Content Shape
 
-`indexContent()` walks a directory tree, reads Markdown files, parses
-frontmatter with `gray-matter`, and stores:
+`indexContent()` walks a directory tree, reads Markdown files, parses frontmatter with `gray-matter`, and stores:
 
 - `slug`
 - `title`
@@ -16,47 +15,113 @@ The slug is derived from the file path relative to `contentPath`.
 
 ## Rebuild Behavior
 
-`indexContent()` clears the target table before rebuilding:
+`indexContent()` replaces the whole target table:
 
 ```ts
 await indexContent({
   client,
   contentPath: "./content",
-  tableName: "articles",
+  tableName: "articles_local_384",
   embeddingOptions: {
     provider: "local",
   },
 });
 ```
 
-That keeps the implementation simple, but it also means a failed rebuild can
-leave the index partially repopulated.
+The rebuild runs in two phases:
 
-Changing an embedding provider or dimension count requires a full re-embed.
-For existing local indexes created with the older padded-local behavior, create
-or recreate a 384-dimensional table before rebuilding. Those older local
-768-dimensional tables stored 384 model values followed by zero padding;
-`indexContent()` clears rows but does not change the table's `F32_BLOB` width.
+1. build: every file is read, parsed, and embedded in memory, touching no database state
+2. replace: the delete and all inserts run in a single write transaction
 
-For Gemini specifically, indexes created with the retired `text-embedding-004`
-model must be rebuilt for `gemini-embedding-2` even when staying at 768
-dimensions, because the model and query/document formatting both changed. If
-you adopt Gemini's 3072-dimensional default, recreate the vector table or build
-into a separate table first; clearing rows with `indexContent()` does not change
-the table's `F32_BLOB` width.
+That means:
+
+- a failed rebuild leaves the previously indexed rows exactly as they were
+- provider or dimension changes should still use a parallel table migration
+- `createTable()` does not resize an existing vector column
+
+Files are discovered and indexed in sorted path order, so a rebuild is deterministic.
+
+If provider, dimensions, model, endpoint, or embedding-space assumptions change, fully reindex into a new table. See the canonical [Migration and reindexing guide](./MIGRATIONS.md).
+
+### Costs Of The Two-Phase Rebuild
+
+Atomicity is not free, and both costs scale with corpus size:
+
+- **Peak memory holds the whole corpus.** The build phase keeps every document in memory: content, frontmatter, and one embedding array per document. The replace phase then builds insert statements including a JSON copy of each embedding, roughly 5-8 KB per document at 384 dimensions and considerably more at 3072. Documents are released as their statements are built, but peak usage is still proportional to the entire corpus rather than to one file.
+- **Remote clients send one request.** Against Turso or any remote client, the delete and every insert travel as a single batch. There is no chunking fallback, because splitting the batch would give up the atomicity this design exists to provide. A corpus large enough to exceed a remote request-size limit fails as an opaque `phase: "replace"` error.
+
+For very large corpora, index into a parallel table and switch reads over once it validates, rather than rebuilding a live table in place. See the [Migration and reindexing guide](./MIGRATIONS.md).
+
+## Content Requirements
+
+Two authoring mistakes fail a file at the `parse` stage rather than corrupting the rebuild:
+
+- **Frontmatter `title` must be a scalar.** Strings, numbers, booleans, and dates are accepted; dates are stored as ISO strings. A structured title such as a YAML list fails the file. A missing or empty title still falls back to the filename.
+- **Slugs must be unique.** The slug comes from the path with the extension removed, so `foo.md` and `foo.markdown` collide. Files are processed in sorted path order and the first file to claim a slug keeps it, so `foo.markdown` wins and `foo.md` is reported as the failure.
+
+Both are governed by `failurePolicy` like any other build failure, so they abort by default and are skippable.
+
+## Failure Handling
+
+`indexContent()` throws `IndexingError` instead of reporting a partially applied rebuild. The error carries `phase` (`"build"` or `"replace"`), a `failures` array, and the underlying error as `cause`.
+
+```ts
+import { indexContent, IndexingError } from "libsql-search";
+
+try {
+  await indexContent({ client, contentPath: "./content" });
+} catch (error) {
+  if (error instanceof IndexingError) {
+    for (const failure of error.failures) {
+      console.error(`${failure.file} failed during ${failure.stage}`);
+    }
+  }
+
+  throw error;
+}
+```
+
+By default one bad file aborts the whole rebuild. To index everything that can be indexed, opt into `failurePolicy: "skip"`:
+
+```ts
+const result = await indexContent({
+  client,
+  contentPath: "./content",
+  failurePolicy: "skip",
+});
+
+if (result.partial) {
+  console.warn(`Indexed ${result.success} of ${result.total} files`);
+}
+```
+
+Skipped rebuilds still replace the table, so treat `partial: true` as a build warning rather than a clean rebuild. If every discovered file fails, the rebuild throws rather than trading a valid index for an empty one.
+
+## Empty Source Directories
+
+An empty source directory throws by default, because silently leaving stale rows in place serves search traffic from content that no longer exists. Emptying an index has to be intentional:
+
+```ts
+await indexContent({
+  client,
+  contentPath: "./content",
+  allowEmptyIndex: true,
+});
+```
+
+Both behaviors changed in a breaking way: partial failures used to be counted and reported, and an empty directory used to return zeros without clearing the table.
 
 ## Quality Guidelines
 
 - include descriptive frontmatter titles
 - add meaningful `tags` when they help retrieval
-- use the same embedding provider and dimensions at index and query time
+- use the same provider and dimensions at index and query time
 - keep `maxLength` intentional if your content is very large
 - start with a smaller search `limit` and tune from real query behavior
 
 ## Build Integration
 
-Many projects wire indexing into a dedicated script and call it before their
-site build:
+Many projects wire indexing into a dedicated script and call it before their site build:
 
 ```json
 {
@@ -69,14 +134,11 @@ site build:
 
 ## Table Names
 
-`tableName` must be an ASCII SQLite identifier matching
-`[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index
-SQL, so reserved words such as `"select"` work safely. Invalid names fail before
-database calls or embedding generation.
+`tableName` must be an ASCII SQLite identifier matching `[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index SQL, so reserved words such as `"select"` work safely. Invalid names fail before database calls or embedding generation.
 
 ## Runtime Notes
 
 - local embeddings may download and cache a model on the first run
 - Node users need `@libsql/client` installed alongside the package
-- the repository validates both the npm package build and `deno check`, but the
-  indexing flow itself still depends on filesystem access
+- hosted providers send indexed or queried text to external services
+- the repository validates package build and `deno check`, but indexing still depends on filesystem access