libsql-search 0.1.5 → 0.2.0

package/docs/API.md

@@ -12,12 +12,25 @@
 - `getArticlesByFolder`
 - `getFolders`
 - `generateEmbedding`
+- `generateEmbeddings`
+- `createEmbeddingProvider`
+- `getEmbeddingProviderMetadata`
+- `validateEmbeddingBatch`
 - `padEmbedding`
 - `prepareTextForEmbedding`
 
 It also exports these types:
 
 - `EmbeddingProvider`
+- `EmbeddingIntent`
+- `EmbeddingBatchMode`
+- `EmbeddingBatchBehavior`
+- `EmbeddingProviderMetadata`
+- `EmbeddingRequestOptions`
+- `EmbeddingProviderClient`
+- `EmbeddingBatchResult`
+- `EmbeddingBatchItemResult`
+- `EmbeddingBatchItem`
 - `EmbeddingOptions`
 - `IndexerOptions`
 - `IndexedDocument`
@@ -37,6 +50,11 @@ Defaults:
 - `tableName`: `"articles"`
 - `dimensions`: `768`
 
+`tableName` must be an ASCII SQLite identifier matching
+`[A-Za-z_][A-Za-z0-9_]*`. Valid identifiers are quoted internally, so reserved
+words such as `"select"` are safe to use. `dimensions` must be a positive
+integer.
+
 The created schema includes:
 
 - `id` primary key
@@ -71,6 +89,8 @@ Defaults:
 - `exclude`: ["node_modules", ".git", "dist", "build"]
 - `tableName`: `"articles"`
 
+`tableName` follows the same identifier policy as `createTable()`.
+
 Return shape:
 
 ```ts
@@ -86,6 +106,8 @@ Behavior notes:
 - `indexContent()` deletes existing rows in the target table before rebuilding
 - frontmatter `title`, `description`, and `tags` are folded into the embedding
   text
+- embeddings default to `intent: "document"` unless `embeddingOptions.intent`
+  is set explicitly
 - if a file has no frontmatter title, the filename becomes the title
 
 ## `search(options)`
@@ -107,6 +129,10 @@ Defaults:
 - `limit`: `10`
 - `tableName`: `"articles"`
 
+`limit` must be an integer from `1` through `100`; invalid values are rejected
+before query embedding generation. `tableName` follows the same identifier
+policy as `createTable()`.
+
 Result shape:
 
 ```ts
@@ -124,6 +150,9 @@ interface SearchResult {
 
 Lower `distance` values are better matches.
 
+Search embeddings default to `intent: "query"` unless
+`embeddingOptions.intent` is set explicitly.
+
 ## Article Retrieval Helpers
 
 ### `getAllArticles(client, tableName?)`
@@ -142,12 +171,99 @@ Returns articles in a specific folder.
 
 Returns distinct folder names from the index.
 
+All article retrieval helpers validate `tableName` before executing SQL.
+
 ## Embedding Helpers
 
 ### `generateEmbedding(text, options?)`
 
 Generates an embedding for arbitrary text using the selected provider.
 
+### `generateEmbeddings(texts, options?)`
+
+Generates an ordered batch of embeddings. Empty batches return `[]` without
+creating a hosted provider client or making a network request.
+
+### `createEmbeddingProvider(options?)`
+
+Creates a provider client with immutable metadata and an `embed(texts, options?)`
+method. Provider clients return a rich `EmbeddingBatchResult`; the compatibility
+helpers `generateEmbedding()` and `generateEmbeddings()` continue returning only
+vectors.
+
+```ts
+const provider = createEmbeddingProvider({
+  provider: "openai",
+  apiKey: process.env.OPENAI_API_KEY,
+  dimensions: 1536,
+});
+
+console.log(provider.metadata);
+```
+
+Provider metadata includes:
+
+- `name`
+- `model`
+- `dimensions`
+- `batch.mode`
+- `batch.maxSize`, when the provider has a hard maximum
+
+Hosted provider clients are scoped to their options. The library does not reuse
+a Gemini or OpenAI client created with a different API key or configuration.
+
+### `getEmbeddingProviderMetadata(options?)`
+
+Returns the same metadata exposed by `createEmbeddingProvider(options).metadata`
+without resolving hosted-provider credentials.
+
+Provider batch metadata uses:
+
+```ts
+type EmbeddingBatchMode = "native" | "sequential";
+
+interface EmbeddingBatchBehavior {
+  mode: EmbeddingBatchMode;
+  maxSize?: number;
+}
+```
+
+`"native"` means the upstream provider accepts the batch in one request.
+`"sequential"` means the library accepts an input batch but processes items one
+at a time. If `maxSize` is present, the library enforces it before provider or
+network work.
+
+Provider clients return:
+
+```ts
+interface EmbeddingBatchResult {
+  embeddings: number[][];
+  provider: "local" | "gemini" | "openai";
+  model: string;
+  dimensions: number;
+  intent: "document" | "query";
+}
+```
+
+### `validateEmbeddingBatch(items, expectedCount, expectedDimensions, provider)`
+
+Validates provider results before they are written to the database. It checks
+cardinality, dimensions, finite numeric values, and indexed batch ordering.
+
+`EmbeddingOptions` supports:
+
+```ts
+interface EmbeddingOptions {
+  provider?: "local" | "gemini" | "openai";
+  apiKey?: string;
+  dimensions?: number;
+  maxLength?: number;
+  intent?: "document" | "query";
+  timeoutMs?: number;
+  signal?: AbortSignal;
+}
+```
+
 ### `padEmbedding(embedding, targetDimensions)`
 
 Pads or truncates an embedding array to the requested length.

package/docs/INDEXING.md

@@ -57,8 +57,10 @@ site build:
 
 ## Table Names
 
-`tableName` is interpolated into SQL. Treat it as a trusted identifier coming
-from your own configuration, not from user input.
+`tableName` must be an ASCII SQLite identifier matching
+`[A-Za-z_][A-Za-z0-9_]*`. Valid names are quoted internally for table and index
+SQL, so reserved words such as `"select"` work safely. Invalid names fail before
+database calls or embedding generation.
 
 ## Runtime Notes
 

package/docs/PROVIDERS.md

@@ -18,15 +18,74 @@ interface EmbeddingOptions {
   apiKey?: string;
   dimensions?: number;
   maxLength?: number;
+  intent?: "document" | "query";
+  timeoutMs?: number;
+  signal?: AbortSignal;
 }
 ```
 
 - `provider` defaults to `"local"`
 - `dimensions` defaults to `768`
 - `maxLength` defaults to `8000`
+- `intent` can be `"document"` or `"query"`; indexing defaults to
+  `"document"` and search defaults to `"query"` unless explicitly set
+- `timeoutMs` defaults to `30000`
 - `apiKey` is optional in code, but required for hosted providers unless the
   matching environment variable is available
 
+## Provider Contract
+
+Each provider exposes immutable metadata:
+
+```ts
+interface EmbeddingProviderMetadata {
+  name: "local" | "gemini" | "openai";
+  model: string;
+  dimensions: number;
+  batch: {
+    mode: "native" | "sequential";
+    maxSize?: number;
+  };
+}
+```
+
+Use `getEmbeddingProviderMetadata(options)` or
+`createEmbeddingProvider(options).metadata` to inspect the effective model,
+dimensions, and batch behavior. Metadata inspection does not require hosted
+provider credentials.
+
+Batch modes:
+
+- `"native"` means the upstream provider accepts the batch in one request
+- `"sequential"` means the library accepts a batch and processes items one at a
+  time
+- when `maxSize` is present, it is a hard maximum enforced before provider or
+  network work
+
+`generateEmbeddings(texts, options)` returns vectors in the same order as the
+input texts. Provider responses are validated before database writes:
+
+- result count must match input count
+- each vector must match the provider's effective dimensions
+- every vector value must be a finite number
+- indexed batch responses must contain unique contiguous indices and are
+  reordered before being returned
+
+Empty batches return `[]` without loading a local model, creating hosted clients,
+or making network calls.
+
+Lower-level provider clients return an `EmbeddingBatchResult` with the validated
+vectors plus provider, model, dimensions, and intent. The compatibility helpers
+`generateEmbedding()` and `generateEmbeddings()` return only arrays.
+
+Gemini and OpenAI clients are scoped to their current options. They are not
+cached globally across different credentials or configurations. The local Xenova
+model can be cached by model name.
+
+Hosted provider failures are reported with bounded provider/status/request-id
+context and without raw upstream bodies, credentials, Authorization headers, or
+full URLs with query strings.
+
 ## Local
 
 Provider value: `local`
@@ -45,6 +104,8 @@ Notes:
 
 - the model emits 384 dimensions and `libsql-search` pads or truncates to your
   requested size
+- metadata reports the requested output dimensions
+- batch metadata is `{ mode: "sequential" }`
 - the first run downloads the model and can take longer on a fresh machine
 - no API key is required
 
@@ -65,6 +126,8 @@ Behavior:
 
 - if `apiKey` is omitted, the library reads `GEMINI_API_KEY`
 - Gemini returns 768 dimensions natively
+- metadata reports `text-embedding-004` and 768 dimensions
+- batch metadata is `{ mode: "sequential" }`
 - the current implementation does not expose model selection
 
 ## OpenAI
@@ -86,6 +149,9 @@ Behavior:
 
 - if `apiKey` is omitted, the library reads `OPENAI_API_KEY`
 - the request sends the `dimensions` value to the OpenAI embeddings API
+- metadata reports `text-embedding-3-small` when `dimensions <= 1536` and
+  `text-embedding-3-large` when `dimensions > 1536`
+- batch metadata is `{ mode: "native", maxSize: 2048 }`
 - use the same dimension count in `createTable()`
 
 ## Dimension Guidelines

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.1.5",
+  "version": "0.2.0",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",