libsql-search 0.1.4 → 0.1.5

package/README.md

@@ -2,482 +2,124 @@
 
 [![npm version](https://img.shields.io/npm/v/libsql-search.svg)](https://www.npmjs.com/package/libsql-search)
 [![JSR](https://jsr.io/badges/@logan/libsql-search)](https://jsr.io/@logan/libsql-search)
-[![npm downloads](https://img.shields.io/npm/dm/libsql-search.svg)](https://www.npmjs.com/package/libsql-search)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 [![CI](https://github.com/llbbl/libsql-search/actions/workflows/ci.yml/badge.svg)](https://github.com/llbbl/libsql-search/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+`libsql-search` adds semantic search to Markdown-backed sites using libSQL/Turso.
+It indexes frontmatter and content from files on disk, stores vectors in your
+database, and lets you query by meaning instead of exact keywords.
 
-Semantic search for static sites using libSQL/Turso with multi-provider embeddings.
+Use it when you want:
 
-Add AI-powered vector search to your Astro, Next.js, or any static site with minimal configuration. Index markdown content, generate embeddings locally or via API, and provide lightning-fast semantic search to your users.
+- a small TypeScript library instead of a hosted search product
+- one search index shared across static-site builds and app routes
+- local or API-based embeddings behind the same indexing/search API
+- direct control over table names, dimensions, content shape, and deployment
 
-## Features
+## What It Supports
 
-- 🔍 **Semantic Search** - Find content by meaning, not just keywords
-- 🌐 **Multi-Provider Embeddings** - Choose local (Xenova), Gemini, or OpenAI
-- ⚡ **Edge-Ready** - Works with Turso's global edge database
-- 📝 **Markdown Support** - Built-in gray-matter parsing
-- 🎯 **Type-Safe** - Full TypeScript support
-- 🆓 **Free Tier Friendly** - Local embeddings require no API keys
+- Markdown indexing from local directories with frontmatter via `gray-matter`
+- libSQL/Turso storage and vector search
+- Embedding providers that exist in the code today: local
+  `Xenova/all-MiniLM-L6-v2`, Google Gemini `text-embedding-004`, and OpenAI
+  `text-embedding-3-small` and `text-embedding-3-large`
+- npm distribution plus JSR publishing
 
-## Installation
+## Install
+
+`@libsql/client` is a peer dependency.
 
-**npm:**
 ```bash
-npm install libsql-search @libsql/client
+pnpm add libsql-search @libsql/client
 ```
 
-**pnpm:**
 ```bash
-pnpm add libsql-search @libsql/client
+npm install libsql-search @libsql/client
 ```
 
-**JSR:**
 ```bash
-deno add @logan/libsql-search
+deno add jsr:@logan/libsql-search npm:@libsql/client
 ```
 
+For npm usage, the package requires Node `>=22.12.0`.
+Node examples in this README import from `libsql-search` and `@libsql/client`.
+In Deno, after `deno add`, import from `@logan/libsql-search` and
+`@libsql/client`.
+
 ## Quick Start
 
-### 1. Set Up Your Database
+The shortest working flow is:
+
+1. create a libSQL client
+2. create the search table
+3. index a Markdown directory
+4. query it with the same embedding provider and dimensions
 
-```typescript
-import { createClient } from '@libsql/client';
-import { createTable } from 'libsql-search';
+```ts
+import { createClient } from "@libsql/client";
+import { createTable, indexContent, search } from "libsql-search";
 
 const client = createClient({
-  url: 'libsql://your-db.turso.io',
-  authToken: 'your-auth-token'
+  url: "libsql://your-db.turso.io",
+  authToken: "your-auth-token",
 });
 
-// Create the articles table with vector index
-await createTable(client, 'articles', 768);
-```
-
-### 2. Index Your Content
-
-```typescript
-import { indexContent } from 'libsql-search';
+await createTable(client, "articles", 768);
 
-const result = await indexContent({
+await indexContent({
   client,
-  contentPath: './content',
+  contentPath: "./content",
   embeddingOptions: {
-    provider: 'local', // or 'gemini', 'openai'
-    dimensions: 768
+    provider: "local",
+    dimensions: 768,
   },
-  onProgress: (current, total, file) => {
-    console.log(`[${current}/${total}] Indexing: ${file}`);
-  }
 });
 
-console.log(`Indexed ${result.success}/${result.total} documents`);
-```
-
-### 3. Search Your Content
-
-```typescript
-import { search } from 'libsql-search';
-
 const results = await search({
   client,
-  query: 'how to deploy astro',
+  query: "how do I deploy my docs site",
   limit: 5,
   embeddingOptions: {
-    provider: 'local'
-  }
-});
-
-results.forEach(result => {
-  console.log(`${result.title} (${result.distance})`);
-});
-```
-
-## Embedding Providers
-
-### Local (Xenova/Transformers.js)
-
-**Free, no API key required**. Runs `all-MiniLM-L6-v2` in Node.js using ONNX.
-
-```typescript
-embeddingOptions: {
-  provider: 'local',
-  dimensions: 768  // 384 native, padded to 768
-}
-```
-
-**Pros:**
-- ✅ No API costs
-- ✅ No rate limits
-- ✅ Works offline
-- ✅ Privacy-friendly
-
-**Cons:**
-- ⚠️ First run downloads model (~50MB)
-- ⚠️ Slower than API-based options
-- ⚠️ Lower quality than large models
-
-### Google Gemini
-
-**Free tier: 1,500 requests/day**. Uses `text-embedding-004` model.
-
-```typescript
-embeddingOptions: {
-  provider: 'gemini',
-  apiKey: process.env.GEMINI_API_KEY,
-  dimensions: 768  // native
-}
-```
-
-**Pros:**
-- ✅ Generous free tier
-- ✅ High quality embeddings
-- ✅ Fast
-
-**Cons:**
-- ⚠️ Requires API key
-- ⚠️ Rate limited
-
-### OpenAI
-
-**Paid only**. Uses `text-embedding-3-small` or `text-embedding-3-large`.
-
-```typescript
-embeddingOptions: {
-  provider: 'openai',
-  apiKey: process.env.OPENAI_API_KEY,
-  dimensions: 1536  // or 3072 for large
-}
-```
-
-**Pros:**
-- ✅ Highest quality
-- ✅ Very fast
-- ✅ Configurable dimensions
-
-**Cons:**
-- ⚠️ Costs money ($0.02 per 1M tokens)
-- ⚠️ Requires API key
-
-## API Reference
-
-### Indexing
-
-#### `indexContent(options)`
-
-Index markdown files from a directory.
-
-```typescript
-interface IndexerOptions {
-  client: Client;                    // libSQL client
-  contentPath: string;               // Path to content directory
-  embeddingOptions?: EmbeddingOptions;
-  fileExtensions?: string[];         // Default: ['.md', '.markdown']
-  exclude?: string[];                // Default: ['node_modules', '.git']
-  tableName?: string;                // Default: 'articles'
-  onProgress?: (current, total, file) => void;
-}
-```
-
-#### `createTable(client, tableName?, dimensions?)`
-
-Create the articles table with vector index.
-
-### Searching
-
-#### `search(options)`
-
-Perform semantic search.
-
-```typescript
-interface SearchOptions {
-  client: Client;
-  query: string;
-  limit?: number;                    // Default: 10
-  tableName?: string;                // Default: 'articles'
-  embeddingOptions?: EmbeddingOptions;
-}
-```
-
-Returns `SearchResult[]`:
-
-```typescript
-interface SearchResult {
-  id: number;
-  slug: string;
-  title: string;
-  content: string;
-  folder: string;
-  tags: string[];
-  distance: number;  // Lower is better
-  created_at: string;
-}
-```
-
-#### `getAllArticles(client, tableName?)`
-
-Get all articles (useful for building static pages).
-
-#### `getArticleBySlug(client, slug, tableName?)`
-
-Get a single article by slug.
-
-#### `getArticlesByFolder(client, folder, tableName?)`
-
-Get all articles in a folder.
-
-#### `getFolders(client, tableName?)`
-
-Get all unique folders.
-
-### Embeddings
-
-#### `generateEmbedding(text, options?)`
-
-Generate embeddings for arbitrary text.
-
-```typescript
-interface EmbeddingOptions {
-  provider?: 'local' | 'gemini' | 'openai';
-  apiKey?: string;
-  dimensions?: number;
-  maxLength?: number;  // Default: 8000
-}
-```
-
-#### `prepareTextForEmbedding(fields)`
-
-Combine multiple fields into embedding text.
-
-```typescript
-const text = prepareTextForEmbedding({
-  title: 'My Article',
-  description: 'A description',
-  content: '# Content here',
-  tags: ['astro', 'turso']
-});
-```
-
-## Framework Integration
-
-### Astro
-
-**Search API Endpoint** (`src/pages/api/search.json.ts`):
-
-```typescript
-import type { APIRoute } from 'astro';
-import { createClient } from '@libsql/client';
-import { search } from 'libsql-search';
-
-export const prerender = false;
-
-const client = createClient({
-  url: import.meta.env.TURSO_DB_URL,
-  authToken: import.meta.env.TURSO_AUTH_TOKEN
-});
-
-export const POST: APIRoute = async ({ request }) => {
-  const { query, limit = 10 } = await request.json();
-
-  const results = await search({
-    client,
-    query,
-    limit,
-    embeddingOptions: { provider: 'local' }
-  });
-
-  return new Response(JSON.stringify({ results }), {
-    headers: { 'Content-Type': 'application/json' }
-  });
-};
-```
-
-**Static Page Generation** (`src/pages/[...slug].astro`):
-
-```astro
----
-import { createClient } from '@libsql/client';
-import { getAllArticles, getArticleBySlug } from 'libsql-search';
-
-export const prerender = true;
-
-const client = createClient({
-  url: import.meta.env.TURSO_DB_URL,
-  authToken: import.meta.env.TURSO_AUTH_TOKEN
-});
-
-export async function getStaticPaths() {
-  const articles = await getAllArticles(client);
-  return articles.map(article => ({
-    params: { slug: article.slug }
-  }));
-}
-
-const { slug } = Astro.params;
-const article = await getArticleBySlug(client, slug);
----
-
-<article>
-  <h1>{article.title}</h1>
-  <div set:html={article.content} />
-</article>
-```
-
-### Next.js
-
-**API Route** (`app/api/search/route.ts`):
-
-```typescript
-import { createClient } from '@libsql/client';
-import { search } from 'libsql-search';
-import { NextRequest } from 'next/server';
-
-const client = createClient({
-  url: process.env.TURSO_DB_URL!,
-  authToken: process.env.TURSO_AUTH_TOKEN!
-});
-
-export async function POST(request: NextRequest) {
-  const { query, limit = 10 } = await request.json();
-
-  const results = await search({
-    client,
-    query,
-    limit,
-    embeddingOptions: { provider: 'local' }
-  });
-
-  return Response.json({ results });
-}
-```
-
-**Static Generation** (`app/[slug]/page.tsx`):
-
-```typescript
-import { createClient } from '@libsql/client';
-import { getAllArticles, getArticleBySlug } from 'libsql-search';
-
-const client = createClient({
-  url: process.env.TURSO_DB_URL!,
-  authToken: process.env.TURSO_AUTH_TOKEN!
+    provider: "local",
+    dimensions: 768,
+  },
 });
 
-export async function generateStaticParams() {
-  const articles = await getAllArticles(client);
-  return articles.map(article => ({
-    slug: article.slug
-  }));
-}
-
-export default async function Page({ params }: { params: { slug: string } }) {
-  const article = await getArticleBySlug(client, params.slug);
-
-  return (
-    <article>
-      <h1>{article.title}</h1>
-      <div dangerouslySetInnerHTML={{ __html: article.content }} />
-    </article>
-  );
-}
-```
-
-## Best Practices
-
-### Embedding Dimensions
-
-- Use **768 dimensions** for best compatibility
-- Local model outputs 384, automatically padded to 768
-- Gemini outputs 768 natively
-- OpenAI supports custom dimensions
-
-### Index Updates
-
-Create a script to re-index content:
-
-```json
-{
-  "scripts": {
-    "index": "node scripts/index.js",
-    "build": "npm run index && astro build"
-  }
-}
+console.log(results.map((result) => ({
+  slug: result.slug,
+  title: result.title,
+  distance: result.distance,
+})));
 ```
 
-### Search Quality
-
-Improve search results:
-
-1. **Include relevant fields** in embedding text (title, description, tags)
-2. **Truncate long content** to avoid noise
-3. **Use the same provider** for indexing and search
-4. **Experiment with distance thresholds** (lower is better)
-
-### Performance
-
-- **Cache the embedding model** (done automatically)
-- **Use edge databases** (Turso) for low latency
-- **Implement search debouncing** in the UI
-- **Limit result count** to 5-10 for best UX
-
-## Examples
-
-See the `/examples` directory for complete implementations:
-
-- [Astro Documentation Site](./examples/astro-docs)
-- [Next.js Blog](./examples/nextjs-blog)
-- [CLI Indexer](./examples/cli-indexer)
+Important behavior:
 
-## CLI Usage
+- Call `createTable()` before indexing or searching.
+- Keep dimensions aligned across table creation, indexing, and search queries.
+- `indexContent()` clears existing rows before rebuilding the index.
 
-For a standalone indexing script:
+## Core API
 
-```javascript
-// scripts/index.js
-import { createClient } from '@libsql/client';
-import { createTable, indexContent } from 'libsql-search';
+- `createTable(client, tableName?, dimensions?)`
+- `indexContent(options)`
+- `search(options)`
+- `getAllArticles(client, tableName?)`
+- `getArticleBySlug(client, slug, tableName?)`
+- `getArticlesByFolder(client, folder, tableName?)`
+- `getFolders(client, tableName?)`
+- `generateEmbedding(text, options?)`
+- `prepareTextForEmbedding(fields)`
 
-const client = createClient({
-  url: process.env.TURSO_DB_URL,
-  authToken: process.env.TURSO_AUTH_TOKEN
-});
-
-await createTable(client);
-
-const result = await indexContent({
-  client,
-  contentPath: './content',
-  embeddingOptions: {
-    provider: process.env.EMBEDDING_PROVIDER || 'local'
-  },
-  onProgress: (current, total, file) => {
-    console.log(`[${current}/${total}] ${file}`);
-  }
-});
-
-console.log(`✅ Indexed ${result.success} documents`);
-```
+## Docs
 
-Run with:
-```bash
-node --env-file=.env scripts/index.js
-```
+- [Docs index](./docs/README.md)
+- [Provider guide](./docs/PROVIDERS.md)
+- [API reference](./docs/API.md)
+- [Integration examples](./docs/INTEGRATIONS.md)
+- [Indexing and operations](./docs/INDEXING.md)
+- [Troubleshooting](./docs/TROUBLESHOOTING.md)
+- [Release workflow](./docs/RELEASING.md)
 
 ## License
 
 MIT
-
-## Contributing
-
-Contributions welcome! Please open an issue or PR on [GitHub](https://github.com/llbbl/libsql-search).
-
-## Related Projects
-
-- [Turso](https://turso.tech) - Edge SQLite database
-- [libSQL](https://github.com/tursodatabase/libsql) - Open source SQLite fork
-- [Astro](https://astro.build) - Static site framework
-- [Transformers.js](https://huggingface.co/docs/transformers.js) - ML models in JavaScript
-
-## Support
-
-- [Documentation](https://github.com/llbbl/libsql-search)
-- [Issues](https://github.com/llbbl/libsql-search/issues)
-- [Discussions](https://github.com/llbbl/libsql-search/discussions)

package/docs/API.md

@@ -0,0 +1,167 @@
+# API Reference
+
+## Exports
+
+`libsql-search` exports:
+
+- `createTable`
+- `indexContent`
+- `search`
+- `getAllArticles`
+- `getArticleBySlug`
+- `getArticlesByFolder`
+- `getFolders`
+- `generateEmbedding`
+- `padEmbedding`
+- `prepareTextForEmbedding`
+
+It also exports these types:
+
+- `EmbeddingProvider`
+- `EmbeddingOptions`
+- `IndexerOptions`
+- `IndexedDocument`
+- `SearchOptions`
+- `SearchResult`
+
+## `createTable(client, tableName?, dimensions?)`
+
+Creates the table and supporting indexes used by search.
+
+```ts
+await createTable(client, "articles", 768);
+```
+
+Defaults:
+
+- `tableName`: `"articles"`
+- `dimensions`: `768`
+
+The created schema includes:
+
+- `id` primary key
+- `slug`
+- `title`
+- `content`
+- `folder`
+- `tags`
+- `embedding`
+- `created_at`
+- `updated_at`
+
+## `indexContent(options)`
+
+Indexes Markdown files from a directory on disk.
+
+```ts
+interface IndexerOptions {
+  client: Client;
+  contentPath: string;
+  embeddingOptions?: EmbeddingOptions;
+  fileExtensions?: string[];
+  exclude?: string[];
+  tableName?: string;
+  onProgress?: (current: number, total: number, file: string) => void;
+}
+```
+
+Defaults:
+
+- `fileExtensions`: [".md", ".markdown"]
+- `exclude`: ["node_modules", ".git", "dist", "build"]
+- `tableName`: `"articles"`
+
+Return shape:
+
+```ts
+{
+  success: number;
+  failed: number;
+  total: number;
+}
+```
+
+Behavior notes:
+
+- `indexContent()` deletes existing rows in the target table before rebuilding
+- frontmatter `title`, `description`, and `tags` are folded into the embedding
+  text
+- if a file has no frontmatter title, the filename becomes the title
+
+## `search(options)`
+
+Generates a query embedding and performs vector similarity search.
+
+```ts
+interface SearchOptions {
+  client: Client;
+  query: string;
+  limit?: number;
+  tableName?: string;
+  embeddingOptions?: EmbeddingOptions;
+}
+```
+
+Defaults:
+
+- `limit`: `10`
+- `tableName`: `"articles"`
+
+Result shape:
+
+```ts
+interface SearchResult {
+  id: number;
+  slug: string;
+  title: string;
+  content: string;
+  folder: string;
+  tags: string[];
+  distance: number;
+  created_at: string;
+}
+```
+
+Lower `distance` values are better matches.
+
+## Article Retrieval Helpers
+
+### `getAllArticles(client, tableName?)`
+
+Returns all indexed articles ordered by title.
+
+### `getArticleBySlug(client, slug, tableName?)`
+
+Returns one article or `null`.
+
+### `getArticlesByFolder(client, folder, tableName?)`
+
+Returns articles in a specific folder.
+
+### `getFolders(client, tableName?)`
+
+Returns distinct folder names from the index.
+
+## Embedding Helpers
+
+### `generateEmbedding(text, options?)`
+
+Generates an embedding for arbitrary text using the selected provider.
+
+### `padEmbedding(embedding, targetDimensions)`
+
+Pads or truncates an embedding array to the requested length.
+
+### `prepareTextForEmbedding(fields)`
+
+Combines title, description, tags, and content into the text sent to the
+embedding model.
+
+```ts
+const text = prepareTextForEmbedding({
+  title: "My Article",
+  description: "How semantic search works",
+  tags: ["search", "turso"],
+  content: "# Content",
+});
+```

package/docs/INDEXING.md

@@ -0,0 +1,68 @@
+# Indexing And Operations
+
+## Content Shape
+
+`indexContent()` walks a directory tree, reads Markdown files, parses
+frontmatter with `gray-matter`, and stores:
+
+- `slug`
+- `title`
+- `content`
+- `folder`
+- `tags`
+- `embedding`
+
+The slug is derived from the file path relative to `contentPath`.
+
+## Rebuild Behavior
+
+`indexContent()` clears the target table before rebuilding:
+
+```ts
+await indexContent({
+  client,
+  contentPath: "./content",
+  tableName: "articles",
+  embeddingOptions: {
+    provider: "local",
+    dimensions: 768,
+  },
+});
+```
+
+That keeps the implementation simple, but it also means a failed rebuild can
+leave the index partially repopulated.
+
+## Quality Guidelines
+
+- include descriptive frontmatter titles
+- add meaningful `tags` when they help retrieval
+- use the same embedding provider and dimensions at index and query time
+- keep `maxLength` intentional if your content is very large
+- start with a smaller search `limit` and tune from real query behavior
+
+## Build Integration
+
+Many projects wire indexing into a dedicated script and call it before their
+site build:
+
+```json
+{
+  "scripts": {
+    "index": "node ./scripts/index.js",
+    "build": "pnpm index && astro build"
+  }
+}
+```
+
+## Table Names
+
+`tableName` is interpolated into SQL. Treat it as a trusted identifier coming
+from your own configuration, not from user input.
+
+## Runtime Notes
+
+- local embeddings may download a model on the first run
+- Node users need `@libsql/client` installed alongside the package
+- the repository validates both the npm package build and `deno check`, but the
+  indexing flow itself still depends on filesystem access

package/docs/INTEGRATIONS.md

@@ -0,0 +1,143 @@
+# Integration Examples
+
+These examples show the current exported API wired into typical server-side
+routes. They are intentionally small so you can adapt them to your app.
+
+## Astro Search Endpoint
+
+```ts
+import type { APIRoute } from "astro";
+import { createClient } from "@libsql/client";
+import { search } from "libsql-search";
+
+export const prerender = false;
+
+const client = createClient({
+  url: import.meta.env.TURSO_DB_URL,
+  authToken: import.meta.env.TURSO_AUTH_TOKEN,
+});
+
+export const POST: APIRoute = async ({ request }) => {
+  const { query, limit = 10 } = await request.json();
+
+  const results = await search({
+    client,
+    query,
+    limit,
+    embeddingOptions: {
+      provider: "local",
+      dimensions: 768,
+    },
+  });
+
+  return new Response(JSON.stringify({ results }), {
+    headers: { "Content-Type": "application/json" },
+  });
+};
+```
+
+## Astro Static Paths
+
+```ts
+import { createClient } from "@libsql/client";
+import { getAllArticles, getArticleBySlug } from "libsql-search";
+
+const client = createClient({
+  url: import.meta.env.TURSO_DB_URL,
+  authToken: import.meta.env.TURSO_AUTH_TOKEN,
+});
+
+export async function getStaticPaths() {
+  const articles = await getAllArticles(client);
+
+  return articles.map((article) => ({
+    params: { slug: article.slug },
+  }));
+}
+
+const article = await getArticleBySlug(client, "guides/getting-started");
+```
+
+## Next.js Route Handler
+
+```ts
+import { createClient } from "@libsql/client";
+import { search } from "libsql-search";
+import { NextRequest } from "next/server";
+
+const client = createClient({
+  url: process.env.TURSO_DB_URL!,
+  authToken: process.env.TURSO_AUTH_TOKEN!,
+});
+
+export async function POST(request: NextRequest) {
+  const { query, limit = 10 } = await request.json();
+
+  const results = await search({
+    client,
+    query,
+    limit,
+    embeddingOptions: {
+      provider: "local",
+      dimensions: 768,
+    },
+  });
+
+  return Response.json({ results });
+}
+```
+
+## Next.js Static Params
+
+```tsx
+import { createClient } from "@libsql/client";
+import { getAllArticles, getArticleBySlug } from "libsql-search";
+
+const client = createClient({
+  url: process.env.TURSO_DB_URL!,
+  authToken: process.env.TURSO_AUTH_TOKEN!,
+});
+
+export async function generateStaticParams() {
+  const articles = await getAllArticles(client);
+
+  return articles.map((article) => ({
+    slug: article.slug,
+  }));
+}
+
+export default async function Page({ params }: { params: Promise<{ slug: string }> }) {
+  const { slug } = await params;
+  const article = await getArticleBySlug(client, slug);
+
+  return <article>{article?.title}</article>;
+}
+```
+
+## Build-Time Index Script
+
+A small script is usually enough to rebuild the index before a site build.
+
+```ts
+import { createClient } from "@libsql/client";
+import { createTable, indexContent } from "libsql-search";
+
+const client = createClient({
+  url: process.env.TURSO_DB_URL!,
+  authToken: process.env.TURSO_AUTH_TOKEN!,
+});
+
+await createTable(client, "articles", 768);
+
+await indexContent({
+  client,
+  contentPath: "./content",
+  embeddingOptions: {
+    provider: process.env.EMBEDDING_PROVIDER as "local" | "gemini" | "openai" | undefined,
+    dimensions: 768,
+  },
+});
+```
+
+Pair this with your framework build command so indexed content and deployed code
+stay in sync.

package/docs/PROVIDERS.md

@@ -0,0 +1,100 @@
+# Embedding Providers
+
+`libsql-search` currently supports three embedding providers:
+
+- `local`
+- `gemini`
+- `openai`
+
+Use the same provider and dimensions for both indexing and querying. A mismatch
+between stored vectors and query vectors will break search quality or fail at
+query time.
+
+## Shared Options
+
+```ts
+interface EmbeddingOptions {
+  provider?: "local" | "gemini" | "openai";
+  apiKey?: string;
+  dimensions?: number;
+  maxLength?: number;
+}
+```
+
+- `provider` defaults to `"local"`
+- `dimensions` defaults to `768`
+- `maxLength` defaults to `8000`
+- `apiKey` is optional in code, but required for hosted providers unless the
+  matching environment variable is available
+
+## Local
+
+Provider value: `local`
+
+The local provider loads `Xenova/all-MiniLM-L6-v2` through
+`@xenova/transformers`.
+
+```ts
+embeddingOptions: {
+  provider: "local",
+  dimensions: 768,
+}
+```
+
+Notes:
+
+- the model emits 384 dimensions and `libsql-search` pads or truncates to your
+  requested size
+- the first run downloads the model and can take longer on a fresh machine
+- no API key is required
+
+## Gemini
+
+Provider value: `gemini`
+
+Gemini uses Google `text-embedding-004`.
+
+```ts
+embeddingOptions: {
+  provider: "gemini",
+  apiKey: process.env.GEMINI_API_KEY,
+}
+```
+
+Behavior:
+
+- if `apiKey` is omitted, the library reads `GEMINI_API_KEY`
+- Gemini returns 768 dimensions natively
+- the current implementation does not expose model selection
+
+## OpenAI
+
+Provider value: `openai`
+
+OpenAI uses `text-embedding-3-small` when `dimensions <= 1536` and
+`text-embedding-3-large` when `dimensions > 1536`.
+
+```ts
+embeddingOptions: {
+  provider: "openai",
+  apiKey: process.env.OPENAI_API_KEY,
+  dimensions: 1536,
+}
+```
+
+Behavior:
+
+- if `apiKey` is omitted, the library reads `OPENAI_API_KEY`
+- the request sends the `dimensions` value to the OpenAI embeddings API
+- use the same dimension count in `createTable()`
+
+## Dimension Guidelines
+
+- `768` is the easiest cross-provider target in the current implementation
+- local embeddings are padded from 384 to your target size
+- Gemini stays at 768
+- OpenAI can be used at 1536 or 3072, or another supported OpenAI dimension
+  value you explicitly set
+
+If you switch provider or dimensions for an existing table, recreate the table
+or rebuild the index into a separate table so stored vectors stay consistent.

package/docs/README.md

@@ -0,0 +1,16 @@
+# Documentation
+
+This directory holds the longer-form reference material for `libsql-search`.
+Start with the page that matches the job you are doing:
+
+- [Provider guide](./PROVIDERS.md): local, Gemini, and OpenAI embedding options,
+  dimensions, and API key behavior
+- [API reference](./API.md): exported functions, option shapes, and result data
+- [Integration examples](./INTEGRATIONS.md): Astro and Next.js server-side usage
+- [Indexing and operations](./INDEXING.md): content layout, rebuild scripts,
+  search quality tips, and indexing gotchas
+- [Troubleshooting](./TROUBLESHOOTING.md): known install/runtime issues
+- [Releasing](./RELEASING.md): maintainer release workflow
+
+For the shortest first-use path, go back to the repository
+[README](../README.md).

package/docs/RELEASING.md

@@ -0,0 +1,91 @@
+# Releasing
+
+Normal releases are automatic. Merge a reviewed PR to `main`; the `Publish
+Package` workflow decides the next stable SemVer version, creates any required
+manifest-only release commit, creates an annotated `vX.Y.Z` tag, publishes npm
+and JSR, then creates GitHub Release notes after both registries succeed.
+
+## Automatic release flow
+
+1. Every push to `main` starts `.github/workflows/publish.yml`.
+2. The workflow ignores self-generated `chore(release): ...` commits.
+3. A serialized release-writer job fetches the current `origin/main` after it
+   has the release slot. If `origin/main` moved beyond the triggering commit,
+   that older run exits and the newer run handles the accumulated changes.
+4. The release planner compares commits since the highest stable `vX.Y.Z` tag
+   merged into the release commit. Higher tags that are not reachable from
+   `main` are ignored so a stray or divergent tag cannot make the default
+   release line jump. Commits whose changed files are confined to `docs/**` or
+   `.github/**` do not count as release-eligible and do not affect the bump. If
+   the newest `main` commit is docs-only but an earlier untagged code or README
+   commit is still pending, that newest run releases the accumulated eligible
+   changes. Breaking changes bump major, `feat:` bumps minor, and all other
+   eligible commits bump patch.
+5. `package.json`, `jsr.json`, and `deno.json` are synchronized to the chosen
+   version. If they already match the chosen version, no release commit is
+   created.
+6. The workflow validates the candidate, creates an annotated tag, and atomically
+   pushes the release commit plus tag.
+7. npm publishes `libsql-search` through trusted publishing OIDC with the GitHub
+   environment named `npm`.
+8. JSR publishes `@logan/libsql-search` through a separate OIDC job.
+9. GitHub Release notes are generated only after both registry jobs complete.
+
+The current bootstrap case is supported: if the latest tag is `v0.1.3` and the
+manifests already say `0.1.4`, the first qualifying `main` release tags and
+publishes `v0.1.4` without creating an empty release commit.
+
+## Manual overrides
+
+Manual version and tag commands are overrides, not the default path. Use them
+only when you intentionally need to publish a specific already-reviewed `main`
+commit.
+
+1. Synchronize `package.json`, `jsr.json`, and `deno.json` to the target
+   `X.Y.Z` version.
+2. Merge that manifest change to `main`.
+3. Create annotated tag `vX.Y.Z` on the merged `main` commit.
+4. Push the tag.
+
+The same `Publish Package` workflow validates manual tags before publishing. It
+checks that the tag target is on `origin/main`, that the tag and manifests agree,
+and that the remote tag still resolves to the checked-out commit immediately
+before each registry publish.
+
+## Recovery dispatch
+
+Use the guarded workflow dispatch path when an existing `vX.Y.Z` tag needs a
+registry or GitHub Release recovery without creating a new version. Example:
+
+```bash
+gh workflow run publish.yml --ref main -f release_tag=v0.1.4
+```
+
+Dispatch validates that `release_tag` is strict `vX.Y.Z`, fetches the remote tag,
+peels it to a commit, checks that commit is on `origin/main`, checks manifests
+against the tag, and then publishes from the tag target SHA. Unlike a tag-push
+run, dispatch does not require the tag target to equal the workflow dispatch
+SHA; this allows a newer `main` workflow fix to recover an older release tag.
+
+npm and GitHub Release recovery steps remain idempotent and skip when the
+version or release already exists. JSR recovery always invokes
+`pnpm dlx jsr@0.14.3 publish`; do not use `deno info jsr:...` as an existence
+gate, because it can resolve successfully even when the public JSR package
+version metadata is not actually published.
+
+## GitHub and registry settings
+
+- npm trusted publishing must point at workflow filename `publish.yml` and use
+  the GitHub Actions environment named `npm`.
+- The workflow does not use `NPM_TOKEN`.
+- JSR publishing uses its own OIDC job and does not depend on the npm
+  environment.
+- The repository settings must allow GitHub Actions to push the generated
+  `chore(release): ...` commit to `main`; otherwise the automatic release job
+  will validate successfully and then fail at the atomic push step.
+- Configure a repository ruleset or tag protection rule for `v*` tags.
+- Restrict the `npm` environment to the `main` deployment branch for automatic
+  releases and `v*` tags for intentional manual overrides.
+
+GitHub Releases are an after-publish record. They are not a prerequisite for npm
+or JSR publishing.

package/docs/TROUBLESHOOTING-SHARP.md

@@ -0,0 +1,64 @@
+# Troubleshooting: Transitive `sharp` Install Errors
+
+`libsql-search` does not directly depend on `sharp`. If you see an install error
+mentioning `sharp`, it is coming from another dependency in your application or
+toolchain.
+
+This page exists because the error can show up in environments that also use
+`libsql-search`, and it is easy to misattribute the failure to this package.
+
+## Typical Error
+
+```text
+Cannot find module '../build/Release/sharp-*.node'
+```
+
+Or:
+
+```text
+Error: Something went wrong installing the "sharp" module
+```
+
+## Why It Happens
+
+With pnpm, native packages may need explicit build-script approval. If the
+relevant install script is blocked, the native binary is never downloaded or
+built.
+
+## What To Do
+
+First inspect which build scripts pnpm blocked:
+
+```bash
+pnpm ignored-builds
+```
+
+Then approve the package that is actually failing and reinstall:
+
+```bash
+pnpm approve-builds
+pnpm install
+```
+
+In the interactive `pnpm approve-builds` prompt, select `sharp` if that is the
+package reporting the native-module failure.
+
+For a committed repository-level fix, you can also allow the package explicitly
+in `pnpm-workspace.yaml` with `onlyBuiltDependencies`.
+
+## Relation To `libsql-search`
+
+- local embeddings use `@xenova/transformers`
+- the first local embedding run may download a model at runtime
+- that runtime model download is separate from a pnpm native-module install
+  failure
+
+## Verification
+
+After reinstalling, rerun the command that originally failed. If your app uses
+`sharp` directly, verify that import in your own project context.
+
+## Additional Resources
+
+- [pnpm approve-builds](https://pnpm.io/10.x/cli/approve-builds)
+- [Sharp installation docs](https://sharp.pixelplumbing.com/install)

package/docs/TROUBLESHOOTING.md

@@ -0,0 +1,12 @@
+# Troubleshooting
+
+Use the page that matches the failure mode:
+
+- [Sharp native module issues](./TROUBLESHOOTING-SHARP.md)
+
+Common operational checks:
+
+- verify you called `createTable()` before indexing or searching
+- verify the table dimension matches the embedding dimension in your code
+- verify the same provider is used for indexing and querying
+- verify hosted providers have `GEMINI_API_KEY` or `OPENAI_API_KEY` available

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",
@@ -15,7 +15,8 @@
   },
   "files": [
     "dist",
-    "README.md"
+    "README.md",
+    "docs"
   ],
   "scripts": {
     "build": "tsc --noEmit && rollup -c && rollup -c rollup.dts.config.js",