libsql-search 0.7.1 → 0.9.0

package/docs/INDEXING.md

@@ -15,7 +15,7 @@ The slug is derived from the file path relative to `contentPath`.
 
 ## Rebuild Behavior
 
-`indexContent()` clears the target table before rebuilding:
+`indexContent()` replaces the whole target table:
 
 ```ts
 await indexContent({
@@ -28,14 +28,123 @@ await indexContent({
 });
 ```
 
-That keeps the implementation simple, but it also means:
+The rebuild runs in two phases:
 
-- failed rebuilds can leave the table partially repopulated
-- provider or dimension changes should use a parallel table migration
+1. build: every file is read, parsed, and embedded in memory, touching no database state
+2. replace: the delete and all inserts run in a single write transaction
+
+That means:
+
+- a failed rebuild leaves the previously indexed rows exactly as they were
+- provider or dimension changes should still use a parallel table migration
 - `createTable()` does not resize an existing vector column
 
+Files are discovered and indexed in sorted path order, so a rebuild is deterministic.
+
 If provider, dimensions, model, endpoint, or embedding-space assumptions change, fully reindex into a new table. See the canonical [Migration and reindexing guide](./MIGRATIONS.md).
 
+### Costs Of The Two-Phase Rebuild
+
+Atomicity is not free, and both costs scale with corpus size:
+
+- **Peak memory holds the whole corpus.** The build phase keeps every document in memory: content, frontmatter, and one embedding array per document. The replace phase then builds insert statements including a JSON copy of each embedding, roughly 5-8 KB per document at 384 dimensions and considerably more at 3072. Documents are released as their statements are built, but peak usage is still proportional to the entire corpus rather than to one file.
+- **Remote clients send one request.** Against Turso or any remote client, the delete and every insert travel as a single batch. There is no chunking fallback, because splitting the batch would give up the atomicity this design exists to provide. A corpus large enough to exceed a remote request-size limit fails as an opaque `phase: "replace"` error.
+
+For very large corpora, index into a parallel table and switch reads over once it validates, rather than rebuilding a live table in place. See the [Migration and reindexing guide](./MIGRATIONS.md).
+
+## Content Requirements
+
+Two authoring mistakes fail a file at the `parse` stage rather than corrupting the rebuild:
+
+- **Frontmatter `title` must be a scalar.** Strings, numbers, booleans, and dates are accepted; dates are stored as ISO strings. A structured title such as a YAML list fails the file. A missing or empty title still falls back to the filename.
+- **Slugs must be unique.** The slug comes from the path with the extension removed, so `foo.md` and `foo.markdown` collide. Files are processed in sorted path order and the first file to claim a slug keeps it, so `foo.markdown` wins and `foo.md` is reported as the failure.
+
+Both are governed by `failurePolicy` like any other build failure, so they abort by default and are skippable.
+
+## Failure Handling
+
+`indexContent()` throws `IndexingError` instead of reporting a partially applied rebuild. The error carries `phase` (`"build"` or `"replace"`), a `failures` array, and the underlying error as `cause`.
+
+```ts
+import { indexContent, IndexingError } from "libsql-search";
+
+try {
+  await indexContent({ client, contentPath: "./content" });
+} catch (error) {
+  if (error instanceof IndexingError) {
+    for (const failure of error.failures) {
+      console.error(`${failure.file} failed during ${failure.stage}`);
+    }
+  }
+
+  throw error;
+}
+```
+
+By default one bad file aborts the whole rebuild. To index everything that can be indexed, opt into `failurePolicy: "skip"`:
+
+```ts
+const result = await indexContent({
+  client,
+  contentPath: "./content",
+  failurePolicy: "skip",
+});
+
+if (result.partial) {
+  console.warn(`Indexed ${result.success} of ${result.total} files`);
+}
+```
+
+Skipped rebuilds still replace the table, so treat `partial: true` as a build warning rather than a clean rebuild. If every discovered file fails, the rebuild throws rather than trading a valid index for an empty one.
+
+## Empty Source Directories
+
+An empty source directory throws by default, because silently leaving stale rows in place serves search traffic from content that no longer exists. Emptying an index has to be intentional:
+
+```ts
+await indexContent({
+  client,
+  contentPath: "./content",
+  allowEmptyIndex: true,
+});
+```
+
+Both behaviors changed in a breaking way: partial failures used to be counted and reported, and an empty directory used to return zeros without clearing the table.
+
+## The Embedding Vector Index
+
+`createTable()` creates `<tableName>_embedding_idx` alongside the table:
+
+```sql
+CREATE INDEX IF NOT EXISTS "<tableName>_embedding_idx"
+ON "<tableName>"(libsql_vector_idx(embedding))
+```
+
+`search()` requires that index by default. It queries the index through `vector_top_k()` instead of scoring every row, so query cost no longer grows linearly with the size of the index.
+
+That index is approximate. `search()` compensates by over-fetching candidates and re-ranking them exactly; see [`search(options)`](./API.md#searchoptions) for the recall and ordering semantics and for the `candidates` and `exact` options.
+
+### Tables Built Before The Index Existed
+
+A table created by hand, or by a version of this package that predated the embedding index, has no `<tableName>_embedding_idx`. The default search path fails on such a table with an error naming the missing index — libSQL's own message for the case ("failed to parse vector index parameters") does not mention it.
+
+`indexContent()` does not create the index; it only replaces rows. Two ways forward:
+
+```ts
+// Preferred: createTable() is idempotent and adds only what is missing
+await createTable(client, "articles", 384);
+```
+
+```sql
+-- Or create the index directly against the existing table
+CREATE INDEX IF NOT EXISTS "articles_embedding_idx"
+ON "articles"(libsql_vector_idx(embedding));
+```
+
+`createTable()` uses `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`, so calling it against an existing table adds the missing index without touching rows. It still does not resize an existing vector column.
+
+Until the index exists, pass `exact: true` to `search()` to keep queries working on the full-scan path.
+
 ## Quality Guidelines
 
 - include descriptive frontmatter titles
@@ -43,6 +152,7 @@ If provider, dimensions, model, endpoint, or embedding-space assumptions change,
 - use the same provider and dimensions at index and query time
 - keep `maxLength` intentional if your content is very large
 - start with a smaller search `limit` and tune from real query behavior
+- raise `candidates` if the approximate index path misses results the exact path finds; compare the two with `exact: true` on a fixed set of queries
 
 ## Build Integration
 

package/docs/MIGRATIONS.md

@@ -22,7 +22,7 @@ References:
 - if dimensions change, create a new table or recreate the old table, then fully re-embed
 - if dimensions stay the same but provider, model, endpoint, model revision, pooling, normalization, or input formatting changes, fully reindex anyway
 - never mix two embedding spaces in one table
-- prefer a parallel table migration because `indexContent()` clears rows first and is not transactional
+- prefer a parallel table migration because `indexContent()` replaces the whole target table, so an in-place rebuild leaves no way back to the old vectors
 
 In practice, this means:
 
@@ -39,6 +39,8 @@ In practice, this means:
 5. Switch application reads and writes to the new table.
 6. Retire the old table in a separate cleanup step.
 
+Step 3 is all or nothing. `indexContent()` throws `IndexingError` and leaves the target table untouched when a file or the replacement transaction fails, so a failed migration step can be retried without cleanup. See [Indexing and operational behavior](./INDEXING.md) for `failurePolicy` and `allowEmptyIndex`.
+
 Example:
 
 ```ts
@@ -73,6 +75,26 @@ const results = await search({
 });
 ```
 
+## Tables Without The Embedding Vector Index
+
+`search()` queries the `<tableName>_embedding_idx` vector index by default rather than scanning the whole table. A table created by hand, or by a version of this package that predated that index, does not have it, and the default search path fails against such a table.
+
+Re-running `createTable()` with the table's existing width is the fix. It is idempotent — `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS` — so it adds the missing index without touching rows and without resizing the vector column:
+
+```ts
+// Same name and same width as the existing table
+await createTable(client, "articles_local_384", 384);
+```
+
+Equivalently, in SQL:
+
+```sql
+CREATE INDEX IF NOT EXISTS "articles_local_384_embedding_idx"
+ON "articles_local_384"(libsql_vector_idx(embedding));
+```
+
+Reindexing does not create the index; `indexContent()` only replaces rows. Any new table created by `createTable()` as part of a migration already has it, so this applies only to pre-existing tables you are carrying forward. Until the index exists, `search({ ..., exact: true })` keeps queries working on the exact full-scan path.
+
 ## Common Migration Paths
 
 | From | To | Why a rebuild is required | Recommended table move |

package/docs/RELEASING.md

@@ -19,11 +19,19 @@ and JSR, then creates GitHub Release notes after both registries succeed.
    `.github/**` do not count as release-eligible and do not affect the bump. If
    the newest `main` commit is docs-only but an earlier untagged code or README
    commit is still pending, that newest run releases the accumulated eligible
-   changes. Breaking changes bump major, `feat:` bumps minor, and all other
-   eligible commits bump patch.
+   changes. `feat:` bumps minor and all other eligible commits bump patch.
+   Breaking changes (a `subject!:` prefix or a `BREAKING CHANGE:` footer) bump
+   major only once the package is `1.0.0` or higher. While the package is still
+   on the `0.x` line a breaking change bumps the minor instead, so an unattended
+   `fix!:` cannot auto-promote the package to `1.0.0`. Promoting off `0.x` is
+   deliberate and manual: see step 5.
 5. `package.json`, `jsr.json`, and `deno.json` are synchronized to the chosen
    version. If they already match the chosen version, no release commit is
-   created.
+   created. A manifest version that is already ahead of the latest tag wins over
+   the computed bump; this is the supported escape hatch for a deliberate
+   version jump, including promoting off the `0.x` line. Set all three manifests
+   to `1.0.0` on a reviewed PR and the next qualifying release publishes
+   `v1.0.0`.
 6. The workflow validates the candidate, creates an annotated tag, and atomically
    pushes the release commit plus tag.
 7. npm publishes `libsql-search` through trusted publishing OIDC with the GitHub

package/docs/TROUBLESHOOTING.md

@@ -17,3 +17,24 @@ Common operational checks:
   or use a new table name before rebuilding
 - after upgrading an existing local 768-dimensional padded index, create or
   recreate a 384-dimensional table and fully re-index before querying it
+- if `search()` reports that the `<tableName>_embedding_idx` vector index could
+  not be used, the table has no embedding vector index: re-run `createTable()`
+  with the table's existing name and width to add it without touching rows, or
+  pass `exact: true` to search on the full-scan path meanwhile. See
+  [Tables without the embedding vector index](./MIGRATIONS.md#tables-without-the-embedding-vector-index).
+  The underlying libSQL message, preserved as the error's `cause`, reads
+  "failed to parse vector index parameters" and does not mention the index
+- if `search()` reports "no vector index support: vector_top_k() is
+  unavailable", the libSQL build or server itself has no vector support, so
+  creating an index will not help. Pass `exact: true` to search on the
+  full-scan path, or move to a deployment with vector support
+- a libSQL error reading "dimensions are different: 384 != 4" is a different
+  problem and is passed through unchanged: the index exists, but the query
+  embedding's width does not match the `embedding` column's width. Creating an
+  index will not help and neither will `exact: true` — the exact path reports
+  the same root cause in different words, as
+  "vector_distance: vectors must have the same length: 4 != 384", with the
+  operands in the opposite order. Align the widths across
+  `createTable()`, `indexContent()`, and `search()`, and re-index if the stored
+  vectors are in the wrong space; see the
+  [Migration and reindexing guide](./MIGRATIONS.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "libsql-search",
-  "version": "0.7.1",
+  "version": "0.9.0",
   "description": "Semantic search for static sites using libSQL/Turso with multi-provider embeddings",
   "type": "module",
   "packageManager": "pnpm@10.34.5",