@delali/narsil 0.1.8 → 0.1.10

package/README.md

@@ -1,3 +1,5 @@
+![Narsil, a distributed search engine](https://raw.githubusercontent.com/assetcorp/narsil/main/assets/banner.png)
+
 # Narsil
 
 [![CI](https://github.com/assetcorp/narsil/actions/workflows/ci.yml/badge.svg)](https://github.com/assetcorp/narsil/actions/workflows/ci.yml)
@@ -8,17 +10,48 @@
 
 Distributed search, reforged.
 
-Narsil is a distributed search engine with full-text, vector, hybrid, and geosearch. It auto-partitions large indexes across workers, serializes them into a cross-language binary format (.nrsl), and merges results back into a single ranked answer. The TypeScript package is the first implementation.
+Narsil is a distributed search engine with full-text, vector, hybrid, and geosearch. One codebase runs in two contexts: embedded in your application process, where queries answer without a network hop, and as a standalone search server with a REST API, a write-ahead log, and bulk NDJSON ingest. Both contexts run the same engine and store indexes in the same cross-language binary format (.nrsl), so an index built in one loads in the other.
+
+The engine partitions large indexes across workers and merges partition results into a single ranked answer. Its BM25 ranking matches the Anserini reference within 0.005 nDCG@10 on the BEIR datasets, and one node answers 1,020 keyword queries per second on BEIR SciFact ([benchmarks](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md)). The TypeScript package is the first implementation.
 
 > *narsil* is the sword of Elendil in Tolkien's Lord of the Rings, shattered into shards and later reforged. The name maps to the architecture: data shatters into partitions, each shard is independently persisted, and every query reforges them into a unified result.
 
+## Contents
+
+- [Install](#install)
+- [Quick start](#quick-start)
+- [Configuration](#configuration)
+- [Indexes](#indexes)
+- [Documents](#documents)
+- [Batch operations](#batch-operations)
+- [Search](#search)
+- [Vector search](#vector-search)
+- [Hybrid search](#hybrid-search)
+- [Geosearch](#geosearch)
+- [Embedding adapters](#embedding-adapters)
+- [Persistence](#persistence)
+- [Durability](#durability)
+- [Snapshots and restore](#snapshots-and-restore)
+- [Partitions and rebalancing](#partitions-and-rebalancing)
+- [Workers](#workers)
+- [Multi-instance invalidation](#multi-instance-invalidation)
+- [Plugins](#plugins)
+- [Events](#events)
+- [Errors](#errors)
+- [Memory reporting](#memory-reporting)
+- [Language support](#language-support)
+- [HTTP server](#http-server)
+- [Distribution](#distribution)
+- [Search quality](#search-quality)
+- [Runtime support](#runtime-support)
+
 ## Install
 
 ```bash
 pnpm add @delali/narsil
 ```
 
-Requires Node.js >= 22.
+Narsil requires Node.js 22 or newer. It also runs in Bun, Deno, and browsers; see [Runtime support](#runtime-support).
 
 ## Quick start
 
@@ -51,23 +84,28 @@ await narsil.insert('products', {
 const results = await narsil.query('products', {
   term: 'mechanical keyboard',
   filters: {
-    inStock: { eq: true },
-    price: { lte: 200 },
+    fields: {
+      inStock: { eq: true },
+      price: { lte: 200 },
+    },
   },
   boost: { title: 2.0 },
   limit: 10,
 })
 ```
 
+Every hit carries the document, its id, and its BM25 score. `results.count` reports how many documents matched in total, and `results.elapsed` reports the query time in milliseconds.
+
 ## Configuration
 
 `createNarsil` accepts an optional `NarsilConfig` object. All fields are optional.
 
 ```ts
 import { createNarsil } from '@delali/narsil'
+import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
 
 const narsil = await createNarsil({
-  persistence: myAdapter,
+  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
   workers: { enabled: true, count: 4 },
   flush: { interval: 5000, mutationThreshold: 100 },
 })
@@ -77,36 +115,37 @@ const narsil = await createNarsil({
 
 | Field | Type | Description |
 | --- | --- | --- |
-| `persistence` | `PersistenceAdapter` | Storage backend for durable indexes (filesystem, IndexedDB, or custom) |
-| `invalidation` | `InvalidationAdapter` | Cross-instance cache coordination adapter for multi-process or multi-tab setups |
-| `plugins` | `NarsilPlugin[]` | Lifecycle hooks for insert, remove, update, search, and index events |
-| `idGenerator` | `() => string` | Custom function for generating document IDs (defaults to UUID v7) |
-| `workers` | `WorkerConfig` | Worker thread configuration for parallel search |
-| `flush` | `FlushConfig` | Controls when dirty partitions persist to storage |
-| `eagerLoad` | `boolean` | When `true`, loads all persisted data into memory at creation time |
-| `embedding` | `EmbeddingAdapter` | Default adapter for auto-embedding text fields into vectors (see [Embedding adapters](#embedding-adapters)) |
+| `persistence` | `PersistenceAdapter` | Sets the storage backend for persisted partitions. See [Persistence](#persistence). |
+| `invalidation` | `InvalidationAdapter` | Coordinates cache eviction across processes or tabs. See [Multi-instance invalidation](#multi-instance-invalidation). |
+| `plugins` | `NarsilPlugin[]` | Registers lifecycle hooks for document and search operations. See [Plugins](#plugins). |
+| `idGenerator` | `() => string` | Replaces the default UUID v7 generator for document ids. |
+| `workers` | `WorkerConfig` | Controls the worker thread pool for parallel search. See [Workers](#workers). |
+| `flush` | `FlushConfig` | Controls when dirty partitions persist to storage. |
+| `embedding` | `EmbeddingAdapter` | Sets the default adapter for auto-embedding text into vectors. See [Embedding adapters](#embedding-adapters). |
+| `embeddingAdapters` | `Record<string, EmbeddingAdapter>` | Registers named adapters that index configs reference by name. Names persist in index metadata, so durability recovery can rebind them. |
+| `durability` | `DurabilityConfig` | Enables write-ahead logging and snapshots. See [Durability](#durability). |
 
 ### WorkerConfig
 
 | Field | Type | Default | Description |
 | --- | --- | --- | --- |
-| `enabled` | `boolean` | `false` | Enables worker thread pool for search |
-| `count` | `number` | CPU count | Number of worker threads to spawn |
-| `promotionThreshold` | `number` | - | Document count per index that triggers auto-promotion to workers |
-| `totalPromotionThreshold` | `number` | - | Total document count across all indexes that triggers auto-promotion |
+| `enabled` | `boolean` | `false` | Turns on the worker thread pool for search. |
+| `count` | `number` | CPU cores minus one, clamped between 2 and 8 | Sets the number of worker threads to spawn. |
+| `promotionThreshold` | `number` | `10000` | Sets the per-index document count that triggers promotion to workers. |
+| `totalPromotionThreshold` | `number` | `50000` | Sets the document count across all indexes that triggers promotion. |
 
 ### FlushConfig
 
-| Field | Type | Default | Description |
-| --- | --- | --- | --- |
-| `interval` | `number` | - | Milliseconds between persistence flushes |
-| `mutationThreshold` | `number` | - | Number of mutations that triggers a flush |
+| Field | Type | Description |
+| --- | --- | --- |
+| `interval` | `number` | Sets the milliseconds between persistence flushes. |
+| `mutationThreshold` | `number` | Sets the mutation count that triggers a flush. |
 
 ### Tokenizer cache
 
-The stemmer normalization cache auto-sizes based on the runtime environment. On Node.js it reads container memory limits (via `process.constrainedMemory()`), in browsers it checks `navigator.deviceMemory`, and falls back to a sensible default elsewhere.
+The stemmer normalization cache auto-sizes based on the runtime environment. On Node.js it reads container memory limits through `process.constrainedMemory()`, in browsers it checks `navigator.deviceMemory`, and it falls back to a fixed default elsewhere.
 
-This cache is process-global, shared across all Narsil instances in the same process. You can override the size by calling `configureNormalizationCache` once at startup, before creating any instances:
+This cache is process-global and shared across all Narsil instances in the same process. Override the size by calling `configureNormalizationCache` once at startup, before creating any instances:
 
 ```ts
 import { configureNormalizationCache, createNarsil } from '@delali/narsil'
@@ -116,110 +155,202 @@ configureNormalizationCache(500_000)
 const narsil = await createNarsil()
 ```
 
-The value clamps to a floor of 50,000 and a ceiling of 2,000,000 entries. Invalid values (NaN, Infinity, negative numbers, zero) throw a `NarsilError` with code `CONFIG_INVALID`.
+The value clamps to a floor of 50,000 and a ceiling of 2,000,000 entries. Invalid values (NaN, Infinity, negative numbers, and zero) throw a `NarsilError` with code `CONFIG_INVALID`.
 
-Four utility functions are available for cache management:
+Four functions manage the cache:
 
-- `configureNormalizationCache(maxSize)` - set the maximum cache size
-- `clearNormalizationCache()` - drop all cached entries (reclaim memory after one-off indexing)
-- `resetNormalizationCache()` - clear the cache and reset the size to the auto-detected default
-- `getNormalizationCacheSize()` - return the current number of cached entries (useful for monitoring)
+- `configureNormalizationCache(maxSize)` sets the maximum cache size.
+- `clearNormalizationCache()` drops all cached entries, which reclaims memory after one-off indexing.
+- `resetNormalizationCache()` clears the cache and resets the size to the auto-detected default.
+- `getNormalizationCacheSize()` returns the current number of cached entries for monitoring.
 
-## Features
+## Indexes
 
-### Full-text search
+`createIndex(name, config)` creates an index from a schema. The schema supports `string`, `number`, `boolean`, `enum`, `geopoint`, `vector[N]`, and the array variants `string[]`, `number[]`, `boolean[]`, and `enum[]`. Objects nest up to 4 levels deep. Narsil validates every document against the schema at insertion time.
 
-BM25 scoring with field boosting, fuzzy matching via bounded Levenshtein distance, exact phrase search, and configurable term matching policies. Specify how many query terms a document must match, set minimum score thresholds, and sort by any field.
+```ts
+await narsil.createIndex('articles', {
+  schema: {
+    title: 'string',
+    body: 'string',
+    author: {
+      name: 'string',
+      verified: 'boolean',
+    },
+    publishedYear: 'number',
+  },
+  language: 'english',
+  required: ['title'],
+})
+```
 
-### Auto-partitioning
+### IndexConfig
 
-Indexes split into shards when they grow beyond a configurable threshold. Partition assignment uses FNV-1a hashing for deterministic routing. Background rebalancing redistributes documents across new partitions without blocking queries, using a write-ahead queue for zero-downtime resharding.
+| Field | Type | Description |
+| --- | --- | --- |
+| `schema` | `SchemaDefinition` | Declares the fields and their types. This field is required. |
+| `language` | `string` | Selects the language module for tokenization and stemming. The default is `english`. |
+| `partitions` | `PartitionConfig` | Sets `maxDocsPerPartition` and `maxPartitions`. See [Partitions and rebalancing](#partitions-and-rebalancing). |
+| `defaultScoring` | `'local' \| 'dfs' \| 'broadcast'` | Sets the scoring mode used when a query does not pass one. See [Scoring modes](#scoring-modes). |
+| `bm25` | `BM25Params` | Overrides the BM25 `k1` and `b` parameters. |
+| `stopWords` | `Set<string> \| (defaults: Set<string>) => Set<string>` | Replaces or transforms the language module's stop word set. |
+| `tokenizer` | `CustomTokenizer` | Replaces the built-in tokenizer with your own `tokenize(text)` implementation. |
+| `trackPositions` | `boolean` | Stores token positions for highlighting. The default is `true`. |
+| `vectorPromotion` | `VectorIndexConfig` | Tunes the HNSW promotion threshold, graph parameters, and quantization. See [Vector search](#vector-search). |
+| `strict` | `boolean` | Rejects documents that carry fields missing from the schema. |
+| `embedding` | `EmbeddingFieldConfig` | Maps text fields to vector fields for auto-embedding. See [Embedding adapters](#embedding-adapters). |
+| `required` | `string[]` | Lists fields a document must carry; inserts without them fail with `DOC_MISSING_REQUIRED_FIELD`. |
+
+### Index management
+
+```ts
+const indexes = narsil.listIndexes()
+// [{ name: 'articles', documentCount: 1204, partitionCount: 1, language: 'english' }]
 
-**Single-process overhead.** Partitioning pays off when shards live on separate processes or hosts. Inside one Node.js process, going from 1 to 20 partitions costs about 14% of insert throughput, 28% of median search latency, and 27% at p95, with no scaling upside. Keep `partitions.maxPartitions` low (or unset) for single-process deployments and raise it once you fan partitions out across workers or distributed nodes.
+const stats = narsil.getStats('articles')
+// { documentCount, partitionCount, estimatedMemoryBytes, language, schema }
 
-**Rebalance latency spikes.** While a partition reshape is running, worst-tick p95 latency can climb to about 25ms compared with around 11ms in steady state. Schedule large bulk inserts that trigger rebalancing during low-traffic windows, or pre-size the index with `partitions.maxDocsPerPartition` to avoid mid-load reshapes.
+await narsil.clear('articles')
 
-### Worker isolation
+await narsil.dropIndex('articles')
+```
 
-Search operations move off the main thread through worker threads (Node.js, Bun) or Web Workers (browser, Deno). The engine starts in direct mode for small indexes and auto-promotes to workers once document counts cross a threshold. You keep using the same API throughout.
+`clear` removes every document but keeps the index and its schema. `dropIndex` removes the index entirely, including its persisted data. Call `shutdown()` when the process is done with the engine; it stops workers, flushes durability state, and rejects further calls.
 
-### Memory reporting
+## Documents
 
-Narsil reports memory at three levels: per index, per partition, and per runtime. `getStats(indexName)` returns `estimatedMemoryBytes`, a formula-based estimate for the index's main-thread partitions and vector structures.
+### Insert
+
+`insert(indexName, document, docId?, options?)` resolves the document id in this order: the explicit `docId` argument wins, then a string `id` field on the document itself, and otherwise Narsil generates a UUID v7. The method returns the resolved id.
 
 ```ts
-const indexStats = narsil.getStats('products')
+const generatedId = await narsil.insert('products', { title: 'Trackball Mouse' })
 
-console.log(indexStats.estimatedMemoryBytes)
+const explicitId = await narsil.insert('products', { title: 'Split Keyboard' }, 'kb-042')
+
+await narsil.insert('products', { id: 'kb-043', title: 'Tenkeyless Keyboard' })
 ```
 
-The estimate comes from document counts, posting lists, field indexes, and vector indexes. It excludes V8 object headers, allocator overhead, and other host runtime costs, so use it for comparing indexes inside one process.
+Inserting an id that already exists fails with `DOC_ALREADY_EXISTS`, and `update` fails with `DOC_NOT_FOUND` when the id is missing, so an upsert checks `has()` first and picks the right call. The HTTP server's PUT endpoint packages that check as one request.
 
-`getPartitionStats(indexName)` returns the same estimate for each partition. The partition values sum to `getStats(indexName).estimatedMemoryBytes`.
+### Read
 
 ```ts
-const partitions = narsil.getPartitionStats('products')
+const doc = await narsil.get('products', 'kb-042')
+// the document, or undefined when the id is unknown
 
-for (const partition of partitions) {
-  console.log(partition.partitionId, partition.estimatedMemoryBytes)
-}
+const docs = await narsil.getMultiple('products', ['kb-042', 'kb-043'])
+// Map<string, AnyDocument> holding only the ids that exist
+
+const exists = await narsil.has('products', 'kb-042')
+
+const count = await narsil.countDocuments('products')
 ```
 
-`getMemoryStats()` returns a runtime snapshot and worker reports. It is async because workers report their heap usage through the worker message channel.
+### Update and remove
+
+`update` replaces the whole document under an id. Internally it removes the old document and inserts the new one, with a fast path when the change touches nothing the index depends on.
 
 ```ts
-const memory = await narsil.getMemoryStats()
+await narsil.update('products', 'kb-042', { title: 'Split Ergonomic Keyboard' })
 
-console.log(memory.estimatedIndexBytes)
-console.log(memory.process?.heapUsed)
-console.log(memory.workers)
+await narsil.remove('products', 'kb-042')
 ```
 
-`memory.estimatedIndexBytes` sums `estimatedMemoryBytes` across every index in this Narsil instance. `memory.process` comes from `process.memoryUsage()` when the runtime exposes it. It measures the whole host process, so two Narsil instances in one Node.js process report the same process numbers. Browser runtimes return `null` for `memory.process`.
+Both methods throw `DOC_NOT_FOUND` for an unknown id.
+
+## Batch operations
+
+`insertBatch`, `updateBatch`, and `removeBatch` process many documents in one call and return partial results. One bad document never aborts the batch: every failure is collected with its id and error, and every success is applied.
+
+```ts
+const result = await narsil.insertBatch('products', [
+  { id: 'p1', title: 'USB-C Hub', price: 49 },
+  { id: 'p2', title: 'Laptop Stand', price: 89 },
+  { id: 'p3', title: 'Broken Doc', price: 'not-a-number' },
+])
+
+// result.succeeded => ['p1', 'p2']
+// result.failed => [{ docId: 'p3', error: NarsilError(DOC_VALIDATION_FAILED) }]
+
+await narsil.updateBatch('products', [
+  { docId: 'p1', document: { title: 'USB-C Hub, 8 ports', price: 59 } },
+])
+
+await narsil.removeBatch('products', ['p1', 'p2'])
+```
+
+Batch inserts resolve ids from each document's `id` field and generate UUID v7 ids for the rest. Large batches process in chunks and yield the event loop between chunks, so searches stay responsive during a bulk load.
+
+## Search
 
-`memory.workers` lists `heapUsed`, `heapTotal`, and `external` for each active worker. The array is empty before worker promotion.
+`query(indexName, params)` runs every search. The `mode` parameter selects `'fulltext'` (the default), `'vector'`, or `'hybrid'`.
 
-### Distributed BM25 scoring
+### Full-text search
 
-Three scoring modes handle the partition-skew problem:
+Full-text search scores with BM25. `fields` restricts the search to specific fields, and `boost` multiplies per-field scores.
 
-- **Local**: each partition scores with its own statistics (fastest, default)
-- **DFS (Distributed Frequency Statistics)**: a two-phase query collects global term statistics first, then scores with unified IDF values
-- **Broadcast**: instances share statistics through the invalidation adapter for pre-computed global scoring
+```ts
+const results = await narsil.query('products', {
+  term: 'wireless keyboard',
+  fields: ['title', 'description'],
+  boost: { title: 2.0 },
+  limit: 10,
+  offset: 0,
+})
+```
 
-### Vector search
+`limit` defaults to 10. Each hit has the shape `{ id, score, document, highlights?, scoreComponents? }`. Pass `includeScoreComponents: true` to receive per-term frequencies, field lengths, and IDF values for debugging a ranking.
 
-Store and query high-dimensional embeddings with cosine similarity, dot product, or Euclidean distance. Small vector sets use an exact brute-force scan. Once a field crosses 1,024 vectors, the engine builds an HNSW graph and switches the field to approximate search. The threshold is configurable per index via `vectorPromotion.threshold`.
+### Fuzzy matching
 
-### Hybrid search
+`tolerance` sets the maximum Levenshtein edit distance between a query term and an indexed term. It defaults to 0, which requires exact matches. `prefixLength` limits fuzzy candidates to terms sharing that many leading characters with the query term; it defaults to 2, and raising it makes fuzzy lookups faster and stricter. `exact: true` turns fuzzy expansion off for the whole query.
 
-Combine full-text BM25 scores with vector similarity scores using weighted alpha blending. A single query searches both text and embeddings, normalizes both score sets to [0,1], and returns a merged ranking.
+```ts
+const results = await narsil.query('products', {
+  term: 'keybaord',
+  tolerance: 2,
+  prefixLength: 3,
+})
+```
 
-### Geosearch
+### Score and coverage thresholds
 
-Index latitude/longitude points and query by radius (Haversine or Vincenty distance) or polygon containment (ray casting). High-precision mode switches to Vincenty's iterative formula for long-distance accuracy.
+`minScore` drops hits scoring below a floor. `termMatch` sets how many query terms a document must match: `'any'` (the default) accepts one term, `'all'` requires every term, and a number requires at least that many terms.
+
+```ts
+const results = await narsil.query('products', {
+  term: 'mechanical gaming keyboard',
+  termMatch: 2,
+  minScore: 1.5,
+})
+```
 
 ### Filters
 
-Filter on any indexed field using comparison operators (`eq`, `ne`, `gt`, `lt`, `gte`, `lte`, `between`), string operators (`in`, `nin`, `startsWith`, `endsWith`), array operators (`containsAll`, `matchesAny`, `size`), and presence checks (`exists`, `isEmpty`). Combine filters with explicit `and`, `or`, and `not` boolean combinators.
+Filter on any indexed field with comparison operators (`eq`, `ne`, `gt`, `lt`, `gte`, `lte`, `between`), string operators (`in`, `nin`, `startsWith`, `endsWith`), array operators (`containsAll`, `matchesAny`, `size`), and presence checks (`exists`, `notExists`, `isEmpty`, `isNotEmpty`). Combine filter expressions with `and`, `or`, and `not`.
 
 ```ts
 const results = await narsil.query('products', {
   term: 'wireless',
   filters: {
     or: [
-      { category: { eq: 'electronics' } },
-      { category: { eq: 'accessories' } },
+      { fields: { category: { eq: 'electronics' } } },
+      { fields: { category: { eq: 'accessories' } } },
     ],
-    price: { between: [10, 100] },
-    tags: { containsAll: ['bluetooth'] },
+    fields: {
+      price: { between: [10, 100] },
+      tags: { containsAll: ['bluetooth'] },
+    },
   },
 })
 ```
 
-### Faceted search
+Field conditions live under `fields`, and the `and`, `or`, and `not` combinators nest whole filter expressions, so any boolean shape is expressible. Filters narrow the candidates a search scores: a full-text query needs a `term` to produce hits, and in vector and hybrid modes the filters restrict which documents the vector search considers.
+
+### Facets
 
-Get aggregate value counts alongside search results for building filter UIs.
+Facets return value counts alongside the hits for building filter UIs. String and enum facets take a `limit` and a `sort` direction, and numeric facets take explicit `ranges`.
 
 ```ts
 const results = await narsil.query('products', {
@@ -229,16 +360,51 @@ const results = await narsil.query('products', {
     price: { ranges: [{ from: 0, to: 500 }, { from: 500, to: 1000 }, { from: 1000, to: 5000 }] },
   },
 })
-// results.facets.category.values => { electronics: 42, computers: 28, ... }
+
+// results.facets => { category: { values: { electronics: 42, computers: 28 }, count: 70 }, ... }
+```
+
+### Sort
+
+`sort` orders hits by field values instead of score. Multiple entries apply in order, so the second field breaks ties in the first.
+
+```ts
+const results = await narsil.query('products', {
+  term: 'keyboard',
+  sort: { price: 'asc', title: 'asc' },
+})
 ```
 
 ### Grouping
 
-Group results by field values with optional custom reducers.
+`group` collapses hits that share field values. `maxPerGroup` caps how many hits each group keeps, and an optional reducer folds every grouped document into an accumulated value.
+
+```ts
+const results = await narsil.query('products', {
+  term: 'keyboard',
+  group: {
+    fields: ['category'],
+    maxPerGroup: 3,
+  },
+})
+
+// results.groups => [{ values: { category: 'electronics' }, hits: [...] }, ...]
+
+const withTotals = await narsil.query('products', {
+  term: 'keyboard',
+  group: {
+    fields: ['category'],
+    reduce: {
+      initialValue: () => 0,
+      reducer: (total, doc) => (total as number) + ((doc.price as number) ?? 0),
+    },
+  },
+})
+```
 
-### Match highlighting
+### Highlighting
 
-Get snippets with customizable pre/post tags marking where query terms appear in the original text.
+`highlight` returns snippets with tags marking where query terms appear. Highlighting needs `trackPositions` left at its default of `true`.
 
 ```ts
 const results = await narsil.query('products', {
@@ -247,14 +413,208 @@ const results = await narsil.query('products', {
     fields: ['title', 'description'],
     preTag: '<mark>',
     postTag: '</mark>',
+    maxSnippetLength: 160,
+  },
+})
+
+// hit.highlights?.title.snippet => '<mark>Mechanical</mark> Keyboard'
+```
+
+### Pagination
+
+Shallow pagination uses `limit` and `offset`. Deep pagination uses `searchAfter` cursors, which track a position per partition and keep latency flat at any depth. Every page's result carries a `cursor` string; pass it back as `searchAfter` to fetch the next page.
+
+```ts
+const firstPage = await narsil.query('products', { term: 'keyboard', limit: 20 })
+
+if (firstPage.cursor) {
+  const secondPage = await narsil.query('products', {
+    term: 'keyboard',
+    limit: 20,
+    searchAfter: firstPage.cursor,
+  })
+}
+```
+
+A cursor is only valid for the same query parameters it came from. A malformed cursor fails with `SEARCH_INVALID_CURSOR`.
+
+### Pinning
+
+`pinned` places specific documents at fixed positions in the ranked results, which serves sponsored or editorial placements. Positions are zero-based.
+
+```ts
+const results = await narsil.query('products', {
+  term: 'keyboard',
+  pinned: [{ docId: 'kb-editorial-pick', position: 0 }],
+})
+```
+
+### Scoring modes
+
+Three scoring modes handle the statistics-skew problem that appears when an index spans partitions or instances:
+
+- `'local'` scores each partition with its own statistics. It is the fastest mode and the default.
+- `'dfs'` runs a two-phase query that first collects global term statistics, then scores with unified IDF values.
+- `'broadcast'` has instances share statistics through the invalidation adapter, so scoring uses pre-computed global values.
+
+Set a per-index default with `defaultScoring` in the index config, or set `scoring` per query:
+
+```ts
+const results = await narsil.query('products', {
+  term: 'keyboard',
+  scoring: 'dfs',
+})
+```
+
+### Preflight
+
+`preflight(indexName, params)` returns the match count for a query without materializing, ranking, or paginating hits. Use it to size a result set before running an expensive query.
+
+```ts
+const { count, elapsed } = await narsil.preflight('products', { term: 'keyboard' })
+```
+
+### Suggestions
+
+`suggest(indexName, params)` returns autocomplete candidates from the index's term dictionary. It tokenizes the input, takes the last word as the prefix, and ranks matching terms by document frequency.
+
+```ts
+const suggestions = await narsil.suggest('products', { prefix: 'mech', limit: 5 })
+// suggestions.terms => [{ term: 'mechan', documentFrequency: 12 }, ...]
+```
+
+## Vector search
+
+Declare a `vector[N]` field in the schema and insert documents carrying arrays of that exact length. Small fields use an exact brute-force scan. Once a field reaches 1,024 vectors, the engine builds an HNSW graph in the background and switches the field to approximate search. The cutoff and graph parameters are configurable per index through `vectorPromotion`.
+
+```ts
+await narsil.createIndex('docs', {
+  schema: {
+    title: 'string',
+    embedding: 'vector[768]',
+  },
+  vectorPromotion: {
+    threshold: 2048,
+    hnswConfig: { m: 16, efConstruction: 200, metric: 'cosine' },
+    quantization: 'sq8',
+  },
+})
+
+await narsil.insert('docs', {
+  title: 'Distributed consensus',
+  embedding: myPrecomputedVector,
+})
+
+const results = await narsil.query('docs', {
+  mode: 'vector',
+  vector: {
+    field: 'embedding',
+    value: myQueryVector,
+    metric: 'cosine',
+    similarity: 0.35,
+    efSearch: 100,
+  },
+  limit: 10,
+})
+```
+
+The `vector` parameter takes either a raw `value` array or a `text` string for auto-embedding, and passing both fails with `EMBEDDING_CONFIG_INVALID`. `metric` selects `'cosine'` (the default), `'dotProduct'`, or `'euclidean'`. `similarity` sets a score floor; hits below it drop before `limit` applies, so a page can come back short. For `euclidean`, the floor applies to the similarity mapping `1 / (1 + distance)`. `efSearch` raises HNSW recall at the cost of latency and has no effect while the field still uses the brute-force backend. A `value` whose length differs from the field's declared dimension fails with `VECTOR_DIMENSION_MISMATCH`.
+
+Quantization mode `'sq8'` is the default; it stores scalar-quantized int8 vectors, which cuts vector memory roughly 4x. Set `quantization: 'none'` to keep full float32 vectors.
+
+### Vector maintenance
+
+Removed and updated vectors leave tombstones in the HNSW graph, which slows queries as they accumulate. Two maintenance calls clean up, and a status call reports whether either is worth running:
+
+```ts
+const status = narsil.vectorMaintenanceStatus('docs')
+// [{ fieldName, tombstoneRatio, graphCount, bufferSize, building, estimatedCompactMs, estimatedOptimizeMs }]
+
+await narsil.compactVectors('docs', 'embedding')
+
+await narsil.optimizeVectors('docs', 'embedding')
+```
+
+`compactVectors` drops tombstones without rebuilding the graph and runs synchronously. `optimizeVectors` rebuilds the graph from live vectors, which takes longer and restores full query speed. Omit the field name to run maintenance on every vector field in the index.
+
+## Hybrid search
+
+Hybrid mode runs the full-text and vector searches in one query and fuses the two rankings.
+
+```ts
+const results = await narsil.query('docs', {
+  mode: 'hybrid',
+  term: 'how do search engines scale',
+  vector: { field: 'embedding', text: 'how do search engines scale' },
+  hybrid: { strategy: 'rrf', k: 60 },
+  limit: 10,
+})
+```
+
+Two fusion strategies are available:
+
+- `'rrf'` (the default) applies reciprocal rank fusion, which combines the two rankings by position instead of by score. `k` dampens the contribution of lower ranks and defaults to 60.
+- `'linear'` normalizes both score sets to [0, 1] and blends them as `alpha * vector + (1 - alpha) * text`. `alpha` defaults to 0.5 and clamps to [0, 1].
+
+```ts
+const weighted = await narsil.query('docs', {
+  mode: 'hybrid',
+  term: 'partition rebalancing',
+  vector: { field: 'embedding', text: 'partition rebalancing' },
+  hybrid: { strategy: 'linear', alpha: 0.7 },
+})
+```
+
+## Geosearch
+
+Declare a `geopoint` field and insert documents with `{ lat, lon }` values. Geo conditions are filters, so they compose with terms, other filters, facets, and every other query feature.
+
+```ts
+await narsil.createIndex('stores', {
+  schema: {
+    name: 'string',
+    location: 'geopoint',
+  },
+})
+
+await narsil.insert('stores', {
+  name: 'Osu Night Market',
+  location: { lat: 5.5571, lon: -0.1824 },
+})
+
+const nearby = await narsil.query('stores', {
+  filters: {
+    fields: {
+      location: {
+        radius: { lat: 5.556, lon: -0.1969, distance: 5, unit: 'km' },
+      },
+    },
+  },
+})
+
+const inArea = await narsil.query('stores', {
+  filters: {
+    fields: {
+      location: {
+        polygon: {
+          points: [
+            { lat: 5.52, lon: -0.25 },
+            { lat: 5.52, lon: -0.15 },
+            { lat: 5.62, lon: -0.15 },
+            { lat: 5.62, lon: -0.25 },
+          ],
+        },
+      },
+    },
   },
 })
-// hit.highlights.title.snippet => '<mark>Mechanical</mark> Keyboard'
 ```
 
-### Embedding adapters
+Radius filters measure Haversine distance by default and accept `unit: 'km' | 'mi' | 'm'`. Set `highPrecision: true` to switch to Vincenty's iterative formula for long-distance accuracy. Polygon filters test containment with ray casting. Both filter shapes accept `inside: false` to invert the match and return documents outside the area.
 
-Auto-embed text fields into vectors on insert and query. Pass an `EmbeddingAdapter` at the instance level (shared default) or per-index. When a document is inserted without a pre-computed vector, Narsil concatenates the configured source text fields and calls the adapter to generate the embedding. On query, pass `text` instead of `value` to auto-embed the search phrase.
+## Embedding adapters
+
+Embedding adapters turn text into vectors automatically, on insert and at query time. Configure a default adapter for the whole engine, register named adapters, or set one per index. Map each vector field to the text fields it embeds; multiple source fields concatenate before embedding.
 
 ```ts
 import { createNarsil } from '@delali/narsil'
@@ -262,7 +622,8 @@ import { createOpenAIEmbedding } from '@delali/narsil/embeddings/openai'
 
 const narsil = await createNarsil({
   embedding: createOpenAIEmbedding({
-    apiKey: process.env.OPENAI_API_KEY,
+    baseUrl: 'https://api.openai.com/v1',
+    apiKey: process.env.OPENAI_API_KEY ?? '',
     model: 'text-embedding-3-small',
     dimensions: 1536,
   }),
@@ -292,16 +653,51 @@ const results = await narsil.query('articles', {
 })
 ```
 
-Two adapters ship with the project:
+### Named adapters
+
+An adapter instance is a function and cannot be serialized, so an index that names its adapter survives durability recovery: the engine persists the name in index metadata and rebinds it on the next start. Register names through the config or at runtime, and reference them from the index config:
+
+```ts
+import { createNarsil } from '@delali/narsil'
+import { createOpenAIEmbedding } from '@delali/narsil/embeddings/openai'
+
+const engine = await createNarsil({
+  embeddingAdapters: {
+    'openai-small': createOpenAIEmbedding({
+      baseUrl: 'https://api.openai.com/v1',
+      apiKey: process.env.OPENAI_API_KEY ?? '',
+      model: 'text-embedding-3-small',
+      dimensions: 1536,
+    }),
+  },
+  durability: { directory: './narsil-data' },
+})
+
+await engine.createIndex('articles', {
+  schema: { title: 'string', embedding: 'vector[1536]' },
+  embedding: {
+    adapter: 'openai-small',
+    fields: { embedding: ['title'] },
+  },
+})
+
+engine.registerEmbeddingAdapter('openai-small', myReplacementAdapter)
+```
+
+`registerEmbeddingAdapter` rebinds every index referencing that name, which lets you rotate credentials or swap models without recreating indexes.
+
+### Shipped adapters
 
 | Adapter | Package | Dependencies |
 | --- | --- | --- |
-| OpenAI | `@delali/narsil/embeddings/openai` | None (uses `fetch`) |
-| Transformers.js | `@delali/narsil-embeddings-transformers` | `@huggingface/transformers` (peer) |
+| OpenAI | `@delali/narsil/embeddings/openai` | The adapter has no dependencies and uses `fetch`. |
+| Transformers.js | `@delali/narsil-embeddings-transformers` | The adapter needs `@huggingface/transformers` as a peer dependency. |
+
+The OpenAI adapter retries retryable failures with exponential backoff and jitter, chunks batches at 2,048 inputs per request, and accepts a timeout and a retry cap. `baseUrl` points at any OpenAI-compatible endpoint, and `apiKey` accepts a string or a function returning one, so short-lived credentials work. The Transformers.js adapter runs models locally with lazy pipeline initialization, supports WebGPU, WASM, and CPU backends, and handles asymmetric models such as E5 and BGE through `documentPrefix` and `queryPrefix`. Its own [README](../embeddings-transformers/README.md) documents every option.
 
-The OpenAI adapter includes retry logic with exponential backoff and jitter, batch chunking (up to 2,048 inputs per request), and configurable timeouts. The Transformers.js adapter runs models locally with lazy pipeline initialization, supports WebGPU/WASM/CPU backends, and handles asymmetric models through document/query prefixes.
+### Custom adapters
 
-Build a custom adapter by implementing the `EmbeddingAdapter` interface:
+Build an adapter by satisfying the `EmbeddingAdapter` interface:
 
 ```ts
 interface EmbeddingAdapter {
@@ -312,51 +708,260 @@ interface EmbeddingAdapter {
 }
 ```
 
-If `embedBatch` is not provided, Narsil falls back to parallel `embed()` calls with a concurrency limit of 8.
+When `embedBatch` is missing, Narsil falls back to parallel `embed()` calls with a concurrency limit of 8.
+
+## Persistence
 
-### Term suggestions
+Persistence stores serialized partitions through a pluggable adapter. Three adapters ship with the package:
 
-Get autocomplete suggestions from the index's term dictionary. The API tokenizes the input, extracts the last word, and matches it against stored terms across all partitions. Results are ranked by document frequency.
+| Adapter | Import | Environment |
+| --- | --- | --- |
+| Memory | `@delali/narsil/adapters/memory` | The adapter works everywhere and is meant for tests. |
+| Filesystem | `@delali/narsil/adapters/filesystem` | The adapter runs on Node.js, Bun, and Deno. |
+| IndexedDB | `@delali/narsil/adapters/indexeddb` | The adapter runs in browsers. |
 
 ```ts
-const suggestions = await narsil.suggest('products', {
-  prefix: 'mech',
-  limit: 5,
+import { createNarsil } from '@delali/narsil'
+import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
+
+const narsil = await createNarsil({
+  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
+  flush: { interval: 5000, mutationThreshold: 100 },
 })
-// suggestions.terms => [{ term: 'mechan', documentFrequency: 12 }, ...]
 ```
 
-### Cursor-based pagination
+In the browser, `createIndexedDBPersistence({ dbName, storeName })` takes the same place, and both config fields are optional. Flushing is debounced: dirty partitions serialize on the `flush.interval` timer or after `flush.mutationThreshold` mutations, whichever fires first.
 
-Deep pagination using `searchAfter` cursors that track position across partitions. No offset-based performance degradation for deep result sets.
+The serialization format is `.nrsl`, a 32-byte header followed by a MessagePack payload. The format is cross-language portable and specified in [`packages/spec`](../spec), so a Python or Rust implementation can read and write the same files.
 
-### Results pinning
+A custom backend satisfies the `PersistenceAdapter` interface: `save(key, data)`, `load(key)`, `delete(key)`, and `list(prefix)`, all returning promises.
 
-Inject specific documents at fixed positions in the result set, useful for sponsored or editorial content.
+## Durability
 
-### Persistence
+Debounced persistence can lose the writes made after the last flush. Durability closes that window with a write-ahead log: every mutation appends to the log before it is acknowledged, periodic checkpoints capture the index state, and recovery replays the log over the newest checkpoint. Enable it with a directory:
 
-Plug in any storage backend through the persistence adapter interface. Three built-in adapters ship with the package:
+```ts
+const narsil = await createNarsil({
+  durability: {
+    directory: './narsil-data',
+    mode: 'sync',
+  },
+})
+```
 
-| Adapter | Import | Environment |
+`createNarsil` runs recovery before it resolves, so indexes, documents, and named embedding adapter bindings are back before the first call. `checkpoint(indexName)` forces a checkpoint outside the automatic schedule:
+
+```ts
+await narsil.checkpoint('products')
+```
+
+### DurabilityConfig
+
+| Field | Type | Default | Description |
+| --- | --- | --- | --- |
+| `directory` | `string` | none | Sets the root directory for the log and checkpoints. |
+| `mode` | `'sync' \| 'async'` | `'sync'` | Selects the acknowledgement contract described below. |
+| `flushIntervalMs` | `number` | `1000` | Sets how often the async mode flushes the log to disk. |
+| `segmentMaxBytes` | `number` | `67108864` | Caps a log segment at this size (64 MiB) before rolling to a new one. |
+| `checkpointIntervalMs` | `number` | `300000` | Sets the time between automatic checkpoints (5 minutes). |
+| `checkpointMutationThreshold` | `number` | `100000` | Sets the mutation count that triggers a checkpoint early. |
+| `compactionThreshold` | `number` | `12` | Sets the checkpoint segment count that triggers compaction. |
+
+In `sync` mode a write is not acknowledged until it is on disk, so a crash never loses a write your caller saw succeed. In `async` mode writes acknowledge immediately while the log flushes on `flushIntervalMs`, which is faster but can lose the final interval on a hard crash. Durability failures surface through the `durabilityError` event; see [Events](#events).
+
+## Snapshots and restore
+
+`snapshot(indexName)` serializes a whole index, including its documents, schema, and vector data, into one portable byte array. `restore(indexName, data)` rebuilds an index from those bytes, replacing the index if it already exists.
+
+```ts
+import { readFile, writeFile } from 'node:fs/promises'
+
+const bytes = await narsil.snapshot('products')
+await writeFile('./products.nrsl', bytes)
+
+const saved = await readFile('./products.nrsl')
+await narsil.restore('products', new Uint8Array(saved))
+```
+
+Snapshots use the same cross-language `.nrsl` envelope as persistence, so one engine's snapshot restores in another process, another machine, or another language implementation. Restoring bytes from an incompatible envelope version fails with `ENVELOPE_VERSION_MISMATCH`.
+
+## Partitions and rebalancing
+
+An index starts with the partition count set by `partitions.maxPartitions`, which defaults to 1. Documents route to partitions by FNV-1a hash of their id, which keeps routing deterministic across processes and languages. `partitions.maxDocsPerPartition` caps capacity: when an index holds `maxDocsPerPartition * partitionCount` documents, further inserts fail with `PARTITION_CAPACITY_EXCEEDED`.
+
+```ts
+await narsil.createIndex('logs', {
+  schema: { message: 'string' },
+  partitions: { maxPartitions: 4, maxDocsPerPartition: 250_000 },
+})
+
+await narsil.rebalance('logs', 8)
+
+await narsil.updatePartitionConfig('logs', { maxDocsPerPartition: 500_000 })
+```
+
+`rebalance(indexName, targetPartitionCount)` reshapes the index to a new partition count while it stays online. Writes arriving during the reshape buffer in a write-ahead queue and replay in order when the reshape completes, and queries keep answering throughout. `updatePartitionConfig` adjusts the caps at runtime; it rejects a new capacity below the current document count with `PARTITION_CAPACITY_EXCEEDED` and rejects changes while a rebalance is running with `PARTITION_REBALANCING_BACKPRESSURE`.
+
+Two measured costs are worth knowing before raising partition counts:
+
+**Single-process overhead.** Partitioning pays off when shards live on separate workers or hosts. Inside one Node.js thread, going from 1 to 20 partitions costs about 14% of insert throughput, 28% of median search latency, and 27% at p95, with no scaling upside. Keep `maxPartitions` low for single-process deployments and raise it once partitions fan out.
+
+**Rebalance latency spikes.** While a reshape runs, worst-tick p95 latency can climb to about 25ms compared with around 11ms in steady state. Schedule reshapes during low-traffic windows, or pre-size the index with `maxDocsPerPartition` so mid-load reshapes never become necessary.
+
+## Workers
+
+Search can move off the main thread through worker threads on Node.js and Bun, or Web Workers in browsers and Deno. With `workers.enabled: true`, the engine starts in direct mode and promotes itself to the worker pool once any index passes `promotionThreshold` documents or all indexes together pass `totalPromotionThreshold`. The API stays identical before and after promotion.
+
+```ts
+const narsil = await createNarsil({
+  workers: {
+    enabled: true,
+    count: 4,
+    promotionThreshold: 10_000,
+    totalPromotionThreshold: 50_000,
+  },
+})
+```
+
+Promotion emits the `workerPromote` event, and a crashed worker emits `workerCrash`; see [Events](#events). Worker heap usage appears in `getMemoryStats()`.
+
+## Multi-instance invalidation
+
+When several engine instances share one persistence backend, the invalidation adapter tells the others which partitions changed so they evict stale cache instead of serving old data. Two adapters ship with the package, and `@delali/narsil/invalidation/noop` stubs the interface for single-instance deployments:
+
+| Adapter | Import | Use case |
 | --- | --- | --- |
-| Memory | `@delali/narsil/adapters/memory` | All |
-| Filesystem | `@delali/narsil/adapters/filesystem` | Node.js, Bun, Deno |
-| IndexedDB | `@delali/narsil/adapters/indexeddb` | Browser |
+| Filesystem | `@delali/narsil/invalidation/filesystem` | The adapter coordinates processes on one machine through marker files. |
+| BroadcastChannel | `@delali/narsil/invalidation/broadcast-channel` | The adapter coordinates browser tabs through a BroadcastChannel. |
+
+```ts
+import { createNarsil } from '@delali/narsil'
+import { createFilesystemPersistence } from '@delali/narsil/adapters/filesystem'
+import { createFilesystemInvalidation } from '@delali/narsil/invalidation/filesystem'
+
+const narsil = await createNarsil({
+  persistence: createFilesystemPersistence({ directory: './narsil-data' }),
+  invalidation: createFilesystemInvalidation({ directory: './narsil-data', pollInterval: 1000 }),
+})
+```
+
+The invalidation channel also carries partition statistics for the `broadcast` scoring mode; see [Scoring modes](#scoring-modes). A custom adapter satisfies the `InvalidationAdapter` interface: `publish(event)`, `subscribe(handler)`, and `shutdown()`.
+
+## Plugins
+
+Plugins hook into the document and search lifecycle. A plugin is an object with a `name` and any of the optional hooks:
+
+| Hook | Fires |
+| --- | --- |
+| `beforeInsert` / `afterInsert` | The hooks fire around every document insert, including each document of a batch. |
+| `beforeUpdate` / `afterUpdate` | The hooks fire around every document update. |
+| `beforeRemove` / `afterRemove` | The hooks fire around every document removal. |
+| `beforeSearch` / `afterSearch` | The hooks fire around every query; `afterSearch` receives the results. |
+| `onIndexCreate` / `onIndexDrop` | The hooks fire when an index is created or dropped. |
+
+The interface also declares `onPartitionSplit` and `onWorkerPromote`, which are reserved for partition and worker lifecycle notifications; the engine does not fire them yet. Subscribe to the `partitionRebalance` and `workerPromote` [events](#events) for those signals today.
+
+```ts
+import { createNarsil, type NarsilPlugin } from '@delali/narsil'
+
+const auditLog: NarsilPlugin = {
+  name: 'audit-log',
+  async afterInsert(ctx) {
+    console.log(`indexed ${ctx.docId} into ${ctx.indexName}`)
+  },
+  async afterSearch(ctx) {
+    console.log(`query on ${ctx.indexName} returned ${ctx.results?.hits.length ?? 0} hits`)
+  },
+}
+
+const narsil = await createNarsil({ plugins: [auditLog] })
+```
+
+Hooks can be async, and `before*` hooks run to completion before the operation applies, so a thrown error in `beforeInsert` rejects the insert. Errors thrown in `after*` hooks log a warning and never fail the operation that already succeeded.
+
+## Events
+
+`on(event, handler)` subscribes to engine events and `off(event, handler)` unsubscribes. The payloads are typed through `NarsilEventMap`.
+
+| Event | Payload | Meaning |
+| --- | --- | --- |
+| `persistenceError` | `{ indexName, partitionId, error, retriesExhausted }` | A partition flush failed; `retriesExhausted` reports whether the engine gave up. |
+| `durabilityError` | `{ error }` | A write-ahead log or checkpoint operation failed. |
+| `workerCrash` | `{ workerId, indexNames, error }` | A worker died; the engine reassigns its indexes. |
+| `workerPromote` | `{ workerCount, reason }` | The engine moved search onto the worker pool. |
+| `partitionRebalance` | `{ indexName, oldCount, newCount }` | A partition reshape completed. |
+
+```ts
+narsil.on('persistenceError', payload => {
+  console.error(`flush failed for ${payload.indexName}:`, payload.error)
+})
+```
+
+Subscribe to `persistenceError` and `durabilityError` in any deployment that persists data; they are the engine's only channel for reporting background write failures.
+
+## Errors
+
+Every failure throws a `NarsilError` carrying a stable string `code`, a human-readable `message`, and a `details` object with the values that produced the failure. The full set of codes is exported as `ErrorCodes`.
+
+```ts
+import { ErrorCodes, NarsilError } from '@delali/narsil'
+
+try {
+  await narsil.insert('products', { title: 42 })
+} catch (err) {
+  if (err instanceof NarsilError && err.code === ErrorCodes.DOC_VALIDATION_FAILED) {
+    console.error(err.message, err.details)
+  }
+}
+```
+
+The codes you handle most often:
+
+| Code | Thrown when |
+| --- | --- |
+| `INDEX_NOT_FOUND` / `INDEX_ALREADY_EXISTS` | An operation names an unknown index, or `createIndex` reuses a name. |
+| `DOC_NOT_FOUND` / `DOC_ALREADY_EXISTS` | A read, update, or removal names an unknown id, or an insert reuses one. |
+| `DOC_VALIDATION_FAILED` / `DOC_MISSING_REQUIRED_FIELD` | A document does not match the schema or omits a required field. |
+| `SEARCH_INVALID_FIELD` / `SEARCH_INVALID_FILTER` / `SEARCH_INVALID_CURSOR` | A query names an unknown field, passes a malformed filter, or replays a bad cursor. |
+| `VECTOR_DIMENSION_MISMATCH` | A vector's length differs from the field's declared dimension. |
+| `EMBEDDING_FAILED` / `EMBEDDING_CONFIG_INVALID` | An adapter call failed, or the embedding configuration is contradictory. |
+| `PARTITION_CAPACITY_EXCEEDED` / `PARTITION_REBALANCING_BACKPRESSURE` | An insert passes the capacity cap, or a config change collides with a running reshape. |
+| `LANGUAGE_NOT_SUPPORTED` | An index config names a language module that was never imported. |
+| `CONFIG_INVALID` | A configuration value is out of range or contradictory. |
+
+## Memory reporting
+
+Narsil reports memory at three levels: per index, per partition, and per runtime. `getStats(indexName)` returns `estimatedMemoryBytes`, a formula-based estimate for the index's main-thread partitions and vector structures.
+
+```ts
+const indexStats = narsil.getStats('products')
 
-Persistence uses debounced flushing: dirty partitions serialize on a timer or after a mutation count threshold, whichever fires first. The serialization format is `.nrsl`, a 32-byte header followed by a MessagePack payload. The format is cross-language portable, so a Python or Rust implementation can read and write the same files.
+console.log(indexStats.estimatedMemoryBytes)
+```
 
-### Multi-instance invalidation
+The estimate comes from document counts, posting lists, field indexes, and vector indexes. It excludes V8 object headers, allocator overhead, and other host runtime costs, so use it for comparing indexes inside one process rather than for sizing host memory.
+
+`getPartitionStats(indexName)` returns the same estimate for each partition. The partition values sum to `getStats(indexName).estimatedMemoryBytes`.
+
+```ts
+const partitions = narsil.getPartitionStats('products')
 
-When multiple instances share the same persistence backend, the invalidation adapter coordinates cache eviction. A filesystem adapter uses marker files for multi-process deployments. A BroadcastChannel adapter handles cross-tab coordination in browsers.
+for (const partition of partitions) {
+  console.log(partition.partitionId, partition.estimatedMemoryBytes)
+}
+```
 
-### Plugin system
+`getMemoryStats()` returns a runtime snapshot and worker reports. It is async because workers report their heap usage through the worker message channel.
 
-Hook into the document and search lifecycle with plugins. Plugins can run before/after insert, remove, update, and search operations, and respond to index creation, partition splits, and worker promotions.
+```ts
+const memory = await narsil.getMemoryStats()
 
-### Schema validation
+console.log(memory.estimatedIndexBytes)
+console.log(memory.process?.heapUsed)
+console.log(memory.workers)
+```
 
-Define typed schemas with support for `string`, `number`, `boolean`, `enum`, `geopoint`, `vector[N]`, and array variants (`string[]`, `number[]`, `boolean[]`, `enum[]`). Nested objects up to 4 levels deep. Documents are validated against the schema at insertion time.
+`memory.estimatedIndexBytes` sums `estimatedMemoryBytes` across every index in this Narsil instance. `memory.process` comes from `process.memoryUsage()` when the runtime exposes it. It measures the whole host process, so two Narsil instances in one Node.js process report the same process numbers, and browser runtimes return `null`. `memory.workers` lists `heapUsed`, `heapTotal`, and `external` for each active worker, and the array is empty before worker promotion.
 
 ## Language support
 
@@ -382,38 +987,78 @@ import '@delali/narsil/languages/swahili'
 import '@delali/narsil/languages/twi'
 ```
 
-## Search modes
+Import the module before creating an index that names it; naming a language that was never imported fails with `LANGUAGE_NOT_SUPPORTED`. English is always available. `registerLanguage(module)` adds a language of your own: a module carries a name, a tokenizer, a stemmer, and a stop word set, and any of the shipped modules serves as a reference.
 
-| Mode | Method | Scoring |
-| --- | --- | --- |
-| Full-text | `mode: 'fulltext'` | BM25 with field boosting |
-| Vector | `mode: 'vector'` | Cosine similarity, dot product, or Euclidean distance |
-| Hybrid | `mode: 'hybrid'` | Weighted combination of BM25 + vector similarity |
+## HTTP server
 
-## Search quality
+`@delali/narsil/server` wraps an engine you build in a REST API. You own the engine and its configuration (durability, embedding adapters, workers), and the server shares it across requests. The HTTP layer runs on `uWebSockets.js`, an optional peer dependency:
 
-Ranking quality is measured against the [Cranfield Collection](https://ir-datasets.com/cranfield.html), the foundational information retrieval benchmark created in the 1960s. It contains 1,400 documents and 225 queries with exhaustive human relevance judgments, meaning every query-document pair was scored by domain experts. All engines in this comparison use identical stop words (Lucene English, 35 terms), Porter stemming, and default BM25 parameters.
+```bash
+pnpm add -E uWebSockets.js@github:uNetworking/uWebSockets.js#v20.58.0
+```
 
-| Engine | nDCG@10 | P@10 | MAP | MRR |
-| --- | ---: | ---: | ---: | ---: |
-| **Narsil** | **0.3739** | **0.2458** | **0.2614** | **0.5638** |
-| Orama 3.1.18 | 0.2911 | 0.1836 | 0.1846 | 0.4821 |
-| MiniSearch 7.2.0 | 0.0077 | 0.0067 | 0.0027 | 0.0139 |
+```ts
+import { createNarsil } from '@delali/narsil'
+import { createServer } from '@delali/narsil/server'
+
+const engine = await createNarsil({ durability: { directory: './narsil-data' } })
+
+const server = createServer(engine, {
+  host: '127.0.0.1',
+  port: 7700,
+})
+
+await server.listen()
+```
+
+`ServerOptions` also accepts `cors`, an `onRequest` hook for authentication, `limits` for body-size and concurrency caps, `embeddingAdapters` that JSON index configs reference by name, a `taskStore` that keeps long-running task status across restarts, an `instanceId` for task recovery, and `allowInsecure` for trusted private networks. The server refuses to bind a non-loopback address without an `onRequest` hook, because the admin endpoints can destroy data.
+
+The full surface:
+
+| Method and path | Purpose |
+| --- | --- |
+| `GET /livez`, `GET /readyz`, `GET /health` | The probes report liveness and readiness without authentication. |
+| `GET /version` | The endpoint reports the build identity stamped at startup. |
+| `GET /stats/memory` | The endpoint returns `getMemoryStats()`. |
+| `POST /indexes`, `GET /indexes`, `DELETE /indexes/{name}` | The endpoints create, list, and drop indexes. |
+| `GET /indexes/{name}/stats`, `GET /indexes/{name}/partitions`, `GET /indexes/{name}/count` | The endpoints report index, partition, and document-count statistics. |
+| `POST /indexes/{name}/_clear` | The endpoint removes every document but keeps the index. |
+| `POST /indexes/{name}/documents` | The endpoint inserts one document. |
+| `GET`, `PUT`, `PATCH`, `DELETE /indexes/{name}/documents/{id}` | The endpoints read, upsert, update, and remove one document. |
+| `GET /indexes/{name}/documents/{id}/_exists` | The endpoint reports whether the id exists. |
+| `POST /indexes/{name}/documents/_batch` | The endpoint runs a batch insert, update, or delete with partial results. |
+| `POST /indexes/{name}/documents/_multi-get` | The endpoint fetches many documents by id. |
+| `POST /indexes/{name}/documents/_import` | The endpoint streams an NDJSON corpus in bounded batches. |
+| `POST /indexes/{name}/search`, `POST /indexes/{name}/search/preflight`, `POST /indexes/{name}/suggest` | The endpoints run queries, match counts, and autocomplete. |
+| `POST /indexes/{name}/_checkpoint`, `GET /indexes/{name}/snapshot`, `POST /indexes/{name}/restore` | The endpoints force a checkpoint, download a snapshot, and restore one. |
+| `GET /indexes/{name}/vector-maintenance`, `POST /indexes/{name}/vectors/_compact`, `POST /indexes/{name}/vectors/_optimize` | The endpoints report and run vector maintenance. |
+| `POST /indexes/{name}/_rebalance`, `POST /indexes/{name}/partition-config` | The endpoints reshape partitions and adjust partition caps. |
+| `GET /tasks`, `GET /tasks/{id}` | The endpoints report long-running task status. |
+
+The [HTTP server example](examples/http-server/README.md) documents every endpoint with request and response bodies, curl walkthroughs, Docker packaging, and the environment-driven configuration of a production launcher.
+
+## Distribution
+
+`@delali/narsil/distribution` holds the building blocks of Narsil's multi-node cluster mode: node roles, replication, coordinator adapters, and query routing. The distribution layer is under active development and highly experimental. It currently runs only in-process, its APIs change without notice, and it is not ready for production deployments. The design is specified in [`packages/spec/distribution`](../spec/distribution), and this section will grow into full documentation once the cluster mode is runnable.
+
+## Search quality
+
+Ranking quality is measured against the [BEIR](https://github.com/beir-cellar/beir) SciFact corpus, 5,183 documents with 300 judged queries, where a human relevance judgment scores each query-document pair. Narsil runs in one process against Orama and MiniSearch, and every engine uses identical stop words (Lucene English, 35 terms), Porter stemming, and default BM25 parameters. Narsil takes the top nDCG@10 of the three.
 
 **What these metrics mean:**
 
-- **nDCG@10** measures whether the most relevant documents appear near the top of the results. A score of 1.0 would mean perfect ranking; 0.0 means no relevant documents in the top 10.
-- **P@10** is the fraction of the top 10 results that are relevant. Narsil averages about 2.5 relevant results in every top-10 list.
-- **MAP** tracks precision at every rank where a relevant document appears. Higher MAP means relevant documents cluster near the top rather than being spread through the result set.
-- **MRR** measures how far a user scrolls to find the first relevant result. Narsil's 0.56 means the first relevant document typically appears at position 2.
+- **nDCG@10** measures whether the most relevant documents appear near the top of the results. A score of 1.0 means perfect ranking, and 0.0 means no relevant documents appear in the top 10.
+- **P@10** is the fraction of the top 10 results that are relevant.
+- **MAP** tracks precision at every rank where a relevant document appears. A higher MAP means relevant documents cluster near the top of the ranking.
+- **MRR** measures how soon the first relevant result appears. A higher value means the first relevant document sits closer to the top.
 
-These scores fall within the expected range for BM25 on standard IR benchmarks (published BM25 baselines on BEIR datasets range from 0.24 to 0.63 nDCG@10). Narsil also runs a [Cranfield regression test](src/__tests__/relevance/cranfield.test.ts) in CI that fails the build if ranking quality drops below calibrated thresholds.
+A separate [SciFact regression test](src/__tests__/relevance/scifact.test.ts) runs in CI on the same corpus and fails the build if ranking quality drops below calibrated thresholds.
 
-Reproduce: `pnpm -C packages/benchmarks bench --tiers quality`
+Reproduce these scores with `pnpm --filter benchmarks bench -- --tiers relevance`. The full quality, throughput, latency, and memory tables for all three engines are in [BENCHMARKS.md](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md).
 
 Narsil also runs as a search server. On the BEIR information-retrieval datasets it is measured against Elasticsearch, OpenSearch, Qdrant, Weaviate, Typesense, and Meilisearch across keyword, vector, and hybrid retrieval. See [the full benchmarks](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md) for those results.
 
-## Runtime support (TypeScript)
+## Runtime support
 
 | Runtime | Concurrency | Persistence | Invalidation |
 | --- | --- | --- | --- |
@@ -422,6 +1067,8 @@ Narsil also runs as a search server. On the BEIR information-retrieval datasets
 | Deno | Web Workers | Filesystem | BroadcastChannel |
 | Browser | Web Workers | IndexedDB | BroadcastChannel |
 
+The [browser example](examples/browser/README.md) shows an embedded engine with IndexedDB persistence, and the [server app example](examples/server-app/README.md) shows a full search UI backed by the HTTP server.
+
 ## License
 
 Apache-2.0