@delali/narsil 0.1.11 → 0.1.13

package/README.md

@@ -12,7 +12,9 @@ Distributed search, reforged.
 
 Narsil is a distributed search engine with full-text, vector, hybrid, and geosearch. One codebase runs in two contexts: embedded in your application process, where queries answer without a network hop, and as a standalone search server with a REST API, a write-ahead log, and bulk NDJSON ingest. Both contexts run the same engine and store indexes in the same cross-language binary format (.nrsl), so an index built in one loads in the other.
 
-The engine partitions large indexes across workers and merges partition results into a single ranked answer. Its BM25 ranking matches the Anserini reference within 0.005 nDCG@10 on the BEIR datasets, and one node answers 1,020 keyword queries per second on BEIR SciFact ([benchmarks](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md)). The TypeScript package is the first implementation.
+The engine partitions large indexes across workers and merges partition results into a single ranked answer. Its BM25 ranking matches the Anserini reference within 0.005 nDCG@10 on the BEIR datasets. On BEIR SciFact it ranks level with Elasticsearch and OpenSearch at 0.678 nDCG@10 and answers 1,020 keyword queries per second, about a quarter more than either ([benchmarks](https://github.com/assetcorp/narsil/blob/main/BENCHMARKS.md)). This TypeScript package is the reference implementation of the cross-language `.nrsl` format, and a second-language port in Go or Rust is the headline item on the [roadmap](https://github.com/assetcorp/narsil/blob/main/ROADMAP.md).
+
+Try it in your browser at [narsil.sondelali.com/demo](https://narsil.sondelali.com/demo).
 
 > *narsil* is the sword of Elendil in Tolkien's Lord of the Rings, shattered into shards and later reforged. The name maps to the architecture: data shatters into partitions, each shard is independently persisted, and every query reforges them into a unified result.
 
@@ -196,6 +198,7 @@ await narsil.createIndex('articles', {
 | `stopWords` | `Set<string> \| (defaults: Set<string>) => Set<string>` | Replaces or transforms the language module's stop word set. |
 | `tokenizer` | `CustomTokenizer` | Replaces the built-in tokenizer with your own `tokenize(text)` implementation. |
 | `trackPositions` | `boolean` | Stores token positions for highlighting. The default is `true`. |
+| `surfaceForms` | `boolean` | Records the original spellings of stemmed words for suggestions and prefix completions. The default is `false`. See [Suggestions](#suggestions). |
 | `vectorPromotion` | `VectorIndexConfig` | Tunes the HNSW promotion threshold, graph parameters, and quantization. See [Vector search](#vector-search). |
 | `strict` | `boolean` | Rejects documents that carry fields missing from the schema. |
 | `embedding` | `EmbeddingFieldConfig` | Maps text fields to vector fields for auto-embedding. See [Embedding adapters](#embedding-adapters). |
@@ -314,6 +317,19 @@ const results = await narsil.query('products', {
 })
 ```
 
+### Search as you type
+
+`prefix: true` treats the last word of the query as an unfinished word, so `secur` matches documents containing `security`. Earlier words must match fully, and `tolerance` keeps applying to them while the unfinished word is completed instead of typo-corrected. Completions score against a shared document frequency and rank below full-word matches, so a document containing the exact typed word comes first. The option is off by default; turn it on for queries fired on every keystroke.
+
+Completions match against the term dictionary, which stores stemmed tokens, so a typed word that runs past the end of a stem stops matching: `security` is indexed as `secur`, and the query `securi` finds nothing. Create the index with `surfaceForms: true` to match completions against the original spellings instead; the same setting gives [suggestions](#suggestions) their display words.
+
+```ts
+const results = await narsil.query('products', {
+  term: 'mechanical keyb',
+  prefix: true,
+})
+```
+
 ### Score and coverage thresholds
 
 `minScore` drops hits scoring below a floor. `termMatch` sets how many query terms a document must match: `'any'` (the default) accepts one term, `'all'` requires every term, and a number requires at least that many terms.
@@ -476,11 +492,16 @@ const { count, elapsed } = await narsil.preflight('products', { term: 'keyboard'
 
 ### Suggestions
 
-`suggest(indexName, params)` returns autocomplete candidates from the index's term dictionary. It tokenizes the input, takes the last word as the prefix, and ranks matching terms by document frequency.
+`suggest(indexName, params)` returns autocomplete candidates. It tokenizes the input, takes the last word as the prefix, and ranks completions by the number of documents they match. By default the candidates are the stemmed tokens the index stores, so a catalogue containing "mechanical" suggests the stem `mechan`. Create the index with `surfaceForms: true` to suggest the words as they appear in your documents; the engine then records the original spelling of every word the stemmer changed and suggests `mechanical` instead.
 
 ```ts
+await narsil.createIndex('products', {
+  schema: { title: 'string', description: 'string' },
+  surfaceForms: true,
+})
+
 const suggestions = await narsil.suggest('products', { prefix: 'mech', limit: 5 })
-// suggestions.terms => [{ term: 'mechan', documentFrequency: 12 }, ...]
+// suggestions.terms => [{ term: 'mechanical', documentFrequency: 12 }, ...]
 ```
 
 ## Vector search