lucivy-wasm 4.0.1 → 4.1.0

package/README.md

@@ -1,18 +1,40 @@
-# lucivy-wasm 4.0.1
+# lucivy-wasm 4.1.0
 
-Full-text search for code and technical text, in the browser. Substrings, fuzzy and regex **across token boundaries**, BM25, exact byte spans, every answer checked against the files — the same engine as the native bindings, built with emscripten: **threads** (pthreads over SharedArrayBuffer), OPFS persistence, snapshot import. Runs in a Web Worker. MIT.
+**One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. In the browser — the same engine as the native bindings, built with emscripten: **threads** (pthreads over SharedArrayBuffer), OPFS persistence, snapshot import. Runs in a Web Worker. MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — it clones lucivy's own source from GitHub and indexes it in your browser.
 
+### What's new in 4.1
+
+- **`positions: false` in `IndexConfig` — about 40 % less OPFS** (typed in `lucivy.d.ts`). The postings keep each token's documents and frequencies instead of its positions, and no position sidecar is written; every match is verified on the stored text — same documents, spans and scores. Measured in Chrome on Linux kernel files: 2 000 files 268 → 174 MB, **10 000 files 1 052 → 638 MB**, indexed in 37 s instead of 43, the tab's memory peak unchanged (1.5 GB); the playground's 21-query parity panel returns the same counts, top 10, scores and spans on both indexes (the differences: equal scores ordered otherwise, and spans the default index of 4.0.2 returned twice — fixed in 4.1). Natively the whole kernel goes 5 289 → 2 598 MB. Every text field must be stored (the default); excludes `derived_in_ram`; fixed at creation. The playground builds its index this way with `?nopos`.
+- **Fix: one occurrence, one span.** In relaxed mode a needle that ends a word cut into chunks — `lock` in `superblock` — came back twice, and counted twice in the score (542 duplicated spans for `lock` over 10 000 kernel files, since the v3 engine, 4.0.2 included); the ground-truth harness now counts duplicates too
+
 ### What's new in 4.0.0
 
-- **The index is 3.7× smaller**, and a tab holds a whole repository: the playground's prompt indexes MDN Web Docs (14 611 pages, 14 s), the **entire Linux 2.6.0 kernel** (14 032 files, 28 s, 1.1 GB held in memory), Go, Godot, **TypeScript (39 044 files, 33 s)**, PostgreSQL, CPython, Redis, Git, curl, SQLite, nginx — `index <name>`, kept in OPFS, reopened in seconds. The ceiling of a tab is about 200 MB of text. Browser against native on the 2.6.0 kernel: same counts and spans, 41 s to index against 23, substrings 10-18 ms against 2 ([README](../../README.md#browser-against-native)).
+- **The index is 3.7× smaller**, and a tab holds a whole repository: the playground's prompt indexes MDN Web Docs (14 611 pages, 14 s), the **entire Linux 2.6.0 kernel** (14 032 files, 28 s, 1.1 GB held in memory), Go, Godot, **TypeScript (39 044 files, 33 s)**, PostgreSQL, CPython, Redis, Git, curl, SQLite, nginx — `index <name>`, kept in OPFS, reopened in seconds. The ceiling of a tab is about 200 MB of text. Browser against native on the 2.6.0 kernel: same counts and spans, 41 s to index against 23, substrings 10-18 ms against 2 ([README](https://github.com/L-Defraiteur/lucivy/blob/main/README.md#browser-against-native)).
 - **`shared_dictionary` and `derived_in_ram` in `IndexConfig`** (typed in `lucivy.d.ts`): the shard dictionary is the playground's default for big corpora (23-25 % smaller); `derived_in_ram` stays an option here — it saves a quarter of OPFS but raises the tab's memory peak.
 - **`dictionary_wait` in `IndexConfig`** (typed): shared dictionary only — natively a commit returns before the shard's new texts are merged and a search waits for that merge by default; in the browser the merge stays inside the commit (few threads, and building the segments' dictionary pieces in parallel raised the memory high-water mark for no time gained), so the option changes nothing there yet
 - **`Lucivy.dropIndex(path)`** — delete an index directory through the worker: WASMFS caches what it mounted, so a directory removed from the main thread still exists for it and the next `create` at that path fails.
 - **A commit every 8 MB of text** in the playground's indexing loop (`?commitmb=M`), not only every 2 000 files: a segment's size, not its document count, sets the memory peak (Godot: 3.3 GB → 1.8 GB).
 - **Compatibility contract** — 4.0 opens a 3.0.x index and returns what 3.0.x returned; 3.0.x does not open a 4.0 index; the first commit converts for good.
 
+### Against Elasticsearch and tantivy — one corpus, one truth
+
+Same 93 983 Linux kernel files, 857 MB of text. Each engine is configured at its best for substring search, not at its default: Elasticsearch 8.19 with a trigram analyzer plus a `wildcard` field for regexes, tantivy 0.25 with its `NgramTokenizer`. The truth of every row is the same byte-by-byte scan of the files; a lucivy count is right only when its documents **and** its byte spans match it. On the substring itself all three agree to the document; where they part:
+
+| asked | truth | lucivy | Elasticsearch | tantivy |
+|---|---|---|---|---|
+| `spin_lock`, separators relaxed (also `spin lock`, `spin-lock`, `spinlock`) | 9 552 | **9 552**, 23 ms | 6 577 — not with this analyzer: its trigrams carry the underscore | 6 601 — relaxed is the only mode it has: the separator never enters its index |
+| `spinlokc`, two edits, across the token boundary | 10 034 | **10 034**, 148 ms | 3 549 — fuzziness compares whole terms | 6 557 — same |
+| `spin_lock_[a-z]+`, a regex, case folded | 5 510 | **5 510**, 219 ms | 5 510 on the wildcard field, 1 ms warm — as `[a-zA-Z]+`: Lucene's `case_insensitive` folds a pattern's literals, not its character classes | 0 — terms are already cut |
+| `de`, two characters | 93 009 | **93 009**, 7.7 M spans, 561 ms | 0, silently | 0, silently |
+| `retur -ENOMEM`, a fuzzy phrase | 14 449 | **14 449**, 30 ms | 14 446 (`span_near`), 24 ms — it does this well | — |
+| **where it matched**: `mutex_lock`, 5 145 documents | 20 797 spans | **all 20 797, 15 ms** | `highlight` on the top 200: 179 ms | verifying 5 145 stored texts: 96 ms |
+| your index **in your transaction** | — | **yes**: pluggable store, one commit for your rows and the index, rollback included | no: a server next to your database, a synchronisation to write | no: its own directory, its own commit |
+| shards and nodes scoring **as one index**, as a library | — | **yes**, asserted by `test_federated_search` | yes, as a cluster | no: one index, one scale of scores |
+
+Where a cell says "not with this analyzer", a purpose-built analyzer or plugin may get closer, at the price of designing it, configuring it and reindexing. Every question in the table is answered by lucivy's default index with nothing to configure, and each answer is checked. Sizes, indexing times and the full generated report: [docs/compare-engines-2026-09-05.md](https://github.com/L-Defraiteur/lucivy/blob/main/docs/compare-engines-2026-09-05.md); `benches/compare_engines.sh` replays it.
+
 ### What 3.0.0 brought
 
 Measured on 10 000 Linux kernel files, in the browser, 8 threads:
@@ -109,6 +131,10 @@ const index = await lucivy.create('/my-index', {
     // derived_in_ram: true — the three derived sidecars of each segment
     // (about a third of the index) are rebuilt in memory when the index
     // opens instead of written to OPFS. Same answers. Off by default.
+    // positions: false (4.1) — documents and frequencies instead of
+    // positions, no position sidecar: about 40 % less OPFS; every match is
+    // verified on the stored text, same answers. Text fields must be stored
+    // (the default); excludes derived_in_ram. On by default.
 });
 
 // Open an existing index from OPFS

package/js/lucivy.d.ts

@@ -327,6 +327,14 @@ export interface IndexConfig {
    * Off by default; fixed at creation.
    */
   derived_in_ram?: boolean;
+  /**
+   * `false` keeps each token's documents and frequencies instead of its
+   * positions and writes no position sidecar (37 % smaller on 10 000 kernel
+   * files); every match is verified on the stored text, so every text field
+   * must be stored. Same documents, spans and scores. Excludes
+   * `derived_in_ram`. On by default; fixed at creation.
+   */
+  positions?: boolean;
   /**
    * Shared dictionary only. A commit returns before the shard's new texts
    * are merged into the dictionary (a background task does it); a search

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy-wasm",
-  "version": "4.0.1",
+  "version": "4.1.0",
   "type": "module",
   "description": "Fast BM25 full-text search for browsers — WASM build of Lucivy with threading, OPFS persistence, and LUCE snapshot support",
   "license": "MIT",