lucivy-wasm 4.0.2 → 4.2.0

package/README.md

@@ -1,9 +1,20 @@
-# lucivy-wasm 4.0.2
+# lucivy-wasm 4.2.0
 
 **One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. In the browser — the same engine as the native bindings, built with emscripten: **threads** (pthreads over SharedArrayBuffer), OPFS persistence, snapshot import. Runs in a Web Worker. MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — it clones lucivy's own source from GitHub and indexes it in your browser.
 
+### What's new in 4.2
+
+- **`search(query, { fields: false })` no longer reads any document**: the id comes from the fast field; with `fields: true` the documents are read through the scheduler (verified on 10 000 kernel files in Chrome: 968 hits with their content in 260 ms, 500 without in 12 ms).
+- **Spans only when `highlights: true`** — they were built for every search.
+- **Indexing**: the dictionary's compaction is three times faster natively and runs the same stages in sequence here; the single indexing thread and its 15 MB heap are unchanged, and so is the index on disk — 4.1 opens a 4.2 index.
+
+### What's new in 4.1
+
+- **`positions: false` in `IndexConfig` — about 40 % less OPFS** (typed in `lucivy.d.ts`). The postings keep each token's documents and frequencies instead of its positions, and no position sidecar is written; every match is verified on the stored text — same documents, spans and scores. Measured in Chrome on Linux kernel files: 2 000 files 268 → 174 MB, **10 000 files 1 052 → 638 MB**, indexed in 37 s instead of 43, the tab's memory peak unchanged (1.5 GB); the playground's 21-query parity panel returns the same counts, top 10, scores and spans on both indexes (the differences: equal scores ordered otherwise, and spans the default index of 4.0.2 returned twice — fixed in 4.1). Natively the whole kernel goes 5 289 → 2 598 MB. Every text field must be stored (the default); excludes `derived_in_ram`; fixed at creation. The playground builds its index this way with `?nopos`.
+- **Fix: one occurrence, one span.** In relaxed mode a needle that ends a word cut into chunks — `lock` in `superblock` — came back twice, and counted twice in the score (542 duplicated spans for `lock` over 10 000 kernel files, since the v3 engine, 4.0.2 included); the ground-truth harness now counts duplicates too
+
 ### What's new in 4.0.0
 
 - **The index is 3.7× smaller**, and a tab holds a whole repository: the playground's prompt indexes MDN Web Docs (14 611 pages, 14 s), the **entire Linux 2.6.0 kernel** (14 032 files, 28 s, 1.1 GB held in memory), Go, Godot, **TypeScript (39 044 files, 33 s)**, PostgreSQL, CPython, Redis, Git, curl, SQLite, nginx — `index <name>`, kept in OPFS, reopened in seconds. The ceiling of a tab is about 200 MB of text. Browser against native on the 2.6.0 kernel: same counts and spans, 41 s to index against 23, substrings 10-18 ms against 2 ([README](https://github.com/L-Defraiteur/lucivy/blob/main/README.md#browser-against-native)).
@@ -21,7 +32,7 @@ Same 93 983 Linux kernel files, 857 MB of text. Each engine is configured at its
 |---|---|---|---|---|
 | `spin_lock`, separators relaxed (also `spin lock`, `spin-lock`, `spinlock`) | 9 552 | **9 552**, 23 ms | 6 577 — not with this analyzer: its trigrams carry the underscore | 6 601 — relaxed is the only mode it has: the separator never enters its index |
 | `spinlokc`, two edits, across the token boundary | 10 034 | **10 034**, 148 ms | 3 549 — fuzziness compares whole terms | 6 557 — same |
-| `spin_lock_[a-z]+`, a regex | 5 510 | **5 510**, 219 ms | 5 440 (wildcard field, 70 short), 480 ms | 0 — terms are already cut |
+| `spin_lock_[a-z]+`, a regex, case folded | 5 510 | **5 510**, 219 ms | 5 510 on the wildcard field, 1 ms warm — as `[a-zA-Z]+`: Lucene's `case_insensitive` folds a pattern's literals, not its character classes | 0 — terms are already cut |
 | `de`, two characters | 93 009 | **93 009**, 7.7 M spans, 561 ms | 0, silently | 0, silently |
 | `retur -ENOMEM`, a fuzzy phrase | 14 449 | **14 449**, 30 ms | 14 446 (`span_near`), 24 ms — it does this well | — |
 | **where it matched**: `mutex_lock`, 5 145 documents | 20 797 spans | **all 20 797, 15 ms** | `highlight` on the top 200: 179 ms | verifying 5 145 stored texts: 96 ms |
@@ -126,6 +137,10 @@ const index = await lucivy.create('/my-index', {
     // derived_in_ram: true — the three derived sidecars of each segment
     // (about a third of the index) are rebuilt in memory when the index
     // opens instead of written to OPFS. Same answers. Off by default.
+    // positions: false (4.1) — documents and frequencies instead of
+    // positions, no position sidecar: about 40 % less OPFS; every match is
+    // verified on the stored text, same answers. Text fields must be stored
+    // (the default); excludes derived_in_ram. On by default.
 });
 
 // Open an existing index from OPFS

package/js/lucivy.d.ts

@@ -327,6 +327,14 @@ export interface IndexConfig {
    * Off by default; fixed at creation.
    */
   derived_in_ram?: boolean;
+  /**
+   * `false` keeps each token's documents and frequencies instead of its
+   * positions and writes no position sidecar (37 % smaller on 10 000 kernel
+   * files); every match is verified on the stored text, so every text field
+   * must be stored. Same documents, spans and scores. Excludes
+   * `derived_in_ram`. On by default; fixed at creation.
+   */
+  positions?: boolean;
   /**
    * Shared dictionary only. A commit returns before the shard's new texts
    * are merged into the dictionary (a background task does it); a search

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy-wasm",
-  "version": "4.0.2",
+  "version": "4.2.0",
   "type": "module",
   "description": "Fast BM25 full-text search for browsers — WASM build of Lucivy with threading, OPFS persistence, and LUCE snapshot support",
   "license": "MIT",