lucivy 4.1.0 → 4.2.0

package/README.md

@@ -1,9 +1,16 @@
-# lucivy 4.1.0
+# lucivy 4.2.0
 
 **One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. From Node.js. Runs in your process, in your transaction (bring your own storage), and the same engine runs in the browser. Powered by Rust via napi-rs, MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
 
+### What's new in 4.2
+
+- **Indexing twice as fast, the same index** — the whole Linux kernel on a 24-core machine: 97 → 48 s, same files, same answers, same query times. The writer's threads follow the cores up to 16 with unchanged per-thread budgets (8 cores or fewer: no change; `LUCIVY_WRITER_THREADS`, `LUCIVY_WRITER_HEAP`, `LUCIVY_SFX_HEAP` still override), a lock-free cache of the dictionary ids already found (every hit verified on the stored texts), background folds kept in the background, and a three-times-faster dictionary compaction.
+- **`search(query, { fields: false })` no longer reads any document**: the id comes from the fast field; with `fields: true` the documents are read in parallel past 64 hits (5 202 documents in 15 ms instead of 114, a top-200 in 0.6 ms).
+- **Spans only when `highlights: true`** — they were built for every search; `de` on the kernel 574 → 442 ms, 337 → 190 ms without positions.
+- **Compatibility**: nothing changes on disk; 4.1 opens a 4.2 index.
+
 ### What's new in 4.1
 
 - **`Index.create(path, fields, { positions: false })` — an index half the size.** The postings keep each token's documents and frequencies instead of its positions, and no position sidecar is written: the whole Linux kernel (101 141 files, 941 MB of text) 5 289 → 2 598 MB, ×2.8 its text instead of ×5.6; 10 000 files −37 %. Nothing is rebuilt when the index opens, and indexing is a little faster (109 → 101 s). Every match is found as candidate documents and verified on the stored text with the ground truth's own definitions: same documents, same spans, same scores, the ground-truth panel 10/10. Some queries pay for it (whole kernel, file cache warm): literal substrings 11-19 → 27-56 ms (the documents found are re-read), a one-edit fuzzy 50 → 200 ms; others get faster — a regex 237 → 22 ms, a two-character needle 630-700 → 330 ms. Every text field must be stored (the default); excludes `derivedInRam`; fixed at creation, also as the `BlobIndex` option `positions`. An index created this way cannot be searched by 4.0.x (it opens, and every search fails: `sfxpost: invalid V2 format`)
@@ -49,7 +56,7 @@ Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `merg
 ## Install
 
 ```bash
-npm install lucivy   # 4.1.0
+npm install lucivy   # 4.2.0
 ```
 
 ## Quick start

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy",
-  "version": "4.1.0",
+  "version": "4.2.0",
   "description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
   "main": "index.js",
   "types": "index.d.ts",
@@ -39,10 +39,10 @@
     "build:debug": "napi build"
   },
   "optionalDependencies": {
-    "lucivy-linux-x64-gnu": "4.1.0",
-    "lucivy-linux-arm64-gnu": "4.1.0",
-    "lucivy-darwin-x64": "4.1.0",
-    "lucivy-darwin-arm64": "4.1.0",
-    "lucivy-windows-x64": "4.1.0"
+    "lucivy-linux-x64-gnu": "4.2.0",
+    "lucivy-linux-arm64-gnu": "4.2.0",
+    "lucivy-darwin-x64": "4.2.0",
+    "lucivy-darwin-arm64": "4.2.0",
+    "lucivy-windows-x64": "4.2.0"
   }
 }