lucivy 4.2.0 → 4.3.1

package/README.md

@@ -1,9 +1,52 @@
-# lucivy 4.2.0
+# lucivy 4.3.1
 
 **One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. From Node.js. Runs in your process, in your transaction (bring your own storage), and the same engine runs in the browser. Powered by Rust via napi-rs, MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
 
+### What's new in 4.3.1
+
+A patch release that fixes concurrency between searches and writes. The index format is unchanged.
+
+- **Searches during commits no longer lose writes.** A BM25 statistic read across two states of the index made a search panic. The panic took down the shard serving it, and documents disappeared behind commits that returned `Ok`, or later commits froze. The statistic is now bounded, and a search that panics answers an error instead of killing the shard.
+- **One commit in flight per shard.** Writes, commits and drains that arrive meanwhile wait, and run when it ends, in their order.
+- **Closing commits, always.** Dropping an index without `close()` now commits what was still queued. Writes to a shard without a writer are reported by the next commit instead of being dropped silently.
+- The browser playground starts its engine only on a cross-origin isolated page. A hard reload no longer leaves it dead.
+
+### What's new in 4.3
+
+- **Indexing 48 → 35 s on the same index.** The whole kernel (Linux 7.2,
+  101 141 files, 941 MB) on a 24-core machine, the same 308 segments, the
+  same ids minted, the same answers. Four pieces, none of them an option: a
+  derived group index next to every dictionary generation (`.pidx`, 2.5 % of
+  the `.sfx`, rebuilt in RAM when an older index lacks it) that reaches the
+  right parent group by binary search instead of reading every header
+  before it; the commit no longer stops the world to forget its pending
+  texts (they live by commit epoch and are dropped whole — 6.7 s of serial
+  path gone); the suffix collector allocates per token instead of per
+  occurrence; and a text minted since the last fold is found in the pending
+  table before every FST part is walked (FST walks 68 → 38 s of CPU).
+- **Two exactness fixes, present since 4.0.0.** A segment holding few of a
+  candidate list's ids kept one item per id and lost the repeated
+  occurrences of a needle inside one token (`0xdedede00` for `de`) — whether
+  a segment took that path depended on its size, so the same corpus was
+  exact at 16 writer threads and lost 3 spans of 7.9 M at 24. And relaxed
+  separators could report a match ending inside a multi-byte character
+  (`D\n,,“underscan`): a word's content overlap was taken from the word
+  after the next when the next one began with a three-byte character, and
+  an anchored word candidate was accepted at any suffix. Both fixed on the
+  query side (an existing index answers exactly with the new reader), the
+  second at indexing too; tests red on the old code.
+- **The segment's dictionary pair is named for what it is:**
+  `<uuid>.<field>.minted.termtexts` and `.minted.sfx` (the texts a segment
+  minted first and the FST over them, folded into the next generation),
+  instead of `.newtexts` / `.newsfx`. Every reader still opens the old names.
+- **Compatibility**: 4.2 opens a 4.3 index and 4.3 opens 3.0.x to 4.2 indexes
+  as before; the only thing 4.2 cannot do is fold the pairs a 4.3 writer left
+  pending after a crash (a cleanly closed index has none). Every binding
+  already had `update` (delete by `_node_id` and add) — editing one file
+  costs that file, not a rebuild.
+
 ### What's new in 4.2
 
 - **Indexing twice as fast, the same index** — the whole Linux kernel on a 24-core machine: 97 → 48 s, same files, same answers, same query times. The writer's threads follow the cores up to 16 with unchanged per-thread budgets (8 cores or fewer: no change; `LUCIVY_WRITER_THREADS`, `LUCIVY_WRITER_HEAP`, `LUCIVY_SFX_HEAP` still override), a lock-free cache of the dictionary ids already found (every hit verified on the stored texts), background folds kept in the background, and a three-times-faster dictionary compaction.
@@ -56,7 +99,7 @@ Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `merg
 ## Install
 
 ```bash
-npm install lucivy   # 4.2.0
+npm install lucivy   # 4.3.1
 ```
 
 ## Quick start

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy",
-  "version": "4.2.0",
+  "version": "4.3.1",
   "description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
   "main": "index.js",
   "types": "index.d.ts",
@@ -39,10 +39,10 @@
     "build:debug": "napi build"
   },
   "optionalDependencies": {
-    "lucivy-linux-x64-gnu": "4.2.0",
-    "lucivy-linux-arm64-gnu": "4.2.0",
-    "lucivy-darwin-x64": "4.2.0",
-    "lucivy-darwin-arm64": "4.2.0",
-    "lucivy-windows-x64": "4.2.0"
+    "lucivy-linux-x64-gnu": "4.3.1",
+    "lucivy-linux-arm64-gnu": "4.3.1",
+    "lucivy-darwin-x64": "4.3.1",
+    "lucivy-darwin-arm64": "4.3.1",
+    "lucivy-windows-x64": "4.3.1"
   }
 }