lucivy 4.2.0 → 4.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +36 -2
- package/package.json +6 -6
package/README.md
CHANGED
|
@@ -1,9 +1,43 @@
|
|
|
1
|
-
# lucivy 4.
|
|
1
|
+
# lucivy 4.3.0
|
|
2
2
|
|
|
3
3
|
**One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. From Node.js. Runs in your process, in your transaction (bring your own storage), and the same engine runs in the browser. Powered by Rust via napi-rs, MIT.
|
|
4
4
|
|
|
5
5
|
[**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
|
|
6
6
|
|
|
7
|
+
### What's new in 4.3
|
|
8
|
+
|
|
9
|
+
- **Indexing 48 → 35 s on the same index.** The whole kernel (Linux 7.2,
|
|
10
|
+
101 141 files, 941 MB) on a 24-core machine, the same 308 segments, the
|
|
11
|
+
same ids minted, the same answers. Four pieces, none of them an option: a
|
|
12
|
+
derived group index next to every dictionary generation (`.pidx`, 2.5 % of
|
|
13
|
+
the `.sfx`, rebuilt in RAM when an older index lacks it) that reaches the
|
|
14
|
+
right parent group by binary search instead of reading every header
|
|
15
|
+
before it; the commit no longer stops the world to forget its pending
|
|
16
|
+
texts (they live by commit epoch and are dropped whole — 6.7 s of serial
|
|
17
|
+
path gone); the suffix collector allocates per token instead of per
|
|
18
|
+
occurrence; and a text minted since the last fold is found in the pending
|
|
19
|
+
table before every FST part is walked (FST walks 68 → 38 s of CPU).
|
|
20
|
+
- **Two exactness fixes, present since 4.0.0.** A segment holding few of a
|
|
21
|
+
candidate list's ids kept one item per id and lost the repeated
|
|
22
|
+
occurrences of a needle inside one token (`0xdedede00` for `de`) — whether
|
|
23
|
+
a segment took that path depended on its size, so the same corpus was
|
|
24
|
+
exact at 16 writer threads and lost 3 spans of 7.9 M at 24. And relaxed
|
|
25
|
+
separators could report a match ending inside a multi-byte character
|
|
26
|
+
(`D\n,,“underscan`): a word's content overlap was taken from the word
|
|
27
|
+
after the next when the next one began with a three-byte character, and
|
|
28
|
+
an anchored word candidate was accepted at any suffix. Both fixed on the
|
|
29
|
+
query side (an existing index answers exactly with the new reader), the
|
|
30
|
+
second at indexing too; tests red on the old code.
|
|
31
|
+
- **The segment's dictionary pair is named for what it is:**
|
|
32
|
+
`<uuid>.<field>.minted.termtexts` and `.minted.sfx` (the texts a segment
|
|
33
|
+
minted first and the FST over them, folded into the next generation),
|
|
34
|
+
instead of `.newtexts` / `.newsfx`. Every reader still opens the old names.
|
|
35
|
+
- **Compatibility**: 4.2 opens a 4.3 index and 4.3 opens 3.0.x to 4.2 indexes
|
|
36
|
+
as before; the only thing 4.2 cannot do is fold the pairs a 4.3 writer left
|
|
37
|
+
pending after a crash (a cleanly closed index has none). Every binding
|
|
38
|
+
already had `update` (delete by `_node_id` and add) — editing one file
|
|
39
|
+
costs that file, not a rebuild.
|
|
40
|
+
|
|
7
41
|
### What's new in 4.2
|
|
8
42
|
|
|
9
43
|
- **Indexing twice as fast, the same index** — the whole Linux kernel on a 24-core machine: 97 → 48 s, same files, same answers, same query times. The writer's threads follow the cores up to 16 with unchanged per-thread budgets (8 cores or fewer: no change; `LUCIVY_WRITER_THREADS`, `LUCIVY_WRITER_HEAP`, `LUCIVY_SFX_HEAP` still override), a lock-free cache of the dictionary ids already found (every hit verified on the stored texts), background folds kept in the background, and a three-times-faster dictionary compaction.
|
|
@@ -56,7 +90,7 @@ Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `merg
|
|
|
56
90
|
## Install
|
|
57
91
|
|
|
58
92
|
```bash
|
|
59
|
-
npm install lucivy # 4.
|
|
93
|
+
npm install lucivy # 4.3.0
|
|
60
94
|
```
|
|
61
95
|
|
|
62
96
|
## Quick start
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "lucivy",
|
|
3
|
-
"version": "4.
|
|
3
|
+
"version": "4.3.0",
|
|
4
4
|
"description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
|
|
5
5
|
"main": "index.js",
|
|
6
6
|
"types": "index.d.ts",
|
|
@@ -39,10 +39,10 @@
|
|
|
39
39
|
"build:debug": "napi build"
|
|
40
40
|
},
|
|
41
41
|
"optionalDependencies": {
|
|
42
|
-
"lucivy-linux-x64-gnu": "4.
|
|
43
|
-
"lucivy-linux-arm64-gnu": "4.
|
|
44
|
-
"lucivy-darwin-x64": "4.
|
|
45
|
-
"lucivy-darwin-arm64": "4.
|
|
46
|
-
"lucivy-windows-x64": "4.
|
|
42
|
+
"lucivy-linux-x64-gnu": "4.3.0",
|
|
43
|
+
"lucivy-linux-arm64-gnu": "4.3.0",
|
|
44
|
+
"lucivy-darwin-x64": "4.3.0",
|
|
45
|
+
"lucivy-darwin-arm64": "4.3.0",
|
|
46
|
+
"lucivy-windows-x64": "4.3.0"
|
|
47
47
|
}
|
|
48
48
|
}
|