lucivy 3.0.8 → 4.0.0

package/README.md

@@ -1,10 +1,18 @@
-# lucivy 3.0.8
+# lucivy 4.0.0
 
-Fast BM25 full-text search for Node.js — with substring matching, fuzzy search, regex, and highlights. Powered by Rust via napi-rs.
+Full-text search for code and technical text, as a library — from Node.js. Substrings, fuzzy and regex **across token boundaries**, BM25, exact byte spans, every answer checked against the files. Runs in your process, in your transaction (bring your own storage), and the same engine runs in the browser. Powered by Rust via napi-rs, MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
 
-### What's new in 3.0.0
+### What's new in 4.0.0
+
+- **The index is 3.7× smaller** — the whole Linux kernel: 18 057 MB in 3.0.8, 4 938 MB in 4.0, 3 344 MB with `derivedInRam`; same answers, same spans, checked against the files ([the comparison with Elasticsearch and tantivy](../../docs/compare-engines-2026-09-05.md))
+- **The shared dictionary is the default** (`Index.create(path, fields, shards, false)` / `BlobIndex` option `sharedDictionary: false` keeps a suffix FST per segment: indexing ×1.5 faster, an index 23 % bigger) — one dictionary of token texts per shard instead of one per segment: 23 % smaller on the kernel, cold queries ×0.8-1.6, same answers; off by default, fixed at creation
+- **`Index.create(path, fields, shards, sharedDictionary, derivedInRam, dictionaryWait)`** and **`BlobIndex` option `dictionaryWait`** — shared dictionary only: a commit returns before the shard's new texts are merged into the dictionary (a background task does it) and, by default, a search waits for that merge so that its cost never depends on when it runs; `false` searches at once over the not-yet-merged parts. Indexing with the dictionary costs ×1.5 (the kernel: 107 s against 56)
+- **`Index.create(path, fields, shards, sharedDictionary, derivedInRam)`** and **`BlobIndex` option `derivedInRam`** — the three derived sidecars of each segment rebuilt byte for byte when the index opens instead of written: about a third less on disk, the open pays (the kernel: 2 s), never a query; off by default
+- **Compatibility contract** — 4.0 opens a 3.0.x index and returns what 3.0.x returned (checked against a fixture the published 3.0.8 wheel built); 3.0.x does not open a 4.0 index; the first commit converts for good
+
+### What 3.0.0 brought
 
 - **SFX v3 segment format** — per-field suffix FST files, the default for every new index; v2 indexes still open
 - **`parse` query type** — boolean syntax (`AND` / `OR` / `NOT`, quotes, `+` / `-`, parentheses) on top of substring matching
@@ -18,7 +26,7 @@ Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `merg
 ## Install
 
 ```bash
-npm install lucivy@3.0.0
+npm install lucivy   # 4.0.0
 ```
 
 ## Quick start
@@ -55,6 +63,18 @@ const index = Index.create('/tmp/my_index', [
 // Sharded (4 shards)
 const sharded = Index.create('/tmp/sharded', [...], 4);
 
+// Smaller index: one dictionary per shard instead of one per segment.
+// About 20 % less disk and RAM; queries slightly slower at cold cache
+// (roughly x1.2 to x1.6 on exact queries, fuzzy ones faster); same answers.
+// Fixed at creation. (BlobIndex.create: `{ sharedDictionary: true }` in options.)
+const compact = Index.create('/tmp/compact', [...], 1, true);
+
+// Smaller still on disk: the three derived sidecars of each segment (about
+// a third of the index) are rebuilt in RAM, byte for byte, when the index is
+// opened, instead of being written. Same answers; opening pays the rebuild,
+// never a query. (BlobIndex.create: `{ derivedInRam: true }`.)
+const lean = Index.create('/tmp/lean', [...], 1, true, true);
+
 // Open existing
 const index2 = Index.open('/tmp/my_index');
 ```

package/index.d.ts

@@ -83,6 +83,21 @@ export interface BlobIndexOptions {
   lazy?: boolean
   /** `create()` only: number of shards (default 1). */
   shards?: number
+  /**
+   * `create()` only: one dictionary per shard instead of one per segment
+   * (about 20 % smaller, slightly slower queries) — see `Index.create()`.
+   */
+  sharedDictionary?: boolean
+  /**
+   * `create()` only: the derived sidecars rebuilt in RAM at open instead
+   * of written (about a third smaller on disk) — see `Index.create()`.
+   */
+  derivedInRam?: boolean
+  /**
+   * `create()` only, shared dictionary: a search waits for the background
+   * merge of the last commit's texts (default `true`) — see `Index.create()`.
+   */
+  dictionaryWait?: boolean
 }
 export declare class Index {
   /**
@@ -92,8 +107,26 @@ export declare class Index {
    * @param fields - Field definitions: `[{name: "body", type: "text", stored: true}]`.
    *   Types: `"text"` (full-text), `"u64"`, `"i64"`, `"f64"`, `"bool"`, `"date"`.
    * @param shards - Number of shards (default 1). More shards = faster search on large datasets.
-   */
-  static create(path: string, fields: Array<FieldDef>, shards?: number | undefined | null): Index
+   * @param sharedDictionary - Store each distinct token text once per shard
+   *   instead of once per segment: the index is about 20 % smaller on disk
+   *   and in RAM, queries are slightly slower at cold cache (roughly x1.2
+   *   to x1.6 on exact queries, fuzzy ones faster) and a commit also writes
+   *   the shard's new texts. Same answers either way. On by default since
+   *   4.0.0; `false` keeps a suffix FST per segment (indexing ×1.5 faster,
+   *   an index 23 % bigger). Fixed at creation.
+   * @param derivedInRam - Do not write the three derived sidecars of each
+   *   segment (`.posmap`, `.word_pos_map`, `.sibling_v3`, about a third
+   *   of the index on disk); they are rebuilt in RAM, byte for byte, when
+   *   the index is opened or reloaded. Same answers; opening pays the
+   *   rebuild (never a query) and the rebuilt structures stay resident.
+   *   Off by default; fixed at creation.
+   * @param dictionaryWait - Shared dictionary only. A commit returns before
+   *   the shard's new texts are merged into the dictionary (a background
+   *   task does it); a search waits for that merge, so that its cost never
+   *   depends on when it runs. `false` searches at once over the
+   *   not-yet-merged parts. On by default; fixed at creation.
+   */
+  static create(path: string, fields: Array<FieldDef>, shards?: number | undefined | null, sharedDictionary?: boolean | undefined | null, derivedInRam?: boolean | undefined | null, dictionaryWait?: boolean | undefined | null): Index
   /**
    * Open an existing index at the given path.
    *
@@ -404,7 +437,7 @@ export declare class BlobIndex {
    * @param store - Object implementing the store protocol (`load`, `save`, `delete`, `exists`, `list`, optional `blobLen` / `loadRange`).
    * @param indexName - Name of the index inside the store.
    * @param fields - Field definitions, as for `Index.create()`.
-   * @param options - `{cacheDir?, lazy?, shards?}`.
+   * @param options - `{cacheDir?, lazy?, shards?, sharedDictionary?, derivedInRam?, dictionaryWait?}`.
    */
   static create(store: BlobStoreCallbacks, indexName: string, fields: Array<FieldDef>, options?: BlobIndexOptions | undefined | null): Promise<BlobIndex>
   /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy",
-  "version": "3.0.8",
+  "version": "4.0.0",
   "description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
   "main": "index.js",
   "types": "index.d.ts",
@@ -39,10 +39,10 @@
     "build:debug": "napi build"
   },
   "optionalDependencies": {
-    "lucivy-linux-x64-gnu": "3.0.8",
-    "lucivy-linux-arm64-gnu": "3.0.8",
-    "lucivy-darwin-x64": "3.0.8",
-    "lucivy-darwin-arm64": "3.0.8",
-    "lucivy-windows-x64": "3.0.8"
+    "lucivy-linux-x64-gnu": "4.0.0",
+    "lucivy-linux-arm64-gnu": "4.0.0",
+    "lucivy-darwin-x64": "4.0.0",
+    "lucivy-darwin-arm64": "4.0.0",
+    "lucivy-windows-x64": "4.0.0"
   }
 }