lucivy 4.0.2 → 4.1.0

package/README.md

@@ -1,9 +1,15 @@
-# lucivy 4.0.2
+# lucivy 4.1.0
 
 **One index answers every question, and every answer is checked.** The default index answers exact substrings, matches across separators, typos across token boundaries, regular expressions and two-character needles — with BM25 and the exact bytes of every match — and nothing to configure per question; the ground-truth harness compares every answer to a scan of the files. From Node.js. Runs in your process, in your transaction (bring your own storage), and the same engine runs in the browser. Powered by Rust via napi-rs, MIT.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
 
+### What's new in 4.1
+
+- **`Index.create(path, fields, { positions: false })` — an index half the size.** The postings keep each token's documents and frequencies instead of its positions, and no position sidecar is written: the whole Linux kernel (101 141 files, 941 MB of text) 5 289 → 2 598 MB, ×2.8 its text instead of ×5.6; 10 000 files −37 %. Nothing is rebuilt when the index opens, and indexing is a little faster (109 → 101 s). Every match is found as candidate documents and verified on the stored text with the ground truth's own definitions: same documents, same spans, same scores, the ground-truth panel 10/10. Some queries pay for it (whole kernel, file cache warm): literal substrings 11-19 → 27-56 ms (the documents found are re-read), a one-edit fuzzy 50 → 200 ms; others get faster — a regex 237 → 22 ms, a two-character needle 630-700 → 330 ms. Every text field must be stored (the default); excludes `derivedInRam`; fixed at creation, also as the `BlobIndex` option `positions`. An index created this way cannot be searched by 4.0.x (it opens, and every search fails: `sfxpost: invalid V2 format`)
+- **The options of `Index.create()` as one object** — `{ shards, sharedDictionary, derivedInRam, dictionaryWait, positions }` (`IndexOptions`, typed) in place of the arguments after `fields`; the positional form still works, giving both is refused
+- **Fix: one occurrence, one span.** In relaxed mode a needle that ends a word cut into chunks — `lock` in `superblock` — came back twice, and counted twice in the score (542 duplicated spans for `lock` over 10 000 kernel files, since the v3 engine, 4.0.2 included); the ground-truth harness now counts duplicates too
+
 ### What's new in 4.0.0
 
 - **The index is 3.7× smaller** — the whole Linux kernel: 18 057 MB in 3.0.8, 4 938 MB in 4.0, 3 344 MB with `derivedInRam`; same answers, same spans, checked against the files ([the comparison with Elasticsearch and tantivy](https://github.com/L-Defraiteur/lucivy/blob/main/docs/compare-engines-2026-09-05.md))
@@ -20,7 +26,7 @@ Same 93 983 Linux kernel files, 857 MB of text. Each engine is configured at its
 |---|---|---|---|---|
 | `spin_lock`, separators relaxed (also `spin lock`, `spin-lock`, `spinlock`) | 9 552 | **9 552**, 23 ms | 6 577 — not with this analyzer: its trigrams carry the underscore | 6 601 — relaxed is the only mode it has: the separator never enters its index |
 | `spinlokc`, two edits, across the token boundary | 10 034 | **10 034**, 148 ms | 3 549 — fuzziness compares whole terms | 6 557 — same |
-| `spin_lock_[a-z]+`, a regex | 5 510 | **5 510**, 219 ms | 5 440 (wildcard field, 70 short), 480 ms | 0 — terms are already cut |
+| `spin_lock_[a-z]+`, a regex, case folded | 5 510 | **5 510**, 219 ms | 5 510 on the wildcard field, 1 ms warm — as `[a-zA-Z]+`: Lucene's `case_insensitive` folds a pattern's literals, not its character classes | 0 — terms are already cut |
 | `de`, two characters | 93 009 | **93 009**, 7.7 M spans, 561 ms | 0, silently | 0, silently |
 | `retur -ENOMEM`, a fuzzy phrase | 14 449 | **14 449**, 30 ms | 14 446 (`span_near`), 24 ms — it does this well | — |
 | **where it matched**: `mutex_lock`, 5 145 documents | 20 797 spans | **all 20 797, 15 ms** | `highlight` on the top 200: 179 ms | verifying 5 145 stored texts: 96 ms |
@@ -43,7 +49,7 @@ Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `merg
 ## Install
 
 ```bash
-npm install lucivy   # 4.0.2
+npm install lucivy   # 4.1.0
 ```
 
 ## Quick start
@@ -92,6 +98,18 @@ const compact = Index.create('/tmp/compact', [...], 1, true);
 // never a query. (BlobIndex.create: `{ derivedInRam: true }`.)
 const lean = Index.create('/tmp/lean', [...], 1, true, true);
 
+// Half the size (4.1): the postings keep documents and frequencies, not
+// positions, and no position sidecar is written. Every match is verified on
+// the stored text: same documents, spans and scores; literal queries pay a
+// re-read of the documents found (tens of milliseconds on the whole
+// kernel), regexes get faster. Text fields must be stored (the default);
+// excludes derivedInRam. Fixed at creation. (BlobIndex: `{ positions: false }`.)
+const half = Index.create('/tmp/half', [...], { positions: false });
+
+// Every option can be given this way, in place of the positional arguments:
+// { shards, sharedDictionary, derivedInRam, dictionaryWait, positions }
+const wide = Index.create('/tmp/wide', [...], { shards: 4, positions: false });
+
 // Open existing
 const index2 = Index.open('/tmp/my_index');
 ```

package/index.d.ts

@@ -39,6 +39,28 @@ export interface SearchOptions {
  * @returns JSON string of merged `ExportableStats` ready for `searchWithGlobalStats()`.
  */
 export declare function mergeStats(statsList: Array<string>): string
+/**
+ * The options of `Index.create()` as one object, in place of its arguments
+ * after `fields`: `Index.create(path, fields, { positions: false })`.
+ */
+export interface IndexOptions {
+  /** Number of shards (default 1). */
+  shards?: number
+  /** One dictionary per shard instead of one per segment (default `true`). */
+  sharedDictionary?: boolean
+  /** The derived sidecars rebuilt in RAM at open instead of written. */
+  derivedInRam?: boolean
+  /**
+   * Shared dictionary: a search waits for the background merge of the
+   * last commit's texts (default `true`).
+   */
+  dictionaryWait?: boolean
+  /**
+   * `false`: documents and frequencies instead of positions, about half
+   * the index; every match verified on the stored text.
+   */
+  positions?: boolean
+}
 /**
  * The object a JavaScript program hands to `BlobIndex.create()` /
  * `BlobIndex.open()`. Methods are called with the object as `this`. Each may
@@ -98,6 +120,11 @@ export interface BlobIndexOptions {
    * merge of the last commit's texts (default `true`) — see `Index.create()`.
    */
   dictionaryWait?: boolean
+  /**
+   * `create()` only: `false` keeps documents and frequencies instead of
+   * positions, every match verified on the stored text — see `Index.create()`.
+   */
+  positions?: boolean
 }
 export declare class Index {
   /**
@@ -107,6 +134,8 @@ export declare class Index {
    * @param fields - Field definitions: `[{name: "body", type: "text", stored: true}]`.
    *   Types: `"text"` (full-text), `"u64"`, `"i64"`, `"f64"`, `"bool"`, `"date"`.
    * @param shards - Number of shards (default 1). More shards = faster search on large datasets.
+   *   Or an options object in place of this argument and all that follow:
+   *   `Index.create(path, fields, { shards: 4, positions: false })`.
    * @param sharedDictionary - Store each distinct token text once per shard
    *   instead of once per segment: the index is about 20 % smaller on disk
    *   and in RAM, queries are slightly slower at cold cache (roughly x1.2
@@ -125,8 +154,16 @@ export declare class Index {
    *   task does it); a search waits for that merge, so that its cost never
    *   depends on when it runs. `false` searches at once over the
    *   not-yet-merged parts. On by default; fixed at creation.
+   * @param positions - `false` keeps each token's documents and frequencies
+   *   instead of its positions, and writes no position sidecar (`.posmap`,
+   *   `.word_pos_map`, `.sibling_v3`): 37 % smaller on 10 000 kernel files.
+   *   Every match is then verified on the stored text — same documents,
+   *   spans and scores; literal queries pay a re-read of the documents
+   *   found (tens of milliseconds on the whole kernel), regexes get
+   *   faster. Every text field must be stored (the default); excludes
+   *   `derivedInRam`. On by default; fixed at creation.
    */
-  static create(path: string, fields: Array<FieldDef>, shards?: number | undefined | null, sharedDictionary?: boolean | undefined | null, derivedInRam?: boolean | undefined | null, dictionaryWait?: boolean | undefined | null): Index
+  static create(path: string, fields: Array<FieldDef>, shards?: number | IndexOptions | undefined | null, sharedDictionary?: boolean | undefined | null, derivedInRam?: boolean | undefined | null, dictionaryWait?: boolean | undefined | null, positions?: boolean | undefined | null): Index
   /**
    * Open an existing index at the given path.
    *
@@ -437,7 +474,7 @@ export declare class BlobIndex {
    * @param store - Object implementing the store protocol (`load`, `save`, `delete`, `exists`, `list`, optional `blobLen` / `loadRange`).
    * @param indexName - Name of the index inside the store.
    * @param fields - Field definitions, as for `Index.create()`.
-   * @param options - `{cacheDir?, lazy?, shards?, sharedDictionary?, derivedInRam?, dictionaryWait?}`.
+   * @param options - `{cacheDir?, lazy?, shards?, sharedDictionary?, derivedInRam?, dictionaryWait?, positions?}`.
    */
   static create(store: BlobStoreCallbacks, indexName: string, fields: Array<FieldDef>, options?: BlobIndexOptions | undefined | null): Promise<BlobIndex>
   /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy",
-  "version": "4.0.2",
+  "version": "4.1.0",
   "description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
   "main": "index.js",
   "types": "index.d.ts",
@@ -39,10 +39,10 @@
     "build:debug": "napi build"
   },
   "optionalDependencies": {
-    "lucivy-linux-x64-gnu": "4.0.2",
-    "lucivy-linux-arm64-gnu": "4.0.2",
-    "lucivy-darwin-x64": "4.0.2",
-    "lucivy-darwin-arm64": "4.0.2",
-    "lucivy-windows-x64": "4.0.2"
+    "lucivy-linux-x64-gnu": "4.1.0",
+    "lucivy-linux-arm64-gnu": "4.1.0",
+    "lucivy-darwin-x64": "4.1.0",
+    "lucivy-darwin-arm64": "4.1.0",
+    "lucivy-windows-x64": "4.1.0"
   }
 }