lucivy 2.0.2 → 3.0.1

package/README.md

@@ -1,21 +1,24 @@
-# lucivy v2
+# lucivy 3.0.1
 
 Fast BM25 full-text search for Node.js — with substring matching, fuzzy search, regex, and highlights. Powered by Rust via napi-rs.
 
 [**Try the live playground**](https://l-defraiteur.github.io/lucivy/) — runs entirely in your browser via WASM.
 
-### What's new in v2
+### What's new in 3.0.0
 
-- **SFX-only engine** — all queries route through the Suffix FST, no legacy code paths
-- **Distributed search** — `exportStats` / `mergeStats` / `searchWithGlobalStats`
-- **Incremental sync** — LUCIDS sharded delta export/apply
-- **Correct BM25 cross-shard** — identical scores whether 1 shard or 4
-- **5 bindings** — Python, Node.js, C++, WASM, Rust
+- **SFX v3 segment format** — per-field suffix FST files, the default for every new index; v2 indexes still open
+- **`parse` query type** — boolean syntax (`AND` / `OR` / `NOT`, quotes, `+` / `-`, parentheses) on top of substring matching
+- **`queryWarnings()`** — what the engine will really search, and where it falls back to a scan, before running the query
+- **Snapshots served in place** — `Index.openSnapshot()` reads a LUCE blob without extracting it
+- **Maintenance** — `compact()`, `waitMergesQuiet()`, `indexBytes()`, `dropIndex()`
+- **Bring your own storage** — `BlobIndex` keeps an index in the transactional store of your choice through a plain object of callbacks; asynchronous API
+
+Still there from 2.x: SFX-only engine, distributed search (`exportStats` / `mergeStats` / `searchWithGlobalStats`), incremental LUCIDS delta sync, BM25 scores identical across 1 or N shards, bindings for Python, Node.js, C++, WASM and Rust.
 
 ## Install
 
 ```bash
-npm install lucivy
+npm install lucivy@3.0.0
 ```
 
 ## Quick start
@@ -100,6 +103,11 @@ index.search({ type: 'contains', field: 'body', value: 'program' });
 // Fuzzy substring (Levenshtein distance)
 index.search({ type: 'contains', field: 'body', value: 'mutx', distance: 1 });
 
+// Fuzzy with Jaro-Winkler instead of Levenshtein: candidates come from the
+// trigram pigeonhole at `distance` (default 2), Jaro-Winkler decides, and
+// hits are tiered by similarity (a typo at the end ranks above one at the start)
+index.search({ type: 'fuzzy', field: 'body', value: 'kmalloc', fuzzy_metric: 'jaro_winkler', min_similarity: 0.9 });
+
 // Regex substring — cross-token regex matching
 index.search({ type: 'contains', field: 'body', value: 'lock.*mutex', regex: true });
 
@@ -141,6 +149,45 @@ index.search({
 });
 ```
 
+#### parse — boolean syntax in one string
+
+A plain value (no operators) is an OR of substring `contains` queries, one per word
+and per field. With boolean syntax the string is compiled into a `boolean` query of
+`contains` clauses: `AND` / `OR` / `NOT`, `"quoted phrases"`, `+required` / `-excluded`,
+and parentheses. Precedence is `NOT` > `AND` > `OR`; words side by side are OR'd.
+Highlights work in both cases. `fields` takes several fields at once.
+
+```javascript
+// Plain value: OR of contains per word x field
+index.search({ type: 'parse', field: 'body', value: 'kmalloc spin_lock' });
+
+// Boolean syntax
+index.search({ type: 'parse', field: 'body', value: 'kmalloc AND NOT vfree' });
+index.search({ type: 'parse', field: 'body', value: '"spin_lock" -vfree' });
+index.search({ type: 'parse', fields: ['title', 'body'], value: '(mutex OR spinlock) AND init' });
+```
+
+#### queryWarnings — know what will run before running it
+
+Returns plain-text warnings for a query, without executing it: separators ignored
+in relaxed mode, a fuzzy distance too loose for the query length, a regex without a
+usable literal (full scan), segments written by the legacy indexer. Empty array when
+nothing applies. For `parse`, the warnings say which of the two modes the value
+selected.
+
+```javascript
+index.queryWarnings({ type: 'contains', field: 'body', value: 'kmalloc' });
+// []
+index.queryWarnings({ type: 'contains', field: 'body', value: '__init' });
+// ['separators are ignored (strict_separators=false): "__init" is searched as "init"']
+index.queryWarnings({ type: 'regex', field: 'body', value: '[0-9]{8}' });
+// ['"[0-9]{8}" requires no literal the index can look up: every document is scanned whole (full scan, ...)']
+index.queryWarnings({ type: 'fuzzy', field: 'body', value: 'init' });
+// ['distance 1 on "init" (4 chars) rewrites a quarter of the query or more: unrelated short words will match (...)']
+index.queryWarnings({ type: 'parse', field: 'body', value: 'kmalloc spin' });
+// ['parse without boolean operators: "kmalloc spin" runs as OR of substring contains, one per word']
+```
+
 #### Filtering
 
 Filter on non-text fields (combined with AND):
@@ -180,6 +227,35 @@ const restored = Index.importSnapshotFrom('./backup.luce', './restored_index');
 
 // Import from Buffer
 const restored2 = Index.importSnapshot(buf, './restored_index');
+
+// Serve a snapshot in place — nothing extracted, nothing written to disk.
+// Memory cost = the blob's length. Read-only: add / delete / commit / compact
+// and the snapshot or delta exports throw. Use importSnapshot() for a writable copy.
+const served = Index.openSnapshot(buf);
+const served2 = Index.openSnapshotFrom('./backup.luce');
+served.search('programming');   // same results as the source index
+served.path;                    // '' — a served snapshot has no directory
+```
+
+### Maintenance
+
+```javascript
+// Once after a bulk load: merge every shard's segments into segments of at
+// most maxDocs documents (default 10000), then commit. Returns the number of
+// merge rounds that reduced a shard's segment count.
+const merges = index.compact();          // or index.compact(50000)
+
+// Block until no background merge is running or about to start.
+// Returns the number of rounds that still saw activity (0 = already quiet).
+index.waitMergesQuiet();
+
+// On-disk bytes of every searchable segment, all shards. Call waitMergesQuiet()
+// first for a stable figure.
+const bytes = index.indexBytes();
+
+// Delete the whole index: close, then remove its files. The instance is
+// consumed — every later call on it throws.
+index.dropIndex();
 ```
 
 ### Delta sync (incremental)
@@ -229,9 +305,149 @@ index.numShards    // number of shards (getter)
 index.path         // index directory path (getter)
 index.schema       // array of {name, type} objects (getter)
 index.shardVersions // per-shard version info for delta sync (getter)
+index.indexBytes() // on-disk size of all searchable segments
 index.close()      // flush + release writer lock
+index.dropIndex()  // close + delete the index files (instance unusable afterwards)
 ```
 
+## Bring your own storage (ACID)
+
+`Index` stores its files on the filesystem. `BlobIndex` stores them wherever
+you say: every file of the index becomes a blob that your JavaScript object
+loads and saves — a transactional database, an object store, a `Map`. The
+store is the source of truth; a local directory only caches the blobs for
+mmap reads and can be thrown away. `meta.json` is written last at each commit,
+so a store with transactions gives you an index that is either at the previous
+commit or at the new one, never in between.
+
+### The store object
+
+```javascript
+const store = {
+    load(indexName, fileName)          // → Buffer | Uint8Array | null (null = does not exist)
+    save(indexName, fileName, data)    // data: Buffer; create or overwrite
+    delete(indexName, fileName)        // a missing blob is not an error
+    exists(indexName, fileName)        // → boolean
+    list(indexName)                    // → string[]: every fileName under indexName
+    // optional, for lazy loading:
+    blobLen(indexName, fileName)       // → number | null (null = unknown)
+    loadRange(indexName, fileName, offset, length)  // → Buffer | Uint8Array | null (null = unsupported)
+};
+```
+
+Each method may return its value directly or a Promise of it. A thrown error
+or a rejection is reported as the rejection of the `BlobIndex` call that
+needed it. Methods are called with the store as `this`.
+
+`indexName` is a namespace, not just the name you gave: segment files live
+under `"Lucivy_<name>/shard_<i>"`, the root files (`_shard_config.json`,
+`_shard_stats.bin`) under the bare `"<name>"`. Keep blobs keyed by the pair
+`(indexName, fileName)` and `list()` cheap. Opening an index takes the writer
+lock through the store (`.lucivy-writer.lock` is saved, then deleted) and
+`close()` always commits, so `save` and `delete` must work even in a process
+that only searches.
+
+### Why `BlobIndex` is asynchronous
+
+The engine calls the store from its own threads — segment writers, merges,
+lazy loads — while a JavaScript callback can only run on the JavaScript
+thread. If that thread were blocked inside a synchronous call such as
+`index.commit()`, the callbacks it waits for could never run. So every
+`BlobIndex` method runs its work on the Node.js thread pool and returns a
+Promise; while you `await` it, the event loop is free and the store
+callbacks are dispatched through it (one `ThreadsafeFunction` per method;
+the engine thread blocks on a channel until the callback answered).
+
+The corollary: **the JavaScript thread must stay free while a `BlobIndex`
+call is pending.** Never `await` a `BlobIndex` call from inside a store
+callback, and do not block the event loop with synchronous work (a
+`readFileSync` inside `load` is fine; a `while` loop waiting for something
+is not). Store callbacks that return Promises are awaited normally.
+
+### Example: a Map
+
+```javascript
+const { BlobIndex } = require('lucivy');
+
+const blobs = new Map();
+const key = (indexName, fileName) => `${indexName}|${fileName}`;
+const store = {
+    load: (i, f) => blobs.get(key(i, f)) ?? null,
+    save: (i, f, data) => { blobs.set(key(i, f), Buffer.from(data)); },
+    delete: (i, f) => { blobs.delete(key(i, f)); },
+    exists: (i, f) => blobs.has(key(i, f)),
+    list: (i) => [...blobs.keys()].filter(k => k.startsWith(i + '|')).map(k => k.slice(i.length + 1)),
+};
+
+const index = await BlobIndex.create(store, 'articles', [
+    { name: 'body', type: 'text', stored: true },
+], { shards: 2 });
+await index.addMany([{ docId: 1, body: 'kmalloc' }, { docId: 2, body: 'spin_lock_init' }]);
+await index.commit();
+const hits = await index.search('kmalloc', { highlights: true });
+await index.close();
+
+// Later, elsewhere — same blobs, same answers:
+const again = await BlobIndex.open(store, 'articles');
+await again.search({ type: 'contains', field: 'body', value: 'lock' });
+await again.close();
+```
+
+With a database, `save` is an upsert, `load` a select, and `commit()` can be
+wrapped in a transaction by the store itself.
+
+### API
+
+```javascript
+BlobIndex.create(store, indexName, fields, options?)   // → Promise<BlobIndex>
+BlobIndex.open(store, indexName, options?)             // → Promise<BlobIndex>
+// options: { cacheDir?: string, lazy?: boolean, shards?: number (create only) }
+
+await index.add(docId, fields);       await index.addMany(docs);
+await index.delete(docId);            await index.update(docId, fields);
+await index.commit();                 await index.search(query, options);
+await index.queryWarnings(query);     await index.numDocs();
+await index.compact(maxDocs?);        await index.waitMergesQuiet();
+await index.indexBytes();             await index.close();
+await index.dropIndex();
+index.indexName; index.numShards; index.schema   // synchronous getters
+```
+
+Queries, options and results are exactly those of `Index`. `dropIndex()`
+closes the index, then deletes every blob it owns: it lists and deletes the
+`Lucivy_<name>/shard_<i>` namespaces and the `<name>` namespace through
+`store.list()` / `store.delete()`; other indexes in the same store are
+untouched. As with `Index`, the instance is consumed — every later call
+rejects.
+
+### Cache directory and lazy loading
+
+`cacheDir` (default: `lucivy_blob_cache` under the OS temp dir) receives a
+`<pid>/<namespace>_<n>/` directory per shard, filled from the store and
+removed when the index is dropped from memory. It is disposable; it must be
+on a local filesystem that supports mmap.
+
+By default every blob is pulled at `open()`. With `lazy: true` — worth it
+only when the store implements `blobLen` and `loadRange` — `open()` reads
+the metadata files and probes segment footers with small `loadRange` calls;
+each other file is loaded whole the first time a query needs more than a few
+kilobytes of it. Searches give the same answers either way; the first query
+pays for what it touches.
+
+### Close before exit
+
+`close()` waits for background merges, commits, releases the lock and
+guarantees that nothing touches the store afterwards — call it before tearing
+down a database connection and before the process exits. A store that stays
+idle does not keep the process alive; if an index is garbage-collected
+without `close()`, a flush that would have needed the JavaScript thread is
+refused rather than deadlocked, and only the last uncommitted changes are
+lost.
+
+A store error — thrown or rejected, from a segment write in the background
+or from `meta.json` at the commit point — comes back as the rejection of the
+`BlobIndex` call that needed it, with the store's own message.
+
 ## License
 
 MIT

package/index.d.ts

@@ -30,11 +30,60 @@ export interface SearchOptions {
 /**
  * Merge BM25 stats from multiple nodes into global stats (for distributed search).
  *
+ * Each node calls `index.exportStats(queryJson)` which returns a JSON string.
+ * The coordinator collects all JSON strings and merges them with this function.
+ * The merged result is then passed back to each node via
+ * `index.searchWithGlobalStats(queryJson, mergedJson)`.
+ *
  * @param statsList - Array of JSON strings, one per node (from `exportStats()`).
  * @returns JSON string of merged `ExportableStats` ready for `searchWithGlobalStats()`.
  */
 export declare function mergeStats(statsList: Array<string>): string
-
+/**
+ * The object a JavaScript program hands to `BlobIndex.create()` /
+ * `BlobIndex.open()`. Methods are called with the object as `this`. Each may
+ * return its value directly or a Promise of it.
+ *
+ * Keys: `indexName` is `"Lucivy_<name>/shard_<i>"` for segment files and the
+ * bare `<name>` for the root files (`_shard_config.json`, `_shard_stats.bin`);
+ * `fileName` is the file within that namespace.
+ */
+export interface BlobStoreCallbacks {
+  /** Bytes of a blob, or `null` when it does not exist. */
+  load: (indexName: string, fileName: string) => Buffer | Uint8Array | null | Promise<Buffer | Uint8Array | null>
+  /** Create or overwrite a blob. */
+  save: (indexName: string, fileName: string, data: Buffer) => void | Promise<void>
+  /** Remove a blob; a missing blob is not an error. */
+  delete: (indexName: string, fileName: string) => void | Promise<void>
+  exists: (indexName: string, fileName: string) => boolean | Promise<boolean>
+  /** Every file name stored under `indexName`. */
+  list: (indexName: string) => string[] | Promise<string[]>
+  /**
+   * Optional, for `lazy: true`: size of a blob without loading it
+   * (`null` = unknown, the file is then loaded whole on first open).
+   */
+  blobLen?: (indexName: string, fileName: string) => number | null | Promise<number | null>
+  /**
+   * Optional, for `lazy: true`: `length` bytes of a blob from `offset`
+   * (`null` = unsupported, the file is then loaded whole).
+   */
+  loadRange?: (indexName: string, fileName: string, offset: number, length: number) => Buffer | Uint8Array | null | Promise<Buffer | Uint8Array | null>
+}
+/** Options of `BlobIndex.create()` / `BlobIndex.open()`. */
+export interface BlobIndexOptions {
+  /**
+   * Local directory for the mmap cache of the blobs (default: `lucivy_blob_cache`
+   * under the OS temp dir). Disposable: the store is the source of truth.
+   */
+  cacheDir?: string
+  /**
+   * Pull blobs on first use instead of all at open. Needs `blobLen` and
+   * `loadRange` on the store to be worth it.
+   */
+  lazy?: boolean
+  /** `create()` only: number of shards (default 1). */
+  shards?: number
+}
 export declare class Index {
   /**
    * Create a new index at the given path.
@@ -100,6 +149,9 @@ export declare class Index {
    *
    * After `close()`, the index data remains on disk and can be re-opened
    * with `Index.open()`. No further mutations are allowed on this instance.
+   *
+   * On a snapshot served with `openSnapshot()` there is nothing to flush
+   * and no lock to release: `close()` is a no-op.
    */
   close(): void
   /**
@@ -124,7 +176,7 @@ export declare class Index {
    * - `{type: "more_like_this", field: "body", value: "sample text", min_doc_frequency: 1}` — similarity
    *
    * **Filtering:**
-   * - `allowed_ids` in options: pre-filter by _node_id (fast, bitmap-based)
+   * - `allowedIds` in options: pre-filter by _node_id (fast, bitmap-based)
    * - `filters` key in query: filter on non-text fields (AND'd with search):
    *   ```json
    *   {type: "contains", field: "body", value: "lock",
@@ -137,8 +189,16 @@ export declare class Index {
    *   Ops: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`, `between`, `starts_with`, `contains`.
    *   Composite: `must`, `should`, `must_not` with nested `clauses`.
    *
-   * @param options - `{limit?: number, highlights?: boolean, fields?: boolean, allowed_ids?: number[]}`
+   * Honest warnings for a query, without running it.
+   *
+   * Plain-text warnings describing what the engine will actually search
+   * and where it falls back to brute force: separators ignored in relaxed
+   * mode, fuzzy distance too loose for the query length, regex without a
+   * usable literal (full scan), segments written by the legacy indexer.
+   * Empty array when nothing applies.
    */
+  queryWarnings(query: any): Array<string>
+  /** @param options - `{limit?: number, highlights?: boolean, fields?: boolean, allowedIds?: number[]}` */
   search(query: any, options?: SearchOptions | undefined | null): Array<SearchResult>
   /**
    * Number of documents in the index (getter, access as `index.numDocs`).
@@ -196,6 +256,69 @@ export declare class Index {
    * @returns A new `Index` instance ready for search.
    */
   static importSnapshotFrom(path: string, destPath?: string | undefined | null): Index
+  /**
+   * Serve a LUCE snapshot (Buffer) directly, without extracting it.
+   *
+   * The blob itself is the index: readers get slices of it, nothing is
+   * written to disk and the memory cost is the blob's own length. The
+   * result is **read-only** — `add()`, `delete()`, `commit()`, `compact()`
+   * and the delta/snapshot exports fail with a clear error. To get a
+   * writable index back, use `Index.importSnapshot()` instead.
+   *
+   * @param data - Raw LUCE snapshot bytes (Buffer), as produced by `exportSnapshot()`.
+   * @returns A read-only `Index` ready for search. Its `path` is `""`.
+   */
+  static openSnapshot(data: Buffer): Index
+  /**
+   * Serve a LUCE snapshot file (.luce) directly, without extracting it.
+   *
+   * Convenience wrapper that reads the file then calls `openSnapshot()`.
+   * Same read-only semantics.
+   *
+   * @param path - Path to the `.luce` snapshot file.
+   * @returns A read-only `Index` ready for search.
+   */
+  static openSnapshotFrom(path: string): Index
+  /**
+   * Merge every shard's segments into segments of at most `maxDocs`
+   * documents, then commit.
+   *
+   * Bulk loading leaves many small segments behind; one `compact()` after
+   * the load makes searches faster and the index smaller on disk. Not
+   * something to call on every commit.
+   *
+   * @param maxDocs - Upper bound on documents per merged segment (default 10000).
+   * @returns Number of merge rounds that actually reduced a shard's segment count.
+   */
+  compact(maxDocs?: number | undefined | null): number
+  /**
+   * Block until no background merge is running or about to start.
+   *
+   * Segment merges run in the background after commits. Call this before
+   * anything that needs a stable set of files or the full address space —
+   * measuring `indexBytes()`, copying the directory, exporting a snapshot
+   * under memory pressure.
+   *
+   * @returns Number of rounds that still saw merge activity (0 = already quiet).
+   */
+  waitMergesQuiet(): number
+  /**
+   * On-disk bytes of every searchable segment of every shard.
+   *
+   * Sums the segment files; the number moves while merges run, so call
+   * `waitMergesQuiet()` first for a stable figure.
+   *
+   * @returns Total size in bytes (a `number`; exact below 2^53).
+   */
+  indexBytes(): number
+  /**
+   * Delete the whole index: commit and release everything (like `close()`),
+   * then remove the index files from disk.
+   *
+   * This consumes the underlying handle. After `dropIndex()` every other
+   * method on this instance throws; create or open a new `Index` instead.
+   */
+  dropIndex(): void
   /**
    * Schema as a list of field definitions (getter, access as `index.schema`).
    *
@@ -259,3 +382,75 @@ export declare class Index {
    */
   searchWithGlobalStats(queryJson: string, globalStatsJson: string, limit?: number | undefined | null, highlights?: boolean | undefined | null): Array<SearchResult>
 }
+/**
+ * An index whose files live in a storage you provide — a transactional
+ * database, an object store, a Map — through the `BlobStoreCallbacks`
+ * object. Every method returns a Promise: the work runs off the JS thread,
+ * which stays free to serve the store callbacks.
+ */
+export declare class BlobIndex {
+  /**
+   * Create a new index in the given store.
+   *
+   * @param store - Object implementing the store protocol (`load`, `save`, `delete`, `exists`, `list`, optional `blobLen` / `loadRange`).
+   * @param indexName - Name of the index inside the store.
+   * @param fields - Field definitions, as for `Index.create()`.
+   * @param options - `{cacheDir?, lazy?, shards?}`.
+   */
+  static create(store: BlobStoreCallbacks, indexName: string, fields: Array<FieldDef>, options?: BlobIndexOptions | undefined | null): Promise<BlobIndex>
+  /**
+   * Open an index that already exists in the store.
+   *
+   * @param store - Same protocol as for `create()`.
+   * @param indexName - Name given at creation.
+   * @param options - `{cacheDir?, lazy?}`.
+   */
+  static open(store: BlobStoreCallbacks, indexName: string, options?: BlobIndexOptions | undefined | null): Promise<BlobIndex>
+  /** Add a document. Same arguments as `Index.add()`. */
+  add(docId: number, fields: Record<string, any>): Promise<void>
+  /** Add multiple documents. Same arguments as `Index.addMany()`. */
+  addMany(docs: Array<Record<string, any>>): Promise<void>
+  /** Delete a document by its `_node_id` (staged until `commit()`). */
+  delete(docId: number): Promise<void>
+  /** Update a document (delete old + re-add with new fields). */
+  update(docId: number, fields: Record<string, any>): Promise<void>
+  /**
+   * Commit pending changes to the store: segment files are saved through
+   * `store.save()`, `meta.json` last (the commit point).
+   */
+  commit(): Promise<void>
+  /** Search. Same arguments and results as `Index.search()`. */
+  search(query: any, options?: SearchOptions | undefined | null): Promise<Array<SearchResult>>
+  /** Honest warnings for a query, without running it. See `Index.queryWarnings()`. */
+  queryWarnings(query: any): Promise<Array<string>>
+  /** Number of documents across all shards. */
+  numDocs(): Promise<number>
+  /**
+   * Flush pending writes, wait for merges, release the writer lock.
+   * After `close()` the store is not touched again: it is safe to tear
+   * down whatever backs it. Always call it before the process exits.
+   */
+  close(): Promise<void>
+  /**
+   * Merge every shard's segments into segments of at most `maxDocs`
+   * documents, then commit. See `Index.compact()`.
+   */
+  compact(maxDocs?: number | undefined | null): Promise<number>
+  /** Block until no background merge is running or about to start. */
+  waitMergesQuiet(): Promise<number>
+  /** Bytes of every searchable segment of every shard, as cached locally. */
+  indexBytes(): Promise<number>
+  /**
+   * Delete the whole index: `close()`, then every blob the store holds
+   * for it — the `Lucivy_<name>/shard_<i>` namespaces and the root
+   * `<name>` namespace, each listed with `store.list()` and removed with
+   * `store.delete()`. Consumes the handle: every later call throws.
+   */
+  dropIndex(): Promise<void>
+  /** Name of the index inside the store (getter). */
+  get indexName(): string
+  /** Number of shards (getter). */
+  get numShards(): number
+  /** Schema as a list of field definitions (getter). */
+  get schema(): Array<FieldDef>
+}

package/index.js

@@ -1,3 +1,3 @@
-const { Index, mergeStats } = require('./lucivy.node');
+const { Index, BlobIndex, mergeStats } = require('./lucivy.node');
 
-module.exports = { Index, mergeStats };
+module.exports = { Index, BlobIndex, mergeStats };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "lucivy",
-  "version": "2.0.2",
+  "version": "3.0.1",
   "description": "Fast BM25 full-text search with substring matching, fuzzy search, and regex — powered by Rust",
   "main": "index.js",
   "types": "index.d.ts",