@yoch/frozenminisearch 1.0.2 → 1.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,18 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## v1.1.0 — `@yoch/frozenminisearch`
6
+
7
+ Minor release: MiniSearch JSON wire export and clearer JSON import API. MSv5 binary format unchanged.
8
+
9
+ ### Added
10
+
11
+ - **`toJSON()`** — export MiniSearch wire snapshots (`serializationVersion: 2`); import via `fromJson` / `fromMiniSearchSnapshot`. Production persistence remains `saveBinarySync`.
12
+
13
+ ### Breaking
14
+
15
+ - **`fromMiniSearchJson` → `fromJson`** — rename for clearer semantics (JSON import vs binary load). Update call sites: `FrozenMiniSearch.fromMiniSearchJson(json)` → `FrozenMiniSearch.fromJson(json)`.
16
+
5
17
  ## v1.0.2 — `@yoch/frozenminisearch`
6
18
 
7
19
  Patch release: lower retained heap when `storeFields` has one field. No API or MSv5 wire-format changes.
@@ -23,7 +35,7 @@ Patch release: lower build-time peak memory and migration ergonomics. No API or
23
35
 
24
36
  ### Fixed
25
37
 
26
- - **Default tokenizer parity** — leading delimiter produces an empty token (e.g. `::a` → `["", "a"]`), matching lucaong `split` behaviour.
38
+ - **Default tokenizer parity** — leading delimiter produces an empty token (e.g. `::a` → `["", "a"]`), matching MiniSearch `split` behaviour.
27
39
  - **Named export** — `FrozenMiniSearch` is exported again alongside the default export (ESM and CJS).
28
40
 
29
41
  ## v1.0.0 — `@yoch/frozenminisearch`
@@ -32,7 +44,7 @@ First stable release on npm. Frozen-only read-only search for Node.js.
32
44
 
33
45
  ### Breaking
34
46
 
35
- - **Binary snapshots** — `loadBinarySync` / `loadBinaryAsync` read only the current frozen binary format; re-build from lucaong JSON if an older snapshot fails to load.
47
+ - **Binary snapshots** — `loadBinarySync` / `loadBinaryAsync` read only the current frozen binary format; re-build from MiniSearch JSON if an older snapshot fails to load.
36
48
  - **Removed `saveBinary()` / `loadBinary()`** — use `saveBinarySync` / `saveBinaryAsync` and `loadBinarySync` / `loadBinaryAsync`.
37
49
 
38
50
  ## v1.0.0-beta.0 — `@yoch/frozenminisearch`
@@ -42,16 +54,16 @@ New standalone package (frozen-only) for read-only serving workloads.
42
54
  ### Added
43
55
 
44
56
  - **`FrozenMiniSearch`** as the default export — `fromDocuments`, builder, `saveBinarySync` / `loadBinarySync`
45
- - **Migration loaders** — `fromMiniSearch`, `fromMiniSearchJson`, `fromMiniSearchSnapshot` (lucaong JSON wire format)
57
+ - **Migration loaders** — `fromMiniSearch`, `fromJson`, `fromMiniSearchSnapshot` (MiniSearch JSON wire format)
46
58
  - **Modular benchmarks** — `npm run bench` with profiles `vs-reference`, `regression`, `dev`
47
59
  - **Parity suite** — `dev/parity/` vs `minisearch` npm (functional invariants)
48
60
 
49
61
  ### Removed from published API
50
62
 
51
63
  - Mutable `MiniSearch` class and `freeze()` on the fork
52
- - `freezeFromMiniSearch` (use `fromMiniSearchJson`)
64
+ - `freezeFromMiniSearch` (use `fromJson`)
53
65
  - Read-only mutation stubs (`add`, `remove`, …)
54
66
 
55
67
  ### Migration
56
68
 
57
- - `new MiniSearch(opts).addAll(docs)` (lucaong) → `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` — see README
69
+ - `new MiniSearch(opts).addAll(docs)` → `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` — see README
package/README.md CHANGED
@@ -1,21 +1,36 @@
1
- # @yoch/frozenminisearch
1
+ # FrozenMiniSearch
2
2
 
3
- **Read-only full-text search for Node.js** — compact frozen indexes, fast binary snapshots, and a **drop-in** search API for frozen workloads: same `search`, `autoSuggest`, scoring, and query options as [MiniSearch](https://github.com/lucaong/minisearch) by [Luca Ongaro](https://github.com/lucaong).
3
+ [![npm version](https://img.shields.io/npm/v/@yoch/frozenminisearch.svg)](https://www.npmjs.com/package/@yoch/frozenminisearch)
4
+ [![coverage](https://codecov.io/gh/yoch/frozenminisearch/graph/badge.svg)](https://codecov.io/gh/yoch/frozenminisearch)
5
+ [![CI](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml/badge.svg)](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml)
6
+ [![Socket Badge](https://socket.dev/api/badge/npm/package/@yoch/frozenminisearch)](https://socket.dev/npm/package/%40yoch%2Ffrozenminisearch)
4
7
 
5
- > **Current release:** `1.0.2` on npm
8
+ **Memory-optimized, read-only full-text search for Node.js** the same BM25, prefix/fuzzy, and `autoSuggest` API as [MiniSearch](https://github.com/lucaong/minisearch), with **up to ~98% less index RAM** on real corpora and compact binary snapshots you ship instead of JSON.
6
9
 
7
- **Design goal:** once an index is built or loaded, migrate with the minimum code change package name and index construction only; serving code stays the same. No mutable `MiniSearch` class is published here; build indexes with `fromDocuments`, the incremental builder, or migrate from an existing lucaong index via `fromMiniSearchJson`.
10
+ **Why it exists:** [MiniSearch](https://github.com/lucaong/minisearch) optimizes for a mutable in-memory index. FrozenMiniSearch optimizes for **retained heap, disk footprint, and cold load** once the corpus is fixed packed radix postings, columnar `storeFields`, typed-array layouts, and MSv5 binary wire format instead of per-document JS objects.
11
+
12
+ **Design goal:** migrate with minimal code change — package name and index construction only; serving code stays the same. Build with `fromDocuments`, the incremental builder, or `fromJson`; no mutable `MiniSearch` class is published here.
8
13
 
9
14
  ---
10
15
 
11
16
  ## Why frozen instead of MiniSearch?
12
17
 
13
- **Mutable** lucaong `minisearch` when documents change (`add`, `remove`, `discard`). **Frozen** when the corpus is fixed or shipped as a binary snapshot same BM25, prefix/fuzzy, `autoSuggest`, wildcard, and `AND` / `OR` / `AND_NOT`. Parity with `minisearch@7` is validated in `dev/parity/` (scores `toBeCloseTo` precision 6).
18
+ Choose **mutable MiniSearch** when documents change at runtime (`add`, `remove`, `discard`). Choose **frozen** when memory and snapshot size matter: fixed corpus, deploy from binary, many replicas loading the same index. Search semantics stay the same — BM25, prefix/fuzzy, `autoSuggest`, wildcard, `AND` / `OR` / `AND_NOT` with parity vs MiniSearch 7 validated in `dev/parity/` (scores `toBeCloseTo` precision 6).
19
+
20
+ ### Memory-first design
21
+
22
+ | Technique | What it saves |
23
+ |-----------|---------------|
24
+ | **Packed radix tree + flat postings** | Term dictionary and posting lists without per-entry JS wrappers |
25
+ | **Columnar `storeFields`** | One dense column per field instead of a `Record` per document (~75% less heap for a single stored field) |
26
+ | **MSv5 binary snapshots** | ~73–94% smaller on disk than MiniSearch JSON; faster cold load |
27
+ | **Read-only freeze** | No mutation bookkeeping — layouts sized for serve-time, not incremental edit |
28
+ | **Incremental builder** | Typed-array accumulators during build; lower peak heap than materializing `number[][]` per term |
14
29
 
15
30
  <!-- vs-reference:start — npm run bench:readme -->
16
- ### Measured vs lucaong MiniSearch (reference baseline)
31
+ ### Measured vs MiniSearch (reference baseline)
17
32
 
18
- Same BM25 queries on identical corpora. **Frozen wins on what we optimize for**: RAM, disk, cold load, and search throughput on real workloads.
33
+ Same BM25 queries on identical corpora. **Index RAM is the headline metric** frozen uses a fraction of mutable heap on every scenario below; disk and cold load follow from the compact binary format.
19
34
 
20
35
  | Scenario | Docs | Index RAM¹ | Disk (binary vs JSON)² | Cold load³ | Search p50⁴ |
21
36
  |----------|-----:|------------|------------------------:|-----------:|------------:|
@@ -29,9 +44,10 @@ Same BM25 queries on identical corpora. **Frozen wins on what we optimize for**:
29
44
 
30
45
  Decomposition (Divina exact): L0 lookup ~300 ns frozen, L1 `executeQuery` ~8.3 µs, L2 full `search` ~11.6 µs (finalize ≈ 3 µs).
31
46
 
32
- | | lucaong `minisearch` | `@yoch/frozenminisearch` |
47
+ | | MiniSearch | `@yoch/frozenminisearch` |
33
48
  |---|------------------------|---------------------------|
34
- | **Sweet spot** | Live index mutations | Fixed corpus, deploy from binary |
49
+ | **Optimizes for** | Live mutations, flexibility | **Retained RAM**, snapshot size, cold load |
50
+ | **Sweet spot** | Documents change at runtime | Fixed corpus, many replicas, tight memory budget |
35
51
  | **Production path** | `addAll` → `toJSON` | `fromDocuments` / `fromMiniSearch` → `saveBinarySync` → `loadBinarySync` |
36
52
  | **Typical trade-off** | Higher RAM, JSON snapshots | One-time freeze, then compact binary |
37
53
 
@@ -91,17 +107,17 @@ ESM and CommonJS are both supported (`main` → CJS, `module` → ESM).
91
107
 
92
108
  ## Drop-in
93
109
 
94
- For **fixed corpora** (build once, serve read-only), treat this package as a drop-in replacement for lucaong `minisearch` on the serving path.
110
+ For **fixed corpora** (build once, serve read-only), treat this package as a drop-in replacement for MiniSearch on the serving path — same queries, far less memory per replica.
95
111
 
96
112
  **Change only:**
97
113
 
98
114
  | What | Before | After |
99
115
  |------|--------|-------|
100
- | Package | lucaong `minisearch` | `@yoch/frozenminisearch` |
116
+ | Package | `minisearch` | `@yoch/frozenminisearch` |
101
117
  | Construction | `new MiniSearch(opts).addAll(docs)` | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
102
- | JSON snapshot | `MiniSearch.loadJSON(json)` / `toJSON()` wire format | `FrozenMiniSearch.fromMiniSearchJson(json, opts)` or `fromMiniSearchSnapshot(obj)` — no runtime dependency on lucaong `minisearch` |
118
+ | JSON snapshot | `toJSON()` / `loadJSON()` wire format | `FrozenMiniSearch.toJSON()` / `fromJson(json, opts)` or `fromMiniSearchSnapshot(obj)` — no runtime dependency on `minisearch` |
103
119
 
104
- **Keep unchanged** after load: `search`, `autoSuggest`, `has`, `getStoredFields`, query options (`prefix`, `fuzzy`, `AND` / `OR` / `AND_NOT`, filters, boosts). Parity vs `minisearch@7` is enforced in `dev/parity/`.
120
+ **Keep unchanged** after load: `search`, `autoSuggest`, `has`, `getStoredFields`, query options (`prefix`, `fuzzy`, `AND` / `OR` / `AND_NOT`, filters, boosts). Parity vs MiniSearch 7 is enforced in `dev/parity/`.
105
121
 
106
122
  **Imports** — default and named both work (ESM and CJS):
107
123
 
@@ -121,7 +137,7 @@ const { FrozenMiniSearch } = require('@yoch/frozenminisearch')
121
137
 
122
138
  ## Migration
123
139
 
124
- ### From lucaong `minisearch` JSON
140
+ ### From MiniSearch JSON
125
141
 
126
142
  ```javascript
127
143
  import MiniSearch from 'minisearch' // build-time only
@@ -135,18 +151,18 @@ const frozen = FrozenMiniSearch.fromMiniSearch(mutable, options)
135
151
 
136
152
  // Option B — serialized index (offline / ETL)
137
153
  const json = JSON.stringify(mutable)
138
- const frozen2 = FrozenMiniSearch.fromMiniSearchJson(json, options)
154
+ const frozen2 = FrozenMiniSearch.fromJson(json, options)
139
155
  ```
140
156
 
141
157
  `options.fields` must match the indexed fields in the snapshot when provided.
142
158
 
143
- ### From lucaong `minisearch` (mutable → frozen)
159
+ ### From MiniSearch (mutable → frozen)
144
160
 
145
161
  | Before (mutable) | After (`@yoch/frozenminisearch`) |
146
162
  |------------------|----------------------------------|
147
163
  | `new MiniSearch(opts).addAll(docs)` then serve | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
148
- | lucaong JSON snapshot | `FrozenMiniSearch.fromMiniSearchJson(json)` or `fromMiniSearchSnapshot(obj)` |
149
- | `import MiniSearch from 'minisearch'` | `import FrozenMiniSearch from '@yoch/frozenminisearch'` (+ lucaong `minisearch` only if you still build mutable indexes) |
164
+ | MiniSearch JSON snapshot | `FrozenMiniSearch.fromJson(json)` or `fromMiniSearchSnapshot(obj)` |
165
+ | `import MiniSearch from 'minisearch'` | `import FrozenMiniSearch from '@yoch/frozenminisearch'` (+ `minisearch` only if you still build mutable indexes) |
150
166
 
151
167
  ---
152
168
 
@@ -163,13 +179,15 @@ Indexing is **not** available on a frozen instance — use `fromDocuments`, the
163
179
 
164
180
  ## Binary snapshots
165
181
 
182
+ The primary way to **persist and ship a memory-compact index** — smaller than MiniSearch JSON and faster to load into a low-RAM serving process.
183
+
166
184
  ```javascript
167
185
  const buf = index.saveBinarySync()
168
186
  const loaded = FrozenMiniSearch.loadBinarySync(buf, {}) // field names embedded in snapshot
169
187
  ```
170
188
 
171
189
  - **Node ≥ 22.15.0** (zstd via `node:zlib`)
172
- - Snapshots produced by this package version are forward-compatible; re-build from lucaong JSON if an older binary fails to load
190
+ - Snapshots produced by this package version are forward-compatible; re-build from MiniSearch JSON if an older binary fails to load
173
191
  - `tokenize` / `processTerm` are not stored — pass the same functions at load when customized
174
192
 
175
193
  ---
@@ -206,6 +224,6 @@ Design notes (freq adaptive, AND gating): [dev/docs/README.md](dev/docs/README.m
206
224
  See [CHANGELOG.md](./CHANGELOG.md).
207
225
 
208
226
  - **MiniSearch** — [Luca Ongaro](https://github.com/lucaong/minisearch) (MIT)
209
- - **@yoch/frozenminisearch** — frozen indexes, packed radix tree, compact binary snapshots
227
+ - **@yoch/frozenminisearch** — memory-optimized frozen indexes, packed radix tree, compact binary snapshots
210
228
 
211
229
  Upstream docs: [MiniSearch](https://lucaong.github.io/minisearch/)
@@ -1938,7 +1938,7 @@ function postingsTypedBytes(layout) {
1938
1938
  slotCount,
1939
1939
  };
1940
1940
  }
1941
- function validateFrozenPostingsLayout(layout, documentCount, nextId, fail = detail => { throw new Error(detail); }) {
1941
+ function validateFrozenPostingsLayout(layout, documentCount, nextId, fail = (detail) => { throw new Error(detail); }) {
1942
1942
  if (layout.fieldCount <= 0)
1943
1943
  fail('fieldCount must be positive');
1944
1944
  if (layout.nextId !== nextId)
@@ -2209,7 +2209,7 @@ function forEachDefaultToken(text, onToken) {
2209
2209
  /** Default tokenizer into a reusable buffer (avoids `text.split()` array allocation). */
2210
2210
  function tokenizeDefaultInto(out, text) {
2211
2211
  out.length = 0;
2212
- forEachDefaultToken(text, (token) => out.push(token));
2212
+ forEachDefaultToken(text, token => out.push(token));
2213
2213
  }
2214
2214
  /** Tokenize field text into `out` (reused). Fast path when `tokenize` is the library default. */
2215
2215
  function tokenizeFieldInto(out, tokenize, text, fieldName) {
@@ -2288,7 +2288,7 @@ function validateFrozenSnapshotNumeric(snap) {
2288
2288
  if (snap.avgFieldLength.length !== snap.fieldCount) {
2289
2289
  throw invalidFrozenIndex('avgFieldLength size mismatch');
2290
2290
  }
2291
- validateFrozenPostingsLayout(snap.postings, snap.documentCount, snap.nextId, detail => {
2291
+ validateFrozenPostingsLayout(snap.postings, snap.documentCount, snap.nextId, (detail) => {
2292
2292
  throw invalidFrozenIndex(detail);
2293
2293
  });
2294
2294
  const indexedFields = Object.keys(snap.fieldIds);
@@ -2503,7 +2503,7 @@ function cloneStoredFields(layout) {
2503
2503
  }
2504
2504
  return { kind: 'multi', rows: layout.rows.slice() };
2505
2505
  }
2506
- /** Import from wire rows or lucaong snapshot. Empty storeFields + non-empty rows → multi (binary load without options). */
2506
+ /** Import from wire rows or MiniSearch snapshot. Empty storeFields + non-empty rows → multi (binary load without options). */
2507
2507
  function storedFieldsFromRows(rows, storeFields) {
2508
2508
  if (storeFields.length === 0) {
2509
2509
  const hasAny = rows.some(row => row != null && Object.keys(row).length > 0);
@@ -2687,7 +2687,7 @@ function buildFlatPostingsFromSearchableMap(searchableMap, fieldCount, nextId, s
2687
2687
  });
2688
2688
  return { termCount, index: packedIndex, postings };
2689
2689
  }
2690
- /** Build frozen assemble params from a lucaong MiniSearch JSON snapshot. */
2690
+ /** Build frozen assemble params from a MiniSearch JSON snapshot. */
2691
2691
  function buildFrozenAssembleParamsFromMiniSearchSnapshot(snapshot, options) {
2692
2692
  var _a, _b, _c;
2693
2693
  if (!SUPPORTED_SERIALIZATION_VERSIONS.has(snapshot.serializationVersion)) {
@@ -3601,7 +3601,7 @@ function decodeFrozenSnapshot(buf, hints) {
3601
3601
  return decodeFrozenSnapshotMsv5(buf, hints);
3602
3602
  }
3603
3603
  if (LEGACY_MAGICS.has(magic)) {
3604
- throw invalidFrozenIndex('Unsupported frozen binary snapshot; re-build with saveBinarySync() or from lucaong JSON');
3604
+ throw invalidFrozenIndex('Unsupported frozen binary snapshot; re-build with saveBinarySync() or from MiniSearch JSON');
3605
3605
  }
3606
3606
  throw invalidFrozenIndex('Unsupported frozen binary snapshot');
3607
3607
  }
@@ -4302,7 +4302,7 @@ function executeQueryInternal(query, searchOptions, params, allowedDocs, run) {
4302
4302
  return executeWildcardQuery(searchOptions, params);
4303
4303
  }
4304
4304
  if (isQueryCombination(query)) {
4305
- // Spread inherits parent combineWith into child branches (lucaong 7.2 behavior).
4305
+ // Spread inherits parent combineWith into child branches (MiniSearch 7.2 behavior).
4306
4306
  const options = { ...searchOptions, ...query, queries: undefined };
4307
4307
  const operator = ((_b = (_a = query.combineWith) !== null && _a !== void 0 ? _a : options.combineWith) !== null && _b !== void 0 ? _b : params.globalSearchOptions.combineWith);
4308
4308
  if (useGatedEvaluation(run, query.queries.length, operator, combinationHasWildcard(query))) {
@@ -4352,6 +4352,73 @@ function autoSuggestFromSearch(search, queryString, options = {}) {
4352
4352
  return suggestFromSearchResults(search(queryString, options));
4353
4353
  }
4354
4354
 
4355
+ /** Visit shortIds with a defined external id (holes in `externalIds` are skipped). */
4356
+ function forEachLiveShortId(nextId, externalIds, callback) {
4357
+ for (let shortId = 0; shortId < nextId; shortId++) {
4358
+ const externalId = externalIds[shortId];
4359
+ if (externalId === undefined)
4360
+ continue;
4361
+ callback(shortId, externalId);
4362
+ }
4363
+ }
4364
+
4365
+ /**
4366
+ * Build a MiniSearch `toJSON` wire snapshot (`serializationVersion: 2`) from frozen index parts.
4367
+ * Alloc-heavy (plain objects per term/field) — migration/interop only, not production persistence.
4368
+ * All input parts must belong to the same frozen index instance.
4369
+ */
4370
+ function miniSearchSnapshotFromFrozen(input) {
4371
+ const { documentCount, nextId, fieldIds, fieldCount, externalIds, fieldLengthMatrix, avgFieldLength, storedFields, index, fieldTermFlyweight, } = input;
4372
+ const documentIds = {};
4373
+ const fieldLength = {};
4374
+ const storedFieldsOut = {};
4375
+ const hasStoredFields = storedFields.kind !== 'none';
4376
+ forEachLiveShortId(nextId, externalIds, (shortId, externalId) => {
4377
+ var _a;
4378
+ const shortIdStr = String(shortId);
4379
+ documentIds[shortIdStr] = externalId;
4380
+ const lengths = new Array(fieldCount);
4381
+ const rowBase = shortId * fieldCount;
4382
+ for (let f = 0; f < fieldCount; f++) {
4383
+ lengths[f] = (_a = fieldLengthMatrix[rowBase + f]) !== null && _a !== void 0 ? _a : 0;
4384
+ }
4385
+ fieldLength[shortIdStr] = lengths;
4386
+ if (hasStoredFields) {
4387
+ storedFieldsOut[shortIdStr] = readStoredFields(storedFields, shortId);
4388
+ }
4389
+ });
4390
+ const indexEntries = [];
4391
+ for (const [term, termIndex] of index.entries()) {
4392
+ fieldTermFlyweight.bind(termIndex);
4393
+ const fieldData = {};
4394
+ for (let f = 0; f < fieldCount; f++) {
4395
+ const segment = fieldTermFlyweight.get(f);
4396
+ if (segment == null || segment.size === 0)
4397
+ continue;
4398
+ const entry = {};
4399
+ segment.forEachDoc((docId, freq) => {
4400
+ entry[String(docId)] = freq;
4401
+ });
4402
+ fieldData[String(f)] = entry;
4403
+ }
4404
+ if (Object.keys(fieldData).length > 0) {
4405
+ indexEntries.push([term, fieldData]);
4406
+ }
4407
+ }
4408
+ return {
4409
+ documentCount,
4410
+ nextId,
4411
+ documentIds,
4412
+ fieldIds,
4413
+ fieldLength,
4414
+ averageFieldLength: Array.from(avgFieldLength),
4415
+ storedFields: storedFieldsOut,
4416
+ dirtCount: 0,
4417
+ index: indexEntries,
4418
+ serializationVersion: 2,
4419
+ };
4420
+ }
4421
+
4355
4422
  function ownedIndexArray(arr) {
4356
4423
  if (arr instanceof Uint8Array)
4357
4424
  return new Uint8Array(arr);
@@ -4510,12 +4577,9 @@ class FrozenMiniSearch {
4510
4577
  tokenize: this._options.tokenize,
4511
4578
  processTerm: this._options.processTerm,
4512
4579
  indexView: createFrozenQueryIndexView(this._index, this._postings, this._fieldTermFlyweight, (callback) => {
4513
- for (let shortId = 0; shortId < this._nextId; shortId++) {
4514
- const id = this._externalIds[shortId];
4515
- if (id === undefined)
4516
- continue;
4580
+ forEachLiveShortId(this._nextId, this._externalIds, (shortId, id) => {
4517
4581
  callback(shortId, id, readStoredFields(this._storedFields, shortId));
4518
- }
4582
+ });
4519
4583
  }),
4520
4584
  aggregateContext: this._aggregateContext,
4521
4585
  };
@@ -4669,21 +4733,43 @@ class FrozenMiniSearch {
4669
4733
  return buildFrozenFromDocuments(documents, options);
4670
4734
  }
4671
4735
  /**
4672
- * Convert a lucaong MiniSearch JSON snapshot (`toJSON` / `loadJSON` wire format) into a
4673
- * frozen index. No runtime dependency on the `minisearch` package.
4736
+ * Export this index as a MiniSearch wire snapshot (`serializationVersion: 2`).
4737
+ * Use for migration or interchange with the `minisearch` package (`JSON.stringify` works via this method).
4738
+ * Not the primary persistence format — prefer {@link saveBinarySync} for production (size and load time).
4739
+ * Term order in `index` may differ from MiniSearch native `toJSON`; search scores stay equivalent.
4740
+ */
4741
+ toJSON() {
4742
+ return miniSearchSnapshotFromFrozen({
4743
+ documentCount: this._documentCount,
4744
+ nextId: this._nextId,
4745
+ fieldIds: this._fieldIds,
4746
+ fieldCount: this._fieldCount,
4747
+ externalIds: this._externalIds,
4748
+ fieldLengthMatrix: this._fieldLengthMatrix,
4749
+ avgFieldLength: this._avgFieldLength,
4750
+ storedFields: this._storedFields,
4751
+ index: this._index,
4752
+ fieldTermFlyweight: this._fieldTermFlyweight,
4753
+ });
4754
+ }
4755
+ /**
4756
+ * Build a new frozen index **from** a MiniSearch JSON snapshot string (import / migration).
4757
+ * Accepts the wire format produced by MiniSearch `toJSON` or by {@link toJSON} on this class.
4758
+ * Distinct from {@link loadBinarySync}: JSON is MiniSearch interchange, not the native frozen binary.
4759
+ * No runtime dependency on the `minisearch` package.
4674
4760
  */
4675
- static fromMiniSearchJson(json, options = {}) {
4761
+ static fromJson(json, options = {}) {
4676
4762
  return FrozenMiniSearch.fromMiniSearchSnapshot(JSON.parse(json), options);
4677
4763
  }
4678
4764
  /**
4679
- * Same as {@link fromMiniSearchJson} with a pre-parsed snapshot object.
4765
+ * Same as {@link fromJson} with a pre-parsed snapshot object.
4680
4766
  * `storedFields` are shallow-copied; callers must not mutate nested values
4681
4767
  * after load if they intend to keep the index immutable.
4682
4768
  */
4683
4769
  static fromMiniSearchSnapshot(snapshot, options = {}) {
4684
4770
  return assembleFrozenTrusted(buildFrozenAssembleParamsFromMiniSearchSnapshot(snapshot, options), 'minisearch-json');
4685
4771
  }
4686
- /** Accepts any object exposing `toJSON()` in lucaong MiniSearch snapshot shape. */
4772
+ /** Accepts any object exposing `toJSON()` in MiniSearch snapshot shape. */
4687
4773
  static fromMiniSearch(source, options = {}) {
4688
4774
  return FrozenMiniSearch.fromMiniSearchSnapshot(source.toJSON(), options);
4689
4775
  }
@@ -459,7 +459,7 @@ interface FrozenMemoryBreakdown {
459
459
  /**
460
460
  * Low-level parameters for {@link assembleFrozen} (custom frozen index pipelines).
461
461
  * Field types are part of the public surface for advanced assembly; typical apps use
462
- * {@link buildFrozenFromDocuments}, {@link FrozenMiniSearch.fromMiniSearchJson}, or binary load instead.
462
+ * {@link buildFrozenFromDocuments}, {@link FrozenMiniSearch.fromJson}, or binary load instead.
463
463
  */
464
464
  interface FrozenAssembleParams<T = any> {
465
465
  options: OptionsWithDefaults<T>;
@@ -479,7 +479,7 @@ interface FrozenAssembleParams<T = any> {
479
479
  postings: FrozenPostingsLayout;
480
480
  }
481
481
 
482
- /** Lucaong MiniSearch JSON snapshot (`toJSON` / `loadJSON` wire format). */
482
+ /** MiniSearch JSON snapshot (`toJSON` wire format, `serializationVersion` 1 or 2). */
483
483
  type SerializedIndexEntry = Record<string, number>;
484
484
  type MiniSearchSnapshot = {
485
485
  documentCount: number;
@@ -607,17 +607,26 @@ declare class FrozenMiniSearch<T = any> {
607
607
  /** Build a read-only index in one pass from documents. */
608
608
  static fromDocuments<T>(documents: readonly T[], options: Options<T>): FrozenMiniSearch<T>;
609
609
  /**
610
- * Convert a lucaong MiniSearch JSON snapshot (`toJSON` / `loadJSON` wire format) into a
611
- * frozen index. No runtime dependency on the `minisearch` package.
610
+ * Export this index as a MiniSearch wire snapshot (`serializationVersion: 2`).
611
+ * Use for migration or interchange with the `minisearch` package (`JSON.stringify` works via this method).
612
+ * Not the primary persistence format — prefer {@link saveBinarySync} for production (size and load time).
613
+ * Term order in `index` may differ from MiniSearch native `toJSON`; search scores stay equivalent.
612
614
  */
613
- static fromMiniSearchJson<T>(json: string, options?: Options<T>): FrozenMiniSearch<T>;
615
+ toJSON(): MiniSearchSnapshot;
614
616
  /**
615
- * Same as {@link fromMiniSearchJson} with a pre-parsed snapshot object.
617
+ * Build a new frozen index **from** a MiniSearch JSON snapshot string (import / migration).
618
+ * Accepts the wire format produced by MiniSearch `toJSON` or by {@link toJSON} on this class.
619
+ * Distinct from {@link loadBinarySync}: JSON is MiniSearch interchange, not the native frozen binary.
620
+ * No runtime dependency on the `minisearch` package.
621
+ */
622
+ static fromJson<T>(json: string, options?: Options<T>): FrozenMiniSearch<T>;
623
+ /**
624
+ * Same as {@link fromJson} with a pre-parsed snapshot object.
616
625
  * `storedFields` are shallow-copied; callers must not mutate nested values
617
626
  * after load if they intend to keep the index immutable.
618
627
  */
619
628
  static fromMiniSearchSnapshot<T>(snapshot: MiniSearchSnapshot, options?: Options<T>): FrozenMiniSearch<T>;
620
- /** Accepts any object exposing `toJSON()` in lucaong MiniSearch snapshot shape. */
629
+ /** Accepts any object exposing `toJSON()` in MiniSearch snapshot shape. */
621
630
  static fromMiniSearch<T>(source: {
622
631
  toJSON(): MiniSearchSnapshot;
623
632
  }, options?: Options<T>): FrozenMiniSearch<T>;
package/dist/es/index.js CHANGED
@@ -1934,7 +1934,7 @@ function postingsTypedBytes(layout) {
1934
1934
  slotCount,
1935
1935
  };
1936
1936
  }
1937
- function validateFrozenPostingsLayout(layout, documentCount, nextId, fail = detail => { throw new Error(detail); }) {
1937
+ function validateFrozenPostingsLayout(layout, documentCount, nextId, fail = (detail) => { throw new Error(detail); }) {
1938
1938
  if (layout.fieldCount <= 0)
1939
1939
  fail('fieldCount must be positive');
1940
1940
  if (layout.nextId !== nextId)
@@ -2205,7 +2205,7 @@ function forEachDefaultToken(text, onToken) {
2205
2205
  /** Default tokenizer into a reusable buffer (avoids `text.split()` array allocation). */
2206
2206
  function tokenizeDefaultInto(out, text) {
2207
2207
  out.length = 0;
2208
- forEachDefaultToken(text, (token) => out.push(token));
2208
+ forEachDefaultToken(text, token => out.push(token));
2209
2209
  }
2210
2210
  /** Tokenize field text into `out` (reused). Fast path when `tokenize` is the library default. */
2211
2211
  function tokenizeFieldInto(out, tokenize, text, fieldName) {
@@ -2284,7 +2284,7 @@ function validateFrozenSnapshotNumeric(snap) {
2284
2284
  if (snap.avgFieldLength.length !== snap.fieldCount) {
2285
2285
  throw invalidFrozenIndex('avgFieldLength size mismatch');
2286
2286
  }
2287
- validateFrozenPostingsLayout(snap.postings, snap.documentCount, snap.nextId, detail => {
2287
+ validateFrozenPostingsLayout(snap.postings, snap.documentCount, snap.nextId, (detail) => {
2288
2288
  throw invalidFrozenIndex(detail);
2289
2289
  });
2290
2290
  const indexedFields = Object.keys(snap.fieldIds);
@@ -2499,7 +2499,7 @@ function cloneStoredFields(layout) {
2499
2499
  }
2500
2500
  return { kind: 'multi', rows: layout.rows.slice() };
2501
2501
  }
2502
- /** Import from wire rows or lucaong snapshot. Empty storeFields + non-empty rows → multi (binary load without options). */
2502
+ /** Import from wire rows or MiniSearch snapshot. Empty storeFields + non-empty rows → multi (binary load without options). */
2503
2503
  function storedFieldsFromRows(rows, storeFields) {
2504
2504
  if (storeFields.length === 0) {
2505
2505
  const hasAny = rows.some(row => row != null && Object.keys(row).length > 0);
@@ -2683,7 +2683,7 @@ function buildFlatPostingsFromSearchableMap(searchableMap, fieldCount, nextId, s
2683
2683
  });
2684
2684
  return { termCount, index: packedIndex, postings };
2685
2685
  }
2686
- /** Build frozen assemble params from a lucaong MiniSearch JSON snapshot. */
2686
+ /** Build frozen assemble params from a MiniSearch JSON snapshot. */
2687
2687
  function buildFrozenAssembleParamsFromMiniSearchSnapshot(snapshot, options) {
2688
2688
  var _a, _b, _c;
2689
2689
  if (!SUPPORTED_SERIALIZATION_VERSIONS.has(snapshot.serializationVersion)) {
@@ -3597,7 +3597,7 @@ function decodeFrozenSnapshot(buf, hints) {
3597
3597
  return decodeFrozenSnapshotMsv5(buf, hints);
3598
3598
  }
3599
3599
  if (LEGACY_MAGICS.has(magic)) {
3600
- throw invalidFrozenIndex('Unsupported frozen binary snapshot; re-build with saveBinarySync() or from lucaong JSON');
3600
+ throw invalidFrozenIndex('Unsupported frozen binary snapshot; re-build with saveBinarySync() or from MiniSearch JSON');
3601
3601
  }
3602
3602
  throw invalidFrozenIndex('Unsupported frozen binary snapshot');
3603
3603
  }
@@ -4298,7 +4298,7 @@ function executeQueryInternal(query, searchOptions, params, allowedDocs, run) {
4298
4298
  return executeWildcardQuery(searchOptions, params);
4299
4299
  }
4300
4300
  if (isQueryCombination(query)) {
4301
- // Spread inherits parent combineWith into child branches (lucaong 7.2 behavior).
4301
+ // Spread inherits parent combineWith into child branches (MiniSearch 7.2 behavior).
4302
4302
  const options = { ...searchOptions, ...query, queries: undefined };
4303
4303
  const operator = ((_b = (_a = query.combineWith) !== null && _a !== void 0 ? _a : options.combineWith) !== null && _b !== void 0 ? _b : params.globalSearchOptions.combineWith);
4304
4304
  if (useGatedEvaluation(run, query.queries.length, operator, combinationHasWildcard(query))) {
@@ -4348,6 +4348,73 @@ function autoSuggestFromSearch(search, queryString, options = {}) {
4348
4348
  return suggestFromSearchResults(search(queryString, options));
4349
4349
  }
4350
4350
 
4351
+ /** Visit shortIds with a defined external id (holes in `externalIds` are skipped). */
4352
+ function forEachLiveShortId(nextId, externalIds, callback) {
4353
+ for (let shortId = 0; shortId < nextId; shortId++) {
4354
+ const externalId = externalIds[shortId];
4355
+ if (externalId === undefined)
4356
+ continue;
4357
+ callback(shortId, externalId);
4358
+ }
4359
+ }
4360
+
4361
+ /**
4362
+ * Build a MiniSearch `toJSON` wire snapshot (`serializationVersion: 2`) from frozen index parts.
4363
+ * Alloc-heavy (plain objects per term/field) — migration/interop only, not production persistence.
4364
+ * All input parts must belong to the same frozen index instance.
4365
+ */
4366
+ function miniSearchSnapshotFromFrozen(input) {
4367
+ const { documentCount, nextId, fieldIds, fieldCount, externalIds, fieldLengthMatrix, avgFieldLength, storedFields, index, fieldTermFlyweight, } = input;
4368
+ const documentIds = {};
4369
+ const fieldLength = {};
4370
+ const storedFieldsOut = {};
4371
+ const hasStoredFields = storedFields.kind !== 'none';
4372
+ forEachLiveShortId(nextId, externalIds, (shortId, externalId) => {
4373
+ var _a;
4374
+ const shortIdStr = String(shortId);
4375
+ documentIds[shortIdStr] = externalId;
4376
+ const lengths = new Array(fieldCount);
4377
+ const rowBase = shortId * fieldCount;
4378
+ for (let f = 0; f < fieldCount; f++) {
4379
+ lengths[f] = (_a = fieldLengthMatrix[rowBase + f]) !== null && _a !== void 0 ? _a : 0;
4380
+ }
4381
+ fieldLength[shortIdStr] = lengths;
4382
+ if (hasStoredFields) {
4383
+ storedFieldsOut[shortIdStr] = readStoredFields(storedFields, shortId);
4384
+ }
4385
+ });
4386
+ const indexEntries = [];
4387
+ for (const [term, termIndex] of index.entries()) {
4388
+ fieldTermFlyweight.bind(termIndex);
4389
+ const fieldData = {};
4390
+ for (let f = 0; f < fieldCount; f++) {
4391
+ const segment = fieldTermFlyweight.get(f);
4392
+ if (segment == null || segment.size === 0)
4393
+ continue;
4394
+ const entry = {};
4395
+ segment.forEachDoc((docId, freq) => {
4396
+ entry[String(docId)] = freq;
4397
+ });
4398
+ fieldData[String(f)] = entry;
4399
+ }
4400
+ if (Object.keys(fieldData).length > 0) {
4401
+ indexEntries.push([term, fieldData]);
4402
+ }
4403
+ }
4404
+ return {
4405
+ documentCount,
4406
+ nextId,
4407
+ documentIds,
4408
+ fieldIds,
4409
+ fieldLength,
4410
+ averageFieldLength: Array.from(avgFieldLength),
4411
+ storedFields: storedFieldsOut,
4412
+ dirtCount: 0,
4413
+ index: indexEntries,
4414
+ serializationVersion: 2,
4415
+ };
4416
+ }
4417
+
4351
4418
  function ownedIndexArray(arr) {
4352
4419
  if (arr instanceof Uint8Array)
4353
4420
  return new Uint8Array(arr);
@@ -4506,12 +4573,9 @@ class FrozenMiniSearch {
4506
4573
  tokenize: this._options.tokenize,
4507
4574
  processTerm: this._options.processTerm,
4508
4575
  indexView: createFrozenQueryIndexView(this._index, this._postings, this._fieldTermFlyweight, (callback) => {
4509
- for (let shortId = 0; shortId < this._nextId; shortId++) {
4510
- const id = this._externalIds[shortId];
4511
- if (id === undefined)
4512
- continue;
4576
+ forEachLiveShortId(this._nextId, this._externalIds, (shortId, id) => {
4513
4577
  callback(shortId, id, readStoredFields(this._storedFields, shortId));
4514
- }
4578
+ });
4515
4579
  }),
4516
4580
  aggregateContext: this._aggregateContext,
4517
4581
  };
@@ -4665,21 +4729,43 @@ class FrozenMiniSearch {
4665
4729
  return buildFrozenFromDocuments(documents, options);
4666
4730
  }
4667
4731
  /**
4668
- * Convert a lucaong MiniSearch JSON snapshot (`toJSON` / `loadJSON` wire format) into a
4669
- * frozen index. No runtime dependency on the `minisearch` package.
4732
+ * Export this index as a MiniSearch wire snapshot (`serializationVersion: 2`).
4733
+ * Use for migration or interchange with the `minisearch` package (`JSON.stringify` works via this method).
4734
+ * Not the primary persistence format — prefer {@link saveBinarySync} for production (size and load time).
4735
+ * Term order in `index` may differ from MiniSearch native `toJSON`; search scores stay equivalent.
4736
+ */
4737
+ toJSON() {
4738
+ return miniSearchSnapshotFromFrozen({
4739
+ documentCount: this._documentCount,
4740
+ nextId: this._nextId,
4741
+ fieldIds: this._fieldIds,
4742
+ fieldCount: this._fieldCount,
4743
+ externalIds: this._externalIds,
4744
+ fieldLengthMatrix: this._fieldLengthMatrix,
4745
+ avgFieldLength: this._avgFieldLength,
4746
+ storedFields: this._storedFields,
4747
+ index: this._index,
4748
+ fieldTermFlyweight: this._fieldTermFlyweight,
4749
+ });
4750
+ }
4751
+ /**
4752
+ * Build a new frozen index **from** a MiniSearch JSON snapshot string (import / migration).
4753
+ * Accepts the wire format produced by MiniSearch `toJSON` or by {@link toJSON} on this class.
4754
+ * Distinct from {@link loadBinarySync}: JSON is MiniSearch interchange, not the native frozen binary.
4755
+ * No runtime dependency on the `minisearch` package.
4670
4756
  */
4671
- static fromMiniSearchJson(json, options = {}) {
4757
+ static fromJson(json, options = {}) {
4672
4758
  return FrozenMiniSearch.fromMiniSearchSnapshot(JSON.parse(json), options);
4673
4759
  }
4674
4760
  /**
4675
- * Same as {@link fromMiniSearchJson} with a pre-parsed snapshot object.
4761
+ * Same as {@link fromJson} with a pre-parsed snapshot object.
4676
4762
  * `storedFields` are shallow-copied; callers must not mutate nested values
4677
4763
  * after load if they intend to keep the index immutable.
4678
4764
  */
4679
4765
  static fromMiniSearchSnapshot(snapshot, options = {}) {
4680
4766
  return assembleFrozenTrusted(buildFrozenAssembleParamsFromMiniSearchSnapshot(snapshot, options), 'minisearch-json');
4681
4767
  }
4682
- /** Accepts any object exposing `toJSON()` in lucaong MiniSearch snapshot shape. */
4768
+ /** Accepts any object exposing `toJSON()` in MiniSearch snapshot shape. */
4683
4769
  static fromMiniSearch(source, options = {}) {
4684
4770
  return FrozenMiniSearch.fromMiniSearchSnapshot(source.toJSON(), options);
4685
4771
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@yoch/frozenminisearch",
3
- "version": "1.0.2",
3
+ "version": "1.1.0",
4
4
  "description": "Read-only Node.js full-text search — compact frozen indexes and binary snapshots",
5
5
  "main": "dist/cjs/index.cjs",
6
6
  "module": "dist/es/index.js",
@@ -49,7 +49,6 @@
49
49
  "@types/benchmark": "^2.1.1",
50
50
  "benchmark": "^2.1.4",
51
51
  "core-js": "^3.1.4",
52
- "coveralls-next": "^6.0.1",
53
52
  "crc-32": "^1.2.2",
54
53
  "eslint": "^10.4.0",
55
54
  "fast-check": "^4.8.0",
@@ -88,6 +87,15 @@
88
87
  "<rootDir>/src/**/*.test.(ts|js)",
89
88
  "<rootDir>/dev/parity/**/*.test.(ts|js)"
90
89
  ],
90
+ "collectCoverageFrom": [
91
+ "src/**/*.{ts,js}",
92
+ "!src/**/*.test.{ts,js}"
93
+ ],
94
+ "coveragePathIgnorePatterns": [
95
+ "/benchmarks/",
96
+ "/dev/",
97
+ "/testSupport/"
98
+ ],
91
99
  "setupFilesAfterEnv": []
92
100
  },
93
101
  "scripts": {