@yoch/frozenminisearch 1.0.1 → 1.1.0

package/CHANGELOG.md

@@ -2,6 +2,28 @@
 
 ## Unreleased
 
+## v1.1.0 — `@yoch/frozenminisearch`
+
+Minor release: MiniSearch JSON wire export and clearer JSON import API. MSv5 binary format unchanged.
+
+### Added
+
+- **`toJSON()`** — export MiniSearch wire snapshots (`serializationVersion: 2`); import via `fromJson` / `fromMiniSearchSnapshot`. Production persistence remains `saveBinarySync`.
+
+### Breaking
+
+- **`fromMiniSearchJson` → `fromJson`** — rename for clearer semantics (JSON import vs binary load). Update call sites: `FrozenMiniSearch.fromMiniSearchJson(json)` → `FrozenMiniSearch.fromJson(json)`.
+
+## v1.0.2 — `@yoch/frozenminisearch`
+
+Patch release: lower retained heap when `storeFields` has one field. No API or MSv5 wire-format changes.
+
+### Improved
+
+- **Single-field `storeFields` at rest** — values live in a dense column instead of one `Record` per document (~75% less retained heap on Divina with `storeFields: ['txt']`; ~1.0 → ~0.3 MB).
+- **Binary save/load** — encode and decode skip intermediate row arrays when the in-memory layout or load `storeFields` hint allows direct wire paths (same bytes on disk).
+- **Posting slice lookups** — scoring flyweight reuses a scratch buffer instead of allocating `{ offset, length }` per lookup.
+
 ## v1.0.1 — `@yoch/frozenminisearch`
 
 Patch release: lower build-time peak memory and migration ergonomics. No API or wire-format changes.
@@ -13,7 +35,7 @@ Patch release: lower build-time peak memory and migration ergonomics. No API or
 
 ### Fixed
 
-- **Default tokenizer parity** — leading delimiter produces an empty token (e.g. `::a` → `["", "a"]`), matching lucaong `split` behaviour.
+- **Default tokenizer parity** — leading delimiter produces an empty token (e.g. `::a` → `["", "a"]`), matching MiniSearch `split` behaviour.
 - **Named export** — `FrozenMiniSearch` is exported again alongside the default export (ESM and CJS).
 
 ## v1.0.0 — `@yoch/frozenminisearch`
@@ -22,7 +44,7 @@ First stable release on npm. Frozen-only read-only search for Node.js.
 
 ### Breaking
 
-- **Binary snapshots** — `loadBinarySync` / `loadBinaryAsync` read only the current frozen binary format; re-build from lucaong JSON if an older snapshot fails to load.
+- **Binary snapshots** — `loadBinarySync` / `loadBinaryAsync` read only the current frozen binary format; re-build from MiniSearch JSON if an older snapshot fails to load.
 - **Removed `saveBinary()` / `loadBinary()`** — use `saveBinarySync` / `saveBinaryAsync` and `loadBinarySync` / `loadBinaryAsync`.
 
 ## v1.0.0-beta.0 — `@yoch/frozenminisearch`
@@ -32,16 +54,16 @@ New standalone package (frozen-only) for read-only serving workloads.
 ### Added
 
 - **`FrozenMiniSearch`** as the default export — `fromDocuments`, builder, `saveBinarySync` / `loadBinarySync`
-- **Migration loaders** — `fromMiniSearch`, `fromMiniSearchJson`, `fromMiniSearchSnapshot` (lucaong JSON wire format)
+- **Migration loaders** — `fromMiniSearch`, `fromJson`, `fromMiniSearchSnapshot` (MiniSearch JSON wire format)
 - **Modular benchmarks** — `npm run bench` with profiles `vs-reference`, `regression`, `dev`
 - **Parity suite** — `dev/parity/` vs `minisearch` npm (functional invariants)
 
 ### Removed from published API
 
 - Mutable `MiniSearch` class and `freeze()` on the fork
-- `freezeFromMiniSearch` (use `fromMiniSearchJson`)
+- `freezeFromMiniSearch` (use `fromJson`)
 - Read-only mutation stubs (`add`, `remove`, …)
 
 ### Migration
 
-- `new MiniSearch(opts).addAll(docs)` (lucaong) → `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` — see README
+- `new MiniSearch(opts).addAll(docs)` → `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` — see README

package/README.md

@@ -1,21 +1,36 @@
-# @yoch/frozenminisearch
+# FrozenMiniSearch
 
-**Read-only full-text search for Node.js** — compact frozen indexes, fast binary snapshots, and a **drop-in** search API for frozen workloads: same `search`, `autoSuggest`, scoring, and query options as [MiniSearch](https://github.com/lucaong/minisearch) by [Luca Ongaro](https://github.com/lucaong).
+[![npm version](https://img.shields.io/npm/v/@yoch/frozenminisearch.svg)](https://www.npmjs.com/package/@yoch/frozenminisearch)
+[![coverage](https://codecov.io/gh/yoch/frozenminisearch/graph/badge.svg)](https://codecov.io/gh/yoch/frozenminisearch)
+[![CI](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml/badge.svg)](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml)
+[![Socket Badge](https://socket.dev/api/badge/npm/package/@yoch/frozenminisearch)](https://socket.dev/npm/package/%40yoch%2Ffrozenminisearch)
 
-> **Current release:** `1.0.1` on npm
+**Memory-optimized, read-only full-text search for Node.js** — the same BM25, prefix/fuzzy, and `autoSuggest` API as [MiniSearch](https://github.com/lucaong/minisearch), with **up to ~98% less index RAM** on real corpora and compact binary snapshots you ship instead of JSON.
 
-**Design goal:** once an index is built or loaded, migrate with the minimum code change — package name and index construction only; serving code stays the same. No mutable `MiniSearch` class is published here; build indexes with `fromDocuments`, the incremental builder, or migrate from an existing lucaong index via `fromMiniSearchJson`.
+**Why it exists:** [MiniSearch](https://github.com/lucaong/minisearch) optimizes for a mutable in-memory index. FrozenMiniSearch optimizes for **retained heap, disk footprint, and cold load** once the corpus is fixed — packed radix postings, columnar `storeFields`, typed-array layouts, and MSv5 binary wire format instead of per-document JS objects.
+
+**Design goal:** migrate with minimal code change — package name and index construction only; serving code stays the same. Build with `fromDocuments`, the incremental builder, or `fromJson`; no mutable `MiniSearch` class is published here.
 
 ---
 
 ## Why frozen instead of MiniSearch?
 
-**Mutable** lucaong `minisearch` when documents change (`add`, `remove`, `discard`). **Frozen** when the corpus is fixed or shipped as a binary snapshot — same BM25, prefix/fuzzy, `autoSuggest`, wildcard, and `AND` / `OR` / `AND_NOT`. Parity with `minisearch@7` is validated in `dev/parity/` (scores `toBeCloseTo` precision 6).
+Choose **mutable MiniSearch** when documents change at runtime (`add`, `remove`, `discard`). Choose **frozen** when memory and snapshot size matter: fixed corpus, deploy from binary, many replicas loading the same index. Search semantics stay the same — BM25, prefix/fuzzy, `autoSuggest`, wildcard, `AND` / `OR` / `AND_NOT` — with parity vs MiniSearch 7 validated in `dev/parity/` (scores `toBeCloseTo` precision 6).
+
+### Memory-first design
+
+| Technique | What it saves |
+|-----------|---------------|
+| **Packed radix tree + flat postings** | Term dictionary and posting lists without per-entry JS wrappers |
+| **Columnar `storeFields`** | One dense column per field instead of a `Record` per document (~75% less heap for a single stored field) |
+| **MSv5 binary snapshots** | ~73–94% smaller on disk than MiniSearch JSON; faster cold load |
+| **Read-only freeze** | No mutation bookkeeping — layouts sized for serve-time, not incremental edit |
+| **Incremental builder** | Typed-array accumulators during build; lower peak heap than materializing `number[][]` per term |
 
 <!-- vs-reference:start — npm run bench:readme -->
-### Measured vs lucaong MiniSearch (reference baseline)
+### Measured vs MiniSearch (reference baseline)
 
-Same BM25 queries on identical corpora. **Frozen wins on what we optimize for**: RAM, disk, cold load, and search throughput on real workloads.
+Same BM25 queries on identical corpora. **Index RAM is the headline metric** — frozen uses a fraction of mutable heap on every scenario below; disk and cold load follow from the compact binary format.
 
 | Scenario | Docs | Index RAM¹ | Disk (binary vs JSON)² | Cold load³ | Search p50⁴ |
 |----------|-----:|------------|------------------------:|-----------:|------------:|
@@ -29,9 +44,10 @@ Same BM25 queries on identical corpora. **Frozen wins on what we optimize for**:
 
 Decomposition (Divina exact): L0 lookup ~300 ns frozen, L1 `executeQuery` ~8.3 µs, L2 full `search` ~11.6 µs (finalize ≈ 3 µs).
 
-| | lucaong `minisearch` | `@yoch/frozenminisearch` |
+| | MiniSearch | `@yoch/frozenminisearch` |
 |---|------------------------|---------------------------|
-| **Sweet spot** | Live index mutations | Fixed corpus, deploy from binary |
+| **Optimizes for** | Live mutations, flexibility | **Retained RAM**, snapshot size, cold load |
+| **Sweet spot** | Documents change at runtime | Fixed corpus, many replicas, tight memory budget |
 | **Production path** | `addAll` → `toJSON` | `fromDocuments` / `fromMiniSearch` → `saveBinarySync` → `loadBinarySync` |
 | **Typical trade-off** | Higher RAM, JSON snapshots | One-time freeze, then compact binary |
 
@@ -91,17 +107,17 @@ ESM and CommonJS are both supported (`main` → CJS, `module` → ESM).
 
 ## Drop-in
 
-For **fixed corpora** (build once, serve read-only), treat this package as a drop-in replacement for lucaong `minisearch` on the serving path.
+For **fixed corpora** (build once, serve read-only), treat this package as a drop-in replacement for MiniSearch on the serving path — same queries, far less memory per replica.
 
 **Change only:**
 
 | What | Before | After |
 |------|--------|-------|
-| Package | lucaong `minisearch` | `@yoch/frozenminisearch` |
+| Package | `minisearch` | `@yoch/frozenminisearch` |
 | Construction | `new MiniSearch(opts).addAll(docs)` | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
-| JSON snapshot | `MiniSearch.loadJSON(json)` / `toJSON()` wire format | `FrozenMiniSearch.fromMiniSearchJson(json, opts)` or `fromMiniSearchSnapshot(obj)` — no runtime dependency on lucaong `minisearch` |
+| JSON snapshot | `toJSON()` / `loadJSON()` wire format | `FrozenMiniSearch.toJSON()` / `fromJson(json, opts)` or `fromMiniSearchSnapshot(obj)` — no runtime dependency on `minisearch` |
 
-**Keep unchanged** after load: `search`, `autoSuggest`, `has`, `getStoredFields`, query options (`prefix`, `fuzzy`, `AND` / `OR` / `AND_NOT`, filters, boosts). Parity vs `minisearch@7` is enforced in `dev/parity/`.
+**Keep unchanged** after load: `search`, `autoSuggest`, `has`, `getStoredFields`, query options (`prefix`, `fuzzy`, `AND` / `OR` / `AND_NOT`, filters, boosts). Parity vs MiniSearch 7 is enforced in `dev/parity/`.
 
 **Imports** — default and named both work (ESM and CJS):
 
@@ -121,7 +137,7 @@ const { FrozenMiniSearch } = require('@yoch/frozenminisearch')
 
 ## Migration
 
-### From lucaong `minisearch` JSON
+### From MiniSearch JSON
 
 ```javascript
 import MiniSearch from 'minisearch' // build-time only
@@ -135,18 +151,18 @@ const frozen = FrozenMiniSearch.fromMiniSearch(mutable, options)
 
 // Option B — serialized index (offline / ETL)
 const json = JSON.stringify(mutable)
-const frozen2 = FrozenMiniSearch.fromMiniSearchJson(json, options)
+const frozen2 = FrozenMiniSearch.fromJson(json, options)
 ```
 
 `options.fields` must match the indexed fields in the snapshot when provided.
 
-### From lucaong `minisearch` (mutable → frozen)
+### From MiniSearch (mutable → frozen)
 
 | Before (mutable) | After (`@yoch/frozenminisearch`) |
 |------------------|----------------------------------|
 | `new MiniSearch(opts).addAll(docs)` then serve | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
-| lucaong JSON snapshot | `FrozenMiniSearch.fromMiniSearchJson(json)` or `fromMiniSearchSnapshot(obj)` |
-| `import MiniSearch from 'minisearch'` | `import FrozenMiniSearch from '@yoch/frozenminisearch'` (+ lucaong `minisearch` only if you still build mutable indexes) |
+| MiniSearch JSON snapshot | `FrozenMiniSearch.fromJson(json)` or `fromMiniSearchSnapshot(obj)` |
+| `import MiniSearch from 'minisearch'` | `import FrozenMiniSearch from '@yoch/frozenminisearch'` (+ `minisearch` only if you still build mutable indexes) |
 
 ---
 
@@ -163,13 +179,15 @@ Indexing is **not** available on a frozen instance — use `fromDocuments`, the
 
 ## Binary snapshots
 
+The primary way to **persist and ship a memory-compact index** — smaller than MiniSearch JSON and faster to load into a low-RAM serving process.
+
 ```javascript
 const buf = index.saveBinarySync()
 const loaded = FrozenMiniSearch.loadBinarySync(buf, {}) // field names embedded in snapshot
 ```
 
 - **Node ≥ 22.15.0** (zstd via `node:zlib`)
-- Snapshots produced by this package version are forward-compatible; re-build from lucaong JSON if an older binary fails to load
+- Snapshots produced by this package version are forward-compatible; re-build from MiniSearch JSON if an older binary fails to load
 - `tokenize` / `processTerm` are not stored — pass the same functions at load when customized
 
 ---
@@ -206,6 +224,6 @@ Design notes (freq adaptive, AND gating): [dev/docs/README.md](dev/docs/README.m
 See [CHANGELOG.md](./CHANGELOG.md).
 
 - **MiniSearch** — [Luca Ongaro](https://github.com/lucaong/minisearch) (MIT)
-- **@yoch/frozenminisearch** — frozen indexes, packed radix tree, compact binary snapshots
+- **@yoch/frozenminisearch** — memory-optimized frozen indexes, packed radix tree, compact binary snapshots
 
 Upstream docs: [MiniSearch](https://lucaong.github.io/minisearch/)