@yoch/frozenminisearch 1.2.2 → 1.2.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/CHANGELOG.md +11 -0
  2. package/README.md +43 -98
  3. package/dist/cjs/index.cjs +130 -15
  4. package/dist/es/index.js +130 -15
  5. package/package.json +1 -1
package/CHANGELOG.md CHANGED
@@ -2,6 +2,17 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## v1.2.3 — `@yoch/frozenminisearch`
6
+
7
+ Patch release: broad-first exact AND / AND_NOT paths, seek-based gated doc-id collection, and README benchmark copy refresh. No API or MSv5 wire-format changes.
8
+
9
+ ### Improved
10
+
11
+ - **Broad-first exact AND** — on exact-only combined queries where the first branch posting is large and a later branch is selective enough, collect the final doc-id gate by estimated posting length, then score branches in query order (parity with naive score-then-intersect unchanged).
12
+ - **Broad-first AND_NOT** — when the positive branch is large and a negated branch is comparably large, collect exclusions first and score the positive branch only on survivors.
13
+ - **Gated doc-id collection** — `DocIdGate` lazy views and seek over sorted postings when the gate is much smaller than the posting list (`scoring.ts`, `frozenPostings.ts`).
14
+ - **AND gate posting estimate** — skip upfront posting-length estimation on prefix/fuzzy AND branches; keep absolute-gate skip on the sequential path so Divina `AND+fuzzy` stays fast.
15
+
5
16
  ## v1.2.2 — `@yoch/frozenminisearch`
6
17
 
7
18
  Patch release: faster frozen AND scoring on large posting lists (gated seek + posting-ratio gate) and BM25 segment hoisting. No API or MSv5 wire-format changes.
package/README.md CHANGED
@@ -3,67 +3,44 @@
3
3
  [![npm version](https://img.shields.io/npm/v/@yoch/frozenminisearch.svg)](https://www.npmjs.com/package/@yoch/frozenminisearch)
4
4
  [![coverage](https://codecov.io/gh/yoch/frozenminisearch/graph/badge.svg)](https://codecov.io/gh/yoch/frozenminisearch)
5
5
  [![CI](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml/badge.svg)](https://github.com/yoch/frozenminisearch/actions/workflows/main.yml)
6
- [![Socket Badge](https://socket.dev/api/badge/npm/package/@yoch/frozenminisearch)](https://socket.dev/npm/package/%40yoch%2Ffrozenminisearch)
7
6
 
8
7
  [API documentation](https://yoch.github.io/frozenminisearch/)
9
8
 
10
- **Memory-optimized, read-only full-text search for Node.js** — the same BM25, prefix/fuzzy, and `autoSuggest` API as [MiniSearch](https://github.com/lucaong/minisearch), with **up to ~98% less index RAM** on real corpora and compact binary snapshots you ship instead of JSON.
9
+ **Memory-optimized, read-only full-text search for Node.js.** FrozenMiniSearch keeps the serving API close to [MiniSearch](https://github.com/lucaong/minisearch) while using compact, immutable indexes for fixed corpora.
11
10
 
12
- **Why it exists:** [MiniSearch](https://github.com/lucaong/minisearch) optimizes for a mutable in-memory index. FrozenMiniSearch optimizes for **retained heap, disk footprint, and cold load** once the corpus is fixed — packed radix postings, columnar `storeFields`, typed-array layouts, and MSv5 binary wire format instead of per-document JS objects.
11
+ Use it when your documents are built offline, shipped to production, and queried many times. In that shape, frozen indexes use **~98-99% less index RAM** in the main benchmark set, save to compact binary snapshots, and load faster than MiniSearch JSON.
13
12
 
14
- **Design goal:** migrate with minimal code change — package name and index construction only; serving code stays the same. Build with `fromDocuments`, the incremental builder, or `fromJson`; no mutable `MiniSearch` class is published here.
13
+ If you need live `add`, `remove`, or `discard`, use MiniSearch. If the corpus is fixed, this package is designed to keep the search experience familiar while making each serving replica much smaller.
15
14
 
16
15
  ---
17
16
 
18
- ## Why frozen instead of MiniSearch?
17
+ ## Why FrozenMiniSearch?
19
18
 
20
- Choose **mutable MiniSearch** when documents change at runtime (`add`, `remove`, `discard`). Choose **frozen** when memory and snapshot size matter: fixed corpus, deploy from binary, many replicas loading the same index. Search semantics stay the same — BM25, prefix/fuzzy, `autoSuggest`, wildcard, `AND` / `OR` / `AND_NOT` — with parity vs MiniSearch 7 validated in `dev/parity/` (scores `toBeCloseTo` precision 6).
19
+ FrozenMiniSearch is for the common production path where search data changes elsewhere, not inside the web process:
21
20
 
22
- ### Memory-first design
21
+ - Build or import the index offline.
22
+ - Save it as a compact binary snapshot.
23
+ - Load it in many read-only Node.js processes.
24
+ - Query with MiniSearch-style `search`, `autoSuggest`, filters, boosts, prefix/fuzzy search, wildcard, and `AND` / `OR` / `AND_NOT`.
23
25
 
24
- | Technique | What it saves |
25
- |-----------|---------------|
26
- | **Packed radix tree + flat postings** | Term dictionary and posting lists without per-entry JS wrappers |
27
- | **Columnar `storeFields`** | One dense column per field instead of a `Record` per document (~75% less heap for a single stored field) |
28
- | **MSv5 binary snapshots** | ~73–94% smaller on disk than MiniSearch JSON; faster cold load |
29
- | **Read-only freeze** | No mutation bookkeeping — layouts sized for serve-time, not incremental edit |
30
- | **Incremental builder** | Typed-array accumulators during build; lower peak heap than materializing `number[][]` per term |
26
+ Internally it replaces mutable JavaScript object graphs with packed radix postings, typed arrays, and columnar stored fields. The result is less flexible than MiniSearch, but much cheaper to keep resident.
31
27
 
32
28
  <!-- vs-reference:start — npm run bench:readme -->
33
- ### Measured vs lucaong MiniSearch (reference baseline)
29
+ ### Measured vs MiniSearch
34
30
 
35
- Same BM25 queries on identical corpora. **Frozen wins on what we optimize for**: RAM, disk, cold load, and search throughput on real workloads.
31
+ Same corpora, same BM25-style queries, MiniSearch 7.2.0 as the reference.
36
32
 
37
- | Scenario | Docs | Index RAM¹ | Disk (binary vs JSON)² | Cold load³ | Search p50⁴ |
38
- |----------|-----:|------------|------------------------:|-----------:|------------:|
39
- | Divina with storeFields | 14,097 | 0.3 vs 16.1 MB (~98% less) | ~73% less | ~65% faster | ~21% faster |
40
- | Divina index only | 14,097 | 0.2 vs 14.9 MB (~99% less) | ~77% less | ~85% faster | ~17% faster |
41
- | high-frequency terms (10k docs) | 10,000 | 0.1 vs 7.4 MB (~99% less) | ~94% less | ~90% faster | ~38% faster |
42
- | Dense numeric ids (100k, identity lookup) | 100,000 | 0.9 vs 91.3 MB (~99% less) | ~88% less | ~90% faster | ~27% faster |
43
- | Doc id Uint16 boundary (65535 docs) | 65,535 | 0.6 vs 58.6 MB (~99% less) | ~91% less | ~93% faster | ~44% faster |
33
+ | Scenario | Docs | Index RAM | Binary size | Load time | Search p50 |
34
+ |----------|-----:|-----------|------------:|----------:|-----------:|
35
+ | Divina, with stored text | 14,097 | 0.3 vs 16.1 MB (~98% less) | ~73% less | ~75% faster | ~14% faster |
36
+ | Divina, index only | 14,097 | 0.2 vs 14.9 MB (~99% less) | ~77% less | ~89% faster | ~23% faster |
37
+ | High-frequency terms | 10,000 | 4.4 vs 7.4 MB (~41% less) | ~94% less | ~91% faster | ~40% faster |
38
+ | Dense numeric ids | 100,000 | 0.9 vs 91.3 MB (~99% less) | ~88% less | ~94% faster | ~27% faster |
39
+ | Uint16 doc id boundary | 65,535 | 0.6 vs 58.6 MB (~99% less) | ~91% less | ~93% faster | ~43% faster |
44
40
 
45
- **Headline:** 26/27 query benchmarks favor frozen (paired **hrtime** protocol v2). Divina `inferno` (exact, paired p50): mutable 15.7 µs → frozen 13.4 µs (**-2 µs**, ratio 0.80).
41
+ Across this full run, frozen is faster on **25/27** search cases. Divina `inferno` (exact, paired p50): mutable 15.0 µs → frozen 13.4 µs (**-2 µs**, ratio 0.78).
46
42
 
47
- Decomposition (Divina exact): L0 lookup ~300 ns frozen, L1 `executeQuery` ~6.6 µs, L2 full `search` ~10.1 µs (finalize ≈ 3 µs).
48
-
49
- | | lucaong `minisearch` | `@yoch/frozenminisearch` |
50
- |---|------------------------|---------------------------|
51
- | **Sweet spot** | Live index mutations | Fixed corpus, deploy from binary |
52
- | **Production path** | `addAll` → `toJSON` | `fromDocuments` / `fromMiniSearch` → `saveBinarySync` → `loadBinarySync` |
53
- | **Typical trade-off** | Higher RAM, JSON snapshots | One-time freeze, then compact binary |
54
-
55
- <details>
56
- <summary><strong>How to read these numbers (limits &amp; protocol)</strong></summary>
57
-
58
- - **Captured:** 2026-06-18 · commit `d05d8e9` · Node v24.16.0 · minisearch **7.2.0** · **3** run(s)/scenario · protocol **v2** (hrtime-paired, batch target 3 ms).
59
- - ¹ **Index RAM** — `measureHeap` with `--expose-gc`, one index alive. V8 overhead is extra; treat as **trend**, not accounting. Sporadic outliers happen (e.g. index-only Divina).
60
- - ² **Disk** — `JSON.stringify(mutable)` vs `saveBinarySync`.
61
- - ³ **Cold load** — median wall time to searchable index after read from disk format.
62
- - ⁴ **Search p50** — paired mutable/frozen samples per iteration; sub-0.1 ms baselines reported in **µs** in full reports. Fast queries use **50** iterations, others **20**.
63
- - **Not shown:** mutable `add`/`remove` (frozen is read-only by design). Freeze time is offline — see full suite for build metrics.
64
- - **Reproduce:** `npm run bench -- run --profile=vs-reference` · **Update this block:** `npm run bench:readme` after refreshing `benchmarks/baselines/reference.json`.
65
-
66
- </details>
43
+ Numbers are from `benchmarks/baselines/reference.json`, captured 2026-06-18 on Node v24.16.0, 3 runs per scenario. Heap is measured with one index alive and should be read as a trend, not exact accounting.
67
44
  <!-- vs-reference:end -->
68
45
 
69
46
  ---
@@ -74,8 +51,6 @@ Decomposition (Divina exact): L0 lookup ~300 ns frozen, L1 `executeQuery` ~6.6
74
51
  npm install @yoch/frozenminisearch
75
52
  ```
76
53
 
77
- **Build from documents:**
78
-
79
54
  ```javascript
80
55
  import FrozenMiniSearch from '@yoch/frozenminisearch'
81
56
 
@@ -89,7 +64,7 @@ const buf = index.saveBinarySync()
89
64
  const loaded = FrozenMiniSearch.loadBinarySync(buf, options)
90
65
  ```
91
66
 
92
- **Incremental builder:**
67
+ For larger imports, use the incremental builder:
93
68
 
94
69
  ```javascript
95
70
  import FrozenMiniSearch, {
@@ -106,21 +81,11 @@ ESM and CommonJS are both supported (`main` → CJS, `module` → ESM).
106
81
 
107
82
  ---
108
83
 
109
- ## Drop-in
110
-
111
- For **fixed corpora** (build once, serve read-only), treat this package as a drop-in replacement for MiniSearch on the serving path — same queries, far less memory per replica.
112
-
113
- **Change only:**
114
-
115
- | What | Before | After |
116
- |------|--------|-------|
117
- | Package | `minisearch` | `@yoch/frozenminisearch` |
118
- | Construction | `new MiniSearch(opts).addAll(docs)` | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
119
- | JSON snapshot | `toJSON()` / `loadJSON()` wire format | `FrozenMiniSearch.toJSON()` / `fromJson(json, opts)` or `fromMiniSearchSnapshot(obj)` — no runtime dependency on `minisearch` |
84
+ ## Migration
120
85
 
121
- **Keep unchanged** after load: `search`, `autoSuggest`, `has`, `getStoredFields`, query options (`prefix`, `fuzzy`, `AND` / `OR` / `AND_NOT`, filters, boosts). Parity vs MiniSearch 7 is enforced in `dev/parity/`.
86
+ For fixed corpora, most serving code can stay the same. Change how the index is built or loaded, then keep calling `search`, `autoSuggest`, `has`, and `getStoredFields`.
122
87
 
123
- **Imports** — default and named both work (ESM and CJS):
88
+ Default and named imports both work:
124
89
 
125
90
  ```javascript
126
91
  // ESM
@@ -132,38 +97,28 @@ const FrozenMiniSearch = require('@yoch/frozenminisearch')
132
97
  const { FrozenMiniSearch } = require('@yoch/frozenminisearch')
133
98
  ```
134
99
 
135
- **Intentionally not drop-in:** live `add` / `remove` / `discard` (frozen is read-only); browser builds; custom `tokenize` / `processTerm` are not stored in JSON or binary snapshots — pass the same functions at load when you customized them.
100
+ Build directly:
136
101
 
137
- ---
102
+ ```javascript
103
+ import FrozenMiniSearch from '@yoch/frozenminisearch'
138
104
 
139
- ## Migration
105
+ const frozen = FrozenMiniSearch.fromDocuments(documents, options)
106
+ ```
140
107
 
141
- ### From MiniSearch JSON
108
+ Or freeze an existing MiniSearch index:
142
109
 
143
110
  ```javascript
144
- import MiniSearch from 'minisearch' // build-time only
111
+ import MiniSearch from 'minisearch'
145
112
  import FrozenMiniSearch from '@yoch/frozenminisearch'
146
113
 
147
114
  const mutable = new MiniSearch(options)
148
115
  mutable.addAll(documents)
149
116
 
150
- // Option A — live instance
151
117
  const frozen = FrozenMiniSearch.fromMiniSearch(mutable, options)
152
-
153
- // Option B — serialized index (offline / ETL)
154
- const json = JSON.stringify(mutable)
155
- const frozen2 = FrozenMiniSearch.fromJson(json, options)
118
+ const fromJson = FrozenMiniSearch.fromJson(JSON.stringify(mutable), options)
156
119
  ```
157
120
 
158
- `options.fields` must match the indexed fields in the snapshot when provided.
159
-
160
- ### From MiniSearch (mutable → frozen)
161
-
162
- | Before (mutable) | After (`@yoch/frozenminisearch`) |
163
- |------------------|----------------------------------|
164
- | `new MiniSearch(opts).addAll(docs)` then serve | `FrozenMiniSearch.fromDocuments(docs, opts)` or `fromMiniSearch(mutable, opts)` |
165
- | MiniSearch JSON snapshot | `FrozenMiniSearch.fromJson(json)` or `fromMiniSearchSnapshot(obj)` |
166
- | `import MiniSearch from 'minisearch'` | `import FrozenMiniSearch from '@yoch/frozenminisearch'` (+ `minisearch` only if you still build mutable indexes) |
121
+ MiniSearch is only needed if you still build mutable indexes. Frozen instances do not support live `add`, `remove`, or `discard`.
167
122
 
168
123
  ---
169
124
 
@@ -174,13 +129,13 @@ const frozen2 = FrozenMiniSearch.fromJson(json, options)
174
129
  - `has(id)`, `getStoredFields(id)`
175
130
  - `saveBinarySync` / `loadBinarySync` / async variants
176
131
 
177
- Indexing is **not** available on a frozen instance — use `fromDocuments`, the builder, `fromMiniSearch*`, or `loadBinary*`.
132
+ Custom `tokenize` and `processTerm` functions are not stored in snapshots; pass the same functions again when loading.
178
133
 
179
134
  ---
180
135
 
181
136
  ## Binary snapshots
182
137
 
183
- The primary way to **persist and ship a memory-compact index** — smaller than MiniSearch JSON and faster to load into a low-RAM serving process.
138
+ Binary snapshots are the preferred production format.
184
139
 
185
140
  ```javascript
186
141
  const buf = index.saveBinarySync()
@@ -188,12 +143,8 @@ const loaded = FrozenMiniSearch.loadBinarySync(buf, {}) // field names embedded
188
143
  ```
189
144
 
190
145
  - **Node ≥ 20**
191
- - Default snapshot compression (`compression: 'auto'`, one pass):
192
- - payloads under 64 B stay raw
193
- - `zstd` on Node 22.15+ when it strictly shrinks the payload
194
- - otherwise `zlib` on Node 20–22.14 when it strictly shrinks the payload
195
- - otherwise `raw` (uncompressed)
196
- - Explicit snapshot compression always writes the chosen codec, even when compression would not shrink the payload (useful for portability):
146
+ - `compression: 'auto'` chooses `zstd` on Node 22.15+, otherwise `zlib`, and falls back to raw when compression does not help.
147
+ - Use explicit compression when you need a portable artifact:
197
148
 
198
149
  ```javascript
199
150
  const portable = index.saveBinarySync({ compression: 'zlib' })
@@ -201,11 +152,7 @@ const uncompressed = index.saveBinarySync({ compression: 'raw' })
201
152
  const bestRatio = index.saveBinarySync({ compression: 'zstd' }) // Node 22.15+
202
153
  ```
203
154
 
204
- - Snapshot readability depends on the embedded codec:
205
- - `raw` and `zlib` snapshots load on Node 20+
206
- - `zstd` snapshots require Node 22.15+
207
- - Snapshots produced by this package version are forward-compatible; re-build from MiniSearch JSON if an older binary fails to load
208
- - `tokenize` / `processTerm` are not stored — pass the same functions at load when customized
155
+ Raw and zlib snapshots load on Node 20+. zstd snapshots require Node 22.15+.
209
156
 
210
157
  ---
211
158
 
@@ -215,8 +162,8 @@ See [benchmarks/README.md](benchmarks/README.md).
215
162
 
216
163
  ```bash
217
164
  npm run bench -- run --profile=vs-reference # compare frozen vs minisearch
218
- npm run bench:diff # regression vs reference.json
219
- npm run bench:readme # refresh comparison table above
165
+ npm run bench:diff # regression vs reference.json
166
+ npm run bench:readme -- --from=benchmarks/baselines/latest.json
220
167
  ```
221
168
 
222
169
  ---
@@ -230,9 +177,7 @@ yarn build
230
177
  node scripts/verify-npm-pack.cjs
231
178
  ```
232
179
 
233
- Parity tests import `minisearch` as a devDependency (reference). Optional upstream clone: `git submodule update --init vendor/minisearch`.
234
-
235
- Design notes (freq adaptive, AND gating): [dev/docs/README.md](dev/docs/README.md).
180
+ Parity tests compare against MiniSearch 7. Longer notes and performance work live under [dev/docs/README.md](dev/docs/README.md) and [benchmarks/README.md](benchmarks/README.md).
236
181
 
237
182
  ---
238
183
 
@@ -241,6 +186,6 @@ Design notes (freq adaptive, AND gating): [dev/docs/README.md](dev/docs/README.m
241
186
  See [CHANGELOG.md](./CHANGELOG.md).
242
187
 
243
188
  - **MiniSearch** — [Luca Ongaro](https://github.com/lucaong/minisearch) (MIT)
244
- - **@yoch/frozenminisearch** — memory-optimized frozen indexes, packed radix tree, compact binary snapshots
189
+ - **@yoch/frozenminisearch** — memory-optimized frozen indexes and compact binary snapshots
245
190
 
246
191
  Upstream docs: [MiniSearch](https://lucaong.github.io/minisearch/)
package/dist/cjs/index.cjs CHANGED
@@ -37,6 +37,21 @@ function gateIsSelectiveEnough(gateSize, documentCount, limits = DEFAULT_AND_GAT
37
37
  }
38
38
  return false;
39
39
  }
40
+ /** True when passing gate as allowedDocs can skip docs vs scanning the full branch posting. */
41
+ function gateFilterShrinksScan(gateSize, postingListLength) {
42
+ return postingListLength > gateSize;
43
+ }
44
+ /**
45
+ * Whether to pass the AND gate as allowedDocs to the next branch (perf only; scores unchanged if false).
46
+ * Distinct from gateIsSelectiveEnough: a selective gate may still be too large to filter a short posting.
47
+ */
48
+ function shouldPassGateAsAllowedDocs(selective, gateSize, postingListLength) {
49
+ if (!selective || gateSize === 0)
50
+ return false;
51
+ if (postingListLength == null || postingListLength <= 0)
52
+ return false;
53
+ return gateFilterShrinksScan(gateSize, postingListLength);
54
+ }
40
55
 
41
56
  const MAX_FREQ = 65535;
42
57
  function readDocId(docIds, index) {
@@ -2138,6 +2153,16 @@ function createFrozenFieldTermFlyweight(layout) {
2138
2153
  return flyweight;
2139
2154
  }
2140
2155
  function collectDocIdsFromFrozenSegment(allDocIds, offset, length, context, docIds, allowedDocs) {
2156
+ if (allowedDocs != null && shouldSeekAllowedDocs(allowedDocs.size, length)) {
2157
+ for (const docId of allowedDocs) {
2158
+ if (context.isDocActive != null && !context.isDocActive(docId))
2159
+ continue;
2160
+ if (findDocIndexInSortedSegment(allDocIds, offset, length, docId) >= 0) {
2161
+ docIds.add(docId);
2162
+ }
2163
+ }
2164
+ return;
2165
+ }
2141
2166
  for (let i = 0; i < length; i++) {
2142
2167
  const docId = readDocId(allDocIds, offset + i);
2143
2168
  if (context.isDocActive != null && !context.isDocActive(docId))
@@ -4127,8 +4152,18 @@ function buildFrozenParamsFromDocuments(documents, options) {
4127
4152
  function useGatedEvaluation(run, branchCount, operator, hasWildcard) {
4128
4153
  return shouldUseGatedEvaluation(branchCount, operator, hasWildcard);
4129
4154
  }
4130
- function docIdsFromResult(result) {
4131
- return new Set(result.keys());
4155
+ function gateFromResult(result) {
4156
+ return {
4157
+ get size() {
4158
+ return result.size;
4159
+ },
4160
+ has(docId) {
4161
+ return result.has(docId);
4162
+ },
4163
+ [Symbol.iterator]() {
4164
+ return result.keys();
4165
+ },
4166
+ };
4132
4167
  }
4133
4168
  function isQueryCombination(query) {
4134
4169
  return typeof query === 'object'
@@ -4200,6 +4235,7 @@ function normalizeStringQuery(query, searchOptions, params) {
4200
4235
  function lazyIndexedTerm(indexView, termIndex) {
4201
4236
  return { kind: 'lazy', resolve: () => indexView.resolveTermByIndex(termIndex) };
4202
4237
  }
4238
+ const TWO_PHASE_AND_NOT_MIN_FRACTION = 0.5;
4203
4239
  function forEachQuerySpecTermRef(query, normalized, params, visit) {
4204
4240
  const { indexView } = params;
4205
4241
  const { options } = normalized;
@@ -4330,6 +4366,70 @@ function subtractDocIdsFromResult(result, excludedDocIds) {
4330
4366
  for (const docId of excludedDocIds)
4331
4367
  result.delete(docId);
4332
4368
  }
4369
+ function twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength) {
4370
+ if (!allowTwoPhase || estimateBranchPostingLength == null)
4371
+ return undefined;
4372
+ const lengths = new Array(branches.length);
4373
+ for (let i = 0; i < branches.length; i++) {
4374
+ lengths[i] = estimateBranchPostingLength(branches[i]);
4375
+ }
4376
+ return lengths;
4377
+ }
4378
+ function shouldUseTwoPhaseAnd(branchPostingLengths, allowedDocs) {
4379
+ if (branchPostingLengths.length <= 1)
4380
+ return false;
4381
+ const firstLength = branchPostingLengths[0];
4382
+ const effectiveFirstLength = allowedDocs == null
4383
+ ? firstLength
4384
+ : Math.min(firstLength, allowedDocs.size);
4385
+ if (effectiveFirstLength < DEFAULT_POSTING_GATE_MIN_LENGTH)
4386
+ return false;
4387
+ const targetLength = effectiveFirstLength >>> DEFAULT_POSTING_GATE_RATIO_SHIFT;
4388
+ for (let i = 1; i < branchPostingLengths.length; i++) {
4389
+ const len = branchPostingLengths[i];
4390
+ if (len > 0 && len <= targetLength)
4391
+ return true;
4392
+ }
4393
+ return false;
4394
+ }
4395
+ function shouldUseTwoPhaseAndNot(branchPostingLengths, allowedDocs, documentCount) {
4396
+ if (branchPostingLengths.length <= 1)
4397
+ return false;
4398
+ const firstLength = branchPostingLengths[0];
4399
+ const effectiveFirstLength = allowedDocs == null
4400
+ ? firstLength
4401
+ : Math.min(firstLength, allowedDocs.size);
4402
+ const largeThreshold = Math.max(DEFAULT_POSTING_GATE_MIN_LENGTH, Math.floor(documentCount * TWO_PHASE_AND_NOT_MIN_FRACTION));
4403
+ if (effectiveFirstLength < largeThreshold)
4404
+ return false;
4405
+ for (let i = 1; i < branchPostingLengths.length; i++) {
4406
+ if (branchPostingLengths[i] >= largeThreshold)
4407
+ return true;
4408
+ }
4409
+ return false;
4410
+ }
4411
+ function executeAndWithFinalGate(branches, finalGate, executeBranch) {
4412
+ if (finalGate.size === 0)
4413
+ return new Map();
4414
+ let result = executeBranch(branches[0], finalGate);
4415
+ for (let i = 1; i < branches.length; i++) {
4416
+ if (result.size === 0)
4417
+ return result;
4418
+ result = combineResults([result, executeBranch(branches[i], finalGate)], AND);
4419
+ }
4420
+ return result;
4421
+ }
4422
+ function collectAndDocIdsByEstimatedLength(branches, branchPostingLengths, collectBranch, allowedDocs) {
4423
+ const order = branches.map((_, i) => i);
4424
+ order.sort((a, b) => branchPostingLengths[a] - branchPostingLengths[b] || a - b);
4425
+ const docIds = collectBranch(branches[order[0]], allowedDocs);
4426
+ for (let i = 1; i < order.length; i++) {
4427
+ if (docIds.size === 0)
4428
+ return docIds;
4429
+ intersectDocIdsInPlace(docIds, collectBranch(branches[order[i]], docIds));
4430
+ }
4431
+ return docIds;
4432
+ }
4333
4433
  function collectCombinedDocIds(branches, operator, collectBranch, allowedDocs) {
4334
4434
  if (branches.length === 0)
4335
4435
  return new Set();
@@ -4359,11 +4459,12 @@ function collectCombinedDocIds(branches, operator, collectBranch, allowedDocs) {
4359
4459
  throw new Error(`Invalid combination operator: ${operator}`);
4360
4460
  }
4361
4461
  /**
4362
- * AND: score every branch (with optional docId gate on later branches), then intersect scores.
4462
+ * AND: normally score left-to-right with optional docId gates; for broad-first selective
4463
+ * exact queries, collect the final gate first, then score branches in original order.
4363
4464
  * AND_NOT: score the positive branch only; negated branches are collected as docId sets and
4364
- * subtracted without scoring (avoids term materialization on excluded branches).
4465
+ * subtracted without scoring. Large exact exclusions may collect survivors before positive scoring.
4365
4466
  */
4366
- function executeCombinedBranches(branches, operator, params, executeBranch, collectBranch, allowedDocs, run, estimateBranchPostingLength) {
4467
+ function executeCombinedBranches(branches, operator, params, executeBranch, collectBranch, allowedDocs, run, estimateBranchPostingLength, allowTwoPhase = false) {
4367
4468
  var _a;
4368
4469
  if (branches.length === 0)
4369
4470
  return new Map();
@@ -4371,9 +4472,14 @@ function executeCombinedBranches(branches, operator, params, executeBranch, coll
4371
4472
  if (op === 'or') {
4372
4473
  return combineResults(branches.map(branch => executeBranch(branch, allowedDocs)), operator);
4373
4474
  }
4374
- let result = executeBranch(branches[0], allowedDocs);
4375
- let gate = docIdsFromResult(result);
4376
4475
  if (op === 'and') {
4476
+ const branchPostingLengths = twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength);
4477
+ if (branchPostingLengths != null && shouldUseTwoPhaseAnd(branchPostingLengths, allowedDocs)) {
4478
+ const finalGate = collectAndDocIdsByEstimatedLength(branches, branchPostingLengths, collectBranch, allowedDocs);
4479
+ return executeAndWithFinalGate(branches, finalGate, executeBranch);
4480
+ }
4481
+ let result = executeBranch(branches[0], allowedDocs);
4482
+ let gate = gateFromResult(result);
4377
4483
  const limits = void 0 ;
4378
4484
  const documentCount = params.aggregateContext.documentCount;
4379
4485
  const postingGatePolicy = (_a = void 0 ) !== null && _a !== void 0 ? _a : DEFAULT_POSTING_GATE_POLICY;
@@ -4381,21 +4487,30 @@ function executeCombinedBranches(branches, operator, params, executeBranch, coll
4381
4487
  for (let i = 1; i < branches.length; i++) {
4382
4488
  if (gate.size === 0)
4383
4489
  return result;
4384
- const ratioPath = gate.size > maxGateSize;
4385
- const postingListLength = ratioPath
4386
- ? estimateBranchPostingLength === null || estimateBranchPostingLength === void 0 ? void 0 : estimateBranchPostingLength(branches[i])
4387
- : undefined;
4490
+ const absoluteSelective = gate.size <= maxGateSize;
4491
+ const postingListLength = absoluteSelective
4492
+ ? undefined
4493
+ : estimateBranchPostingLength === null || estimateBranchPostingLength === void 0 ? void 0 : estimateBranchPostingLength(branches[i]);
4388
4494
  const selective = gateIsSelectiveEnough(gate.size, documentCount, limits, postingListLength, postingGatePolicy);
4389
- const branchAllowed = selective ? gate : allowedDocs;
4495
+ const branchAllowed = absoluteSelective || shouldPassGateAsAllowedDocs(selective, gate.size, postingListLength)
4496
+ ? gate
4497
+ : allowedDocs;
4390
4498
  result = combineResults([result, executeBranch(branches[i], branchAllowed)], AND);
4391
- gate = docIdsFromResult(result);
4499
+ gate = gateFromResult(result);
4392
4500
  }
4393
4501
  return result;
4394
4502
  }
4395
4503
  if (op === 'and_not') {
4504
+ const branchPostingLengths = twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength);
4505
+ if (branchPostingLengths != null && shouldUseTwoPhaseAndNot(branchPostingLengths, allowedDocs, params.aggregateContext.documentCount)) {
4506
+ const finalGate = collectCombinedDocIds(branches, operator, collectBranch, allowedDocs);
4507
+ return finalGate.size === 0 ? new Map() : executeBranch(branches[0], finalGate);
4508
+ }
4509
+ const result = executeBranch(branches[0], allowedDocs);
4510
+ let gate = gateFromResult(result);
4396
4511
  for (let i = 1; i < branches.length; i++) {
4397
4512
  subtractDocIdsFromResult(result, collectBranch(branches[i], gate));
4398
- gate = docIdsFromResult(result);
4513
+ gate = gateFromResult(result);
4399
4514
  }
4400
4515
  return result;
4401
4516
  }
@@ -4491,7 +4606,7 @@ function executeQueryInternal(query, searchOptions, params, allowedDocs, run) {
4491
4606
  const { specs, operator } = normalized;
4492
4607
  const combineWith = (operator !== null && operator !== void 0 ? operator : params.globalSearchOptions.combineWith);
4493
4608
  if (useGatedEvaluation(run, specs.length, combineWith, false)) {
4494
- return executeCombinedBranches(specs, combineWith, params, (spec, branchAllowed) => executeQuerySpecInternal(spec, normalized, params, branchAllowed), (spec, branchAllowed) => collectDocIdsForQuerySpec(spec, normalized, params, branchAllowed), allowedDocs, run, spec => estimateMaxPostingLengthForQuerySpec(spec, normalized, params));
4609
+ return executeCombinedBranches(specs, combineWith, params, (spec, branchAllowed) => executeQuerySpecInternal(spec, normalized, params, branchAllowed), (spec, branchAllowed) => collectDocIdsForQuerySpec(spec, normalized, params, branchAllowed), allowedDocs, run, spec => estimateMaxPostingLengthForQuerySpec(spec, normalized, params), specs.every(spec => !spec.prefix && !spec.fuzzy));
4495
4610
  }
4496
4611
  const results = specs.map(spec => executeQuerySpecInternal(spec, normalized, params, allowedDocs));
4497
4612
  return combineResults(results, combineWith);
package/dist/es/index.js CHANGED
@@ -33,6 +33,21 @@ function gateIsSelectiveEnough(gateSize, documentCount, limits = DEFAULT_AND_GAT
33
33
  }
34
34
  return false;
35
35
  }
36
+ /** True when passing gate as allowedDocs can skip docs vs scanning the full branch posting. */
37
+ function gateFilterShrinksScan(gateSize, postingListLength) {
38
+ return postingListLength > gateSize;
39
+ }
40
+ /**
41
+ * Whether to pass the AND gate as allowedDocs to the next branch (perf only; scores unchanged if false).
42
+ * Distinct from gateIsSelectiveEnough: a selective gate may still be too large to filter a short posting.
43
+ */
44
+ function shouldPassGateAsAllowedDocs(selective, gateSize, postingListLength) {
45
+ if (!selective || gateSize === 0)
46
+ return false;
47
+ if (postingListLength == null || postingListLength <= 0)
48
+ return false;
49
+ return gateFilterShrinksScan(gateSize, postingListLength);
50
+ }
36
51
 
37
52
  const MAX_FREQ = 65535;
38
53
  function readDocId(docIds, index) {
@@ -2134,6 +2149,16 @@ function createFrozenFieldTermFlyweight(layout) {
2134
2149
  return flyweight;
2135
2150
  }
2136
2151
  function collectDocIdsFromFrozenSegment(allDocIds, offset, length, context, docIds, allowedDocs) {
2152
+ if (allowedDocs != null && shouldSeekAllowedDocs(allowedDocs.size, length)) {
2153
+ for (const docId of allowedDocs) {
2154
+ if (context.isDocActive != null && !context.isDocActive(docId))
2155
+ continue;
2156
+ if (findDocIndexInSortedSegment(allDocIds, offset, length, docId) >= 0) {
2157
+ docIds.add(docId);
2158
+ }
2159
+ }
2160
+ return;
2161
+ }
2137
2162
  for (let i = 0; i < length; i++) {
2138
2163
  const docId = readDocId(allDocIds, offset + i);
2139
2164
  if (context.isDocActive != null && !context.isDocActive(docId))
@@ -4123,8 +4148,18 @@ function buildFrozenParamsFromDocuments(documents, options) {
4123
4148
  function useGatedEvaluation(run, branchCount, operator, hasWildcard) {
4124
4149
  return shouldUseGatedEvaluation(branchCount, operator, hasWildcard);
4125
4150
  }
4126
- function docIdsFromResult(result) {
4127
- return new Set(result.keys());
4151
+ function gateFromResult(result) {
4152
+ return {
4153
+ get size() {
4154
+ return result.size;
4155
+ },
4156
+ has(docId) {
4157
+ return result.has(docId);
4158
+ },
4159
+ [Symbol.iterator]() {
4160
+ return result.keys();
4161
+ },
4162
+ };
4128
4163
  }
4129
4164
  function isQueryCombination(query) {
4130
4165
  return typeof query === 'object'
@@ -4196,6 +4231,7 @@ function normalizeStringQuery(query, searchOptions, params) {
4196
4231
  function lazyIndexedTerm(indexView, termIndex) {
4197
4232
  return { kind: 'lazy', resolve: () => indexView.resolveTermByIndex(termIndex) };
4198
4233
  }
4234
+ const TWO_PHASE_AND_NOT_MIN_FRACTION = 0.5;
4199
4235
  function forEachQuerySpecTermRef(query, normalized, params, visit) {
4200
4236
  const { indexView } = params;
4201
4237
  const { options } = normalized;
@@ -4326,6 +4362,70 @@ function subtractDocIdsFromResult(result, excludedDocIds) {
4326
4362
  for (const docId of excludedDocIds)
4327
4363
  result.delete(docId);
4328
4364
  }
4365
+ function twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength) {
4366
+ if (!allowTwoPhase || estimateBranchPostingLength == null)
4367
+ return undefined;
4368
+ const lengths = new Array(branches.length);
4369
+ for (let i = 0; i < branches.length; i++) {
4370
+ lengths[i] = estimateBranchPostingLength(branches[i]);
4371
+ }
4372
+ return lengths;
4373
+ }
4374
+ function shouldUseTwoPhaseAnd(branchPostingLengths, allowedDocs) {
4375
+ if (branchPostingLengths.length <= 1)
4376
+ return false;
4377
+ const firstLength = branchPostingLengths[0];
4378
+ const effectiveFirstLength = allowedDocs == null
4379
+ ? firstLength
4380
+ : Math.min(firstLength, allowedDocs.size);
4381
+ if (effectiveFirstLength < DEFAULT_POSTING_GATE_MIN_LENGTH)
4382
+ return false;
4383
+ const targetLength = effectiveFirstLength >>> DEFAULT_POSTING_GATE_RATIO_SHIFT;
4384
+ for (let i = 1; i < branchPostingLengths.length; i++) {
4385
+ const len = branchPostingLengths[i];
4386
+ if (len > 0 && len <= targetLength)
4387
+ return true;
4388
+ }
4389
+ return false;
4390
+ }
4391
+ function shouldUseTwoPhaseAndNot(branchPostingLengths, allowedDocs, documentCount) {
4392
+ if (branchPostingLengths.length <= 1)
4393
+ return false;
4394
+ const firstLength = branchPostingLengths[0];
4395
+ const effectiveFirstLength = allowedDocs == null
4396
+ ? firstLength
4397
+ : Math.min(firstLength, allowedDocs.size);
4398
+ const largeThreshold = Math.max(DEFAULT_POSTING_GATE_MIN_LENGTH, Math.floor(documentCount * TWO_PHASE_AND_NOT_MIN_FRACTION));
4399
+ if (effectiveFirstLength < largeThreshold)
4400
+ return false;
4401
+ for (let i = 1; i < branchPostingLengths.length; i++) {
4402
+ if (branchPostingLengths[i] >= largeThreshold)
4403
+ return true;
4404
+ }
4405
+ return false;
4406
+ }
4407
+ function executeAndWithFinalGate(branches, finalGate, executeBranch) {
4408
+ if (finalGate.size === 0)
4409
+ return new Map();
4410
+ let result = executeBranch(branches[0], finalGate);
4411
+ for (let i = 1; i < branches.length; i++) {
4412
+ if (result.size === 0)
4413
+ return result;
4414
+ result = combineResults([result, executeBranch(branches[i], finalGate)], AND);
4415
+ }
4416
+ return result;
4417
+ }
4418
+ function collectAndDocIdsByEstimatedLength(branches, branchPostingLengths, collectBranch, allowedDocs) {
4419
+ const order = branches.map((_, i) => i);
4420
+ order.sort((a, b) => branchPostingLengths[a] - branchPostingLengths[b] || a - b);
4421
+ const docIds = collectBranch(branches[order[0]], allowedDocs);
4422
+ for (let i = 1; i < order.length; i++) {
4423
+ if (docIds.size === 0)
4424
+ return docIds;
4425
+ intersectDocIdsInPlace(docIds, collectBranch(branches[order[i]], docIds));
4426
+ }
4427
+ return docIds;
4428
+ }
4329
4429
  function collectCombinedDocIds(branches, operator, collectBranch, allowedDocs) {
4330
4430
  if (branches.length === 0)
4331
4431
  return new Set();
@@ -4355,11 +4455,12 @@ function collectCombinedDocIds(branches, operator, collectBranch, allowedDocs) {
4355
4455
  throw new Error(`Invalid combination operator: ${operator}`);
4356
4456
  }
4357
4457
  /**
4358
- * AND: score every branch (with optional docId gate on later branches), then intersect scores.
4458
+ * AND: normally score left-to-right with optional docId gates; for broad-first selective
4459
+ * exact queries, collect the final gate first, then score branches in original order.
4359
4460
  * AND_NOT: score the positive branch only; negated branches are collected as docId sets and
4360
- * subtracted without scoring (avoids term materialization on excluded branches).
4461
+ * subtracted without scoring. Large exact exclusions may collect survivors before positive scoring.
4361
4462
  */
4362
- function executeCombinedBranches(branches, operator, params, executeBranch, collectBranch, allowedDocs, run, estimateBranchPostingLength) {
4463
+ function executeCombinedBranches(branches, operator, params, executeBranch, collectBranch, allowedDocs, run, estimateBranchPostingLength, allowTwoPhase = false) {
4363
4464
  var _a;
4364
4465
  if (branches.length === 0)
4365
4466
  return new Map();
@@ -4367,9 +4468,14 @@ function executeCombinedBranches(branches, operator, params, executeBranch, coll
4367
4468
  if (op === 'or') {
4368
4469
  return combineResults(branches.map(branch => executeBranch(branch, allowedDocs)), operator);
4369
4470
  }
4370
- let result = executeBranch(branches[0], allowedDocs);
4371
- let gate = docIdsFromResult(result);
4372
4471
  if (op === 'and') {
4472
+ const branchPostingLengths = twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength);
4473
+ if (branchPostingLengths != null && shouldUseTwoPhaseAnd(branchPostingLengths, allowedDocs)) {
4474
+ const finalGate = collectAndDocIdsByEstimatedLength(branches, branchPostingLengths, collectBranch, allowedDocs);
4475
+ return executeAndWithFinalGate(branches, finalGate, executeBranch);
4476
+ }
4477
+ let result = executeBranch(branches[0], allowedDocs);
4478
+ let gate = gateFromResult(result);
4373
4479
  const limits = void 0 ;
4374
4480
  const documentCount = params.aggregateContext.documentCount;
4375
4481
  const postingGatePolicy = (_a = void 0 ) !== null && _a !== void 0 ? _a : DEFAULT_POSTING_GATE_POLICY;
@@ -4377,21 +4483,30 @@ function executeCombinedBranches(branches, operator, params, executeBranch, coll
4377
4483
  for (let i = 1; i < branches.length; i++) {
4378
4484
  if (gate.size === 0)
4379
4485
  return result;
4380
- const ratioPath = gate.size > maxGateSize;
4381
- const postingListLength = ratioPath
4382
- ? estimateBranchPostingLength === null || estimateBranchPostingLength === void 0 ? void 0 : estimateBranchPostingLength(branches[i])
4383
- : undefined;
4486
+ const absoluteSelective = gate.size <= maxGateSize;
4487
+ const postingListLength = absoluteSelective
4488
+ ? undefined
4489
+ : estimateBranchPostingLength === null || estimateBranchPostingLength === void 0 ? void 0 : estimateBranchPostingLength(branches[i]);
4384
4490
  const selective = gateIsSelectiveEnough(gate.size, documentCount, limits, postingListLength, postingGatePolicy);
4385
- const branchAllowed = selective ? gate : allowedDocs;
4491
+ const branchAllowed = absoluteSelective || shouldPassGateAsAllowedDocs(selective, gate.size, postingListLength)
4492
+ ? gate
4493
+ : allowedDocs;
4386
4494
  result = combineResults([result, executeBranch(branches[i], branchAllowed)], AND);
4387
- gate = docIdsFromResult(result);
4495
+ gate = gateFromResult(result);
4388
4496
  }
4389
4497
  return result;
4390
4498
  }
4391
4499
  if (op === 'and_not') {
4500
+ const branchPostingLengths = twoPhasePostingLengths(branches, allowTwoPhase, estimateBranchPostingLength);
4501
+ if (branchPostingLengths != null && shouldUseTwoPhaseAndNot(branchPostingLengths, allowedDocs, params.aggregateContext.documentCount)) {
4502
+ const finalGate = collectCombinedDocIds(branches, operator, collectBranch, allowedDocs);
4503
+ return finalGate.size === 0 ? new Map() : executeBranch(branches[0], finalGate);
4504
+ }
4505
+ const result = executeBranch(branches[0], allowedDocs);
4506
+ let gate = gateFromResult(result);
4392
4507
  for (let i = 1; i < branches.length; i++) {
4393
4508
  subtractDocIdsFromResult(result, collectBranch(branches[i], gate));
4394
- gate = docIdsFromResult(result);
4509
+ gate = gateFromResult(result);
4395
4510
  }
4396
4511
  return result;
4397
4512
  }
@@ -4487,7 +4602,7 @@ function executeQueryInternal(query, searchOptions, params, allowedDocs, run) {
4487
4602
  const { specs, operator } = normalized;
4488
4603
  const combineWith = (operator !== null && operator !== void 0 ? operator : params.globalSearchOptions.combineWith);
4489
4604
  if (useGatedEvaluation(run, specs.length, combineWith, false)) {
4490
- return executeCombinedBranches(specs, combineWith, params, (spec, branchAllowed) => executeQuerySpecInternal(spec, normalized, params, branchAllowed), (spec, branchAllowed) => collectDocIdsForQuerySpec(spec, normalized, params, branchAllowed), allowedDocs, run, spec => estimateMaxPostingLengthForQuerySpec(spec, normalized, params));
4605
+ return executeCombinedBranches(specs, combineWith, params, (spec, branchAllowed) => executeQuerySpecInternal(spec, normalized, params, branchAllowed), (spec, branchAllowed) => collectDocIdsForQuerySpec(spec, normalized, params, branchAllowed), allowedDocs, run, spec => estimateMaxPostingLengthForQuerySpec(spec, normalized, params), specs.every(spec => !spec.prefix && !spec.fuzzy));
4491
4606
  }
4492
4607
  const results = specs.map(spec => executeQuerySpecInternal(spec, normalized, params, allowedDocs));
4493
4608
  return combineResults(results, combineWith);
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@yoch/frozenminisearch",
3
- "version": "1.2.2",
3
+ "version": "1.2.3",
4
4
  "description": "Read-only Node.js full-text search — compact frozen indexes and binary snapshots",
5
5
  "main": "dist/cjs/index.cjs",
6
6
  "module": "dist/es/index.js",