@yoch/minisearch 8.0.0-beta.2 → 8.0.0-beta.3

package/CHANGELOG.md

@@ -2,6 +2,20 @@
 
 `MiniSearch` follows [semantic versioning](https://semver.org/spec/v2.0.0.html).
 
+## v8.0.0-beta.3
+
+Incremental frozen index construction without a temporary `documents[]` array.
+
+  - Add `FrozenIndexBuilder` and `createFrozenIndexBuilder(options, hints?)` with `.add(doc)`
+    and optional `estimatedDocumentCount` pre-sizing
+  - Add `freezeFrozenIndexBuilder(builder)` to finalize into `FrozenMiniSearch` (avoids a
+    circular import between build and assembly modules)
+  - Add `FrozenMiniSearch.fromAsyncIterable(iterable, options)` for async document streams
+    (e.g. CSV parsers)
+  - Refactor `buildFrozenParamsFromDocuments` to use the builder internally (same output)
+  - Trim per-document arrays when `estimatedDocumentCount` exceeds the actual document count
+  - Export `FrozenIndexBuilderHints` type
+
 ## v8.0.0-beta.2
 
 Consolidated beta on npm. Supersedes `8.0.0-beta.0` and `8.0.0-beta.1` (unpublished).

package/README.md

@@ -2,7 +2,7 @@
 
 **In-memory full-text search for Node.js** — a fork of [MiniSearch](https://github.com/lucaong/minisearch) by [Luca Ongaro](https://github.com/lucaong/minisearch), extended for **production serving**: smaller indexes, faster loads, and a read-only fast path.
 
-> **Current release:** `8.0.0-beta.2` · install with `npm install @yoch/minisearch`
+> **Current release:** `8.0.0-beta.3` · install with `npm install @yoch/minisearch`
 
 ---
 
@@ -74,17 +74,73 @@ const { FrozenMiniSearch } = require('@yoch/minisearch')
 |------|-----|
 | Live index that changes over time | `MiniSearch` → `freeze()` when you need read-only serving |
 | Fixed corpus, build frozen directly | **`FrozenMiniSearch.fromDocuments(documents, options)`** |
+| Build doc-by-doc (no `documents[]` buffer) | **`createFrozenIndexBuilder(options)`** → `.add(doc)` → **`freezeFrozenIndexBuilder(builder)`** |
+| Async stream of documents | **`FrozenMiniSearch.fromAsyncIterable(iterable, options)`** |
 | Load a snapshot from disk | `FrozenMiniSearch.loadBinary(buffer, options)` |
 | Custom assembly pipeline | `buildFrozenFromDocuments`, `assembleFrozen`, `freezeFromMiniSearch` |
 
 `fromDocuments` matches `new MiniSearch(opts).addAll(docs).freeze()` for search ranking on the same corpus and options (`fields`, `tokenize`, `processTerm`, …). Frozen indexes do not support `add` / `remove`.
 
+**External corpus (e.g. lookup by id after search):** keep full rows in your own store (`dataCache`, DB, etc.) and use minimal `storeFields` (often `['id']` only) so the frozen index does not duplicate payload text:
+
+```javascript
+import { createFrozenIndexBuilder, freezeFrozenIndexBuilder } from '@yoch/minisearch'
+
+function buildFrozenIndexFromRows (rows, options) {
+  const builder = createFrozenIndexBuilder(options, {
+    estimatedDocumentCount: rows.length
+  })
+  for (let i = 0; i < rows.length; i++) {
+    builder.add(buildIndexDocument(rows[i], i))
+  }
+  return freezeFrozenIndexBuilder(builder)
+}
+
+// After search: enrich from your store — frozen.getStoredFields(res.id) or dataCache[type][res.id]
+```
+
+**Async stream** (no intermediate array; documents are indexed as they arrive):
+
+```javascript
+import { createReadStream } from 'node:fs'
+import { parse } from 'csv-parse'
+import { FrozenMiniSearch } from '@yoch/minisearch'
+
+async function buildFromCsv (path, options) {
+  async function * documents () {
+    const parser = createReadStream(path).pipe(parse({ columns: true }))
+    for await (const row of parser) {
+      yield { id: row.cis, denomination: row.denomination, /* … */ }
+    }
+  }
+  return FrozenMiniSearch.fromAsyncIterable(documents(), options)
+}
+```
+
+For a **sync** iterable (`for...of` on an array or generator), use the builder directly:
+
+```javascript
+import { createFrozenIndexBuilder, freezeFrozenIndexBuilder } from '@yoch/minisearch'
+
+const builder = createFrozenIndexBuilder(options)
+for (const doc of documentGenerator()) {
+  builder.add(doc)
+}
+const frozen = freezeFrozenIndexBuilder(builder)
+```
+
+`estimatedDocumentCount` in the second argument to `createFrozenIndexBuilder` pre-allocates
+per-document arrays when the final size is known; internal buffers are trimmed to the actual
+count on freeze if the hint was too large.
+
 ---
 
 ## FrozenMiniSearch in a bit more detail
 
 - **`freeze()`** — snapshot a mutable index into compact typed postings + a radix tree keyed by term index.
 - **`fromDocuments()`** — build that structure in one pass (skips nested `Map` postings and radix cloning at freeze time).
+- **`createFrozenIndexBuilder()`** — same output without a temporary `documents[]` array; finalize with `freezeFrozenIndexBuilder(builder)` (or `assembleFrozen(builder.freezeParams())` for custom assembly).
+- **`fromAsyncIterable()`** — async document stream (e.g. CSV parser) into a frozen index; equivalent to builder + `for await` + `freezeFrozenIndexBuilder`.
 - **`saveBinary()` / `loadBinary()`** — MSv2 on write, MSv1 still readable; pass the same `fields` (and custom `tokenize` / `processTerm` if used at build time).
 - **Term frequencies** — stored as `Uint8` (max 255 per doc/term); only affects scores for extreme term repetition.
 - **`frozenMemoryBreakdown()`** — introspect postings, radix tree, and stored-field footprint.
@@ -94,6 +150,10 @@ Advanced exports:
 ```javascript
 import {
   FrozenMiniSearch,
+  createFrozenIndexBuilder,
+  freezeFrozenIndexBuilder,
+  FrozenIndexBuilder,
+  type FrozenIndexBuilderHints,
   buildFrozenFromDocuments,
   assembleFrozen,
   freezeFromMiniSearch,
@@ -124,8 +184,8 @@ TypeScript definitions: `dist/es/index.d.ts`.
 Reproducible comparisons (heap, load time, search latency) live under [`benchmarks/`](benchmarks/README.md):
 
 ```bash
-yarn benchmark:compare    # terminal report
-yarn benchmark:diff       # vs versioned baseline
+npm run benchmark:compare    # terminal report
+npm run benchmark:diff       # vs versioned baseline
 ```
 
 ---
@@ -133,11 +193,13 @@ yarn benchmark:diff       # vs versioned baseline
 ## Development
 
 ```bash
-yarn install
-yarn test
-yarn build
+npm install
+npm test
+npm run build
 ```
 
+Use `npm run` for scripts (Yarn 1.x on Node 22 prints `url.parse` deprecation noise when invoking `yarn test` / `yarn build`).
+
 **Requirements:** Node.js **ES2018+**. No browser UMD/CDN build in this fork (Node-only ESM + CJS).
 
 ---

package/dist/cjs/index.cjs

@@ -1085,41 +1085,41 @@ function saveStoredFieldsForDocument(storeFields, extractField, document) {
     return documentFields;
 }
 
-function getOrCreateTermIndex(builder, term) {
-    const existing = builder.index.get(term);
+function getOrCreateTermIndex(state, index, term) {
+    const existing = index.get(term);
     if (existing != null)
         return existing;
-    const ti = builder.terms.length;
-    builder.terms.push(term);
-    builder.index.set(term, ti);
+    const ti = state.terms.length;
+    state.terms.push(term);
+    index.set(term, ti);
     return ti;
 }
-function appendPosting(builder, termIndex, fieldId, docId, freq) {
-    const slot = termIndex * builder.fieldCount + fieldId;
-    let docIds = builder.postingsDocIds[slot];
-    let freqs = builder.postingsFreqs[slot];
+function appendPosting(state, termIndex, fieldId, docId, freq) {
+    const slot = termIndex * state.fieldCount + fieldId;
+    let docIds = state.postingsDocIds[slot];
+    let freqs = state.postingsFreqs[slot];
     if (docIds == null) {
         docIds = [];
         freqs = [];
-        builder.postingsDocIds[slot] = docIds;
-        builder.postingsFreqs[slot] = freqs;
+        state.postingsDocIds[slot] = docIds;
+        state.postingsFreqs[slot] = freqs;
     }
     docIds.push(docId);
     freqs.push(clampFreq(freq));
 }
-function finalizeFlatPostings(builder) {
-    const termCount = builder.terms.length;
-    const slotCount = termCount * builder.fieldCount;
+function finalizeFlatPostings(state) {
+    const termCount = state.terms.length;
+    const slotCount = termCount * state.fieldCount;
     const postingsOffsets = new Uint32Array(slotCount);
     const postingsLengths = new Uint32Array(slotCount);
     const docScratch = [];
     const freqScratch = [];
     for (let ti = 0; ti < termCount; ti++) {
-        const base = ti * builder.fieldCount;
-        for (let f = 0; f < builder.fieldCount; f++) {
+        const base = ti * state.fieldCount;
+        for (let f = 0; f < state.fieldCount; f++) {
             const offset = docScratch.length;
-            const docIds = builder.postingsDocIds[base + f];
-            const freqs = builder.postingsFreqs[base + f];
+            const docIds = state.postingsDocIds[base + f];
+            const freqs = state.postingsFreqs[base + f];
             if (docIds == null || docIds.length === 0) {
                 postingsOffsets[base + f] = offset;
                 postingsLengths[base + f] = 0;
@@ -1140,84 +1140,132 @@ function finalizeFlatPostings(builder) {
         allFreqs: new Uint8Array(freqScratch)
     };
 }
-function indexDocument(builder, document, shortId) {
-    const { extractField, stringifyField, tokenize, processTerm, fields, idField, storeFields } = builder.options;
-    const id = extractField(document, idField);
-    if (id == null) {
-        throw new Error(`MiniSearch: document does not have ID field "${idField}"`);
-    }
-    if (builder.idToShortId.has(id)) {
-        throw new Error(`MiniSearch: duplicate ID ${id}`);
-    }
-    builder.idToShortId.set(id, shortId);
-    builder.externalIds[shortId] = id;
-    builder.storedFields[shortId] = saveStoredFieldsForDocument(storeFields, extractField, document);
-    const documentCount = shortId + 1;
-    for (const field of fields) {
-        const fieldValue = extractField(document, field);
-        if (fieldValue == null)
-            continue;
-        const tokens = tokenize(stringifyField(fieldValue, field), field);
-        const fieldId = builder.fieldIds[field];
-        const uniqueTerms = new Set(tokens).size;
-        const localFreqs = collectFieldTermFreqs(tokens, field, processTerm);
-        builder.fieldLengthMatrix[shortId * builder.fieldCount + fieldId] = uniqueTerms;
-        updateAvgFieldLength(builder.avgFieldLength, fieldId, documentCount - 1, uniqueTerms);
-        for (const [term, freq] of localFreqs) {
-            const ti = getOrCreateTermIndex(builder, term);
-            appendPosting(builder, ti, fieldId, shortId, freq);
+/** Incremental builder for {@link FrozenMiniSearch} without materializing a full `documents[]` array. */
+class FrozenIndexBuilder {
+    constructor(options, hints) {
+        this._options = resolveIndexingOptions(options);
+        this._fieldIds = buildFieldIds(this._options.fields);
+        this._fieldCount = this._options.fields.length;
+        this._index = new SearchableMap();
+        this._terms = [];
+        this._postingsDocIds = [];
+        this._postingsFreqs = [];
+        this._idToShortId = new Map();
+        this._avgFieldLength = [];
+        this._nextId = 0;
+        this._frozen = false;
+        const estimated = hints === null || hints === void 0 ? void 0 : hints.estimatedDocumentCount;
+        if (estimated != null && estimated > 0) {
+            this._externalIds = new Array(estimated);
+            this._storedFields = new Array(estimated);
+            this._fieldLengthData = new Array(estimated * this._fieldCount).fill(0);
+        }
+        else {
+            this._externalIds = [];
+            this._storedFields = [];
+            this._fieldLengthData = [];
         }
+        this._postingsState = {
+            fieldCount: this._fieldCount,
+            terms: this._terms,
+            postingsDocIds: this._postingsDocIds,
+            postingsFreqs: this._postingsFreqs
+        };
+    }
+    /** Number of documents indexed so far (not yet frozen). */
+    get documentCount() {
+        return this._nextId;
+    }
+    add(document) {
+        if (this._frozen) {
+            throw new Error('FrozenIndexBuilder: cannot add after freezeParams()');
+        }
+        const { extractField, stringifyField, tokenize, processTerm, fields, idField, storeFields } = this._options;
+        const id = extractField(document, idField);
+        if (id == null) {
+            throw new Error(`MiniSearch: document does not have ID field "${idField}"`);
+        }
+        if (this._idToShortId.has(id)) {
+            throw new Error(`MiniSearch: duplicate ID ${id}`);
+        }
+        const shortId = this._nextId++;
+        this._idToShortId.set(id, shortId);
+        this._externalIds[shortId] = id;
+        this._storedFields[shortId] = saveStoredFieldsForDocument(storeFields, extractField, document);
+        const documentCount = shortId + 1;
+        for (const field of fields) {
+            const fieldValue = extractField(document, field);
+            if (fieldValue == null)
+                continue;
+            const tokens = tokenize(stringifyField(fieldValue, field), field);
+            const fieldId = this._fieldIds[field];
+            const uniqueTerms = new Set(tokens).size;
+            const localFreqs = collectFieldTermFreqs(tokens, field, processTerm);
+            this._fieldLengthData[shortId * this._fieldCount + fieldId] = uniqueTerms;
+            updateAvgFieldLength(this._avgFieldLength, fieldId, documentCount - 1, uniqueTerms);
+            for (const [term, freq] of localFreqs) {
+                const ti = getOrCreateTermIndex(this._postingsState, this._index, term);
+                appendPosting(this._postingsState, ti, fieldId, shortId, freq);
+            }
+        }
+    }
+    /**
+     * Finalize this builder into assembly params. Call {@link assembleFrozen} or
+     * {@link freezeFrozenIndexBuilder} to obtain a {@link FrozenMiniSearch} instance.
+     */
+    freezeParams() {
+        var _a;
+        if (this._frozen) {
+            throw new Error('FrozenIndexBuilder: freezeParams() already called');
+        }
+        this._frozen = true;
+        const documentCount = this._nextId;
+        const flat = finalizeFlatPostings(this._postingsState);
+        const avgFieldLength = new Float32Array(this._fieldCount);
+        for (let f = 0; f < this._fieldCount; f++) {
+            avgFieldLength[f] = (_a = this._avgFieldLength[f]) !== null && _a !== void 0 ? _a : 0;
+        }
+        // Ensure exact size regardless of over- or under-estimated documentCount.
+        this._fieldLengthData.length = documentCount * this._fieldCount;
+        // Trim per-document arrays to actual count when estimatedDocumentCount was too large.
+        const externalIds = this._externalIds.length > documentCount
+            ? this._externalIds.slice(0, documentCount)
+            : this._externalIds;
+        const storedFields = this._storedFields.length > documentCount
+            ? this._storedFields.slice(0, documentCount)
+            : this._storedFields;
+        return {
+            options: this._options,
+            documentCount,
+            nextId: documentCount,
+            fieldIds: this._fieldIds,
+            fieldCount: this._fieldCount,
+            externalIds,
+            idToShortId: this._idToShortId,
+            storedFields,
+            fieldLengthMatrix: new Uint32Array(this._fieldLengthData),
+            avgFieldLength,
+            index: this._index,
+            terms: this._terms,
+            postingsOffsets: flat.postingsOffsets,
+            postingsLengths: flat.postingsLengths,
+            allDocIds: flat.allDocIds,
+            allFreqs: flat.allFreqs
+        };
     }
 }
-function createBuilder(options, documentCount) {
-    const fieldCount = options.fields.length;
-    return {
-        options,
-        fieldIds: buildFieldIds(options.fields),
-        fieldCount,
-        documentCount,
-        index: new SearchableMap(),
-        terms: [],
-        postingsDocIds: [],
-        postingsFreqs: [],
-        externalIds: new Array(documentCount),
-        idToShortId: new Map(),
-        storedFields: new Array(documentCount),
-        fieldLengthMatrix: new Uint32Array(documentCount * fieldCount),
-        avgFieldLength: []
-    };
+/** Create an incremental builder for {@link FrozenMiniSearch}. */
+function createFrozenIndexBuilder(options, hints) {
+    return new FrozenIndexBuilder(options, hints);
 }
 function buildFrozenParamsFromDocuments(documents, options) {
-    var _a;
-    const resolved = resolveIndexingOptions(options);
-    const documentCount = documents.length;
-    const builder = createBuilder(resolved, documentCount);
-    for (let d = 0; d < documentCount; d++) {
-        indexDocument(builder, documents[d], d);
-    }
-    const flat = finalizeFlatPostings(builder);
-    const avgFieldLength = new Float32Array(builder.fieldCount);
-    for (let f = 0; f < builder.fieldCount; f++) {
-        avgFieldLength[f] = (_a = builder.avgFieldLength[f]) !== null && _a !== void 0 ? _a : 0;
+    const builder = createFrozenIndexBuilder(options, {
+        estimatedDocumentCount: documents.length
+    });
+    for (let d = 0; d < documents.length; d++) {
+        builder.add(documents[d]);
     }
-    return {
-        options: resolved,
-        documentCount,
-        nextId: documentCount,
-        fieldIds: builder.fieldIds,
-        fieldCount: builder.fieldCount,
-        externalIds: builder.externalIds,
-        idToShortId: builder.idToShortId,
-        storedFields: builder.storedFields,
-        fieldLengthMatrix: builder.fieldLengthMatrix,
-        avgFieldLength,
-        index: builder.index,
-        terms: builder.terms,
-        postingsOffsets: flat.postingsOffsets,
-        postingsLengths: flat.postingsLengths,
-        allDocIds: flat.allDocIds,
-        allFreqs: flat.allFreqs
-    };
+    return builder.freezeParams();
 }
 
 /** Shared wildcard query symbol for MiniSearch and FrozenMiniSearch */
@@ -1370,6 +1418,10 @@ function freezeFromMiniSearch(source) {
 function buildFrozenFromDocuments(documents, options) {
     return assembleFrozen(buildFrozenParamsFromDocuments(documents, options));
 }
+/** Finalize a {@link FrozenIndexBuilder} into a read-only index. */
+function freezeFrozenIndexBuilder(builder) {
+    return assembleFrozen(builder.freezeParams());
+}
 class FrozenMiniSearch {
     constructor(params) {
         this._options = params.options;
@@ -1560,6 +1612,17 @@ class FrozenMiniSearch {
     static fromDocuments(documents, options) {
         return buildFrozenFromDocuments(documents, options);
     }
+    /**
+     * Build a read-only index from an async stream of documents (e.g. CSV parser).
+     * For sync iterables, use {@link createFrozenIndexBuilder} with `for...of` instead.
+     */
+    static async fromAsyncIterable(iterable, options) {
+        const builder = createFrozenIndexBuilder(options);
+        for await (const document of iterable) {
+            builder.add(document);
+        }
+        return freezeFrozenIndexBuilder(builder);
+    }
     getFieldLength(docId, fieldId) {
         var _a;
         return (_a = this._fieldLengthMatrix[docId * this._fieldCount + fieldId]) !== null && _a !== void 0 ? _a : 0;
@@ -2952,10 +3015,13 @@ const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
 
 exports.AND = AND;
 exports.AND_NOT = AND_NOT;
+exports.FrozenIndexBuilder = FrozenIndexBuilder;
 exports.FrozenMiniSearch = FrozenMiniSearch;
 exports.OR = OR;
 exports.assembleFrozen = assembleFrozen;
 exports.buildFrozenFromDocuments = buildFrozenFromDocuments;
+exports.createFrozenIndexBuilder = createFrozenIndexBuilder;
 exports.default = MiniSearch;
 exports.freezeFromMiniSearch = freezeFromMiniSearch;
+exports.freezeFrozenIndexBuilder = freezeFrozenIndexBuilder;
 exports.frozenMemoryBreakdown = frozenMemoryBreakdown;

package/dist/es/index.d.ts

@@ -254,6 +254,40 @@ declare class SearchableMap<T = any> {
     }): SearchableMap<any>;
 }
 
+interface FrozenIndexBuilderHints {
+    /** Pre-size per-document arrays when the final document count is known. */
+    estimatedDocumentCount?: number;
+}
+/** Incremental builder for {@link FrozenMiniSearch} without materializing a full `documents[]` array. */
+declare class FrozenIndexBuilder<T> {
+    private readonly _options;
+    private readonly _fieldIds;
+    private readonly _fieldCount;
+    private readonly _index;
+    private readonly _terms;
+    private readonly _postingsDocIds;
+    private readonly _postingsFreqs;
+    private readonly _externalIds;
+    private readonly _idToShortId;
+    private readonly _storedFields;
+    private readonly _fieldLengthData;
+    private readonly _avgFieldLength;
+    private readonly _postingsState;
+    private _nextId;
+    private _frozen;
+    constructor(options: Options<T>, hints?: FrozenIndexBuilderHints);
+    /** Number of documents indexed so far (not yet frozen). */
+    get documentCount(): number;
+    add(document: T): void;
+    /**
+     * Finalize this builder into assembly params. Call {@link assembleFrozen} or
+     * {@link freezeFrozenIndexBuilder} to obtain a {@link FrozenMiniSearch} instance.
+     */
+    freezeParams(): FrozenAssembleParams<T>;
+}
+/** Create an incremental builder for {@link FrozenMiniSearch}. */
+declare function createFrozenIndexBuilder<T>(options: Options<T>, hints?: FrozenIndexBuilderHints): FrozenIndexBuilder<T>;
+
 /** Shared wildcard query symbol for MiniSearch and FrozenMiniSearch */
 declare const WILDCARD_QUERY: unique symbol;
 
@@ -345,6 +379,8 @@ interface FrozenAssembleParams<T = any> {
 declare function assembleFrozen<T>(params: FrozenAssembleParams<T>): FrozenMiniSearch<T>;
 declare function freezeFromMiniSearch<T>(source: FreezeSource<T>): FrozenMiniSearch<T>;
 declare function buildFrozenFromDocuments<T>(documents: readonly T[], options: Options<T>): FrozenMiniSearch<T>;
+/** Finalize a {@link FrozenIndexBuilder} into a read-only index. */
+declare function freezeFrozenIndexBuilder<T>(builder: FrozenIndexBuilder<T>): FrozenMiniSearch<T>;
 declare class FrozenMiniSearch<T = any> {
     private readonly _options;
     private readonly _index;
@@ -408,6 +444,11 @@ declare class FrozenMiniSearch<T = any> {
      * incremental updates before freezing.
      */
     static fromDocuments<T>(documents: readonly T[], options: Options<T>): FrozenMiniSearch<T>;
+    /**
+     * Build a read-only index from an async stream of documents (e.g. CSV parser).
+     * For sync iterables, use {@link createFrozenIndexBuilder} with `for...of` instead.
+     */
+    static fromAsyncIterable<T>(iterable: AsyncIterable<T>, options: Options<T>): Promise<FrozenMiniSearch<T>>;
     private getFieldLength;
     private fieldTermDataFor;
     private aggregateContext;
@@ -1632,4 +1673,4 @@ interface SerializedIndexEntry {
     [key: string]: number;
 }
 
-export { AND, AND_NOT, type AsPlainObject, type AutoVacuumOptions, type BM25Params, type CombinationOperator, type FrozenAssembleParams, type FrozenMemoryBreakdown, FrozenMiniSearch, type LowercaseCombinationOperator, type MatchInfo, OR, type Options, type Query, type QueryCombination, type SearchOptions, type SearchResult, type Suggestion, type VacuumConditions, type VacuumOptions, type Wildcard, assembleFrozen, buildFrozenFromDocuments, MiniSearch as default, freezeFromMiniSearch, frozenMemoryBreakdown };
+export { AND, AND_NOT, type AsPlainObject, type AutoVacuumOptions, type BM25Params, type CombinationOperator, type FrozenAssembleParams, FrozenIndexBuilder, type FrozenIndexBuilderHints, type FrozenMemoryBreakdown, FrozenMiniSearch, type LowercaseCombinationOperator, type MatchInfo, OR, type Options, type Query, type QueryCombination, type SearchOptions, type SearchResult, type Suggestion, type VacuumConditions, type VacuumOptions, type Wildcard, assembleFrozen, buildFrozenFromDocuments, createFrozenIndexBuilder, MiniSearch as default, freezeFromMiniSearch, freezeFrozenIndexBuilder, frozenMemoryBreakdown };

package/dist/es/index.js

@@ -1081,41 +1081,41 @@ function saveStoredFieldsForDocument(storeFields, extractField, document) {
     return documentFields;
 }
 
-function getOrCreateTermIndex(builder, term) {
-    const existing = builder.index.get(term);
+function getOrCreateTermIndex(state, index, term) {
+    const existing = index.get(term);
     if (existing != null)
         return existing;
-    const ti = builder.terms.length;
-    builder.terms.push(term);
-    builder.index.set(term, ti);
+    const ti = state.terms.length;
+    state.terms.push(term);
+    index.set(term, ti);
     return ti;
 }
-function appendPosting(builder, termIndex, fieldId, docId, freq) {
-    const slot = termIndex * builder.fieldCount + fieldId;
-    let docIds = builder.postingsDocIds[slot];
-    let freqs = builder.postingsFreqs[slot];
+function appendPosting(state, termIndex, fieldId, docId, freq) {
+    const slot = termIndex * state.fieldCount + fieldId;
+    let docIds = state.postingsDocIds[slot];
+    let freqs = state.postingsFreqs[slot];
     if (docIds == null) {
         docIds = [];
         freqs = [];
-        builder.postingsDocIds[slot] = docIds;
-        builder.postingsFreqs[slot] = freqs;
+        state.postingsDocIds[slot] = docIds;
+        state.postingsFreqs[slot] = freqs;
     }
     docIds.push(docId);
     freqs.push(clampFreq(freq));
 }
-function finalizeFlatPostings(builder) {
-    const termCount = builder.terms.length;
-    const slotCount = termCount * builder.fieldCount;
+function finalizeFlatPostings(state) {
+    const termCount = state.terms.length;
+    const slotCount = termCount * state.fieldCount;
     const postingsOffsets = new Uint32Array(slotCount);
     const postingsLengths = new Uint32Array(slotCount);
     const docScratch = [];
     const freqScratch = [];
     for (let ti = 0; ti < termCount; ti++) {
-        const base = ti * builder.fieldCount;
-        for (let f = 0; f < builder.fieldCount; f++) {
+        const base = ti * state.fieldCount;
+        for (let f = 0; f < state.fieldCount; f++) {
             const offset = docScratch.length;
-            const docIds = builder.postingsDocIds[base + f];
-            const freqs = builder.postingsFreqs[base + f];
+            const docIds = state.postingsDocIds[base + f];
+            const freqs = state.postingsFreqs[base + f];
             if (docIds == null || docIds.length === 0) {
                 postingsOffsets[base + f] = offset;
                 postingsLengths[base + f] = 0;
@@ -1136,84 +1136,132 @@ function finalizeFlatPostings(builder) {
         allFreqs: new Uint8Array(freqScratch)
     };
 }
-function indexDocument(builder, document, shortId) {
-    const { extractField, stringifyField, tokenize, processTerm, fields, idField, storeFields } = builder.options;
-    const id = extractField(document, idField);
-    if (id == null) {
-        throw new Error(`MiniSearch: document does not have ID field "${idField}"`);
-    }
-    if (builder.idToShortId.has(id)) {
-        throw new Error(`MiniSearch: duplicate ID ${id}`);
-    }
-    builder.idToShortId.set(id, shortId);
-    builder.externalIds[shortId] = id;
-    builder.storedFields[shortId] = saveStoredFieldsForDocument(storeFields, extractField, document);
-    const documentCount = shortId + 1;
-    for (const field of fields) {
-        const fieldValue = extractField(document, field);
-        if (fieldValue == null)
-            continue;
-        const tokens = tokenize(stringifyField(fieldValue, field), field);
-        const fieldId = builder.fieldIds[field];
-        const uniqueTerms = new Set(tokens).size;
-        const localFreqs = collectFieldTermFreqs(tokens, field, processTerm);
-        builder.fieldLengthMatrix[shortId * builder.fieldCount + fieldId] = uniqueTerms;
-        updateAvgFieldLength(builder.avgFieldLength, fieldId, documentCount - 1, uniqueTerms);
-        for (const [term, freq] of localFreqs) {
-            const ti = getOrCreateTermIndex(builder, term);
-            appendPosting(builder, ti, fieldId, shortId, freq);
+/** Incremental builder for {@link FrozenMiniSearch} without materializing a full `documents[]` array. */
+class FrozenIndexBuilder {
+    constructor(options, hints) {
+        this._options = resolveIndexingOptions(options);
+        this._fieldIds = buildFieldIds(this._options.fields);
+        this._fieldCount = this._options.fields.length;
+        this._index = new SearchableMap();
+        this._terms = [];
+        this._postingsDocIds = [];
+        this._postingsFreqs = [];
+        this._idToShortId = new Map();
+        this._avgFieldLength = [];
+        this._nextId = 0;
+        this._frozen = false;
+        const estimated = hints === null || hints === void 0 ? void 0 : hints.estimatedDocumentCount;
+        if (estimated != null && estimated > 0) {
+            this._externalIds = new Array(estimated);
+            this._storedFields = new Array(estimated);
+            this._fieldLengthData = new Array(estimated * this._fieldCount).fill(0);
+        }
+        else {
+            this._externalIds = [];
+            this._storedFields = [];
+            this._fieldLengthData = [];
         }
+        this._postingsState = {
+            fieldCount: this._fieldCount,
+            terms: this._terms,
+            postingsDocIds: this._postingsDocIds,
+            postingsFreqs: this._postingsFreqs
+        };
+    }
+    /** Number of documents indexed so far (not yet frozen). */
+    get documentCount() {
+        return this._nextId;
+    }
+    add(document) {
+        if (this._frozen) {
+            throw new Error('FrozenIndexBuilder: cannot add after freezeParams()');
+        }
+        const { extractField, stringifyField, tokenize, processTerm, fields, idField, storeFields } = this._options;
+        const id = extractField(document, idField);
+        if (id == null) {
+            throw new Error(`MiniSearch: document does not have ID field "${idField}"`);
+        }
+        if (this._idToShortId.has(id)) {
+            throw new Error(`MiniSearch: duplicate ID ${id}`);
+        }
+        const shortId = this._nextId++;
+        this._idToShortId.set(id, shortId);
+        this._externalIds[shortId] = id;
+        this._storedFields[shortId] = saveStoredFieldsForDocument(storeFields, extractField, document);
+        const documentCount = shortId + 1;
+        for (const field of fields) {
+            const fieldValue = extractField(document, field);
+            if (fieldValue == null)
+                continue;
+            const tokens = tokenize(stringifyField(fieldValue, field), field);
+            const fieldId = this._fieldIds[field];
+            const uniqueTerms = new Set(tokens).size;
+            const localFreqs = collectFieldTermFreqs(tokens, field, processTerm);
+            this._fieldLengthData[shortId * this._fieldCount + fieldId] = uniqueTerms;
+            updateAvgFieldLength(this._avgFieldLength, fieldId, documentCount - 1, uniqueTerms);
+            for (const [term, freq] of localFreqs) {
+                const ti = getOrCreateTermIndex(this._postingsState, this._index, term);
+                appendPosting(this._postingsState, ti, fieldId, shortId, freq);
+            }
+        }
+    }
+    /**
+     * Finalize this builder into assembly params. Call {@link assembleFrozen} or
+     * {@link freezeFrozenIndexBuilder} to obtain a {@link FrozenMiniSearch} instance.
+     */
+    freezeParams() {
+        var _a;
+        if (this._frozen) {
+            throw new Error('FrozenIndexBuilder: freezeParams() already called');
+        }
+        this._frozen = true;
+        const documentCount = this._nextId;
+        const flat = finalizeFlatPostings(this._postingsState);
+        const avgFieldLength = new Float32Array(this._fieldCount);
+        for (let f = 0; f < this._fieldCount; f++) {
+            avgFieldLength[f] = (_a = this._avgFieldLength[f]) !== null && _a !== void 0 ? _a : 0;
+        }
+        // Ensure exact size regardless of over- or under-estimated documentCount.
+        this._fieldLengthData.length = documentCount * this._fieldCount;
+        // Trim per-document arrays to actual count when estimatedDocumentCount was too large.
+        const externalIds = this._externalIds.length > documentCount
+            ? this._externalIds.slice(0, documentCount)
+            : this._externalIds;
+        const storedFields = this._storedFields.length > documentCount
+            ? this._storedFields.slice(0, documentCount)
+            : this._storedFields;
+        return {
+            options: this._options,
+            documentCount,
+            nextId: documentCount,
+            fieldIds: this._fieldIds,
+            fieldCount: this._fieldCount,
+            externalIds,
+            idToShortId: this._idToShortId,
+            storedFields,
+            fieldLengthMatrix: new Uint32Array(this._fieldLengthData),
+            avgFieldLength,
+            index: this._index,
+            terms: this._terms,
+            postingsOffsets: flat.postingsOffsets,
+            postingsLengths: flat.postingsLengths,
+            allDocIds: flat.allDocIds,
+            allFreqs: flat.allFreqs
+        };
     }
 }
-function createBuilder(options, documentCount) {
-    const fieldCount = options.fields.length;
-    return {
-        options,
-        fieldIds: buildFieldIds(options.fields),
-        fieldCount,
-        documentCount,
-        index: new SearchableMap(),
-        terms: [],
-        postingsDocIds: [],
-        postingsFreqs: [],
-        externalIds: new Array(documentCount),
-        idToShortId: new Map(),
-        storedFields: new Array(documentCount),
-        fieldLengthMatrix: new Uint32Array(documentCount * fieldCount),
-        avgFieldLength: []
-    };
+/** Create an incremental builder for {@link FrozenMiniSearch}. */
+function createFrozenIndexBuilder(options, hints) {
+    return new FrozenIndexBuilder(options, hints);
 }
 function buildFrozenParamsFromDocuments(documents, options) {
-    var _a;
-    const resolved = resolveIndexingOptions(options);
-    const documentCount = documents.length;
-    const builder = createBuilder(resolved, documentCount);
-    for (let d = 0; d < documentCount; d++) {
-        indexDocument(builder, documents[d], d);
-    }
-    const flat = finalizeFlatPostings(builder);
-    const avgFieldLength = new Float32Array(builder.fieldCount);
-    for (let f = 0; f < builder.fieldCount; f++) {
-        avgFieldLength[f] = (_a = builder.avgFieldLength[f]) !== null && _a !== void 0 ? _a : 0;
+    const builder = createFrozenIndexBuilder(options, {
+        estimatedDocumentCount: documents.length
+    });
+    for (let d = 0; d < documents.length; d++) {
+        builder.add(documents[d]);
     }
-    return {
-        options: resolved,
-        documentCount,
-        nextId: documentCount,
-        fieldIds: builder.fieldIds,
-        fieldCount: builder.fieldCount,
-        externalIds: builder.externalIds,
-        idToShortId: builder.idToShortId,
-        storedFields: builder.storedFields,
-        fieldLengthMatrix: builder.fieldLengthMatrix,
-        avgFieldLength,
-        index: builder.index,
-        terms: builder.terms,
-        postingsOffsets: flat.postingsOffsets,
-        postingsLengths: flat.postingsLengths,
-        allDocIds: flat.allDocIds,
-        allFreqs: flat.allFreqs
-    };
+    return builder.freezeParams();
 }
 
 /** Shared wildcard query symbol for MiniSearch and FrozenMiniSearch */
@@ -1366,6 +1414,10 @@ function freezeFromMiniSearch(source) {
 function buildFrozenFromDocuments(documents, options) {
     return assembleFrozen(buildFrozenParamsFromDocuments(documents, options));
 }
+/** Finalize a {@link FrozenIndexBuilder} into a read-only index. */
+function freezeFrozenIndexBuilder(builder) {
+    return assembleFrozen(builder.freezeParams());
+}
 class FrozenMiniSearch {
     constructor(params) {
         this._options = params.options;
@@ -1556,6 +1608,17 @@ class FrozenMiniSearch {
     static fromDocuments(documents, options) {
         return buildFrozenFromDocuments(documents, options);
     }
+    /**
+     * Build a read-only index from an async stream of documents (e.g. CSV parser).
+     * For sync iterables, use {@link createFrozenIndexBuilder} with `for...of` instead.
+     */
+    static async fromAsyncIterable(iterable, options) {
+        const builder = createFrozenIndexBuilder(options);
+        for await (const document of iterable) {
+            builder.add(document);
+        }
+        return freezeFrozenIndexBuilder(builder);
+    }
     getFieldLength(docId, fieldId) {
         var _a;
         return (_a = this._fieldLengthMatrix[docId * this._fieldCount + fieldId]) !== null && _a !== void 0 ? _a : 0;
@@ -2946,4 +3009,4 @@ const objectToNumericMapAsync = async (object) => {
 };
 const wait = (ms) => new Promise((resolve) => setTimeout(resolve, ms));
 
-export { AND, AND_NOT, FrozenMiniSearch, OR, assembleFrozen, buildFrozenFromDocuments, MiniSearch as default, freezeFromMiniSearch, frozenMemoryBreakdown };
+export { AND, AND_NOT, FrozenIndexBuilder, FrozenMiniSearch, OR, assembleFrozen, buildFrozenFromDocuments, createFrozenIndexBuilder, MiniSearch as default, freezeFromMiniSearch, freezeFrozenIndexBuilder, frozenMemoryBreakdown };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yoch/minisearch",
-  "version": "8.0.0-beta.2",
+  "version": "8.0.0-beta.3",
   "description": "Node.js full-text search with FrozenMiniSearch and binary index snapshots",
   "main": "dist/cjs/index.cjs",
   "module": "dist/es/index.js",
@@ -94,23 +94,23 @@
     "test": "jest",
     "test-watch": "jest --watch",
     "coverage": "jest --coverage",
-    "benchmark": "yarn build-benchmark && NODE_ENV=production node --expose-gc benchmarks/dist/index.cjs",
-    "benchmark:baseline": "yarn build && node --expose-gc benchmarks/baseline.js",
-    "benchmark:compare": "yarn build && node --expose-gc benchmarks/compare.js",
-    "benchmark:record": "yarn build && node --expose-gc benchmarks/captureBaseline.js",
-    "benchmark:record:reference": "yarn build && node --expose-gc benchmarks/captureBaseline.js --reference",
-    "benchmark:diff": "yarn build && node --expose-gc benchmarks/diffBaseline.js",
+    "benchmark": "npm run build-benchmark && NODE_ENV=production node --expose-gc benchmarks/dist/index.cjs",
+    "benchmark:baseline": "npm run build && node --expose-gc benchmarks/baseline.js",
+    "benchmark:compare": "npm run build && node --expose-gc benchmarks/compare.js",
+    "benchmark:record": "npm run build && node --expose-gc benchmarks/captureBaseline.js",
+    "benchmark:record:reference": "npm run build && node --expose-gc benchmarks/captureBaseline.js --reference",
+    "benchmark:diff": "npm run build && node --expose-gc benchmarks/diffBaseline.js",
     "benchmark:diff:latest": "node --expose-gc benchmarks/diffBaseline.js --latest",
-    "benchmark:baseline:update": "yarn benchmark:record:reference",
-    "build-benchmark": "BENCHMARKS=true yarn build",
-    "build": "yarn clean-build && NODE_ENV=production rollup -c && node scripts/postbuild-cjs.cjs",
+    "benchmark:baseline:update": "npm run benchmark:record:reference",
+    "build-benchmark": "BENCHMARKS=true npm run build",
+    "build": "npm run clean-build && NODE_ENV=production rollup -c && node scripts/postbuild-cjs.cjs",
     "clean-build": "rm -rf dist",
-    "build-minified": "MINIFY=true yarn build",
-    "build-docs": "typedoc --options typedoc.json && yarn build-demo",
+    "build-minified": "MINIFY=true npm run build",
+    "build-docs": "typedoc --options typedoc.json && npm run build-demo",
     "build-demo": "mkdir -p ./docs/demo && cp -r ./examples/plain_js/. ./docs/demo",
     "lint": "eslint 'src/**/*.{js,ts}'",
     "lintfix": "eslint --fix 'src/**/*.{js,ts}'",
-    "prepublishOnly": "yarn test && yarn build"
+    "prepublishOnly": "npm test && npm run build"
   },
   "sideEffects": false
 }