@lde/search-typesense 0.3.0 → 0.4.0

package/README.md

@@ -6,7 +6,8 @@ domain-agnostic** – you supply a `SearchType`; this package never names your
 domain. It is the Typesense implementation of the `SearchEngine` port: it derives
 a collection schema from the field model, compiles the neutral `SearchQuery` into
 Typesense search params, runs it, reconstructs the engine-neutral `SearchResult`,
-and manages the search index lifecycle (blue/green rebuild).
+and manages the search index lifecycle as transactional `@lde/pipeline`
+writers (Blue/green Rebuild and In-place Rebuild).
 
 ## Collection schema and engine
 
@@ -39,56 +40,100 @@ direct use and testing.
 
 ## Indexing
 
-`rebuild` blue/green-rebuilds a search index in one call, straight from the
-declaration: it derives the collection schema from your `SearchType` (via
-`buildCollectionSchema`), creates a fresh versioned collection
-(`${name}_<timestamp>`), streams the documents into it in batches, atomically
-repoints the `name` alias to it, then drops the collection it superseded. The
-caller passes the `SearchType`, the logical index `name` and a stream of
-documents; the versioned collection and the alias are managed for them.
+Indexing runs through two transactional writers, one per update mode – the
+[NDE Stack](https://docs.nde.nl/stack/patterns) patterns of the same names:
+
+- [**Blue/green Rebuild**](#bluegreen-rebuild: build a fresh index, then swap to it atomically;
+- [**In-place Rebuild**](#in-place-rebuild): update the live index directly by
+  upserting changed sources and sweeping the rest.
+
+Both implement `@lde/pipeline`’s `Writer` – each run is
+`openRun(context)` → `write` per dataset → `commit()` or `abort(error)` – so
+an `@lde/pipeline` `Pipeline` drives them without branching on the mode. Both
+derive the collection schema from your `SearchType` (via
+`buildCollectionSchema`), and their options accept everything
+`buildCollectionSchema` does (`defaultLocale`, `defaultSortingField`,
+`synonymSets`) plus the tuning knobs (`batchSize`, `lockTtlMs`).
+
+### Blue/green Rebuild
+
+`BlueGreenRebuild` rebuilds the index from zero and goes live atomically:
+`openRun` creates a fresh versioned collection (`${name}_<timestamp>`),
+`write` streams documents into it in batches (each stamped with its `source`
+dataset IRI), and `commit` atomically repoints the `name` alias and drops the
+collection it superseded. Until commit, the live alias never points at a
+partial build; `abort` drops the half-built collection. Deletion is implicit –
+whatever a run does not write does not exist in the new collection. A dataset
+that fails (or is reset before a dump re-run) is rolled back out of the
+not-yet-live collection by `source`, so the swap never ships a half-processed
+dataset. Right-sized for small collections (e.g. one document per dataset
+description).
 
 ```ts
 import { Client } from 'typesense';
-import { rebuild } from '@lde/search-typesense';
+import { BlueGreenRebuild } from '@lde/search-typesense';
 
 const client = new Client({
   nodes: [{ host, port, protocol: 'https' }],
   apiKey,
 });
 
-// `documents` is an async iterable (e.g. a streaming projection); only one
-// batch is held in memory at a time. `rebuild` returns the live collection name
-// and the imported count (or `null` if another rebuild was already running).
-const result = await rebuild(client, documents, DATASET, { name: 'datasets' });
+const writer = new BlueGreenRebuild(client, DATASET, { name: 'datasets' });
+// Standalone use; under @lde/pipeline the Pipeline drives this lifecycle.
+const run = await writer.openRun(context);
+await run.write(dataset, documents);
+await run.commit();
 ```
 
-The options accept everything `buildCollectionSchema` does (`defaultLocale`,
-`defaultSortingField`, `synonymSets`) plus the rebuild knobs (`batchSize`,
-`lockTtlMs`).
+### In-place Rebuild
 
-`rebuild` takes a `Client` the caller owns (and reuses for queries), so this
+`InPlaceRebuild` maintains one long-lived collection with per-source
+atomicity – no swap, no staging – for large, mostly-static corpora (e.g.
+millions of objects across many datasets, where a daily run touches only the
+changed ones). Every document is stamped with its `source` (the dataset IRI)
+and `last_seen` (the run id); deletion is a sweep, never special-cased:
+
+- a successful dataset flush deletes the source’s documents the run did not
+  rewrite (`source = dataset && last_seen != runId`); a failed dataset is not
+  swept – its output is incomplete – and the next successful run reconciles;
+- `commit` deletes every document whose source left the run’s selection (the
+  registry-membership sweep over `RunContext.selectedSources()`, which
+  includes datasets skipped as unchanged);
+- `abort` only releases the lock: upserts are idempotent, so whatever landed
+  stays until the next run reconciles.
+
+Document ids must be unique per (source, entity) – the caller keys them.
+
+Both writers take a `Client` the caller owns (and reuses for queries), so this
 package adds no connection or document type of its own – any object with an `id`
 is a valid document, including the `SearchDocument`s `@lde/search` produces.
 
 ## Concurrency
 
-`rebuild` is **single-flight per index**: it first takes a lock (a marker
+Rebuilds are **single-flight per index**: `openRun` takes a lock (a marker
 document in a `rebuild_locks` collection, created on demand) via Typesense’s
-atomic create, so concurrent callers across pods never rebuild the same index at
-once. A call made while another rebuild for that index is in flight returns
-`null` instead of a count. This keeps blue/green safe under replication: without
-it, two same-millisecond rebuilds would collide on the versioned collection name
-and one would delete the other’s in-flight build.
+atomic create, so concurrent runs across pods never rebuild the same index at
+once – a run opened while another holds the lock throws
+`RebuildAlreadyRunning` (catch it to treat a concurrent rebuild as a graceful
+skip). This keeps blue/green safe under replication: without it, two
+same-millisecond rebuilds would collide on the versioned collection name and
+one would delete the other’s in-flight build.
 
 Limitations to design around:
 
 - **Advisory, not a strict mutex.** The lock is built on Typesense, not a
   consensus store. Under a TTL-reclaim race two rebuilds can briefly run at
   once; this is safe because blue/green is idempotent (worst case: redundant
-  work and a transient orphaned collection).
-- **Single-flight, not coalescing.** A call skipped with `null` is _not_ queued.
-  If you must capture state that changed mid-build, re-trigger after the running
-  rebuild finishes.
+  work and a transient orphaned collection) and in-place upserts are
+  idempotent.
+- **Single-flight, not coalescing.** A run refused with `RebuildAlreadyRunning`
+  is _not_ queued. If you must capture state that changed mid-build,
+  re-trigger after the running rebuild finishes.
 - **Lock TTL.** A rebuild running longer than `lockTtlMs` (default 10 minutes)
   can be reclaimed by another caller and run concurrently; size the TTL above
   your longest rebuild.
+- **Membership-sweep cap.** The in-place membership sweep enumerates distinct
+  sources via a single facet, capped at `maxSweepableSources` (default
+  10 000); beyond that the commit throws rather than sweeping blind. Raise the
+  option (up to the engine’s `max_facet_values` limit) before an index nears
+  the cap so it stays a tunable guard rather than a hard wall.

package/dist/blue-green-rebuild.d.ts

@@ -0,0 +1,46 @@
+import type { Client } from 'typesense';
+import type { SearchType } from '@lde/search';
+import type { RunContext, RunWriter, Writer } from '@lde/pipeline';
+import { type RebuildOptions } from './rebuild-support.js';
+/** {@link BlueGreenRebuild} options: the collection-schema options (`name` is
+ *  the logical index name the alias is kept on) plus the rebuild tuning knobs. */
+export type BlueGreenRebuildOptions = RebuildOptions;
+/**
+ * Blue/green Rebuild (build a fresh index alongside the live one, then swap to
+ * it atomically) as a transactional `Writer`: each run builds the search
+ * index `options.name` from zero in a fresh versioned collection and goes
+ * live atomically on commit. Deletion is implicit – whatever the run does not
+ * write does not exist in the new collection. The name is the NDE Stack’s
+ * pattern ({@link https://docs.nde.nl/stack/patterns | Stack patterns}).
+ *
+ * - `openRun` takes the single-flight cross-pod lock ({@link openLockedRun},
+ *   throwing `RebuildAlreadyRunning` when another rebuild holds it) and
+ *   creates the versioned collection (`${name}_<timestamp>`) with the schema
+ *   derived from the {@link SearchType}.
+ * - `write` streams documents into the fresh collection, batched across write
+ *   calls, each stamped with its `source` (the dataset IRI).
+ * - `flush` on a **failed** dataset, and `reset` (the pipeline’s dump-fallback
+ *   discard), roll that dataset’s documents back out of the not-yet-live
+ *   collection by `source`, so a swap never ships a half-processed dataset;
+ *   a successful flush leaves the streamed documents in place.
+ * - `commit` imports the remainder, atomically repoints the `name` alias to
+ *   the new collection, drops the collection it superseded, and releases the
+ *   lock. Until commit, the live alias never points at a partial build.
+ * - `abort` drops the half-built collection and releases the lock; the live
+ *   index is untouched.
+ *
+ * The caller passes only the logical index `name`; the versioned collection
+ * name and the alias are managed here.
+ */
+export declare class BlueGreenRebuild<TDocument extends {
+    id: string;
+}> implements Writer<TDocument> {
+    private readonly client;
+    private readonly searchType;
+    private readonly options;
+    constructor(client: Client, searchType: SearchType, options: BlueGreenRebuildOptions);
+    openRun(context: RunContext): Promise<RunWriter<TDocument>>;
+    /** The collection an alias currently points at, or `undefined` if unset. */
+    private aliasTarget;
+}
+//# sourceMappingURL=blue-green-rebuild.d.ts.map

package/dist/blue-green-rebuild.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"blue-green-rebuild.d.ts","sourceRoot":"","sources":["../src/blue-green-rebuild.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,KAAK,EAEV,UAAU,EACV,SAAS,EACT,MAAM,EACP,MAAM,eAAe,CAAC;AAMvB,OAAO,EAKL,KAAK,cAAc,EACpB,MAAM,sBAAsB,CAAC;AAE9B;kFACkF;AAClF,MAAM,MAAM,uBAAuB,GAAG,cAAc,CAAC;AAErD;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AACH,qBAAa,gBAAgB,CAC3B,SAAS,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE,CAChC,YAAW,MAAM,CAAC,SAAS,CAAC;IAE1B,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAC3B,OAAO,CAAC,QAAQ,CAAC,OAAO;gBAFP,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,UAAU,EACtB,OAAO,EAAE,uBAAuB;IAM7C,OAAO,CAAC,OAAO,EAAE,UAAU,GAAG,OAAO,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC;IAoFjE,4EAA4E;YAC9D,WAAW;CAW1B"}

package/dist/blue-green-rebuild.js

@@ -0,0 +1,119 @@
+import { buildCollectionSchema } from './collection-schema.js';
+import { BatchImporter } from './import.js';
+import { httpStatus, openLockedRun, releaseLock } from './lock.js';
+import { SOURCE_FIELD, sourceDocumentsFilter } from './sweep.js';
+import { assertNoReservedFields, deleteByFilter, resolveRebuildOptions, stampDocuments, } from './rebuild-support.js';
+/**
+ * Blue/green Rebuild (build a fresh index alongside the live one, then swap to
+ * it atomically) as a transactional `Writer`: each run builds the search
+ * index `options.name` from zero in a fresh versioned collection and goes
+ * live atomically on commit. Deletion is implicit – whatever the run does not
+ * write does not exist in the new collection. The name is the NDE Stack’s
+ * pattern ({@link https://docs.nde.nl/stack/patterns | Stack patterns}).
+ *
+ * - `openRun` takes the single-flight cross-pod lock ({@link openLockedRun},
+ *   throwing `RebuildAlreadyRunning` when another rebuild holds it) and
+ *   creates the versioned collection (`${name}_<timestamp>`) with the schema
+ *   derived from the {@link SearchType}.
+ * - `write` streams documents into the fresh collection, batched across write
+ *   calls, each stamped with its `source` (the dataset IRI).
+ * - `flush` on a **failed** dataset, and `reset` (the pipeline’s dump-fallback
+ *   discard), roll that dataset’s documents back out of the not-yet-live
+ *   collection by `source`, so a swap never ships a half-processed dataset;
+ *   a successful flush leaves the streamed documents in place.
+ * - `commit` imports the remainder, atomically repoints the `name` alias to
+ *   the new collection, drops the collection it superseded, and releases the
+ *   lock. Until commit, the live alias never points at a partial build.
+ * - `abort` drops the half-built collection and releases the lock; the live
+ *   index is untouched.
+ *
+ * The caller passes only the logical index `name`; the versioned collection
+ * name and the alias are managed here.
+ */
+export class BlueGreenRebuild {
+    client;
+    searchType;
+    options;
+    constructor(client, searchType, options) {
+        this.client = client;
+        this.searchType = searchType;
+        this.options = options;
+        // `source` is stamped on every document for per-dataset rollback.
+        assertNoReservedFields(searchType, [SOURCE_FIELD]);
+    }
+    async openRun(context) {
+        const { name, batchSize, lockTtlMs, schemaOptions } = resolveRebuildOptions(this.options);
+        return openLockedRun(this.client, name, lockTtlMs, async () => {
+            // Create the fresh (blue) collection up front, so a failure surfaces
+            // before any dataset is processed. startedAt orders the versioned names;
+            // concurrent same-name runs are excluded by the lock.
+            const collection = `${name}_${Date.parse(context.startedAt)}`;
+            const previous = await this.aliasTarget(name);
+            const schema = buildCollectionSchema(this.searchType, schemaOptions);
+            await this.client.collections().create({
+                ...schema,
+                name: collection,
+                fields: [
+                    ...(schema.fields ?? []),
+                    { name: SOURCE_FIELD, type: 'string' },
+                ],
+            });
+            const importer = new BatchImporter(this.client, collection, batchSize);
+            // Drop a dataset’s streamed documents back out of the not-yet-live
+            // collection (a failed dataset, or a reset before the dump re-run). Any
+            // still buffered must land first, so the delete filter sees them.
+            const rollback = async (dataset) => {
+                await importer.flush();
+                await deleteByFilter(this.client, collection, sourceDocumentsFilter(dataset.iri.toString()));
+            };
+            return {
+                write: async (dataset, documents) => importer.add(stampDocuments(documents, {
+                    [SOURCE_FIELD]: dataset.iri.toString(),
+                })),
+                flush: async (dataset, outcome) => {
+                    // A successful dataset keeps its streamed documents (they go live at
+                    // the swap); a failed one is rolled back so the swap never ships it.
+                    if (outcome !== 'success') {
+                        await rollback(dataset);
+                    }
+                },
+                reset: async (dataset) => rollback(dataset),
+                commit: async () => {
+                    await importer.flush();
+                    await this.client
+                        .aliases()
+                        .upsert(name, { collection_name: collection });
+                    if (previous !== undefined && previous !== collection) {
+                        await this.client
+                            .collections(previous)
+                            .delete()
+                            .catch(() => undefined);
+                    }
+                    await releaseLock(this.client, name);
+                },
+                abort: async () => {
+                    // The live alias is untouched; just drop the orphaned half-built
+                    // collection rather than let it accumulate.
+                    await this.client
+                        .collections(collection)
+                        .delete()
+                        .catch(() => undefined);
+                    await releaseLock(this.client, name);
+                },
+            };
+        });
+    }
+    /** The collection an alias currently points at, or `undefined` if unset. */
+    async aliasTarget(alias) {
+        try {
+            const { collection_name } = await this.client.aliases(alias).retrieve();
+            return collection_name;
+        }
+        catch (error) {
+            if (httpStatus(error) === 404) {
+                return undefined;
+            }
+            throw error;
+        }
+    }
+}

package/dist/import.d.ts

@@ -0,0 +1,31 @@
+import type { Client } from 'typesense';
+/** Documents imported per Typesense request unless a writer is told otherwise. */
+export declare const DEFAULT_BATCH_SIZE = 1000;
+/**
+ * Accumulates documents across writes and upserts them into a collection in
+ * fixed-size batches, so many small writes (one dataset at a time) still land
+ * as few Typesense requests. Call {@link flush} to import the remainder.
+ */
+export declare class BatchImporter<TDocument extends {
+    id: string;
+}> {
+    private readonly client;
+    private readonly collection;
+    private readonly batchSize;
+    private buffer;
+    constructor(client: Client, collection: string, batchSize: number);
+    /** Add a stream of documents, importing a batch whenever one fills up. */
+    add(documents: AsyncIterable<TDocument>): Promise<void>;
+    /** Import any remaining documents. */
+    flush(): Promise<void>;
+    private importBuffer;
+}
+/**
+ * Create-or-replace one batch of whole documents, keyed on `id`, throwing if any
+ * individual document fails (Typesense’s bulk import otherwise reports
+ * per-document failures without rejecting).
+ */
+export declare function importBatch(client: Client, collection: string, batch: readonly {
+    id: string;
+}[]): Promise<void>;
+//# sourceMappingURL=import.d.ts.map

package/dist/import.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"import.d.ts","sourceRoot":"","sources":["../src/import.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAkB,MAAM,WAAW,CAAC;AAExD,kFAAkF;AAClF,eAAO,MAAM,kBAAkB,OAAO,CAAC;AAEvC;;;;GAIG;AACH,qBAAa,aAAa,CAAC,SAAS,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE;IAIvD,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAC3B,OAAO,CAAC,QAAQ,CAAC,SAAS;IAL5B,OAAO,CAAC,MAAM,CAAmB;gBAGd,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,MAAM,EAClB,SAAS,EAAE,MAAM;IAGpC,0EAA0E;IACpE,GAAG,CAAC,SAAS,EAAE,aAAa,CAAC,SAAS,CAAC,GAAG,OAAO,CAAC,IAAI,CAAC;IAS7D,sCAAsC;IAChC,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;YAMd,YAAY;CAK3B;AAED;;;;GAIG;AACH,wBAAsB,WAAW,CAC/B,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,MAAM,EAClB,KAAK,EAAE,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE,EAAE,GAC/B,OAAO,CAAC,IAAI,CAAC,CAkBf"}

package/dist/import.js

@@ -0,0 +1,60 @@
+/** Documents imported per Typesense request unless a writer is told otherwise. */
+export const DEFAULT_BATCH_SIZE = 1000;
+/**
+ * Accumulates documents across writes and upserts them into a collection in
+ * fixed-size batches, so many small writes (one dataset at a time) still land
+ * as few Typesense requests. Call {@link flush} to import the remainder.
+ */
+export class BatchImporter {
+    client;
+    collection;
+    batchSize;
+    buffer = [];
+    constructor(client, collection, batchSize) {
+        this.client = client;
+        this.collection = collection;
+        this.batchSize = batchSize;
+    }
+    /** Add a stream of documents, importing a batch whenever one fills up. */
+    async add(documents) {
+        for await (const document of documents) {
+            this.buffer.push(document);
+            if (this.buffer.length >= this.batchSize) {
+                await this.importBuffer();
+            }
+        }
+    }
+    /** Import any remaining documents. */
+    async flush() {
+        if (this.buffer.length > 0) {
+            await this.importBuffer();
+        }
+    }
+    async importBuffer() {
+        const batch = this.buffer;
+        this.buffer = [];
+        await importBatch(this.client, this.collection, batch);
+    }
+}
+/**
+ * Create-or-replace one batch of whole documents, keyed on `id`, throwing if any
+ * individual document fails (Typesense’s bulk import otherwise reports
+ * per-document failures without rejecting).
+ */
+export async function importBatch(client, collection, batch) {
+    const results = (await client
+        .collections(collection)
+        .documents()
+        .import(batch, {
+        action: 'upsert',
+        // Collect per-document outcomes instead of throwing the client’s opaque
+        // ImportError, so we can report which documents failed and why.
+        throwOnFail: false,
+    }));
+    const failures = results.filter((result) => !result.success);
+    if (failures.length > 0) {
+        throw new Error(`Typesense upsert into “${collection}” failed for ${failures.length}/${results.length} documents: ${failures
+            .map((failure) => failure.error)
+            .join('; ')}`);
+    }
+}

package/dist/in-place-rebuild.d.ts

@@ -0,0 +1,62 @@
+import type { Client } from 'typesense';
+import type { SearchType } from '@lde/search';
+import type { RunContext, RunWriter, Writer } from '@lde/pipeline';
+import { type RebuildOptions } from './rebuild-support.js';
+/** {@link InPlaceRebuild} options: the shared rebuild options plus the
+ *  membership-sweep ceiling. */
+export interface InPlaceRebuildOptions extends RebuildOptions {
+    /** Most distinct sources the membership sweep may enumerate before it throws
+     *  rather than risk missing departed sources (default 10 000). */
+    readonly maxSweepableSources?: number;
+}
+/**
+ * In-place Rebuild (update the live index directly – upsert changed sources,
+ * sweep the rest – rather than swap in a fresh one) as a transactional
+ * `Writer`: documents are upserted into one long-lived collection with
+ * per-source atomicity – no swap, no staging. Every document is stamped with
+ * its `source` (the dataset IRI) and `last_seen` (the run id); deletion is a
+ * sweep, never special-cased:
+ *
+ * - a **successful dataset flush** deletes the source’s documents the run did
+ *   not rewrite (`source = dataset && last_seen != runId`). A failed dataset
+ *   is not swept – its output is incomplete, and the next successful run
+ *   reconciles;
+ * - **reset** (the pipeline’s dump-fallback discard) deletes only *this run’s*
+ *   writes for the dataset (`source = dataset && last_seen = runId`), so the
+ *   dump re-run rebuilds it cleanly while the source’s prior-run documents are
+ *   left for the success sweep to reconcile;
+ * - **commit** deletes every document whose source left the run’s selection
+ *   (registry-membership sweep over {@link RunContext.selectedSources}, which
+ *   includes datasets skipped as unchanged) and releases the lock;
+ * - **abort** only releases the lock: upserts are idempotent, so whatever
+ *   landed stays until the next run reconciles.
+ *
+ * `openRun` takes the single-flight cross-pod lock (throwing
+ * `RebuildAlreadyRunning` when another rebuild holds it) and creates
+ * the collection on demand from the {@link SearchType} plus the two
+ * bookkeeping fields.
+ *
+ * Document ids must be unique per (source, entity) – the caller keys them –
+ * or documents from different sources overwrite each other.
+ *
+ * The name is the NDE Stack’s pattern
+ * ({@link https://docs.nde.nl/stack/patterns | Stack patterns}).
+ */
+export declare class InPlaceRebuild<TDocument extends {
+    id: string;
+}> implements Writer<TDocument> {
+    private readonly client;
+    private readonly searchType;
+    private readonly options;
+    constructor(client: Client, searchType: SearchType, options: InPlaceRebuildOptions);
+    openRun(context: RunContext): Promise<RunWriter<TDocument>>;
+    /**
+     * The distinct sources present in the collection, via a single `source`
+     * facet. Requests one bucket beyond `maxSources` so genuine truncation is
+     * distinguishable from an exactly-full result: `maxSources` buckets are
+     * returned intact, `maxSources + 1` proves more exist and the sweep would
+     * miss some, so it throws rather than delete blind.
+     */
+    private indexedSources;
+}
+//# sourceMappingURL=in-place-rebuild.d.ts.map

package/dist/in-place-rebuild.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"in-place-rebuild.d.ts","sourceRoot":"","sources":["../src/in-place-rebuild.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AAExC,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAE9C,OAAO,KAAK,EAEV,UAAU,EACV,SAAS,EACT,MAAM,EACP,MAAM,eAAe,CAAC;AAYvB,OAAO,EAKL,KAAK,cAAc,EACpB,MAAM,sBAAsB,CAAC;AAa9B;gCACgC;AAChC,MAAM,WAAW,qBAAsB,SAAQ,cAAc;IAC3D;sEACkE;IAClE,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAC;CACvC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAgCG;AACH,qBAAa,cAAc,CACzB,SAAS,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE,CAChC,YAAW,MAAM,CAAC,SAAS,CAAC;IAE1B,OAAO,CAAC,QAAQ,CAAC,MAAM;IACvB,OAAO,CAAC,QAAQ,CAAC,UAAU;IAC3B,OAAO,CAAC,QAAQ,CAAC,OAAO;gBAFP,MAAM,EAAE,MAAM,EACd,UAAU,EAAE,UAAU,EACtB,OAAO,EAAE,qBAAqB;IAK3C,OAAO,CAAC,OAAO,EAAE,UAAU,GAAG,OAAO,CAAC,SAAS,CAAC,SAAS,CAAC,CAAC;IAuFjE;;;;;;OAMG;YACW,cAAc;CAsB7B"}

package/dist/in-place-rebuild.js

@@ -0,0 +1,140 @@
+import { buildCollectionSchema } from './collection-schema.js';
+import { BatchImporter } from './import.js';
+import { ensureCollectionExists, openLockedRun, releaseLock } from './lock.js';
+import { LAST_SEEN_FIELD, SOURCE_FIELD, departedSources, membershipSweepFilters, staleDocumentsFilter, thisRunDocumentsFilter, } from './sweep.js';
+import { assertNoReservedFields, deleteByFilter, resolveRebuildOptions, stampDocuments, } from './rebuild-support.js';
+/**
+ * Default ceiling on the distinct sources the membership sweep enumerates via
+ * a single `source` facet. The sweep needs the complete source set to spot
+ * departed ones, so a truncated facet would silently miss deletions;
+ * {@link InPlaceRebuild} throws instead. Raise it (up to the engine’s
+ * `max_facet_values` limit) with {@link InPlaceRebuildOptions.maxSweepableSources}
+ * before an index approaches it, so the ceiling is a tunable guard rather than
+ * a hard wall.
+ */
+const DEFAULT_MAX_SWEEPABLE_SOURCES = 10_000;
+/**
+ * In-place Rebuild (update the live index directly – upsert changed sources,
+ * sweep the rest – rather than swap in a fresh one) as a transactional
+ * `Writer`: documents are upserted into one long-lived collection with
+ * per-source atomicity – no swap, no staging. Every document is stamped with
+ * its `source` (the dataset IRI) and `last_seen` (the run id); deletion is a
+ * sweep, never special-cased:
+ *
+ * - a **successful dataset flush** deletes the source’s documents the run did
+ *   not rewrite (`source = dataset && last_seen != runId`). A failed dataset
+ *   is not swept – its output is incomplete, and the next successful run
+ *   reconciles;
+ * - **reset** (the pipeline’s dump-fallback discard) deletes only *this run’s*
+ *   writes for the dataset (`source = dataset && last_seen = runId`), so the
+ *   dump re-run rebuilds it cleanly while the source’s prior-run documents are
+ *   left for the success sweep to reconcile;
+ * - **commit** deletes every document whose source left the run’s selection
+ *   (registry-membership sweep over {@link RunContext.selectedSources}, which
+ *   includes datasets skipped as unchanged) and releases the lock;
+ * - **abort** only releases the lock: upserts are idempotent, so whatever
+ *   landed stays until the next run reconciles.
+ *
+ * `openRun` takes the single-flight cross-pod lock (throwing
+ * `RebuildAlreadyRunning` when another rebuild holds it) and creates
+ * the collection on demand from the {@link SearchType} plus the two
+ * bookkeeping fields.
+ *
+ * Document ids must be unique per (source, entity) – the caller keys them –
+ * or documents from different sources overwrite each other.
+ *
+ * The name is the NDE Stack’s pattern
+ * ({@link https://docs.nde.nl/stack/patterns | Stack patterns}).
+ */
+export class InPlaceRebuild {
+    client;
+    searchType;
+    options;
+    constructor(client, searchType, options) {
+        this.client = client;
+        this.searchType = searchType;
+        this.options = options;
+        assertNoReservedFields(searchType, [SOURCE_FIELD, LAST_SEEN_FIELD]);
+    }
+    async openRun(context) {
+        const { maxSweepableSources = DEFAULT_MAX_SWEEPABLE_SOURCES, ...rebuildOptions } = this.options;
+        const { name, batchSize, lockTtlMs, schemaOptions } = resolveRebuildOptions(rebuildOptions);
+        return openLockedRun(this.client, name, lockTtlMs, async () => {
+            // Create the collection on demand: SearchType schema + the bookkeeping
+            // fields, `source` faceted so the membership sweep can enumerate the
+            // distinct sources.
+            await ensureCollectionExists(this.client, name, () => {
+                const schema = buildCollectionSchema(this.searchType, schemaOptions);
+                const bookkeeping = [
+                    { name: SOURCE_FIELD, type: 'string', facet: true },
+                    { name: LAST_SEEN_FIELD, type: 'string' },
+                ];
+                return {
+                    ...schema,
+                    fields: [...(schema.fields ?? []), ...bookkeeping],
+                };
+            });
+            const importer = new BatchImporter(this.client, name, batchSize);
+            return {
+                write: async (dataset, documents) => importer.add(stampDocuments(documents, {
+                    [SOURCE_FIELD]: dataset.iri.toString(),
+                    [LAST_SEEN_FIELD]: context.runId,
+                })),
+                flush: async (dataset, outcome) => {
+                    // Land the buffered documents first, so the sweep below never
+                    // deletes what this run just rewrote.
+                    await importer.flush();
+                    if (outcome !== 'success') {
+                        // A failed dataset’s output is incomplete: sweeping against it
+                        // would delete documents the run never got to rewrite. Leave the
+                        // stale ones for the next successful run to reconcile.
+                        return;
+                    }
+                    await deleteByFilter(this.client, name, staleDocumentsFilter(dataset.iri.toString(), context.runId));
+                },
+                reset: async (dataset) => {
+                    // Discard only this run’s partial writes for the source (the failed
+                    // endpoint attempt) so the dump re-run rebuilds it cleanly; the
+                    // source’s prior-run documents stay for the success sweep.
+                    await importer.flush();
+                    await deleteByFilter(this.client, name, thisRunDocumentsFilter(dataset.iri.toString(), context.runId));
+                },
+                commit: async () => {
+                    await importer.flush();
+                    const departed = departedSources(await this.indexedSources(name, maxSweepableSources), context.selectedSources());
+                    for (const filter of membershipSweepFilters(departed)) {
+                        await deleteByFilter(this.client, name, filter);
+                    }
+                    await releaseLock(this.client, name);
+                },
+                abort: async () => {
+                    await releaseLock(this.client, name);
+                },
+            };
+        });
+    }
+    /**
+     * The distinct sources present in the collection, via a single `source`
+     * facet. Requests one bucket beyond `maxSources` so genuine truncation is
+     * distinguishable from an exactly-full result: `maxSources` buckets are
+     * returned intact, `maxSources + 1` proves more exist and the sweep would
+     * miss some, so it throws rather than delete blind.
+     */
+    async indexedSources(name, maxSources) {
+        const response = await this.client
+            .collections(name)
+            .documents()
+            .search({
+            q: '*',
+            query_by: SOURCE_FIELD,
+            per_page: 0,
+            facet_by: SOURCE_FIELD,
+            max_facet_values: maxSources + 1,
+        });
+        const counts = response.facet_counts?.[0]?.counts ?? [];
+        if (counts.length > maxSources) {
+            throw new Error(`Membership sweep cannot see beyond ${maxSources} distinct sources in “${name}”; raise maxSweepableSources or departed sources might be missed`);
+        }
+        return counts.map((count) => count.value);
+    }
+}

package/dist/index.d.ts

@@ -1,5 +1,9 @@
-export { rebuild } from './adapter.js';
-export type { RebuildOptions } from './adapter.js';
+export { BlueGreenRebuild } from './blue-green-rebuild.js';
+export type { BlueGreenRebuildOptions } from './blue-green-rebuild.js';
+export { InPlaceRebuild } from './in-place-rebuild.js';
+export type { InPlaceRebuildOptions } from './in-place-rebuild.js';
+export { RebuildAlreadyRunning } from './lock.js';
+export { departedSources, membershipSweepFilters, sourceDocumentsFilter, staleDocumentsFilter, } from './sweep.js';
 export { buildCollectionSchema } from './collection-schema.js';
 export type { CollectionSchemaOptions } from './collection-schema.js';
 export { buildSearchParams } from './query-compiler.js';

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,qBAAqB,EAAE,MAAM,wBAAwB,CAAC;AAC/D,YAAY,EAAE,uBAAuB,EAAE,MAAM,wBAAwB,CAAC;AACtE,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAC;AACxD,YAAY,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpE,OAAO,EAAE,2BAA2B,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAC/E,YAAY,EACV,4BAA4B,EAC5B,uBAAuB,GACxB,MAAM,aAAa,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,gBAAgB,EAAE,MAAM,yBAAyB,CAAC;AAC3D,YAAY,EAAE,uBAAuB,EAAE,MAAM,yBAAyB,CAAC;AACvE,OAAO,EAAE,cAAc,EAAE,MAAM,uBAAuB,CAAC;AACvD,YAAY,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AACnE,OAAO,EAAE,qBAAqB,EAAE,MAAM,WAAW,CAAC;AAClD,OAAO,EACL,eAAe,EACf,sBAAsB,EACtB,qBAAqB,EACrB,oBAAoB,GACrB,MAAM,YAAY,CAAC;AACpB,OAAO,EAAE,qBAAqB,EAAE,MAAM,wBAAwB,CAAC;AAC/D,YAAY,EAAE,uBAAuB,EAAE,MAAM,wBAAwB,CAAC;AACtE,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAC;AACxD,YAAY,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpE,OAAO,EAAE,2BAA2B,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAC/E,YAAY,EACV,4BAA4B,EAC5B,uBAAuB,GACxB,MAAM,aAAa,CAAC"}

package/dist/index.js

@@ -1,4 +1,7 @@
-export { rebuild } from './adapter.js';
+export { BlueGreenRebuild } from './blue-green-rebuild.js';
+export { InPlaceRebuild } from './in-place-rebuild.js';
+export { RebuildAlreadyRunning } from './lock.js';
+export { departedSources, membershipSweepFilters, sourceDocumentsFilter, staleDocumentsFilter, } from './sweep.js';
 export { buildCollectionSchema } from './collection-schema.js';
 export { buildSearchParams } from './query-compiler.js';
 export { createTypesenseSearchEngine, parseSearchResponse } from './search.js';