@lde/search-pipeline 0.3.1 → 0.4.0

package/README.md

@@ -11,11 +11,13 @@ The division of labour ([ADR 6](../../docs/decisions/0006-make-the-writer-transa
 - [`@lde/search`](../search) owns the engine-agnostic projection (framed
   document + field model) and stays pipeline-free;
 - `@lde/search-<engine>` (e.g. [`@lde/search-typesense`](../search-typesense))
-  is the engine adapter: a transactional `Writer` consuming projected
-  documents, owning the run lifecycle (Blue/green alias swap or In-place
-  sweeps, cross-pod lock);
-- **this package** composes them: `searchIndexWriter` is the one
-  type-changing step (quad → document), shared across engines.
+  is the engine adapter: a transactional, **single-collection** `Writer`
+  consuming projected documents, owning the run lifecycle (Blue/green alias swap
+  or In-place sweeps, cross-pod lock);
+- **this package** composes them: `searchIndexWriter` is the one type-changing
+  step (quad → document), shared across engines, and – since one schema declares
+  several root types – it fans each document out to the engine writer for **its**
+  type’s collection ([ADR 9](../../docs/decisions/0009-route-a-whole-schema-projection-to-per-type-collections.md)).
 
 ## Usage
 
@@ -25,20 +27,41 @@ import { searchSchema } from '@lde/search';
 import { BlueGreenRebuild } from '@lde/search-typesense';
 import { searchIndexWriter } from '@lde/search-pipeline';
 
-const schema = searchSchema({
-  name: 'dataset',
-  type: 'http://www.w3.org/ns/dcat#Dataset',
-  fields: [
-    {
-      name: 'title',
-      kind: 'text',
-      path: 'http://purl.org/dc/terms/title',
-      locales: ['und'],
-      output: true,
-      searchable: true,
-    },
-  ],
-});
+// One schema, several root types: the `datasets` catalog plus the Organization
+// label collection its references resolve against.
+const schema = searchSchema(
+  {
+    name: 'Dataset',
+    type: 'http://www.w3.org/ns/dcat#Dataset',
+    fields: [
+      {
+        name: 'title',
+        kind: 'text',
+        path: 'http://purl.org/dc/terms/title',
+        locales: ['und'],
+        output: true,
+        searchable: { weight: 5 },
+      },
+    ],
+  },
+  {
+    name: 'Organization',
+    type: 'http://xmlns.com/foaf/0.1/Organization',
+    fields: [
+      {
+        name: 'label',
+        kind: 'text',
+        path: 'http://xmlns.com/foaf/0.1/name',
+        locales: ['und'],
+        output: true,
+        searchable: { weight: 1 },
+      },
+    ],
+  },
+);
+
+// Each type maps to its own collection: an independent blue/green rebuild.
+const collectionName = { Dataset: 'datasets', Organization: 'organizations' };
 
 const pipeline = new Pipeline({
   datasetSelector,
@@ -50,13 +73,10 @@ const pipeline = new Pipeline({
   ],
   writers: searchIndexWriter({
     schema,
-    writer: new BlueGreenRebuild(
-      typesenseClient,
-      schema.get('http://www.w3.org/ns/dcat#Dataset')!,
-      {
-        name: 'datasets',
-      },
-    ),
+    writerFor: (searchType) =>
+      new BlueGreenRebuild(typesenseClient, searchType, {
+        name: collectionName[searchType.name],
+      }),
   }),
 });
 
@@ -64,12 +84,31 @@ await pipeline.run();
 ```
 
 Each dataset’s extracted CONSTRUCT quads are buffered until the dataset
-completes, then framed by root type and projected into flat search documents
-(`@lde/search`’s `projectGraph`), and written to the engine run – before the
-engine acts on the dataset’s completion, so an In-place stale sweep never
-races its own documents. The run lifecycle (run context, per-dataset flush
-outcome, commit/abort) passes through unchanged: the engine writer’s update
-mode governs, and the pipeline never branches on it.
+completes, then projected once over the **whole** schema (`@lde/search`’s
+`projectGraph`) into one mixed, type-tagged stream, and each document is
+dispatched to the engine run for its type’s collection – before the engine acts
+on the dataset’s completion, so an In-place stale sweep never races its own
+documents. The run lifecycle (run context, per-dataset flush outcome,
+commit/abort) passes through unchanged: the engine writer’s update mode governs,
+and the pipeline never branches on it, nor on how many collections there are.
+
+## Per-collection isolation
+
+Each root type is an independent engine run – its own collection, alias and
+cross-pod lock – so the collections commit, sweep and fail in isolation:
+
+- a type whose projection is empty this run affects only its own collection,
+  never another’s – in particular the `datasets` index still goes live;
+- `commit` finalizes every collection independently and, if any fails, throws an
+  `AggregateError` _after_ attempting them all, so a non-critical
+  label-collection failure never blocks the collections that did commit, while
+  the failure is still surfaced (the run is marked failed);
+- because the pipeline aborts a run whose `commit` throws, `abort` finalizes only
+  the collections that did **not** already go live (aborting a committed
+  blue/green rebuild would drop its now-live collection), dropping the half-built
+  ones and releasing their locks.
+
+See [ADR 9](../../docs/decisions/0009-route-a-whole-schema-projection-to-per-type-collections.md).
 
 ## Memory
 

package/dist/search-index-writer.d.ts

@@ -1,31 +1,55 @@
 import type { Quad } from '@rdfjs/types';
 import type { Writer } from '@lde/pipeline';
-import { type SearchDocument, type SearchSchema } from '@lde/search';
+import { type SearchDocument, type SearchSchema, type SearchType } from '@lde/search';
 /** Options for {@link searchIndexWriter}. */
 export interface SearchIndexWriterOptions {
     /**
      * The declarative schema driving the projection: one {@link SearchType} per
-     * root type, each mapping framed RDF to flat search-document fields.
+     * root type, each mapping framed RDF to flat search-document fields. The
+     * writer opens one engine run per type in it.
      */
     schema: SearchSchema;
     /**
-     * The engine adapter the projected documents are written to – e.g.
-     * `@lde/search-typesense`’s `BlueGreenRebuild` or `InPlaceRebuild`. The
-     * run lifecycle (context, per-dataset flush outcome, commit/abort) is
-     * forwarded unchanged, so the engine writer’s update mode governs.
+     * The engine writer that owns a given root type’s collection – e.g. a
+     * `@lde/search-typesense` `BlueGreenRebuild` bound to that type and the
+     * collection name it keeps its alias on. Called once per {@link SearchType}
+     * in the schema when the writer is built.
+     *
+     * A single-collection deployment returns one writer for its one type; a
+     * multi-collection deployment (the Dataset Register’s `datasets` plus its
+     * Organization / Class / TerminologySource label collections) returns a
+     * distinct writer per type, each an independent blue/green rebuild with its
+     * own collection, alias and cross-pod lock. The projection is whole-schema
+     * either way; the per-collection fan-out is this writer’s job.
      */
-    writer: Writer<SearchDocument>;
+    writerFor: (searchType: SearchType) => Writer<SearchDocument>;
 }
 /**
- * The projection step of a search-indexing pipeline, as a quad `Writer`: it
- * turns each dataset’s extracted CONSTRUCT quads into engine-agnostic search
- * documents ({@link projectGraph}: frame by root type, then project each node
- * with its type’s declaration) and feeds them to the engine writer. This is
- * the one type-changing step (quad → document), shared across engines –
- * which is why it lives here and not inside an engine adapter.
+ * The projection step of a search-indexing pipeline, as a quad `Writer`, fanned
+ * out across a type’s collections. It turns each dataset’s extracted CONSTRUCT
+ * quads into engine-agnostic search documents ({@link projectGraph}: frame by
+ * root type, then project each node with its type’s declaration) and dispatches
+ * each document to the engine run for **its** type. This is the one
+ * type-changing step (quad → document), shared across engines – which is why it
+ * lives here and not inside an engine adapter.
+ *
+ * Each root type is an independent engine run (its own collection, alias and
+ * lock), so the collections commit, sweep and fail in isolation:
+ *
+ * - a type whose projection is empty this run affects only its own collection,
+ *   never another’s – in particular the `datasets` index still goes live;
+ * - `commit` finalizes every collection independently and, if any fails, throws
+ *   an `AggregateError` *after* attempting them all, so a non-critical
+ *   label-collection failure never blocks the collections that did commit,
+ *   while the failure is still surfaced (the pipeline marks the run failed).
+ *   Because the pipeline then calls {@link RunWriter.abort}, `abort` finalizes
+ *   only the collections that did **not** already go live – aborting a
+ *   committed blue/green rebuild would drop its now-live collection;
+ * - `abort` (a run failure, or a partial commit) drops every half-built
+ *   collection that has not committed and leaves the live ones untouched.
  *
  * A dataset’s quads are buffered until its flush and projected then, so the
- * documents land before the engine acts on the dataset’s completion (e.g. an
+ * documents land before every engine acts on the dataset’s completion (e.g. an
  * In-place stale sweep). Memory is bounded by one dataset’s extraction, and
  * released at each flush – nothing accumulates across datasets. Streaming
  * bounded entity batches *within* one huge dataset needs the two-level

package/dist/search-index-writer.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"search-index-writer.d.ts","sourceRoot":"","sources":["../src/search-index-writer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAEzC,OAAO,KAAK,EAIV,MAAM,EACP,MAAM,eAAe,CAAC;AACvB,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,YAAY,EAClB,MAAM,aAAa,CAAC;AAErB,6CAA6C;AAC7C,MAAM,WAAW,wBAAwB;IACvC;;;OAGG;IACH,MAAM,EAAE,YAAY,CAAC;IACrB;;;;;OAKG;IACH,MAAM,EAAE,MAAM,CAAC,cAAc,CAAC,CAAC;CAChC;AAED;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,iBAAiB,CAC/B,OAAO,EAAE,wBAAwB,GAChC,MAAM,CAAC,IAAI,CAAC,CA8Dd"}
+{"version":3,"file":"search-index-writer.d.ts","sourceRoot":"","sources":["../src/search-index-writer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAEzC,OAAO,KAAK,EAIV,MAAM,EACP,MAAM,eAAe,CAAC;AACvB,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,YAAY,EACjB,KAAK,UAAU,EAChB,MAAM,aAAa,CAAC;AAErB,6CAA6C;AAC7C,MAAM,WAAW,wBAAwB;IACvC;;;;OAIG;IACH,MAAM,EAAE,YAAY,CAAC;IACrB;;;;;;;;;;;;OAYG;IACH,SAAS,EAAE,CAAC,UAAU,EAAE,UAAU,KAAK,MAAM,CAAC,cAAc,CAAC,CAAC;CAC/D;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AACH,wBAAgB,iBAAiB,CAC/B,OAAO,EAAE,wBAAwB,GAChC,MAAM,CAAC,IAAI,CAAC,CA2Id"}

package/dist/search-index-writer.js

@@ -1,34 +1,90 @@
 import { projectGraph, } from '@lde/search';
 /**
- * The projection step of a search-indexing pipeline, as a quad `Writer`: it
- * turns each dataset’s extracted CONSTRUCT quads into engine-agnostic search
- * documents ({@link projectGraph}: frame by root type, then project each node
- * with its type’s declaration) and feeds them to the engine writer. This is
- * the one type-changing step (quad → document), shared across engines –
- * which is why it lives here and not inside an engine adapter.
+ * The projection step of a search-indexing pipeline, as a quad `Writer`, fanned
+ * out across a type’s collections. It turns each dataset’s extracted CONSTRUCT
+ * quads into engine-agnostic search documents ({@link projectGraph}: frame by
+ * root type, then project each node with its type’s declaration) and dispatches
+ * each document to the engine run for **its** type. This is the one
+ * type-changing step (quad → document), shared across engines – which is why it
+ * lives here and not inside an engine adapter.
+ *
+ * Each root type is an independent engine run (its own collection, alias and
+ * lock), so the collections commit, sweep and fail in isolation:
+ *
+ * - a type whose projection is empty this run affects only its own collection,
+ *   never another’s – in particular the `datasets` index still goes live;
+ * - `commit` finalizes every collection independently and, if any fails, throws
+ *   an `AggregateError` *after* attempting them all, so a non-critical
+ *   label-collection failure never blocks the collections that did commit,
+ *   while the failure is still surfaced (the pipeline marks the run failed).
+ *   Because the pipeline then calls {@link RunWriter.abort}, `abort` finalizes
+ *   only the collections that did **not** already go live – aborting a
+ *   committed blue/green rebuild would drop its now-live collection;
+ * - `abort` (a run failure, or a partial commit) drops every half-built
+ *   collection that has not committed and leaves the live ones untouched.
  *
  * A dataset’s quads are buffered until its flush and projected then, so the
- * documents land before the engine acts on the dataset’s completion (e.g. an
+ * documents land before every engine acts on the dataset’s completion (e.g. an
  * In-place stale sweep). Memory is bounded by one dataset’s extraction, and
  * released at each flush – nothing accumulates across datasets. Streaming
  * bounded entity batches *within* one huge dataset needs the two-level
  * iteration (dataset → entity-URI batches) and is not implemented yet.
  */
 export function searchIndexWriter(options) {
-    const { schema, writer } = options;
+    const { schema, writerFor } = options;
+    // One engine writer per root type, built once; each run opens them all. Keyed
+    // by the type IRI, which is also how a projected document names its type.
+    const writers = new Map([...schema.values()].map((searchType) => [
+        searchType.type,
+        writerFor(searchType),
+    ]));
     return {
         async openRun(context) {
-            const run = await writer.openRun(context);
+            const runs = new Map();
+            try {
+                for (const [typeIri, writer] of writers) {
+                    runs.set(typeIri, await writer.openRun(context));
+                }
+            }
+            catch (error) {
+                // One engine run failed to open (e.g. its lock is held); roll the
+                // already-opened ones back so no collection or lock is left dangling.
+                await Promise.allSettled([...runs.values()].map((run) => run.abort(error)));
+                throw error;
+            }
+            // The collections that have gone live, so `abort` never re-finalizes one
+            // (a committed blue/green rebuild’s abort drops its now-live collection).
+            const committed = new Set();
             // One buffered pass per dataset, keyed by IRI. The pipeline processes
-            // datasets sequentially, but keeping the key explicit makes a write
-            // after another dataset's flush safe too.
+            // datasets sequentially, but keeping the key explicit makes a write after
+            // another dataset's flush safe too.
             const passes = new Map();
             const project = async (pass) => {
                 passes.delete(pass.dataset.iri.toString());
                 if (pass.quads.length === 0) {
                     return;
                 }
-                await run.write(pass.dataset, projectGraph(pass.quads, schema));
+                // Whole-schema projection into one mixed stream, split by type here so
+                // each type’s documents reach its own collection’s run.
+                const byType = new Map();
+                for await (const { searchType, document } of projectGraph(pass.quads, schema)) {
+                    const documents = byType.get(searchType.type);
+                    if (documents === undefined) {
+                        byType.set(searchType.type, [document]);
+                    }
+                    else {
+                        documents.push(document);
+                    }
+                }
+                // Every yielded type is a schema type, so it has a run; iterate the
+                // runs and write each the documents projected for its type (none, for a
+                // type absent from this pass).
+                for (const [typeIri, run] of runs) {
+                    const documents = byType.get(typeIri);
+                    if (documents !== undefined) {
+                        await run.write(pass.dataset, stream(documents));
+                    }
+                }
             };
             return {
                 write: async (dataset, quads) => {
@@ -47,24 +103,66 @@ export function searchIndexWriter(options) {
                     if (pass !== undefined) {
                         await project(pass);
                     }
-                    await run.flush?.(dataset, outcome);
+                    // Flush every collection independently: one collection’s flush failure
+                    // (a rollback, or an In-place stale sweep) must not skip another’s –
+                    // the pipeline isolates a flush error per dataset and still commits,
+                    // so a skipped collection would swap live with a dataset it should
+                    // have rolled back or swept. Every collection is flushed, not just the
+                    // ones that received documents this dataset: a collection an earlier
+                    // run held documents for still needs its sweep to reconcile.
+                    await settleAll(runs.values(), 'flush', (run) => run.flush?.(dataset, outcome));
                 },
                 reset: async (dataset) => {
-                    // Discard the buffered pass so the re-run replaces it, and let the
-                    // engine writer discard whatever it already holds for the dataset.
+                    // Discard the buffered pass so the re-run replaces it, and let every
+                    // collection’s run discard whatever it already holds for the dataset –
+                    // independently, so one collection’s reset failure never leaves
+                    // another holding the discarded pass’s documents into the re-run.
                     passes.delete(dataset.iri.toString());
-                    await run.reset?.(dataset);
+                    await settleAll(runs.values(), 'reset', (run) => run.reset?.(dataset));
                 },
                 commit: async () => {
-                    // Safety net: project passes that were never flushed, so a
-                    // committed run never silently drops written quads.
+                    // Safety net: project passes that were never flushed, so a committed
+                    // run never silently drops written quads.
                     for (const pass of [...passes.values()]) {
                         await project(pass);
                     }
-                    await run.commit();
+                    // Commit every collection independently, so one failure neither
+                    // blocks nor wipes another – the `datasets` index goes live even if a
+                    // label collection cannot. Record each collection that went live, so
+                    // the abort that follows a failed commit never drops it.
+                    await settleAll(runs, 'commit', async ([typeIri, run]) => {
+                        await run.commit();
+                        committed.add(typeIri);
+                    });
+                },
+                abort: async (error) => {
+                    // Finalize only the collections that have not gone live: aborting a
+                    // committed blue/green rebuild would drop its now-live collection.
+                    // Best-effort – cleanup failures must not mask the original error.
+                    await Promise.allSettled([...runs]
+                        .filter(([typeIri]) => !committed.has(typeIri))
+                        .map(([, run]) => run.abort(error)));
                 },
-                abort: (error) => run.abort(error),
             };
         },
     };
 }
+/**
+ * Run `operation` on every collection concurrently and independently, so one
+ * collection’s failure never skips another’s (a flush’s rollback, a reset’s
+ * discard, a commit’s alias swap). Surface the failures together once all have
+ * been attempted, as an `AggregateError` – the run is still marked failed, but
+ * every collection got its chance to reconcile or go live first.
+ */
+async function settleAll(items, verb, operation) {
+    const targets = [...items];
+    const outcomes = await Promise.allSettled(targets.map((item) => operation(item)));
+    const failures = outcomes.flatMap((outcome) => outcome.status === 'rejected' ? [outcome.reason] : []);
+    if (failures.length > 0) {
+        throw new AggregateError(failures, `${failures.length} of ${targets.length} search collections failed to ${verb}`);
+    }
+}
+/** Yield the given documents as an async iterable, like a streaming source. */
+async function* stream(items) {
+    yield* items;
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@lde/search-pipeline",
-  "version": "0.3.1",
-  "description": "Search indexing as an @lde/pipeline instance: a Writer<Quad> that applies the @lde/search projection to extracted RDF and streams the documents to a transactional search-engine writer. Glue between the pipeline quad world and the search-engine document world; owns no projection or engine logic itself.",
+  "version": "0.4.0",
+  "description": "Search indexing as an @lde/pipeline instance: a Writer<Quad> that applies the @lde/search projection to extracted RDF and fans each document out to the transactional engine writer for its type's collection. Glue between the pipeline quad world and the search-engine document world; owns no projection or engine logic itself.",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
     "directory": "packages/search-pipeline"
@@ -25,12 +25,15 @@
     "!**/*.tsbuildinfo"
   ],
   "dependencies": {
-    "@lde/search": "^0.4.1",
+    "@lde/search": "^0.5.0",
     "@rdfjs/types": "^2.0.1",
     "tslib": "^2.3.0"
   },
   "devDependencies": {
-    "n3": "^2.1.1"
+    "@lde/search-typesense": "^0.7.0",
+    "n3": "^2.1.1",
+    "testcontainers": "^12.0.3",
+    "typesense": "^3.0.6"
   },
   "peerDependencies": {
     "@lde/dataset": "^0.7.8",