@lde/search-indexer 0.0.2 → 0.0.4

package/README.md

@@ -7,18 +7,11 @@ projects each root type per the schema, and rebuilds the engine collections –
 then exits. Run it as a cron job; the rebuild writers’ cross-pod lock makes an
 overlapping run fail fast instead of corrupting a collection.
 
-This is the write-path counterpart of
-[`@lde/search-api-server`](../search-api-server)
-([#652](https://github.com/ldelements/lde/issues/652)): the composition layer
-that binds the engine-agnostic `searchIndexerPipeline` to the
-[`@lde/search-typesense`](../search-typesense) rebuild writers. Mount the
-**same schema module** into both images and point both at the same
-`TYPESENSE_*` coordinates, and the write and the read side cannot disagree
-about the schema. A deployment that needs more – a bespoke root selector,
-per-stage tuning, another engine – composes `searchStages`,
-`searchIndexWriter` and `Pipeline` directly instead.
+## Installation
 
-## Run
+```sh
+npm install @lde/search-indexer
+```
 
 ```sh
 docker run --rm \
@@ -29,109 +22,6 @@ docker run --rm \
   ghcr.io/ldelements/search-indexer
 ```
 
-Or without Docker (the same environment variables apply):
-
-```sh
-npx @lde/search-indexer
-```
-
-Dataset IRIs can also be passed as arguments, which override `DATASETS`:
-
-```sh
-npx @lde/search-indexer https://example.org/id/dataset/1
-```
-
-`--check` validates the configuration and schema module, then exits without
-indexing – for CI and init containers.
-
-## The schema module
-
-The mounted module default-exports the deployment’s search type declarations
-as **plain data** – see [the API server’s
-README](../search-api-server/README.md#the-schema-module) for the format and
-authoring guidance; both images load the module with the same
-`@lde/search/module` loader. The indexer reads only the default export: the
-declarations drive one pipeline stage and one engine collection per root type,
-and each field’s `path` drives the extraction CONSTRUCT. The read-side extras
-(`schemaOptions`, `engineOptions`) are ignored here, so one file serves both
-images.
-
-## Configuration
-
-| Variable             | Default                     | Meaning                                                                      |
-| -------------------- | --------------------------- | ---------------------------------------------------------------------------- |
-| `SCHEMA_MODULE`      | `/config/search-schema.mjs` | Path of the mounted schema-declaration module                                |
-| `REGISTRY_ENDPOINT`  | **required**                | SPARQL endpoint of the DCAT dataset registry                                 |
-| `DATASETS`           | all registry datasets       | Dataset IRIs to index (whitespace- or comma-separated)                       |
-| `DATASET_CRITERIA`   | all registry datasets       | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`) |
-| `TYPESENSE_HOST`     | **required**                | Typesense host                                                               |
-| `TYPESENSE_PORT`     | `8108`                      | Typesense port                                                               |
-| `TYPESENSE_PROTOCOL` | `http`                      | `http` or `https`                                                            |
-| `TYPESENSE_API_KEY`  | **required**                | An admin key: the indexer creates, writes and swaps collections              |
-| `REBUILD_MODE`       | `in-place`                  | `in-place` (update the live collection) or `blue-green` (swap on commit)     |
-| `COLLECTION_PREFIX`  | none                        | Prefix for every derived collection name (configure the read side to match)  |
-| `PROVENANCE_FILE`    | none                        | JSON file remembering per-dataset processing, to skip unchanged datasets     |
-| `PIPELINE_VERSION`   | none                        | Version keying the skip decisions; required with `PROVENANCE_FILE`           |
-| `QLEVER_IMAGE`       | none                        | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest`  |
-| `IMPORT_STRATEGY`    | `sparql`                    | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE`    |
-| `DATA_DIR`           | `/data`                     | Directory for downloaded dumps and QLever index caches                       |
-
-A misconfigured boot reports **all** problems in one error, not one per crash
-loop. `PROVENANCE_FILE` must sit on a durable volume, and cannot be combined
-with `REBUILD_MODE=blue-green`: a skipped dataset would be missing from the
-fresh collection the swap makes live. Its directory must be writable by the
-image’s runtime user (uid 1000) – the image pre-creates `/provenance` with
-that ownership, so a named volume mounted there just works, while a bind
-mount must be `chown`ed on the host. A run fails at start when the file is
-not writable, instead of silently never skipping.
+## Documentation
 
-## The QLever import path
-
-By default the indexer serves only datasets that publish a live SPARQL
-endpoint (the pipeline’s endpoint-only default). Setting `QLEVER_IMAGE`
-enables the import path: data dumps are downloaded to `DATA_DIR` and imported
-into a QLever instance the **pipeline itself creates and controls** – one
-sibling container per dataset, spawned over the Docker socket
-([ADR 16](../../docs/decisions/0016-ship-the-search-indexer-as-a-config-driven-image.md)
-explains why a statically-declared QLever service cannot work). That requires:
-
-- the Docker socket mounted into the indexer container
-  (`--volume /var/run/docker.sock:/var/run/docker.sock`), plus a group grant
-  to reach it as the image’s non-root user
-  (`--group-add $(stat -c %g /var/run/docker.sock)`);
-- `DATA_DIR` on a volume whose **host path is identical** for the indexer and
-  the spawned QLever containers – the pipeline passes `DATA_DIR` as a bind
-  mount to a sibling container, where it resolves against the host.
-
-This mode does not work on container runtimes without a Docker socket
-(containerd-based Kubernetes); there, run endpoint-only for now.
-
-## Building the image
-
-The image is built from the workspace’s own outputs – the compiled package,
-the same-commit builds of its `@lde/*` dependencies and a pruned lockfile –
-never from npm, so it exists for any commit
-([ADR 15](../../docs/decisions/0015-ship-the-served-search-api-as-a-docker-image-built-from-the-workspace.md)):
-
-```sh
-npx nx run @lde/search-indexer:docker:build   # → packages-search-indexer
-npx nx run @lde/search-indexer:docker:smoke   # boots it with --check
-```
-
-CI runs `docker:smoke` for affected PRs; each release rebuilds and pushes
-`ghcr.io/ldelements/search-indexer:<version>` (`.github/workflows/docker.yml`).
-
-## Programmatic use
-
-The bin is a thin wrapper over the exported API, usable in tests or a custom
-boot:
-
-```ts
-import {
-  configFromEnvironment,
-  createSearchIndexer,
-} from '@lde/search-indexer';
-
-const pipeline = await createSearchIndexer(configFromEnvironment(process.env));
-await pipeline.run();
-```
+See the [full documentation](https://ldelements.org/reference/search-indexer).

package/dist/composition.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CAclC"}
+{"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CA0BlC"}

package/dist/composition.js

@@ -40,11 +40,23 @@ export function distributionResolverFrom(config) {
     if (!config.qlever) {
         return undefined;
     }
-    const { importer, server } = createQlever({
-        mode: 'docker',
-        image: config.qlever.image,
-        dataDir: config.qlever.dataDir,
-    });
+    const { importer, server } = config.qlever.network
+        ? createQlever({
+            mode: 'docker',
+            image: config.qlever.image,
+            dataDir: config.qlever.dataDir,
+            network: config.qlever.network,
+            // The network-scoped name doubles as the endpoint hostname. It keeps
+            // stacks on different networks from removing each other’s QLever;
+            // indexers sharing one network would still collide, so run at most
+            // one indexer per network.
+            containerName: `qlever-${config.qlever.network}`,
+        })
+        : createQlever({
+            mode: 'docker',
+            image: config.qlever.image,
+            dataDir: config.qlever.dataDir,
+        });
     return new ImportResolver(new SparqlDistributionResolver(), {
         importer,
         server,

package/dist/config.d.ts

@@ -55,6 +55,11 @@ export interface QleverConfig {
     readonly strategy: 'sparql' | 'sparqlWithImportFallback' | 'import';
     /** Directory for downloaded dumps and QLever index caches. */
     readonly dataDir: string;
+    /** Docker network the spawned QLever joins (`QLEVER_NETWORK`), reached by
+     *  container name. Required when the indexer itself runs containerized on
+     *  a bridge network; without it QLever is published on the host and
+     *  addressed as `localhost`. */
+    readonly network?: string;
 }
 /**
  * Read the {@link IndexerConfig} from environment variables. Reports **all**

package/dist/config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB;;;oCAGgC;IAChC,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;CAC3B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}

package/dist/config.js

@@ -132,10 +132,14 @@ function provenanceFromEnvironment(environment, rebuildMode, problems) {
 function qleverFromEnvironment(environment, problems) {
     const image = environment['QLEVER_IMAGE'];
     const strategy = environment['IMPORT_STRATEGY'];
+    const network = environment['QLEVER_NETWORK'];
     if (!image) {
         if (strategy) {
             problems.push('IMPORT_STRATEGY requires QLEVER_IMAGE: without an import engine there is nothing to import into');
         }
+        if (network) {
+            problems.push('QLEVER_NETWORK requires QLEVER_IMAGE: without an import engine there is no QLever container to attach');
+        }
         return undefined;
     }
     if (strategy !== undefined &&
@@ -146,5 +150,6 @@ function qleverFromEnvironment(environment, problems) {
         image,
         strategy: (strategy ?? 'sparql'),
         dataDir: environment['DATA_DIR'] ?? '/data',
+        network,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lde/search-indexer",
-  "version": "0.0.2",
+  "version": "0.0.4",
   "description": "The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: loads a mounted schema-declaration module, dataset selection, Typesense connection and rebuild mode from the environment, then runs the object-grain searchIndexerPipeline once to completion. The composition layer that binds the engine-agnostic @lde/search-pipeline stages to @lde/search-typesense – the write-path counterpart of @lde/search-api-server.",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
@@ -31,10 +31,10 @@
     "@lde/dataset-registry-client": "^0.9.1",
     "@lde/pipeline": "^0.35.4",
     "@lde/pipeline-console-reporter": "^0.26.4",
-    "@lde/search": "^0.12.2",
-    "@lde/search-pipeline": "^0.12.5",
-    "@lde/search-typesense": "^0.15.5",
-    "@lde/sparql-qlever": "^0.15.2",
+    "@lde/search": "^0.12.3",
+    "@lde/search-pipeline": "^0.12.6",
+    "@lde/search-typesense": "^0.15.6",
+    "@lde/sparql-qlever": "^0.15.3",
     "commander": "^15.0.0",
     "tslib": "^2.3.0",
     "typesense": "^3.0.6"