@lde/search-indexer 0.0.2 → 0.0.3

package/README.md

@@ -58,23 +58,24 @@ images.
 
 ## Configuration
 
-| Variable             | Default                     | Meaning                                                                      |
-| -------------------- | --------------------------- | ---------------------------------------------------------------------------- |
-| `SCHEMA_MODULE`      | `/config/search-schema.mjs` | Path of the mounted schema-declaration module                                |
-| `REGISTRY_ENDPOINT`  | **required**                | SPARQL endpoint of the DCAT dataset registry                                 |
-| `DATASETS`           | all registry datasets       | Dataset IRIs to index (whitespace- or comma-separated)                       |
-| `DATASET_CRITERIA`   | all registry datasets       | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`) |
-| `TYPESENSE_HOST`     | **required**                | Typesense host                                                               |
-| `TYPESENSE_PORT`     | `8108`                      | Typesense port                                                               |
-| `TYPESENSE_PROTOCOL` | `http`                      | `http` or `https`                                                            |
-| `TYPESENSE_API_KEY`  | **required**                | An admin key: the indexer creates, writes and swaps collections              |
-| `REBUILD_MODE`       | `in-place`                  | `in-place` (update the live collection) or `blue-green` (swap on commit)     |
-| `COLLECTION_PREFIX`  | none                        | Prefix for every derived collection name (configure the read side to match)  |
-| `PROVENANCE_FILE`    | none                        | JSON file remembering per-dataset processing, to skip unchanged datasets     |
-| `PIPELINE_VERSION`   | none                        | Version keying the skip decisions; required with `PROVENANCE_FILE`           |
-| `QLEVER_IMAGE`       | none                        | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest`  |
-| `IMPORT_STRATEGY`    | `sparql`                    | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE`    |
-| `DATA_DIR`           | `/data`                     | Directory for downloaded dumps and QLever index caches                       |
+| Variable             | Default                     | Meaning                                                                                             |
+| -------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- |
+| `SCHEMA_MODULE`      | `/config/search-schema.mjs` | Path of the mounted schema-declaration module                                                       |
+| `REGISTRY_ENDPOINT`  | **required**                | SPARQL endpoint of the DCAT dataset registry                                                        |
+| `DATASETS`           | all registry datasets       | Dataset IRIs to index (whitespace- or comma-separated)                                              |
+| `DATASET_CRITERIA`   | all registry datasets       | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`)                        |
+| `TYPESENSE_HOST`     | **required**                | Typesense host                                                                                      |
+| `TYPESENSE_PORT`     | `8108`                      | Typesense port                                                                                      |
+| `TYPESENSE_PROTOCOL` | `http`                      | `http` or `https`                                                                                   |
+| `TYPESENSE_API_KEY`  | **required**                | An admin key: the indexer creates, writes and swaps collections                                     |
+| `REBUILD_MODE`       | `in-place`                  | `in-place` (update the live collection) or `blue-green` (swap on commit)                            |
+| `COLLECTION_PREFIX`  | none                        | Prefix for every derived collection name (configure the read side to match)                         |
+| `PROVENANCE_FILE`    | none                        | JSON file remembering per-dataset processing, to skip unchanged datasets                            |
+| `PIPELINE_VERSION`   | none                        | Version keying the skip decisions; required with `PROVENANCE_FILE`                                  |
+| `QLEVER_IMAGE`       | none                        | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest`                         |
+| `IMPORT_STRATEGY`    | `sparql`                    | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE`                           |
+| `DATA_DIR`           | `/data`                     | Directory for downloaded dumps and QLever index caches                                              |
+| `QLEVER_NETWORK`     | none                        | Docker network the spawned QLever joins; set when the indexer itself runs containerized (see below) |
 
 A misconfigured boot reports **all** problems in one error, not one per crash
 loop. `PROVENANCE_FILE` must sit on a durable volume, and cannot be combined
@@ -103,6 +104,27 @@ explains why a statically-declared QLever service cannot work). That requires:
   the spawned QLever containers – the pipeline passes `DATA_DIR` as a bind
   mount to a sibling container, where it resolves against the host.
 
+How the indexer reaches the QLever it spawned depends on where the indexer
+itself runs:
+
+- **On the host** (`npx @lde/search-indexer` with Docker available): leave
+  `QLEVER_NETWORK` unset. QLever’s port is published on the host and
+  addressed as `localhost`.
+- **In a container on a bridge network** (the `docker compose` default): set
+  `QLEVER_NETWORK` to a network the indexer is attached to. QLever joins that
+  network and is addressed by container name (`qlever-<network>`) instead – a
+  containerized indexer’s `localhost` is its own network namespace, so it
+  cannot reach a host-published port. Alternatively, run the indexer with
+  `network_mode: host` and leave `QLEVER_NETWORK` unset.
+
+`QLEVER_NETWORK` must name a network the indexer container is actually
+attached to – for compose, the prefixed name Docker creates (e.g.
+`myproject_default`), not the short service-file name. A wrong or unattached
+network is not caught at boot: each dataset imports fully, then fails when
+the endpoint never becomes reachable. And run at most one indexer per
+network – the QLever container name is derived from the network, so two
+indexers sharing one would remove each other’s QLever.
+
 This mode does not work on container runtimes without a Docker socket
 (containerd-based Kubernetes); there, run endpoint-only for now.
 

package/dist/composition.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CAclC"}
+{"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CA0BlC"}

package/dist/composition.js

@@ -40,11 +40,23 @@ export function distributionResolverFrom(config) {
     if (!config.qlever) {
         return undefined;
     }
-    const { importer, server } = createQlever({
-        mode: 'docker',
-        image: config.qlever.image,
-        dataDir: config.qlever.dataDir,
-    });
+    const { importer, server } = config.qlever.network
+        ? createQlever({
+            mode: 'docker',
+            image: config.qlever.image,
+            dataDir: config.qlever.dataDir,
+            network: config.qlever.network,
+            // The network-scoped name doubles as the endpoint hostname. It keeps
+            // stacks on different networks from removing each other’s QLever;
+            // indexers sharing one network would still collide, so run at most
+            // one indexer per network.
+            containerName: `qlever-${config.qlever.network}`,
+        })
+        : createQlever({
+            mode: 'docker',
+            image: config.qlever.image,
+            dataDir: config.qlever.dataDir,
+        });
     return new ImportResolver(new SparqlDistributionResolver(), {
         importer,
         server,

package/dist/config.d.ts

@@ -55,6 +55,11 @@ export interface QleverConfig {
     readonly strategy: 'sparql' | 'sparqlWithImportFallback' | 'import';
     /** Directory for downloaded dumps and QLever index caches. */
     readonly dataDir: string;
+    /** Docker network the spawned QLever joins (`QLEVER_NETWORK`), reached by
+     *  container name. Required when the indexer itself runs containerized on
+     *  a bridge network; without it QLever is published on the host and
+     *  addressed as `localhost`. */
+    readonly network?: string;
 }
 /**
  * Read the {@link IndexerConfig} from environment variables. Reports **all**

package/dist/config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB;;;oCAGgC;IAChC,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;CAC3B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}

package/dist/config.js

@@ -132,10 +132,14 @@ function provenanceFromEnvironment(environment, rebuildMode, problems) {
 function qleverFromEnvironment(environment, problems) {
     const image = environment['QLEVER_IMAGE'];
     const strategy = environment['IMPORT_STRATEGY'];
+    const network = environment['QLEVER_NETWORK'];
     if (!image) {
         if (strategy) {
             problems.push('IMPORT_STRATEGY requires QLEVER_IMAGE: without an import engine there is nothing to import into');
         }
+        if (network) {
+            problems.push('QLEVER_NETWORK requires QLEVER_IMAGE: without an import engine there is no QLever container to attach');
+        }
         return undefined;
     }
     if (strategy !== undefined &&
@@ -146,5 +150,6 @@ function qleverFromEnvironment(environment, problems) {
         image,
         strategy: (strategy ?? 'sparql'),
         dataDir: environment['DATA_DIR'] ?? '/data',
+        network,
     };
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lde/search-indexer",
-  "version": "0.0.2",
+  "version": "0.0.3",
   "description": "The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: loads a mounted schema-declaration module, dataset selection, Typesense connection and rebuild mode from the environment, then runs the object-grain searchIndexerPipeline once to completion. The composition layer that binds the engine-agnostic @lde/search-pipeline stages to @lde/search-typesense – the write-path counterpart of @lde/search-api-server.",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
@@ -34,7 +34,7 @@
     "@lde/search": "^0.12.2",
     "@lde/search-pipeline": "^0.12.5",
     "@lde/search-typesense": "^0.15.5",
-    "@lde/sparql-qlever": "^0.15.2",
+    "@lde/sparql-qlever": "^0.15.3",
     "commander": "^15.0.0",
     "tslib": "^2.3.0",
     "typesense": "^3.0.6"