@lde/search-indexer 0.0.1 → 0.0.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (7) hide show
  1. package/README.md +45 -19
  2. package/dist/composition.d.ts.map +1 -1
  3. package/dist/composition.js +17 -5
  4. package/dist/config.d.ts +5 -0
  5. package/dist/config.d.ts.map +1 -1
  6. package/dist/config.js +5 -0
  7. package/package.json +6 -6
package/README.md CHANGED
@@ -58,28 +58,33 @@ images.
58
58
 
59
59
  ## Configuration
60
60
 
61
- | Variable | Default | Meaning |
62
- | -------------------- | --------------------------- | ---------------------------------------------------------------------------- |
63
- | `SCHEMA_MODULE` | `/config/search-schema.mjs` | Path of the mounted schema-declaration module |
64
- | `REGISTRY_ENDPOINT` | **required** | SPARQL endpoint of the DCAT dataset registry |
65
- | `DATASETS` | all registry datasets | Dataset IRIs to index (whitespace- or comma-separated) |
66
- | `DATASET_CRITERIA` | all registry datasets | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`) |
67
- | `TYPESENSE_HOST` | **required** | Typesense host |
68
- | `TYPESENSE_PORT` | `8108` | Typesense port |
69
- | `TYPESENSE_PROTOCOL` | `http` | `http` or `https` |
70
- | `TYPESENSE_API_KEY` | **required** | An admin key: the indexer creates, writes and swaps collections |
71
- | `REBUILD_MODE` | `in-place` | `in-place` (update the live collection) or `blue-green` (swap on commit) |
72
- | `COLLECTION_PREFIX` | none | Prefix for every derived collection name (configure the read side to match) |
73
- | `PROVENANCE_FILE` | none | JSON file remembering per-dataset processing, to skip unchanged datasets |
74
- | `PIPELINE_VERSION` | none | Version keying the skip decisions; required with `PROVENANCE_FILE` |
75
- | `QLEVER_IMAGE` | none | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest` |
76
- | `IMPORT_STRATEGY` | `sparql` | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE` |
77
- | `DATA_DIR` | `/data` | Directory for downloaded dumps and QLever index caches |
61
+ | Variable | Default | Meaning |
62
+ | -------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- |
63
+ | `SCHEMA_MODULE` | `/config/search-schema.mjs` | Path of the mounted schema-declaration module |
64
+ | `REGISTRY_ENDPOINT` | **required** | SPARQL endpoint of the DCAT dataset registry |
65
+ | `DATASETS` | all registry datasets | Dataset IRIs to index (whitespace- or comma-separated) |
66
+ | `DATASET_CRITERIA` | all registry datasets | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`) |
67
+ | `TYPESENSE_HOST` | **required** | Typesense host |
68
+ | `TYPESENSE_PORT` | `8108` | Typesense port |
69
+ | `TYPESENSE_PROTOCOL` | `http` | `http` or `https` |
70
+ | `TYPESENSE_API_KEY` | **required** | An admin key: the indexer creates, writes and swaps collections |
71
+ | `REBUILD_MODE` | `in-place` | `in-place` (update the live collection) or `blue-green` (swap on commit) |
72
+ | `COLLECTION_PREFIX` | none | Prefix for every derived collection name (configure the read side to match) |
73
+ | `PROVENANCE_FILE` | none | JSON file remembering per-dataset processing, to skip unchanged datasets |
74
+ | `PIPELINE_VERSION` | none | Version keying the skip decisions; required with `PROVENANCE_FILE` |
75
+ | `QLEVER_IMAGE` | none | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest` |
76
+ | `IMPORT_STRATEGY` | `sparql` | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE` |
77
+ | `DATA_DIR` | `/data` | Directory for downloaded dumps and QLever index caches |
78
+ | `QLEVER_NETWORK` | none | Docker network the spawned QLever joins; set when the indexer itself runs containerized (see below) |
78
79
 
79
80
  A misconfigured boot reports **all** problems in one error, not one per crash
80
81
  loop. `PROVENANCE_FILE` must sit on a durable volume, and cannot be combined
81
82
  with `REBUILD_MODE=blue-green`: a skipped dataset would be missing from the
82
- fresh collection the swap makes live.
83
+ fresh collection the swap makes live. Its directory must be writable by the
84
+ image’s runtime user (uid 1000) – the image pre-creates `/provenance` with
85
+ that ownership, so a named volume mounted there just works, while a bind
86
+ mount must be `chown`ed on the host. A run fails at start when the file is
87
+ not writable, instead of silently never skipping.
83
88
 
84
89
  ## The QLever import path
85
90
 
@@ -99,6 +104,27 @@ explains why a statically-declared QLever service cannot work). That requires:
99
104
  the spawned QLever containers – the pipeline passes `DATA_DIR` as a bind
100
105
  mount to a sibling container, where it resolves against the host.
101
106
 
107
+ How the indexer reaches the QLever it spawned depends on where the indexer
108
+ itself runs:
109
+
110
+ - **On the host** (`npx @lde/search-indexer` with Docker available): leave
111
+ `QLEVER_NETWORK` unset. QLever’s port is published on the host and
112
+ addressed as `localhost`.
113
+ - **In a container on a bridge network** (the `docker compose` default): set
114
+ `QLEVER_NETWORK` to a network the indexer is attached to. QLever joins that
115
+ network and is addressed by container name (`qlever-<network>`) instead – a
116
+ containerized indexer’s `localhost` is its own network namespace, so it
117
+ cannot reach a host-published port. Alternatively, run the indexer with
118
+ `network_mode: host` and leave `QLEVER_NETWORK` unset.
119
+
120
+ `QLEVER_NETWORK` must name a network the indexer container is actually
121
+ attached to – for compose, the prefixed name Docker creates (e.g.
122
+ `myproject_default`), not the short service-file name. A wrong or unattached
123
+ network is not caught at boot: each dataset imports fully, then fails when
124
+ the endpoint never becomes reachable. And run at most one indexer per
125
+ network – the QLever container name is derived from the network, so two
126
+ indexers sharing one would remove each other’s QLever.
127
+
102
128
  This mode does not work on container runtimes without a Docker socket
103
129
  (containerd-based Kubernetes); there, run endpoint-only for now.
104
130
 
@@ -114,7 +140,7 @@ npx nx run @lde/search-indexer:docker:build # → packages-search-indexer
114
140
  npx nx run @lde/search-indexer:docker:smoke # boots it with --check
115
141
  ```
116
142
 
117
- CI runs `docker:smoke` for affected PRs; each release tag rebuilds and pushes
143
+ CI runs `docker:smoke` for affected PRs; each release rebuilds and pushes
118
144
  `ghcr.io/ldelements/search-indexer:<version>` (`.github/workflows/docker.yml`).
119
145
 
120
146
  ## Programmatic use
package/dist/composition.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CAclC"}
1
+ {"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CA0BlC"}
package/dist/composition.js CHANGED
@@ -40,11 +40,23 @@ export function distributionResolverFrom(config) {
40
40
  if (!config.qlever) {
41
41
  return undefined;
42
42
  }
43
- const { importer, server } = createQlever({
44
- mode: 'docker',
45
- image: config.qlever.image,
46
- dataDir: config.qlever.dataDir,
47
- });
43
+ const { importer, server } = config.qlever.network
44
+ ? createQlever({
45
+ mode: 'docker',
46
+ image: config.qlever.image,
47
+ dataDir: config.qlever.dataDir,
48
+ network: config.qlever.network,
49
+ // The network-scoped name doubles as the endpoint hostname. It keeps
50
+ // stacks on different networks from removing each other’s QLever;
51
+ // indexers sharing one network would still collide, so run at most
52
+ // one indexer per network.
53
+ containerName: `qlever-${config.qlever.network}`,
54
+ })
55
+ : createQlever({
56
+ mode: 'docker',
57
+ image: config.qlever.image,
58
+ dataDir: config.qlever.dataDir,
59
+ });
48
60
  return new ImportResolver(new SparqlDistributionResolver(), {
49
61
  importer,
50
62
  server,
package/dist/config.d.ts CHANGED
@@ -55,6 +55,11 @@ export interface QleverConfig {
55
55
  readonly strategy: 'sparql' | 'sparqlWithImportFallback' | 'import';
56
56
  /** Directory for downloaded dumps and QLever index caches. */
57
57
  readonly dataDir: string;
58
+ /** Docker network the spawned QLever joins (`QLEVER_NETWORK`), reached by
59
+ * container name. Required when the indexer itself runs containerized on
60
+ * a bridge network; without it QLever is published on the host and
61
+ * addressed as `localhost`. */
62
+ readonly network?: string;
58
63
  }
59
64
  /**
60
65
  * Read the {@link IndexerConfig} from environment variables. Reports **all**
package/dist/config.d.ts.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}
1
+ {"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;IACzB;;;oCAGgC;IAChC,QAAQ,CAAC,OAAO,CAAC,EAAE,MAAM,CAAC;CAC3B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}
package/dist/config.js CHANGED
@@ -132,10 +132,14 @@ function provenanceFromEnvironment(environment, rebuildMode, problems) {
132
132
  function qleverFromEnvironment(environment, problems) {
133
133
  const image = environment['QLEVER_IMAGE'];
134
134
  const strategy = environment['IMPORT_STRATEGY'];
135
+ const network = environment['QLEVER_NETWORK'];
135
136
  if (!image) {
136
137
  if (strategy) {
137
138
  problems.push('IMPORT_STRATEGY requires QLEVER_IMAGE: without an import engine there is nothing to import into');
138
139
  }
140
+ if (network) {
141
+ problems.push('QLEVER_NETWORK requires QLEVER_IMAGE: without an import engine there is no QLever container to attach');
142
+ }
139
143
  return undefined;
140
144
  }
141
145
  if (strategy !== undefined &&
@@ -146,5 +150,6 @@ function qleverFromEnvironment(environment, problems) {
146
150
  image,
147
151
  strategy: (strategy ?? 'sparql'),
148
152
  dataDir: environment['DATA_DIR'] ?? '/data',
153
+ network,
149
154
  };
150
155
  }
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@lde/search-indexer",
3
- "version": "0.0.1",
3
+ "version": "0.0.3",
4
4
  "description": "The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: loads a mounted schema-declaration module, dataset selection, Typesense connection and rebuild mode from the environment, then runs the object-grain searchIndexerPipeline once to completion. The composition layer that binds the engine-agnostic @lde/search-pipeline stages to @lde/search-typesense – the write-path counterpart of @lde/search-api-server.",
5
5
  "repository": {
6
6
  "url": "git+https://github.com/ldelements/lde.git",
@@ -29,12 +29,12 @@
29
29
  ],
30
30
  "dependencies": {
31
31
  "@lde/dataset-registry-client": "^0.9.1",
32
- "@lde/pipeline": "^0.35.3",
33
- "@lde/pipeline-console-reporter": "^0.26.3",
32
+ "@lde/pipeline": "^0.35.4",
33
+ "@lde/pipeline-console-reporter": "^0.26.4",
34
34
  "@lde/search": "^0.12.2",
35
- "@lde/search-pipeline": "^0.12.4",
36
- "@lde/search-typesense": "^0.15.4",
37
- "@lde/sparql-qlever": "^0.15.2",
35
+ "@lde/search-pipeline": "^0.12.5",
36
+ "@lde/search-typesense": "^0.15.5",
37
+ "@lde/sparql-qlever": "^0.15.3",
38
38
  "commander": "^15.0.0",
39
39
  "tslib": "^2.3.0",
40
40
  "typesense": "^3.0.6"