@lde/search-indexer 0.0.3 → 0.1.0

package/README.md

@@ -7,18 +7,11 @@ projects each root type per the schema, and rebuilds the engine collections –
 then exits. Run it as a cron job; the rebuild writers’ cross-pod lock makes an
 overlapping run fail fast instead of corrupting a collection.
 
-This is the write-path counterpart of
-[`@lde/search-api-server`](../search-api-server)
-([#652](https://github.com/ldelements/lde/issues/652)): the composition layer
-that binds the engine-agnostic `searchIndexerPipeline` to the
-[`@lde/search-typesense`](../search-typesense) rebuild writers. Mount the
-**same schema module** into both images and point both at the same
-`TYPESENSE_*` coordinates, and the write and the read side cannot disagree
-about the schema. A deployment that needs more – a bespoke root selector,
-per-stage tuning, another engine – composes `searchStages`,
-`searchIndexWriter` and `Pipeline` directly instead.
+## Installation
 
-## Run
+```sh
+npm install @lde/search-indexer
+```
 
 ```sh
 docker run --rm \
@@ -29,131 +22,6 @@ docker run --rm \
   ghcr.io/ldelements/search-indexer
 ```
 
-Or without Docker (the same environment variables apply):
-
-```sh
-npx @lde/search-indexer
-```
-
-Dataset IRIs can also be passed as arguments, which override `DATASETS`:
-
-```sh
-npx @lde/search-indexer https://example.org/id/dataset/1
-```
-
-`--check` validates the configuration and schema module, then exits without
-indexing – for CI and init containers.
-
-## The schema module
-
-The mounted module default-exports the deployment’s search type declarations
-as **plain data** – see [the API server’s
-README](../search-api-server/README.md#the-schema-module) for the format and
-authoring guidance; both images load the module with the same
-`@lde/search/module` loader. The indexer reads only the default export: the
-declarations drive one pipeline stage and one engine collection per root type,
-and each field’s `path` drives the extraction CONSTRUCT. The read-side extras
-(`schemaOptions`, `engineOptions`) are ignored here, so one file serves both
-images.
-
-## Configuration
-
-| Variable             | Default                     | Meaning                                                                                             |
-| -------------------- | --------------------------- | --------------------------------------------------------------------------------------------------- |
-| `SCHEMA_MODULE`      | `/config/search-schema.mjs` | Path of the mounted schema-declaration module                                                       |
-| `REGISTRY_ENDPOINT`  | **required**                | SPARQL endpoint of the DCAT dataset registry                                                        |
-| `DATASETS`           | all registry datasets       | Dataset IRIs to index (whitespace- or comma-separated)                                              |
-| `DATASET_CRITERIA`   | all registry datasets       | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`)                        |
-| `TYPESENSE_HOST`     | **required**                | Typesense host                                                                                      |
-| `TYPESENSE_PORT`     | `8108`                      | Typesense port                                                                                      |
-| `TYPESENSE_PROTOCOL` | `http`                      | `http` or `https`                                                                                   |
-| `TYPESENSE_API_KEY`  | **required**                | An admin key: the indexer creates, writes and swaps collections                                     |
-| `REBUILD_MODE`       | `in-place`                  | `in-place` (update the live collection) or `blue-green` (swap on commit)                            |
-| `COLLECTION_PREFIX`  | none                        | Prefix for every derived collection name (configure the read side to match)                         |
-| `PROVENANCE_FILE`    | none                        | JSON file remembering per-dataset processing, to skip unchanged datasets                            |
-| `PIPELINE_VERSION`   | none                        | Version keying the skip decisions; required with `PROVENANCE_FILE`                                  |
-| `QLEVER_IMAGE`       | none                        | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest`                         |
-| `IMPORT_STRATEGY`    | `sparql`                    | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE`                           |
-| `DATA_DIR`           | `/data`                     | Directory for downloaded dumps and QLever index caches                                              |
-| `QLEVER_NETWORK`     | none                        | Docker network the spawned QLever joins; set when the indexer itself runs containerized (see below) |
-
-A misconfigured boot reports **all** problems in one error, not one per crash
-loop. `PROVENANCE_FILE` must sit on a durable volume, and cannot be combined
-with `REBUILD_MODE=blue-green`: a skipped dataset would be missing from the
-fresh collection the swap makes live. Its directory must be writable by the
-image’s runtime user (uid 1000) – the image pre-creates `/provenance` with
-that ownership, so a named volume mounted there just works, while a bind
-mount must be `chown`ed on the host. A run fails at start when the file is
-not writable, instead of silently never skipping.
-
-## The QLever import path
-
-By default the indexer serves only datasets that publish a live SPARQL
-endpoint (the pipeline’s endpoint-only default). Setting `QLEVER_IMAGE`
-enables the import path: data dumps are downloaded to `DATA_DIR` and imported
-into a QLever instance the **pipeline itself creates and controls** – one
-sibling container per dataset, spawned over the Docker socket
-([ADR 16](../../docs/decisions/0016-ship-the-search-indexer-as-a-config-driven-image.md)
-explains why a statically-declared QLever service cannot work). That requires:
-
-- the Docker socket mounted into the indexer container
-  (`--volume /var/run/docker.sock:/var/run/docker.sock`), plus a group grant
-  to reach it as the image’s non-root user
-  (`--group-add $(stat -c %g /var/run/docker.sock)`);
-- `DATA_DIR` on a volume whose **host path is identical** for the indexer and
-  the spawned QLever containers – the pipeline passes `DATA_DIR` as a bind
-  mount to a sibling container, where it resolves against the host.
-
-How the indexer reaches the QLever it spawned depends on where the indexer
-itself runs:
-
-- **On the host** (`npx @lde/search-indexer` with Docker available): leave
-  `QLEVER_NETWORK` unset. QLever’s port is published on the host and
-  addressed as `localhost`.
-- **In a container on a bridge network** (the `docker compose` default): set
-  `QLEVER_NETWORK` to a network the indexer is attached to. QLever joins that
-  network and is addressed by container name (`qlever-<network>`) instead – a
-  containerized indexer’s `localhost` is its own network namespace, so it
-  cannot reach a host-published port. Alternatively, run the indexer with
-  `network_mode: host` and leave `QLEVER_NETWORK` unset.
-
-`QLEVER_NETWORK` must name a network the indexer container is actually
-attached to – for compose, the prefixed name Docker creates (e.g.
-`myproject_default`), not the short service-file name. A wrong or unattached
-network is not caught at boot: each dataset imports fully, then fails when
-the endpoint never becomes reachable. And run at most one indexer per
-network – the QLever container name is derived from the network, so two
-indexers sharing one would remove each other’s QLever.
-
-This mode does not work on container runtimes without a Docker socket
-(containerd-based Kubernetes); there, run endpoint-only for now.
-
-## Building the image
-
-The image is built from the workspace’s own outputs – the compiled package,
-the same-commit builds of its `@lde/*` dependencies and a pruned lockfile –
-never from npm, so it exists for any commit
-([ADR 15](../../docs/decisions/0015-ship-the-served-search-api-as-a-docker-image-built-from-the-workspace.md)):
-
-```sh
-npx nx run @lde/search-indexer:docker:build   # → packages-search-indexer
-npx nx run @lde/search-indexer:docker:smoke   # boots it with --check
-```
-
-CI runs `docker:smoke` for affected PRs; each release rebuilds and pushes
-`ghcr.io/ldelements/search-indexer:<version>` (`.github/workflows/docker.yml`).
+## Documentation
 
-## Programmatic use
-
-The bin is a thin wrapper over the exported API, usable in tests or a custom
-boot:
-
-```ts
-import {
-  configFromEnvironment,
-  createSearchIndexer,
-} from '@lde/search-indexer';
-
-const pipeline = await createSearchIndexer(configFromEnvironment(process.env));
-await pipeline.run();
-```
+See the [full documentation](https://ldelements.org/reference/search-indexer).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lde/search-indexer",
-  "version": "0.0.3",
+  "version": "0.1.0",
   "description": "The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: loads a mounted schema-declaration module, dataset selection, Typesense connection and rebuild mode from the environment, then runs the object-grain searchIndexerPipeline once to completion. The composition layer that binds the engine-agnostic @lde/search-pipeline stages to @lde/search-typesense – the write-path counterpart of @lde/search-api-server.",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
@@ -31,9 +31,9 @@
     "@lde/dataset-registry-client": "^0.9.1",
     "@lde/pipeline": "^0.35.4",
     "@lde/pipeline-console-reporter": "^0.26.4",
-    "@lde/search": "^0.12.2",
-    "@lde/search-pipeline": "^0.12.5",
-    "@lde/search-typesense": "^0.15.5",
+    "@lde/search": "^0.13.0",
+    "@lde/search-pipeline": "^0.13.0",
+    "@lde/search-typesense": "^0.16.0",
     "@lde/sparql-qlever": "^0.15.3",
     "commander": "^15.0.0",
     "tslib": "^2.3.0",