@lde/search-indexer 0.0.1

package/README.md

@@ -0,0 +1,133 @@
+# @lde/search-indexer
+
+The [@lde/search-pipeline](../search-pipeline) indexer as a bootable process
+and prebuilt Docker image: mount a schema-declaration module, point it at a
+dataset registry and Typesense, and one run selects the datasets, extracts and
+projects each root type per the schema, and rebuilds the engine collections –
+then exits. Run it as a cron job; the rebuild writers’ cross-pod lock makes an
+overlapping run fail fast instead of corrupting a collection.
+
+This is the write-path counterpart of
+[`@lde/search-api-server`](../search-api-server)
+([#652](https://github.com/ldelements/lde/issues/652)): the composition layer
+that binds the engine-agnostic `searchIndexerPipeline` to the
+[`@lde/search-typesense`](../search-typesense) rebuild writers. Mount the
+**same schema module** into both images and point both at the same
+`TYPESENSE_*` coordinates, and the write and the read side cannot disagree
+about the schema. A deployment that needs more – a bespoke root selector,
+per-stage tuning, another engine – composes `searchStages`,
+`searchIndexWriter` and `Pipeline` directly instead.
+
+## Run
+
+```sh
+docker run --rm \
+  --volume ./search-schema.mjs:/config/search-schema.mjs:ro \
+  --env REGISTRY_ENDPOINT=https://registry.example.org/sparql \
+  --env TYPESENSE_HOST=typesense.internal \
+  --env TYPESENSE_API_KEY=admin-key \
+  ghcr.io/ldelements/search-indexer
+```
+
+Or without Docker (the same environment variables apply):
+
+```sh
+npx @lde/search-indexer
+```
+
+Dataset IRIs can also be passed as arguments, which override `DATASETS`:
+
+```sh
+npx @lde/search-indexer https://example.org/id/dataset/1
+```
+
+`--check` validates the configuration and schema module, then exits without
+indexing – for CI and init containers.
+
+## The schema module
+
+The mounted module default-exports the deployment’s search type declarations
+as **plain data** – see [the API server’s
+README](../search-api-server/README.md#the-schema-module) for the format and
+authoring guidance; both images load the module with the same
+`@lde/search/module` loader. The indexer reads only the default export: the
+declarations drive one pipeline stage and one engine collection per root type,
+and each field’s `path` drives the extraction CONSTRUCT. The read-side extras
+(`schemaOptions`, `engineOptions`) are ignored here, so one file serves both
+images.
+
+## Configuration
+
+| Variable             | Default                     | Meaning                                                                      |
+| -------------------- | --------------------------- | ---------------------------------------------------------------------------- |
+| `SCHEMA_MODULE`      | `/config/search-schema.mjs` | Path of the mounted schema-declaration module                                |
+| `REGISTRY_ENDPOINT`  | **required**                | SPARQL endpoint of the DCAT dataset registry                                 |
+| `DATASETS`           | all registry datasets       | Dataset IRIs to index (whitespace- or comma-separated)                       |
+| `DATASET_CRITERIA`   | all registry datasets       | Registry search criteria as a JSON object (mutually exclusive w/ `DATASETS`) |
+| `TYPESENSE_HOST`     | **required**                | Typesense host                                                               |
+| `TYPESENSE_PORT`     | `8108`                      | Typesense port                                                               |
+| `TYPESENSE_PROTOCOL` | `http`                      | `http` or `https`                                                            |
+| `TYPESENSE_API_KEY`  | **required**                | An admin key: the indexer creates, writes and swaps collections              |
+| `REBUILD_MODE`       | `in-place`                  | `in-place` (update the live collection) or `blue-green` (swap on commit)     |
+| `COLLECTION_PREFIX`  | none                        | Prefix for every derived collection name (configure the read side to match)  |
+| `PROVENANCE_FILE`    | none                        | JSON file remembering per-dataset processing, to skip unchanged datasets     |
+| `PIPELINE_VERSION`   | none                        | Version keying the skip decisions; required with `PROVENANCE_FILE`           |
+| `QLEVER_IMAGE`       | none                        | Enables the QLever import path (see below), e.g. `adfreiburg/qlever:latest`  |
+| `IMPORT_STRATEGY`    | `sparql`                    | `sparql`, `sparqlWithImportFallback` or `import`; requires `QLEVER_IMAGE`    |
+| `DATA_DIR`           | `/data`                     | Directory for downloaded dumps and QLever index caches                       |
+
+A misconfigured boot reports **all** problems in one error, not one per crash
+loop. `PROVENANCE_FILE` must sit on a durable volume, and cannot be combined
+with `REBUILD_MODE=blue-green`: a skipped dataset would be missing from the
+fresh collection the swap makes live.
+
+## The QLever import path
+
+By default the indexer serves only datasets that publish a live SPARQL
+endpoint (the pipeline’s endpoint-only default). Setting `QLEVER_IMAGE`
+enables the import path: data dumps are downloaded to `DATA_DIR` and imported
+into a QLever instance the **pipeline itself creates and controls** – one
+sibling container per dataset, spawned over the Docker socket
+([ADR 16](../../docs/decisions/0016-ship-the-search-indexer-as-a-config-driven-image.md)
+explains why a statically-declared QLever service cannot work). That requires:
+
+- the Docker socket mounted into the indexer container
+  (`--volume /var/run/docker.sock:/var/run/docker.sock`), plus a group grant
+  to reach it as the image’s non-root user
+  (`--group-add $(stat -c %g /var/run/docker.sock)`);
+- `DATA_DIR` on a volume whose **host path is identical** for the indexer and
+  the spawned QLever containers – the pipeline passes `DATA_DIR` as a bind
+  mount to a sibling container, where it resolves against the host.
+
+This mode does not work on container runtimes without a Docker socket
+(containerd-based Kubernetes); there, run endpoint-only for now.
+
+## Building the image
+
+The image is built from the workspace’s own outputs – the compiled package,
+the same-commit builds of its `@lde/*` dependencies and a pruned lockfile –
+never from npm, so it exists for any commit
+([ADR 15](../../docs/decisions/0015-ship-the-served-search-api-as-a-docker-image-built-from-the-workspace.md)):
+
+```sh
+npx nx run @lde/search-indexer:docker:build   # → packages-search-indexer
+npx nx run @lde/search-indexer:docker:smoke   # boots it with --check
+```
+
+CI runs `docker:smoke` for affected PRs; each release tag rebuilds and pushes
+`ghcr.io/ldelements/search-indexer:<version>` (`.github/workflows/docker.yml`).
+
+## Programmatic use
+
+The bin is a thin wrapper over the exported API, usable in tests or a custom
+boot:
+
+```ts
+import {
+  configFromEnvironment,
+  createSearchIndexer,
+} from '@lde/search-indexer';
+
+const pipeline = await createSearchIndexer(configFromEnvironment(process.env));
+await pipeline.run();
+```

package/dist/cli.d.ts

@@ -0,0 +1,3 @@
+#!/usr/bin/env node
+export {};
+//# sourceMappingURL=cli.d.ts.map

package/dist/cli.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":""}

package/dist/cli.js

@@ -0,0 +1,30 @@
+#!/usr/bin/env node
+import { readFileSync } from 'node:fs';
+import { Command } from 'commander';
+import { configFromEnvironment } from './config.js';
+import { createSearchIndexer } from './indexer.js';
+const { version } = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
+const program = new Command()
+    .name('search-indexer')
+    .description('Run the object-grain search indexer: select datasets, extract and project each root type per the mounted schema module, and rebuild the Typesense collections. Configuration comes from environment variables; see the README.')
+    .version(version)
+    .argument('[datasets...]', 'dataset IRIs to index (overrides the DATASETS environment variable)')
+    .option('--check', 'validate the configuration and schema module, then exit without indexing');
+program.parse();
+try {
+    const datasets = program.args;
+    const config = configFromEnvironment(datasets.length > 0
+        ? { ...process.env, DATASETS: datasets.join(' ') }
+        : process.env);
+    const pipeline = await createSearchIndexer(config);
+    if (program.opts().check) {
+        console.info(`@lde/search-indexer ${version}: configuration and schema module “${config.schemaModulePath}” are valid.`);
+    }
+    else {
+        await pipeline.run();
+    }
+}
+catch (error) {
+    console.error(error instanceof Error ? error.message : String(error));
+    process.exit(1);
+}

package/dist/composition.d.ts

@@ -0,0 +1,22 @@
+import { type DatasetSelector, type DistributionResolver, type Writer } from '@lde/pipeline';
+import type { RootType, SearchDocument } from '@lde/search';
+import type { Client } from 'typesense';
+import type { IndexerConfig } from './config.js';
+/** The registry-backed dataset selection the configuration describes. */
+export declare function datasetSelectorFrom(config: IndexerConfig): DatasetSelector;
+/**
+ * The engine-writer factory the configuration describes: an
+ * {@link InPlaceRebuild} or {@link BlueGreenRebuild} per root type, its
+ * collection name derived from the type – prefixed when the deployment says
+ * so, so the read side must be configured with the same prefix.
+ */
+export declare function writerFactoryFrom(client: Client, config: IndexerConfig): (searchType: RootType) => Writer<SearchDocument>;
+/**
+ * The distribution resolver the configuration describes: with `QLEVER_IMAGE`
+ * set, an {@link ImportResolver} that imports data dumps into a
+ * pipeline-controlled QLever sibling container over the mounted Docker
+ * socket; without it, `undefined`, leaving the pipeline its endpoint-only
+ * default (a bare {@link SparqlDistributionResolver}).
+ */
+export declare function distributionResolverFrom(config: IndexerConfig): DistributionResolver | undefined;
+//# sourceMappingURL=composition.d.ts.map

package/dist/composition.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"composition.d.ts","sourceRoot":"","sources":["../src/composition.ts"],"names":[],"mappings":"AAGA,OAAO,EAIL,KAAK,eAAe,EACpB,KAAK,oBAAoB,EACzB,KAAK,MAAM,EACZ,MAAM,eAAe,CAAC;AACvB,OAAO,KAAK,EAAE,QAAQ,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAO5D,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD,yEAAyE;AACzE,wBAAgB,mBAAmB,CAAC,MAAM,EAAE,aAAa,GAAG,eAAe,CAK1E;AAED;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAC/B,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,aAAa,GACpB,CAAC,UAAU,EAAE,QAAQ,KAAK,MAAM,CAAC,cAAc,CAAC,CAWlD;AAED;;;;;;GAMG;AACH,wBAAgB,wBAAwB,CACtC,MAAM,EAAE,aAAa,GACpB,oBAAoB,GAAG,SAAS,CAclC"}

package/dist/composition.js

@@ -0,0 +1,53 @@
+// Internal composition pieces of createSearchIndexer, split out so each can
+// be tested without running a pipeline (Pipeline keeps its wiring private).
+import { Client as RegistryClient } from '@lde/dataset-registry-client';
+import { ImportResolver, RegistrySelector, SparqlDistributionResolver, } from '@lde/pipeline';
+import { BlueGreenRebuild, deriveCollectionName, InPlaceRebuild, } from '@lde/search-typesense';
+import { createQlever } from '@lde/sparql-qlever';
+/** The registry-backed dataset selection the configuration describes. */
+export function datasetSelectorFrom(config) {
+    return new RegistrySelector({
+        registry: new RegistryClient(config.registryEndpoint),
+        criteria: config.datasetCriteria,
+    });
+}
+/**
+ * The engine-writer factory the configuration describes: an
+ * {@link InPlaceRebuild} or {@link BlueGreenRebuild} per root type, its
+ * collection name derived from the type – prefixed when the deployment says
+ * so, so the read side must be configured with the same prefix.
+ */
+export function writerFactoryFrom(client, config) {
+    return (searchType) => {
+        const options = config.collectionPrefix
+            ? {
+                name: `${config.collectionPrefix}${deriveCollectionName(searchType)}`,
+            }
+            : {};
+        return config.rebuildMode === 'blue-green'
+            ? new BlueGreenRebuild(client, searchType, options)
+            : new InPlaceRebuild(client, searchType, options);
+    };
+}
+/**
+ * The distribution resolver the configuration describes: with `QLEVER_IMAGE`
+ * set, an {@link ImportResolver} that imports data dumps into a
+ * pipeline-controlled QLever sibling container over the mounted Docker
+ * socket; without it, `undefined`, leaving the pipeline its endpoint-only
+ * default (a bare {@link SparqlDistributionResolver}).
+ */
+export function distributionResolverFrom(config) {
+    if (!config.qlever) {
+        return undefined;
+    }
+    const { importer, server } = createQlever({
+        mode: 'docker',
+        image: config.qlever.image,
+        dataDir: config.qlever.dataDir,
+    });
+    return new ImportResolver(new SparqlDistributionResolver(), {
+        importer,
+        server,
+        strategy: config.qlever.strategy,
+    });
+}

package/dist/config.d.ts

@@ -0,0 +1,65 @@
+import type { SearchCriteria } from '@lde/dataset-registry-client';
+/**
+ * The indexer’s complete runtime configuration, read from environment
+ * variables ({@link configFromEnvironment}) – the contract of the Docker
+ * image. Everything schema-shaped lives in the mounted schema module instead,
+ * the same file the served-API image mounts.
+ */
+export interface IndexerConfig {
+    /** Path of the mounted schema-declaration module (`SCHEMA_MODULE`). */
+    readonly schemaModulePath: string;
+    /** SPARQL endpoint of the DCAT dataset registry the selection queries
+     *  (`REGISTRY_ENDPOINT`). */
+    readonly registryEndpoint: URL;
+    /** Dataset selection within the registry: explicit IRIs (`DATASETS`),
+     *  search criteria (`DATASET_CRITERIA`), or every dataset when neither is
+     *  set. */
+    readonly datasetCriteria: SearchCriteria;
+    /** Where the Typesense engine writes (`TYPESENSE_*`). */
+    readonly typesense: TypesenseConnection;
+    /** How each type’s collection is rebuilt (`REBUILD_MODE`): update the live
+     *  collection in place, or build a fresh one and swap on commit. */
+    readonly rebuildMode: 'in-place' | 'blue-green';
+    /** Prefix prepended to every derived collection name
+     *  (`COLLECTION_PREFIX`), e.g. for a shared multi-tenant engine. */
+    readonly collectionPrefix?: string;
+    /** Per-dataset processing memory: skip unchanged datasets
+     *  (`PROVENANCE_FILE` + `PIPELINE_VERSION`). */
+    readonly provenance?: ProvenanceConfig;
+    /** The QLever import path (`QLEVER_IMAGE` + `IMPORT_STRATEGY` +
+     *  `DATA_DIR`): import data dumps into a pipeline-controlled QLever
+     *  container instead of relying on live SPARQL endpoints alone. */
+    readonly qlever?: QleverConfig;
+}
+/** Connection details of the Typesense node the indexer writes to. */
+export interface TypesenseConnection {
+    readonly host: string;
+    readonly port: number;
+    readonly protocol: 'http' | 'https';
+    /** An admin key: the indexer creates, writes and swaps collections. */
+    readonly apiKey: string;
+}
+/** File-backed provenance: both halves required, so a skip-enabled pipeline
+ *  can never run without the version that keys its skip decisions. */
+export interface ProvenanceConfig {
+    /** Path of the JSON provenance file; must sit on a durable volume. */
+    readonly path: string;
+    /** Consumer-declared version of the indexer’s output-affecting logic. */
+    readonly pipelineVersion: string;
+}
+/** The pipeline-controlled QLever sibling container (Docker socket mode). */
+export interface QleverConfig {
+    /** QLever Docker image to run, e.g. `adfreiburg/qlever:latest`. */
+    readonly image: string;
+    /** When to import a data dump instead of querying a live endpoint. */
+    readonly strategy: 'sparql' | 'sparqlWithImportFallback' | 'import';
+    /** Directory for downloaded dumps and QLever index caches. */
+    readonly dataDir: string;
+}
+/**
+ * Read the {@link IndexerConfig} from environment variables. Reports **all**
+ * problems in one error – a misconfigured deployment learns everything
+ * wrong with it from a single boot attempt, not one variable per crash loop.
+ */
+export declare function configFromEnvironment(environment: Readonly<Record<string, string | undefined>>): IndexerConfig;
+//# sourceMappingURL=config.d.ts.map

package/dist/config.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,8BAA8B,CAAC;AAEnE;;;;;GAKG;AACH,MAAM,WAAW,aAAa;IAC5B,uEAAuE;IACvE,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;IAClC;iCAC6B;IAC7B,QAAQ,CAAC,gBAAgB,EAAE,GAAG,CAAC;IAC/B;;eAEW;IACX,QAAQ,CAAC,eAAe,EAAE,cAAc,CAAC;IACzC,yDAAyD;IACzD,QAAQ,CAAC,SAAS,EAAE,mBAAmB,CAAC;IACxC;wEACoE;IACpE,QAAQ,CAAC,WAAW,EAAE,UAAU,GAAG,YAAY,CAAC;IAChD;wEACoE;IACpE,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;oDACgD;IAChD,QAAQ,CAAC,UAAU,CAAC,EAAE,gBAAgB,CAAC;IACvC;;uEAEmE;IACnE,QAAQ,CAAC,MAAM,CAAC,EAAE,YAAY,CAAC;CAChC;AAED,sEAAsE;AACtE,MAAM,WAAW,mBAAmB;IAClC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC;IACpC,uEAAuE;IACvE,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC;CACzB;AAED;sEACsE;AACtE,MAAM,WAAW,gBAAgB;IAC/B,sEAAsE;IACtE,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,yEAAyE;IACzE,QAAQ,CAAC,eAAe,EAAE,MAAM,CAAC;CAClC;AAED,6EAA6E;AAC7E,MAAM,WAAW,YAAY;IAC3B,mEAAmE;IACnE,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,sEAAsE;IACtE,QAAQ,CAAC,QAAQ,EAAE,QAAQ,GAAG,0BAA0B,GAAG,QAAQ,CAAC;IACpE,8DAA8D;IAC9D,QAAQ,CAAC,OAAO,EAAE,MAAM,CAAC;CAC1B;AAQD;;;;GAIG;AACH,wBAAgB,qBAAqB,CACnC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,GAAG,SAAS,CAAC,CAAC,GACxD,aAAa,CAoFf"}

package/dist/config.js

@@ -0,0 +1,150 @@
+const IMPORT_STRATEGIES = [
+    'sparql',
+    'sparqlWithImportFallback',
+    'import',
+];
+/**
+ * Read the {@link IndexerConfig} from environment variables. Reports **all**
+ * problems in one error – a misconfigured deployment learns everything
+ * wrong with it from a single boot attempt, not one variable per crash loop.
+ */
+export function configFromEnvironment(environment) {
+    const problems = [];
+    const required = (name) => {
+        const value = environment[name];
+        if (value === undefined || value.length === 0) {
+            problems.push(`${name} is required`);
+            return '';
+        }
+        return value;
+    };
+    const integer = (name, fallback) => {
+        const value = environment[name];
+        if (value === undefined || value.length === 0) {
+            return fallback;
+        }
+        const parsed = Number(value);
+        if (!Number.isInteger(parsed) || parsed < 0) {
+            problems.push(`${name} must be a non-negative integer, got “${value}”`);
+            return fallback;
+        }
+        return parsed;
+    };
+    const url = (name, value) => {
+        try {
+            return new URL(value);
+        }
+        catch {
+            // An empty value is already reported as missing by required().
+            if (value.length > 0) {
+                problems.push(`${name} must be an absolute URL, got “${value}”`);
+            }
+            return new URL('http://invalid.invalid');
+        }
+    };
+    const registryEndpoint = url('REGISTRY_ENDPOINT', required('REGISTRY_ENDPOINT'));
+    const protocol = environment['TYPESENSE_PROTOCOL'] ?? 'http';
+    if (protocol !== 'http' && protocol !== 'https') {
+        problems.push(`TYPESENSE_PROTOCOL must be “http” or “https”, got “${protocol}”`);
+    }
+    const rebuildMode = environment['REBUILD_MODE'] ?? 'in-place';
+    if (rebuildMode !== 'in-place' && rebuildMode !== 'blue-green') {
+        problems.push(`REBUILD_MODE must be “in-place” or “blue-green”, got “${rebuildMode}”`);
+    }
+    const datasetCriteria = criteriaFromEnvironment(environment, problems);
+    const provenance = provenanceFromEnvironment(environment, rebuildMode, problems);
+    const qlever = qleverFromEnvironment(environment, problems);
+    const config = {
+        schemaModulePath: environment['SCHEMA_MODULE'] ?? '/config/search-schema.mjs',
+        registryEndpoint,
+        datasetCriteria,
+        typesense: {
+            host: required('TYPESENSE_HOST'),
+            port: integer('TYPESENSE_PORT', 8108),
+            protocol: protocol,
+            apiKey: required('TYPESENSE_API_KEY'),
+        },
+        rebuildMode: rebuildMode,
+        collectionPrefix: environment['COLLECTION_PREFIX'],
+        provenance,
+        qlever,
+    };
+    if (problems.length > 0) {
+        throw new Error(`Invalid configuration:\n${problems.map((problem) => `- ${problem}`).join('\n')}`);
+    }
+    return config;
+}
+/** `DATASETS` (whitespace-separated IRIs) or `DATASET_CRITERIA` (a JSON
+ *  object) – never both; neither selects the whole registry. */
+function criteriaFromEnvironment(environment, problems) {
+    const datasets = environment['DATASETS'];
+    const criteriaJson = environment['DATASET_CRITERIA'];
+    if (datasets && criteriaJson) {
+        problems.push('DATASETS and DATASET_CRITERIA are mutually exclusive; set at most one');
+        return {};
+    }
+    if (datasets) {
+        const iris = datasets.split(/[\s,]+/).filter((iri) => iri.length > 0);
+        for (const iri of iris) {
+            try {
+                new URL(iri);
+            }
+            catch {
+                problems.push(`DATASETS contains “${iri}”, which is not an IRI`);
+            }
+        }
+        return { $id: iris };
+    }
+    if (criteriaJson) {
+        try {
+            const parsed = JSON.parse(criteriaJson);
+            if (typeof parsed !== 'object' ||
+                parsed === null ||
+                Array.isArray(parsed)) {
+                problems.push('DATASET_CRITERIA must be a JSON object');
+                return {};
+            }
+            return parsed;
+        }
+        catch {
+            problems.push('DATASET_CRITERIA must be valid JSON');
+            return {};
+        }
+    }
+    return {};
+}
+function provenanceFromEnvironment(environment, rebuildMode, problems) {
+    const path = environment['PROVENANCE_FILE'];
+    const pipelineVersion = environment['PIPELINE_VERSION'];
+    if (!path && !pipelineVersion) {
+        return undefined;
+    }
+    if (!path || !pipelineVersion) {
+        problems.push('PROVENANCE_FILE and PIPELINE_VERSION must be set together: the version keys the skip decisions the file remembers');
+        return undefined;
+    }
+    if (rebuildMode === 'blue-green') {
+        problems.push('PROVENANCE_FILE cannot be combined with REBUILD_MODE=blue-green: a skipped dataset would be missing from the fresh collection the swap makes live');
+        return undefined;
+    }
+    return { path, pipelineVersion };
+}
+function qleverFromEnvironment(environment, problems) {
+    const image = environment['QLEVER_IMAGE'];
+    const strategy = environment['IMPORT_STRATEGY'];
+    if (!image) {
+        if (strategy) {
+            problems.push('IMPORT_STRATEGY requires QLEVER_IMAGE: without an import engine there is nothing to import into');
+        }
+        return undefined;
+    }
+    if (strategy !== undefined &&
+        !IMPORT_STRATEGIES.includes(strategy)) {
+        problems.push(`IMPORT_STRATEGY must be one of ${IMPORT_STRATEGIES.map((name) => `“${name}”`).join(', ')}, got “${strategy}”`);
+    }
+    return {
+        image,
+        strategy: (strategy ?? 'sparql'),
+        dataDir: environment['DATA_DIR'] ?? '/data',
+    };
+}

package/dist/index.d.ts

@@ -0,0 +1,4 @@
+export { createSearchIndexer } from './indexer.js';
+export { configFromEnvironment } from './config.js';
+export type { IndexerConfig, ProvenanceConfig, QleverConfig, TypesenseConnection, } from './config.js';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAKA,OAAO,EAAE,mBAAmB,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,qBAAqB,EAAE,MAAM,aAAa,CAAC;AACpD,YAAY,EACV,aAAa,EACb,gBAAgB,EAChB,YAAY,EACZ,mBAAmB,GACpB,MAAM,aAAa,CAAC"}

package/dist/index.js

@@ -0,0 +1,7 @@
+// The bootable search indexer: the composition layer binding the
+// engine-agnostic @lde/search-pipeline indexer to Typesense rebuild writers,
+// a mounted schema-declaration module and environment config. The
+// `search-indexer` bin (src/cli.ts) and the Docker image are thin wrappers
+// over these.
+export { createSearchIndexer } from './indexer.js';
+export { configFromEnvironment } from './config.js';

package/dist/indexer.d.ts

@@ -0,0 +1,13 @@
+import { type Pipeline } from '@lde/pipeline';
+import { type TypedSearchDocument } from '@lde/search-pipeline';
+import type { IndexerConfig } from './config.js';
+/**
+ * Compose the ready-to-run search indexer from an {@link IndexerConfig}: load
+ * and validate the mounted schema module (the same file the served-API image
+ * mounts), select datasets from the registry, bind the Typesense rebuild
+ * writers, and wire the optional QLever import path and provenance store into
+ * `searchIndexerPipeline`. Every misconfiguration – an invalid schema, an
+ * underivable collection name – throws here, at boot, never mid-run.
+ */
+export declare function createSearchIndexer(config: IndexerConfig): Promise<Pipeline<TypedSearchDocument>>;
+//# sourceMappingURL=indexer.d.ts.map

package/dist/indexer.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"indexer.d.ts","sourceRoot":"","sources":["../src/indexer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAuB,KAAK,QAAQ,EAAE,MAAM,eAAe,CAAC;AAGnE,OAAO,EAEL,KAAK,mBAAmB,EACzB,MAAM,sBAAsB,CAAC;AAO9B,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAEjD;;;;;;;GAOG;AACH,wBAAsB,mBAAmB,CACvC,MAAM,EAAE,aAAa,GACpB,OAAO,CAAC,QAAQ,CAAC,mBAAmB,CAAC,CAAC,CAuBxC"}

package/dist/indexer.js

@@ -0,0 +1,38 @@
+import { FileProvenanceStore } from '@lde/pipeline';
+import { ConsoleReporter } from '@lde/pipeline-console-reporter';
+import { loadSchemaModule } from '@lde/search/module';
+import { searchIndexerPipeline, } from '@lde/search-pipeline';
+import { Client } from 'typesense';
+import { datasetSelectorFrom, distributionResolverFrom, writerFactoryFrom, } from './composition.js';
+/**
+ * Compose the ready-to-run search indexer from an {@link IndexerConfig}: load
+ * and validate the mounted schema module (the same file the served-API image
+ * mounts), select datasets from the registry, bind the Typesense rebuild
+ * writers, and wire the optional QLever import path and provenance store into
+ * `searchIndexerPipeline`. Every misconfiguration – an invalid schema, an
+ * underivable collection name – throws here, at boot, never mid-run.
+ */
+export async function createSearchIndexer(config) {
+    const { schema } = await loadSchemaModule(config.schemaModulePath);
+    const client = new Client({
+        nodes: [
+            {
+                host: config.typesense.host,
+                port: config.typesense.port,
+                protocol: config.typesense.protocol,
+            },
+        ],
+        apiKey: config.typesense.apiKey,
+    });
+    return searchIndexerPipeline({
+        schema,
+        datasets: datasetSelectorFrom(config),
+        distributionResolver: distributionResolverFrom(config),
+        writerFor: writerFactoryFrom(client, config),
+        provenanceStore: config.provenance
+            ? new FileProvenanceStore({ path: config.provenance.path })
+            : undefined,
+        pipelineVersion: config.provenance?.pipelineVersion,
+        reporter: new ConsoleReporter(),
+    });
+}

package/package.json

@@ -0,0 +1,65 @@
+{
+  "name": "@lde/search-indexer",
+  "version": "0.0.1",
+  "description": "The @lde/search-pipeline indexer as a bootable process and prebuilt Docker image: loads a mounted schema-declaration module, dataset selection, Typesense connection and rebuild mode from the environment, then runs the object-grain searchIndexerPipeline once to completion. The composition layer that binds the engine-agnostic @lde/search-pipeline stages to @lde/search-typesense – the write-path counterpart of @lde/search-api-server.",
+  "repository": {
+    "url": "git+https://github.com/ldelements/lde.git",
+    "directory": "packages/search-indexer"
+  },
+  "license": "MIT",
+  "type": "module",
+  "exports": {
+    "./package.json": "./package.json",
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js",
+      "development": "./src/index.ts",
+      "default": "./dist/index.js"
+    }
+  },
+  "main": "./dist/index.js",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "bin": {
+    "search-indexer": "dist/cli.js"
+  },
+  "files": [
+    "dist",
+    "!**/*.tsbuildinfo"
+  ],
+  "dependencies": {
+    "@lde/dataset-registry-client": "^0.9.1",
+    "@lde/pipeline": "^0.35.3",
+    "@lde/pipeline-console-reporter": "^0.26.3",
+    "@lde/search": "^0.12.2",
+    "@lde/search-pipeline": "^0.12.4",
+    "@lde/search-typesense": "^0.15.4",
+    "@lde/sparql-qlever": "^0.15.2",
+    "commander": "^15.0.0",
+    "tslib": "^2.3.0",
+    "typesense": "^3.0.6"
+  },
+  "nx": {
+    "targets": {
+      "docker:stage": {
+        "options": {
+          "cwd": "{projectRoot}",
+          "command": "rm -rf dist-docker && mkdir dist-docker && cp package.json dist-docker/ && cp -R dist dist-docker/dist"
+        }
+      },
+      "copy-workspace-modules": {},
+      "prune-lockfile": {},
+      "docker:lockfile": {},
+      "docker:smoke": {
+        "executor": "nx:run-commands",
+        "dependsOn": [
+          "docker:build"
+        ],
+        "cache": false,
+        "options": {
+          "command": "./tools/docker-smoke-indexer.sh packages-search-indexer"
+        }
+      }
+    }
+  }
+}