@lde/pipeline 0.35.3 → 0.35.5

package/README.md

@@ -1,453 +1,13 @@
-# Pipeline
+# @lde/pipeline
 
-A framework for transforming large RDF datasets, primarily using [SPARQL](https://www.w3.org/TR/sparql11-query/) queries with TypeScript for the parts that are hard to express in SPARQL alone.
+A framework for transforming large RDF datasets, primarily using [SPARQL](https://www.w3.org/TR/sparql11-query/) queries with TypeScript for the parts that are hard to express in SPARQL alone. Transformations are plain SPARQL query files – portable, transparent, testable and version-controlled – and memory stays bounded no matter how large the dataset.
 
-- **SPARQL-native.** Data transformations are plain SPARQL query files – portable, transparent, testable and version-controlled.
-- **Composable.** Executors are an interface: wrap a SPARQL reader with custom TypeScript to handle edge cases like date parsing or string normalisation (see [Reader](#reader)).
-- **Extensible.** A plugin system lets packages like [@lde/pipeline-void](../pipeline-void) (or your own plugins) hook into the pipeline lifecycle.
+## Installation
 
-## Components
-
-A **Pipeline** consists of:
-
-- a **Dataset Selector** that selects which datasets to process
-- a **Distribution Resolver** that resolves each dataset to a usable SPARQL endpoint
-- one or more **Stages**, each consisting of:
-  - an optional **Item Selector** that selects resources (as variable bindings) for fan-out
-  - one or more **Executors** that generate triples
-
-### Dataset Selector
-
-Selects datasets, either manually or by querying a DCAT Dataset Registry:
-
-```typescript
-// From a registry
-const selector = new RegistrySelector({
-  registry: new Client(new URL('https://example.com/sparql')),
-});
-
-// Manual
-const selector = new ManualDatasetSelection([dataset]);
-```
-
-### Distribution Resolver
-
-Resolves each dataset to a usable SPARQL endpoint. `SparqlDistributionResolver` probes a dataset’s own endpoint; wrap it in `ImportResolver` to add the ability to import a data dump into a local SPARQL server.
-
-`ImportResolver`’s `strategy` controls how the source is chosen, ordered by how eagerly a dump is imported:
-
-```typescript
-const resolver = new ImportResolver(new SparqlDistributionResolver(), {
-  importer,
-  server,
-  strategy: 'sparqlWithImportFallback',
-});
-```
-
-- `'sparql'` (default) – use the dataset’s own SPARQL endpoint when one is available; import a data dump only when no endpoint responds.
-- `'sparqlWithImportFallback'` – like `'sparql'`, but also fall back to the data dump when the endpoint passes probing yet a stage fails against it at runtime. The pipeline discards the endpoint-sourced partial output and re-runs all stages against the import. Use this when endpoints are present but unreliable for heavy aggregate queries.
-- `'import'` – always import the data dump, even when a working endpoint is advertised.
-
-### Stage
-
-A stage groups an item selector, one or more readers, and configuration:
-
-```typescript
-new Stage({
-  name: 'per-class',
-  itemSelector: new SparqlItemSelector({
-    query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
-  }),
-  readers: reader,
-  batchSize: 100,
-  maxConcurrency: 5,
-});
-```
-
-#### Batch size
-
-`batchSize` (default: 10) controls how many variable bindings are passed to each reader call as a `VALUES` clause. It also sets the page size for the item selector's SPARQL requests, so that each paginated request fills exactly one reader batch.
-
-Some SPARQL endpoints enforce different result limits for SELECT and CONSTRUCT queries. Since the selector uses SELECT and the reader uses CONSTRUCT, a `LIMIT` clause in the selector query overrides `batchSize` as the page size. Use this when the endpoint caps SELECT results below your desired batch size:
-
-```typescript
-// Endpoint caps SELECT results at 500, but each CONSTRUCT can handle 1000 bindings.
-new Stage({
-  name: 'per-class',
-  itemSelector: new SparqlItemSelector({
-    query: 'SELECT DISTINCT ?class WHERE { ?s a ?class } LIMIT 500',
-  }),
-  readers: reader,
-  batchSize: 1000, // Two SELECT pages fill one CONSTRUCT batch.
-});
-```
-
-#### Concurrency
-
-`maxConcurrency` (default: 10) limits the total number of concurrent SPARQL queries. Within each batch, all readers run in parallel; the number of concurrent batches is automatically reduced to `⌊maxConcurrency / executorCount⌋` so the total query pressure stays within the limit. For example, with `maxConcurrency: 10` and two readers per stage, up to 5 batches run concurrently (10 SPARQL queries total).
-
-#### Expecting output
-
-`expectsOutput` (default: `false`) marks a stage whose query must yield at least one quad. A supported stage that produces none is then treated as a hard failure rather than a legitimately empty result.
-
-Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always return exactly one row – so zero output can only mean the endpoint truncated or aborted the response (e.g. a timeout surfaced as an empty `HTTP 200`). The failure flows through like any other hard stage failure, triggering the [reactive dump fallback](#distribution-resolver) when `strategy: 'sparqlWithImportFallback'` is configured. Leave it `false` for stages that may legitimately be empty, such as class or property partitions of a dataset that lacks that structure.
-
-### Item Selector
-
-Selects resources from the distribution and fans out reader calls per batch of results. Implements the `ItemSelector` interface:
-
-```typescript
-interface ItemSelector {
-  select(
-    distribution: Distribution,
-    batchSize?: number,
-  ): AsyncIterable<VariableBindings>;
-}
-```
-
-The distribution is received at run time, so selectors don't need the endpoint URL at construction time. The `batchSize` parameter is set by the stage. Use `SparqlItemSelector` for SPARQL-based selection with automatic pagination:
-
-```typescript
-new SparqlItemSelector({
-  query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
-});
-```
-
-A row is yielded only when **every** projected variable binds a `NamedNode` – binding values double as stable item identities downstream, which a blank-node label or literal cannot provide. Other rows are dropped client-side but still count toward pagination, so they are fetched first; to skip them at the endpoint, filter in the query itself (e.g. `FILTER(isIRI(?s))`, as `selectByClass` in `@lde/search-pipeline` does with `FILTER(!isBlank(?root))`).
-
-#### Capping total results with `maxResults`
-
-By default, `SparqlItemSelector` paginates through **all** matching rows: any `LIMIT` clause in the query is interpreted as the page size, then it walks pages with `OFFSET` until the source is exhausted. To cap the total bindings yielded across all pages – for sampling, testing, prototyping, or just safety – set `maxResults`:
-
-```typescript
-new SparqlItemSelector({
-  query: 'SELECT DISTINCT ?s WHERE { ?s a <http://example.com/Class> }',
-  maxResults: 50,
-});
-```
-
-When `maxResults` is set:
-
-- Pagination stops as soon as `maxResults` bindings have been yielded – no wasted page request after the cap is hit.
-- As long as no row has been dropped, the last (partial) page's `LIMIT` is shrunk to the remaining cap so the endpoint doesn't over-fetch on the remainder (e.g. with `maxResults: 85` and `pageSize: 10`, the 9th page request is `LIMIT 5`, not `LIMIT 10`). Once rows have been dropped, pages stay at full size – shrinking to the yielded remainder would crawl a dropped-row region one row per request.
-- The first page uses the configured page size as-is; `maxResults` and page size stay orthogonal. If `maxResults < pageSize`, the first page may return a few rows that aren't yielded.
-- `maxResults: 0` is a valid no-op; the selector yields nothing without issuing any SPARQL request.
-- `maxResults` is independent of any `LIMIT` clause in the query, which still controls page size when the cap is larger than one page.
-
-For dynamic queries that depend on the distribution, implement `ItemSelector` directly:
-
-```typescript
-const itemSelector: ItemSelector = {
-  select: (distribution, batchSize) => {
-    const query = buildQuery(distribution);
-    return new SparqlItemSelector({ query }).select(distribution, batchSize);
-  },
-};
-```
-
-### Reader
-
-Generates RDF triples. The built-in `SparqlConstructReader` runs a SPARQL CONSTRUCT query with template substitution and variable bindings:
-
-```typescript
-const reader = new SparqlConstructReader({
-  query: 'CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }',
-});
-```
-
-When querying endpoints that return line-oriented formats like N-Triples (e.g. QLever), enable `lineBuffer` to work around an [N3.js chunk-splitting bug](https://github.com/rdfjs/N3.js/issues/578) that causes intermittent parse errors on large responses:
-
-```typescript
-const reader = new SparqlConstructReader({
-  query: 'CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }',
-  lineBuffer: true,
-});
-```
-
-SPARQL CONSTRUCT queries can produce duplicate triples – for example, constant triples (like `?dataset a edm:ProvidedCHO`) are emitted for every solution row. Enable `deduplicate` to remove duplicates inline on the stream using a string-based identity set (inspired by [Comunica's `distinctConstruct`](https://comunica.dev/docs/query/advanced/context/#14--distinct-construct)):
-
-```typescript
-const reader = new SparqlConstructReader({
-  query: 'CONSTRUCT { ?s a edm:ProvidedCHO . ?s ?p ?o } WHERE { ?s ?p ?o }',
-  deduplicate: true,
-});
-```
-
-The dedup set is scoped to each `read()` call, so memory stays bounded to the number of unique quads per batch. A standalone `deduplicateQuads()` function is also exported for use outside the reader.
-
-### Extending a stage with a quad transform
-
-Some logic is hard to express in pure SPARQL – cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling a reader’s output and firing follow-up queries. Rather than subclass `Reader`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one reader’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
-
-A transform receives an `ReaderContext` – the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per reader call, so **write it to accept being called more than once**: a global stage calls it once over the reader’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations – or keep the transform per-quad, where the number of calls makes no difference.
-
-```typescript
-import { DataFactory } from 'n3';
-import {
-  Stage,
-  SparqlConstructReader,
-  type QuadTransform,
-  type ReaderContext,
-} from '@lde/pipeline';
-
-const cleanDates: QuadTransform<ReaderContext> = async function* (quads) {
-  for await (const quad of quads) {
-    if (quad.object.termType === 'Literal' && isMessyDate(quad.object)) {
-      yield DataFactory.quad(
-        quad.subject,
-        quad.predicate,
-        DataFactory.literal(
-          parseDutchDate(quad.object.value),
-          DataFactory.namedNode('http://www.w3.org/2001/XMLSchema#date'),
-        ),
-      );
-    } else {
-      yield quad;
-    }
-  }
-};
-
-new Stage({
-  name: 'dates',
-  readers: {
-    reader: await SparqlConstructReader.fromFile('dates.rq'),
-    transform: cleanDates,
-  },
-});
-```
-
-`transform` accepts a single transform or an array applied in order, so a stage can compose several. This keeps SPARQL doing the heavy lifting while TypeScript handles the edge cases. See [@lde/pipeline-void](../pipeline-void)'s `withVocabularies` for a real-world example of this pattern.
-
-#### Adaptive timeouts
-
-By default, every SPARQL request uses the same 5-minute budget. When a pipeline runs against many third-party endpoints, that fixed budget can cost ~80 minutes on a single dataset whose endpoint times out repeatedly on heavy queries – light stages on the same endpoint then sit behind the heavy ones that will never succeed.
-
-A `TimeoutPolicy` decides the budget for each SPARQL request and observes the outcome. Two are built in:
-
-- **`ConstantTimeoutPolicy(timeoutMs)`** – returns the same budget for every request. The implicit default when `PipelineOptions.timeout` is omitted (`constantTimeoutPolicy(300_000)`).
-- **`AdaptiveTimeoutPolicy({ defaultMs, tightenedMs, tightenAfterTimeouts })`** – per-endpoint state machine. Each endpoint is either _healthy_ (use `defaultMs`) or _tightened_ (use `tightenedMs`). After `tightenAfterTimeouts` consecutive `timeout` outcomes the endpoint flips to _tightened_; a single `ok` flips it back to _healthy_.
-
-`PipelineOptions.timeout` accepts a `() => TimeoutPolicy` factory. The pipeline invokes it once per dataset, so policy state resets between datasets and one bad dataset can’t poison the next:
-
-```typescript
-import { adaptiveTimeoutPolicy } from '@lde/pipeline';
-
-new Pipeline({
-  // …
-  timeout: adaptiveTimeoutPolicy({
-    defaultMs: 300_000, //         5 min while the endpoint is healthy
-    tightenedMs: 10_000, //        10 s once the endpoint is tightened
-    tightenAfterTimeouts: 2, //    flip to tightened after 2 consecutive timeouts
-  }),
-});
-```
-
-Outcomes are classified as:
-
-| outcome   | source                                                                   |
-| --------- | ------------------------------------------------------------------------ |
-| `ok`      | the request resolved                                                     |
-| `timeout` | client-side `AbortSignal.timeout()` fired, or upstream returned HTTP 504 |
-| `error`   | anything else (other HTTP errors, parse errors, …) – neutral             |
-
-Transitions are forwarded to the `ProgressReporter` via `timeoutTightened` / `timeoutRelaxed`; `ConsoleReporter` prints them as `↘ Tightened` / `↗ Relaxed` lines so operators can tell a fast-failed stage from an unexpected speedup.
-
-Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets).
-
-Timeouts live at the pipeline level – neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
-
-### Validation
-
-Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set – a self-contained cluster of linked resources – that can be meaningfully matched against SHACL shapes. Even with a single reader, each batch is a complete unit; with multiple readers, shapes that reference triples from different readers are validated correctly.
-
-Validating individual quads would be meaningless, since a single quad carries no structural context for shape matching. Validating the full pipeline output would also be problematic: because the pipeline streams results in batches, it doesn’t know where resource cluster boundaries fall. Batching the output could split a valid cluster across two batches, causing partial resources to fail validation even though the complete cluster is valid.
-
-Quads are buffered, validated, and then written or discarded based on the `onInvalid` policy. When no validator is configured, quads stream directly with zero overhead.
-
-```typescript
-import { ShaclValidator } from '@lde/pipeline-shacl-validator';
-import { FileWriter } from '@lde/pipeline';
-
-new Stage({
-  name: 'transform',
-  readers: await SparqlConstructReader.fromFile('transform.rq'),
-  validation: {
-    validator: new ShaclValidator({
-      shapesFile: './shapes.ttl',
-      reportWriters: [
-        new FileWriter({ outputDir: './validation', format: 'turtle' }),
-      ],
-    }),
-    onInvalid: 'write', // 'write' (default) | 'skip' | 'halt'
-  },
-});
-```
-
-| `onInvalid` | Behaviour                                          |
-| ----------- | -------------------------------------------------- |
-| `'write'`   | Write quads even if validation fails **(default)** |
-| `'skip'`    | Discard the batch silently                         |
-| `'halt'`    | Throw an error, stopping the pipeline              |
-
-`Validator` is an interface, so you can implement your own validation strategy. See [@lde/pipeline-shacl-validator](../pipeline-shacl-validator) for the SHACL implementation.
-
-#### Per-dataset reporting
-
-After all stages for a dataset have run, the pipeline calls `validator.report(dataset)` once for each distinct validator attached to any stage and emits a `datasetValidated(dataset, report)` event on the reporter. The call happens **regardless of whether any stage actually invoked `validate()`** – for SHACL that means a dataset whose stages produced no input typically reports `quadsValidated: 0` and `conforms: true` (the SHACL vacuous-truth default). Consumers that want to distinguish ‘not tested’ from ‘tested and passed’ can read `quadsValidated`.
-
-### Writer
-
-Writes generated quads to a destination. A `Writer` is transactional: each pipeline run opens one run on it (`openRun(context)`), writes every dataset through the resulting `RunWriter`, and ends with exactly one `commit()` (on success) or `abort(error)` (on failure). The run lifecycle is the home of destination-level concerns such as atomic swaps, deletion sweeps and cross-pod locks; the pipeline drives `openRun → write* → commit/abort` uniformly and never branches on the writer’s update mode. After a dataset’s stages complete, its `flush(dataset, outcome)` says whether the dataset succeeded, so a writer can gate destructive finalization (e.g. an In-place stale-document sweep) on `'success'`.
-
-- `SparqlUpdateWriter` – writes to a SPARQL endpoint via UPDATE queries; writes are visible as they land, so `commit`/`abort` are no-ops
-- `FileWriter` – writes to local files; each file is streamed to a sibling temp file and atomically renamed on flush, `commit` finalizes any files still open, and `abort` discards temp output. In `n-quads` format it skolemises blank nodes to dataset-scoped IRIs, because such files are typically `cat`-indexed into one served graph where document-scoped blank-node labels from different datasets would otherwise fuse (see `blankNodes`/`failOnBlankNodes` and ldelements/lde#474); Turtle/N-Triples output is left unchanged
-
-The `RunContext` handed to `openRun` carries the run’s identity (`runId`, `startedAt`), the full selection (`selectedSources()`, complete by commit time – including datasets skipped as unchanged) and the pipeline’s provenance store, when configured.
-
-A destination without run-level state implements the same contract with no-op `commit`/`abort` – `SparqlUpdateWriter` shows the pattern:
-
-```typescript
-async openRun(): Promise<RunWriter> {
-  return {
-    write: async (dataset, quads) => {
-      // send quads somewhere
-    },
-    commit: () => Promise.resolve(),
-    abort: () => Promise.resolve(),
-  };
-}
-```
-
-### Reporter
-
-A `ProgressReporter` observes the run, receiving lifecycle events such as `pipelineStart`, `stageComplete`, `datasetValidated` and `pipelineComplete`. Every method is optional, so a reporter implements only the events it cares about.
-
-Pass a single reporter, or an array to have several observe the same run – for example a console reporter alongside one that collects validation verdicts:
-
-```typescript
-new Pipeline({
-  // …
-  reporter: [new ConsoleReporter(), verdictCollector],
-});
-```
-
-Each reporter receives every event, in array order; a reporter that does not implement a given event is skipped for it.
-
-### Provenance store
-
-A `ProvenanceStore` gives the pipeline a small per-dataset memory, so a future run can skip datasets that are genuinely unchanged. It is purely a storage seam: the framework owns the skip decision (see [`sourceFingerprint`](#source-change-fingerprint) and `shouldReprocess`), the store owns only how each record is persisted.
-
-```typescript
-interface ProvenanceStore {
-  get(datasetUri: URL): Promise<ProcessingRecord | null>;
-  set(datasetUri: URL, record: ProcessingRecord): Promise<void>;
-}
-```
-
-A `ProcessingRecord` holds the two opaque change fields – `sourceFingerprint` (derived automatically from source metadata) and `pipelineVersion` (consumer-declared) – plus `generatedAt` and a `status` of `'success'` or `'failed'`. The two change fields are compared only for equality, never parsed or ordered.
-
-#### `FileLoadedSparqlProvenanceStore`
-
-The reference implementation targets a triplestore that is served read-only and rebuilt by bulk-loading files (e.g. [QLever](https://github.com/ad-freiburg/qlever)). It reads through SPARQL queries against the live endpoint, and writes records as files for the next bulk-load – because the endpoint accepts no SPARQL UPDATE.
-
-```typescript
-import { FileLoadedSparqlProvenanceStore } from '@lde/pipeline';
-
-const store = new FileLoadedSparqlProvenanceStore({
-  queryEndpoint: new URL('http://localhost:7001/sparql'),
-  pipelineIri: new URL('https://example.org/pipelines/dkg'),
-  outputDir: './provenance',
-});
+```sh
+npm install @lde/pipeline
 ```
 
-- `get` runs a named-graph-scoped SPARQL `SELECT` against `queryEndpoint`, reading the records a previous run loaded.
-- `set` writes one flat [PROV-O](https://www.w3.org/TR/prov-o/) N-Quads file per dataset into `outputDir`, in the pipeline-scoped named graph, to be bulk-loaded after the run.
-
-Each record is stored as flat PROV-O on the dataset entity – `prov:generatedAtTime` plus `sourceFingerprint`, `pipelineVersion` and `status` under the `https://w3id.org/lde/provenance#` namespace. Scoping every record by `pipelineIri` (used as the named graph) lets multiple pipelines share one triplestore without colliding.
-
-#### `FileProvenanceStore`
-
-For pipelines that run without a triplestore, `FileProvenanceStore` persists all records to a single JSON file, keyed by dataset URI:
-
-```typescript
-import { FileProvenanceStore } from '@lde/pipeline';
-
-const store = new FileProvenanceStore({ path: './state/provenance.json' });
-```
-
-Writes are atomic (temp file + rename), so a run killed mid-write cannot corrupt the next run’s skip decisions. Safe for a single writer: concurrent pipeline processes writing the same file lose each other’s updates – use the SPARQL-backed store (or file locking) instead. The file must sit on a durable volume to survive across runs, the same requirement the SPARQL-backed store already has.
-
-#### Enabling skipping
-
-Skipping is opt-in. Pass a `provenanceStore` and a `pipelineVersion` to the `Pipeline`:
-
-```typescript
-new Pipeline({
-  // …
-  provenanceStore: store,
-  pipelineVersion: 'v3', // rotate only on releases that change output
-});
-```
-
-For each dataset the pipeline probes its distributions, derives the source-change fingerprint, reads the stored record, and **skips before importing** when both change fields match:
-
-```
-skip iff  recorded.sourceFingerprint === current.sourceFingerprint
-     AND  recorded.pipelineVersion   === current.pipelineVersion
-```
-
-Otherwise it imports (if needed), runs the stages, and writes an updated record. `pipelineVersion` is consumer-owned and opaque: rotate it only on releases that change output, and every dataset reprocesses on the next run. It is **required** when a `provenanceStore` is configured (a skip-enabled pipeline with no version would silently freeze); when no store is configured, every dataset is reprocessed – today’s behaviour. A dataset that failed but whose source is unchanged is recorded as `'failed'` and skipped on later runs until its source changes or the version rotates, so a deterministically failing import is not retried every run.
-
-### Source-change fingerprint
-
-`sourceFingerprint(distribution, probeResult)` derives a cheap, opaque change signal for a distribution from metadata the probe already collected – no body download. For a data dump it combines the most recent of the register’s `dct:modified` and the artifact’s HTTP `Last-Modified` with the byte size (the probe’s `Content-Length`, falling back to the declared `dcat:byteSize`). It returns `null` for a live SPARQL endpoint, or when no date and no size can be established – a `null` fingerprint never compares equal, so such a distribution is always reprocessed.
-
-### Plugins
-
-Plugins hook into the pipeline lifecycle via the `PipelinePlugin` interface. Register them in the `plugins` array when constructing a `Pipeline`. A plugin can implement either or both of two hooks, each a `QuadTransform`:
-
-- **`beforeStageWrite`** runs over each stage’s output as it is written – scoped to one stage’s quads for one dataset.
-- **`beforeDatasetWrite`** runs once over a dataset’s combined cross-stage output, just before it is written – scoped to the whole dataset, so it can reconcile quads that different stages produced. The output is streamed through a single queue rather than materialized, so memory stays bounded by whatever the transform itself buffers; it honors the writer’s run lifecycle (`flush`/`reset`/`commit`/`abort`), and a failure is isolated to the dataset being written.
-
-Both hooks are domain-agnostic – dedup, roll-ups, provenance stamping, or namespace rewriting can use them. This package ships `provenancePlugin()`, `namespaceNormalizationPlugin()`, and `schemaOrgNormalizationPlugin()` (all `beforeStageWrite` plugins). Plugins that must understand a particular RDF shape live with that vocabulary: for example, [@lde/pipeline-void](../pipeline-void) ships `schemaOrgPartitionMergePlugin()`, a `beforeDatasetWrite` plugin that merges the VoID partition _nodes_ that Schema.org `http`/`https` variants produce.
-
-#### `provenancePlugin()`
-
-Appends [PROV-O](https://www.w3.org/TR/prov-o/) provenance quads (`prov:Entity`, `prov:Activity`, `prov:startedAtTime`, `prov:endedAtTime`) to every stage’s output. The `prov:Activity` is a stable IRI keyed on `(dataset, stage)`, not a blank node, so activities stay distinct – and a re-run stays idempotent – when per-dataset outputs are merged into one graph (blank-node labels are not unique across separately serialised documents and would fuse unrelated activities).
-
-#### `namespaceNormalizationPlugin(options)`
-
-Rewrites every IRI in the `from` namespace to the `to` namespace, in subject, predicate and object position alike. A blanket, vocabulary-agnostic rewrite for standardizing a namespace across a dataset’s own quads – for example when mapping instance data to an application profile.
-
-```typescript
-import { namespaceNormalizationPlugin } from '@lde/pipeline';
-
-new Pipeline({
-  // ...
-  plugins: [
-    namespaceNormalizationPlugin({
-      from: 'http://example.org/',
-      to: 'https://example.org/',
-    }),
-  ],
-});
-```
-
-#### `schemaOrgNormalizationPlugin(options?)`
-
-Convenience wrapper around `namespaceNormalizationPlugin` for the common case: rewrites `http://schema.org/` to `https://schema.org/` (pass `{ reverse: true }` for the opposite direction).
-
-```typescript
-import { schemaOrgNormalizationPlugin, provenancePlugin } from '@lde/pipeline';
-
-new Pipeline({
-  // ...
-  plugins: [schemaOrgNormalizationPlugin(), provenancePlugin()],
-});
-```
-
-This is a plain rewrite and knows nothing about VoID. If your pipeline emits VoID partitions and a dataset mixes both schema.org namespaces, rewriting the `void:class` objects alone still leaves two `void:classPartition` nodes for one class – use [`schemaOrgPartitionMergePlugin`](../pipeline-void) from `@lde/pipeline-void`, which also merges those nodes.
-
 ## Usage
 
 ```typescript
@@ -481,3 +41,7 @@ const pipeline = new Pipeline({
 
 await pipeline.run();
 ```
+
+## Documentation
+
+See the [full documentation](https://ldelements.org/reference/pipeline).

package/dist/pipeline.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"pipeline.d.ts","sourceRoot":"","sources":["../src/pipeline.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,OAAO,EAAgB,MAAM,cAAc,CAAC;AACrD,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAEzC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AACrD,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAEhD,OAAO,KAAK,EAKV,MAAM,EACP,MAAM,oBAAoB,CAAC;AAE5B,OAAO,EACL,KAAK,oBAAoB,EAI1B,MAAM,4BAA4B,CAAC;AAKpC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,0BAA0B,CAAC;AACpE,OAAO,KAAK,EAEV,gBAAgB,EACjB,MAAM,uBAAuB,CAAC;AAG/B,OAAO,EAEL,KAAK,aAAa,EACnB,MAAM,2BAA2B,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED;;;;;GAKG;AACH,MAAM,WAAW,yBAAyB;IACxC,OAAO,EAAE,OAAO,CAAC;CAClB;AAED,wDAAwD;AACxD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,gBAAgB,CAAC,EAAE,aAAa,CAAC,uBAAuB,CAAC,CAAC;IAC1D;;;;;;;;;;;;OAYG;IACH,kBAAkB,CAAC,EAAE,aAAa,CAAC,yBAAyB,CAAC,CAAC;CAC/D;AAED,MAAM,WAAW,eAAe,CAAC,GAAG,GAAG,IAAI;IACzC,eAAe,EAAE,eAAe,CAAC;IACjC,MAAM,EAAE,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;IACrB,OAAO,EAAE,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;IACrC;;;;;;;OAOG;IACH,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,cAAc,EAAE,GAAG,KAAK,CAAC;IAC1D,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,QAAQ,CAAC,EAAE;QACT,mBAAmB,EAAE,mBAAmB,CAAC;QACzC,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;IACF;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,gBAAgB,GAAG,SAAS,gBAAgB,EAAE,CAAC;IAC1D;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,eAAe,CAAC;IAClC;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,aAAa,CAAC;CAC/B;AA0OD,qBAAa,QAAQ,CAAC,GAAG,GAAG,IAAI;IAC9B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAClD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAe;IACtC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAc;IACrC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAyC;IAC3E,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAC,CAA2C;IAC/E,OAAO,CAAC,QAAQ,CAAC,oBAAoB,CAAuB;IAC5D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmC;IAC7D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmB;IAC7C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAsB;IACrD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAkB;IACnD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAS;gBAE9B,OAAO,EAAE,eAAe,CAAC,GAAG,CAAC;IA6DnC,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;YAoDZ,cAAc;IAgO5B,+EAA+E;YACjE,aAAa;YAmBb,gBAAgB;IAa9B,OAAO,CAAE,aAAa;IAStB;;;;;OAKG;IACH,OAAO,CAAC,WAAW;IAgBnB;;;;;;;;OAQG;IACH,OAAO,CAAC,0BAA0B;IA4BlC;;;;;OAKG;YACW,SAAS;IAuCvB;;;OAGG;YACW,QAAQ;IA0CtB,2EAA2E;YAC7D,eAAe;YAqBf,QAAQ;YA8EP,SAAS;CAczB"}
+{"version":3,"file":"pipeline.d.ts","sourceRoot":"","sources":["../src/pipeline.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,OAAO,EAAgB,MAAM,cAAc,CAAC;AACrD,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAEzC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AACrD,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAEhD,OAAO,KAAK,EAKV,MAAM,EACP,MAAM,oBAAoB,CAAC;AAE5B,OAAO,EACL,KAAK,oBAAoB,EAI1B,MAAM,4BAA4B,CAAC;AAKpC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,0BAA0B,CAAC;AACpE,OAAO,KAAK,EAEV,gBAAgB,EACjB,MAAM,uBAAuB,CAAC;AAG/B,OAAO,EAEL,KAAK,aAAa,EACnB,MAAM,2BAA2B,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED;;;;;GAKG;AACH,MAAM,WAAW,yBAAyB;IACxC,OAAO,EAAE,OAAO,CAAC;CAClB;AAED,wDAAwD;AACxD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;;OAMG;IACH,gBAAgB,CAAC,EAAE,aAAa,CAAC,uBAAuB,CAAC,CAAC;IAC1D;;;;;;;;;;;;OAYG;IACH,kBAAkB,CAAC,EAAE,aAAa,CAAC,yBAAyB,CAAC,CAAC;CAC/D;AAED,MAAM,WAAW,eAAe,CAAC,GAAG,GAAG,IAAI;IACzC,eAAe,EAAE,eAAe,CAAC;IACjC,MAAM,EAAE,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC;IACrB,OAAO,EAAE,MAAM,CAAC,GAAG,CAAC,GAAG,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC;IACrC;;;;;;;OAOG;IACH,OAAO,CAAC,EAAE,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,GAAG,cAAc,EAAE,GAAG,KAAK,CAAC;IAC1D,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,QAAQ,CAAC,EAAE;QACT,mBAAmB,EAAE,mBAAmB,CAAC;QACzC,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;IACF;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,gBAAgB,GAAG,SAAS,gBAAgB,EAAE,CAAC;IAC1D;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,eAAe,CAAC;IAClC;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,aAAa,CAAC;CAC/B;AA0OD,qBAAa,QAAQ,CAAC,GAAG,GAAG,IAAI;IAC9B,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAClD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAe;IACtC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAc;IACrC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAyC;IAC3E,OAAO,CAAC,QAAQ,CAAC,kBAAkB,CAAC,CAA2C;IAC/E,OAAO,CAAC,QAAQ,CAAC,oBAAoB,CAAuB;IAC5D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmC;IAC7D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmB;IAC7C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAsB;IACrD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAkB;IACnD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAS;gBAE9B,OAAO,EAAE,eAAe,CAAC,GAAG,CAAC;IA6DnC,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;YA0DZ,cAAc;IAgO5B,+EAA+E;YACjE,aAAa;YAyBb,gBAAgB;IAa9B,OAAO,CAAE,aAAa;IAStB;;;;;OAKG;IACH,OAAO,CAAC,WAAW;IAgBnB;;;;;;;;OAQG;IACH,OAAO,CAAC,0BAA0B;IA4BlC;;;;;OAKG;YACW,SAAS;IAuCvB;;;OAGG;YACW,QAAQ;IA0CtB,2EAA2E;YAC7D,eAAe;YAqBf,QAAQ;YA8EP,SAAS;CAczB"}

package/dist/pipeline.js

@@ -273,6 +273,11 @@ export class Pipeline {
     async run() {
         const start = Date.now();
         this.reporter?.pipelineStart?.(this.name);
+        // Fail fast on a store that cannot persist records (e.g. a provenance
+        // file on a mount the runtime user cannot write to): every dataset would
+        // process fine, but the records would be lost and skip-unchanged would
+        // silently never engage on any later run.
+        await this.provenanceStore?.check?.();
         const selectStart = Date.now();
         const datasets = await this.datasetSelector.select();
         this.reporter?.datasetsSelected?.(datasets.total, Date.now() - selectStart);
@@ -486,9 +491,12 @@ export class Pipeline {
                 status,
             });
         }
-        catch {
-            // A failed write must not abort the run; the dataset simply reprocesses
-            // next run, its record not yet updated.
+        catch (error) {
+            // A failed write must not abort the run – the dataset simply
+            // reprocesses next run, its record not yet updated – but it must be
+            // surfaced: a store that fails every write silently disables
+            // skip-unchanged.
+            this.reporter?.stageFailed?.('provenance', error instanceof Error ? error : new Error(String(error)));
         }
     }
     async reportValidators(dataset) {

package/dist/provenance/fileProvenanceStore.d.ts

@@ -26,6 +26,13 @@ export interface FileProvenanceStoreOptions {
 export declare class FileProvenanceStore implements ProvenanceStore {
     private readonly path;
     constructor(options: FileProvenanceStoreOptions);
+    /**
+     * Verify the file can be written by actually creating (and removing) a
+     * probe file next to it. Catches the mount that a non-root runtime user
+     * cannot write to (e.g. a root-owned Docker volume, see issue #661) at the
+     * start of the run, before writes would silently fail per dataset.
+     */
+    check(): Promise<void>;
     get(datasetUri: URL): Promise<ProcessingRecord | null>;
     set(datasetUri: URL, record: ProcessingRecord): Promise<void>;
     private readAll;

package/dist/provenance/fileProvenanceStore.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"fileProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileProvenanceStore.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,0BAA0B;IACzC;;;;OAIG;IACH,IAAI,EAAE,MAAM,CAAC;CACd;AAED;;;;;;;;;;;;;;GAcG;AACH,qBAAa,mBAAoB,YAAW,eAAe;IACzD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;gBAElB,OAAO,EAAE,0BAA0B;IAIzC,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAKtD,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YAMrD,OAAO;YAcP,QAAQ;CAQvB"}
+{"version":3,"file":"fileProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileProvenanceStore.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,0BAA0B;IACzC;;;;OAIG;IACH,IAAI,EAAE,MAAM,CAAC;CACd;AAED;;;;;;;;;;;;;;GAcG;AACH,qBAAa,mBAAoB,YAAW,eAAe;IACzD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;gBAElB,OAAO,EAAE,0BAA0B;IAI/C;;;;;OAKG;IACG,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IActB,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAKtD,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YAMrD,OAAO;YAcP,QAAQ;CAQvB"}

package/dist/provenance/fileProvenanceStore.js

@@ -1,4 +1,4 @@
-import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';
+import { mkdir, readFile, rename, unlink, writeFile } from 'node:fs/promises';
 import { dirname } from 'node:path';
 /**
  * A {@link ProvenanceStore} that persists all records to a single JSON file,
@@ -20,6 +20,23 @@ export class FileProvenanceStore {
     constructor(options) {
         this.path = options.path;
     }
+    /**
+     * Verify the file can be written by actually creating (and removing) a
+     * probe file next to it. Catches the mount that a non-root runtime user
+     * cannot write to (e.g. a root-owned Docker volume, see issue #661) at the
+     * start of the run, before writes would silently fail per dataset.
+     */
+    async check() {
+        try {
+            await mkdir(dirname(this.path), { recursive: true });
+            const probePath = `${this.path}.${process.pid}.check`;
+            await writeFile(probePath, '');
+            await unlink(probePath);
+        }
+        catch (error) {
+            throw new Error(`Provenance file ${this.path} is not writable; make its directory writable by the user the pipeline runs as: ${String(error)}`, { cause: error });
+        }
+    }
     async get(datasetUri) {
         const records = await this.readAll();
         return records[datasetUri.toString()] ?? null;

package/dist/provenance/store.d.ts

@@ -8,6 +8,14 @@ import type { ProcessingRecord } from './record.js';
  * triplestore, files, or anything else.
  */
 export interface ProvenanceStore {
+    /**
+     * Verify the store can persist records, throwing when it cannot. The
+     * pipeline calls this once at the start of a run, so a store that would
+     * fail every write – e.g. a provenance file on a mount the runtime user
+     * cannot write to – fails the run immediately instead of silently
+     * disabling skip-unchanged.
+     */
+    check?(): Promise<void>;
     /**
      * The record from the dataset’s last processing, or `null` if it has never
      * been processed (or the store was wiped). A `null` result drives a

package/dist/provenance/store.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"store.d.ts","sourceRoot":"","sources":["../../src/provenance/store.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAEpD;;;;;;;GAOG;AACH,MAAM,WAAW,eAAe;IAC9B;;;;OAIG;IACH,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC,CAAC;IACvD,oEAAoE;IACpE,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;CAC/D"}
+{"version":3,"file":"store.d.ts","sourceRoot":"","sources":["../../src/provenance/store.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAEpD;;;;;;;GAOG;AACH,MAAM,WAAW,eAAe;IAC9B;;;;;;OAMG;IACH,KAAK,CAAC,IAAI,OAAO,CAAC,IAAI,CAAC,CAAC;IACxB;;;;OAIG;IACH,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC,CAAC;IACvD,oEAAoE;IACpE,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC;CAC/D"}

package/dist/sparql/graph.d.ts

@@ -1,8 +1,18 @@
-import { type QueryConstruct } from '@traqula/rules-sparql-1-1';
+import { type QueryConstruct, type QuerySelect } from '@traqula/rules-sparql-1-1';
 /**
- * Set the default graph (FROM clause) on a parsed CONSTRUCT query.
+ * Set the default graph (FROM clause) on a parsed CONSTRUCT or SELECT query.
  *
  * Mutates the query in place, replacing any existing FROM clause.
+ *
+ * Both query forms scope the same way, and a stage reading a
+ * {@link Distribution} with a `namedGraph` must scope **both**: were only the
+ * reader to honour it, an item selector would pick its roots from the whole
+ * endpoint and the reader would then find nothing for most of them.
+ *
+ * The graph IRI is {@link assertSafeIri}-checked before it becomes an `<…>`
+ * reference: a `namedGraph` is a plain string, typically carried in from
+ * third-party registry data, and one containing an angle bracket or whitespace
+ * would break out of the IRI reference and rewrite the query around it.
  */
-export declare function withDefaultGraph(query: QueryConstruct, graphIri: string): void;
+export declare function withDefaultGraph(query: QueryConstruct | QuerySelect, graphIri: string): void;
 //# sourceMappingURL=graph.d.ts.map

package/dist/sparql/graph.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"graph.d.ts","sourceRoot":"","sources":["../../src/sparql/graph.ts"],"names":[],"mappings":"AAAA,OAAO,EAAc,KAAK,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAI5E;;;;GAIG;AACH,wBAAgB,gBAAgB,CAC9B,KAAK,EAAE,cAAc,EACrB,QAAQ,EAAE,MAAM,GACf,IAAI,CAKN"}
+{"version":3,"file":"graph.d.ts","sourceRoot":"","sources":["../../src/sparql/graph.ts"],"names":[],"mappings":"AACA,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,WAAW,EACjB,MAAM,2BAA2B,CAAC;AAInC;;;;;;;;;;;;;;GAcG;AACH,wBAAgB,gBAAgB,CAC9B,KAAK,EAAE,cAAc,GAAG,WAAW,EACnC,QAAQ,EAAE,MAAM,GACf,IAAI,CAMN"}

package/dist/sparql/graph.js

@@ -1,10 +1,22 @@
-import { AstFactory } from '@traqula/rules-sparql-1-1';
+import { assertSafeIri } from '@lde/dataset';
+import { AstFactory, } from '@traqula/rules-sparql-1-1';
 const F = new AstFactory();
 /**
- * Set the default graph (FROM clause) on a parsed CONSTRUCT query.
+ * Set the default graph (FROM clause) on a parsed CONSTRUCT or SELECT query.
  *
  * Mutates the query in place, replacing any existing FROM clause.
+ *
+ * Both query forms scope the same way, and a stage reading a
+ * {@link Distribution} with a `namedGraph` must scope **both**: were only the
+ * reader to honour it, an item selector would pick its roots from the whole
+ * endpoint and the reader would then find nothing for most of them.
+ *
+ * The graph IRI is {@link assertSafeIri}-checked before it becomes an `<…>`
+ * reference: a `namedGraph` is a plain string, typically carried in from
+ * third-party registry data, and one containing an angle bracket or whitespace
+ * would break out of the IRI reference and rewrite the query around it.
  */
 export function withDefaultGraph(query, graphIri) {
+    assertSafeIri(graphIri);
     query.datasets = F.datasetClauses([{ clauseType: 'default', value: F.termNamed(F.gen(), graphIri) }], F.gen());
 }

package/dist/sparql/selector.d.ts

@@ -7,7 +7,7 @@ export interface SparqlItemSelectorOptions {
      * SELECT query projecting at least one named variable.
      *
      * A `LIMIT` clause in the query overrides the stage's `batchSize` as the
-     * page size — use this when the SPARQL endpoint enforces a result limit.
+     * page size – use this when the SPARQL endpoint enforces a result limit.
      * It does **not** cap the total number of bindings the selector yields;
      * pagination continues with `OFFSET` until the source is exhausted. Use
      * {@link maxResults} to cap the total.
@@ -15,7 +15,7 @@ export interface SparqlItemSelectorOptions {
     query: string;
     /**
      * Maximum number of bindings the selector yields across all pages.
-     * Use this for sampling — “give me at most N items, don’t walk the full
+     * Use this for sampling – “give me at most N items, don’t walk the full
      * source”. Independent of {@link query}’s `LIMIT`, which controls page
      * size. Pagination stops as soon as `maxResults` bindings have been
      * yielded. Must not be negative.
@@ -36,8 +36,12 @@ export interface SparqlItemSelectorOptions {
  * pagination early. Dropped rows are still fetched; to skip them at the
  * endpoint, filter in the query itself (e.g. `FILTER(isIRI(?s))`).
  *
- * The endpoint URL comes from the {@link Distribution} passed to {@link select}.
- * Pagination is an internal detail — consumers iterate binding rows directly.
+ * The endpoint URL comes from the {@link Distribution} passed to {@link select},
+ * as does the graph to scope to: a distribution declaring a `namedGraph` selects
+ * within that graph (a `FROM` clause), the same scoping `SparqlConstructReader`
+ * applies to the extraction. The two must scope alike, or the roots come from a
+ * wider source than the quads read for them.
+ * Pagination is an internal detail – consumers iterate binding rows directly.
  *
  * The page size (results per SPARQL request) is determined by, in order:
  * 1. A `LIMIT` clause in the selector query (for endpoints with hard result limits)

package/dist/sparql/selector.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"selector.d.ts","sourceRoot":"","sources":["../../src/sparql/selector.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAEjD,OAAO,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAQ9D,OAAO,KAAK,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAC/D,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAoBpD,MAAM,WAAW,yBAAyB;IACxC;;;;;;;;OAQG;IACH,KAAK,EAAE,MAAM,CAAC;IACd;;;;;;OAMG;IACH,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,+BAA+B;IAC/B,OAAO,CAAC,EAAE,qBAAqB,CAAC;CACjC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GA8BG;AACH,qBAAa,kBAAmB,YAAW,YAAY;IACrD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAc;IACrC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAoB;IAClD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAwB;gBAEzC,OAAO,EAAE,yBAAyB;IA0BvC,MAAM,CACX,YAAY,EAAE,YAAY,EAC1B,SAAS,CAAC,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE,aAAa,GACtB,qBAAqB,CAAC,gBAAgB,CAAC;IAwE1C;;;;OAIG;YACW,uBAAuB;CA8BtC"}
+{"version":3,"file":"selector.d.ts","sourceRoot":"","sources":["../../src/sparql/selector.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAEjD,OAAO,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAQ9D,OAAO,KAAK,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,aAAa,CAAC;AAE/D,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AAoBpD,MAAM,WAAW,yBAAyB;IACxC;;;;;;;;OAQG;IACH,KAAK,EAAE,MAAM,CAAC;IACd;;;;;;OAMG;IACH,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,+BAA+B;IAC/B,OAAO,CAAC,EAAE,qBAAqB,CAAC;CACjC;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AACH,qBAAa,kBAAmB,YAAW,YAAY;IACrD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAc;IACrC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAoB;IAClD,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAS;IACrC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAwB;gBAEzC,OAAO,EAAE,yBAAyB;IA0BvC,MAAM,CACX,YAAY,EAAE,YAAY,EAC1B,SAAS,CAAC,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE,aAAa,GACtB,qBAAqB,CAAC,gBAAgB,CAAC;IA8E1C;;;;OAIG;YACW,uBAAuB;CA8BtC"}

package/dist/sparql/selector.js

@@ -2,6 +2,7 @@ import { SparqlEndpointFetcher } from 'fetch-sparql-endpoint';
 import { Parser } from '@traqula/parser-sparql-1-1';
 import { Generator } from '@traqula/generator-sparql-1-1';
 import { AstFactory, } from '@traqula/rules-sparql-1-1';
+import { withDefaultGraph } from './graph.js';
 import { ConstantTimeoutPolicy, } from './timeoutPolicy.js';
 const transientStatusPattern = /HTTP status (\d+)/;
 /**
@@ -25,8 +26,12 @@ const F = new AstFactory();
  * pagination early. Dropped rows are still fetched; to skip them at the
  * endpoint, filter in the query itself (e.g. `FILTER(isIRI(?s))`).
  *
- * The endpoint URL comes from the {@link Distribution} passed to {@link select}.
- * Pagination is an internal detail — consumers iterate binding rows directly.
+ * The endpoint URL comes from the {@link Distribution} passed to {@link select},
+ * as does the graph to scope to: a distribution declaring a `namedGraph` selects
+ * within that graph (a `FROM` clause), the same scoping `SparqlConstructReader`
+ * applies to the extraction. The two must scope alike, or the roots come from a
+ * wider source than the quads read for them.
+ * Pagination is an internal detail – consumers iterate binding rows directly.
  *
  * The page size (results per SPARQL request) is determined by, in order:
  * 1. A `LIMIT` clause in the selector query (for endpoints with hard result limits)
@@ -76,6 +81,12 @@ export class SparqlItemSelector {
             throw new Error(`Page size must be positive; got ${basePageSize} (from the query’s LIMIT or the stage’s batchSize)`);
         }
         const endpoint = distribution.accessUrl;
+        // Per call, so concurrent selections over different datasets cannot see
+        // each other's graph scope or page window.
+        const query = structuredClone(this.parsed);
+        if (distribution.namedGraph) {
+            withDefaultGraph(query, distribution.namedGraph);
+        }
         const policy = options?.timeout ?? defaultTimeoutPolicy;
         let offset = 0;
         let totalFetched = 0;
@@ -92,8 +103,8 @@ export class SparqlItemSelector {
             const effectivePageSize = offset === 0 || totalFetched > totalYielded
                 ? basePageSize
                 : Math.min(basePageSize, remaining);
-            this.parsed.solutionModifiers.limitOffset = F.solutionModifierLimitOffset(effectivePageSize, offset, F.gen());
-            const paginatedQuery = generator.generate(this.parsed);
+            query.solutionModifiers.limitOffset = F.solutionModifierLimitOffset(effectivePageSize, offset, F.gen());
+            const paginatedQuery = generator.generate(query);
             const stream = await this.fetchBindingsWithPolicy(endpoint, paginatedQuery, policy);
             // Fetched rows drive pagination, yielded rows only `maxResults` – see
             // the class JSDoc for why dropped rows must keep their page slot.

package/dist/stage.d.ts

@@ -102,6 +102,35 @@ export interface StageOptions<Out = Quad> {
      * @default 10
      */
     maxConcurrency?: number;
+    /**
+     * Where this stage reads, when that is **not** the dataset’s own
+     * distribution: given the dataset and the distribution the pipeline resolved
+     * for it, return the {@link Distribution} this stage should use instead.
+     *
+     * One hook covers the whole stage – the item selector and every reader are
+     * handed the same substitute – because a stage that reads elsewhere must
+     * *select* there too, or its roots and its quads come from different sources.
+     *
+     * The dataset is passed so the substitute can be derived per dataset. The
+     * motivating case is a stage sourced from the dataset registry rather than
+     * from the dataset’s data: the registry endpoint is fixed, but the graph to
+     * scope to is the dataset in hand.
+     *
+     * ```ts
+     * sourceFor: (dataset) => {
+     *   const source = new Distribution(registryEndpoint, SPARQL_MEDIA_TYPE);
+     *   source.namedGraph = dataset.iri.toString();
+     *   return source;
+     * }
+     * ```
+     *
+     * Omit it – the default – to read the dataset’s own distribution.
+     *
+     * A **chained child** stage may not declare one: a child is handed its
+     * parent’s output as its distribution, so substituting a source would discard
+     * the chain it exists to continue. The parent’s constructor rejects that.
+     */
+    sourceFor?: (dataset: Dataset, distribution: Distribution) => Distribution;
     /** Child stages that chain off this stage's output. */
     stages?: Stage[];
     /**
@@ -148,6 +177,9 @@ export declare class Stage<Out = Quad> {
     readonly stages: readonly Stage[];
     /** Whether an empty result is treated as a hard failure. @see {@link StageOptions.expectsOutput} */
     readonly expectsOutput: boolean;
+    /** Whether this stage reads a source of its own rather than the dataset’s
+     *  resolved distribution. @see {@link StageOptions.sourceFor} */
+    readonly sourcesOwnData: boolean;
     private readonly readers;
     private readonly itemSelector?;
     private readonly batchSize;
@@ -155,10 +187,11 @@ export declare class Stage<Out = Quad> {
     private readonly validation?;
     private readonly project?;
     private readonly queueCapacity?;
+    private readonly sourceFor?;
     constructor(options: StageOptions<Out>);
     /** The validator for this stage, if configured. */
     get validator(): Validator | undefined;
-    run(dataset: Dataset, distribution: Distribution, writer: DatasetWriter<Out>, options?: RunOptions): Promise<NotSupported | void>;
+    run(dataset: Dataset, resolved: Distribution, writer: DatasetWriter<Out>, options?: RunOptions): Promise<NotSupported | void>;
     /**
      * Throw when {@link StageOptions.expectsOutput} is set but the stage produced
      * no quads – a supported-but-empty result that signals a truncated or aborted

package/dist/stage.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AACrD,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,KAAK,EAAE,MAAM,EAAE,gBAAgB,EAAE,MAAM,oBAAoB,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAC;AAClD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,2BAA2B,CAAC;AAE/D,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,oBAAoB,CAAC;AAGxD;;;;;;;GAOG;AACH,MAAM,MAAM,aAAa,CAAC,GAAG,IAAI,CAC/B,KAAK,EAAE,aAAa,CAAC,IAAI,CAAC,EAC1B,OAAO,EAAE,GAAG,KACT,aAAa,CAAC,IAAI,CAAC,CAAC;AAEzB;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,OAAO,EAAE,OAAO,CAAC;IACjB,YAAY,EAAE,YAAY,CAAC;IAC3B,KAAK,EAAE,MAAM,CAAC;CACf;AAED;;;;;;GAMG;AACH,MAAM,MAAM,cAAc,CAAC,GAAG,IAAI,CAChC,KAAK,EAAE,SAAS,IAAI,EAAE,EACtB,OAAO,EAAE,YAAY,KAClB,QAAQ,CAAC,GAAG,CAAC,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;AAExC;;;;;GAKG;AACH,MAAM,WAAW,YAAa,SAAQ,aAAa;IACjD,QAAQ,EAAE,SAAS,gBAAgB,EAAE,CAAC;CACvC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,aAAa,CAAC,aAAa,CAAC,GAAG,aAAa,CAAC,aAAa,CAAC,EAAE,CAAC;CAC3E;AAED,yEAAyE;AACzE,MAAM,MAAM,YAAY,GACpB,MAAM,GACN,cAAc,GACd,CAAC,MAAM,GAAG,cAAc,CAAC,EAAE,CAAC;AAQhC,MAAM,WAAW,YAAY,CAAC,GAAG,GAAG,IAAI;IACtC,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,EAAE,YAAY,CAAC;IACtB,YAAY,CAAC,EAAE,YAAY,CAAC;IAC5B;;;;;;OAMG;IACH,OAAO,CAAC,EAAE,cAAc,CAAC,GAAG,CAAC,CAAC;IAC9B;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;OAQG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,uDAAuD;IACvD,MAAM,CAAC,EAAE,KAAK,EAAE,CAAC;IACjB;;;;;;;;;;;;;;;OAeG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,mFAAmF;IACnF,UAAU,CAAC,EAAE;QACX,SAAS,EAAE,SAAS,CAAC;QACrB,iEAAiE;QACjE,SAAS,CAAC,EAAE,OAAO,GAAG,MAAM,GAAG,MAAM,CAAC;KACvC,CAAC;CACH;AAED,MAAM,WAAW,UAAU;IACzB,UAAU,CAAC,EAAE,CAAC,cAAc,EAAE,MAAM,EAAE,cAAc,EAAE,MAAM,KAAK,IAAI,CAAC;IACtE;;;;;OAKG;IACH,OAAO,CAAC,EAAE,aAAa,CAAC;CACzB;AAED,uDAAuD;AACvD,MAAM,WAAW,aAAa;IAC5B,+BAA+B;IAC/B,OAAO,CAAC,EAAE,aAAa,CAAC;CACzB;AAED,qBAAa,KAAK,CAAC,GAAG,GAAG,IAAI;IAC3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,MAAM,EAAE,SAAS,KAAK,EAAE,CAAC;IAClC,oGAAoG;IACpG,QAAQ,CAAC,aAAa,EAAE,OAAO,CAAC;IAChC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqB;IAC7C,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAe;IAC7C,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;IACxC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAkC;IAC9D,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAsB;IAC/C,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAS;gBAE5B,OAAO,EAAE,YAAY,CAAC,GAAG,CAAC;IAuBtC,mDAAmD;IACnD,IAAI,SAAS,IAAI,SAAS,GAAG,SAAS,CAErC;IAEK,GAAG,CACP,OAAO,EAAE,OAAO,EAChB,YAAY,EAAE,YAAY,EAC1B,MAAM,EAAE,aAAa,CAAC,GAAG,CAAC,EAC1B,OAAO,CAAC,EAAE,UAAU,GACnB,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC;IA2E/B;;;;OAIG;IACH,OAAO,CAAC,cAAc;YAMR,eAAe;IAiL7B;;;OAGG;YACW,cAAc;YAqBd,OAAO;IA6BrB;;;;;OAKG;IACH,OAAO,CAAC,eAAe;CAiBxB;AAiDD,2GAA2G;AAC3G,MAAM,WAAW,YAAY;IAC3B,MAAM,CACJ,YAAY,EAAE,YAAY,EAC1B,SAAS,CAAC,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE,aAAa,GACtB,aAAa,CAAC,gBAAgB,CAAC,CAAC;CACpC"}
+{"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AACrD,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AACzC,OAAO,KAAK,EAAE,MAAM,EAAE,gBAAgB,EAAE,MAAM,oBAAoB,CAAC;AACnE,OAAO,EAAE,YAAY,EAAE,MAAM,oBAAoB,CAAC;AAClD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,2BAA2B,CAAC;AAE/D,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,oBAAoB,CAAC;AAGxD;;;;;;;GAOG;AACH,MAAM,MAAM,aAAa,CAAC,GAAG,IAAI,CAC/B,KAAK,EAAE,aAAa,CAAC,IAAI,CAAC,EAC1B,OAAO,EAAE,GAAG,KACT,aAAa,CAAC,IAAI,CAAC,CAAC;AAEzB;;;;;;GAMG;AACH,MAAM,WAAW,aAAa;IAC5B,OAAO,EAAE,OAAO,CAAC;IACjB,YAAY,EAAE,YAAY,CAAC;IAC3B,KAAK,EAAE,MAAM,CAAC;CACf;AAED;;;;;;GAMG;AACH,MAAM,MAAM,cAAc,CAAC,GAAG,IAAI,CAChC,KAAK,EAAE,SAAS,IAAI,EAAE,EACtB,OAAO,EAAE,YAAY,KAClB,QAAQ,CAAC,GAAG,CAAC,GAAG,aAAa,CAAC,GAAG,CAAC,CAAC;AAExC;;;;;GAKG;AACH,MAAM,WAAW,YAAa,SAAQ,aAAa;IACjD,QAAQ,EAAE,SAAS,gBAAgB,EAAE,CAAC;CACvC;AAED;;;;;;;;;;;;GAYG;AACH,MAAM,WAAW,cAAc;IAC7B,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,CAAC,EAAE,aAAa,CAAC,aAAa,CAAC,GAAG,aAAa,CAAC,aAAa,CAAC,EAAE,CAAC;CAC3E;AAED,yEAAyE;AACzE,MAAM,MAAM,YAAY,GACtB,MAAM,GAAG,cAAc,GAAG,CAAC,MAAM,GAAG,cAAc,CAAC,EAAE,CAAC;AAQxD,MAAM,WAAW,YAAY,CAAC,GAAG,GAAG,IAAI;IACtC,IAAI,EAAE,MAAM,CAAC;IACb,OAAO,EAAE,YAAY,CAAC;IACtB,YAAY,CAAC,EAAE,YAAY,CAAC;IAC5B;;;;;;OAMG;IACH,OAAO,CAAC,EAAE,cAAc,CAAC,GAAG,CAAC,CAAC;IAC9B;;;;;;;OAOG;IACH,aAAa,CAAC,EAAE,MAAM,CAAC;IACvB;;;;;;;;OAQG;IACH,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB;;;;;;;OAOG;IACH,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB;;;;;;;;;;;;;;;;;;;;;;;;;;;OA2BG;IACH,SAAS,CAAC,EAAE,CAAC,OAAO,EAAE,OAAO,EAAE,YAAY,EAAE,YAAY,KAAK,YAAY,CAAC;IAC3E,uDAAuD;IACvD,MAAM,CAAC,EAAE,KAAK,EAAE,CAAC;IACjB;;;;;;;;;;;;;;;OAeG;IACH,aAAa,CAAC,EAAE,OAAO,CAAC;IACxB,mFAAmF;IACnF,UAAU,CAAC,EAAE;QACX,SAAS,EAAE,SAAS,CAAC;QACrB,iEAAiE;QACjE,SAAS,CAAC,EAAE,OAAO,GAAG,MAAM,GAAG,MAAM,CAAC;KACvC,CAAC;CACH;AAED,MAAM,WAAW,UAAU;IACzB,UAAU,CAAC,EAAE,CAAC,cAAc,EAAE,MAAM,EAAE,cAAc,EAAE,MAAM,KAAK,IAAI,CAAC;IACtE;;;;;OAKG;IACH,OAAO,CAAC,EAAE,aAAa,CAAC;CACzB;AAED,uDAAuD;AACvD,MAAM,WAAW,aAAa;IAC5B,+BAA+B;IAC/B,OAAO,CAAC,EAAE,aAAa,CAAC;CACzB;AAED,qBAAa,KAAK,CAAC,GAAG,GAAG,IAAI;IAC3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,MAAM,EAAE,SAAS,KAAK,EAAE,CAAC;IAClC,oGAAoG;IACpG,QAAQ,CAAC,aAAa,EAAE,OAAO,CAAC;IAChC;qEACiE;IACjE,QAAQ,CAAC,cAAc,EAAE,OAAO,CAAC;IACjC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqB;IAC7C,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC,CAAe;IAC7C,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAS;IACnC,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAS;IACxC,OAAO,CAAC,QAAQ,CAAC,UAAU,CAAC,CAAkC;IAC9D,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAsB;IAC/C,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAC,CAAS;IACxC,OAAO,CAAC,QAAQ,CAAC,SAAS,CAAC,CAAiC;gBAEhD,OAAO,EAAE,YAAY,CAAC,GAAG,CAAC;IAmCtC,mDAAmD;IACnD,IAAI,SAAS,IAAI,SAAS,GAAG,SAAS,CAErC;IAEK,GAAG,CACP,OAAO,EAAE,OAAO,EAChB,QAAQ,EAAE,YAAY,EACtB,MAAM,EAAE,aAAa,CAAC,GAAG,CAAC,EAC1B,OAAO,CAAC,EAAE,UAAU,GACnB,OAAO,CAAC,YAAY,GAAG,IAAI,CAAC;IAgF/B;;;;OAIG;IACH,OAAO,CAAC,cAAc;YAMR,eAAe;IAiL7B;;;OAGG;YACW,cAAc;YAqBd,OAAO;IA6BrB;;;;;OAKG;IACH,OAAO,CAAC,eAAe;CAiBxB;AAiDD,2GAA2G;AAC3G,MAAM,WAAW,YAAY;IAC3B,MAAM,CACJ,YAAY,EAAE,YAAY,EAC1B,SAAS,CAAC,EAAE,MAAM,EAClB,OAAO,CAAC,EAAE,aAAa,GACtB,aAAa,CAAC,gBAAgB,CAAC,CAAC;CACpC"}

package/dist/stage.js

@@ -6,6 +6,9 @@ export class Stage {
     stages;
     /** Whether an empty result is treated as a hard failure. @see {@link StageOptions.expectsOutput} */
     expectsOutput;
+    /** Whether this stage reads a source of its own rather than the dataset’s
+     *  resolved distribution. @see {@link StageOptions.sourceFor} */
+    sourcesOwnData;
     readers;
     itemSelector;
     batchSize;
@@ -13,6 +16,7 @@ export class Stage {
     validation;
     project;
     queueCapacity;
+    sourceFor;
     constructor(options) {
         if (options.project && !options.itemSelector) {
             throw new Error(`Stage '${options.name}': 'project' requires an 'itemSelector' – without one there is no batch to project, only the readers' whole output.`);
@@ -20,6 +24,14 @@ export class Stage {
         if (options.project && (options.stages?.length ?? 0) > 0) {
             throw new Error(`Stage '${options.name}': 'project' cannot combine with chained 'stages' – a chained stage serializes to N-Triples, which a projected item cannot.`);
         }
+        // A child is handed its parent's output as its distribution; a `sourceFor`
+        // would substitute that away, silently reading elsewhere instead of
+        // continuing the chain. Caught here rather than at run time, where it looks
+        // like a chain that mysteriously produced nothing.
+        const sourcedChild = options.stages?.find((child) => child.sourcesOwnData);
+        if (sourcedChild) {
+            throw new Error(`Stage '${options.name}': chained stage '${sourcedChild.name}' declares 'sourceFor' – a chained stage reads its parent's output, so it cannot source its own data.`);
+        }
         this.name = options.name;
         this.stages = options.stages ?? [];
         this.readers = normalizeReaders(options.readers);
@@ -30,13 +42,20 @@ export class Stage {
         this.expectsOutput = options.expectsOutput ?? false;
         this.project = options.project;
         this.queueCapacity = options.queueCapacity;
+        this.sourceFor = options.sourceFor;
+        this.sourcesOwnData = options.sourceFor !== undefined;
     }
     /** The validator for this stage, if configured. */
     get validator() {
         return this.validation?.validator;
     }
-    async run(dataset, distribution, writer, options) {
+    async run(dataset, resolved, writer, options) {
         const timeout = options?.timeout;
+        // Substituted once, before anything reads: the selector and the readers
+        // must agree on where this stage's data comes from.
+        const distribution = this.sourceFor
+            ? this.sourceFor(dataset, resolved)
+            : resolved;
         if (this.itemSelector) {
             return this.runWithSelector(this.itemSelector.select(distribution, this.batchSize, {
                 timeout,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lde/pipeline",
-  "version": "0.35.3",
+  "version": "0.35.5",
   "repository": {
     "url": "git+https://github.com/ldelements/lde.git",
     "directory": "packages/pipeline"
@@ -39,7 +39,7 @@
     "fetch-sparql-endpoint": "^7.1.1",
     "filenamify-url": "^4.0.0",
     "is-network-error": "^1.3.2",
-    "n3": "^2.1.1",
+    "n3": "^2.1.2",
     "p-retry": "^8.0.0",
     "rdf-string": "^2.0.1",
     "tslib": "^2.3.0"