@lde/pipeline 0.31.5 → 0.33.0

package/README.md

@@ -3,7 +3,7 @@
 A framework for transforming large RDF datasets, primarily using [SPARQL](https://www.w3.org/TR/sparql11-query/) queries with TypeScript for the parts that are hard to express in SPARQL alone.
 
 - **SPARQL-native.** Data transformations are plain SPARQL query files — portable, transparent, testable and version-controlled.
-- **Composable.** Executors are an interface: wrap a SPARQL executor with custom TypeScript to handle edge cases like date parsing or string normalisation (see [Executor](#executor)).
+- **Composable.** Executors are an interface: wrap a SPARQL reader with custom TypeScript to handle edge cases like date parsing or string normalisation (see [Reader](#reader)).
 - **Extensible.** A plugin system lets packages like [@lde/pipeline-void](../pipeline-void) (or your own plugins) hook into the pipeline lifecycle.
 
 ## Components
@@ -50,7 +50,7 @@ const resolver = new ImportResolver(new SparqlDistributionResolver(), {
 
 ### Stage
 
-A stage groups an item selector, one or more executors, and configuration:
+A stage groups an item selector, one or more readers, and configuration:
 
 ```typescript
 new Stage({
@@ -58,7 +58,7 @@ new Stage({
   itemSelector: new SparqlItemSelector({
     query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
   }),
-  executors: executor,
+  readers: reader,
   batchSize: 100,
   maxConcurrency: 5,
 });
@@ -66,9 +66,9 @@ new Stage({
 
 #### Batch size
 
-`batchSize` (default: 10) controls how many variable bindings are passed to each executor call as a `VALUES` clause. It also sets the page size for the item selector's SPARQL requests, so that each paginated request fills exactly one executor batch.
+`batchSize` (default: 10) controls how many variable bindings are passed to each reader call as a `VALUES` clause. It also sets the page size for the item selector's SPARQL requests, so that each paginated request fills exactly one reader batch.
 
-Some SPARQL endpoints enforce different result limits for SELECT and CONSTRUCT queries. Since the selector uses SELECT and the executor uses CONSTRUCT, a `LIMIT` clause in the selector query overrides `batchSize` as the page size. Use this when the endpoint caps SELECT results below your desired batch size:
+Some SPARQL endpoints enforce different result limits for SELECT and CONSTRUCT queries. Since the selector uses SELECT and the reader uses CONSTRUCT, a `LIMIT` clause in the selector query overrides `batchSize` as the page size. Use this when the endpoint caps SELECT results below your desired batch size:
 
 ```typescript
 // Endpoint caps SELECT results at 500, but each CONSTRUCT can handle 1000 bindings.
@@ -77,14 +77,14 @@ new Stage({
   itemSelector: new SparqlItemSelector({
     query: 'SELECT DISTINCT ?class WHERE { ?s a ?class } LIMIT 500',
   }),
-  executors: executor,
+  readers: reader,
   batchSize: 1000, // Two SELECT pages fill one CONSTRUCT batch.
 });
 ```
 
 #### Concurrency
 
-`maxConcurrency` (default: 10) limits the total number of concurrent SPARQL queries. Within each batch, all executors run in parallel; the number of concurrent batches is automatically reduced to `⌊maxConcurrency / executorCount⌋` so the total query pressure stays within the limit. For example, with `maxConcurrency: 10` and two executors per stage, up to 5 batches run concurrently (10 SPARQL queries total).
+`maxConcurrency` (default: 10) limits the total number of concurrent SPARQL queries. Within each batch, all readers run in parallel; the number of concurrent batches is automatically reduced to `⌊maxConcurrency / executorCount⌋` so the total query pressure stays within the limit. For example, with `maxConcurrency: 10` and two readers per stage, up to 5 batches run concurrently (10 SPARQL queries total).
 
 #### Expecting output
 
@@ -94,7 +94,7 @@ Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always ret
 
 ### Item Selector
 
-Selects resources from the distribution and fans out executor calls per batch of results. Implements the `ItemSelector` interface:
+Selects resources from the distribution and fans out reader calls per batch of results. Implements the `ItemSelector` interface:
 
 ```typescript
 interface ItemSelector {
@@ -143,12 +143,12 @@ const itemSelector: ItemSelector = {
 };
 ```
 
-### Executor
+### Reader
 
-Generates RDF triples. The built-in `SparqlConstructExecutor` runs a SPARQL CONSTRUCT query with template substitution and variable bindings:
+Generates RDF triples. The built-in `SparqlConstructReader` runs a SPARQL CONSTRUCT query with template substitution and variable bindings:
 
 ```typescript
-const executor = new SparqlConstructExecutor({
+const reader = new SparqlConstructReader({
   query: 'CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }',
 });
 ```
@@ -156,7 +156,7 @@ const executor = new SparqlConstructExecutor({
 When querying endpoints that return line-oriented formats like N-Triples (e.g. QLever), enable `lineBuffer` to work around an [N3.js chunk-splitting bug](https://github.com/rdfjs/N3.js/issues/578) that causes intermittent parse errors on large responses:
 
 ```typescript
-const executor = new SparqlConstructExecutor({
+const reader = new SparqlConstructReader({
   query: 'CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }',
   lineBuffer: true,
 });
@@ -165,30 +165,30 @@ const executor = new SparqlConstructExecutor({
 SPARQL CONSTRUCT queries can produce duplicate triples — for example, constant triples (like `?dataset a edm:ProvidedCHO`) are emitted for every solution row. Enable `deduplicate` to remove duplicates inline on the stream using a string-based identity set (inspired by [Comunica's `distinctConstruct`](https://comunica.dev/docs/query/advanced/context/#14--distinct-construct)):
 
 ```typescript
-const executor = new SparqlConstructExecutor({
+const reader = new SparqlConstructReader({
   query: 'CONSTRUCT { ?s a edm:ProvidedCHO . ?s ?p ?o } WHERE { ?s ?p ?o }',
   deduplicate: true,
 });
 ```
 
-The dedup set is scoped to each `execute()` call, so memory stays bounded to the number of unique quads per batch. A standalone `deduplicateQuads()` function is also exported for use outside the executor.
+The dedup set is scoped to each `read()` call, so memory stays bounded to the number of unique quads per batch. A standalone `deduplicateQuads()` function is also exported for use outside the reader.
 
 ### Extending a stage with a quad transform
 
-Some logic is hard to express in pure SPARQL — cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling an executor’s output and firing follow-up queries. Rather than subclass `Executor`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one executor’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
+Some logic is hard to express in pure SPARQL — cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling a reader’s output and firing follow-up queries. Rather than subclass `Reader`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one reader’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
 
-A transform receives an `ExecutorContext` — the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per executor call, so **write it to accept being called more than once**: a global stage calls it once over the executor’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations — or keep the transform per-quad, where the number of calls makes no difference.
+A transform receives an `ReaderContext` — the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per reader call, so **write it to accept being called more than once**: a global stage calls it once over the reader’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations — or keep the transform per-quad, where the number of calls makes no difference.
 
 ```typescript
 import { DataFactory } from 'n3';
 import {
   Stage,
-  SparqlConstructExecutor,
+  SparqlConstructReader,
   type QuadTransform,
-  type ExecutorContext,
+  type ReaderContext,
 } from '@lde/pipeline';
 
-const cleanDates: QuadTransform<ExecutorContext> = async function* (quads) {
+const cleanDates: QuadTransform<ReaderContext> = async function* (quads) {
   for await (const quad of quads) {
     if (quad.object.termType === 'Literal' && isMessyDate(quad.object)) {
       yield DataFactory.quad(
@@ -207,8 +207,8 @@ const cleanDates: QuadTransform<ExecutorContext> = async function* (quads) {
 
 new Stage({
   name: 'dates',
-  executors: {
-    executor: await SparqlConstructExecutor.fromFile('dates.rq'),
+  readers: {
+    reader: await SparqlConstructReader.fromFile('dates.rq'),
     transform: cleanDates,
   },
 });
@@ -252,11 +252,11 @@ Transitions are forwarded to the `ProgressReporter` via `timeoutTightened` / `ti
 
 Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets).
 
-Timeouts live at the pipeline level — neither `SparqlConstructExecutor` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
+Timeouts live at the pipeline level — neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
 
 ### Validation
 
-Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all executors per batch**, not on individual quads or per-executor output. A batch produces a complete result set — a self-contained cluster of linked resources — that can be meaningfully matched against SHACL shapes. Even with a single executor, each batch is a complete unit; with multiple executors, shapes that reference triples from different executors are validated correctly.
+Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set — a self-contained cluster of linked resources — that can be meaningfully matched against SHACL shapes. Even with a single reader, each batch is a complete unit; with multiple readers, shapes that reference triples from different readers are validated correctly.
 
 Validating individual quads would be meaningless, since a single quad carries no structural context for shape matching. Validating the full pipeline output would also be problematic: because the pipeline streams results in batches, it doesn’t know where resource cluster boundaries fall. Batching the output could split a valid cluster across two batches, causing partial resources to fail validation even though the complete cluster is valid.
 
@@ -268,7 +268,7 @@ import { FileWriter } from '@lde/pipeline';
 
 new Stage({
   name: 'transform',
-  executors: await SparqlConstructExecutor.fromFile('transform.rq'),
+  readers: await SparqlConstructReader.fromFile('transform.rq'),
   validation: {
     validator: new ShaclValidator({
       shapesFile: './shapes.ttl',
@@ -295,10 +295,26 @@ After all stages for a dataset have run, the pipeline calls `validator.report(da
 
 ### Writer
 
-Writes generated quads to a destination:
+Writes generated quads to a destination. A `Writer` is transactional: each pipeline run opens one run on it (`openRun(context)`), writes every dataset through the resulting `RunWriter`, and ends with exactly one `commit()` (on success) or `abort(error)` (on failure). The run lifecycle is the home of destination-level concerns such as atomic swaps, deletion sweeps and cross-pod locks; the pipeline drives `openRun → write* → commit/abort` uniformly and never branches on the writer’s update mode. After a dataset’s stages complete, its `flush(dataset, outcome)` says whether the dataset succeeded, so a writer can gate destructive finalization (e.g. an In-place stale-document sweep) on `'success'`.
 
-- `SparqlUpdateWriter` — writes to a SPARQL endpoint via UPDATE queries
-- `FileWriter` — writes to local files
+- `SparqlUpdateWriter` — writes to a SPARQL endpoint via UPDATE queries; writes are visible as they land, so `commit`/`abort` are no-ops
+- `FileWriter` — writes to local files; each file is streamed to a sibling temp file and atomically renamed on flush, `commit` finalizes any files still open, and `abort` discards temp output
+
+The `RunContext` handed to `openRun` carries the run’s identity (`runId`, `startedAt`), the full selection (`selectedSources()`, complete by commit time – including datasets skipped as unchanged) and the pipeline’s provenance store, when configured.
+
+A destination without run-level state implements the same contract with no-op `commit`/`abort` – `SparqlUpdateWriter` shows the pattern:
+
+```typescript
+async openRun(): Promise<RunWriter> {
+  return {
+    write: async (dataset, quads) => {
+      // send quads somewhere
+    },
+    commit: () => Promise.resolve(),
+    abort: () => Promise.resolve(),
+  };
+}
+```
 
 ### Reporter
 
@@ -425,7 +441,7 @@ new Pipeline({
 import {
   Pipeline,
   Stage,
-  SparqlConstructExecutor,
+  SparqlConstructReader,
   SparqlItemSelector,
   SparqlUpdateWriter,
   ManualDatasetSelection,
@@ -439,7 +455,7 @@ const pipeline = new Pipeline({
       itemSelector: new SparqlItemSelector({
         query: 'SELECT DISTINCT ?class WHERE { ?s a ?class }',
       }),
-      executors: new SparqlConstructExecutor({
+      readers: new SparqlConstructReader({
         query:
           'CONSTRUCT { ?class a <http://example.org/Class> } WHERE { ?s a ?class }',
       }),

package/dist/pipeline.d.ts

@@ -27,7 +27,7 @@ export interface PipelinePlugin {
      * Transform the merged, post-stage quad stream before writing (extension
      * point 2: pipeline-wide, post-merge). The home of cross-cutting concerns
      * – provenance, namespace normalisation – that apply regardless of which
-     * executor produced a quad.
+     * reader produced a quad.
      */
     beforeStageWrite?: QuadTransform<BeforeStageWriteContext>;
 }
@@ -97,7 +97,7 @@ export declare class Pipeline {
     private reportValidators;
     private collectStages;
     /**
-     * The writer a stage's merged output is written through: the user writer
+     * The writer a stage's merged output is written through: the open run
      * wrapped with the plugins' {@link PipelinePlugin.beforeStageWrite}
      * transforms, carrying this `stage`'s identity so a transform can mint stable
      * per-`(dataset, stage)` IRIs rather than blank nodes.

package/dist/pipeline.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"pipeline.d.ts","sourceRoot":"","sources":["../src/pipeline.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,OAAO,EAAgB,MAAM,cAAc,CAAC;AAGrD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AACrD,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAChD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,oBAAoB,CAAC;AAEjD,OAAO,EACL,KAAK,oBAAoB,EAI1B,MAAM,4BAA4B,CAAC;AAKpC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,0BAA0B,CAAC;AACpE,OAAO,KAAK,EAEV,gBAAgB,EACjB,MAAM,uBAAuB,CAAC;AAG/B,OAAO,EAEL,KAAK,aAAa,EACnB,MAAM,2BAA2B,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED,wDAAwD;AACxD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;OAKG;IACH,gBAAgB,CAAC,EAAE,aAAa,CAAC,uBAAuB,CAAC,CAAC;CAC3D;AAED,MAAM,WAAW,eAAe;IAC9B,eAAe,EAAE,eAAe,CAAC;IACjC,MAAM,EAAE,KAAK,EAAE,CAAC;IAChB,OAAO,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC3B,OAAO,CAAC,EAAE,cAAc,EAAE,CAAC;IAC3B,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,QAAQ,CAAC,EAAE;QACT,mBAAmB,EAAE,mBAAmB,CAAC;QACzC,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;IACF;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,gBAAgB,GAAG,SAAS,gBAAgB,EAAE,CAAC;IAC1D;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,eAAe,CAAC;IAClC;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,aAAa,CAAC;CAC/B;AAsFD,qBAAa,QAAQ;IACnB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAClD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAU;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAS;IAChC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAyC;IAC3E,OAAO,CAAC,QAAQ,CAAC,oBAAoB,CAAuB;IAC5D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAA8B;IACxD,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmB;IAC7C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAsB;IACrD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAkB;IACnD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAS;gBAE9B,OAAO,EAAE,eAAe;IA+C9B,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;YAoBZ,cAAc;IA6M5B,+EAA+E;YACjE,aAAa;YAmBb,gBAAgB;IAW9B,OAAO,CAAE,aAAa;IAOtB;;;;;OAKG;IACH,OAAO,CAAC,WAAW;IAMnB;;;;;;;;OAQG;IACH,OAAO,CAAC,0BAA0B;IA4BlC;;;;;OAKG;YACW,SAAS;IA8BvB;;;OAGG;YACW,QAAQ;IA0CtB,2EAA2E;YAC7D,eAAe;YAqBf,QAAQ;YA+DP,SAAS;CAczB"}
+{"version":3,"file":"pipeline.d.ts","sourceRoot":"","sources":["../src/pipeline.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,OAAO,EAAgB,MAAM,cAAc,CAAC;AAGrD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AACrD,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAC;AACnC,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,YAAY,CAAC;AAChD,OAAO,KAAK,EAKV,MAAM,EACP,MAAM,oBAAoB,CAAC;AAE5B,OAAO,EACL,KAAK,oBAAoB,EAI1B,MAAM,4BAA4B,CAAC;AAKpC,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,uBAAuB,CAAC;AAY7D,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,0BAA0B,CAAC;AACpE,OAAO,KAAK,EAEV,gBAAgB,EACjB,MAAM,uBAAuB,CAAC;AAG/B,OAAO,EAEL,KAAK,aAAa,EACnB,MAAM,2BAA2B,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,uBAAuB;IACtC,OAAO,EAAE,OAAO,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED,wDAAwD;AACxD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb;;;;;OAKG;IACH,gBAAgB,CAAC,EAAE,aAAa,CAAC,uBAAuB,CAAC,CAAC;CAC3D;AAED,MAAM,WAAW,eAAe;IAC9B,eAAe,EAAE,eAAe,CAAC;IACjC,MAAM,EAAE,KAAK,EAAE,CAAC;IAChB,OAAO,EAAE,MAAM,GAAG,MAAM,EAAE,CAAC;IAC3B,OAAO,CAAC,EAAE,cAAc,EAAE,CAAC;IAC3B,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,oBAAoB,CAAC,EAAE,oBAAoB,CAAC;IAC5C,QAAQ,CAAC,EAAE;QACT,mBAAmB,EAAE,mBAAmB,CAAC;QACzC,SAAS,EAAE,MAAM,CAAC;KACnB,CAAC;IACF;;;;;OAKG;IACH,QAAQ,CAAC,EAAE,gBAAgB,GAAG,SAAS,gBAAgB,EAAE,CAAC;IAC1D;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,eAAe,CAAC;IAClC;;;;;;OAMG;IACH,eAAe,CAAC,EAAE,MAAM,CAAC;IACzB;;;;;;;;;OASG;IACH,OAAO,CAAC,EAAE,MAAM,aAAa,CAAC;CAC/B;AAqID,qBAAa,QAAQ;IACnB,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;IAC9B,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAkB;IAClD,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAU;IACjC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAS;IAChC,OAAO,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAyC;IAC3E,OAAO,CAAC,QAAQ,CAAC,oBAAoB,CAAuB;IAC5D,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAA8B;IACxD,OAAO,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAmB;IAC7C,OAAO,CAAC,QAAQ,CAAC,cAAc,CAAsB;IACrD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAkB;IACnD,OAAO,CAAC,QAAQ,CAAC,eAAe,CAAC,CAAS;gBAE9B,OAAO,EAAE,eAAe;IA+C9B,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;YAyCZ,cAAc;IAqN5B,+EAA+E;YACjE,aAAa;YAmBb,gBAAgB;IAW9B,OAAO,CAAE,aAAa;IAOtB;;;;;OAKG;IACH,OAAO,CAAC,WAAW;IAMnB;;;;;;;;OAQG;IACH,OAAO,CAAC,0BAA0B;IA4BlC;;;;;OAKG;YACW,SAAS;IAuCvB;;;OAGG;YACW,QAAQ;IA0CtB,2EAA2E;YAC7D,eAAe;YAqBf,QAAQ;YAuEP,SAAS;CAczB"}

package/dist/pipeline.js

@@ -1,4 +1,5 @@
 import { createReadStream } from 'node:fs';
+import { randomUUID } from 'node:crypto';
 import { StreamParser } from 'n3';
 import { FileWriter } from './writer/fileWriter.js';
 import { NoDistributionAvailable, ResolvedDistribution, } from './distribution/resolver.js';
@@ -8,7 +9,7 @@ import { shouldReprocess } from './provenance/reprocessDecision.js';
 import { NetworkError, SparqlProbeResult, } from '@lde/distribution-probe';
 import { ImportSuccessful } from '@lde/sparql-importer';
 import { importOutcomeToVerdict, probeResultToVerdict, } from '@lde/distribution-health';
-import { NotSupported } from './sparql/executor.js';
+import { NotSupported } from './sparql/reader.js';
 import { combineReporters } from './combineReporters.js';
 import { ConstantTimeoutPolicy, } from './sparql/timeoutPolicy.js';
 /**
@@ -54,17 +55,57 @@ class FanOutWriter {
     constructor(writers) {
         this.writers = writers;
     }
+    async openRun(context) {
+        // Opening a writer's run can have side effects (a cross-pod lock, a fresh
+        // collection). If one writer opens but a sibling then fails, the pipeline
+        // never gets a RunWriter to abort, so roll the opened ones back here
+        // before rethrowing — otherwise their locks and collections leak.
+        const results = await Promise.allSettled(this.writers.map((writer) => writer.openRun(context)));
+        const opened = results.flatMap((result) => result.status === 'fulfilled' ? [result.value] : []);
+        const failure = results.find((result) => result.status === 'rejected');
+        if (failure) {
+            await Promise.allSettled(opened.map((run) => run.abort(failure.reason)));
+            throw failure.reason;
+        }
+        return new FanOutRunWriter(opened);
+    }
+}
+/**
+ * Run independent branch operations concurrently, giving every branch its
+ * chance even when one fails; the first failure is rethrown afterwards.
+ */
+async function settleBranches(tasks) {
+    const results = await Promise.allSettled(tasks);
+    const failed = results.find((result) => result.status === 'rejected');
+    if (failed)
+        throw failed.reason;
+}
+class FanOutRunWriter {
+    runs;
+    constructor(runs) {
+        this.runs = runs;
+    }
     async write(dataset, quads) {
-        const branches = tee(quads, this.writers.length);
-        await Promise.all(this.writers.map((writer, index) => writer.write(dataset, branches[index])));
+        const branches = tee(quads, this.runs.length);
+        await Promise.all(this.runs.map((run, index) => run.write(dataset, branches[index])));
     }
-    async flush(dataset) {
-        for (const w of this.writers)
-            await w.flush?.(dataset);
+    async flush(dataset, outcome) {
+        for (const run of this.runs)
+            await run.flush?.(dataset, outcome);
     }
     async reset(dataset) {
-        for (const w of this.writers)
-            await w.reset?.(dataset);
+        for (const run of this.runs)
+            await run.reset?.(dataset);
+    }
+    async commit() {
+        // Destinations are independent, so their commits (alias swaps, sweeps –
+        // the slowest part of a run) need not queue behind each other.
+        await settleBranches(this.runs.map((run) => run.commit()));
+    }
+    async abort(error) {
+        // Abort every branch even when one abort fails: each destination must get
+        // its chance to clean up.
+        await settleBranches(this.runs.map((run) => run.abort(error)));
     }
 }
 class TransformWriter {
@@ -134,8 +175,28 @@ export class Pipeline {
         const selectStart = Date.now();
         const datasets = await this.datasetSelector.select();
         this.reporter?.datasetsSelected?.(datasets.total, Date.now() - selectStart);
-        for await (const dataset of datasets) {
-            await this.processDataset(dataset);
+        // The run transaction: one openRun → write* → commit/abort per pipeline
+        // run. `selectedSources` accumulates every selected dataset – including
+        // ones skipped as unchanged – so a writer's commit can sweep by registry
+        // membership.
+        const selectedSources = [];
+        const context = {
+            runId: randomUUID(),
+            startedAt: new Date().toISOString(),
+            selectedSources: () => selectedSources,
+            provenance: this.provenanceStore,
+        };
+        const runWriter = await this.writer.openRun(context);
+        try {
+            for await (const dataset of datasets) {
+                selectedSources.push(dataset.iri.toString());
+                await this.processDataset(dataset, runWriter, context);
+            }
+            await runWriter.commit();
+        }
+        catch (error) {
+            await runWriter.abort(error);
+            throw error;
         }
         const finalMemory = process.memoryUsage();
         this.reporter?.pipelineComplete?.({
@@ -144,7 +205,7 @@ export class Pipeline {
             heapUsedBytes: finalMemory.heapUsed,
         });
     }
-    async processDataset(dataset) {
+    async processDataset(dataset, runWriter, context) {
         this.reporter?.datasetStart?.(dataset);
         // Probe phase: gather probe results and the source-to-be, without importing.
         let probed;
@@ -232,7 +293,7 @@ export class Pipeline {
         });
         let stageFailed = false;
         try {
-            stageFailed = await this.runStages(dataset, resolved.distribution, timeout);
+            stageFailed = await this.runStages(dataset, resolved.distribution, timeout, runWriter, context);
             // Reactive fallback: an endpoint that passed probing but could not serve
             // the analysis stages is empirically incapable. Switch to the dataset’s
             // data dump and re-run all stages locally, discarding the
@@ -265,8 +326,8 @@ export class Pipeline {
                         this.reportSelectedDistribution(dataset, fallback, fingerprint);
                         // Discard the endpoint-sourced partial output before the re-run so
                         // the dump-sourced stats replace it rather than appending to it.
-                        await this.writer.reset?.(dataset);
-                        stageFailed = await this.runStages(dataset, fallback.distribution, timeout);
+                        await runWriter.reset?.(dataset);
+                        stageFailed = await this.runStages(dataset, fallback.distribution, timeout, runWriter, context);
                     }
                     else if (fallback.importFailed) {
                         // A failed dump import is a deep validity verdict on that dump –
@@ -284,7 +345,7 @@ export class Pipeline {
             await this.distributionResolver.cleanup?.();
             unsubscribe?.();
         }
-        await this.writer.flush?.(dataset);
+        await runWriter.flush?.(dataset, stageFailed ? 'failed' : 'success');
         await this.reportValidators(dataset);
         // A dataset whose stages threw produced incomplete output; record it as
         // ‘failed’ rather than freezing a broken result under a ‘success’ record.
@@ -331,15 +392,15 @@ export class Pipeline {
         }
     }
     /**
-     * The writer a stage's merged output is written through: the user writer
+     * The writer a stage's merged output is written through: the open run
      * wrapped with the plugins' {@link PipelinePlugin.beforeStageWrite}
      * transforms, carrying this `stage`'s identity so a transform can mint stable
      * per-`(dataset, stage)` IRIs rather than blank nodes.
      */
-    stageWriter(stage) {
+    stageWriter(runWriter, stage) {
         return this.beforeStageWrite
-            ? new TransformWriter(this.writer, this.beforeStageWrite, stage)
-            : this.writer;
+            ? new TransformWriter(runWriter, this.beforeStageWrite, stage)
+            : runWriter;
     }
     /**
      * Report a resolved distribution as the dataset's selected source, plus its
@@ -362,15 +423,15 @@ export class Pipeline {
      * whether any stage hard-failed – the signal the reactive dump fallback
      * reacts to.
      */
-    async runStages(dataset, distribution, timeout) {
+    async runStages(dataset, distribution, timeout, runWriter, context) {
         let stageFailed = false;
         for (const stage of this.stages) {
             try {
                 if (stage.stages.length > 0) {
-                    await this.runChain(dataset, distribution, stage, timeout);
+                    await this.runChain(dataset, distribution, stage, runWriter, context, timeout);
                 }
                 else {
-                    await this.runStage(dataset, distribution, stage, this.stageWriter(stage.name), timeout);
+                    await this.runStage(dataset, distribution, stage, this.stageWriter(runWriter, stage.name), timeout);
                 }
             }
             catch (error) {
@@ -384,7 +445,7 @@ export class Pipeline {
      * Run a stage with reporting and return whether it was supported.
      * Returns `true` if the stage produced results, `false` if NotSupported.
      */
-    async runStage(dataset, distribution, stage, writer = this.writer, timeout) {
+    async runStage(dataset, distribution, stage, writer, timeout) {
         this.reporter?.stageStart?.(stage.name);
         const stageStart = Date.now();
         let itemsProcessed = 0;
@@ -421,34 +482,46 @@ export class Pipeline {
             throw new Error(`Stage '${stage.name}' returned NotSupported in chained mode`);
         }
     }
-    async runChain(dataset, distribution, stage, timeout) {
+    async runChain(dataset, distribution, stage, runWriter, context, timeout) {
         const { stageOutputResolver, outputDir } = this.chaining;
         const outputFiles = [];
-        try {
-            // 1. Run parent stage → FileWriter.
-            const parentWriter = new FileWriter({
-                outputDir: `${outputDir}/${stage.name}`,
+        // Run one chained stage into its scratch FileWriter and flush it, so the
+        // output file exists on its final path before the resolver reads it. The
+        // scratch run is bracketed like any other: committed on success, aborted
+        // on failure so no half-written temp file is left behind.
+        const runScratchStage = async (chainedStage, stageDistribution) => {
+            const scratchWriter = new FileWriter({
+                outputDir: `${outputDir}/${chainedStage.name}`,
                 format: 'n-triples',
             });
-            await this.runChainedStage(dataset, distribution, stage, parentWriter, timeout);
-            outputFiles.push(parentWriter.getOutputPath(dataset));
+            const scratchRun = await scratchWriter.openRun(context);
+            try {
+                await this.runChainedStage(dataset, stageDistribution, chainedStage, scratchRun, timeout);
+                await scratchRun.flush(dataset, 'success');
+                await scratchRun.commit();
+            }
+            catch (error) {
+                await scratchRun.abort(error);
+                throw error;
+            }
+            return scratchWriter.getOutputPath(dataset);
+        };
+        try {
+            // 1. Run parent stage → scratch FileWriter.
+            const parentOutput = await runScratchStage(stage, distribution);
+            outputFiles.push(parentOutput);
             // 2. Chain through children.
-            let currentDistribution = await stageOutputResolver.resolve(parentWriter.getOutputPath(dataset));
+            let currentDistribution = await stageOutputResolver.resolve(parentOutput);
             for (let i = 0; i < stage.stages.length; i++) {
-                const child = stage.stages[i];
-                const childWriter = new FileWriter({
-                    outputDir: `${outputDir}/${child.name}`,
-                    format: 'n-triples',
-                });
-                await this.runChainedStage(dataset, currentDistribution, child, childWriter, timeout);
-                outputFiles.push(childWriter.getOutputPath(dataset));
+                const childOutput = await runScratchStage(stage.stages[i], currentDistribution);
+                outputFiles.push(childOutput);
                 if (i < stage.stages.length - 1) {
-                    currentDistribution = await stageOutputResolver.resolve(childWriter.getOutputPath(dataset));
+                    currentDistribution = await stageOutputResolver.resolve(childOutput);
                 }
             }
-            // 3. Concatenate all output files → user writer, applying the plugins'
+            // 3. Concatenate all output files → the open run, applying the plugins'
             // beforeStageWrite transforms once for the chain under the parent stage.
-            await this.stageWriter(stage.name).write(dataset, this.readFiles(outputFiles));
+            await this.stageWriter(runWriter, stage.name).write(dataset, this.readFiles(outputFiles));
         }
         finally {
             await stageOutputResolver.cleanup();

package/dist/provenance/fileLoadedSparqlProvenanceStore.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"fileLoadedSparqlProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileLoadedSparqlProvenanceStore.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAI9D,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAgBlD,MAAM,WAAW,sCAAsC;IACrD,wEAAwE;IACxE,aAAa,EAAE,GAAG,CAAC;IACnB;;;OAGG;IACH,WAAW,EAAE,GAAG,CAAC;IACjB;;;;OAIG;IACH,SAAS,EAAE,MAAM,CAAC;IAClB;;;OAGG;IACH,OAAO,CAAC,EAAE,qBAAqB,CAAC;CACjC;AAED;;;;;;;;GAQG;AACH,qBAAa,+BAAgC,YAAW,eAAe;IACrE,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAM;IACpC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAM;IAClC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAa;IACpC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAwB;gBAEpC,OAAO,EAAE,sCAAsC;IAWrD,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAoB5D,OAAO,CAAC,WAAW;IAmBb,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YAMpD,OAAO;CAsBvB"}
+{"version":3,"file":"fileLoadedSparqlProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileLoadedSparqlProvenanceStore.ts"],"names":[],"mappings":"AAGA,OAAO,EAAE,qBAAqB,EAAE,MAAM,uBAAuB,CAAC;AAI9D,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAgBlD,MAAM,WAAW,sCAAsC;IACrD,wEAAwE;IACxE,aAAa,EAAE,GAAG,CAAC;IACnB;;;OAGG;IACH,WAAW,EAAE,GAAG,CAAC;IACjB;;;;OAIG;IACH,SAAS,EAAE,MAAM,CAAC;IAClB;;;OAGG;IACH,OAAO,CAAC,EAAE,qBAAqB,CAAC;CACjC;AAED;;;;;;;;GAQG;AACH,qBAAa,+BAAgC,YAAW,eAAe;IACrE,OAAO,CAAC,QAAQ,CAAC,aAAa,CAAM;IACpC,OAAO,CAAC,QAAQ,CAAC,WAAW,CAAM;IAClC,OAAO,CAAC,QAAQ,CAAC,MAAM,CAAa;IACpC,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAwB;gBAEpC,OAAO,EAAE,sCAAsC;IAWrD,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAoB5D,OAAO,CAAC,WAAW;IAmBb,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YASpD,OAAO;CAsBvB"}

package/dist/provenance/fileLoadedSparqlProvenanceStore.js

@@ -68,8 +68,11 @@ export class FileLoadedSparqlProvenanceStore {
     }
     async set(datasetUri, record) {
         const dataset = new Dataset({ iri: datasetUri, distributions: [] });
-        await this.writer.write(dataset, this.toQuads(datasetUri, record));
-        await this.writer.flush(dataset);
+        // Each record is complete in itself, so it gets its own mini run:
+        // write + flush materializes the record file atomically.
+        const run = await this.writer.openRun();
+        await run.write(dataset, this.toQuads(datasetUri, record));
+        await run.flush(dataset);
     }
     async *toQuads(datasetUri, record) {
         const subject = namedNode(datasetUri.toString());

package/dist/sparql/index.d.ts

@@ -1,4 +1,4 @@
-export { deduplicateQuads, SparqlConstructExecutor, LineBufferTransform, NotSupported, readQueryFile, type ExecuteOptions, type Executor, type SparqlConstructExecutorOptions, type VariableBindings, } from './executor.js';
+export { deduplicateQuads, SparqlConstructReader, LineBufferTransform, NotSupported, readQueryFile, type ReadOptions, type Reader, type SparqlConstructReaderOptions, type VariableBindings, } from './reader.js';
 export { SparqlItemSelector, type SparqlItemSelectorOptions, } from './selector.js';
 export { injectValues } from './values.js';
 export { withDefaultGraph } from './graph.js';

package/dist/sparql/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/sparql/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,gBAAgB,EAChB,uBAAuB,EACvB,mBAAmB,EACnB,YAAY,EACZ,aAAa,EACb,KAAK,cAAc,EACnB,KAAK,QAAQ,EACb,KAAK,8BAA8B,EACnC,KAAK,gBAAgB,GACtB,MAAM,eAAe,CAAC;AACvB,OAAO,EACL,kBAAkB,EAClB,KAAK,yBAAyB,GAC/B,MAAM,eAAe,CAAC;AAEvB,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAE3C,OAAO,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAE9C,OAAO,EACL,qBAAqB,EACrB,qBAAqB,EACrB,qBAAqB,EACrB,qBAAqB,EACrB,KAAK,4BAA4B,EACjC,KAAK,mBAAmB,EACxB,KAAK,oBAAoB,EACzB,KAAK,cAAc,EACnB,KAAK,aAAa,EAClB,KAAK,qBAAqB,EAC1B,KAAK,sBAAsB,GAC5B,MAAM,oBAAoB,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/sparql/index.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,gBAAgB,EAChB,qBAAqB,EACrB,mBAAmB,EACnB,YAAY,EACZ,aAAa,EACb,KAAK,WAAW,EAChB,KAAK,MAAM,EACX,KAAK,4BAA4B,EACjC,KAAK,gBAAgB,GACtB,MAAM,aAAa,CAAC;AACrB,OAAO,EACL,kBAAkB,EAClB,KAAK,yBAAyB,GAC/B,MAAM,eAAe,CAAC;AAEvB,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAE3C,OAAO,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAE9C,OAAO,EACL,qBAAqB,EACrB,qBAAqB,EACrB,qBAAqB,EACrB,qBAAqB,EACrB,KAAK,4BAA4B,EACjC,KAAK,mBAAmB,EACxB,KAAK,oBAAoB,EACzB,KAAK,cAAc,EACnB,KAAK,aAAa,EAClB,KAAK,qBAAqB,EAC1B,KAAK,sBAAsB,GAC5B,MAAM,oBAAoB,CAAC"}

package/dist/sparql/index.js

@@ -1,4 +1,4 @@
-export { deduplicateQuads, SparqlConstructExecutor, LineBufferTransform, NotSupported, readQueryFile, } from './executor.js';
+export { deduplicateQuads, SparqlConstructReader, LineBufferTransform, NotSupported, readQueryFile, } from './reader.js';
 export { SparqlItemSelector, } from './selector.js';
 export { injectValues } from './values.js';
 export { withDefaultGraph } from './graph.js';