@lde/pipeline 0.34.2 → 0.34.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  A framework for transforming large RDF datasets, primarily using [SPARQL](https://www.w3.org/TR/sparql11-query/) queries with TypeScript for the parts that are hard to express in SPARQL alone.
4
4
 
5
- - **SPARQL-native.** Data transformations are plain SPARQL query files — portable, transparent, testable and version-controlled.
5
+ - **SPARQL-native.** Data transformations are plain SPARQL query files – portable, transparent, testable and version-controlled.
6
6
  - **Composable.** Executors are an interface: wrap a SPARQL reader with custom TypeScript to handle edge cases like date parsing or string normalisation (see [Reader](#reader)).
7
7
  - **Extensible.** A plugin system lets packages like [@lde/pipeline-void](../pipeline-void) (or your own plugins) hook into the pipeline lifecycle.
8
8
 
@@ -44,9 +44,9 @@ const resolver = new ImportResolver(new SparqlDistributionResolver(), {
44
44
  });
45
45
  ```
46
46
 
47
- - `'sparql'` (default) — use the dataset’s own SPARQL endpoint when one is available; import a data dump only when no endpoint responds.
48
- - `'sparqlWithImportFallback'` — like `'sparql'`, but also fall back to the data dump when the endpoint passes probing yet a stage fails against it at runtime. The pipeline discards the endpoint-sourced partial output and re-runs all stages against the import. Use this when endpoints are present but unreliable for heavy aggregate queries.
49
- - `'import'` — always import the data dump, even when a working endpoint is advertised.
47
+ - `'sparql'` (default) – use the dataset’s own SPARQL endpoint when one is available; import a data dump only when no endpoint responds.
48
+ - `'sparqlWithImportFallback'` – like `'sparql'`, but also fall back to the data dump when the endpoint passes probing yet a stage fails against it at runtime. The pipeline discards the endpoint-sourced partial output and re-runs all stages against the import. Use this when endpoints are present but unreliable for heavy aggregate queries.
49
+ - `'import'` – always import the data dump, even when a working endpoint is advertised.
50
50
 
51
51
  ### Stage
52
52
 
@@ -90,7 +90,7 @@ new Stage({
90
90
 
91
91
  `expectsOutput` (default: `false`) marks a stage whose query must yield at least one quad. A supported stage that produces none is then treated as a hard failure rather than a legitimately empty result.
92
92
 
93
- Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always return exactly one row — so zero output can only mean the endpoint truncated or aborted the response (e.g. a timeout surfaced as an empty `HTTP 200`). The failure flows through like any other hard stage failure, triggering the [reactive dump fallback](#distribution-resolver) when `strategy: 'sparqlWithImportFallback'` is configured. Leave it `false` for stages that may legitimately be empty, such as class or property partitions of a dataset that lacks that structure.
93
+ Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always return exactly one row – so zero output can only mean the endpoint truncated or aborted the response (e.g. a timeout surfaced as an empty `HTTP 200`). The failure flows through like any other hard stage failure, triggering the [reactive dump fallback](#distribution-resolver) when `strategy: 'sparqlWithImportFallback'` is configured. Leave it `false` for stages that may legitimately be empty, such as class or property partitions of a dataset that lacks that structure.
94
94
 
95
95
  ### Item Selector
96
96
 
@@ -115,7 +115,7 @@ new SparqlItemSelector({
115
115
 
116
116
  #### Capping total results with `maxResults`
117
117
 
118
- By default, `SparqlItemSelector` paginates through **all** matching rows: any `LIMIT` clause in the query is interpreted as the page size, then it walks pages with `OFFSET` until the source is exhausted. To cap the total bindings yielded across all pages — for sampling, testing, prototyping, or just safety — set `maxResults`:
118
+ By default, `SparqlItemSelector` paginates through **all** matching rows: any `LIMIT` clause in the query is interpreted as the page size, then it walks pages with `OFFSET` until the source is exhausted. To cap the total bindings yielded across all pages – for sampling, testing, prototyping, or just safety – set `maxResults`:
119
119
 
120
120
  ```typescript
121
121
  new SparqlItemSelector({
@@ -126,7 +126,7 @@ new SparqlItemSelector({
126
126
 
127
127
  When `maxResults` is set:
128
128
 
129
- - Pagination stops as soon as `maxResults` bindings have been yielded — no wasted page request after the cap is hit.
129
+ - Pagination stops as soon as `maxResults` bindings have been yielded – no wasted page request after the cap is hit.
130
130
  - The last (partial) page's `LIMIT` is shrunk to the remaining cap so the endpoint doesn't over-fetch on the remainder (e.g. with `maxResults: 85` and `pageSize: 10`, the 9th page request is `LIMIT 5`, not `LIMIT 10`).
131
131
  - The first page uses the configured page size as-is; `maxResults` and page size stay orthogonal. If `maxResults < pageSize`, the first page may return a few rows that aren't yielded.
132
132
  - `maxResults: 0` is a valid no-op; the selector yields nothing without issuing any SPARQL request.
@@ -162,7 +162,7 @@ const reader = new SparqlConstructReader({
162
162
  });
163
163
  ```
164
164
 
165
- SPARQL CONSTRUCT queries can produce duplicate triples — for example, constant triples (like `?dataset a edm:ProvidedCHO`) are emitted for every solution row. Enable `deduplicate` to remove duplicates inline on the stream using a string-based identity set (inspired by [Comunica's `distinctConstruct`](https://comunica.dev/docs/query/advanced/context/#14--distinct-construct)):
165
+ SPARQL CONSTRUCT queries can produce duplicate triples – for example, constant triples (like `?dataset a edm:ProvidedCHO`) are emitted for every solution row. Enable `deduplicate` to remove duplicates inline on the stream using a string-based identity set (inspired by [Comunica's `distinctConstruct`](https://comunica.dev/docs/query/advanced/context/#14--distinct-construct)):
166
166
 
167
167
  ```typescript
168
168
  const reader = new SparqlConstructReader({
@@ -175,9 +175,9 @@ The dedup set is scoped to each `read()` call, so memory stays bounded to the nu
175
175
 
176
176
  ### Extending a stage with a quad transform
177
177
 
178
- Some logic is hard to express in pure SPARQL — cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling a reader’s output and firing follow-up queries. Rather than subclass `Reader`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one reader’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
178
+ Some logic is hard to express in pure SPARQL – cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling a reader’s output and firing follow-up queries. Rather than subclass `Reader`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one reader’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
179
179
 
180
- A transform receives an `ReaderContext` — the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per reader call, so **write it to accept being called more than once**: a global stage calls it once over the reader’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations — or keep the transform per-quad, where the number of calls makes no difference.
180
+ A transform receives an `ReaderContext` – the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per reader call, so **write it to accept being called more than once**: a global stage calls it once over the reader’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations – or keep the transform per-quad, where the number of calls makes no difference.
181
181
 
182
182
  ```typescript
183
183
  import { DataFactory } from 'n3';
@@ -218,7 +218,7 @@ new Stage({
218
218
 
219
219
  #### Adaptive timeouts
220
220
 
221
- By default, every SPARQL request uses the same 5-minute budget. When a pipeline runs against many third-party endpoints, that fixed budget can cost ~80 minutes on a single dataset whose endpoint times out repeatedly on heavy queries — light stages on the same endpoint then sit behind the heavy ones that will never succeed.
221
+ By default, every SPARQL request uses the same 5-minute budget. When a pipeline runs against many third-party endpoints, that fixed budget can cost ~80 minutes on a single dataset whose endpoint times out repeatedly on heavy queries – light stages on the same endpoint then sit behind the heavy ones that will never succeed.
222
222
 
223
223
  A `TimeoutPolicy` decides the budget for each SPARQL request and observes the outcome. Two are built in:
224
224
 
@@ -252,11 +252,11 @@ Transitions are forwarded to the `ProgressReporter` via `timeoutTightened` / `ti
252
252
 
253
253
  Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets).
254
254
 
255
- Timeouts live at the pipeline level — neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
255
+ Timeouts live at the pipeline level – neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
256
256
 
257
257
  ### Validation
258
258
 
259
- Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set — a self-contained cluster of linked resources — that can be meaningfully matched against SHACL shapes. Even with a single reader, each batch is a complete unit; with multiple readers, shapes that reference triples from different readers are validated correctly.
259
+ Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set – a self-contained cluster of linked resources – that can be meaningfully matched against SHACL shapes. Even with a single reader, each batch is a complete unit; with multiple readers, shapes that reference triples from different readers are validated correctly.
260
260
 
261
261
  Validating individual quads would be meaningless, since a single quad carries no structural context for shape matching. Validating the full pipeline output would also be problematic: because the pipeline streams results in batches, it doesn’t know where resource cluster boundaries fall. Batching the output could split a valid cluster across two batches, causing partial resources to fail validation even though the complete cluster is valid.
262
262
 
@@ -291,14 +291,14 @@ new Stage({
291
291
 
292
292
  #### Per-dataset reporting
293
293
 
294
- After all stages for a dataset have run, the pipeline calls `validator.report(dataset)` once for each distinct validator attached to any stage and emits a `datasetValidated(dataset, report)` event on the reporter. The call happens **regardless of whether any stage actually invoked `validate()`** — for SHACL that means a dataset whose stages produced no input typically reports `quadsValidated: 0` and `conforms: true` (the SHACL vacuous-truth default). Consumers that want to distinguish ‘not tested’ from ‘tested and passed’ can read `quadsValidated`.
294
+ After all stages for a dataset have run, the pipeline calls `validator.report(dataset)` once for each distinct validator attached to any stage and emits a `datasetValidated(dataset, report)` event on the reporter. The call happens **regardless of whether any stage actually invoked `validate()`** – for SHACL that means a dataset whose stages produced no input typically reports `quadsValidated: 0` and `conforms: true` (the SHACL vacuous-truth default). Consumers that want to distinguish ‘not tested’ from ‘tested and passed’ can read `quadsValidated`.
295
295
 
296
296
  ### Writer
297
297
 
298
298
  Writes generated quads to a destination. A `Writer` is transactional: each pipeline run opens one run on it (`openRun(context)`), writes every dataset through the resulting `RunWriter`, and ends with exactly one `commit()` (on success) or `abort(error)` (on failure). The run lifecycle is the home of destination-level concerns such as atomic swaps, deletion sweeps and cross-pod locks; the pipeline drives `openRun → write* → commit/abort` uniformly and never branches on the writer’s update mode. After a dataset’s stages complete, its `flush(dataset, outcome)` says whether the dataset succeeded, so a writer can gate destructive finalization (e.g. an In-place stale-document sweep) on `'success'`.
299
299
 
300
- - `SparqlUpdateWriter` — writes to a SPARQL endpoint via UPDATE queries; writes are visible as they land, so `commit`/`abort` are no-ops
301
- - `FileWriter` — writes to local files; each file is streamed to a sibling temp file and atomically renamed on flush, `commit` finalizes any files still open, and `abort` discards temp output. In `n-quads` format it skolemises blank nodes to dataset-scoped IRIs, because such files are typically `cat`-indexed into one served graph where document-scoped blank-node labels from different datasets would otherwise fuse (see `blankNodes`/`failOnBlankNodes` and ldelements/lde#474); Turtle/N-Triples output is left unchanged
300
+ - `SparqlUpdateWriter` – writes to a SPARQL endpoint via UPDATE queries; writes are visible as they land, so `commit`/`abort` are no-ops
301
+ - `FileWriter` – writes to local files; each file is streamed to a sibling temp file and atomically renamed on flush, `commit` finalizes any files still open, and `abort` discards temp output. In `n-quads` format it skolemises blank nodes to dataset-scoped IRIs, because such files are typically `cat`-indexed into one served graph where document-scoped blank-node labels from different datasets would otherwise fuse (see `blankNodes`/`failOnBlankNodes` and ldelements/lde#474); Turtle/N-Triples output is left unchanged
302
302
 
303
303
  The `RunContext` handed to `openRun` carries the run’s identity (`runId`, `startedAt`), the full selection (`selectedSources()`, complete by commit time – including datasets skipped as unchanged) and the pipeline’s provenance store, when configured.
304
304
 
@@ -320,7 +320,7 @@ async openRun(): Promise<RunWriter> {
320
320
 
321
321
  A `ProgressReporter` observes the run, receiving lifecycle events such as `pipelineStart`, `stageComplete`, `datasetValidated` and `pipelineComplete`. Every method is optional, so a reporter implements only the events it cares about.
322
322
 
323
- Pass a single reporter, or an array to have several observe the same run — for example a console reporter alongside one that collects validation verdicts:
323
+ Pass a single reporter, or an array to have several observe the same run – for example a console reporter alongside one that collects validation verdicts:
324
324
 
325
325
  ```typescript
326
326
  new Pipeline({
@@ -342,11 +342,11 @@ interface ProvenanceStore {
342
342
  }
343
343
  ```
344
344
 
345
- A `ProcessingRecord` holds the two opaque change fields — `sourceFingerprint` (derived automatically from source metadata) and `pipelineVersion` (consumer-declared) — plus `generatedAt` and a `status` of `'success'` or `'failed'`. The two change fields are compared only for equality, never parsed or ordered.
345
+ A `ProcessingRecord` holds the two opaque change fields – `sourceFingerprint` (derived automatically from source metadata) and `pipelineVersion` (consumer-declared) – plus `generatedAt` and a `status` of `'success'` or `'failed'`. The two change fields are compared only for equality, never parsed or ordered.
346
346
 
347
347
  #### `FileLoadedSparqlProvenanceStore`
348
348
 
349
- The reference implementation targets a triplestore that is served read-only and rebuilt by bulk-loading files (e.g. [QLever](https://github.com/ad-freiburg/qlever)). It reads through SPARQL queries against the live endpoint, and writes records as files for the next bulk-load — because the endpoint accepts no SPARQL UPDATE.
349
+ The reference implementation targets a triplestore that is served read-only and rebuilt by bulk-loading files (e.g. [QLever](https://github.com/ad-freiburg/qlever)). It reads through SPARQL queries against the live endpoint, and writes records as files for the next bulk-load – because the endpoint accepts no SPARQL UPDATE.
350
350
 
351
351
  ```typescript
352
352
  import { FileLoadedSparqlProvenanceStore } from '@lde/pipeline';
@@ -361,7 +361,19 @@ const store = new FileLoadedSparqlProvenanceStore({
361
361
  - `get` runs a named-graph-scoped SPARQL `SELECT` against `queryEndpoint`, reading the records a previous run loaded.
362
362
  - `set` writes one flat [PROV-O](https://www.w3.org/TR/prov-o/) N-Quads file per dataset into `outputDir`, in the pipeline-scoped named graph, to be bulk-loaded after the run.
363
363
 
364
- Each record is stored as flat PROV-O on the dataset entity — `prov:generatedAtTime` plus `sourceFingerprint`, `pipelineVersion` and `status` under the `https://w3id.org/lde/provenance#` namespace. Scoping every record by `pipelineIri` (used as the named graph) lets multiple pipelines share one triplestore without colliding.
364
+ Each record is stored as flat PROV-O on the dataset entity – `prov:generatedAtTime` plus `sourceFingerprint`, `pipelineVersion` and `status` under the `https://w3id.org/lde/provenance#` namespace. Scoping every record by `pipelineIri` (used as the named graph) lets multiple pipelines share one triplestore without colliding.
365
+
366
+ #### `FileProvenanceStore`
367
+
368
+ For pipelines that run without a triplestore, `FileProvenanceStore` persists all records to a single JSON file, keyed by dataset URI:
369
+
370
+ ```typescript
371
+ import { FileProvenanceStore } from '@lde/pipeline';
372
+
373
+ const store = new FileProvenanceStore({ path: './state/provenance.json' });
374
+ ```
375
+
376
+ Writes are atomic (temp file + rename), so a run killed mid-write cannot corrupt the next run’s skip decisions. Safe for a single writer: concurrent pipeline processes writing the same file lose each other’s updates – use the SPARQL-backed store (or file locking) instead. The file must sit on a durable volume to survive across runs, the same requirement the SPARQL-backed store already has.
365
377
 
366
378
  #### Enabling skipping
367
379
 
@@ -382,11 +394,11 @@ skip iff recorded.sourceFingerprint === current.sourceFingerprint
382
394
  AND recorded.pipelineVersion === current.pipelineVersion
383
395
  ```
384
396
 
385
- Otherwise it imports (if needed), runs the stages, and writes an updated record. `pipelineVersion` is consumer-owned and opaque: rotate it only on releases that change output, and every dataset reprocesses on the next run. It is **required** when a `provenanceStore` is configured (a skip-enabled pipeline with no version would silently freeze); when no store is configured, every dataset is reprocessed — today’s behaviour. A dataset that failed but whose source is unchanged is recorded as `'failed'` and skipped on later runs until its source changes or the version rotates, so a deterministically failing import is not retried every run.
397
+ Otherwise it imports (if needed), runs the stages, and writes an updated record. `pipelineVersion` is consumer-owned and opaque: rotate it only on releases that change output, and every dataset reprocesses on the next run. It is **required** when a `provenanceStore` is configured (a skip-enabled pipeline with no version would silently freeze); when no store is configured, every dataset is reprocessed – today’s behaviour. A dataset that failed but whose source is unchanged is recorded as `'failed'` and skipped on later runs until its source changes or the version rotates, so a deterministically failing import is not retried every run.
386
398
 
387
399
  ### Source-change fingerprint
388
400
 
389
- `sourceFingerprint(distribution, probeResult)` derives a cheap, opaque change signal for a distribution from metadata the probe already collected — no body download. For a data dump it combines the most recent of the register’s `dct:modified` and the artifact’s HTTP `Last-Modified` with the byte size (the probe’s `Content-Length`, falling back to the declared `dcat:byteSize`). It returns `null` for a live SPARQL endpoint, or when no date and no size can be established — a `null` fingerprint never compares equal, so such a distribution is always reprocessed.
401
+ `sourceFingerprint(distribution, probeResult)` derives a cheap, opaque change signal for a distribution from metadata the probe already collected – no body download. For a data dump it combines the most recent of the register’s `dct:modified` and the artifact’s HTTP `Last-Modified` with the byte size (the probe’s `Content-Length`, falling back to the declared `dcat:byteSize`). It returns `null` for a live SPARQL endpoint, or when no date and no size can be established – a `null` fingerprint never compares equal, so such a distribution is always reprocessed.
390
402
 
391
403
  ### Plugins
392
404
 
@@ -0,0 +1,34 @@
1
+ import type { ProcessingRecord } from './record.js';
2
+ import type { ProvenanceStore } from './store.js';
3
+ export interface FileProvenanceStoreOptions {
4
+ /**
5
+ * Path of the JSON file the records are persisted to. Parent directories are
6
+ * created on the first write. Must sit on a durable volume to survive across
7
+ * runs (a Kubernetes CronJob’s container filesystem is discarded).
8
+ */
9
+ path: string;
10
+ }
11
+ /**
12
+ * A {@link ProvenanceStore} that persists all records to a single JSON file,
13
+ * keyed by dataset URI.
14
+ *
15
+ * For pipelines that run without a triplestore – e.g. a single CronJob
16
+ * persisting to a mounted volume – a JSON file is far lighter than standing
17
+ * up a SPARQL server purely to remember processing records.
18
+ *
19
+ * Writes are atomic (temp file + rename), so a run killed mid-write cannot
20
+ * corrupt the next run’s skip decisions. Safe for a single writer only:
21
+ * concurrent pipeline processes writing the same file lose each other’s
22
+ * updates – use {@link FileLoadedSparqlProvenanceStore} (or file locking)
23
+ * instead. Every access reads the whole file into memory; records are tiny,
24
+ * but at very large dataset counts prefer the SPARQL-backed store.
25
+ */
26
+ export declare class FileProvenanceStore implements ProvenanceStore {
27
+ private readonly path;
28
+ constructor(options: FileProvenanceStoreOptions);
29
+ get(datasetUri: URL): Promise<ProcessingRecord | null>;
30
+ set(datasetUri: URL, record: ProcessingRecord): Promise<void>;
31
+ private readAll;
32
+ private writeAll;
33
+ }
34
+ //# sourceMappingURL=fileProvenanceStore.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"fileProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileProvenanceStore.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,0BAA0B;IACzC;;;;OAIG;IACH,IAAI,EAAE,MAAM,CAAC;CACd;AAED;;;;;;;;;;;;;;GAcG;AACH,qBAAa,mBAAoB,YAAW,eAAe;IACzD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;gBAElB,OAAO,EAAE,0BAA0B;IAIzC,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAKtD,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YAMrD,OAAO;YAcP,QAAQ;CAQvB"}
@@ -0,0 +1,52 @@
1
+ import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';
2
+ import { dirname } from 'node:path';
3
+ /**
4
+ * A {@link ProvenanceStore} that persists all records to a single JSON file,
5
+ * keyed by dataset URI.
6
+ *
7
+ * For pipelines that run without a triplestore – e.g. a single CronJob
8
+ * persisting to a mounted volume – a JSON file is far lighter than standing
9
+ * up a SPARQL server purely to remember processing records.
10
+ *
11
+ * Writes are atomic (temp file + rename), so a run killed mid-write cannot
12
+ * corrupt the next run’s skip decisions. Safe for a single writer only:
13
+ * concurrent pipeline processes writing the same file lose each other’s
14
+ * updates – use {@link FileLoadedSparqlProvenanceStore} (or file locking)
15
+ * instead. Every access reads the whole file into memory; records are tiny,
16
+ * but at very large dataset counts prefer the SPARQL-backed store.
17
+ */
18
+ export class FileProvenanceStore {
19
+ path;
20
+ constructor(options) {
21
+ this.path = options.path;
22
+ }
23
+ async get(datasetUri) {
24
+ const records = await this.readAll();
25
+ return records[datasetUri.toString()] ?? null;
26
+ }
27
+ async set(datasetUri, record) {
28
+ const records = await this.readAll();
29
+ records[datasetUri.toString()] = record;
30
+ await this.writeAll(records);
31
+ }
32
+ async readAll() {
33
+ let json;
34
+ try {
35
+ json = await readFile(this.path, 'utf8');
36
+ }
37
+ catch (error) {
38
+ // A missing file is the empty store: nothing has been processed yet.
39
+ if (error.code === 'ENOENT') {
40
+ return {};
41
+ }
42
+ throw error;
43
+ }
44
+ return JSON.parse(json);
45
+ }
46
+ async writeAll(records) {
47
+ await mkdir(dirname(this.path), { recursive: true });
48
+ const temporaryPath = `${this.path}.${process.pid}.tmp`;
49
+ await writeFile(temporaryPath, `${JSON.stringify(records, null, 2)}\n`);
50
+ await rename(temporaryPath, this.path); // Atomic on POSIX filesystems.
51
+ }
52
+ }
@@ -3,4 +3,5 @@ export { shouldReprocess } from './reprocessDecision.js';
3
3
  export type { ProcessingRecord, ChangeKey } from './record.js';
4
4
  export type { ProvenanceStore } from './store.js';
5
5
  export { FileLoadedSparqlProvenanceStore, type FileLoadedSparqlProvenanceStoreOptions, } from './fileLoadedSparqlProvenanceStore.js';
6
+ export { FileProvenanceStore, type FileProvenanceStoreOptions, } from './fileProvenanceStore.js';
6
7
  //# sourceMappingURL=index.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/provenance/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AACzD,YAAY,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAC/D,YAAY,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAClD,OAAO,EACL,+BAA+B,EAC/B,KAAK,sCAAsC,GAC5C,MAAM,sCAAsC,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/provenance/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AACzD,YAAY,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAC/D,YAAY,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAClD,OAAO,EACL,+BAA+B,EAC/B,KAAK,sCAAsC,GAC5C,MAAM,sCAAsC,CAAC;AAC9C,OAAO,EACL,mBAAmB,EACnB,KAAK,0BAA0B,GAChC,MAAM,0BAA0B,CAAC"}
@@ -1,3 +1,4 @@
1
1
  export { sourceFingerprint } from './sourceFingerprint.js';
2
2
  export { shouldReprocess } from './reprocessDecision.js';
3
3
  export { FileLoadedSparqlProvenanceStore, } from './fileLoadedSparqlProvenanceStore.js';
4
+ export { FileProvenanceStore, } from './fileProvenanceStore.js';
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@lde/pipeline",
3
- "version": "0.34.2",
3
+ "version": "0.34.4",
4
4
  "repository": {
5
5
  "url": "git+https://github.com/ldelements/lde.git",
6
6
  "directory": "packages/pipeline"
@@ -26,9 +26,9 @@
26
26
  "dependencies": {
27
27
  "@lde/dataset": "^0.7.8",
28
28
  "@lde/dataset-registry-client": "^0.8.6",
29
- "@lde/distribution-health": "^0.2.6",
29
+ "@lde/distribution-health": "^0.2.7",
30
30
  "@lde/distribution-probe": "^0.2.6",
31
- "@lde/sparql-importer": "^0.6.6",
31
+ "@lde/sparql-importer": "^0.6.7",
32
32
  "@lde/sparql-server": "^0.4.12",
33
33
  "@rdfjs/namespace": "^2.0.1",
34
34
  "@rdfjs/types": "^2.0.1",