@lde/pipeline 0.34.2 → 0.34.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -22
- package/dist/provenance/fileProvenanceStore.d.ts +34 -0
- package/dist/provenance/fileProvenanceStore.d.ts.map +1 -0
- package/dist/provenance/fileProvenanceStore.js +52 -0
- package/dist/provenance/index.d.ts +1 -0
- package/dist/provenance/index.d.ts.map +1 -1
- package/dist/provenance/index.js +1 -0
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
A framework for transforming large RDF datasets, primarily using [SPARQL](https://www.w3.org/TR/sparql11-query/) queries with TypeScript for the parts that are hard to express in SPARQL alone.
|
|
4
4
|
|
|
5
|
-
- **SPARQL-native.** Data transformations are plain SPARQL query files
|
|
5
|
+
- **SPARQL-native.** Data transformations are plain SPARQL query files – portable, transparent, testable and version-controlled.
|
|
6
6
|
- **Composable.** Executors are an interface: wrap a SPARQL reader with custom TypeScript to handle edge cases like date parsing or string normalisation (see [Reader](#reader)).
|
|
7
7
|
- **Extensible.** A plugin system lets packages like [@lde/pipeline-void](../pipeline-void) (or your own plugins) hook into the pipeline lifecycle.
|
|
8
8
|
|
|
@@ -44,9 +44,9 @@ const resolver = new ImportResolver(new SparqlDistributionResolver(), {
|
|
|
44
44
|
});
|
|
45
45
|
```
|
|
46
46
|
|
|
47
|
-
- `'sparql'` (default)
|
|
48
|
-
- `'sparqlWithImportFallback'`
|
|
49
|
-
- `'import'`
|
|
47
|
+
- `'sparql'` (default) – use the dataset’s own SPARQL endpoint when one is available; import a data dump only when no endpoint responds.
|
|
48
|
+
- `'sparqlWithImportFallback'` – like `'sparql'`, but also fall back to the data dump when the endpoint passes probing yet a stage fails against it at runtime. The pipeline discards the endpoint-sourced partial output and re-runs all stages against the import. Use this when endpoints are present but unreliable for heavy aggregate queries.
|
|
49
|
+
- `'import'` – always import the data dump, even when a working endpoint is advertised.
|
|
50
50
|
|
|
51
51
|
### Stage
|
|
52
52
|
|
|
@@ -90,7 +90,7 @@ new Stage({
|
|
|
90
90
|
|
|
91
91
|
`expectsOutput` (default: `false`) marks a stage whose query must yield at least one quad. A supported stage that produces none is then treated as a hard failure rather than a legitimately empty result.
|
|
92
92
|
|
|
93
|
-
Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always return exactly one row
|
|
93
|
+
Set it for scalar aggregates such as `SELECT (COUNT(*) AS ?n)`, which always return exactly one row – so zero output can only mean the endpoint truncated or aborted the response (e.g. a timeout surfaced as an empty `HTTP 200`). The failure flows through like any other hard stage failure, triggering the [reactive dump fallback](#distribution-resolver) when `strategy: 'sparqlWithImportFallback'` is configured. Leave it `false` for stages that may legitimately be empty, such as class or property partitions of a dataset that lacks that structure.
|
|
94
94
|
|
|
95
95
|
### Item Selector
|
|
96
96
|
|
|
@@ -115,7 +115,7 @@ new SparqlItemSelector({
|
|
|
115
115
|
|
|
116
116
|
#### Capping total results with `maxResults`
|
|
117
117
|
|
|
118
|
-
By default, `SparqlItemSelector` paginates through **all** matching rows: any `LIMIT` clause in the query is interpreted as the page size, then it walks pages with `OFFSET` until the source is exhausted. To cap the total bindings yielded across all pages
|
|
118
|
+
By default, `SparqlItemSelector` paginates through **all** matching rows: any `LIMIT` clause in the query is interpreted as the page size, then it walks pages with `OFFSET` until the source is exhausted. To cap the total bindings yielded across all pages – for sampling, testing, prototyping, or just safety – set `maxResults`:
|
|
119
119
|
|
|
120
120
|
```typescript
|
|
121
121
|
new SparqlItemSelector({
|
|
@@ -126,7 +126,7 @@ new SparqlItemSelector({
|
|
|
126
126
|
|
|
127
127
|
When `maxResults` is set:
|
|
128
128
|
|
|
129
|
-
- Pagination stops as soon as `maxResults` bindings have been yielded
|
|
129
|
+
- Pagination stops as soon as `maxResults` bindings have been yielded – no wasted page request after the cap is hit.
|
|
130
130
|
- The last (partial) page's `LIMIT` is shrunk to the remaining cap so the endpoint doesn't over-fetch on the remainder (e.g. with `maxResults: 85` and `pageSize: 10`, the 9th page request is `LIMIT 5`, not `LIMIT 10`).
|
|
131
131
|
- The first page uses the configured page size as-is; `maxResults` and page size stay orthogonal. If `maxResults < pageSize`, the first page may return a few rows that aren't yielded.
|
|
132
132
|
- `maxResults: 0` is a valid no-op; the selector yields nothing without issuing any SPARQL request.
|
|
@@ -162,7 +162,7 @@ const reader = new SparqlConstructReader({
|
|
|
162
162
|
});
|
|
163
163
|
```
|
|
164
164
|
|
|
165
|
-
SPARQL CONSTRUCT queries can produce duplicate triples
|
|
165
|
+
SPARQL CONSTRUCT queries can produce duplicate triples – for example, constant triples (like `?dataset a edm:ProvidedCHO`) are emitted for every solution row. Enable `deduplicate` to remove duplicates inline on the stream using a string-based identity set (inspired by [Comunica's `distinctConstruct`](https://comunica.dev/docs/query/advanced/context/#14--distinct-construct)):
|
|
166
166
|
|
|
167
167
|
```typescript
|
|
168
168
|
const reader = new SparqlConstructReader({
|
|
@@ -175,9 +175,9 @@ The dedup set is scoped to each `read()` call, so memory stays bounded to the nu
|
|
|
175
175
|
|
|
176
176
|
### Extending a stage with a quad transform
|
|
177
177
|
|
|
178
|
-
Some logic is hard to express in pure SPARQL
|
|
178
|
+
Some logic is hard to express in pure SPARQL – cleaning up messy date notations, converting locale-specific dates to ISO 8601, or sampling a reader’s output and firing follow-up queries. Rather than subclass `Reader`, attach a `QuadTransform` to it as data: a plain function `(quads, context) => quads` that post-processes one reader’s output before the stage merges it with its siblings. This is extension point 1 of [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md).
|
|
179
179
|
|
|
180
|
-
A transform receives an `ReaderContext`
|
|
180
|
+
A transform receives an `ReaderContext` – the `dataset`, the `distribution` (so it can fire its own SPARQL queries), and the `stage` name. It runs once per reader call, so **write it to accept being called more than once**: a global stage calls it once over the reader’s complete output, but a per-class stage with batching enabled calls it once per batch (one class at `batchSize: 1`). Accumulate within an invocation, not across invocations – or keep the transform per-quad, where the number of calls makes no difference.
|
|
181
181
|
|
|
182
182
|
```typescript
|
|
183
183
|
import { DataFactory } from 'n3';
|
|
@@ -218,7 +218,7 @@ new Stage({
|
|
|
218
218
|
|
|
219
219
|
#### Adaptive timeouts
|
|
220
220
|
|
|
221
|
-
By default, every SPARQL request uses the same 5-minute budget. When a pipeline runs against many third-party endpoints, that fixed budget can cost ~80 minutes on a single dataset whose endpoint times out repeatedly on heavy queries
|
|
221
|
+
By default, every SPARQL request uses the same 5-minute budget. When a pipeline runs against many third-party endpoints, that fixed budget can cost ~80 minutes on a single dataset whose endpoint times out repeatedly on heavy queries – light stages on the same endpoint then sit behind the heavy ones that will never succeed.
|
|
222
222
|
|
|
223
223
|
A `TimeoutPolicy` decides the budget for each SPARQL request and observes the outcome. Two are built in:
|
|
224
224
|
|
|
@@ -252,11 +252,11 @@ Transitions are forwarded to the `ProgressReporter` via `timeoutTightened` / `ti
|
|
|
252
252
|
|
|
253
253
|
Implement `TimeoutPolicy` directly for custom strategies (closing over shared state in the factory if you want it to span datasets).
|
|
254
254
|
|
|
255
|
-
Timeouts live at the pipeline level
|
|
255
|
+
Timeouts live at the pipeline level – neither `SparqlConstructReader` nor `SparqlItemSelector` accept their own `timeout` option. Per-endpoint state belongs in the adaptive policy, and per-stage budgets aren’t supported. Reusable stage facades (`@lde/pipeline-void`, `@lde/pipeline-shacl-sampler`) follow the same convention.
|
|
256
256
|
|
|
257
257
|
### Validation
|
|
258
258
|
|
|
259
|
-
Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set
|
|
259
|
+
Stages can optionally validate their output quads against a `Validator`. Validation operates on the **combined output of all readers per batch**, not on individual quads or per-reader output. A batch produces a complete result set – a self-contained cluster of linked resources – that can be meaningfully matched against SHACL shapes. Even with a single reader, each batch is a complete unit; with multiple readers, shapes that reference triples from different readers are validated correctly.
|
|
260
260
|
|
|
261
261
|
Validating individual quads would be meaningless, since a single quad carries no structural context for shape matching. Validating the full pipeline output would also be problematic: because the pipeline streams results in batches, it doesn’t know where resource cluster boundaries fall. Batching the output could split a valid cluster across two batches, causing partial resources to fail validation even though the complete cluster is valid.
|
|
262
262
|
|
|
@@ -291,14 +291,14 @@ new Stage({
|
|
|
291
291
|
|
|
292
292
|
#### Per-dataset reporting
|
|
293
293
|
|
|
294
|
-
After all stages for a dataset have run, the pipeline calls `validator.report(dataset)` once for each distinct validator attached to any stage and emits a `datasetValidated(dataset, report)` event on the reporter. The call happens **regardless of whether any stage actually invoked `validate()`**
|
|
294
|
+
After all stages for a dataset have run, the pipeline calls `validator.report(dataset)` once for each distinct validator attached to any stage and emits a `datasetValidated(dataset, report)` event on the reporter. The call happens **regardless of whether any stage actually invoked `validate()`** – for SHACL that means a dataset whose stages produced no input typically reports `quadsValidated: 0` and `conforms: true` (the SHACL vacuous-truth default). Consumers that want to distinguish ‘not tested’ from ‘tested and passed’ can read `quadsValidated`.
|
|
295
295
|
|
|
296
296
|
### Writer
|
|
297
297
|
|
|
298
298
|
Writes generated quads to a destination. A `Writer` is transactional: each pipeline run opens one run on it (`openRun(context)`), writes every dataset through the resulting `RunWriter`, and ends with exactly one `commit()` (on success) or `abort(error)` (on failure). The run lifecycle is the home of destination-level concerns such as atomic swaps, deletion sweeps and cross-pod locks; the pipeline drives `openRun → write* → commit/abort` uniformly and never branches on the writer’s update mode. After a dataset’s stages complete, its `flush(dataset, outcome)` says whether the dataset succeeded, so a writer can gate destructive finalization (e.g. an In-place stale-document sweep) on `'success'`.
|
|
299
299
|
|
|
300
|
-
- `SparqlUpdateWriter`
|
|
301
|
-
- `FileWriter`
|
|
300
|
+
- `SparqlUpdateWriter` – writes to a SPARQL endpoint via UPDATE queries; writes are visible as they land, so `commit`/`abort` are no-ops
|
|
301
|
+
- `FileWriter` – writes to local files; each file is streamed to a sibling temp file and atomically renamed on flush, `commit` finalizes any files still open, and `abort` discards temp output. In `n-quads` format it skolemises blank nodes to dataset-scoped IRIs, because such files are typically `cat`-indexed into one served graph where document-scoped blank-node labels from different datasets would otherwise fuse (see `blankNodes`/`failOnBlankNodes` and ldelements/lde#474); Turtle/N-Triples output is left unchanged
|
|
302
302
|
|
|
303
303
|
The `RunContext` handed to `openRun` carries the run’s identity (`runId`, `startedAt`), the full selection (`selectedSources()`, complete by commit time – including datasets skipped as unchanged) and the pipeline’s provenance store, when configured.
|
|
304
304
|
|
|
@@ -320,7 +320,7 @@ async openRun(): Promise<RunWriter> {
|
|
|
320
320
|
|
|
321
321
|
A `ProgressReporter` observes the run, receiving lifecycle events such as `pipelineStart`, `stageComplete`, `datasetValidated` and `pipelineComplete`. Every method is optional, so a reporter implements only the events it cares about.
|
|
322
322
|
|
|
323
|
-
Pass a single reporter, or an array to have several observe the same run
|
|
323
|
+
Pass a single reporter, or an array to have several observe the same run – for example a console reporter alongside one that collects validation verdicts:
|
|
324
324
|
|
|
325
325
|
```typescript
|
|
326
326
|
new Pipeline({
|
|
@@ -342,11 +342,11 @@ interface ProvenanceStore {
|
|
|
342
342
|
}
|
|
343
343
|
```
|
|
344
344
|
|
|
345
|
-
A `ProcessingRecord` holds the two opaque change fields
|
|
345
|
+
A `ProcessingRecord` holds the two opaque change fields – `sourceFingerprint` (derived automatically from source metadata) and `pipelineVersion` (consumer-declared) – plus `generatedAt` and a `status` of `'success'` or `'failed'`. The two change fields are compared only for equality, never parsed or ordered.
|
|
346
346
|
|
|
347
347
|
#### `FileLoadedSparqlProvenanceStore`
|
|
348
348
|
|
|
349
|
-
The reference implementation targets a triplestore that is served read-only and rebuilt by bulk-loading files (e.g. [QLever](https://github.com/ad-freiburg/qlever)). It reads through SPARQL queries against the live endpoint, and writes records as files for the next bulk-load
|
|
349
|
+
The reference implementation targets a triplestore that is served read-only and rebuilt by bulk-loading files (e.g. [QLever](https://github.com/ad-freiburg/qlever)). It reads through SPARQL queries against the live endpoint, and writes records as files for the next bulk-load – because the endpoint accepts no SPARQL UPDATE.
|
|
350
350
|
|
|
351
351
|
```typescript
|
|
352
352
|
import { FileLoadedSparqlProvenanceStore } from '@lde/pipeline';
|
|
@@ -361,7 +361,19 @@ const store = new FileLoadedSparqlProvenanceStore({
|
|
|
361
361
|
- `get` runs a named-graph-scoped SPARQL `SELECT` against `queryEndpoint`, reading the records a previous run loaded.
|
|
362
362
|
- `set` writes one flat [PROV-O](https://www.w3.org/TR/prov-o/) N-Quads file per dataset into `outputDir`, in the pipeline-scoped named graph, to be bulk-loaded after the run.
|
|
363
363
|
|
|
364
|
-
Each record is stored as flat PROV-O on the dataset entity
|
|
364
|
+
Each record is stored as flat PROV-O on the dataset entity – `prov:generatedAtTime` plus `sourceFingerprint`, `pipelineVersion` and `status` under the `https://w3id.org/lde/provenance#` namespace. Scoping every record by `pipelineIri` (used as the named graph) lets multiple pipelines share one triplestore without colliding.
|
|
365
|
+
|
|
366
|
+
#### `FileProvenanceStore`
|
|
367
|
+
|
|
368
|
+
For pipelines that run without a triplestore, `FileProvenanceStore` persists all records to a single JSON file, keyed by dataset URI:
|
|
369
|
+
|
|
370
|
+
```typescript
|
|
371
|
+
import { FileProvenanceStore } from '@lde/pipeline';
|
|
372
|
+
|
|
373
|
+
const store = new FileProvenanceStore({ path: './state/provenance.json' });
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
Writes are atomic (temp file + rename), so a run killed mid-write cannot corrupt the next run’s skip decisions. Safe for a single writer: concurrent pipeline processes writing the same file lose each other’s updates – use the SPARQL-backed store (or file locking) instead. The file must sit on a durable volume to survive across runs, the same requirement the SPARQL-backed store already has.
|
|
365
377
|
|
|
366
378
|
#### Enabling skipping
|
|
367
379
|
|
|
@@ -382,11 +394,11 @@ skip iff recorded.sourceFingerprint === current.sourceFingerprint
|
|
|
382
394
|
AND recorded.pipelineVersion === current.pipelineVersion
|
|
383
395
|
```
|
|
384
396
|
|
|
385
|
-
Otherwise it imports (if needed), runs the stages, and writes an updated record. `pipelineVersion` is consumer-owned and opaque: rotate it only on releases that change output, and every dataset reprocesses on the next run. It is **required** when a `provenanceStore` is configured (a skip-enabled pipeline with no version would silently freeze); when no store is configured, every dataset is reprocessed
|
|
397
|
+
Otherwise it imports (if needed), runs the stages, and writes an updated record. `pipelineVersion` is consumer-owned and opaque: rotate it only on releases that change output, and every dataset reprocesses on the next run. It is **required** when a `provenanceStore` is configured (a skip-enabled pipeline with no version would silently freeze); when no store is configured, every dataset is reprocessed – today’s behaviour. A dataset that failed but whose source is unchanged is recorded as `'failed'` and skipped on later runs until its source changes or the version rotates, so a deterministically failing import is not retried every run.
|
|
386
398
|
|
|
387
399
|
### Source-change fingerprint
|
|
388
400
|
|
|
389
|
-
`sourceFingerprint(distribution, probeResult)` derives a cheap, opaque change signal for a distribution from metadata the probe already collected
|
|
401
|
+
`sourceFingerprint(distribution, probeResult)` derives a cheap, opaque change signal for a distribution from metadata the probe already collected – no body download. For a data dump it combines the most recent of the register’s `dct:modified` and the artifact’s HTTP `Last-Modified` with the byte size (the probe’s `Content-Length`, falling back to the declared `dcat:byteSize`). It returns `null` for a live SPARQL endpoint, or when no date and no size can be established – a `null` fingerprint never compares equal, so such a distribution is always reprocessed.
|
|
390
402
|
|
|
391
403
|
### Plugins
|
|
392
404
|
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import type { ProcessingRecord } from './record.js';
|
|
2
|
+
import type { ProvenanceStore } from './store.js';
|
|
3
|
+
export interface FileProvenanceStoreOptions {
|
|
4
|
+
/**
|
|
5
|
+
* Path of the JSON file the records are persisted to. Parent directories are
|
|
6
|
+
* created on the first write. Must sit on a durable volume to survive across
|
|
7
|
+
* runs (a Kubernetes CronJob’s container filesystem is discarded).
|
|
8
|
+
*/
|
|
9
|
+
path: string;
|
|
10
|
+
}
|
|
11
|
+
/**
|
|
12
|
+
* A {@link ProvenanceStore} that persists all records to a single JSON file,
|
|
13
|
+
* keyed by dataset URI.
|
|
14
|
+
*
|
|
15
|
+
* For pipelines that run without a triplestore – e.g. a single CronJob
|
|
16
|
+
* persisting to a mounted volume – a JSON file is far lighter than standing
|
|
17
|
+
* up a SPARQL server purely to remember processing records.
|
|
18
|
+
*
|
|
19
|
+
* Writes are atomic (temp file + rename), so a run killed mid-write cannot
|
|
20
|
+
* corrupt the next run’s skip decisions. Safe for a single writer only:
|
|
21
|
+
* concurrent pipeline processes writing the same file lose each other’s
|
|
22
|
+
* updates – use {@link FileLoadedSparqlProvenanceStore} (or file locking)
|
|
23
|
+
* instead. Every access reads the whole file into memory; records are tiny,
|
|
24
|
+
* but at very large dataset counts prefer the SPARQL-backed store.
|
|
25
|
+
*/
|
|
26
|
+
export declare class FileProvenanceStore implements ProvenanceStore {
|
|
27
|
+
private readonly path;
|
|
28
|
+
constructor(options: FileProvenanceStoreOptions);
|
|
29
|
+
get(datasetUri: URL): Promise<ProcessingRecord | null>;
|
|
30
|
+
set(datasetUri: URL, record: ProcessingRecord): Promise<void>;
|
|
31
|
+
private readAll;
|
|
32
|
+
private writeAll;
|
|
33
|
+
}
|
|
34
|
+
//# sourceMappingURL=fileProvenanceStore.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"fileProvenanceStore.d.ts","sourceRoot":"","sources":["../../src/provenance/fileProvenanceStore.ts"],"names":[],"mappings":"AAEA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,aAAa,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAElD,MAAM,WAAW,0BAA0B;IACzC;;;;OAIG;IACH,IAAI,EAAE,MAAM,CAAC;CACd;AAED;;;;;;;;;;;;;;GAcG;AACH,qBAAa,mBAAoB,YAAW,eAAe;IACzD,OAAO,CAAC,QAAQ,CAAC,IAAI,CAAS;gBAElB,OAAO,EAAE,0BAA0B;IAIzC,GAAG,CAAC,UAAU,EAAE,GAAG,GAAG,OAAO,CAAC,gBAAgB,GAAG,IAAI,CAAC;IAKtD,GAAG,CAAC,UAAU,EAAE,GAAG,EAAE,MAAM,EAAE,gBAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;YAMrD,OAAO;YAcP,QAAQ;CAQvB"}
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
import { mkdir, readFile, rename, writeFile } from 'node:fs/promises';
|
|
2
|
+
import { dirname } from 'node:path';
|
|
3
|
+
/**
|
|
4
|
+
* A {@link ProvenanceStore} that persists all records to a single JSON file,
|
|
5
|
+
* keyed by dataset URI.
|
|
6
|
+
*
|
|
7
|
+
* For pipelines that run without a triplestore – e.g. a single CronJob
|
|
8
|
+
* persisting to a mounted volume – a JSON file is far lighter than standing
|
|
9
|
+
* up a SPARQL server purely to remember processing records.
|
|
10
|
+
*
|
|
11
|
+
* Writes are atomic (temp file + rename), so a run killed mid-write cannot
|
|
12
|
+
* corrupt the next run’s skip decisions. Safe for a single writer only:
|
|
13
|
+
* concurrent pipeline processes writing the same file lose each other’s
|
|
14
|
+
* updates – use {@link FileLoadedSparqlProvenanceStore} (or file locking)
|
|
15
|
+
* instead. Every access reads the whole file into memory; records are tiny,
|
|
16
|
+
* but at very large dataset counts prefer the SPARQL-backed store.
|
|
17
|
+
*/
|
|
18
|
+
export class FileProvenanceStore {
|
|
19
|
+
path;
|
|
20
|
+
constructor(options) {
|
|
21
|
+
this.path = options.path;
|
|
22
|
+
}
|
|
23
|
+
async get(datasetUri) {
|
|
24
|
+
const records = await this.readAll();
|
|
25
|
+
return records[datasetUri.toString()] ?? null;
|
|
26
|
+
}
|
|
27
|
+
async set(datasetUri, record) {
|
|
28
|
+
const records = await this.readAll();
|
|
29
|
+
records[datasetUri.toString()] = record;
|
|
30
|
+
await this.writeAll(records);
|
|
31
|
+
}
|
|
32
|
+
async readAll() {
|
|
33
|
+
let json;
|
|
34
|
+
try {
|
|
35
|
+
json = await readFile(this.path, 'utf8');
|
|
36
|
+
}
|
|
37
|
+
catch (error) {
|
|
38
|
+
// A missing file is the empty store: nothing has been processed yet.
|
|
39
|
+
if (error.code === 'ENOENT') {
|
|
40
|
+
return {};
|
|
41
|
+
}
|
|
42
|
+
throw error;
|
|
43
|
+
}
|
|
44
|
+
return JSON.parse(json);
|
|
45
|
+
}
|
|
46
|
+
async writeAll(records) {
|
|
47
|
+
await mkdir(dirname(this.path), { recursive: true });
|
|
48
|
+
const temporaryPath = `${this.path}.${process.pid}.tmp`;
|
|
49
|
+
await writeFile(temporaryPath, `${JSON.stringify(records, null, 2)}\n`);
|
|
50
|
+
await rename(temporaryPath, this.path); // Atomic on POSIX filesystems.
|
|
51
|
+
}
|
|
52
|
+
}
|
|
@@ -3,4 +3,5 @@ export { shouldReprocess } from './reprocessDecision.js';
|
|
|
3
3
|
export type { ProcessingRecord, ChangeKey } from './record.js';
|
|
4
4
|
export type { ProvenanceStore } from './store.js';
|
|
5
5
|
export { FileLoadedSparqlProvenanceStore, type FileLoadedSparqlProvenanceStoreOptions, } from './fileLoadedSparqlProvenanceStore.js';
|
|
6
|
+
export { FileProvenanceStore, type FileProvenanceStoreOptions, } from './fileProvenanceStore.js';
|
|
6
7
|
//# sourceMappingURL=index.d.ts.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/provenance/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AACzD,YAAY,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAC/D,YAAY,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAClD,OAAO,EACL,+BAA+B,EAC/B,KAAK,sCAAsC,GAC5C,MAAM,sCAAsC,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/provenance/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAC3D,OAAO,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AACzD,YAAY,EAAE,gBAAgB,EAAE,SAAS,EAAE,MAAM,aAAa,CAAC;AAC/D,YAAY,EAAE,eAAe,EAAE,MAAM,YAAY,CAAC;AAClD,OAAO,EACL,+BAA+B,EAC/B,KAAK,sCAAsC,GAC5C,MAAM,sCAAsC,CAAC;AAC9C,OAAO,EACL,mBAAmB,EACnB,KAAK,0BAA0B,GAChC,MAAM,0BAA0B,CAAC"}
|
package/dist/provenance/index.js
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
1
|
export { sourceFingerprint } from './sourceFingerprint.js';
|
|
2
2
|
export { shouldReprocess } from './reprocessDecision.js';
|
|
3
3
|
export { FileLoadedSparqlProvenanceStore, } from './fileLoadedSparqlProvenanceStore.js';
|
|
4
|
+
export { FileProvenanceStore, } from './fileProvenanceStore.js';
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lde/pipeline",
|
|
3
|
-
"version": "0.34.
|
|
3
|
+
"version": "0.34.4",
|
|
4
4
|
"repository": {
|
|
5
5
|
"url": "git+https://github.com/ldelements/lde.git",
|
|
6
6
|
"directory": "packages/pipeline"
|
|
@@ -26,9 +26,9 @@
|
|
|
26
26
|
"dependencies": {
|
|
27
27
|
"@lde/dataset": "^0.7.8",
|
|
28
28
|
"@lde/dataset-registry-client": "^0.8.6",
|
|
29
|
-
"@lde/distribution-health": "^0.2.
|
|
29
|
+
"@lde/distribution-health": "^0.2.7",
|
|
30
30
|
"@lde/distribution-probe": "^0.2.6",
|
|
31
|
-
"@lde/sparql-importer": "^0.6.
|
|
31
|
+
"@lde/sparql-importer": "^0.6.7",
|
|
32
32
|
"@lde/sparql-server": "^0.4.12",
|
|
33
33
|
"@rdfjs/namespace": "^2.0.1",
|
|
34
34
|
"@rdfjs/types": "^2.0.1",
|