@lde/pipeline-void 0.31.4 → 0.32.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -10,15 +10,14 @@ Returns all VoID stages in their recommended execution order. The ordering is op
10
10
 
11
11
  Accepts an optional `VoidStagesOptions` object:
12
12
 
13
- | Option | Default | Description |
14
- | ------------------ | ------- | --------------------------------------------------------------------------------------------------------------- |
15
- | `batchSize` | 10 | Maximum class bindings per reader call (per-class stages only) |
16
- | `maxConcurrency` | 10 | Maximum concurrent in-flight reader batches (per-class stages only) |
17
- | `perClass` | — | Override per-class iteration for all five per-class stages |
18
- | `uriSpaces` | — | When provided, includes the object URI space stage |
19
- | `vocabularies` | — | Additional vocabulary namespace URIs to detect beyond the built-in defaults |
20
- | `transforms` | — | Transforms to attach to bundled stages, keyed by `VOID_STAGE_NAMES` (see [Stage transforms](#stage-transforms)) |
21
- | `namespaceAliases` | — | Namespace pairs to treat as equivalent when partitioning (see [Namespace aliases](#namespace-aliases)) |
13
+ | Option | Default | Description |
14
+ | ---------------- | ------- | --------------------------------------------------------------------------------------------------------------- |
15
+ | `batchSize` | 10 | Maximum class bindings per reader call (per-class stages only) |
16
+ | `maxConcurrency` | 10 | Maximum concurrent in-flight reader batches (per-class stages only) |
17
+ | `perClass` | — | Override per-class iteration for all five per-class stages |
18
+ | `uriSpaces` | — | When provided, includes the object URI space stage |
19
+ | `vocabularies` | — | Additional vocabulary namespace URIs to detect beyond the built-in defaults |
20
+ | `transforms` | — | Transforms to attach to bundled stages, keyed by `VOID_STAGE_NAMES` (see [Stage transforms](#stage-transforms)) |
22
21
 
23
22
  Per-request timeouts are configured at the `Pipeline` level via `PipelineOptions.timeout`, not per VoID stage.
24
23
 
@@ -73,28 +72,26 @@ Global and domain-specific factories accept `VoidStageOptions` (`transform`) and
73
72
  | `detectVocabularies()` | [`entity-properties.rq`](queries/entity-properties.rq) — Entity properties with automatic `void:vocabulary` detection. Accepts `DetectVocabulariesOptions` with an optional `vocabularies` array to extend the built-in defaults. |
74
73
  | `uriSpaces(uriSpaceMap)` | [`object-uri-space.rq`](queries/object-uri-space.rq) — Object URI namespace linksets, aggregated against a provided URI space map |
75
74
 
76
- ## Namespace aliases
75
+ ## Namespace normalization
77
76
 
78
- Some vocabularies publish under both HTTP and HTTPS variants of the same namespace (notably schema.org), and datasets mix them. Without normalization, each variant gets its own `void:classPartition`/`void:propertyPartition`, so consumers see two partitions for what is semantically one class.
77
+ Some vocabularies publish under both `http://` and `https://` variants of the same namespace (notably schema.org), and datasets mix them. Without normalization each variant gets its own `void:classPartition`/`void:propertyPartition`, so consumers see two partitions for one class which crashed the Dataset Register browser (netwerk-digitaal-erfgoed/dataset-knowledge-graph#334).
79
78
 
80
- Pass `namespaceAliases` to merge the variants into one partition per canonical class/property:
79
+ `schemaOrgPartitionMergePlugin` normalizes `http://schema.org/` to `https://schema.org/` **and** merges the duplicate partition nodes the two variants produced. It is a [`beforeDatasetWrite`](../pipeline) plugin — it runs once over a whole dataset’s output at the pipeline edge, so the analysis queries stay unaware of namespace aliases (see [ADR 7](../../docs/decisions/0007-namespace-merge-as-a-dataset-plugin.md)):
81
80
 
82
81
  ```typescript
83
- const stages = await voidStages({
84
- namespaceAliases: [
85
- { canonical: 'https://schema.org/', alias: 'http://schema.org/' },
86
- ],
87
- });
88
- ```
89
-
90
- Merging happens **after** aggregation, in a single partition-merge transform (`mergeNamespaceVariants`) attached to the class and property partition stages: it re-mints each partition IRI from its canonical key components, collapses the duplicates, and sums `void:entities` / `void:triples`. The analysis queries stay plain — except `class-properties-objects.rq`, which normalizes at query time because its `void:distinctObjects` is a distinct-count over object values that overlap across variants and so cannot be recovered by summing. See [ADR 7](../../docs/decisions/0007-merge-namespace-alias-partitions.md).
82
+ import { voidStages, schemaOrgPartitionMergePlugin } from '@lde/pipeline-void';
83
+ import { Pipeline, provenancePlugin } from '@lde/pipeline';
91
84
 
92
- Summing pre-aggregated counts is exact only under two assumptions, both of which hold for the schema.org datasets that motivate this (the variants appear as disjoint subsets):
85
+ await new Pipeline({
86
+ stages: await voidStages(), // plain, no namespace options
87
+ plugins: [schemaOrgPartitionMergePlugin(), provenancePlugin()],
88
+ // …
89
+ }).run();
90
+ ```
93
91
 
94
- - **Subject/class disjointness** no resource is typed under both namespace variants of a class (guards the class-partition and triple-count sums).
95
- - **Predicate-namespace disjointness** — no subject uses both variants of the same property (guards the property-partition entity sum).
92
+ Use `namespacePartitionMergePlugin(aliases)` for namespaces other than schema.org. The transform streams it buffers only partition quads (bounded by the summary), passing everything else straight through. Datasets typically use a single schema.org namespace, so within one dataset there is one variant per class and every count stays exact; a dataset that mixes both namespaces on one property has its `void:distinctObjects` summed (an over-count for shared objects) rather than deduped.
96
93
 
97
- A resource typed under both variants over-counts its class entities. With no aliases configured the transform is a no-op. The top-level property partitions from `entity-properties.rq` and `void:vocabulary` detection keep the source namespaces, so consumers can still see which namespace the dataset actually uses.
94
+ > This plugin does more than rename IRIs: rewriting the `void:class` objects alone would still leave two `void:classPartition` nodes for one class. If you only need a blanket namespace rewrite over a dataset’s own quads (not a VoID partition merge) for example when mapping instance data to an application profile use the generic [`schemaOrgNormalizationPlugin` / `namespaceNormalizationPlugin`](../pipeline) from `@lde/pipeline` instead.
98
95
 
99
96
  ## Stage transforms
100
97
 
package/dist/index.d.ts CHANGED
@@ -1,8 +1,8 @@
1
1
  export { Stage, NotSupported } from '@lde/pipeline';
2
2
  export type { AttachedReader, NamespaceAlias, ReaderContext, QuadTransform, } from '@lde/pipeline';
3
3
  export * from './stage.js';
4
- export * from './namespaceAliases.js';
5
4
  export * from './partitionMerge.js';
5
+ export * from './partitionIri.js';
6
6
  export * from './vocabularyTransform.js';
7
7
  export * from './uriSpaceTransform.js';
8
8
  //# sourceMappingURL=index.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AACpD,YAAY,EACV,cAAc,EACd,cAAc,EACd,aAAa,EACb,aAAa,GACd,MAAM,eAAe,CAAC;AACvB,cAAc,YAAY,CAAC;AAC3B,cAAc,uBAAuB,CAAC;AACtC,cAAc,qBAAqB,CAAC;AACpC,cAAc,0BAA0B,CAAC;AACzC,cAAc,wBAAwB,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AACpD,YAAY,EACV,cAAc,EACd,cAAc,EACd,aAAa,EACb,aAAa,GACd,MAAM,eAAe,CAAC;AACvB,cAAc,YAAY,CAAC;AAC3B,cAAc,qBAAqB,CAAC;AACpC,cAAc,mBAAmB,CAAC;AAClC,cAAc,0BAA0B,CAAC;AACzC,cAAc,wBAAwB,CAAC"}
package/dist/index.js CHANGED
@@ -1,6 +1,6 @@
1
1
  export { Stage, NotSupported } from '@lde/pipeline';
2
2
  export * from './stage.js';
3
- export * from './namespaceAliases.js';
4
3
  export * from './partitionMerge.js';
4
+ export * from './partitionIri.js';
5
5
  export * from './vocabularyTransform.js';
6
6
  export * from './uriSpaceTransform.js';
@@ -0,0 +1,28 @@
1
+ /**
2
+ * Each partition kind's IRI prefix and the ordered SPARQL string expressions
3
+ * whose MD5 hash keys it. The queries must bind the standard component
4
+ * variables (`?class`, `?p`, `?dt`, `?objectClass`, `?lang`).
5
+ */
6
+ declare const PARTITION_KINDS: {
7
+ readonly class: readonly ["STR(?class)"];
8
+ readonly 'class-property': readonly ["STR(?class)", "STR(?p)"];
9
+ readonly datatype: readonly ["STR(?class)", "STR(?p)", "STR(?dt)"];
10
+ readonly 'object-class': readonly ["STR(?class)", "STR(?p)", "STR(?objectClass)"];
11
+ readonly language: readonly ["STR(?class)", "STR(?p)", "?lang"];
12
+ };
13
+ /** A VoID partition kind; also its IRI prefix. */
14
+ export type PartitionKind = keyof typeof PARTITION_KINDS;
15
+ /**
16
+ * Mint a partition IRI from its canonical component *string values* (the class
17
+ * IRI, property IRI, datatype IRI, or language tag — the `STR()` forms the
18
+ * SPARQL side concatenates). The TypeScript counterpart of the query minting.
19
+ */
20
+ export declare function mintPartitionIri(datasetIri: string, kind: PartitionKind, components: readonly string[]): string;
21
+ /**
22
+ * Replace every `#mint:<kind>#` marker in a query with the SPARQL value
23
+ * expression that mints that kind's partition IRI, e.g.
24
+ * `URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class))))`.
25
+ */
26
+ export declare function substituteMintMarkers(query: string): string;
27
+ export {};
28
+ //# sourceMappingURL=partitionIri.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"partitionIri.d.ts","sourceRoot":"","sources":["../src/partitionIri.ts"],"names":[],"mappings":"AAoBA;;;;GAIG;AACH,QAAA,MAAM,eAAe;;;;;;CAOX,CAAC;AAEX,kDAAkD;AAClD,MAAM,MAAM,aAAa,GAAG,MAAM,OAAO,eAAe,CAAC;AAEzD;;;;GAIG;AACH,wBAAgB,gBAAgB,CAC9B,UAAU,EAAE,MAAM,EAClB,IAAI,EAAE,aAAa,EACnB,UAAU,EAAE,SAAS,MAAM,EAAE,GAC5B,MAAM,CAGR;AAED;;;;GAIG;AACH,wBAAgB,qBAAqB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAY3D"}
@@ -0,0 +1,56 @@
1
+ import { createHash } from 'node:crypto';
2
+ /**
3
+ * Single source of truth for the VoID partition-IRI scheme
4
+ * (`<dataset>/.well-known/void#<prefix>-<MD5(component…)>`), shared by two
5
+ * derivations that must agree byte-for-byte:
6
+ *
7
+ * - {@link mintPartitionIri} mints an IRI in TypeScript (used by the
8
+ * partition-merge transform to re-key a partition to its canonical form);
9
+ * - {@link substituteMintMarkers} generates the equivalent SPARQL `BIND` value
10
+ * expression, injected into the analysis queries in place of `#mint:<kind>#`.
11
+ *
12
+ * Because both come from {@link PARTITION_KINDS}, a change to the scheme (a
13
+ * prefix, a component, its order) updates the query minting and the TS minting
14
+ * together — they cannot silently diverge.
15
+ */
16
+ /** The `/.well-known/void#` path under a dataset IRI where partitions live. */
17
+ const WELL_KNOWN_VOID = '/.well-known/void#';
18
+ /**
19
+ * Each partition kind's IRI prefix and the ordered SPARQL string expressions
20
+ * whose MD5 hash keys it. The queries must bind the standard component
21
+ * variables (`?class`, `?p`, `?dt`, `?objectClass`, `?lang`).
22
+ */
23
+ const PARTITION_KINDS = {
24
+ class: ['STR(?class)'],
25
+ 'class-property': ['STR(?class)', 'STR(?p)'],
26
+ datatype: ['STR(?class)', 'STR(?p)', 'STR(?dt)'],
27
+ 'object-class': ['STR(?class)', 'STR(?p)', 'STR(?objectClass)'],
28
+ // ?lang is a plain string literal, so it is hashed without STR().
29
+ language: ['STR(?class)', 'STR(?p)', '?lang'],
30
+ };
31
+ /**
32
+ * Mint a partition IRI from its canonical component *string values* (the class
33
+ * IRI, property IRI, datatype IRI, or language tag — the `STR()` forms the
34
+ * SPARQL side concatenates). The TypeScript counterpart of the query minting.
35
+ */
36
+ export function mintPartitionIri(datasetIri, kind, components) {
37
+ const hash = createHash('md5').update(components.join('')).digest('hex');
38
+ return `${datasetIri}${WELL_KNOWN_VOID}${kind}-${hash}`;
39
+ }
40
+ /**
41
+ * Replace every `#mint:<kind>#` marker in a query with the SPARQL value
42
+ * expression that mints that kind's partition IRI, e.g.
43
+ * `URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class))))`.
44
+ */
45
+ export function substituteMintMarkers(query) {
46
+ return query.replace(/#mint:([a-z-]+)#/g, (_match, kind) => {
47
+ const components = PARTITION_KINDS[kind];
48
+ if (components === undefined) {
49
+ throw new Error(`Unknown partition kind in #mint:# marker: ${kind}`);
50
+ }
51
+ const hashed = components.length === 1
52
+ ? components[0]
53
+ : `CONCAT(${components.join(', ')})`;
54
+ return `URI(CONCAT(STR(?dataset), "${WELL_KNOWN_VOID}${kind}-", MD5(${hashed})))`;
55
+ });
56
+ }
@@ -1,37 +1,55 @@
1
- import { type NamespaceAlias, type QuadTransform, type ReaderContext } from '@lde/pipeline';
1
+ import { type BeforeDatasetWriteContext, type NamespaceAlias, type PipelinePlugin, type QuadTransform } from '@lde/pipeline';
2
2
  /**
3
- * A {@link QuadTransform} that merges the `void:classPartition` /
4
- * `void:propertyPartition` subtrees of namespace-alias variants (e.g.
5
- * `http://schema.org/CreativeWork` and `https://schema.org/CreativeWork`) into
6
- * a single partition per canonical class/property.
3
+ * A {@link QuadTransform} for the {@link PipelinePlugin.beforeDatasetWrite} hook
4
+ * that merges the `void:classPartition` / `void:propertyPartition` subtrees of
5
+ * namespace-alias variants (e.g. `http://schema.org/CreativeWork` and
6
+ * `https://schema.org/CreativeWork`) into one partition per canonical
7
+ * class/property.
7
8
  *
8
9
  * VoID partitions are keyed by an opaque `MD5(class[, property[, …]])` IRI, so
9
- * two namespace variants yield two partition nodes that both, after
10
- * canonicalization, describe the same class. This transform re-mints every
11
- * partition IRI from its **canonical** key components (replicating the queries’
12
- * SPARQL `MD5(CONCAT(STR(…)))`), collapses the duplicates, and **sums** their
13
- * numeric measures (`void:entities`, `void:triples`).
10
+ * two namespace variants produce two partition nodes that, once the class IRI is
11
+ * canonicalized, describe the same class. Seeing a whole dataset's output at
12
+ * once, this transform re-mints every partition IRI from its **canonical** key
13
+ * components via {@link mintPartitionIri} — the single source of truth the
14
+ * queries' SPARQL minting is also generated from — collapses the duplicates,
15
+ * and sums their numeric measures.
14
16
  *
15
- * ## Correctness assumptions
17
+ * It streams: only partition quads are buffered (bounded by the summary — a
18
+ * handful of classes × properties, not the dataset), and every other quad
19
+ * passes straight through.
16
20
  *
17
- * Summing pre-aggregated counts is exact only under these assumptions; the
18
- * queries whose measures cannot be safely summed keep their normalization at
19
- * query time instead (notably `class-properties-objects.rq`’s
20
- * `void:distinctObjects`, which this transform never sees):
21
- *
22
- * - **Subject/class disjointness** — no resource is typed under two namespace
23
- * variants of the same class. Guards the `void:entities` sum on class
24
- * partitions and every `void:triples` sum (a doubly-typed resource’s triples
25
- * would otherwise count under both variants).
26
- * - **Predicate-namespace disjointness** — no subject uses two namespace
27
- * variants of the same property (e.g. both `http://schema.org/name` and
28
- * `https://schema.org/name`). Guards the `void:entities` sum on property
29
- * partitions.
21
+ * Datasets typically use a single schema.org namespace, so within one dataset
22
+ * there is one variant per class and the transform merely renames and re-keys —
23
+ * `void:distinctObjects` and every count stay exact. A dataset that genuinely
24
+ * mixes both namespaces on the same property collapses the variants by summing,
25
+ * which over-counts shared distinct objects; this is not optimized for.
30
26
  *
31
27
  * With no aliases configured the transform is a no-op.
28
+ */
29
+ export declare function mergeNamespaceVariants(namespaceAliases: readonly NamespaceAlias[]): QuadTransform<BeforeDatasetWriteContext>;
30
+ /**
31
+ * A {@link PipelinePlugin} that canonicalizes schema.org namespace variants in
32
+ * the VoID output — rewriting `http://schema.org/` to `https://schema.org/` —
33
+ * _and_ merges the duplicate partition nodes the two variants produced. Runs on
34
+ * the whole dataset's output via {@link PipelinePlugin.beforeDatasetWrite}, so
35
+ * the analysis queries stay unaware of namespace aliases.
36
+ *
37
+ * This does more than a plain namespace rewrite: rewriting the `void:class`
38
+ * objects alone would leave two `void:classPartition` nodes for the same class.
39
+ * For a non-VoID, blanket namespace rewrite (e.g. mapping instance data to an
40
+ * application profile), use `schemaOrgNormalizationPlugin` from `@lde/pipeline`.
41
+ */
42
+ export declare function schemaOrgPartitionMergePlugin(): PipelinePlugin;
43
+ /**
44
+ * A {@link PipelinePlugin} that canonicalizes the given namespace aliases in the
45
+ * VoID output and merges the duplicate partition nodes their variants produced.
46
+ * Generic form of {@link schemaOrgPartitionMergePlugin}.
32
47
  *
33
- * @see substituteNormalizationMarkers for the query-time normalization used
34
- * where summing is not safe.
48
+ * Required stages: re-keying a datatype/language/object-class partition walks up
49
+ * its `cp pp → dp` chain, reading `void:class` and `void:property` that
50
+ * `classPartitions` and `classPropertySubjects` emit. Use this plugin with a
51
+ * stage set that includes both (as {@link voidStages} does); without them a
52
+ * void-ext partition cannot be re-keyed and its alias variants ship unmerged.
35
53
  */
36
- export declare function mergeNamespaceVariants(namespaceAliases: readonly NamespaceAlias[]): QuadTransform<ReaderContext>;
54
+ export declare function namespacePartitionMergePlugin(namespaceAliases: readonly NamespaceAlias[]): PipelinePlugin;
37
55
  //# sourceMappingURL=partitionMerge.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"partitionMerge.d.ts","sourceRoot":"","sources":["../src/partitionMerge.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,aAAa,EAClB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAyCvB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAiCG;AACH,wBAAgB,sBAAsB,CACpC,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,aAAa,CAAC,aAAa,CAAC,CAY9B"}
1
+ {"version":3,"file":"partitionMerge.d.ts","sourceRoot":"","sources":["../src/partitionMerge.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,yBAAyB,EAC9B,KAAK,cAAc,EACnB,KAAK,cAAc,EACnB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAqEvB;;;;;;;;;;;;;;;;;;;;;;;;;;GA0BG;AACH,wBAAgB,sBAAsB,CACpC,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,aAAa,CAAC,yBAAyB,CAAC,CAgB1C;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,6BAA6B,IAAI,cAAc,CAI9D;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,6BAA6B,CAC3C,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,cAAc,CAKhB"}
@@ -1,14 +1,17 @@
1
1
  import { canonicalizeIri, } from '@lde/pipeline';
2
2
  import { DataFactory } from 'n3';
3
- import { createHash } from 'node:crypto';
3
+ import { mintPartitionIri } from './partitionIri.js';
4
4
  const { namedNode, literal, quad } = DataFactory;
5
5
  const VOID = 'http://rdfs.org/ns/void#';
6
6
  const VOID_EXT = 'http://ldf.fi/void-ext#';
7
7
  const XSD_INTEGER = 'http://www.w3.org/2001/XMLSchema#integer';
8
+ const SCHEMA_HTTP = 'http://schema.org/';
9
+ const SCHEMA_HTTPS = 'https://schema.org/';
8
10
  const VOID_CLASS = `${VOID}class`;
9
11
  const VOID_PROPERTY = `${VOID}property`;
10
12
  const VOID_ENTITIES = `${VOID}entities`;
11
13
  const VOID_TRIPLES = `${VOID}triples`;
14
+ const VOID_DISTINCT_OBJECTS = `${VOID}distinctObjects`;
12
15
  const VOID_CLASS_PARTITION = `${VOID}classPartition`;
13
16
  const VOID_PROPERTY_PARTITION = `${VOID}propertyPartition`;
14
17
  const VOIDEXT_DATATYPE = `${VOID_EXT}datatype`;
@@ -16,11 +19,19 @@ const VOIDEXT_LANGUAGE = `${VOID_EXT}language`;
16
19
  const VOIDEXT_DATATYPE_PARTITION = `${VOID_EXT}datatypePartition`;
17
20
  const VOIDEXT_OBJECTCLASS_PARTITION = `${VOID_EXT}objectClassPartition`;
18
21
  const VOIDEXT_LANGUAGE_PARTITION = `${VOID_EXT}languagePartition`;
19
- // Predicates whose integer-literal objects are summed across merged partitions.
20
- // `void:distinctObjects` is deliberately excluded: distinct-object sets overlap
21
- // across namespace variants, so summing would over-count. It is normalized at
22
- // query time in `class-properties-objects.rq` and never reaches this transform.
23
- const NUMERIC_MEASURES = new Set([VOID_ENTITIES, VOID_TRIPLES]);
22
+ /**
23
+ * Integer-literal measures summed when partitions collapse.
24
+ * `void:distinctObjects` is included: a dataset that uses a single schema.org
25
+ * namespace has one variant per class, so there is nothing to combine and the
26
+ * count is unchanged; only a dataset that mixes both namespaces on the same
27
+ * property (rare) sums two distinct-object counts — an over-count we accept
28
+ * rather than optimize for.
29
+ */
30
+ const NUMERIC_MEASURES = new Set([
31
+ VOID_ENTITIES,
32
+ VOID_TRIPLES,
33
+ VOID_DISTINCT_OBJECTS,
34
+ ]);
24
35
  /** Structural links whose object is a child partition node. */
25
36
  const CHILD_LINKS = new Set([
26
37
  VOID_CLASS_PARTITION,
@@ -29,41 +40,50 @@ const CHILD_LINKS = new Set([
29
40
  VOIDEXT_OBJECTCLASS_PARTITION,
30
41
  VOIDEXT_LANGUAGE_PARTITION,
31
42
  ]);
43
+ /**
44
+ * Every predicate that describes a partition. A quad with one of these
45
+ * predicates is buffered for merging; every other quad streams through
46
+ * untouched, so the transform's memory is bounded by the VoID summary, not the
47
+ * dataset.
48
+ */
49
+ const PARTITION_VOCABULARY = new Set([
50
+ VOID_CLASS,
51
+ VOID_PROPERTY,
52
+ VOIDEXT_DATATYPE,
53
+ VOIDEXT_LANGUAGE,
54
+ // Derived, not re-listed: a measure added to NUMERIC_MEASURES must also be
55
+ // buffered here, or it would stream through un-summed.
56
+ ...NUMERIC_MEASURES,
57
+ ...CHILD_LINKS,
58
+ ]);
32
59
  /** `void:class` / `void:property` objects are IRIs subject to canonicalization. */
33
60
  const CANONICALIZED_OBJECT_PREDICATES = new Set([VOID_CLASS, VOID_PROPERTY]);
34
61
  /**
35
- * A {@link QuadTransform} that merges the `void:classPartition` /
36
- * `void:propertyPartition` subtrees of namespace-alias variants (e.g.
37
- * `http://schema.org/CreativeWork` and `https://schema.org/CreativeWork`) into
38
- * a single partition per canonical class/property.
62
+ * A {@link QuadTransform} for the {@link PipelinePlugin.beforeDatasetWrite} hook
63
+ * that merges the `void:classPartition` / `void:propertyPartition` subtrees of
64
+ * namespace-alias variants (e.g. `http://schema.org/CreativeWork` and
65
+ * `https://schema.org/CreativeWork`) into one partition per canonical
66
+ * class/property.
39
67
  *
40
68
  * VoID partitions are keyed by an opaque `MD5(class[, property[, …]])` IRI, so
41
- * two namespace variants yield two partition nodes that both, after
42
- * canonicalization, describe the same class. This transform re-mints every
43
- * partition IRI from its **canonical** key components (replicating the queries’
44
- * SPARQL `MD5(CONCAT(STR(…)))`), collapses the duplicates, and **sums** their
45
- * numeric measures (`void:entities`, `void:triples`).
69
+ * two namespace variants produce two partition nodes that, once the class IRI is
70
+ * canonicalized, describe the same class. Seeing a whole dataset's output at
71
+ * once, this transform re-mints every partition IRI from its **canonical** key
72
+ * components via {@link mintPartitionIri} — the single source of truth the
73
+ * queries' SPARQL minting is also generated from — collapses the duplicates,
74
+ * and sums their numeric measures.
46
75
  *
47
- * ## Correctness assumptions
76
+ * It streams: only partition quads are buffered (bounded by the summary — a
77
+ * handful of classes × properties, not the dataset), and every other quad
78
+ * passes straight through.
48
79
  *
49
- * Summing pre-aggregated counts is exact only under these assumptions; the
50
- * queries whose measures cannot be safely summed keep their normalization at
51
- * query time instead (notably `class-properties-objects.rq`’s
52
- * `void:distinctObjects`, which this transform never sees):
53
- *
54
- * - **Subject/class disjointness** — no resource is typed under two namespace
55
- * variants of the same class. Guards the `void:entities` sum on class
56
- * partitions and every `void:triples` sum (a doubly-typed resource’s triples
57
- * would otherwise count under both variants).
58
- * - **Predicate-namespace disjointness** — no subject uses two namespace
59
- * variants of the same property (e.g. both `http://schema.org/name` and
60
- * `https://schema.org/name`). Guards the `void:entities` sum on property
61
- * partitions.
80
+ * Datasets typically use a single schema.org namespace, so within one dataset
81
+ * there is one variant per class and the transform merely renames and re-keys —
82
+ * `void:distinctObjects` and every count stay exact. A dataset that genuinely
83
+ * mixes both namespaces on the same property collapses the variants by summing,
84
+ * which over-counts shared distinct objects; this is not optimized for.
62
85
  *
63
86
  * With no aliases configured the transform is a no-op.
64
- *
65
- * @see substituteNormalizationMarkers for the query-time normalization used
66
- * where summing is not safe.
67
87
  */
68
88
  export function mergeNamespaceVariants(namespaceAliases) {
69
89
  if (namespaceAliases.length === 0) {
@@ -71,11 +91,50 @@ export function mergeNamespaceVariants(namespaceAliases) {
71
91
  }
72
92
  return async function* (quads, { dataset }) {
73
93
  const datasetIri = dataset.iri.toString();
74
- const buffered = [];
94
+ const partitionQuads = [];
75
95
  for await (const q of quads) {
76
- buffered.push(q);
96
+ if (PARTITION_VOCABULARY.has(q.predicate.value)) {
97
+ partitionQuads.push(q);
98
+ }
99
+ else {
100
+ yield q;
101
+ }
77
102
  }
78
- yield* mergeBuffered(buffered, datasetIri, namespaceAliases);
103
+ yield* mergeBuffered(partitionQuads, datasetIri, namespaceAliases);
104
+ };
105
+ }
106
+ /**
107
+ * A {@link PipelinePlugin} that canonicalizes schema.org namespace variants in
108
+ * the VoID output — rewriting `http://schema.org/` to `https://schema.org/` —
109
+ * _and_ merges the duplicate partition nodes the two variants produced. Runs on
110
+ * the whole dataset's output via {@link PipelinePlugin.beforeDatasetWrite}, so
111
+ * the analysis queries stay unaware of namespace aliases.
112
+ *
113
+ * This does more than a plain namespace rewrite: rewriting the `void:class`
114
+ * objects alone would leave two `void:classPartition` nodes for the same class.
115
+ * For a non-VoID, blanket namespace rewrite (e.g. mapping instance data to an
116
+ * application profile), use `schemaOrgNormalizationPlugin` from `@lde/pipeline`.
117
+ */
118
+ export function schemaOrgPartitionMergePlugin() {
119
+ return namespacePartitionMergePlugin([
120
+ { canonical: SCHEMA_HTTPS, alias: SCHEMA_HTTP },
121
+ ]);
122
+ }
123
+ /**
124
+ * A {@link PipelinePlugin} that canonicalizes the given namespace aliases in the
125
+ * VoID output and merges the duplicate partition nodes their variants produced.
126
+ * Generic form of {@link schemaOrgPartitionMergePlugin}.
127
+ *
128
+ * Required stages: re-keying a datatype/language/object-class partition walks up
129
+ * its `cp → pp → dp` chain, reading `void:class` and `void:property` that
130
+ * `classPartitions` and `classPropertySubjects` emit. Use this plugin with a
131
+ * stage set that includes both (as {@link voidStages} does); without them a
132
+ * void-ext partition cannot be re-keyed and its alias variants ship unmerged.
133
+ */
134
+ export function namespacePartitionMergePlugin(namespaceAliases) {
135
+ return {
136
+ name: 'void-namespace-partition-merge',
137
+ beforeDatasetWrite: mergeNamespaceVariants(namespaceAliases),
79
138
  };
80
139
  }
81
140
  function* mergeBuffered(buffered, datasetIri, namespaceAliases) {
@@ -83,10 +142,17 @@ function* mergeBuffered(buffered, datasetIri, namespaceAliases) {
83
142
  const remap = buildRemap(nodes, datasetIri, namespaceAliases);
84
143
  const measureSums = new Map();
85
144
  const emitted = new Set();
86
- const structural = [];
87
145
  for (const original of buffered) {
88
146
  const subject = remapTerm(original.subject, remap);
89
- const object = canonicalizeObject(remapTerm(original.object, remap), original.predicate.value, namespaceAliases);
147
+ let object = remapTerm(original.object, remap);
148
+ // Canonicalize a void:class/void:property object only when its own
149
+ // partition node is being re-keyed. The top-level property partitions from
150
+ // entity-properties.rq are never merged (their parent is the dataset, not a
151
+ // class), so they keep the source namespace — consumers can still see which
152
+ // namespace the dataset actually uses (see ADR 7).
153
+ if (remap.has(original.subject.value)) {
154
+ object = canonicalizeObject(object, original.predicate.value, namespaceAliases);
155
+ }
90
156
  const rewritten = quad(subject, original.predicate, object, original.graph);
91
157
  if (NUMERIC_MEASURES.has(original.predicate.value)) {
92
158
  accumulateMeasure(measureSums, rewritten);
@@ -95,10 +161,9 @@ function* mergeBuffered(buffered, datasetIri, namespaceAliases) {
95
161
  const key = quadKey(rewritten);
96
162
  if (!emitted.has(key)) {
97
163
  emitted.add(key);
98
- structural.push(rewritten);
164
+ yield rewritten;
99
165
  }
100
166
  }
101
- yield* structural;
102
167
  for (const sum of measureSums.values()) {
103
168
  yield reconstructMeasure(sum);
104
169
  }
@@ -143,7 +208,7 @@ function buildRemap(nodes, datasetIri, namespaceAliases) {
143
208
  }
144
209
  /**
145
210
  * The canonical partition IRI for a node, or `undefined` if the node is not a
146
- * partition (no incoming structural link). Replicates the queries minting:
211
+ * partition (no incoming structural link). Replicates the queries' minting:
147
212
  * `<dataset>/.well-known/void#<prefix>-<MD5(STR(component)…)>`.
148
213
  */
149
214
  function canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases) {
@@ -155,20 +220,25 @@ function canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases) {
155
220
  switch (link.predicate) {
156
221
  case VOID_CLASS_PARTITION: {
157
222
  const klass = node.values.get(VOID_CLASS)?.value;
158
- return klass ? mint(datasetIri, 'class', [canon(klass)]) : undefined;
223
+ return klass
224
+ ? mintPartitionIri(datasetIri, 'class', [canon(klass)])
225
+ : undefined;
159
226
  }
160
227
  case VOID_PROPERTY_PARTITION: {
161
228
  const klass = classOf(link.parent);
162
229
  const property = node.values.get(VOID_PROPERTY)?.value;
163
230
  return klass && property
164
- ? mint(datasetIri, 'class-property', [canon(klass), canon(property)])
231
+ ? mintPartitionIri(datasetIri, 'class-property', [
232
+ canon(klass),
233
+ canon(property),
234
+ ])
165
235
  : undefined;
166
236
  }
167
237
  case VOIDEXT_DATATYPE_PARTITION: {
168
238
  const [klass, property] = classProperty(link.parent, nodes);
169
239
  const datatype = node.values.get(VOIDEXT_DATATYPE)?.value;
170
240
  return klass && property && datatype
171
- ? mint(datasetIri, 'datatype', [
241
+ ? mintPartitionIri(datasetIri, 'datatype', [
172
242
  canon(klass),
173
243
  canon(property),
174
244
  datatype,
@@ -179,7 +249,7 @@ function canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases) {
179
249
  const [klass, property] = classProperty(link.parent, nodes);
180
250
  const objectClass = node.values.get(VOID_CLASS)?.value;
181
251
  return klass && property && objectClass
182
- ? mint(datasetIri, 'object-class', [
252
+ ? mintPartitionIri(datasetIri, 'object-class', [
183
253
  canon(klass),
184
254
  canon(property),
185
255
  canon(objectClass),
@@ -190,7 +260,7 @@ function canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases) {
190
260
  const [klass, property] = classProperty(link.parent, nodes);
191
261
  const language = node.values.get(VOIDEXT_LANGUAGE)?.value;
192
262
  return klass && property && language !== undefined
193
- ? mint(datasetIri, 'language', [
263
+ ? mintPartitionIri(datasetIri, 'language', [
194
264
  canon(klass),
195
265
  canon(property),
196
266
  language,
@@ -211,10 +281,6 @@ function classProperty(propertyPartition, nodes) {
211
281
  : undefined;
212
282
  return [klass, property];
213
283
  }
214
- function mint(datasetIri, prefix, components) {
215
- const hash = createHash('md5').update(components.join('')).digest('hex');
216
- return `${datasetIri}/.well-known/void#${prefix}-${hash}`;
217
- }
218
284
  function remapTerm(term, remap) {
219
285
  if (term.termType === 'NamedNode') {
220
286
  const canonical = remap.get(term.value);
package/dist/stage.d.ts CHANGED
@@ -1,4 +1,4 @@
1
- import { Stage, type ReaderContext, type NamespaceAlias, type QuadTransform } from '@lde/pipeline';
1
+ import { Stage, type ReaderContext, type QuadTransform } from '@lde/pipeline';
2
2
  import type { Quad } from '@rdfjs/types';
3
3
  /**
4
4
  * Stable names for every VoID stage, equal to the underlying query filename.
@@ -46,27 +46,6 @@ export interface VoidStageOptions {
46
46
  * with these, built-in first.
47
47
  */
48
48
  transform?: VoidStageTransform;
49
- /**
50
- * Namespace pairs to treat as equivalent when partitioning by class and
51
- * property, so alias variants (e.g. `http://schema.org/CreativeWork` and
52
- * `https://schema.org/CreativeWork`) merge into a single partition instead
53
- * of two that both claim the canonical `void:class`.
54
- *
55
- * Merging happens after aggregation, in the {@link mergeNamespaceVariants}
56
- * transform, which sums `void:entities` / `void:triples`; only
57
- * `class-properties-objects.rq` normalizes at query time (its
58
- * `void:distinctObjects` cannot be recovered by summing). Correct under the
59
- * subject/class- and predicate-namespace-disjointness assumptions documented
60
- * on {@link mergeNamespaceVariants}. The top-level property partitions
61
- * ({@link detectVocabularies}’s `entity-properties.rq`) and `void:vocabulary`
62
- * detection keep the source namespaces, so consumers can still see which
63
- * namespace the dataset actually uses.
64
- *
65
- * Defaults to no aliases. To cover schema.org datasets that publish under
66
- * both `http://schema.org/` and `https://schema.org/`, pass
67
- * `[{ canonical: 'https://schema.org/', alias: 'http://schema.org/' }]`.
68
- */
69
- namespaceAliases?: readonly NamespaceAlias[];
70
49
  }
71
50
  /**
72
51
  * Options for per-class VoID stages that iterate over classes.
@@ -1 +1 @@
1
- {"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EAKL,KAAK,aAAa,EAElB,KAAK,cAAc,EACnB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAEvB,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAqBzC;;;;;;GAMG;AACH,eAAO,MAAM,gBAAgB;;;;;;;;;;;;;;;;;CAiBnB,CAAC;AAEX,sDAAsD;AACtD,MAAM,MAAM,aAAa,GACvB,CAAC,OAAO,gBAAgB,CAAC,CAAC,MAAM,OAAO,gBAAgB,CAAC,CAAC;AAE3D,2EAA2E;AAC3E,MAAM,MAAM,kBAAkB,GAC1B,aAAa,CAAC,aAAa,CAAC,GAC5B,aAAa,CAAC,aAAa,CAAC,EAAE,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,gBAAgB;IAC/B;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,kBAAkB,CAAC;IAC/B;;;;;;;;;;;;;;;;;;;OAmBG;IACH,gBAAgB,CAAC,EAAE,SAAS,cAAc,EAAE,CAAC;CAC9C;AAED;;;;;GAKG;AACH,MAAM,WAAW,wBAAyB,SAAQ,gBAAgB;IAChE,oEAAoE;IACpE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,+DAA+D;IAC/D,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,iFAAiF;IACjF,QAAQ,CAAC,EAAE,OAAO,CAAC;CACpB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,iBAAkB,SAAQ,IAAI,CAC7C,wBAAwB,EACxB,WAAW,CACZ;IACC,yEAAyE;IACzE,SAAS,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,CAAC;IACjD,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IACjC;;;;;;;OAOG;IACH,UAAU,CAAC,EAAE,OAAO,CAAC,MAAM,CAAC,aAAa,EAAE,kBAAkB,CAAC,CAAC,CAAC;CACjE;AAwFD,wBAAgB,gBAAgB,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAE3E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAOD,wBAAgB,mBAAmB,CACjC,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,aAAa,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKxE;AAED,wBAAgB,YAAY,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKvE;AAED,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,oBAAoB,CAClC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKzE;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAEzE;AAID,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAID,wBAAgB,SAAS,CACvB,WAAW,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,EACjD,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAOhB;AAED,MAAM,WAAW,yBAA0B,SAAQ,gBAAgB;IACjE,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CAClC;AAED,wBAAgB,kBAAkB,CAChC,OAAO,CAAC,EAAE,yBAAyB,GAClC,OAAO,CAAC,KAAK,CAAC,CAQhB;AAED;;;;;;;GAOG;AACH,wBAAsB,UAAU,CAC9B,OAAO,CAAC,EAAE,iBAAiB,GAC1B,OAAO,CAAC,KAAK,EAAE,CAAC,CAgDlB"}
1
+ {"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EAKL,KAAK,aAAa,EAElB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAEvB,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAgBzC;;;;;;GAMG;AACH,eAAO,MAAM,gBAAgB;;;;;;;;;;;;;;;;;CAiBnB,CAAC;AAEX,sDAAsD;AACtD,MAAM,MAAM,aAAa,GACvB,CAAC,OAAO,gBAAgB,CAAC,CAAC,MAAM,OAAO,gBAAgB,CAAC,CAAC;AAE3D,2EAA2E;AAC3E,MAAM,MAAM,kBAAkB,GAC1B,aAAa,CAAC,aAAa,CAAC,GAC5B,aAAa,CAAC,aAAa,CAAC,EAAE,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,gBAAgB;IAC/B;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,kBAAkB,CAAC;CAChC;AAED;;;;;GAKG;AACH,MAAM,WAAW,wBAAyB,SAAQ,gBAAgB;IAChE,oEAAoE;IACpE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,+DAA+D;IAC/D,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,iFAAiF;IACjF,QAAQ,CAAC,EAAE,OAAO,CAAC;CACpB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,iBAAkB,SAAQ,IAAI,CAC7C,wBAAwB,EACxB,WAAW,CACZ;IACC,yEAAyE;IACzE,SAAS,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,CAAC;IACjD,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IACjC;;;;;;;OAOG;IACH,UAAU,CAAC,EAAE,OAAO,CAAC,MAAM,CAAC,aAAa,EAAE,kBAAkB,CAAC,CAAC,CAAC;CACjE;AAkED,wBAAgB,gBAAgB,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAE3E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAE1E;AAOD,wBAAgB,mBAAmB,CACjC,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,aAAa,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKxE;AAED,wBAAgB,YAAY,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKvE;AAED,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,oBAAoB,CAClC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKzE;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAEzE;AAID,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAID,wBAAgB,SAAS,CACvB,WAAW,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,EACjD,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAOhB;AAED,MAAM,WAAW,yBAA0B,SAAQ,gBAAgB;IACjE,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CAClC;AAED,wBAAgB,kBAAkB,CAChC,OAAO,CAAC,EAAE,yBAAyB,GAClC,OAAO,CAAC,KAAK,CAAC,CAQhB;AAED;;;;;;;GAOG;AACH,wBAAsB,UAAU,CAC9B,OAAO,CAAC,EAAE,iBAAiB,GAC1B,OAAO,CAAC,KAAK,EAAE,CAAC,CAgDlB"}
package/dist/stage.js CHANGED
@@ -4,8 +4,7 @@ import { resolve, dirname } from 'node:path';
4
4
  import { fileURLToPath } from 'node:url';
5
5
  import { withVocabularies, defaultVocabularies, } from './vocabularyTransform.js';
6
6
  import { withUriSpaces } from './uriSpaceTransform.js';
7
- import { canonicalizeClassBindings, substituteNormalizationMarkers, withAliasVariantBindings, } from './namespaceAliases.js';
8
- import { mergeNamespaceVariants } from './partitionMerge.js';
7
+ import { substituteMintMarkers } from './partitionIri.js';
9
8
  const queriesDir = resolve(dirname(fileURLToPath(import.meta.url)), '..', 'queries');
10
9
  /**
11
10
  * Stable names for every VoID stage, equal to the underlying query filename.
@@ -33,30 +32,15 @@ export const VOID_STAGE_NAMES = {
33
32
  objectUriSpace: 'object-uri-space.rq',
34
33
  };
35
34
  async function createVoidStage(filename, options) {
36
- const namespaceAliases = options?.namespaceAliases ?? [];
37
- const query = substituteNormalizationMarkers(await readQueryFile(resolve(queriesDir, filename)), namespaceAliases);
38
- const constructReader = new SparqlConstructReader({ query });
39
- // Partition stages merge their namespace-alias variants (e.g. http:// and
40
- // https://schema.org/) into one partition per canonical class/property.
41
- // Runs first, so any consumer transform sees the already-merged output.
42
- const mergeTransform = options?.mergePartitions && namespaceAliases.length > 0
43
- ? [mergeNamespaceVariants(namespaceAliases)]
44
- : [];
35
+ const query = substituteMintMarkers(await readQueryFile(resolve(queriesDir, filename)));
45
36
  const reader = {
46
- // Per-class stages receive canonical `?class` bindings from the selector;
47
- // expand each into one `?class` binding per alias variant so the query
48
- // (`?s a ?class`) matches instances under either namespace.
49
- reader: options?.perClass
50
- ? withAliasVariantBindings(constructReader, namespaceAliases)
51
- : constructReader,
52
- transform: [...mergeTransform, ...asTransforms(options?.transform)],
37
+ reader: new SparqlConstructReader({ query }),
38
+ transform: options?.transform,
53
39
  };
54
40
  return new Stage({
55
41
  name: filename,
56
42
  readers: reader,
57
- itemSelector: options?.perClass
58
- ? classSelector(namespaceAliases)
59
- : undefined,
43
+ itemSelector: options?.perClass ? classSelector() : undefined,
60
44
  batchSize: options?.batchSize,
61
45
  maxConcurrency: options?.maxConcurrency,
62
46
  expectsOutput: options?.expectsOutput,
@@ -68,7 +52,7 @@ function asTransforms(transform) {
68
52
  return [];
69
53
  return Array.isArray(transform) ? [...transform] : [transform];
70
54
  }
71
- function classSelector(namespaceAliases) {
55
+ function classSelector() {
72
56
  return {
73
57
  // Forward `options` so the Pipeline’s per-dataset TimeoutPolicy
74
58
  // reaches the inner SparqlItemSelector — without this the adaptive
@@ -86,10 +70,9 @@ function classSelector(namespaceAliases) {
86
70
  `WHERE { ${subjectFilter} ?s a ?class . }`,
87
71
  'LIMIT 1000',
88
72
  ].join('\n');
89
- const rows = new SparqlItemSelector({
73
+ return new SparqlItemSelector({
90
74
  query: selectorQuery,
91
75
  }).select(distribution, batchSize, options);
92
- return canonicalizeClassBindings(rows, namespaceAliases);
93
76
  },
94
77
  };
95
78
  }
@@ -98,10 +81,7 @@ export function subjectUriSpaces(options) {
98
81
  return createVoidStage(VOID_STAGE_NAMES.subjectUriSpace, options);
99
82
  }
100
83
  export function classPartitions(options) {
101
- return createVoidStage(VOID_STAGE_NAMES.classPartitions, {
102
- ...options,
103
- mergePartitions: true,
104
- });
84
+ return createVoidStage(VOID_STAGE_NAMES.classPartitions, options);
105
85
  }
106
86
  // Scalar-aggregate counts: each query is a single COUNT with no GROUP BY/HAVING,
107
87
  // so it always returns exactly one row. Zero output therefore means the endpoint
@@ -141,7 +121,6 @@ export function classPropertySubjects(options) {
141
121
  return createVoidStage(VOID_STAGE_NAMES.classPropertySubjects, {
142
122
  ...options,
143
123
  perClass: options?.perClass ?? true,
144
- mergePartitions: true,
145
124
  });
146
125
  }
147
126
  export function classPropertyObjects(options) {
@@ -164,21 +143,18 @@ export function perClassObjectClasses(options) {
164
143
  return createVoidStage(VOID_STAGE_NAMES.perClassObjectClasses, {
165
144
  ...options,
166
145
  perClass: options?.perClass ?? true,
167
- mergePartitions: true,
168
146
  });
169
147
  }
170
148
  export function perClassDatatypes(options) {
171
149
  return createVoidStage(VOID_STAGE_NAMES.perClassDatatypes, {
172
150
  ...options,
173
151
  perClass: options?.perClass ?? true,
174
- mergePartitions: true,
175
152
  });
176
153
  }
177
154
  export function perClassLanguages(options) {
178
155
  return createVoidStage(VOID_STAGE_NAMES.perClassLanguages, {
179
156
  ...options,
180
157
  perClass: options?.perClass ?? true,
181
- mergePartitions: true,
182
158
  });
183
159
  }
184
160
  // Stages with a built-in transform
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@lde/pipeline-void",
3
- "version": "0.31.4",
3
+ "version": "0.32.0",
4
4
  "description": "VOiD (Vocabulary of Interlinked Datasets) statistical analysis for RDF datasets",
5
5
  "repository": {
6
6
  "url": "git+https://github.com/ldelements/lde.git",
@@ -34,6 +34,6 @@
34
34
  },
35
35
  "peerDependencies": {
36
36
  "@lde/dataset": "^0.7.8",
37
- "@lde/pipeline": "^0.33.4"
37
+ "@lde/pipeline": "^0.34.0"
38
38
  }
39
39
  }
@@ -3,17 +3,17 @@ PREFIX void: <http://rdfs.org/ns/void#>
3
3
  CONSTRUCT {
4
4
  ?dataset a void:Dataset ;
5
5
  void:classPartition ?classPartition .
6
- ?classPartition void:class ?type ;
6
+ ?classPartition void:class ?class ;
7
7
  void:entities ?entities .
8
8
  }
9
9
  WHERE {
10
10
  {
11
- SELECT (COUNT(?type) AS ?entities) ?type {
11
+ SELECT (COUNT(?class) AS ?entities) ?class {
12
12
  #subjectFilter#
13
- ?s a ?type .
13
+ ?s a ?class .
14
14
  }
15
- GROUP BY ?type
15
+ GROUP BY ?class
16
16
  }
17
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?type)))) AS ?classPartition)
17
+ BIND(#mint:class# AS ?classPartition)
18
18
  }
19
19
  LIMIT 10000
@@ -5,27 +5,17 @@ CONSTRUCT {
5
5
  }
6
6
  WHERE {
7
7
  # Object counts only. Subject counts in class-properties-subjects.rq.
8
- #
9
- # Unlike the other per-class queries, this one normalizes the class and
10
- # property to their canonical namespace *before* the COUNT(DISTINCT ?o):
11
- # merging distinct-object counts by summing would over-count objects shared
12
- # between namespace variants (e.g. the same name under http:name and
13
- # https:name), so the dedup must happen here rather than in the
14
- # partition-merge transform. `?class` is bound per namespace variant by the
15
- # injected VALUES clause; `?property` is discovered.
16
8
  {
17
- SELECT ?canonicalClass ?canonicalProperty (COUNT(DISTINCT ?o) AS ?objects) {
9
+ SELECT ?class ?p (COUNT(DISTINCT ?o) AS ?objects) {
18
10
  {
19
- SELECT ?canonicalClass ?canonicalProperty ?o {
11
+ SELECT ?class ?p ?o {
20
12
  #subjectFilter#
21
- ?s a ?class ; ?property ?o .
22
- BIND(#normalized:class# AS ?canonicalClass)
23
- BIND(#normalized:property# AS ?canonicalProperty)
13
+ ?s a ?class ; ?p ?o .
24
14
  }
25
15
  }
26
16
  }
27
- GROUP BY ?canonicalClass ?canonicalProperty
17
+ GROUP BY ?class ?p
28
18
  }
29
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?canonicalClass), STR(?canonicalProperty))))) AS ?propertyPartition)
19
+ BIND(#mint:class-property# AS ?propertyPartition)
30
20
  }
31
21
  LIMIT 100000
@@ -21,7 +21,7 @@ WHERE {
21
21
  }
22
22
  GROUP BY ?class ?p
23
23
  }
24
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
25
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
24
+ BIND(#mint:class# AS ?classPartition)
25
+ BIND(#mint:class-property# AS ?propertyPartition)
26
26
  }
27
27
  LIMIT 100000
@@ -2,17 +2,7 @@ PREFIX void: <http://rdfs.org/ns/void#>
2
2
  PREFIX void-ext: <http://ldf.fi/void-ext#>
3
3
 
4
4
  CONSTRUCT {
5
- # Self-describing parent chain so the namespace-merge transform can re-key
6
- # this partition from its (class, property) components. The
7
- # `?dataset void:classPartition ?classPartition` link is what gives the class
8
- # partition an incoming edge, so the transform re-keys it too (without it, the
9
- # alias-variant class partition leaks unmerged).
10
- ?dataset a void:Dataset ;
11
- void:classPartition ?classPartition .
12
- ?classPartition void:class ?class ;
13
- void:propertyPartition ?propertyPartition .
14
- ?propertyPartition void:property ?p ;
15
- void-ext:datatypePartition ?datatypePartition .
5
+ ?propertyPartition void-ext:datatypePartition ?datatypePartition .
16
6
  ?datatypePartition
17
7
  void-ext:datatype ?dt ;
18
8
  void:triples ?count .
@@ -32,8 +22,7 @@ WHERE {
32
22
  }
33
23
  GROUP BY ?class ?p ?dt
34
24
  }
35
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
36
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
37
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#datatype-", MD5(CONCAT(STR(?class), STR(?p), STR(?dt))))) AS ?datatypePartition)
25
+ BIND(#mint:class-property# AS ?propertyPartition)
26
+ BIND(#mint:datatype# AS ?datatypePartition)
38
27
  }
39
28
  LIMIT 100000
@@ -2,17 +2,7 @@ PREFIX void: <http://rdfs.org/ns/void#>
2
2
  PREFIX void-ext: <http://ldf.fi/void-ext#>
3
3
 
4
4
  CONSTRUCT {
5
- # Self-describing parent chain so the namespace-merge transform can re-key
6
- # this partition from its (class, property) components. The
7
- # `?dataset void:classPartition ?classPartition` link is what gives the class
8
- # partition an incoming edge, so the transform re-keys it too (without it, the
9
- # alias-variant class partition leaks unmerged).
10
- ?dataset a void:Dataset ;
11
- void:classPartition ?classPartition .
12
- ?classPartition void:class ?class ;
13
- void:propertyPartition ?propertyPartition .
14
- ?propertyPartition void:property ?p ;
15
- void-ext:languagePartition ?languagePartition .
5
+ ?propertyPartition void-ext:languagePartition ?languagePartition .
16
6
  ?languagePartition
17
7
  void-ext:language ?lang ;
18
8
  void:triples ?count .
@@ -34,8 +24,7 @@ WHERE {
34
24
  }
35
25
  GROUP BY ?class ?p ?lang
36
26
  }
37
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
38
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
39
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#language-", MD5(CONCAT(STR(?class), STR(?p), ?lang)))) AS ?languagePartition)
27
+ BIND(#mint:class-property# AS ?propertyPartition)
28
+ BIND(#mint:language# AS ?languagePartition)
40
29
  }
41
30
  LIMIT 100000
@@ -2,17 +2,7 @@ PREFIX void: <http://rdfs.org/ns/void#>
2
2
  PREFIX void-ext: <http://ldf.fi/void-ext#>
3
3
 
4
4
  CONSTRUCT {
5
- # Self-describing parent chain so the namespace-merge transform can re-key
6
- # this partition from its (class, property) components. The
7
- # `?dataset void:classPartition ?classPartition` link is what gives the class
8
- # partition an incoming edge, so the transform re-keys it too (without it, the
9
- # alias-variant class partition leaks unmerged).
10
- ?dataset a void:Dataset ;
11
- void:classPartition ?classPartition .
12
- ?classPartition void:class ?class ;
13
- void:propertyPartition ?propertyPartition .
14
- ?propertyPartition void:property ?p ;
15
- void-ext:objectClassPartition ?objectClassPartition .
5
+ ?propertyPartition void-ext:objectClassPartition ?objectClassPartition .
16
6
  ?objectClassPartition
17
7
  void:class ?objectClass ;
18
8
  void:triples ?count .
@@ -32,8 +22,7 @@ WHERE {
32
22
  }
33
23
  GROUP BY ?class ?p ?objectClass
34
24
  }
35
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
36
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
37
- BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#object-class-", MD5(CONCAT(STR(?class), STR(?p), STR(?objectClass))))) AS ?objectClassPartition)
25
+ BIND(#mint:class-property# AS ?propertyPartition)
26
+ BIND(#mint:object-class# AS ?objectClassPartition)
38
27
  }
39
28
  LIMIT 100000
@@ -1,23 +0,0 @@
1
- import { type NamespaceAlias, type Reader, type VariableBindings } from '@lde/pipeline';
2
- /**
3
- * Replace every {@link NORMALIZATION_MARKER} in a query template with the
4
- * normalization expression for its raw variable.
5
- */
6
- export declare function substituteNormalizationMarkers(query: string, namespaceAliases: readonly NamespaceAlias[]): string;
7
- /**
8
- * Canonicalize and deduplicate the `?class` bindings a class selector
9
- * yields, so every namespace-alias variant of a class becomes one item and
10
- * its variants are queried together in one batch (split across batches,
11
- * each batch would emit its own partial counts for the same partition IRI).
12
- */
13
- export declare function canonicalizeClassBindings(rows: AsyncIterable<VariableBindings>, namespaceAliases: readonly NamespaceAlias[]): AsyncIterable<VariableBindings>;
14
- /**
15
- * Decorate a {@link Reader} so each canonical `?class` binding is expanded to
16
- * one `?class` binding per namespace-alias variant. The per-class VoID queries
17
- * match `?s a ?class`, so this makes them pick up instances typed under either
18
- * namespace; the partition-merge transform then collapses the variants. The
19
- * expansion happens within one executor call, so the variants of a class are
20
- * co-located in a single batch — a prerequisite for the per-stage merge.
21
- */
22
- export declare function withAliasVariantBindings(inner: Reader, namespaceAliases: readonly NamespaceAlias[]): Reader;
23
- //# sourceMappingURL=namespaceAliases.d.ts.map
@@ -1 +0,0 @@
1
- {"version":3,"file":"namespaceAliases.d.ts","sourceRoot":"","sources":["../src/namespaceAliases.ts"],"names":[],"mappings":"AAAA,OAAO,EAGL,KAAK,cAAc,EACnB,KAAK,MAAM,EACX,KAAK,gBAAgB,EACtB,MAAM,eAAe,CAAC;AAcvB;;;GAGG;AACH,wBAAgB,8BAA8B,CAC5C,KAAK,EAAE,MAAM,EACb,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,MAAM,CAIR;AAED;;;;;GAKG;AACH,wBAAuB,yBAAyB,CAC9C,IAAI,EAAE,aAAa,CAAC,gBAAgB,CAAC,EACrC,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,aAAa,CAAC,gBAAgB,CAAC,CAQjC;AAED;;;;;;;GAOG;AACH,wBAAgB,wBAAwB,CACtC,KAAK,EAAE,MAAM,EACb,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,MAAM,CAWR"}
@@ -1,77 +0,0 @@
1
- import { aliasVariants, canonicalizeIri, } from '@lde/pipeline';
2
- import { assertSafeIri } from '@lde/dataset';
3
- import { DataFactory } from 'n3';
4
- const { namedNode } = DataFactory;
5
- /**
6
- * Marker in a VoID query template that expands to a SPARQL expression
7
- * normalizing the named raw variable to its canonical namespace, e.g.
8
- * `BIND(#normalized:rawClass# AS ?class)`. With no aliases configured the
9
- * marker expands to the raw variable itself.
10
- */
11
- const NORMALIZATION_MARKER = /#normalized:([A-Za-z][A-Za-z0-9]*)#/g;
12
- /**
13
- * Replace every {@link NORMALIZATION_MARKER} in a query template with the
14
- * normalization expression for its raw variable.
15
- */
16
- export function substituteNormalizationMarkers(query, namespaceAliases) {
17
- return query.replace(NORMALIZATION_MARKER, (_match, rawVariable) => normalizedExpression(rawVariable, namespaceAliases));
18
- }
19
- /**
20
- * Canonicalize and deduplicate the `?class` bindings a class selector
21
- * yields, so every namespace-alias variant of a class becomes one item and
22
- * its variants are queried together in one batch (split across batches,
23
- * each batch would emit its own partial counts for the same partition IRI).
24
- */
25
- export async function* canonicalizeClassBindings(rows, namespaceAliases) {
26
- const seen = new Set();
27
- for await (const row of rows) {
28
- const canonical = canonicalizeIri(row.class.value, namespaceAliases);
29
- if (seen.has(canonical))
30
- continue;
31
- seen.add(canonical);
32
- yield { class: namedNode(canonical) };
33
- }
34
- }
35
- /**
36
- * Decorate a {@link Reader} so each canonical `?class` binding is expanded to
37
- * one `?class` binding per namespace-alias variant. The per-class VoID queries
38
- * match `?s a ?class`, so this makes them pick up instances typed under either
39
- * namespace; the partition-merge transform then collapses the variants. The
40
- * expansion happens within one executor call, so the variants of a class are
41
- * co-located in a single batch — a prerequisite for the per-stage merge.
42
- */
43
- export function withAliasVariantBindings(inner, namespaceAliases) {
44
- return {
45
- read(dataset, distribution, options) {
46
- const bindings = options?.bindings?.flatMap((row) => aliasVariants(row.class.value, namespaceAliases).map((iri) => ({
47
- class: namedNode(iri),
48
- })));
49
- return inner.read(dataset, distribution, { ...options, bindings });
50
- },
51
- };
52
- }
53
- /**
54
- * SPARQL expression rewriting `?rawVariable` from any alias namespace to its
55
- * canonical namespace: nested `IF(STRSTARTS(...), IRI(CONCAT(...)), ...)`
56
- * per alias, or the bare variable when no aliases are configured.
57
- */
58
- function normalizedExpression(rawVariable, namespaceAliases) {
59
- let expression = `?${rawVariable}`;
60
- // Build from the inside out so the first alias is the outermost check.
61
- for (const { canonical, alias } of [...namespaceAliases].reverse()) {
62
- assertSafeSparqlString(canonical);
63
- assertSafeSparqlString(alias);
64
- expression = `IF(STRSTARTS(STR(?${rawVariable}), "${alias}"), IRI(CONCAT("${canonical}", STRAFTER(STR(?${rawVariable}), "${alias}"))), ${expression})`;
65
- }
66
- return expression;
67
- }
68
- /**
69
- * Namespaces are interpolated into double-quoted SPARQL string literals, so
70
- * beyond {@link assertSafeIri} they must not contain quotes or backslashes.
71
- */
72
- function assertSafeSparqlString(namespace) {
73
- assertSafeIri(namespace);
74
- if (namespace.includes('"') || namespace.includes('\\')) {
75
- throw new Error(`Namespace contains unsafe characters and cannot be interpolated into SPARQL: ${namespace}`);
76
- }
77
- }