@lde/pipeline-void 0.31.2 → 0.31.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +32 -8
- package/dist/index.d.ts +3 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -0
- package/dist/namespaceAliases.d.ts +23 -0
- package/dist/namespaceAliases.d.ts.map +1 -0
- package/dist/namespaceAliases.js +77 -0
- package/dist/partitionMerge.d.ts +37 -0
- package/dist/partitionMerge.d.ts.map +1 -0
- package/dist/partitionMerge.js +275 -0
- package/dist/stage.d.ts +22 -1
- package/dist/stage.d.ts.map +1 -1
- package/dist/stage.js +32 -7
- package/package.json +2 -2
- package/queries/class-properties-objects.rq +15 -5
- package/queries/class-property-datatypes.rq +12 -1
- package/queries/class-property-languages.rq +12 -1
- package/queries/class-property-object-classes.rq +12 -1
package/README.md
CHANGED
|
@@ -10,14 +10,15 @@ Returns all VoID stages in their recommended execution order. The ordering is op
|
|
|
10
10
|
|
|
11
11
|
Accepts an optional `VoidStagesOptions` object:
|
|
12
12
|
|
|
13
|
-
| Option
|
|
14
|
-
|
|
|
15
|
-
| `batchSize`
|
|
16
|
-
| `maxConcurrency`
|
|
17
|
-
| `perClass`
|
|
18
|
-
| `uriSpaces`
|
|
19
|
-
| `vocabularies`
|
|
20
|
-
| `transforms`
|
|
13
|
+
| Option | Default | Description |
|
|
14
|
+
| ------------------ | ------- | --------------------------------------------------------------------------------------------------------------- |
|
|
15
|
+
| `batchSize` | 10 | Maximum class bindings per reader call (per-class stages only) |
|
|
16
|
+
| `maxConcurrency` | 10 | Maximum concurrent in-flight reader batches (per-class stages only) |
|
|
17
|
+
| `perClass` | — | Override per-class iteration for all five per-class stages |
|
|
18
|
+
| `uriSpaces` | — | When provided, includes the object URI space stage |
|
|
19
|
+
| `vocabularies` | — | Additional vocabulary namespace URIs to detect beyond the built-in defaults |
|
|
20
|
+
| `transforms` | — | Transforms to attach to bundled stages, keyed by `VOID_STAGE_NAMES` (see [Stage transforms](#stage-transforms)) |
|
|
21
|
+
| `namespaceAliases` | — | Namespace pairs to treat as equivalent when partitioning (see [Namespace aliases](#namespace-aliases)) |
|
|
21
22
|
|
|
22
23
|
Per-request timeouts are configured at the `Pipeline` level via `PipelineOptions.timeout`, not per VoID stage.
|
|
23
24
|
|
|
@@ -72,6 +73,29 @@ Global and domain-specific factories accept `VoidStageOptions` (`transform`) and
|
|
|
72
73
|
| `detectVocabularies()` | [`entity-properties.rq`](queries/entity-properties.rq) — Entity properties with automatic `void:vocabulary` detection. Accepts `DetectVocabulariesOptions` with an optional `vocabularies` array to extend the built-in defaults. |
|
|
73
74
|
| `uriSpaces(uriSpaceMap)` | [`object-uri-space.rq`](queries/object-uri-space.rq) — Object URI namespace linksets, aggregated against a provided URI space map |
|
|
74
75
|
|
|
76
|
+
## Namespace aliases
|
|
77
|
+
|
|
78
|
+
Some vocabularies publish under both HTTP and HTTPS variants of the same namespace (notably schema.org), and datasets mix them. Without normalization, each variant gets its own `void:classPartition`/`void:propertyPartition`, so consumers see two partitions for what is semantically one class.
|
|
79
|
+
|
|
80
|
+
Pass `namespaceAliases` to merge the variants into one partition per canonical class/property:
|
|
81
|
+
|
|
82
|
+
```typescript
|
|
83
|
+
const stages = await voidStages({
|
|
84
|
+
namespaceAliases: [
|
|
85
|
+
{ canonical: 'https://schema.org/', alias: 'http://schema.org/' },
|
|
86
|
+
],
|
|
87
|
+
});
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Merging happens **after** aggregation, in a single partition-merge transform (`mergeNamespaceVariants`) attached to the class and property partition stages: it re-mints each partition IRI from its canonical key components, collapses the duplicates, and sums `void:entities` / `void:triples`. The analysis queries stay plain — except `class-properties-objects.rq`, which normalizes at query time because its `void:distinctObjects` is a distinct-count over object values that overlap across variants and so cannot be recovered by summing. See [ADR 7](../../docs/decisions/0007-merge-namespace-alias-partitions.md).
|
|
91
|
+
|
|
92
|
+
Summing pre-aggregated counts is exact only under two assumptions, both of which hold for the schema.org datasets that motivate this (the variants appear as disjoint subsets):
|
|
93
|
+
|
|
94
|
+
- **Subject/class disjointness** — no resource is typed under both namespace variants of a class (guards the class-partition and triple-count sums).
|
|
95
|
+
- **Predicate-namespace disjointness** — no subject uses both variants of the same property (guards the property-partition entity sum).
|
|
96
|
+
|
|
97
|
+
A resource typed under both variants over-counts its class entities. With no aliases configured the transform is a no-op. The top-level property partitions from `entity-properties.rq` and `void:vocabulary` detection keep the source namespaces, so consumers can still see which namespace the dataset actually uses.
|
|
98
|
+
|
|
75
99
|
## Stage transforms
|
|
76
100
|
|
|
77
101
|
A VoID stage decorates its reader’s output with a `QuadTransform<ReaderContext>` attached as data (see [@lde/pipeline](../pipeline)’s extension model and [ADR 2](../../docs/decisions/0002-unify-pipeline-extension-on-quad-transforms.md)). It runs once per reader call and may fire its own SPARQL queries against the `distribution` in scope — so write it to accept being called more than once: a global stage calls it once over the complete output, a per-class stage with batching enabled once per batch (one class at `batchSize: 1`).
|
package/dist/index.d.ts
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
export { Stage, NotSupported } from '@lde/pipeline';
|
|
2
|
-
export type { AttachedReader, ReaderContext, QuadTransform, } from '@lde/pipeline';
|
|
2
|
+
export type { AttachedReader, NamespaceAlias, ReaderContext, QuadTransform, } from '@lde/pipeline';
|
|
3
3
|
export * from './stage.js';
|
|
4
|
+
export * from './namespaceAliases.js';
|
|
5
|
+
export * from './partitionMerge.js';
|
|
4
6
|
export * from './vocabularyTransform.js';
|
|
5
7
|
export * from './uriSpaceTransform.js';
|
|
6
8
|
//# sourceMappingURL=index.d.ts.map
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AACpD,YAAY,EACV,cAAc,EACd,aAAa,EACb,aAAa,GACd,MAAM,eAAe,CAAC;AACvB,cAAc,YAAY,CAAC;AAC3B,cAAc,0BAA0B,CAAC;AACzC,cAAc,wBAAwB,CAAC"}
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,EAAE,YAAY,EAAE,MAAM,eAAe,CAAC;AACpD,YAAY,EACV,cAAc,EACd,cAAc,EACd,aAAa,EACb,aAAa,GACd,MAAM,eAAe,CAAC;AACvB,cAAc,YAAY,CAAC;AAC3B,cAAc,uBAAuB,CAAC;AACtC,cAAc,qBAAqB,CAAC;AACpC,cAAc,0BAA0B,CAAC;AACzC,cAAc,wBAAwB,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
import { type NamespaceAlias, type Reader, type VariableBindings } from '@lde/pipeline';
|
|
2
|
+
/**
|
|
3
|
+
* Replace every {@link NORMALIZATION_MARKER} in a query template with the
|
|
4
|
+
* normalization expression for its raw variable.
|
|
5
|
+
*/
|
|
6
|
+
export declare function substituteNormalizationMarkers(query: string, namespaceAliases: readonly NamespaceAlias[]): string;
|
|
7
|
+
/**
|
|
8
|
+
* Canonicalize and deduplicate the `?class` bindings a class selector
|
|
9
|
+
* yields, so every namespace-alias variant of a class becomes one item and
|
|
10
|
+
* its variants are queried together in one batch (split across batches,
|
|
11
|
+
* each batch would emit its own partial counts for the same partition IRI).
|
|
12
|
+
*/
|
|
13
|
+
export declare function canonicalizeClassBindings(rows: AsyncIterable<VariableBindings>, namespaceAliases: readonly NamespaceAlias[]): AsyncIterable<VariableBindings>;
|
|
14
|
+
/**
|
|
15
|
+
* Decorate a {@link Reader} so each canonical `?class` binding is expanded to
|
|
16
|
+
* one `?class` binding per namespace-alias variant. The per-class VoID queries
|
|
17
|
+
* match `?s a ?class`, so this makes them pick up instances typed under either
|
|
18
|
+
* namespace; the partition-merge transform then collapses the variants. The
|
|
19
|
+
* expansion happens within one executor call, so the variants of a class are
|
|
20
|
+
* co-located in a single batch — a prerequisite for the per-stage merge.
|
|
21
|
+
*/
|
|
22
|
+
export declare function withAliasVariantBindings(inner: Reader, namespaceAliases: readonly NamespaceAlias[]): Reader;
|
|
23
|
+
//# sourceMappingURL=namespaceAliases.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"namespaceAliases.d.ts","sourceRoot":"","sources":["../src/namespaceAliases.ts"],"names":[],"mappings":"AAAA,OAAO,EAGL,KAAK,cAAc,EACnB,KAAK,MAAM,EACX,KAAK,gBAAgB,EACtB,MAAM,eAAe,CAAC;AAcvB;;;GAGG;AACH,wBAAgB,8BAA8B,CAC5C,KAAK,EAAE,MAAM,EACb,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,MAAM,CAIR;AAED;;;;;GAKG;AACH,wBAAuB,yBAAyB,CAC9C,IAAI,EAAE,aAAa,CAAC,gBAAgB,CAAC,EACrC,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,aAAa,CAAC,gBAAgB,CAAC,CAQjC;AAED;;;;;;;GAOG;AACH,wBAAgB,wBAAwB,CACtC,KAAK,EAAE,MAAM,EACb,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,MAAM,CAWR"}
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
import { aliasVariants, canonicalizeIri, } from '@lde/pipeline';
|
|
2
|
+
import { assertSafeIri } from '@lde/dataset';
|
|
3
|
+
import { DataFactory } from 'n3';
|
|
4
|
+
const { namedNode } = DataFactory;
|
|
5
|
+
/**
|
|
6
|
+
* Marker in a VoID query template that expands to a SPARQL expression
|
|
7
|
+
* normalizing the named raw variable to its canonical namespace, e.g.
|
|
8
|
+
* `BIND(#normalized:rawClass# AS ?class)`. With no aliases configured the
|
|
9
|
+
* marker expands to the raw variable itself.
|
|
10
|
+
*/
|
|
11
|
+
const NORMALIZATION_MARKER = /#normalized:([A-Za-z][A-Za-z0-9]*)#/g;
|
|
12
|
+
/**
|
|
13
|
+
* Replace every {@link NORMALIZATION_MARKER} in a query template with the
|
|
14
|
+
* normalization expression for its raw variable.
|
|
15
|
+
*/
|
|
16
|
+
export function substituteNormalizationMarkers(query, namespaceAliases) {
|
|
17
|
+
return query.replace(NORMALIZATION_MARKER, (_match, rawVariable) => normalizedExpression(rawVariable, namespaceAliases));
|
|
18
|
+
}
|
|
19
|
+
/**
|
|
20
|
+
* Canonicalize and deduplicate the `?class` bindings a class selector
|
|
21
|
+
* yields, so every namespace-alias variant of a class becomes one item and
|
|
22
|
+
* its variants are queried together in one batch (split across batches,
|
|
23
|
+
* each batch would emit its own partial counts for the same partition IRI).
|
|
24
|
+
*/
|
|
25
|
+
export async function* canonicalizeClassBindings(rows, namespaceAliases) {
|
|
26
|
+
const seen = new Set();
|
|
27
|
+
for await (const row of rows) {
|
|
28
|
+
const canonical = canonicalizeIri(row.class.value, namespaceAliases);
|
|
29
|
+
if (seen.has(canonical))
|
|
30
|
+
continue;
|
|
31
|
+
seen.add(canonical);
|
|
32
|
+
yield { class: namedNode(canonical) };
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
/**
|
|
36
|
+
* Decorate a {@link Reader} so each canonical `?class` binding is expanded to
|
|
37
|
+
* one `?class` binding per namespace-alias variant. The per-class VoID queries
|
|
38
|
+
* match `?s a ?class`, so this makes them pick up instances typed under either
|
|
39
|
+
* namespace; the partition-merge transform then collapses the variants. The
|
|
40
|
+
* expansion happens within one executor call, so the variants of a class are
|
|
41
|
+
* co-located in a single batch — a prerequisite for the per-stage merge.
|
|
42
|
+
*/
|
|
43
|
+
export function withAliasVariantBindings(inner, namespaceAliases) {
|
|
44
|
+
return {
|
|
45
|
+
read(dataset, distribution, options) {
|
|
46
|
+
const bindings = options?.bindings?.flatMap((row) => aliasVariants(row.class.value, namespaceAliases).map((iri) => ({
|
|
47
|
+
class: namedNode(iri),
|
|
48
|
+
})));
|
|
49
|
+
return inner.read(dataset, distribution, { ...options, bindings });
|
|
50
|
+
},
|
|
51
|
+
};
|
|
52
|
+
}
|
|
53
|
+
/**
|
|
54
|
+
* SPARQL expression rewriting `?rawVariable` from any alias namespace to its
|
|
55
|
+
* canonical namespace: nested `IF(STRSTARTS(...), IRI(CONCAT(...)), ...)`
|
|
56
|
+
* per alias, or the bare variable when no aliases are configured.
|
|
57
|
+
*/
|
|
58
|
+
function normalizedExpression(rawVariable, namespaceAliases) {
|
|
59
|
+
let expression = `?${rawVariable}`;
|
|
60
|
+
// Build from the inside out so the first alias is the outermost check.
|
|
61
|
+
for (const { canonical, alias } of [...namespaceAliases].reverse()) {
|
|
62
|
+
assertSafeSparqlString(canonical);
|
|
63
|
+
assertSafeSparqlString(alias);
|
|
64
|
+
expression = `IF(STRSTARTS(STR(?${rawVariable}), "${alias}"), IRI(CONCAT("${canonical}", STRAFTER(STR(?${rawVariable}), "${alias}"))), ${expression})`;
|
|
65
|
+
}
|
|
66
|
+
return expression;
|
|
67
|
+
}
|
|
68
|
+
/**
|
|
69
|
+
* Namespaces are interpolated into double-quoted SPARQL string literals, so
|
|
70
|
+
* beyond {@link assertSafeIri} they must not contain quotes or backslashes.
|
|
71
|
+
*/
|
|
72
|
+
function assertSafeSparqlString(namespace) {
|
|
73
|
+
assertSafeIri(namespace);
|
|
74
|
+
if (namespace.includes('"') || namespace.includes('\\')) {
|
|
75
|
+
throw new Error(`Namespace contains unsafe characters and cannot be interpolated into SPARQL: ${namespace}`);
|
|
76
|
+
}
|
|
77
|
+
}
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
import { type NamespaceAlias, type QuadTransform, type ReaderContext } from '@lde/pipeline';
|
|
2
|
+
/**
|
|
3
|
+
* A {@link QuadTransform} that merges the `void:classPartition` /
|
|
4
|
+
* `void:propertyPartition` subtrees of namespace-alias variants (e.g.
|
|
5
|
+
* `http://schema.org/CreativeWork` and `https://schema.org/CreativeWork`) into
|
|
6
|
+
* a single partition per canonical class/property.
|
|
7
|
+
*
|
|
8
|
+
* VoID partitions are keyed by an opaque `MD5(class[, property[, …]])` IRI, so
|
|
9
|
+
* two namespace variants yield two partition nodes that both, after
|
|
10
|
+
* canonicalization, describe the same class. This transform re-mints every
|
|
11
|
+
* partition IRI from its **canonical** key components (replicating the queries’
|
|
12
|
+
* SPARQL `MD5(CONCAT(STR(…)))`), collapses the duplicates, and **sums** their
|
|
13
|
+
* numeric measures (`void:entities`, `void:triples`).
|
|
14
|
+
*
|
|
15
|
+
* ## Correctness assumptions
|
|
16
|
+
*
|
|
17
|
+
* Summing pre-aggregated counts is exact only under these assumptions; the
|
|
18
|
+
* queries whose measures cannot be safely summed keep their normalization at
|
|
19
|
+
* query time instead (notably `class-properties-objects.rq`’s
|
|
20
|
+
* `void:distinctObjects`, which this transform never sees):
|
|
21
|
+
*
|
|
22
|
+
* - **Subject/class disjointness** — no resource is typed under two namespace
|
|
23
|
+
* variants of the same class. Guards the `void:entities` sum on class
|
|
24
|
+
* partitions and every `void:triples` sum (a doubly-typed resource’s triples
|
|
25
|
+
* would otherwise count under both variants).
|
|
26
|
+
* - **Predicate-namespace disjointness** — no subject uses two namespace
|
|
27
|
+
* variants of the same property (e.g. both `http://schema.org/name` and
|
|
28
|
+
* `https://schema.org/name`). Guards the `void:entities` sum on property
|
|
29
|
+
* partitions.
|
|
30
|
+
*
|
|
31
|
+
* With no aliases configured the transform is a no-op.
|
|
32
|
+
*
|
|
33
|
+
* @see substituteNormalizationMarkers for the query-time normalization used
|
|
34
|
+
* where summing is not safe.
|
|
35
|
+
*/
|
|
36
|
+
export declare function mergeNamespaceVariants(namespaceAliases: readonly NamespaceAlias[]): QuadTransform<ReaderContext>;
|
|
37
|
+
//# sourceMappingURL=partitionMerge.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"partitionMerge.d.ts","sourceRoot":"","sources":["../src/partitionMerge.ts"],"names":[],"mappings":"AAAA,OAAO,EAEL,KAAK,cAAc,EACnB,KAAK,aAAa,EAClB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAyCvB;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAiCG;AACH,wBAAgB,sBAAsB,CACpC,gBAAgB,EAAE,SAAS,cAAc,EAAE,GAC1C,aAAa,CAAC,aAAa,CAAC,CAY9B"}
|
|
@@ -0,0 +1,275 @@
|
|
|
1
|
+
import { canonicalizeIri, } from '@lde/pipeline';
|
|
2
|
+
import { DataFactory } from 'n3';
|
|
3
|
+
import { createHash } from 'node:crypto';
|
|
4
|
+
const { namedNode, literal, quad } = DataFactory;
|
|
5
|
+
const VOID = 'http://rdfs.org/ns/void#';
|
|
6
|
+
const VOID_EXT = 'http://ldf.fi/void-ext#';
|
|
7
|
+
const XSD_INTEGER = 'http://www.w3.org/2001/XMLSchema#integer';
|
|
8
|
+
const VOID_CLASS = `${VOID}class`;
|
|
9
|
+
const VOID_PROPERTY = `${VOID}property`;
|
|
10
|
+
const VOID_ENTITIES = `${VOID}entities`;
|
|
11
|
+
const VOID_TRIPLES = `${VOID}triples`;
|
|
12
|
+
const VOID_CLASS_PARTITION = `${VOID}classPartition`;
|
|
13
|
+
const VOID_PROPERTY_PARTITION = `${VOID}propertyPartition`;
|
|
14
|
+
const VOIDEXT_DATATYPE = `${VOID_EXT}datatype`;
|
|
15
|
+
const VOIDEXT_LANGUAGE = `${VOID_EXT}language`;
|
|
16
|
+
const VOIDEXT_DATATYPE_PARTITION = `${VOID_EXT}datatypePartition`;
|
|
17
|
+
const VOIDEXT_OBJECTCLASS_PARTITION = `${VOID_EXT}objectClassPartition`;
|
|
18
|
+
const VOIDEXT_LANGUAGE_PARTITION = `${VOID_EXT}languagePartition`;
|
|
19
|
+
// Predicates whose integer-literal objects are summed across merged partitions.
|
|
20
|
+
// `void:distinctObjects` is deliberately excluded: distinct-object sets overlap
|
|
21
|
+
// across namespace variants, so summing would over-count. It is normalized at
|
|
22
|
+
// query time in `class-properties-objects.rq` and never reaches this transform.
|
|
23
|
+
const NUMERIC_MEASURES = new Set([VOID_ENTITIES, VOID_TRIPLES]);
|
|
24
|
+
/** Structural links whose object is a child partition node. */
|
|
25
|
+
const CHILD_LINKS = new Set([
|
|
26
|
+
VOID_CLASS_PARTITION,
|
|
27
|
+
VOID_PROPERTY_PARTITION,
|
|
28
|
+
VOIDEXT_DATATYPE_PARTITION,
|
|
29
|
+
VOIDEXT_OBJECTCLASS_PARTITION,
|
|
30
|
+
VOIDEXT_LANGUAGE_PARTITION,
|
|
31
|
+
]);
|
|
32
|
+
/** `void:class` / `void:property` objects are IRIs subject to canonicalization. */
|
|
33
|
+
const CANONICALIZED_OBJECT_PREDICATES = new Set([VOID_CLASS, VOID_PROPERTY]);
|
|
34
|
+
/**
|
|
35
|
+
* A {@link QuadTransform} that merges the `void:classPartition` /
|
|
36
|
+
* `void:propertyPartition` subtrees of namespace-alias variants (e.g.
|
|
37
|
+
* `http://schema.org/CreativeWork` and `https://schema.org/CreativeWork`) into
|
|
38
|
+
* a single partition per canonical class/property.
|
|
39
|
+
*
|
|
40
|
+
* VoID partitions are keyed by an opaque `MD5(class[, property[, …]])` IRI, so
|
|
41
|
+
* two namespace variants yield two partition nodes that both, after
|
|
42
|
+
* canonicalization, describe the same class. This transform re-mints every
|
|
43
|
+
* partition IRI from its **canonical** key components (replicating the queries’
|
|
44
|
+
* SPARQL `MD5(CONCAT(STR(…)))`), collapses the duplicates, and **sums** their
|
|
45
|
+
* numeric measures (`void:entities`, `void:triples`).
|
|
46
|
+
*
|
|
47
|
+
* ## Correctness assumptions
|
|
48
|
+
*
|
|
49
|
+
* Summing pre-aggregated counts is exact only under these assumptions; the
|
|
50
|
+
* queries whose measures cannot be safely summed keep their normalization at
|
|
51
|
+
* query time instead (notably `class-properties-objects.rq`’s
|
|
52
|
+
* `void:distinctObjects`, which this transform never sees):
|
|
53
|
+
*
|
|
54
|
+
* - **Subject/class disjointness** — no resource is typed under two namespace
|
|
55
|
+
* variants of the same class. Guards the `void:entities` sum on class
|
|
56
|
+
* partitions and every `void:triples` sum (a doubly-typed resource’s triples
|
|
57
|
+
* would otherwise count under both variants).
|
|
58
|
+
* - **Predicate-namespace disjointness** — no subject uses two namespace
|
|
59
|
+
* variants of the same property (e.g. both `http://schema.org/name` and
|
|
60
|
+
* `https://schema.org/name`). Guards the `void:entities` sum on property
|
|
61
|
+
* partitions.
|
|
62
|
+
*
|
|
63
|
+
* With no aliases configured the transform is a no-op.
|
|
64
|
+
*
|
|
65
|
+
* @see substituteNormalizationMarkers for the query-time normalization used
|
|
66
|
+
* where summing is not safe.
|
|
67
|
+
*/
|
|
68
|
+
export function mergeNamespaceVariants(namespaceAliases) {
|
|
69
|
+
if (namespaceAliases.length === 0) {
|
|
70
|
+
return (quads) => quads;
|
|
71
|
+
}
|
|
72
|
+
return async function* (quads, { dataset }) {
|
|
73
|
+
const datasetIri = dataset.iri.toString();
|
|
74
|
+
const buffered = [];
|
|
75
|
+
for await (const q of quads) {
|
|
76
|
+
buffered.push(q);
|
|
77
|
+
}
|
|
78
|
+
yield* mergeBuffered(buffered, datasetIri, namespaceAliases);
|
|
79
|
+
};
|
|
80
|
+
}
|
|
81
|
+
function* mergeBuffered(buffered, datasetIri, namespaceAliases) {
|
|
82
|
+
const nodes = indexNodes(buffered);
|
|
83
|
+
const remap = buildRemap(nodes, datasetIri, namespaceAliases);
|
|
84
|
+
const measureSums = new Map();
|
|
85
|
+
const emitted = new Set();
|
|
86
|
+
const structural = [];
|
|
87
|
+
for (const original of buffered) {
|
|
88
|
+
const subject = remapTerm(original.subject, remap);
|
|
89
|
+
const object = canonicalizeObject(remapTerm(original.object, remap), original.predicate.value, namespaceAliases);
|
|
90
|
+
const rewritten = quad(subject, original.predicate, object, original.graph);
|
|
91
|
+
if (NUMERIC_MEASURES.has(original.predicate.value)) {
|
|
92
|
+
accumulateMeasure(measureSums, rewritten);
|
|
93
|
+
continue;
|
|
94
|
+
}
|
|
95
|
+
const key = quadKey(rewritten);
|
|
96
|
+
if (!emitted.has(key)) {
|
|
97
|
+
emitted.add(key);
|
|
98
|
+
structural.push(rewritten);
|
|
99
|
+
}
|
|
100
|
+
}
|
|
101
|
+
yield* structural;
|
|
102
|
+
for (const sum of measureSums.values()) {
|
|
103
|
+
yield reconstructMeasure(sum);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
function indexNodes(buffered) {
|
|
107
|
+
const nodes = new Map();
|
|
108
|
+
const nodeOf = (value) => {
|
|
109
|
+
let node = nodes.get(value);
|
|
110
|
+
if (!node) {
|
|
111
|
+
node = { values: new Map() };
|
|
112
|
+
nodes.set(value, node);
|
|
113
|
+
}
|
|
114
|
+
return node;
|
|
115
|
+
};
|
|
116
|
+
for (const q of buffered) {
|
|
117
|
+
if (CHILD_LINKS.has(q.predicate.value) &&
|
|
118
|
+
q.object.termType === 'NamedNode') {
|
|
119
|
+
nodeOf(q.object.value).incomingLink = {
|
|
120
|
+
predicate: q.predicate.value,
|
|
121
|
+
parent: q.subject.value,
|
|
122
|
+
};
|
|
123
|
+
}
|
|
124
|
+
else {
|
|
125
|
+
nodeOf(q.subject.value).values.set(q.predicate.value, q.object);
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
return nodes;
|
|
129
|
+
}
|
|
130
|
+
/**
|
|
131
|
+
* Build a raw-IRI → canonical-IRI map for every partition node whose canonical
|
|
132
|
+
* key differs from its current IRI.
|
|
133
|
+
*/
|
|
134
|
+
function buildRemap(nodes, datasetIri, namespaceAliases) {
|
|
135
|
+
const remap = new Map();
|
|
136
|
+
for (const [iri, node] of nodes) {
|
|
137
|
+
const canonicalIri = canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases);
|
|
138
|
+
if (canonicalIri !== undefined && canonicalIri !== iri) {
|
|
139
|
+
remap.set(iri, canonicalIri);
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
return remap;
|
|
143
|
+
}
|
|
144
|
+
/**
|
|
145
|
+
* The canonical partition IRI for a node, or `undefined` if the node is not a
|
|
146
|
+
* partition (no incoming structural link). Replicates the queries’ minting:
|
|
147
|
+
* `<dataset>/.well-known/void#<prefix>-<MD5(STR(component)…)>`.
|
|
148
|
+
*/
|
|
149
|
+
function canonicalPartitionIri(node, nodes, datasetIri, namespaceAliases) {
|
|
150
|
+
const link = node.incomingLink;
|
|
151
|
+
if (link === undefined)
|
|
152
|
+
return undefined;
|
|
153
|
+
const canon = (value) => canonicalizeIri(value, namespaceAliases);
|
|
154
|
+
const classOf = (partition) => nodes.get(partition)?.values.get(VOID_CLASS)?.value;
|
|
155
|
+
switch (link.predicate) {
|
|
156
|
+
case VOID_CLASS_PARTITION: {
|
|
157
|
+
const klass = node.values.get(VOID_CLASS)?.value;
|
|
158
|
+
return klass ? mint(datasetIri, 'class', [canon(klass)]) : undefined;
|
|
159
|
+
}
|
|
160
|
+
case VOID_PROPERTY_PARTITION: {
|
|
161
|
+
const klass = classOf(link.parent);
|
|
162
|
+
const property = node.values.get(VOID_PROPERTY)?.value;
|
|
163
|
+
return klass && property
|
|
164
|
+
? mint(datasetIri, 'class-property', [canon(klass), canon(property)])
|
|
165
|
+
: undefined;
|
|
166
|
+
}
|
|
167
|
+
case VOIDEXT_DATATYPE_PARTITION: {
|
|
168
|
+
const [klass, property] = classProperty(link.parent, nodes);
|
|
169
|
+
const datatype = node.values.get(VOIDEXT_DATATYPE)?.value;
|
|
170
|
+
return klass && property && datatype
|
|
171
|
+
? mint(datasetIri, 'datatype', [
|
|
172
|
+
canon(klass),
|
|
173
|
+
canon(property),
|
|
174
|
+
datatype,
|
|
175
|
+
])
|
|
176
|
+
: undefined;
|
|
177
|
+
}
|
|
178
|
+
case VOIDEXT_OBJECTCLASS_PARTITION: {
|
|
179
|
+
const [klass, property] = classProperty(link.parent, nodes);
|
|
180
|
+
const objectClass = node.values.get(VOID_CLASS)?.value;
|
|
181
|
+
return klass && property && objectClass
|
|
182
|
+
? mint(datasetIri, 'object-class', [
|
|
183
|
+
canon(klass),
|
|
184
|
+
canon(property),
|
|
185
|
+
canon(objectClass),
|
|
186
|
+
])
|
|
187
|
+
: undefined;
|
|
188
|
+
}
|
|
189
|
+
case VOIDEXT_LANGUAGE_PARTITION: {
|
|
190
|
+
const [klass, property] = classProperty(link.parent, nodes);
|
|
191
|
+
const language = node.values.get(VOIDEXT_LANGUAGE)?.value;
|
|
192
|
+
return klass && property && language !== undefined
|
|
193
|
+
? mint(datasetIri, 'language', [
|
|
194
|
+
canon(klass),
|
|
195
|
+
canon(property),
|
|
196
|
+
language,
|
|
197
|
+
])
|
|
198
|
+
: undefined;
|
|
199
|
+
}
|
|
200
|
+
default:
|
|
201
|
+
return undefined;
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
/** The (class, property) of a property partition, following it up to its class partition. */
|
|
205
|
+
function classProperty(propertyPartition, nodes) {
|
|
206
|
+
const node = nodes.get(propertyPartition);
|
|
207
|
+
const property = node?.values.get(VOID_PROPERTY)?.value;
|
|
208
|
+
const classPartition = node?.incomingLink?.parent;
|
|
209
|
+
const klass = classPartition
|
|
210
|
+
? nodes.get(classPartition)?.values.get(VOID_CLASS)?.value
|
|
211
|
+
: undefined;
|
|
212
|
+
return [klass, property];
|
|
213
|
+
}
|
|
214
|
+
function mint(datasetIri, prefix, components) {
|
|
215
|
+
const hash = createHash('md5').update(components.join('')).digest('hex');
|
|
216
|
+
return `${datasetIri}/.well-known/void#${prefix}-${hash}`;
|
|
217
|
+
}
|
|
218
|
+
function remapTerm(term, remap) {
|
|
219
|
+
if (term.termType === 'NamedNode') {
|
|
220
|
+
const canonical = remap.get(term.value);
|
|
221
|
+
if (canonical !== undefined)
|
|
222
|
+
return namedNode(canonical);
|
|
223
|
+
}
|
|
224
|
+
return term;
|
|
225
|
+
}
|
|
226
|
+
function canonicalizeObject(object, predicate, namespaceAliases) {
|
|
227
|
+
if (CANONICALIZED_OBJECT_PREDICATES.has(predicate) &&
|
|
228
|
+
object.termType === 'NamedNode') {
|
|
229
|
+
const canonical = canonicalizeIri(object.value, namespaceAliases);
|
|
230
|
+
if (canonical !== object.value)
|
|
231
|
+
return namedNode(canonical);
|
|
232
|
+
}
|
|
233
|
+
return object;
|
|
234
|
+
}
|
|
235
|
+
function accumulateMeasure(sums, measure) {
|
|
236
|
+
const key = `${measure.subject.value} ${measure.predicate.value} ${measure.graph.value}`;
|
|
237
|
+
const existing = sums.get(key);
|
|
238
|
+
if (existing === undefined) {
|
|
239
|
+
sums.set(key, {
|
|
240
|
+
subject: measure.subject,
|
|
241
|
+
predicate: measure.predicate,
|
|
242
|
+
graph: measure.graph,
|
|
243
|
+
datatype: measure.object.termType === 'Literal'
|
|
244
|
+
? measure.object.datatype.value
|
|
245
|
+
: XSD_INTEGER,
|
|
246
|
+
total: parseInteger(measure.object),
|
|
247
|
+
});
|
|
248
|
+
}
|
|
249
|
+
else {
|
|
250
|
+
existing.total += parseInteger(measure.object);
|
|
251
|
+
}
|
|
252
|
+
}
|
|
253
|
+
function parseInteger(object) {
|
|
254
|
+
if (object.termType === 'Literal' && /^[+-]?\d+$/.test(object.value)) {
|
|
255
|
+
return BigInt(object.value);
|
|
256
|
+
}
|
|
257
|
+
return 0n;
|
|
258
|
+
}
|
|
259
|
+
function reconstructMeasure(sum) {
|
|
260
|
+
return quad(sum.subject, sum.predicate, literal(sum.total.toString(), namedNode(sum.datatype)), sum.graph);
|
|
261
|
+
}
|
|
262
|
+
function quadKey(q) {
|
|
263
|
+
return [
|
|
264
|
+
q.subject.value,
|
|
265
|
+
q.predicate.value,
|
|
266
|
+
termKey(q.object),
|
|
267
|
+
q.graph.value,
|
|
268
|
+
].join(' ');
|
|
269
|
+
}
|
|
270
|
+
function termKey(term) {
|
|
271
|
+
if (term.termType === 'Literal') {
|
|
272
|
+
return `"${term.value}"^^${term.datatype.value}@${term.language}`;
|
|
273
|
+
}
|
|
274
|
+
return term.value;
|
|
275
|
+
}
|
package/dist/stage.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { Stage, type ReaderContext, type QuadTransform } from '@lde/pipeline';
|
|
1
|
+
import { Stage, type ReaderContext, type NamespaceAlias, type QuadTransform } from '@lde/pipeline';
|
|
2
2
|
import type { Quad } from '@rdfjs/types';
|
|
3
3
|
/**
|
|
4
4
|
* Stable names for every VoID stage, equal to the underlying query filename.
|
|
@@ -46,6 +46,27 @@ export interface VoidStageOptions {
|
|
|
46
46
|
* with these, built-in first.
|
|
47
47
|
*/
|
|
48
48
|
transform?: VoidStageTransform;
|
|
49
|
+
/**
|
|
50
|
+
* Namespace pairs to treat as equivalent when partitioning by class and
|
|
51
|
+
* property, so alias variants (e.g. `http://schema.org/CreativeWork` and
|
|
52
|
+
* `https://schema.org/CreativeWork`) merge into a single partition instead
|
|
53
|
+
* of two that both claim the canonical `void:class`.
|
|
54
|
+
*
|
|
55
|
+
* Merging happens after aggregation, in the {@link mergeNamespaceVariants}
|
|
56
|
+
* transform, which sums `void:entities` / `void:triples`; only
|
|
57
|
+
* `class-properties-objects.rq` normalizes at query time (its
|
|
58
|
+
* `void:distinctObjects` cannot be recovered by summing). Correct under the
|
|
59
|
+
* subject/class- and predicate-namespace-disjointness assumptions documented
|
|
60
|
+
* on {@link mergeNamespaceVariants}. The top-level property partitions
|
|
61
|
+
* ({@link detectVocabularies}’s `entity-properties.rq`) and `void:vocabulary`
|
|
62
|
+
* detection keep the source namespaces, so consumers can still see which
|
|
63
|
+
* namespace the dataset actually uses.
|
|
64
|
+
*
|
|
65
|
+
* Defaults to no aliases. To cover schema.org datasets that publish under
|
|
66
|
+
* both `http://schema.org/` and `https://schema.org/`, pass
|
|
67
|
+
* `[{ canonical: 'https://schema.org/', alias: 'http://schema.org/' }]`.
|
|
68
|
+
*/
|
|
69
|
+
namespaceAliases?: readonly NamespaceAlias[];
|
|
49
70
|
}
|
|
50
71
|
/**
|
|
51
72
|
* Options for per-class VoID stages that iterate over classes.
|
package/dist/stage.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EAKL,KAAK,aAAa,EAElB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAEvB,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;
|
|
1
|
+
{"version":3,"file":"stage.d.ts","sourceRoot":"","sources":["../src/stage.ts"],"names":[],"mappings":"AAAA,OAAO,EACL,KAAK,EAKL,KAAK,aAAa,EAElB,KAAK,cAAc,EACnB,KAAK,aAAa,EACnB,MAAM,eAAe,CAAC;AAEvB,OAAO,KAAK,EAAE,IAAI,EAAE,MAAM,cAAc,CAAC;AAqBzC;;;;;;GAMG;AACH,eAAO,MAAM,gBAAgB;;;;;;;;;;;;;;;;;CAiBnB,CAAC;AAEX,sDAAsD;AACtD,MAAM,MAAM,aAAa,GACvB,CAAC,OAAO,gBAAgB,CAAC,CAAC,MAAM,OAAO,gBAAgB,CAAC,CAAC;AAE3D,2EAA2E;AAC3E,MAAM,MAAM,kBAAkB,GAC1B,aAAa,CAAC,aAAa,CAAC,GAC5B,aAAa,CAAC,aAAa,CAAC,EAAE,CAAC;AAEnC;;;;;;;GAOG;AACH,MAAM,WAAW,gBAAgB;IAC/B;;;;;;OAMG;IACH,SAAS,CAAC,EAAE,kBAAkB,CAAC;IAC/B;;;;;;;;;;;;;;;;;;;OAmBG;IACH,gBAAgB,CAAC,EAAE,SAAS,cAAc,EAAE,CAAC;CAC9C;AAED;;;;;GAKG;AACH,MAAM,WAAW,wBAAyB,SAAQ,gBAAgB;IAChE,oEAAoE;IACpE,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,+DAA+D;IAC/D,cAAc,CAAC,EAAE,MAAM,CAAC;IACxB,iFAAiF;IACjF,QAAQ,CAAC,EAAE,OAAO,CAAC;CACpB;AAED;;;;;;GAMG;AACH,MAAM,WAAW,iBAAkB,SAAQ,IAAI,CAC7C,wBAAwB,EACxB,WAAW,CACZ;IACC,yEAAyE;IACzE,SAAS,CAAC,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,CAAC;IACjD,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;IACjC;;;;;;;OAOG;IACH,UAAU,CAAC,EAAE,OAAO,CAAC,MAAM,CAAC,aAAa,EAAE,kBAAkB,CAAC,CAAC,CAAC;CACjE;AAwFD,wBAAgB,gBAAgB,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAE3E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAOD,wBAAgB,mBAAmB,CACjC,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAK1E;AAED,wBAAgB,aAAa,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKxE;AAED,wBAAgB,YAAY,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKvE;AAED,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,oBAAoB,CAClC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAKhB;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAKzE;AAED,wBAAgB,cAAc,CAAC,OAAO,CAAC,EAAE,gBAAgB,GAAG,OAAO,CAAC,KAAK,CAAC,CAEzE;AAID,wBAAgB,qBAAqB,CACnC,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAED,wBAAgB,iBAAiB,CAC/B,OAAO,CAAC,EAAE,wBAAwB,GACjC,OAAO,CAAC,KAAK,CAAC,CAMhB;AAID,wBAAgB,SAAS,CACvB,WAAW,EAAE,WAAW,CAAC,MAAM,EAAE,SAAS,IAAI,EAAE,CAAC,EACjD,OAAO,CAAC,EAAE,gBAAgB,GACzB,OAAO,CAAC,KAAK,CAAC,CAOhB;AAED,MAAM,WAAW,yBAA0B,SAAQ,gBAAgB;IACjE,mFAAmF;IACnF,YAAY,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CAClC;AAED,wBAAgB,kBAAkB,CAChC,OAAO,CAAC,EAAE,yBAAyB,GAClC,OAAO,CAAC,KAAK,CAAC,CAQhB;AAED;;;;;;;GAOG;AACH,wBAAsB,UAAU,CAC9B,OAAO,CAAC,EAAE,iBAAiB,GAC1B,OAAO,CAAC,KAAK,EAAE,CAAC,CAgDlB"}
|
package/dist/stage.js
CHANGED
|
@@ -4,6 +4,8 @@ import { resolve, dirname } from 'node:path';
|
|
|
4
4
|
import { fileURLToPath } from 'node:url';
|
|
5
5
|
import { withVocabularies, defaultVocabularies, } from './vocabularyTransform.js';
|
|
6
6
|
import { withUriSpaces } from './uriSpaceTransform.js';
|
|
7
|
+
import { canonicalizeClassBindings, substituteNormalizationMarkers, withAliasVariantBindings, } from './namespaceAliases.js';
|
|
8
|
+
import { mergeNamespaceVariants } from './partitionMerge.js';
|
|
7
9
|
const queriesDir = resolve(dirname(fileURLToPath(import.meta.url)), '..', 'queries');
|
|
8
10
|
/**
|
|
9
11
|
* Stable names for every VoID stage, equal to the underlying query filename.
|
|
@@ -31,15 +33,30 @@ export const VOID_STAGE_NAMES = {
|
|
|
31
33
|
objectUriSpace: 'object-uri-space.rq',
|
|
32
34
|
};
|
|
33
35
|
async function createVoidStage(filename, options) {
|
|
34
|
-
const
|
|
36
|
+
const namespaceAliases = options?.namespaceAliases ?? [];
|
|
37
|
+
const query = substituteNormalizationMarkers(await readQueryFile(resolve(queriesDir, filename)), namespaceAliases);
|
|
38
|
+
const constructReader = new SparqlConstructReader({ query });
|
|
39
|
+
// Partition stages merge their namespace-alias variants (e.g. http:// and
|
|
40
|
+
// https://schema.org/) into one partition per canonical class/property.
|
|
41
|
+
// Runs first, so any consumer transform sees the already-merged output.
|
|
42
|
+
const mergeTransform = options?.mergePartitions && namespaceAliases.length > 0
|
|
43
|
+
? [mergeNamespaceVariants(namespaceAliases)]
|
|
44
|
+
: [];
|
|
35
45
|
const reader = {
|
|
36
|
-
|
|
37
|
-
|
|
46
|
+
// Per-class stages receive canonical `?class` bindings from the selector;
|
|
47
|
+
// expand each into one `?class` binding per alias variant so the query
|
|
48
|
+
// (`?s a ?class`) matches instances under either namespace.
|
|
49
|
+
reader: options?.perClass
|
|
50
|
+
? withAliasVariantBindings(constructReader, namespaceAliases)
|
|
51
|
+
: constructReader,
|
|
52
|
+
transform: [...mergeTransform, ...asTransforms(options?.transform)],
|
|
38
53
|
};
|
|
39
54
|
return new Stage({
|
|
40
55
|
name: filename,
|
|
41
56
|
readers: reader,
|
|
42
|
-
itemSelector: options?.perClass
|
|
57
|
+
itemSelector: options?.perClass
|
|
58
|
+
? classSelector(namespaceAliases)
|
|
59
|
+
: undefined,
|
|
43
60
|
batchSize: options?.batchSize,
|
|
44
61
|
maxConcurrency: options?.maxConcurrency,
|
|
45
62
|
expectsOutput: options?.expectsOutput,
|
|
@@ -51,7 +68,7 @@ function asTransforms(transform) {
|
|
|
51
68
|
return [];
|
|
52
69
|
return Array.isArray(transform) ? [...transform] : [transform];
|
|
53
70
|
}
|
|
54
|
-
function classSelector() {
|
|
71
|
+
function classSelector(namespaceAliases) {
|
|
55
72
|
return {
|
|
56
73
|
// Forward `options` so the Pipeline’s per-dataset TimeoutPolicy
|
|
57
74
|
// reaches the inner SparqlItemSelector — without this the adaptive
|
|
@@ -69,9 +86,10 @@ function classSelector() {
|
|
|
69
86
|
`WHERE { ${subjectFilter} ?s a ?class . }`,
|
|
70
87
|
'LIMIT 1000',
|
|
71
88
|
].join('\n');
|
|
72
|
-
|
|
89
|
+
const rows = new SparqlItemSelector({
|
|
73
90
|
query: selectorQuery,
|
|
74
91
|
}).select(distribution, batchSize, options);
|
|
92
|
+
return canonicalizeClassBindings(rows, namespaceAliases);
|
|
75
93
|
},
|
|
76
94
|
};
|
|
77
95
|
}
|
|
@@ -80,7 +98,10 @@ export function subjectUriSpaces(options) {
|
|
|
80
98
|
return createVoidStage(VOID_STAGE_NAMES.subjectUriSpace, options);
|
|
81
99
|
}
|
|
82
100
|
export function classPartitions(options) {
|
|
83
|
-
return createVoidStage(VOID_STAGE_NAMES.classPartitions,
|
|
101
|
+
return createVoidStage(VOID_STAGE_NAMES.classPartitions, {
|
|
102
|
+
...options,
|
|
103
|
+
mergePartitions: true,
|
|
104
|
+
});
|
|
84
105
|
}
|
|
85
106
|
// Scalar-aggregate counts: each query is a single COUNT with no GROUP BY/HAVING,
|
|
86
107
|
// so it always returns exactly one row. Zero output therefore means the endpoint
|
|
@@ -120,6 +141,7 @@ export function classPropertySubjects(options) {
|
|
|
120
141
|
return createVoidStage(VOID_STAGE_NAMES.classPropertySubjects, {
|
|
121
142
|
...options,
|
|
122
143
|
perClass: options?.perClass ?? true,
|
|
144
|
+
mergePartitions: true,
|
|
123
145
|
});
|
|
124
146
|
}
|
|
125
147
|
export function classPropertyObjects(options) {
|
|
@@ -142,18 +164,21 @@ export function perClassObjectClasses(options) {
|
|
|
142
164
|
return createVoidStage(VOID_STAGE_NAMES.perClassObjectClasses, {
|
|
143
165
|
...options,
|
|
144
166
|
perClass: options?.perClass ?? true,
|
|
167
|
+
mergePartitions: true,
|
|
145
168
|
});
|
|
146
169
|
}
|
|
147
170
|
export function perClassDatatypes(options) {
|
|
148
171
|
return createVoidStage(VOID_STAGE_NAMES.perClassDatatypes, {
|
|
149
172
|
...options,
|
|
150
173
|
perClass: options?.perClass ?? true,
|
|
174
|
+
mergePartitions: true,
|
|
151
175
|
});
|
|
152
176
|
}
|
|
153
177
|
export function perClassLanguages(options) {
|
|
154
178
|
return createVoidStage(VOID_STAGE_NAMES.perClassLanguages, {
|
|
155
179
|
...options,
|
|
156
180
|
perClass: options?.perClass ?? true,
|
|
181
|
+
mergePartitions: true,
|
|
157
182
|
});
|
|
158
183
|
}
|
|
159
184
|
// Stages with a built-in transform
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@lde/pipeline-void",
|
|
3
|
-
"version": "0.31.
|
|
3
|
+
"version": "0.31.4",
|
|
4
4
|
"description": "VOiD (Vocabulary of Interlinked Datasets) statistical analysis for RDF datasets",
|
|
5
5
|
"repository": {
|
|
6
6
|
"url": "git+https://github.com/ldelements/lde.git",
|
|
@@ -34,6 +34,6 @@
|
|
|
34
34
|
},
|
|
35
35
|
"peerDependencies": {
|
|
36
36
|
"@lde/dataset": "^0.7.8",
|
|
37
|
-
"@lde/pipeline": "^0.33.
|
|
37
|
+
"@lde/pipeline": "^0.33.4"
|
|
38
38
|
}
|
|
39
39
|
}
|
|
@@ -5,17 +5,27 @@ CONSTRUCT {
|
|
|
5
5
|
}
|
|
6
6
|
WHERE {
|
|
7
7
|
# Object counts only. Subject counts in class-properties-subjects.rq.
|
|
8
|
+
#
|
|
9
|
+
# Unlike the other per-class queries, this one normalizes the class and
|
|
10
|
+
# property to their canonical namespace *before* the COUNT(DISTINCT ?o):
|
|
11
|
+
# merging distinct-object counts by summing would over-count objects shared
|
|
12
|
+
# between namespace variants (e.g. the same name under http:name and
|
|
13
|
+
# https:name), so the dedup must happen here rather than in the
|
|
14
|
+
# partition-merge transform. `?class` is bound per namespace variant by the
|
|
15
|
+
# injected VALUES clause; `?property` is discovered.
|
|
8
16
|
{
|
|
9
|
-
SELECT ?
|
|
17
|
+
SELECT ?canonicalClass ?canonicalProperty (COUNT(DISTINCT ?o) AS ?objects) {
|
|
10
18
|
{
|
|
11
|
-
SELECT ?
|
|
19
|
+
SELECT ?canonicalClass ?canonicalProperty ?o {
|
|
12
20
|
#subjectFilter#
|
|
13
|
-
?s a ?class ; ?
|
|
21
|
+
?s a ?class ; ?property ?o .
|
|
22
|
+
BIND(#normalized:class# AS ?canonicalClass)
|
|
23
|
+
BIND(#normalized:property# AS ?canonicalProperty)
|
|
14
24
|
}
|
|
15
25
|
}
|
|
16
26
|
}
|
|
17
|
-
GROUP BY ?
|
|
27
|
+
GROUP BY ?canonicalClass ?canonicalProperty
|
|
18
28
|
}
|
|
19
|
-
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?
|
|
29
|
+
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?canonicalClass), STR(?canonicalProperty))))) AS ?propertyPartition)
|
|
20
30
|
}
|
|
21
31
|
LIMIT 100000
|
|
@@ -2,7 +2,17 @@ PREFIX void: <http://rdfs.org/ns/void#>
|
|
|
2
2
|
PREFIX void-ext: <http://ldf.fi/void-ext#>
|
|
3
3
|
|
|
4
4
|
CONSTRUCT {
|
|
5
|
-
|
|
5
|
+
# Self-describing parent chain so the namespace-merge transform can re-key
|
|
6
|
+
# this partition from its (class, property) components. The
|
|
7
|
+
# `?dataset void:classPartition ?classPartition` link is what gives the class
|
|
8
|
+
# partition an incoming edge, so the transform re-keys it too (without it, the
|
|
9
|
+
# alias-variant class partition leaks unmerged).
|
|
10
|
+
?dataset a void:Dataset ;
|
|
11
|
+
void:classPartition ?classPartition .
|
|
12
|
+
?classPartition void:class ?class ;
|
|
13
|
+
void:propertyPartition ?propertyPartition .
|
|
14
|
+
?propertyPartition void:property ?p ;
|
|
15
|
+
void-ext:datatypePartition ?datatypePartition .
|
|
6
16
|
?datatypePartition
|
|
7
17
|
void-ext:datatype ?dt ;
|
|
8
18
|
void:triples ?count .
|
|
@@ -22,6 +32,7 @@ WHERE {
|
|
|
22
32
|
}
|
|
23
33
|
GROUP BY ?class ?p ?dt
|
|
24
34
|
}
|
|
35
|
+
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
|
|
25
36
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
|
|
26
37
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#datatype-", MD5(CONCAT(STR(?class), STR(?p), STR(?dt))))) AS ?datatypePartition)
|
|
27
38
|
}
|
|
@@ -2,7 +2,17 @@ PREFIX void: <http://rdfs.org/ns/void#>
|
|
|
2
2
|
PREFIX void-ext: <http://ldf.fi/void-ext#>
|
|
3
3
|
|
|
4
4
|
CONSTRUCT {
|
|
5
|
-
|
|
5
|
+
# Self-describing parent chain so the namespace-merge transform can re-key
|
|
6
|
+
# this partition from its (class, property) components. The
|
|
7
|
+
# `?dataset void:classPartition ?classPartition` link is what gives the class
|
|
8
|
+
# partition an incoming edge, so the transform re-keys it too (without it, the
|
|
9
|
+
# alias-variant class partition leaks unmerged).
|
|
10
|
+
?dataset a void:Dataset ;
|
|
11
|
+
void:classPartition ?classPartition .
|
|
12
|
+
?classPartition void:class ?class ;
|
|
13
|
+
void:propertyPartition ?propertyPartition .
|
|
14
|
+
?propertyPartition void:property ?p ;
|
|
15
|
+
void-ext:languagePartition ?languagePartition .
|
|
6
16
|
?languagePartition
|
|
7
17
|
void-ext:language ?lang ;
|
|
8
18
|
void:triples ?count .
|
|
@@ -24,6 +34,7 @@ WHERE {
|
|
|
24
34
|
}
|
|
25
35
|
GROUP BY ?class ?p ?lang
|
|
26
36
|
}
|
|
37
|
+
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
|
|
27
38
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
|
|
28
39
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#language-", MD5(CONCAT(STR(?class), STR(?p), ?lang)))) AS ?languagePartition)
|
|
29
40
|
}
|
|
@@ -2,7 +2,17 @@ PREFIX void: <http://rdfs.org/ns/void#>
|
|
|
2
2
|
PREFIX void-ext: <http://ldf.fi/void-ext#>
|
|
3
3
|
|
|
4
4
|
CONSTRUCT {
|
|
5
|
-
|
|
5
|
+
# Self-describing parent chain so the namespace-merge transform can re-key
|
|
6
|
+
# this partition from its (class, property) components. The
|
|
7
|
+
# `?dataset void:classPartition ?classPartition` link is what gives the class
|
|
8
|
+
# partition an incoming edge, so the transform re-keys it too (without it, the
|
|
9
|
+
# alias-variant class partition leaks unmerged).
|
|
10
|
+
?dataset a void:Dataset ;
|
|
11
|
+
void:classPartition ?classPartition .
|
|
12
|
+
?classPartition void:class ?class ;
|
|
13
|
+
void:propertyPartition ?propertyPartition .
|
|
14
|
+
?propertyPartition void:property ?p ;
|
|
15
|
+
void-ext:objectClassPartition ?objectClassPartition .
|
|
6
16
|
?objectClassPartition
|
|
7
17
|
void:class ?objectClass ;
|
|
8
18
|
void:triples ?count .
|
|
@@ -22,6 +32,7 @@ WHERE {
|
|
|
22
32
|
}
|
|
23
33
|
GROUP BY ?class ?p ?objectClass
|
|
24
34
|
}
|
|
35
|
+
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-", MD5(STR(?class)))) AS ?classPartition)
|
|
25
36
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#class-property-", MD5(CONCAT(STR(?class), STR(?p))))) AS ?propertyPartition)
|
|
26
37
|
BIND(URI(CONCAT(STR(?dataset), "/.well-known/void#object-class-", MD5(CONCAT(STR(?class), STR(?p), STR(?objectClass))))) AS ?objectClassPartition)
|
|
27
38
|
}
|