@factoidal/core 0.7.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +510 -1
- package/NOTICE +2 -21
- package/README.md +117 -993
- package/api.d.ts +53 -0
- package/api.js +5 -0
- package/api.mjs +56 -0
- package/bin/engine.mjs +38 -19
- package/bin/factoidal.mjs +513 -67
- package/bin/pack.mjs +37 -0
- package/bin/serve-worker.mjs +32 -0
- package/bin/serve.mjs +423 -0
- package/bin/store-worker-host.mjs +1 -1
- package/bin/store.mjs +1 -1
- package/browser.d.ts +110 -507
- package/browser.js +151 -2257
- package/docs/fuseki-replacement.md +108 -0
- package/fn.d.ts +24 -95
- package/fn.js +104 -150
- package/index.d.ts +598 -259
- package/index.js +4 -234
- package/index.mjs +176 -84
- package/l4-assets/l4factoidal.js +47 -7
- package/l4-assets/l4factoidal.mjs +1 -1
- package/l4-assets/l4factoidal.wasm +0 -0
- package/l4-assets/version.json +11 -7
- package/lib/api.mjs +2300 -0
- package/lib/browser-engine.mjs +69 -0
- package/lib/dataset.mjs +377 -0
- package/lib/delta-log-hosts.mjs +191 -0
- package/lib/lean-entry.mjs +204 -0
- package/lib/pure.mjs +98 -0
- package/lib/store.mjs +115 -0
- package/lib/version.mjs +5 -0
- package/lws/README.md +97 -0
- package/lws/server.mjs +331 -0
- package/package.json +59 -61
- package/rdfjs.js +5 -536
- package/rdfjs.mjs +594 -0
- package/sample-store.d.ts +8 -16
- package/sample-store.mjs +39 -29
- package/solid/client/README.md +100 -0
- package/solid/client/index.mjs +277 -0
- package/solid/server/README.md +96 -0
- package/solid/server/index.mjs +317 -0
- package/store-host/deno.mjs +8 -0
- package/store-host/index.mjs +12 -0
- package/store-host/node.mjs +8 -0
- package/xmpp/README.md +107 -0
- package/xmpp/index.mjs +229 -0
- package/browser-wasm.js +0 -872
- package/factoidal-npm-entry.js +0 -15477
- package/factoidal-npm-entry.wasm.assets/code-5a7fc68f2ab1323718b8.wasm +0 -0
- package/factoidal-npm-entry.wasm.js +0 -455
- package/factoidal.js +0 -13118
- package/factoidal.wasm.assets/code-bbe6099bfb5b10c4c3ab.wasm +0 -0
- package/factoidal.wasm.js +0 -457
- package/hacl-init.js +0 -92
- package/hacl-wasm/FStar.wasm +0 -0
- package/hacl-wasm/Hacl_Bignum.wasm +0 -0
- package/hacl-wasm/Hacl_Bignum25519_51.wasm +0 -0
- package/hacl-wasm/Hacl_Bignum_Base.wasm +0 -0
- package/hacl-wasm/Hacl_Curve25519_51.wasm +0 -0
- package/hacl-wasm/Hacl_Ed25519.wasm +0 -0
- package/hacl-wasm/Hacl_Ed25519_PrecompTable.wasm +0 -0
- package/hacl-wasm/Hacl_Hash_Base.wasm +0 -0
- package/hacl-wasm/Hacl_Hash_SHA2.wasm +0 -0
- package/hacl-wasm/Hacl_IntTypes_Intrinsics.wasm +0 -0
- package/hacl-wasm/LowStar_Endianness.wasm +0 -0
- package/hacl-wasm/WasmSupport.wasm +0 -0
- package/hacl-wasm/api.js +0 -775
- package/hacl-wasm/api.json +0 -3787
- package/hacl-wasm/layouts.json +0 -1
- package/hacl-wasm/loader.js +0 -568
- package/hacl-wasm/shell.js +0 -12
- package/l4-core.d.ts +0 -159
- package/l4-core.js +0 -188
- package/l4.d.ts +0 -50
- package/l4.js +0 -104
- package/lib/api.js +0 -2300
- package/lib/engine-js.js +0 -165
- package/lib/engine-wasm.js +0 -300
- package/sample-store/CURRENT +0 -1
- package/sample-store/gen-1/manifest.sbm2 +0 -0
- package/sample-store/gen-1/manifest.tsv +0 -14
- package/sample-store/gen-1/predicate-0.ibk3 +0 -0
- package/sample-store/gen-1/predicate-0.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-0.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-0.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-0.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-0.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-0.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-0.ibk3.tli1.merkle +0 -0
- package/sample-store/gen-1/predicate-1.ibk3 +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.merkle +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.oli2.merkle +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-1.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-1.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-10.ibk3 +0 -0
- package/sample-store/gen-1/predicate-10.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-10.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-10.ibk3.oli2.merkle +0 -0
- package/sample-store/gen-1/predicate-10.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-10.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-10.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-10.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-11.ibk3 +0 -0
- package/sample-store/gen-1/predicate-11.ibk3.merkle +0 -0
- package/sample-store/gen-1/predicate-11.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-11.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-11.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-11.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-11.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-11.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-12.ibk3 +0 -0
- package/sample-store/gen-1/predicate-12.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-12.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-12.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-12.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-12.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-12.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-12.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-2.ibk3 +0 -0
- package/sample-store/gen-1/predicate-2.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-2.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-2.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-2.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-2.ibk3.sri2.merkle +0 -0
- package/sample-store/gen-1/predicate-2.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-2.ibk3.tli1.merkle +0 -0
- package/sample-store/gen-1/predicate-3.ibk3 +0 -0
- package/sample-store/gen-1/predicate-3.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-3.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-3.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-3.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-3.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-3.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-3.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-4.ibk3 +0 -0
- package/sample-store/gen-1/predicate-4.ibk3.merkle +0 -0
- package/sample-store/gen-1/predicate-4.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-4.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-4.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-4.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-4.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-4.ibk3.tli1.merkle +0 -2
- package/sample-store/gen-1/predicate-5.ibk3 +0 -0
- package/sample-store/gen-1/predicate-5.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-5.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-5.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-5.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-5.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-5.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-5.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-6.ibk3 +0 -0
- package/sample-store/gen-1/predicate-6.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-6.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-6.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-6.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-6.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-6.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-6.ibk3.tli1.merkle +0 -0
- package/sample-store/gen-1/predicate-7.ibk3 +0 -0
- package/sample-store/gen-1/predicate-7.ibk3.merkle +0 -0
- package/sample-store/gen-1/predicate-7.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-7.ibk3.oli2.merkle +0 -2
- package/sample-store/gen-1/predicate-7.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-7.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-7.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-7.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-8.ibk3 +0 -0
- package/sample-store/gen-1/predicate-8.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-8.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-8.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-8.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-8.ibk3.sri2.merkle +0 -1
- package/sample-store/gen-1/predicate-8.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-8.ibk3.tli1.merkle +0 -1
- package/sample-store/gen-1/predicate-9.ibk3 +0 -0
- package/sample-store/gen-1/predicate-9.ibk3.merkle +0 -1
- package/sample-store/gen-1/predicate-9.ibk3.oli2 +0 -0
- package/sample-store/gen-1/predicate-9.ibk3.oli2.merkle +0 -1
- package/sample-store/gen-1/predicate-9.ibk3.sri2 +0 -0
- package/sample-store/gen-1/predicate-9.ibk3.sri2.merkle +0 -3
- package/sample-store/gen-1/predicate-9.ibk3.tli1 +0 -0
- package/sample-store/gen-1/predicate-9.ibk3.tli1.merkle +0 -2
- package/select.d.ts +0 -116
- package/select.js +0 -492
- package/version.json +0 -102
- package/wasm.d.ts +0 -130
- package/wasm.js +0 -158
package/README.md
CHANGED
|
@@ -1,1012 +1,136 @@
|
|
|
1
1
|
# @factoidal/core
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
> under the placeholder names `factoidal` and `@danbri/foafos`; it was
|
|
6
|
-
> never published under those names. See [CHANGELOG.md](CHANGELOG.md).
|
|
7
|
-
|
|
8
|
-
A formally verified RDF/SPARQL engine for JavaScript and WebAssembly.
|
|
9
|
-
The semantics live in [F\*](https://www.fstar-lang.org/) and are
|
|
10
|
-
compiled via OCaml and js_of_ocaml / wasm_of_ocaml into the engine
|
|
11
|
-
bundles shipped here. There is no hand-written SPARQL evaluator — the
|
|
12
|
-
JavaScript you run was extracted from the same `.fst` specifications
|
|
13
|
-
that verify under Z3 and pass the W3C suites.
|
|
14
|
-
|
|
15
|
-
Verification status qualifier: parser and algebra spec verified in
|
|
16
|
-
F\*; the on-disk backend has unverified OCaml-side optimization layers
|
|
17
|
-
being migrated back to F\* (the npm build does not include the
|
|
18
|
-
on-disk backend).
|
|
19
|
-
|
|
20
|
-
## Why this instead of N3.js / rdflib.js / Comunica?
|
|
21
|
-
|
|
22
|
-
- **Full SPARQL 1.1 client-side** — query and update evaluated in
|
|
23
|
-
your process by conformance-tested code (631 of 631 W3C SPARQL
|
|
24
|
-
tests, 1031 of 1031 RDF parsing tests on the native build of the
|
|
25
|
-
same source); no server, no endpoint.
|
|
26
|
-
- **RDFC-1.0 canonicalization built in**: `canonicalize()` yields
|
|
27
|
-
standard canonical N-Quads (stable under blank-node renaming — for
|
|
28
|
-
content addressing, dataset diffing, cache keys) without a separate
|
|
29
|
-
library.
|
|
30
|
-
- **RDFS / OWL-RL entailment** as a query option (`entail: "RDFS"`).
|
|
31
|
-
- **RDF/JS data model** (`DataFactory`, terms with `.equals`,
|
|
32
|
-
`DatasetCore`) so it composes with the existing ecosystem.
|
|
33
|
-
- Provenance: every release corresponds to a gates-green commit of
|
|
34
|
-
[danbri/factoidal](https://github.com/danbri/factoidal).
|
|
35
|
-
|
|
36
|
-
## Install
|
|
3
|
+
RDF, SPARQL, reasoning, validation and Linked Data servers in one package.
|
|
4
|
+
Runs in Node 22.12+, Deno 2 and browsers. No dependencies.
|
|
37
5
|
|
|
38
6
|
```bash
|
|
39
7
|
npm install @factoidal/core
|
|
40
8
|
```
|
|
41
9
|
|
|
42
|
-
## Quickstart
|
|
43
|
-
|
|
44
|
-
The API is **async throughout** — every engine call returns a Promise,
|
|
45
|
-
so `await` each one (the engine bundle loads lazily on first use).
|
|
46
|
-
|
|
47
|
-
```js
|
|
48
|
-
import { parse, query, serialize, canonicalize, dataFactory }
|
|
49
|
-
from "@factoidal/core";
|
|
50
|
-
|
|
51
|
-
// Parse any supported syntax into an RDF/JS DatasetCore.
|
|
52
|
-
// If your data contains RELATIVE IRIs, pass { baseIRI: ... } —
|
|
53
|
-
// see the note below this example.
|
|
54
|
-
const ds = await parse(`
|
|
55
|
-
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
|
|
56
|
-
_:a foaf:name "Alice" ; foaf:knows _:b .
|
|
57
|
-
_:b foaf:name "Bob" .
|
|
58
|
-
`, { format: "turtle" });
|
|
59
|
-
|
|
60
|
-
// SELECT — bindings are Map<variableName, RDF/JS Term>.
|
|
61
|
-
const rows = await query(ds, `
|
|
62
|
-
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
|
|
63
|
-
SELECT ?name WHERE { ?p foaf:name ?name } ORDER BY ?name
|
|
64
|
-
`);
|
|
65
|
-
for (const b of rows) console.log(b.get("name").value); // Alice, Bob
|
|
66
|
-
|
|
67
|
-
// ASK — plain boolean.
|
|
68
|
-
const yes = await query(ds, "ASK { ?s ?p ?o }"); // true
|
|
69
|
-
|
|
70
|
-
// Canonical N-Quads (RDFC-1.0): identical output for isomorphic
|
|
71
|
-
// inputs regardless of blank-node labels.
|
|
72
|
-
const c14n = await canonicalize(ds);
|
|
73
|
-
|
|
74
|
-
// Round-trip.
|
|
75
|
-
const nq = await serialize(ds, { format: "nquads" });
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
> ⚠️ **Pass `baseIRI` when your input uses relative IRIs.** In the
|
|
79
|
-
> current build, statements whose relative IRIs cannot be resolved are
|
|
80
|
-
> **dropped without an error**, so a document can parse to fewer
|
|
81
|
-
> triples than it contains (`{ baseIRI: "https://example.org/doc" }`
|
|
82
|
-
> fixes it). A count check after parsing is a cheap guard. Surfacing
|
|
83
|
-
> these drops as a throw or a warnings channel is tracked in the
|
|
84
|
-
> repository issues.
|
|
85
|
-
|
|
86
|
-
CommonJS: `const factoidal = require("@factoidal/core")`.
|
|
87
|
-
|
|
88
|
-
### WebAssembly entry
|
|
89
|
-
|
|
90
|
-
```js
|
|
91
|
-
import { parse, query } from "factoidal/wasm";
|
|
92
|
-
// Same API over the wasm_of_ocaml bundle; needs a Wasm-GC engine
|
|
93
|
-
// (Chrome >= 119, Node >= 22). The unit suite asserts byte-parity
|
|
94
|
-
// between the JS and Wasm entries on parse/SELECT/ASK.
|
|
95
|
-
```
|
|
96
|
-
|
|
97
|
-
### Load in the browser without npm
|
|
98
|
-
|
|
99
|
-
You don't need `npm install` (or a bundler) to run this in a browser.
|
|
100
|
-
Two options, in preference order:
|
|
101
|
-
|
|
102
|
-
1. **This site's own mirror (recommended, same-origin, no build step
|
|
103
|
-
for you).** Every push regenerates `docs/npm/factoidal/` from this
|
|
104
|
-
package (`formal/fstar/build-ocaml.sh npm`'s Pages-mirror step), so
|
|
105
|
-
it's always in sync with what's published here:
|
|
106
|
-
|
|
107
|
-
```html
|
|
108
|
-
<script type="module">
|
|
109
|
-
import { query } from 'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
|
|
110
|
-
const r = await query(dataTtl, 'SELECT * WHERE { ?s ?p ?o }');
|
|
111
|
-
console.log(r.results.bindings);
|
|
112
|
-
</script>
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
See [`demo-jsonld-playground.html`](../../docs/fstar-extracted/demo-jsonld-playground.html)
|
|
116
|
-
for a working page built exactly this way (loads `browser.js` from
|
|
117
|
-
`../npm/factoidal/browser.js`, a relative same-origin path).
|
|
118
|
-
|
|
119
|
-
2. **jsDelivr's GitHub proxy** (serves the correct
|
|
120
|
-
`text/javascript` MIME type; no Pages dependency, works from any
|
|
121
|
-
branch/tag):
|
|
122
|
-
|
|
123
|
-
```html
|
|
124
|
-
<script type="module">
|
|
125
|
-
import { query } from 'https://cdn.jsdelivr.net/gh/danbri/factoidal@claude/main/npm/factoidal/browser.js';
|
|
126
|
-
</script>
|
|
127
|
-
```
|
|
128
|
-
|
|
129
|
-
**Do not** link `browser.js` (or `factoidal.js`) via
|
|
130
|
-
`raw.githubusercontent.com`. Raw GitHub serves every file as
|
|
131
|
-
`text/plain`, and browsers refuse to execute a `text/plain` response
|
|
132
|
-
as an ES module (`<script type="module">` fails with a MIME-type
|
|
133
|
-
error) — this bites people who copy a "raw" link expecting it to just
|
|
134
|
-
work. Use the Pages mirror or jsDelivr above instead.
|
|
135
|
-
|
|
136
|
-
`browser.js` also exports `toRdf(text, options)` (parse to sorted
|
|
137
|
-
N-Quads — `--dump-nq` under the hood) and `canonicalize(text,
|
|
138
|
-
options)` (RDFC-1.0 — `--canonicalize`) alongside `query()`; both
|
|
139
|
-
default `options.format` to `'jsonld'` since that's the playground's
|
|
140
|
-
use case, but accept the same formats as `query()`'s `dataFormat`.
|
|
141
|
-
|
|
142
|
-
For multi-file, multi-named-graph datasets (several documents, each
|
|
143
|
-
loaded into its own named graph or the default graph) and a choice of
|
|
144
|
-
extraction target, use `queryDataset(files, queryString, options)`:
|
|
145
|
-
|
|
146
|
-
```js
|
|
147
|
-
import { queryDataset } from 'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
|
|
148
|
-
|
|
149
|
-
const r = await queryDataset(
|
|
150
|
-
[
|
|
151
|
-
{ content: defaultGraphTtl },
|
|
152
|
-
{ content: peopleTtl, graph: 'urn:x:people' },
|
|
153
|
-
],
|
|
154
|
-
'SELECT * WHERE { GRAPH ?g { ?s ?p ?o } }',
|
|
155
|
-
{ entail: 'RDFS', engine: 'js' } // or engine: 'wasm'
|
|
156
|
-
);
|
|
157
|
-
```
|
|
158
|
-
|
|
159
|
-
On `engine: 'wasm'` it merges the named graphs into one TriG document
|
|
160
|
-
(wasm_of_ocaml's `query()` only takes a single data string) via
|
|
161
|
-
`browser-wasm.js`, loaded on demand. Both `query()` and `queryDataset()`
|
|
162
|
-
attach a non-enumerable `engineMs` (bundle-eval wall-clock time) to the
|
|
163
|
-
returned results object — invisible to `JSON.stringify()`/`Object.keys()`
|
|
164
|
-
so it never perturbs a diff against a W3C `.srx`-derived fixture, but
|
|
165
|
-
readable as `result.engineMs` for timing/observability UIs. This is
|
|
166
|
-
what `docs/fstar-extracted/factoidal-sparql-client.js`'s web component
|
|
167
|
-
is built on, rather than duplicating the engine-invocation logic itself.
|
|
168
|
-
|
|
169
|
-
### Durable browser persistence (delta log)
|
|
170
|
-
|
|
171
|
-
`browser.js` also exports a small IndexedDB-backed durable-UPDATE log —
|
|
172
|
-
see [`docs/designissues/2026-07-06-browser-persistence.md`](../../docs/designissues/2026-07-06-browser-persistence.md)
|
|
173
|
-
for the full design (why IndexedDB and not OPFS for v1, the tab-close/
|
|
174
|
-
crash guarantee mapping, and the quota/eviction honesty section). Every
|
|
175
|
-
byte moved through these functions is produced/consumed by the same
|
|
176
|
-
F\*-verified `RDF_Store_Columnar_DeltaLog`/`RDF_Store_Columnar_DeltaMerge`
|
|
177
|
-
modules the native on-disk delta log uses (`factoidal serve --rw
|
|
178
|
-
--delta-log`); this is a browser-native persistence path, not a mock:
|
|
179
|
-
|
|
180
|
-
```js
|
|
181
|
-
import { deltaLogOpen, deltaLogAppend, deltaLogMerge } from
|
|
182
|
-
'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
|
|
183
|
-
|
|
184
|
-
const handle = await deltaLogOpen(); // opens/creates an IndexedDB database
|
|
185
|
-
await deltaLogAppend(handle, 'INSERT DATA { <urn:x:a> <urn:x:p> "1" }');
|
|
186
|
-
await deltaLogAppend(handle, 'INSERT DATA { <urn:x:b> <urn:x:p> "2" }');
|
|
187
|
-
|
|
188
|
-
// ... reload the page, or close and reopen the browser ...
|
|
189
|
-
|
|
190
|
-
const merged = await deltaLogMerge(handle, ''); // '' = empty base dataset
|
|
191
|
-
console.log(merged); // the two INSERT DATA ops, replayed from IndexedDB
|
|
192
|
-
```
|
|
193
|
-
|
|
194
|
-
Supported update ops: `INSERT DATA`, `DELETE DATA`, `CLEAR`, `DROP`,
|
|
195
|
-
`CREATE` — the same subset the native `--rw` commit path accepts;
|
|
196
|
-
anything else (`DELETE/INSERT WHERE`, `COPY`, `MOVE`, `ADD`) rejects
|
|
197
|
-
with `ok:false` rather than silently no-op'ing. `deltaLogReadAllHex`,
|
|
198
|
-
`deltaLogDestroy`, and the test-only `_deltaLogCorruptLastForTest` round
|
|
199
|
-
out the surface (see `browser.js`'s own JSDoc for each). This is a
|
|
200
|
-
prototype: no compaction and no `navigator.storage.persist()` wiring
|
|
201
|
-
yet (both named as staged next steps in the design doc), and it does
|
|
202
|
-
not yet back a hub demo page — that is separate, follow-on work.
|
|
203
|
-
|
|
204
|
-
### RDF/JS interop
|
|
205
|
-
|
|
206
10
|
```js
|
|
207
|
-
import {
|
|
208
|
-
|
|
209
|
-
const
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
const answer = handle.query(`
|
|
268
|
-
PREFIX ex: <http://example.org/fn/>
|
|
269
|
-
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
|
|
270
|
-
SELECT ?c ?l WHERE { ?c skos:prefLabel ?l FILTER(ex:endsWithZed(?l)) }`)
|
|
271
|
-
```
|
|
272
|
-
|
|
273
|
-
The argument and result encoding, the §17.6 error rules and the
|
|
274
|
-
async re-evaluation loop are the same as above, so one function serves
|
|
275
|
-
both engines. Two differences to plan around:
|
|
276
|
-
|
|
277
|
-
- **Registration is per engine instance**, not per handle or per query.
|
|
278
|
-
A server that answers for more than one caller uses
|
|
279
|
-
`withExtensionFunctions(engine, map, body)`, which registers, runs and
|
|
280
|
-
clears in a `finally`, or loads one engine per caller.
|
|
281
|
-
- **Every call crosses into JavaScript**, so a function in a FILTER runs
|
|
282
|
-
once per row. Measure before putting one on a large scan.
|
|
283
|
-
|
|
284
|
-
Design, with the determinism and scope rules in full:
|
|
285
|
-
[`docs/designissues/2026-09-04-lean-extension-functions.md`](../../docs/designissues/2026-09-04-lean-extension-functions.md).
|
|
286
|
-
|
|
287
|
-
## Functional API (fn)
|
|
288
|
-
|
|
289
|
-
`@factoidal/core/fn` is a strictly functional variant of the API above:
|
|
290
|
-
every extracted engine operation is already a value-to-value
|
|
291
|
-
function — `FnDataset` makes the JS surface match that instead of
|
|
292
|
-
papering over it with RDF/JS's mutable `add`/`delete`. Frozen
|
|
293
|
-
snapshots, free functions instead of methods (so they compose), and
|
|
294
|
-
RDFC-1.0 canonical hashes as a first-class, memoized identity — the
|
|
295
|
-
piece that makes dataflow-style recompute-skipping possible. Full
|
|
296
|
-
design rationale (cost model, backend/streaming extension points):
|
|
297
|
-
[`docs/designissues/2026-07-05-functional-dataset-api.md`](../../docs/designissues/2026-07-05-functional-dataset-api.md).
|
|
298
|
-
|
|
299
|
-
```js
|
|
300
|
-
const { parse, filter, hash, cell, derive } = require('@factoidal/core/fn');
|
|
301
|
-
|
|
302
|
-
const ds = await parse(`
|
|
303
|
-
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
|
|
304
|
-
_:a foaf:name "Alice" ; a foaf:Person .
|
|
305
|
-
_:b foaf:name "Bob".
|
|
306
|
-
`);
|
|
307
|
-
|
|
308
|
-
// Every op returns a new FnDataset; ds is never touched.
|
|
309
|
-
const people = filter(ds, (q) => q.predicate.value.endsWith('/type'));
|
|
310
|
-
|
|
311
|
-
// Dataflow: a derive() node recomputes only when its input's content
|
|
312
|
-
// hash actually changes -- not when a new (but equal-content)
|
|
313
|
-
// FnDataset object is set into the cell.
|
|
314
|
-
const source = cell(ds);
|
|
315
|
-
const derived = derive((d) => filter(d, (q) => q.predicate.value.endsWith('name')), source);
|
|
316
|
-
await derived.get(); // computes
|
|
317
|
-
source.set(await parse(sameTextAsBefore));
|
|
318
|
-
await derived.get(); // memoized hit -- same content, new object
|
|
319
|
-
|
|
320
|
-
await hash(ds); // sha256 hex of RDFC-1.0 canonical N-Quads
|
|
321
|
-
```
|
|
322
|
-
|
|
323
|
-
`fromDataset(dataset)` / `toDataset(fnDataset)` convert to and from
|
|
324
|
-
the mutable RDF/JS `Dataset` above, so the two styles compose freely.
|
|
325
|
-
`union`/`difference`/`filter`/`mapQuads` give set algebra without
|
|
326
|
-
methods; `query`/`entail`/`canonicalize`/`graphs` mirror the plain
|
|
327
|
-
API's capability gating (`capabilities()`) exactly. `validate`/`shex`/
|
|
328
|
-
`fromMapping`/`fromCsvw`/`rif` wrap the SHACL/ShEx/RML/CSVW/RIF
|
|
329
|
-
surface below the same way — FnDataset in, FnDataset (or a plain
|
|
330
|
-
verdict) out:
|
|
331
|
-
|
|
332
|
-
```js
|
|
333
|
-
const { parse, validate, shex, fromMapping, fromCsvw, rif } = require('@factoidal/core/fn');
|
|
334
|
-
|
|
335
|
-
const data = await parse(personTtl);
|
|
336
|
-
const shapes = await parse(shapesTtl);
|
|
337
|
-
const { conforms, report } = await validate(data, shapes); // SHACL
|
|
338
|
-
|
|
339
|
-
const ok = await shex(data, shexSchemaJson, 'http://example.org/alice'); // true|false|null
|
|
340
|
-
|
|
341
|
-
const mapped = await fromMapping(await parse(rmlMappingTtl), csvOrJsonText, 'csv'); // RML
|
|
342
|
-
|
|
343
|
-
const tabular = await fromCsvw(csvText, csvwMetadataJson, { mode: 'minimal' }); // CSVW csv2rdf
|
|
344
|
-
|
|
345
|
-
const saturated = await rif(data, rifRulesXml); // RIF Core forward chaining
|
|
346
|
-
```
|
|
347
|
-
|
|
348
|
-
## Lean engine (new in 0.2.0)
|
|
349
|
-
|
|
350
|
-
Two engines now ship in one package. `factoidal` and `factoidal/wasm`
|
|
351
|
-
are the F\*-extracted engine, unchanged. The subpaths below are the
|
|
352
|
-
Lean 4 engine (`L4Factoidal`, compiled to wasm).
|
|
353
|
-
|
|
354
|
-
`factoidal/l4` also exposes two deliberately narrow physical helpers:
|
|
355
|
-
`scanIBK2Predicate(ibk2Hex, predicateIri)` for the predecessor format and
|
|
356
|
-
`scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` for the current
|
|
357
|
-
predicate-local format. They validate one canonical RDF block and scan its
|
|
358
|
-
named predicate, returning N-Triples and a row count. The IBK3 source scope
|
|
359
|
-
must be shared across blocks partitioned from one RDF import unit and differ
|
|
360
|
-
across unrelated units; this preserves document-scoped blank-node identity
|
|
361
|
-
when fragments are composed. The hexadecimal argument is a portable
|
|
362
|
-
diagnostic ABI, not the intended high-throughput buffer interface.
|
|
363
|
-
|
|
364
|
-
```js
|
|
365
|
-
const l4 = require('factoidal/l4-core'); // Lean engine, same API shape
|
|
366
|
-
const { select } = require('factoidal/select'); // choose an engine per call
|
|
367
|
-
```
|
|
368
|
-
|
|
369
|
-
### `factoidal/l4-core`
|
|
370
|
-
|
|
371
|
-
Same call shape as the main API — `parse`, `query`, `update`,
|
|
372
|
-
`serialize`, `canonicalize`, `graphs`, `canonicalHash`, `owlClosure`,
|
|
373
|
-
`coreRdfsClosure`, `coreRdfsCheck`, `rhoDfClosure`,
|
|
374
|
-
`rhoDfFragmentCheck`, `rdfsPlusClosure`, `owlIsConsistent` — plus four
|
|
375
|
-
Common Logic / IKL operations that exist only here, because the F\* tree
|
|
376
|
-
has no CL parser.
|
|
377
|
-
|
|
378
|
-
| Function | In → out | What the answer is worth |
|
|
379
|
-
|---|---|---|
|
|
380
|
-
| `clParse(clifText)` | CLIF text → shape report | sentence count, CL-vs-IKL dialect, canonical re-serialisation. Reads CLIF; never produces RDF |
|
|
381
|
-
| `clSerialize(clifText)` | CLIF → CLIF | canonical writer. Returns `roundTripProved: false` — `clif_roundTrip` is an open lemma and the fragment boundary is measured, not proved |
|
|
382
|
-
| `clAlphaNorm(clifText)` | CLIF → CLIF | alpha-equivalence canonical form (IKL Appendix B condition 1) |
|
|
383
|
-
| `clNormalize(clifText)` | CLIF → CLIF | Hayes's IKL-to-CL reduction. Returns `preserves: "satisfiability"` — **not** equivalence — and `noIntrusion`, the proof hypothesis decided rather than assumed |
|
|
384
|
-
|
|
385
|
-
A fifth op, `clFiniteSat(interpJson, clifText)`, is reachable only
|
|
386
|
-
through the raw dispatch ABI: `l4.call('clFiniteSat', [interpJson,
|
|
387
|
-
clifText])`. It is not in the typed layer.
|
|
388
|
-
|
|
389
|
-
### `factoidal/select`
|
|
390
|
-
|
|
391
|
-
Per-instance backend choice with a per-call override.
|
|
392
|
-
|
|
393
|
-
| Value | Behaviour |
|
|
394
|
-
|---|---|
|
|
395
|
-
| `lean` / `fstar` | that engine only; **throws** if it does not implement the function |
|
|
396
|
-
| `lean1st` / `fstar1st` | prefer that engine, fall through to the other for unimplemented functions |
|
|
397
|
-
| `slowcompareboth` | run both and **report** disagreement rather than throwing |
|
|
398
|
-
|
|
399
|
-
Every result names the engine that answered. `capabilityTable()`
|
|
400
|
-
returns which functions each engine implements.
|
|
401
|
-
|
|
402
|
-
### What this surface is, honestly
|
|
403
|
-
|
|
404
|
-
Four `String → String` operations plus one through raw dispatch. The
|
|
405
|
-
Lean tree behind them is larger than that — the CL/IKL and unified
|
|
406
|
-
model-theory modules run to about 22,000 lines — but only these reach
|
|
407
|
-
JavaScript today. Everything else in the Lean tree is used through
|
|
408
|
-
`parse`/`query`/`closure`, or not exposed at all.
|
|
409
|
-
|
|
410
|
-
## The `factoidal` command: querying a persisted store
|
|
411
|
-
|
|
412
|
-
Installing this package puts a `factoidal` command on PATH. It reads a
|
|
413
|
-
**Shardborough** store — the on-disk format the Lean `l4block-*` tools
|
|
414
|
-
write — with no native binary: JavaScript reads the files and moves the
|
|
415
|
-
bytes, and the Lean engine running as WebAssembly makes every format
|
|
416
|
-
decision (parsing the manifest, choosing the blocks, verifying their
|
|
417
|
-
SHA-256, evaluating the SPARQL).
|
|
418
|
-
|
|
419
|
-
> This command is not the native F\* `factoidal` binary that the API
|
|
420
|
-
> table below refers to. That one is `bin/<platform>/factoidal` in the
|
|
421
|
-
> repository and takes subcommands such as `shex` and `compact`. This
|
|
422
|
-
> one takes `version`, `sample-store`, `inspect` and `query`.
|
|
423
|
-
|
|
424
|
-
### First query, with nothing else to download
|
|
425
|
-
|
|
426
|
-
The package carries an activated store, so a fresh install answers a
|
|
427
|
-
SPARQL query at once:
|
|
11
|
+
import { parse, csvwToRdf } from "@factoidal/core";
|
|
12
|
+
|
|
13
|
+
const ds = await parse('<http://x/a> <http://x/n> "A" .', { format: "ntriples" });
|
|
14
|
+
const rows = await ds.query("SELECT ?n WHERE { ?s <http://x/n> ?n }");
|
|
15
|
+
console.log(rows[0].get("n").value); // A
|
|
16
|
+
console.log(ds.toString("turtle"));
|
|
17
|
+
console.log(await ds.serialize({ format: "rdfxml" }));
|
|
18
|
+
console.log(await ds.canonicalize()); // RDFC-1.0
|
|
19
|
+
|
|
20
|
+
const table = await csvwToRdf("id,name\n1,a\n", "", { mode: "minimal", url: "http://x/t.csv" });
|
|
21
|
+
console.log(table.size); // 2
|
|
22
|
+
|
|
23
|
+
ds.dispose();
|
|
24
|
+
table.dispose();
|
|
25
|
+
```
|
|
26
|
+
|
|
27
|
+
A `Dataset` keeps its data inside the engine. Every producer returns one and
|
|
28
|
+
every consumer accepts one, so a pipeline does not re-parse between steps.
|
|
29
|
+
An unknown option key throws. With no `format`, the format is inferred and
|
|
30
|
+
reported in `dataset.format`; an explicit `format` is never overridden.
|
|
31
|
+
RDF 1.2 input uses `turtle12`, `ntriples12`, `nquads12` or `trig12`.
|
|
32
|
+
|
|
33
|
+
## What it does
|
|
34
|
+
|
|
35
|
+
- Syntaxes: N-Triples, N-Quads, Turtle, TriG, RDF/XML, JSON-LD; RDF 1.1 and 1.2
|
|
36
|
+
- SPARQL 1.1 and 1.2 query and update; GeoSPARQL functions, text search,
|
|
37
|
+
JavaScript extension functions, SERVICE
|
|
38
|
+
- Reasoning: RDFS, OWL 2 RL closure, OWL 2 DL consistency and entailment, RIF Core
|
|
39
|
+
- Validation: SHACL, ShEx, JSON Schema, Schematron
|
|
40
|
+
- Mapping: CSVW, RML, JSON-LD
|
|
41
|
+
- Canonicalization: RDFC-1.0; Verifiable Credentials Data Integrity (eddsa-rdfc-2022), did:key
|
|
42
|
+
- XML: XML 1.0, XPath 1.0, XSLT 1.0, XForms
|
|
43
|
+
- Servers: SPARQL 1.1 Protocol and Graph Store (Fuseki-compatible paths,
|
|
44
|
+
Jena TDB2 read only), Linked Web Storage 1.0, Solid
|
|
45
|
+
- A persisted store with a delta log for updates (early stage)
|
|
46
|
+
- RDF/JS DataFactory and DatasetCore (`/rdfjs`), a functional API (`/fn`)
|
|
47
|
+
|
|
48
|
+
Formats (generated from `formats()`):
|
|
49
|
+
|
|
50
|
+
<!-- formats-table:start (generated by test/formats-table.test.mjs) -->
|
|
51
|
+
| format | read RDF 1.1 | read RDF 1.2 | write RDF 1.1 | write RDF 1.2 | engine |
|
|
52
|
+
|---|---|---|---|---|---|
|
|
53
|
+
| nquads | yes | yes | yes | yes | core |
|
|
54
|
+
| ntriples | yes | yes | yes | yes | core |
|
|
55
|
+
| turtle | yes | yes | yes | yes | core |
|
|
56
|
+
| trig | yes | yes | yes | yes | core |
|
|
57
|
+
| rdfxml | yes | no | yes | no | core |
|
|
58
|
+
| jsonld | yes | no | yes | no | full |
|
|
59
|
+
| rdfc | no | no | yes | no | core |
|
|
60
|
+
| sparql-results-json | no | no | yes | yes | core |
|
|
61
|
+
| sparql-results-xml | no | no | yes | yes | core |
|
|
62
|
+
| sparql-results-csv | no | no | yes | yes | core |
|
|
63
|
+
| sparql-results-tsv | no | no | yes | yes | core |
|
|
64
|
+
<!-- formats-table:end -->
|
|
65
|
+
|
|
66
|
+
Missing writers (HDT, N3, RDFa): https://github.com/danbri/factoidal/issues/883.
|
|
67
|
+
Full API: [API.md](https://github.com/danbri/factoidal-core/blob/main/npm/factoidal/API.md).
|
|
68
|
+
Changes: [CHANGELOG.md](CHANGELOG.md).
|
|
69
|
+
|
|
70
|
+
## Command line
|
|
428
71
|
|
|
429
72
|
```console
|
|
430
|
-
$
|
|
431
|
-
$ npx factoidal
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
LIMIT 4'
|
|
438
|
-
c l
|
|
439
|
-
<http://cv.iptc.org/newscodes/videocodec/c001> "Analogue Black and White"@en-gb
|
|
440
|
-
<http://cv.iptc.org/newscodes/videocodec/c002> "PAL"@en-gb
|
|
441
|
-
<http://cv.iptc.org/newscodes/videocodec/c003> "NTSC"@en-gb
|
|
442
|
-
<http://cv.iptc.org/newscodes/videocodec/c004> "SECAM"@en-gb
|
|
73
|
+
$ npx factoidal serve --file data.ttl /ds # SPARQL endpoint at /ds/sparql
|
|
74
|
+
$ npx factoidal serve --tdb2 DIR /ds # serve a Jena TDB2 database
|
|
75
|
+
$ npx factoidal pack data.ttl store/gen-1 && npx factoidal activate store gen-1
|
|
76
|
+
$ npx factoidal query store 'SELECT * WHERE { ?s ?p ?o } LIMIT 5'
|
|
77
|
+
$ npx factoidal query "$(npx factoidal sample-store)" 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
|
|
78
|
+
$ npx factoidal lws-serve DIR # Linked Web Storage
|
|
79
|
+
$ npx factoidal solid-serve DIR # Solid Protocol
|
|
443
80
|
```
|
|
444
81
|
|
|
445
|
-
`factoidal
|
|
446
|
-
|
|
82
|
+
`npx factoidal --help` lists every command. Replacing Apache Jena Fuseki:
|
|
83
|
+
[docs/fuseki-replacement.md](https://github.com/danbri/factoidal-core/blob/main/npm/factoidal/docs/fuseki-replacement.md).
|
|
447
84
|
|
|
448
|
-
|
|
449
|
-
import { sampleStorePath, sampleStoreFacts } from '@factoidal/core/sample-store'
|
|
450
|
-
```
|
|
451
|
-
|
|
452
|
-
The store holds 4,434 triples in 13 predicate blocks: five IPTC
|
|
453
|
-
NewsCodes vocabularies, published by the IPTC under CC BY 4.0 and taken
|
|
454
|
-
from [danbri/skosdex](https://github.com/danbri/skosdex). See `NOTICE`.
|
|
85
|
+
## Test results
|
|
455
86
|
|
|
456
|
-
|
|
87
|
+
Official test suites, pass / fail / total, 2026-10-09 [B] unless marked.
|
|
457
88
|
|
|
458
|
-
|
|
459
|
-
$ factoidal inspect ./mystore
|
|
460
|
-
store ./mystore
|
|
461
|
-
generation gen-1 (activated through CURRENT)
|
|
462
|
-
manifest manifest.sbm2, 2372 bytes, wire version 6
|
|
463
|
-
layout predicate-ibk3-ptd1-sri2-tli1-oli2-merkle-v0
|
|
464
|
-
blank-node profile (none recorded)
|
|
465
|
-
term registry local-ibk3-ptd1-v0
|
|
466
|
-
fixed-chunk Merkle commitment yes
|
|
467
|
-
5 entries, 393775 bytes, 6455 rows
|
|
468
|
-
generation directory holds 42 files, 846592 bytes
|
|
469
|
-
|
|
470
|
-
# rows bytes kind graphs predicate
|
|
471
|
-
0 1800 110085 IBK3 - http://www.wikidata.org/prop/direct/P31
|
|
472
|
-
1 719 35535 IBK3 - http://www.wikidata.org/prop/direct/P361
|
|
473
|
-
...
|
|
474
|
-
|
|
475
|
-
$ factoidal query ./mystore 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
|
|
476
|
-
mode ibk3-paged-merkle-full-manifest(5), 5 artifacts, 393775 bytes read, plan declares 6455 block rows
|
|
477
|
-
n
|
|
478
|
-
"6455"^^<http://www.w3.org/2001/XMLSchema#integer>
|
|
479
|
-
1 row
|
|
480
|
-
```
|
|
481
|
-
|
|
482
|
-
`STORE` is a collection root: the directory holding `CURRENT`. The plan
|
|
483
|
-
line goes to stderr, so stdout carries only the result; `--quiet`
|
|
484
|
-
removes it.
|
|
485
|
-
|
|
486
|
-
| Option | What it does |
|
|
89
|
+
| Suite | Result |
|
|
487
90
|
|---|---|
|
|
488
|
-
|
|
|
489
|
-
|
|
|
490
|
-
|
|
|
491
|
-
|
|
|
492
|
-
|
|
|
493
|
-
|
|
|
494
|
-
|
|
|
495
|
-
|
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
91
|
+
| SPARQL 1.1 Query, Update, Protocol | 631 / 0 / 631 |
|
|
92
|
+
| SPARQL 1.2 | 254 / 0 / 254 |
|
|
93
|
+
| SPARQL Protocol, Graph Store, Service Description against `serve` | 50 / 0 / 50 [P] |
|
|
94
|
+
| RDF 1.1 N-Triples, N-Quads, Turtle, TriG, RDF/XML | 992 / 0 / 992 |
|
|
95
|
+
| RDF 1.2 syntax suites | all pass |
|
|
96
|
+
| RDF 1.1 Semantics | 39 / 0 / 39 |
|
|
97
|
+
| RDF 1.2 Semantics | 40 / 7 / 47 |
|
|
98
|
+
| JSON-LD 1.1 toRdf, expand, flatten | 910 / 0 / 910 |
|
|
99
|
+
| JSON-LD 1.1 compact, fromRdf | 298 / 0 / 300 (2 not run) |
|
|
100
|
+
| JSON-LD 1.1 Framing | 29 / 63 / 92 |
|
|
101
|
+
| RDFC-1.0 | 86 / 0 / 86 |
|
|
102
|
+
| OWL 2 RL | 126 / 0 / 126 |
|
|
103
|
+
| OWL 2 DL type-inconsistency | 116 / 11 / 128 (1 other) |
|
|
104
|
+
| OWL 2 DL positive entailment | 169 / 35 / 206 (2 other) |
|
|
105
|
+
| SHACL 1.0 Core, SHACL-SPARQL | 120 / 0 / 120 |
|
|
106
|
+
| SHACL 1.2 Core | 103 / 30 / 133 |
|
|
107
|
+
| ShEx 2 validation, ShExC | 1624 / 0 / 1624 |
|
|
108
|
+
| CSVW | 821 / 1 / 822 |
|
|
109
|
+
| RIF Core | 43 / 0 / 46 (3 undecided) |
|
|
110
|
+
| RML Core | 60 / 0 / 76 (16 not attempted) |
|
|
111
|
+
| XML 1.0 (xmlconf) | 1861 / 0 (516 not run) |
|
|
112
|
+
| XSD 1.1 datatypes | 18951 / 5 / 18956 |
|
|
113
|
+
| XSLT 1.0 | 84 / 3 / 88 (1 refused) |
|
|
114
|
+
| JSON Schema draft-07 | 770 / 0 / 770 |
|
|
115
|
+
| VC Data Integrity, did:key, Ed25519 | 58 / 0 / 58 |
|
|
116
|
+
| XPath 1.0 (own battery) | 100 / 0 / 100 [X] |
|
|
117
|
+
|
|
118
|
+
[B] https://github.com/danbri/factoidal-core/blob/03e2e6deee8146ce7e6fdc3f8f34b8245af5ab54/docs/designissues/2026-10-09-lean-conformance-baseline.md
|
|
119
|
+
[P] https://github.com/danbri/factoidal-core/blob/9a4045f1572a032b53ccb867f8395fe0c8d97bb9/docs/designissues/2026-10-10-store-serve.md
|
|
120
|
+
[X] https://github.com/danbri/factoidal-core/blob/cb871c2de8354362ae6226eb57fb6c549624ea1a/docs/designissues/2026-10-10-xml-profile-widened.md
|
|
121
|
+
|
|
122
|
+
## Limits
|
|
123
|
+
|
|
124
|
+
- Size: 2.5 MB packed, 12.4 MB unpacked; one 11.7 MB wasm module.
|
|
125
|
+
- `factoidal update` and `compact` exit 3; updates work through `serve`
|
|
517
126
|
(https://github.com/danbri/factoidal/issues/641).
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
Measured 2026-09-03 on macOS arm64, the 6455-triple `sequence_variant`
|
|
526
|
-
store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
|
|
527
|
-
including start-up: 220 ms through this command, 33 ms through the
|
|
528
|
-
native `l4block-id-v3-query`.
|
|
529
|
-
|
|
530
|
-
### Holding a store open: verify once, answer many
|
|
531
|
-
|
|
532
|
-
`queryStore` is stateless. Every call transfers the artifacts again, and
|
|
533
|
-
the engine hashes, decodes and indexes them again. A process that answers
|
|
534
|
-
many questions against one generation — a chat bot, an MCP server, a
|
|
535
|
-
SPARQL endpoint — pays all of that for every question, and none of it
|
|
536
|
-
depends on the question.
|
|
537
|
-
|
|
538
|
-
A **store handle** holds that work. `openStoreHandle` verifies each
|
|
539
|
-
artifact against the SHA-256 the manifest commits, decodes each block and
|
|
540
|
-
indexes the rows, once. `handle.query()` then answers from what it
|
|
541
|
-
retained. Several stores can be open at the same time.
|
|
542
|
-
|
|
543
|
-
```js
|
|
544
|
-
import { loadEngine } from '@factoidal/core/bin/engine.mjs'
|
|
545
|
-
import { openStore, openStoreHandle, listStoreHandles } from '@factoidal/core/store'
|
|
546
|
-
|
|
547
|
-
const engine = await loadEngine()
|
|
548
|
-
|
|
549
|
-
// Two stores, open at once, held for the life of the process.
|
|
550
|
-
const skos = openStoreHandle(engine, openStore('./skos-store'))
|
|
551
|
-
const docs = openStoreHandle(engine, openStore('./docs-store'))
|
|
552
|
-
|
|
553
|
-
console.log(listStoreHandles(engine))
|
|
554
|
-
// { ok: true, handles: [ {handle:'s1', bytes:…, rows:…}, {handle:'s2', …} ],
|
|
555
|
-
// bytes: …, rows: …, handleCap: 8, bytesCap: 134217728 }
|
|
556
|
-
|
|
557
|
-
const PREFIX = 'PREFIX skos: <http://www.w3.org/2004/02/skos/core#>'
|
|
558
|
-
|
|
559
|
-
function labelsMatching (handle, needle) {
|
|
560
|
-
const answer = handle.query(`${PREFIX}
|
|
561
|
-
SELECT ?c ?l WHERE {
|
|
562
|
-
GRAPH ?g { ?c skos:prefLabel ?l }
|
|
563
|
-
FILTER(CONTAINS(LCASE(STR(?l)), "${needle.toLowerCase()}"))
|
|
564
|
-
} LIMIT 10`)
|
|
565
|
-
return answer.srj.results.bindings
|
|
566
|
-
}
|
|
567
|
-
|
|
568
|
-
// Many questions, each a new search string. None of them re-reads a block.
|
|
569
|
-
for (const needle of ['water', 'forest', 'railway', 'volcano']) {
|
|
570
|
-
console.log(needle, labelsMatching(skos, needle).length)
|
|
571
|
-
}
|
|
572
|
-
console.log(docs.query(`${PREFIX} SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`))
|
|
573
|
-
|
|
574
|
-
skos.close()
|
|
575
|
-
docs.close()
|
|
576
|
-
```
|
|
577
|
-
|
|
578
|
-
`openStoreHandle(engine, store, options)` takes:
|
|
579
|
-
|
|
580
|
-
| option | effect |
|
|
581
|
-
|---|---|
|
|
582
|
-
| (none) | open every artifact the manifest declares |
|
|
583
|
-
| `{ sparql }` | open only the artifacts that query's plan names |
|
|
584
|
-
| `{ keys }` | open exactly these artifact keys |
|
|
585
|
-
|
|
586
|
-
`queryStoreHandle(engine, handle, sparql)`, `closeStoreHandle(engine,
|
|
587
|
-
handle)` and `listStoreHandles(engine)` are the same operations for a
|
|
588
|
-
caller that holds only the handle id.
|
|
589
|
-
|
|
590
|
-
**Measured 2026-09-04**, macOS arm64, Node 22, the 141-graph SKOS store,
|
|
591
|
-
`skos:prefLabel` block of 5,571,302 bytes and 45,806 rows, one
|
|
592
|
-
`CONTAINS` query per row, a DIFFERENT search string every time:
|
|
593
|
-
|
|
594
|
-
| | stateless `queryStore` | handle |
|
|
595
|
-
|---|---|---|
|
|
596
|
-
| first query (open + query) | 1376 ms | 1363 ms |
|
|
597
|
-
| second query, different search string | 1376 ms | 95 ms |
|
|
598
|
-
| tenth query, all different | 1382 ms | 103 ms |
|
|
599
|
-
| ten queries, total | 13962 ms | 2277 ms |
|
|
600
|
-
|
|
601
|
-
**What a handle buys.** It removes the per-query digest check, block
|
|
602
|
-
decode and index build. With the LGI1 literal index (below) a search is
|
|
603
|
-
also sub-linear rather than a scan, so a handle plus the index is what
|
|
604
|
-
makes repeated search on a large store interactive.
|
|
605
|
-
|
|
606
|
-
**Residency.** Retaining a decoded block costs memory. Measured on the
|
|
607
|
-
same store: 76 MiB resident with the engine loaded and no handle, 170 MiB
|
|
608
|
-
with the handle open — about 94 MiB for a 5.5 MB packed block, and
|
|
609
|
-
evaluation peaks higher again (346 MiB during the queries above). At
|
|
610
|
-
corpus scale the marginal figure is smaller, because that one is carrying
|
|
611
|
-
the fixed cost of the process: measured 2026-09-05 on a 7,315,251-quad
|
|
612
|
-
store, one handle over 257 blocks and 103,341,302 retained artifact bytes
|
|
613
|
-
peaked at 1,675,345,920 bytes resident — 16.2 bytes resident per retained
|
|
614
|
-
byte.
|
|
615
|
-
|
|
616
|
-
The caps are on ARTIFACT bytes, which is what the manifest declares: 8
|
|
617
|
-
open handles, and 134217728 (128 MiB) retained artifact bytes across all
|
|
618
|
-
of them. There is NO cap on the number of artifacts a handle retains; a
|
|
619
|
-
count bounds nothing that the bytes do not
|
|
620
|
-
(https://github.com/danbri/factoidal/issues/657). 128 MiB is half the
|
|
621
|
-
wasm32 address space divided by that measured multiplier, and it admits
|
|
622
|
-
the 257-block corpus-wide set above. A cap is a refusal naming the cap
|
|
623
|
-
and the value that tripped it; no handle is ever evicted to make room for
|
|
624
|
-
another. `listStoreHandles` is how a server sees its own residency.
|
|
625
|
-
|
|
626
|
-
**One call at a time.** The WebAssembly module is single-threaded. Two
|
|
627
|
-
`query()` calls cannot overlap; a server queues them.
|
|
628
|
-
|
|
629
|
-
### A handle on a large store: the worker route
|
|
630
|
-
|
|
631
|
-
Several engine paths recurse once per manifest entry and once per row.
|
|
632
|
-
Against a large collection that exceeds the default call stack of Node
|
|
633
|
-
and of Deno, and the failure is `Maximum call stack size exceeded`.
|
|
634
|
-
**Measured 2026-09-05**, macOS arm64, Node 22.22.2, on a 7,315,251-quad
|
|
635
|
-
collection of 3,286 blocks in 204 graphs: `storeQueryPlan` alone
|
|
636
|
-
overflows on plain `node`, before one artifact byte is read.
|
|
637
|
-
`node --stack-size=60000` clears the plan, the open and every query.
|
|
638
|
-
|
|
639
|
-
`openStoreHandleOnWorker` removes the flag. It holds the engine and the
|
|
640
|
-
handle on a `worker_threads` thread with `resourceLimits.stackSizeMb`,
|
|
641
|
-
the route `factoidal pack` already takes
|
|
642
|
-
([issue 649](https://github.com/danbri/factoidal/issues/649)). A handle
|
|
643
|
-
is state inside the wasm instance and an instance does not cross a
|
|
644
|
-
thread boundary, so the handle lives where the raised stack is, and
|
|
645
|
-
`query()` and `close()` are messages to it.
|
|
646
|
-
|
|
647
|
-
```js
|
|
648
|
-
import { openStoreHandleOnWorker, closeSharedStoreWorkerSession }
|
|
649
|
-
from '@factoidal/core/store-worker'
|
|
650
|
-
|
|
651
|
-
// No runtime flag. One worker thread, shared by every handle opened
|
|
652
|
-
// this way, so a caller that opens several stores pays for one thread
|
|
653
|
-
// and one copy of the module.
|
|
654
|
-
const handle = await openStoreHandleOnWorker('/path/to/store', {
|
|
655
|
-
sparql: 'SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c ?p ?l } }'
|
|
656
|
-
})
|
|
657
|
-
const answer = await handle.query(`${PREFIX}
|
|
658
|
-
SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c skos:prefLabel ?l }
|
|
659
|
-
FILTER(CONTAINS(LCASE(STR(?l)), "volcan")) } LIMIT 8`)
|
|
660
|
-
await handle.close()
|
|
661
|
-
await closeSharedStoreWorkerSession()
|
|
662
|
-
```
|
|
663
|
-
|
|
664
|
-
| option | effect |
|
|
665
|
-
|---|---|
|
|
666
|
-
| `{sparql}`, `{keys}` | the same artifact choice `openStoreHandle` takes |
|
|
667
|
-
| `{generation}` | open a generation that has not been activated |
|
|
668
|
-
| `{session}` | open into a session you started with `openStoreWorkerSession()` |
|
|
669
|
-
| `{ownWorker: true}` | give this handle its own thread and its own copy of the module |
|
|
670
|
-
| `{worker: false}` | open in this process, with no thread |
|
|
671
|
-
|
|
672
|
-
What it costs, all **measured 2026-09-05** on the same machine, against
|
|
673
|
-
the bundled sample store so the overhead is not lost in the query:
|
|
674
|
-
|
|
675
|
-
| | in process | on a worker |
|
|
676
|
-
|---|---|---|
|
|
677
|
-
| one-shot query (thread start + engine load + query) | 163 ms | 220 ms |
|
|
678
|
-
| handle open (thread start + engine load + open) | 124 ms | 236 ms |
|
|
679
|
-
| every query after the open | 1 ms | 1 ms |
|
|
680
|
-
|
|
681
|
-
So the thread and its second copy of the engine cost about 110 ms once,
|
|
682
|
-
and the message round trip is under a millisecond. Three further costs:
|
|
683
|
-
every call is asynchronous where the in-process handle is synchronous;
|
|
684
|
-
an extension function registered on the main thread's engine
|
|
685
|
-
(`bin/ext.mjs`) is not visible to the worker's engine; and the worker
|
|
686
|
-
keeps the process alive until `close()`.
|
|
687
|
-
|
|
688
|
-
**Deno takes a different route.** Deno's `node:worker_threads` shim
|
|
689
|
-
accepts `resourceLimits.stackSizeMb` and raises almost nothing with it —
|
|
690
|
-
measured by counting frames to the overflow inside the worker, Node
|
|
691
|
-
reaches 41,195 frames by default and 696,555 at `stackSizeMb` 64, where
|
|
692
|
-
Deno reaches 10,835 and 13,837. `openStoreHandleOnWorker` therefore
|
|
693
|
-
gives a Deno caller an in-process handle behind the same asynchronous
|
|
694
|
-
interface, and the process supplies the stack:
|
|
695
|
-
|
|
696
|
-
```
|
|
697
|
-
deno run --allow-read --v8-flags=--stack-size=65536 your-program.mjs
|
|
698
|
-
```
|
|
699
|
-
|
|
700
|
-
**A one-shot `factoidal query` pays none of this.** It builds no handle,
|
|
701
|
-
so it runs in process and only retries on a worker if the runtime runs
|
|
702
|
-
out of frames; under Deno it re-executes itself once with a raised V8
|
|
703
|
-
stack, which needs `--allow-run` and `--allow-env`. `--no-worker` turns
|
|
704
|
-
the retry off. See
|
|
705
|
-
[issue 653](https://github.com/danbri/factoidal/issues/653).
|
|
706
|
-
|
|
707
|
-
## API (draft)
|
|
708
|
-
|
|
709
|
-
The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to
|
|
710
|
-
`bin/<platform>/factoidal`) calls the exact same F\*-extracted
|
|
711
|
-
functions as this npm surface — see `tests/local/cli_api_parity.sh`
|
|
712
|
-
for the test that diffs CLI output against the npm API on shared
|
|
713
|
-
fixtures.
|
|
714
|
-
|
|
715
|
-
| Function | Signature (informal) | CLI equivalent | Notes |
|
|
716
|
-
|---|---|---|---|
|
|
717
|
-
| `parse` | `(text, {format?, baseIRI?}) => Dataset` | `factoidal dump-nq -d FILE` (or `dump`/`dump-turtle`) | formats: `turtle`, `ntriples`, `nquads`, `trig`, `rdfxml`, `jsonld`\* — auto-detected where possible. Each call is one document: blank-node labels are scoped per RDF 1.1 |
|
|
718
|
-
| `query` | `(Dataset \| string, sparql, {entail?}) => Bindings[] \| boolean \| Dataset` | `factoidal query -d FILE -e 'SPARQL' [--entail RDFS\|OWL-RL]` | SELECT → array of `Map<var, Term>`; ASK → boolean; CONSTRUCT → Dataset\*\*; `entail: "RDFS" \| "OWL-RL"` |
|
|
719
|
-
| `update` | `(Dataset, sparqlUpdate) => Dataset` | `factoidal update -d FILE -e 'SPARQL update'` | \*\* in-memory; no persistence. (Durable UPDATE against a COTTAS store is a separate path: `factoidal serve --rw --delta-log ...` / `factoidal compact`.) |
|
|
720
|
-
| `serialize` | `(Dataset, {format}) => string` | `factoidal dump-nq FILE` (nquads) / `factoidal dump FILE` (ntriples) / `factoidal dump-turtle FILE` | `nquads`, `ntriples` (sorted); `turtle`\*\* (prefix-compacted, subject-grouped — needs the entry bundle, flattens named graphs into the default graph) |
|
|
721
|
-
| `canonicalize` | `(Dataset \| string) => string` | `factoidal canonicalize FILE` | RDFC-1.0 canonical N-Quads\*\* |
|
|
722
|
-
| `graphs` | `(Dataset) => Array<[iri, Dataset]>` | `factoidal graphs list FILE` | enumerate named graphs (default graph excluded); pure enumeration, no engine round-trip |
|
|
723
|
-
| `canonicalHash` | `(Dataset) => string` | `factoidal graphs hash FILE IRI` | RDFC-1.0 canonical hash of one graph\*\*; graph-scoped sibling of `canonicalize` — typically called with one entry of `graphs()`'s output |
|
|
724
|
-
| `shaclValidate` | `(data, shapes) => {conforms, report: Dataset}` | `factoidal shacl --data FILE --shapes FILE [--json]` (alias: `factoidal validate --shapes FILE FILE`) | \*\* SHACL Core validation; `report` is the `sh:ValidationReport` graph; exit code 0 iff `sh:conforms` |
|
|
725
|
-
| `shexValidate` | `(data, schemaText, focus, shape?) => boolean \| null` | `factoidal shex --data FILE --schema FILE.{json,shex} --node N [--shape S]` | \*\* ShEx (Shape Expressions) validation of one focus node; `schemaText` accepts ShExJ or ShExC — dispatched by the first non-whitespace character (`{` ⇒ ShExJ, else ShExC; the CLI additionally honors a `.shex` file extension); `null` = outside this engine's decidable ShEx fragment, never a guessed answer |
|
|
726
|
-
| `owlClosure` | `(data, mode) => Dataset` | `factoidal entail --data FILE --regime RDFS\|OWL-RL` | \*\* materializes the entailment closure (input + derived triples), default graph only. (`query --entail` applies the same closure internally before evaluating a query, but does not dump it on its own.) |
|
|
727
|
-
| `rmlMap` | `(mapping, sourceData, sourceKind) => Dataset` | `factoidal rml --mapping FILE --source FILE --kind json\|csv` | \*\* evaluates an RML mapping graph against one logical source (`sourceKind: "json" \| "csv"`); every triples map reads the SAME source — cross-source joins are out of scope for this entry point |
|
|
728
|
-
| `csvwToRdf` | `(csvText, metadataJson?, {mode?, base?, url?}) => Dataset` | `factoidal csvw --csv FILE [--metadata FILE] [--minimal] [--base IRI] [--url URL]` | \*\* CSVW csv2rdf conversion; metadata omitted = schema inferred from the CSV header row; `mode: "standard" \| "minimal"` (default standard); every table in a multi-table group reads the SAME csvText |
|
|
729
|
-
| `jsonldToRdf` | `(jsonldText, {base?, rdfDirection?, expandContext?, processingMode?}) => Dataset` | `factoidal jsonld --in FILE [--base IRI]` (or `factoidal dump-nq FILE.jsonld`, format auto-detected) | \*\* JSON-LD parsing with options `parse()` has no room for; plain `parse(text, {format:'jsonld'})` also works for the common case |
|
|
730
|
-
| `jsonldFromRdf` | `(data, {useNativeTypes?, useRdfType?}) => JSON-LD` | `factoidal dump-nq` ← (reverse) | \*\* serializes an RDF dataset as expanded-form JSON-LD (the reverse of `jsonldToRdf`); returns the parsed JSON-LD document (array of node objects), JCS-canonical |
|
|
731
|
-
| `didKeyResolve` | `(didString) => Dataset` | N/A (`did_runner`) | \*\* resolves a `did:key:z6Mk...` (Ed25519) to its DID Document as RDF; non-Ed25519 / malformed inputs reject rather than guess |
|
|
732
|
-
| `xmlWellformed` | `(xmlText) => boolean` | N/A (`xml_runner`) | \*\* XML 1.0 well-formedness check (byte-oriented, no DOCTYPE/DTD production — a document with a DOCTYPE reports `false`) |
|
|
733
|
-
| `xpathEval` | `(xmlText, xpathExpr) => {resultType, value \| nodes, ...}` | N/A | \*\* XPath 1.0 evaluation over an XML document; `resultType` is `'nodeset' \| 'string' \| 'number' \| 'boolean'` |
|
|
734
|
-
| `rifEval` | `(data, rifRulesXml) => Dataset` | `factoidal rif --rules FILE --data FILE` | \*\* RIF Core forward-chaining saturation (materializes input + derived triples); accepts real vendored RIF-XML (`<!DOCTYPE>` + `&rif;`/`&xs;`/`&rdf;` entities) unmodified |
|
|
735
|
-
| `toCottas` | `(data, {format?}) => Uint8Array` | `factoidal compact --native-writer` | \*\* serializes a dataset to COTTAS/Parquet bytes via the native writer; round-trips into `openCottas()` and into the native `--data-cottas`/`--data-cottas-mem` CLI flags byte-for-byte |
|
|
736
|
-
| `openCottas` | `(bytes: string \| Uint8Array \| ArrayBuffer) => handle` | `factoidal query --data-cottas-mem FILE` | \*\* opens a whole `.cottas` artifact's bytes as a queryable, read-only, in-memory store; rows decode lazily as `queryCottas()` touches them (no heap `Dataset`, no full parse) |
|
|
737
|
-
| `queryCottas` | `(handle, sparql) => Bindings[] \| boolean \| Dataset` | `factoidal query --data-cottas-mem FILE -e 'SPARQL'` | \*\* SPARQL over a store opened by `openCottas()`; no `entail` option, no write overlay (read-only), no DESCRIBE |
|
|
738
|
-
| `closeCottas` | `(handle) => void` | N/A | releases a handle from this process's registry; does not evict the underlying byte cache |
|
|
739
|
-
| `queryRaw` | `(input, sparql) => string` | `factoidal query -d FILE -e 'SPARQL' -o json` | SPARQL-Results-JSON string, for callers that want the wire form |
|
|
740
|
-
| `capabilities` | `() => {construct, update, canonicalize, graphs, canonicalHash, shacl, shex, owlClosure, rml, csvw, jsonld, jsonldFromRdf, didKey, xml, xpath, rif, cottasBytesStore, ...}` | N/A | runtime feature probe; the CLI is one fixed native binary, not a runtime bundle whose feature set varies |
|
|
741
|
-
| `dataFactory` | RDF/JS DataFactory | N/A | data-model class, not an engine operation |
|
|
742
|
-
| `Dataset` | RDF/JS DatasetCore | N/A | returned by `parse`; accepted everywhere |
|
|
743
|
-
|
|
744
|
-
The `fn.js` functional layer's own combinators — `union`, `difference`,
|
|
745
|
-
`filter`, `mapQuads`, `equals`, `hash`, `builder`/`fromChunks`,
|
|
746
|
-
`cell`/`derive` — are pure client-side set algebra and dataflow
|
|
747
|
-
plumbing over already-materialized `Dataset`s. They have no CLI
|
|
748
|
-
equivalent by design: there is no engine operation to wrap, only JS
|
|
749
|
-
composition on top of the operations already listed above.
|
|
750
|
-
|
|
751
|
-
\* JSON-LD parsing (expanded form, inline `@context`, `@base`
|
|
752
|
-
resolution, `@reverse`, container maps) works through both `parse()`
|
|
753
|
-
and `jsonldToRdf()` when the npm-entry bundle is loaded. Remote
|
|
754
|
-
`@context` URLs need a `documentLoader`, which this package's entries
|
|
755
|
-
don't register (an honest failure, not a silent wrong answer) —
|
|
756
|
-
tracked against the vendored W3C json-ld-api suite.
|
|
757
|
-
\*\* CONSTRUCT, UPDATE, `canonicalize`, `canonicalHash`,
|
|
758
|
-
`shaclValidate`, `shexValidate`, `owlClosure`, `rmlMap`, `csvwToRdf`,
|
|
759
|
-
`jsonldToRdf`, `jsonldFromRdf`, `didKeyResolve`, `xmlWellformed`, `xpathEval`, `rifEval`, `toCottas`, `openCottas`, `queryCottas`, and
|
|
760
|
-
`closeCottas` are probed via `capabilities()`: they activate
|
|
761
|
-
automatically when the dedicated npm-entry engine bundle is present,
|
|
762
|
-
and the package reports their absence honestly against older bundles
|
|
763
|
-
instead of guessing.
|
|
764
|
-
`canonicalHash` rides the same engine support as `canonicalize` (it
|
|
765
|
-
computes `canonicalize()` over one graph's triples); `graphs` is pure
|
|
766
|
-
JS enumeration and is always available.
|
|
767
|
-
|
|
768
|
-
### The db API (openCottas/queryCottas/closeCottas/toCottas)
|
|
769
|
-
|
|
770
|
-
The in-memory COTTAS/Parquet bytes store works identically on both
|
|
771
|
-
engines: `require('@factoidal/core')` (js_of_ocaml) and
|
|
772
|
-
`require('factoidal/wasm')` (wasm_of_ocaml, Node ≥ 22) expose the same
|
|
773
|
-
`toCottas`/`openCottas`/`queryCottas`/`closeCottas` functions, backed
|
|
774
|
-
by the same F\*-verified reader
|
|
775
|
-
(`RDF.CottasStore`/`Parquet.Footer`/`SPARQL11_Store`). A store is
|
|
776
|
-
read-only and holds no `entail` option — see the divergence list on
|
|
777
|
-
`queryCottas`'s doc comment in `lib/api.js`. Zstd-compressed COTTAS
|
|
778
|
-
pages are supported under js_of_ocaml (via the vendored `fzstd`
|
|
779
|
-
decompressor baked into `factoidal.js`); under wasm_of_ocaml, Zstd
|
|
780
|
-
decompression is still an identity-stubbed primitive (a documented
|
|
781
|
-
gap, tracked separately) — write bytes with `toCottas()` (which never
|
|
782
|
-
reaches for Zstd on small in-memory writes) rather than a
|
|
783
|
-
Zstd-compressed on-disk fixture if you need a wasm-portable test case.
|
|
784
|
-
The browser entries mirror this: `browser.js`'s `openCottas`/
|
|
785
|
-
`queryCottas`/`closeCottas`/`toCottas` drive the js_of_ocaml npm-entry
|
|
786
|
-
ABI over `fetch()`; `browser-wasm.js` exposes the same four functions
|
|
787
|
-
against the wasm_of_ocaml npm-entry ABI (`factoidal-npm-entry.wasm.js`).
|
|
788
|
-
|
|
789
|
-
## Capability matrix
|
|
790
|
-
|
|
791
|
-
Every public function, where it runs, and what it needs. "Node" is the
|
|
792
|
-
`require('@factoidal/core')` / `import '@factoidal/core'` entry (`index.js` /
|
|
793
|
-
`index.mjs`); "Browser" is `import 'factoidal/browser'` (`browser.js`);
|
|
794
|
-
"Wasm" is `require('factoidal/wasm')`. **Needs** legend: *CLI bundle* =
|
|
795
|
-
the fresh-eval `factoidal.js` bundle (always present); *entry* = the
|
|
796
|
-
persistent `factoidal-npm-entry.js` ABI bundle (probe with
|
|
797
|
-
`capabilities()`); *HACL\* init* = the wasm crypto backend must be
|
|
798
|
-
initialised (auto on Node, explicit in the browser — see below);
|
|
799
|
-
*IndexedDB* = a browser storage layer.
|
|
800
|
-
|
|
801
|
-
| Function(s) | Node | Browser | Wasm | Needs |
|
|
802
|
-
|---|---|---|---|---|
|
|
803
|
-
| `parse`, `query` (SELECT/ASK), `serialize` (nquads/ntriples), `canonicalize`, `graphs`, `canonicalHash`, `queryHdt`, `queryRaw` | ✓ | ✓\* | ✓ | CLI bundle |
|
|
804
|
-
| `query` (CONSTRUCT), `update`, `serialize` (turtle) | ✓ | ✓\* | ✓ | entry |
|
|
805
|
-
| `shaclValidate`, `shexValidate`, `owlClosure`, `rmlMap`, `csvwToRdf`, `jsonldToRdf`, `jsonldFromRdf`, `didKeyResolve`, `xmlWellformed`, `xpathEval`, `rifEval` | ✓ | ✓ | partial† | entry |
|
|
806
|
-
| `coreRdfsClosure`/`rhoDfClosure`, `coreRdfsCheck`/`rhoDfFragmentCheck`, `rdfsPlusClosure`, `tableauMaterialise`, `tableauDlInconsistent`, `owlIsConsistent`, `owlEntails` | ✓ | ✓ | wrapper only‡ | entry |
|
|
807
|
-
| `xsltTransform`, `mathmlEval`, `xformsRecalc`, `jsonSchemaValidate`, `schematronValidate`, `toan*`, `matrix*` | ✓ | ✓ | partial† | entry |
|
|
808
|
-
| `openCottas`, `queryCottas`, `closeCottas`, `toCottas` | ✓ | ✓ | ✓ | entry |
|
|
809
|
-
| `vcSha256Hex`, `vcEd25519SecretToPublic`, `vcEd25519Sign`, `vcEd25519Verify`, `vcEddsaCreateFromCanonical`, `vcEddsaVerifyFromCanonical` | ✓ | ✓ | ✓ | entry + HACL\* init |
|
|
810
|
-
| `deltaLogOpen`/`Append`/`ReadAllHex`/`Merge`/`Destroy` | ✗ | ✓ | ✗ | entry + IndexedDB |
|
|
811
|
-
| `dataFactory`, `Dataset`, `fn.*` combinators (`union`/`difference`/`filter`/`mapQuads`/`equals`/`hash`/`builder`/`cell`/`derive`/`pipe`) | ✓ | — | ✓ | none (pure JS) |
|
|
812
|
-
| `capabilities` | ✓ | — | ✓ | none (probe) |
|
|
813
|
-
|
|
814
|
-
\* In the browser the same capability is reached through `browser.js`'s
|
|
815
|
-
own function names, which differ from the Node API: SELECT/ASK is
|
|
816
|
-
`query(dataString, sparql, {output})` (returns SPARQL-Results JSON, not
|
|
817
|
-
`Bindings[]`); parse-to-N-Quads is `toRdf()`; multi-graph queries use
|
|
818
|
-
`queryDataset()`. CONSTRUCT/UPDATE/Turtle ride the same `entry` bundle,
|
|
819
|
-
fetched over the network on first use.
|
|
820
|
-
† Wasm (`require('factoidal/wasm')`) today re-exports the core +
|
|
821
|
-
validation/inference/COTTAS set (`parse`/`query`/`update`/`serialize`/
|
|
822
|
-
`canonicalize`/`shaclValidate`/`shexValidate`/`owlClosure`/`rmlMap`/
|
|
823
|
-
`csvwToRdf`/`jsonldToRdf`/`rifEval`/`openCottas`…/`capabilities`); the
|
|
824
|
-
typed-engine `#74` functions and the `vc*`/`did`/`xml`/`xpath` wrappers
|
|
825
|
-
are exposed on the js and browser entries. The underlying wasm ABI
|
|
826
|
-
carries them — the `/wasm` re-export surface is being brought to parity.
|
|
827
|
-
‡ `coreRdfsClosure`/`rhoDfClosure`/`coreRdfsCheck`/`rhoDfFragmentCheck`/
|
|
828
|
-
`rdfsPlusClosure`/`tableauMaterialise`/`tableauDlInconsistent`/
|
|
829
|
-
`owlIsConsistent`/`owlEntails` are exported from `factoidal/wasm`
|
|
830
|
-
(wrapper wiring — the same `buildApi()` `index.js` uses), but the
|
|
831
|
-
committed `factoidal-npm-entry.wasm.js` ABI bundle predates these
|
|
832
|
-
functions (built before they landed on the ABI) — calling them on the
|
|
833
|
-
wasm engine throws the existing "pending npm-entry bundle" error until
|
|
834
|
-
that bundle is rebuilt via a real `wasm_of_ocaml` build (not a copy).
|
|
835
|
-
`capabilities()` on the wasm engine reports this honestly (`tableau:
|
|
836
|
-
false` etc.) rather than guessing.
|
|
837
|
-
|
|
838
|
-
### VC crypto: the init story
|
|
839
|
-
|
|
840
|
-
The `vc*` functions run the F\*-extracted VC Data Integrity pipeline
|
|
841
|
-
over HACL\*'s official WebAssembly build, so the wasm backend has to be
|
|
842
|
-
initialised before the first call (a verify against an uninitialised
|
|
843
|
-
backend **throws** — it never silently returns `true`; issue #286).
|
|
844
|
-
|
|
845
|
-
- **Node** (`index.js`/`index.mjs`, `/wasm`, `fn`): the typed wrappers
|
|
846
|
-
**auto-await `initHacl()` on first call** — a caller writes
|
|
847
|
-
`await vcEd25519Verify(pk, msg, sig)` and never touches init.
|
|
848
|
-
- **Browser** (`browser.js`): **explicit init required** — the
|
|
849
|
-
`hacl-wasm` URL is page-specific. Serve `hacl-wasm/` next to the page
|
|
850
|
-
and `await initHacl({ apiUrl })` from `hacl-init.js` once, then call
|
|
851
|
-
the `vc*` wrappers. (Auto-init isn't possible without a URL to fetch.)
|
|
852
|
-
|
|
853
|
-
### Native-CLI-only capabilities (not on the npm surface, by design)
|
|
854
|
-
|
|
855
|
-
These live only in `bin/factoidal-cli` (the native `factoidal` binary),
|
|
856
|
-
not in the JS package — they are I/O- or process-shaped, not pure
|
|
857
|
-
value transforms:
|
|
858
|
-
|
|
859
|
-
- **SPARQL endpoint / federation.** `factoidal serve` (HTTP endpoint),
|
|
860
|
-
`SERVICE` federation and `--data-cottas`/`--data-hdt` *file-path*
|
|
861
|
-
backends read the filesystem/network at query time. The npm
|
|
862
|
-
`queryHdt`/`openCottas` take *bytes* in-process instead.
|
|
863
|
-
- **Durable UPDATE against on-disk COTTAS.** `factoidal serve --rw
|
|
864
|
-
--delta-log …` + `factoidal compact` — the npm `update()` is
|
|
865
|
-
in-memory only; the browser `deltaLog*` family is the closest
|
|
866
|
-
in-package durable path (IndexedDB, browser-only).
|
|
867
|
-
- **Internal primitives.** `runFactoidalCli` / `loadNpmEntry` (browser)
|
|
868
|
-
are exported but are the low-level bundle drivers, not the intended
|
|
869
|
-
API; the `_*` functions (e.g. `_deltaLogCorruptLastForTest`) are
|
|
870
|
-
test-only and intentionally left untyped.
|
|
871
|
-
|
|
872
|
-
### GeoSPARQL — six topological functions
|
|
873
|
-
|
|
874
|
-
The `geof:` functions below are built into the SPARQL engine and need no
|
|
875
|
-
import. They work through `query()` / `fn.query()` AND against a
|
|
876
|
-
persisted store through `factoidal query`, because both paths evaluate
|
|
877
|
-
in the same environment.
|
|
878
|
-
|
|
879
|
-
geof:sfEquals geof:sfDisjoint geof:sfIntersects
|
|
880
|
-
geof:sfTouches geof:sfWithin geof:sfContains
|
|
881
|
-
|
|
882
|
-
```sparql
|
|
883
|
-
PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
|
|
884
|
-
PREFIX geo: <http://www.opengis.net/ont/geosparql#>
|
|
885
|
-
SELECT ?a WHERE {
|
|
886
|
-
?a :footprint ?w
|
|
887
|
-
FILTER(geof:sfWithin(?w, "POLYGON((0 0,0 2,2 2,2 0,0 0))"^^geo:wktLiteral))
|
|
888
|
-
}
|
|
889
|
-
```
|
|
890
|
-
|
|
891
|
-
**What is NOT there**, stated so nobody plans around it: no
|
|
892
|
-
`geof:distance`, `geof:buffer`, `geof:envelope`, `geof:boundary`,
|
|
893
|
-
`geof:convexHull` or any other non-topological measure; no
|
|
894
|
-
`geof:relate` with a DE-9IM matrix; no coordinate reference system
|
|
895
|
-
handling beyond what the WKT literal carries; no GML literals. Geometry
|
|
896
|
-
comes from a WKT parser, so a shapefile, GeoJSON or GML source must be
|
|
897
|
-
converted to `geo:wktLiteral` before it is loaded.
|
|
898
|
-
|
|
899
|
-
### Full text: SPARQL's own functions, over a character-gram index
|
|
900
|
-
|
|
901
|
-
`CONTAINS`, `STRSTARTS`, `STRENDS` and `REGEX` (SPARQL 1.1 §17.4.3) are
|
|
902
|
-
the way to search text. There is no `text:query`-style extension and you
|
|
903
|
-
do not ask for the index: where the query shape allows it, the planner
|
|
904
|
-
uses the **LGI1** literal index the packer writes beside each block.
|
|
905
|
-
|
|
906
|
-
LGI1 holds character 3-grams of the case-folded lexical form, and it is a
|
|
907
|
-
CANDIDATE FILTER rather than a decider — it answers a superset and the
|
|
908
|
-
engine re-evaluates your original `FILTER` on those rows, so the answer
|
|
909
|
-
is exactly the answer a scan gives. Tokens would not do: `CONTAINS` is a
|
|
910
|
-
substring test, and "underwater" contains "water" without being the
|
|
911
|
-
token "water".
|
|
912
|
-
|
|
913
|
-
Measured 2026-09-05, full skosdex corpus (7,315,251 quads, 3,286 blocks,
|
|
914
|
-
204 named graphs, 1.0 GB), through a store handle on plain `node`:
|
|
915
|
-
|
|
916
|
-
| search | rows | time |
|
|
917
|
-
|---|---|---|
|
|
918
|
-
| `water` | 5 | 645 ms |
|
|
919
|
-
| `glacier` | 0 | 670 ms |
|
|
920
|
-
| `bicycle` | 2 | 617 ms |
|
|
921
|
-
|
|
922
|
-
A miss costs what a hit costs, because both are index lookups rather than
|
|
923
|
-
scans. On a 45,806-row block the same search was about 180 ms of scan
|
|
924
|
-
before the index and 1.8 ms after it.
|
|
925
|
-
|
|
926
|
-
**It falls back to a scan**, silently and correctly, for: a needle under
|
|
927
|
-
3 characters, a variable needle, `REGEX`, `UCASE`, `!CONTAINS`,
|
|
928
|
-
`CONTAINS` under `||`, a filter on a variable not bound in object
|
|
929
|
-
position, and any block whose generation has no `.lgi1` sidecar. The
|
|
930
|
-
index costs about 55% of the block bytes.
|
|
931
|
-
|
|
932
|
-
### Geometry: the GBI1 bounding-box index
|
|
933
|
-
|
|
934
|
-
The same construction for `geof:`. Each `geo:wktLiteral` object's
|
|
935
|
-
bounding box is indexed, and five of the six topological functions are
|
|
936
|
-
filtered by it — `sfIntersects`, `sfWithin`, `sfContains`, `sfTouches`,
|
|
937
|
-
`sfEquals`. Measured 74x to 94x against a scan, with a miss at 82x.
|
|
938
|
-
|
|
939
|
-
**`sfDisjoint` is refused and falls back to a scan**, deliberately: it
|
|
940
|
-
accepts exactly the rows a box can exclude, so a box test inverts and
|
|
941
|
-
would drop answers. A non-overlapping pair of boxes proves geometries
|
|
942
|
-
disjoint, but that saves work inside a scan rather than reducing the
|
|
943
|
-
candidate set.
|
|
944
|
-
|
|
945
|
-
### Wire version 10: large literals, RDF 1.2 terms, zone maps
|
|
946
|
-
|
|
947
|
-
A generation packed at wire version 10 (`--layout ibk5`) carries three
|
|
948
|
-
things earlier versions do not. Everything below is read by the command
|
|
949
|
-
and by `bin/store.mjs` without a flag; wire versions 9 and earlier are
|
|
950
|
-
read exactly as before.
|
|
951
|
-
|
|
952
|
-
**Out-of-line literals.** A literal whose lexical form is at most 65,536
|
|
953
|
-
UTF-8 bytes is stored inside the block. A longer one is stored as one
|
|
954
|
-
file beside it, `blob-<sha256 hex>.lit`, holding exactly those bytes; the
|
|
955
|
-
block keeps only the byte length and the SHA-256, and the manifest blob
|
|
956
|
-
table commits the file. The name is the content address, so the same
|
|
957
|
-
literal in twenty blocks is one file. `factoidal inspect` prints how many
|
|
958
|
-
such files a generation holds and their total size, and the plan a query
|
|
959
|
-
produces lists them under `blobKeys` beside `keys` and `sidecarKeys` — a
|
|
960
|
-
host reading a store itself must fetch all three. A blob that is missing
|
|
961
|
-
or whose bytes hash differently REFUSES the query; it is never answered
|
|
962
|
-
with a shortened literal.
|
|
963
|
-
|
|
964
|
-
Above 4,294,967,295 bytes the packer refuses the literal and names its
|
|
965
|
-
subject and predicate. That ceiling, and every other one, is stated in
|
|
966
|
-
`docs/designissues/2026-09-05-wire-version-10-scale.md` section 2.
|
|
967
|
-
|
|
968
|
-
**RDF 1.2 terms.** A triple term (`<<( :a :p :b )>>`) and a directional
|
|
969
|
-
language literal (`"..."@ar--rtl`) are stored and read back. Wire version
|
|
970
|
-
9 refuses both.
|
|
971
|
-
|
|
972
|
-
**Zone maps.** Each manifest entry carries the smallest and the largest
|
|
973
|
-
subject key and object key of its block, truncated to 64 bytes. A query
|
|
974
|
-
with a constant subject or object skips every block whose range cannot
|
|
975
|
-
hold it, from the manifest alone, with no block read. The plan reports
|
|
976
|
-
how many entries were dropped that way as `zoneExcluded`. Selectivity
|
|
977
|
-
depends on the source order: a subject-grouped or graph-grouped file
|
|
978
|
-
gives disjoint ranges per block, a shuffled one gives overlapping ranges
|
|
979
|
-
and a scan, which is correct and no worse than wire version 9.
|
|
980
|
-
|
|
981
|
-
## Limits (deliberate, documented)
|
|
982
|
-
|
|
983
|
-
- **In-memory only.** ~1.2 KB RAM per quad (measured); 1M quads ≈
|
|
984
|
-
1.2 GB. No streaming parse yet — inputs are whole strings.
|
|
985
|
-
- Lenient Turtle parsing: `parse()` cannot yet reject syntax errors
|
|
986
|
-
(bad input can yield an empty dataset).
|
|
987
|
-
- No *write* persistence in the npm build (SPARQL Update stays
|
|
988
|
-
in-memory; durable UPDATE against a COTTAS store on disk is
|
|
989
|
-
native-only today). *Reading* a COTTAS artifact's bytes is available
|
|
990
|
-
in-process via `openCottas`/`queryCottas`/`closeCottas`/`toCottas`
|
|
991
|
-
(both engines, both npm and browser entries) — see "The db API"
|
|
992
|
-
above.
|
|
993
|
-
- Bundle sizes (measured 2026-07-04): JS engine 554 KB, npm entry
|
|
994
|
-
461 KB, Wasm 43 KB loader + 1.3 MB assets. The Wasm entry trades
|
|
995
|
-
startup cost for throughput.
|
|
996
|
-
|
|
997
|
-
## Testing
|
|
998
|
-
|
|
999
|
-
`npm test` — unit suite covering the parse/query/serialize surface,
|
|
1000
|
-
the RDF/JS contract, canonicalization stability under blank-node
|
|
1001
|
-
renaming, and JS↔Wasm byte-parity. The engine underneath is gated on
|
|
1002
|
-
every landing by the full W3C suites, a cross-backend parity harness,
|
|
1003
|
-
and comparison probes against Apache Jena ARQ — live scores with
|
|
1004
|
-
dates and commit links at
|
|
1005
|
-
[danbri.github.io/factoidal/test-results](https://danbri.github.io/factoidal/test-results/).
|
|
127
|
+
- Fuseki: read-only administration API, one dataset per process, no
|
|
128
|
+
configuration file, no authentication, no JSON-LD from Graph Store GET
|
|
129
|
+
(https://github.com/danbri/factoidal/issues/829 to 834).
|
|
130
|
+
- Large result sets can exceed Node's default stack; use
|
|
131
|
+
`node --stack-size=4000` or Deno.
|
|
132
|
+
- The API can change before 1.0.
|
|
1006
133
|
|
|
1007
|
-
##
|
|
134
|
+
## Licence
|
|
1008
135
|
|
|
1009
|
-
|
|
1010
|
-
(Apache-2.0). This package's JavaScript is a thin consumer layer (API
|
|
1011
|
-
shaping + RDF/JS conversion) containing no RDF/SPARQL semantics of
|
|
1012
|
-
its own — the semantics are F\*-extracted.
|
|
136
|
+
Apache-2.0. See `LICENSE` and `NOTICE`.
|