@factoidal/core 0.7.0 → 0.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (193) hide show
  1. package/CHANGELOG.md +510 -1
  2. package/NOTICE +2 -21
  3. package/README.md +117 -993
  4. package/api.d.ts +53 -0
  5. package/api.js +5 -0
  6. package/api.mjs +56 -0
  7. package/bin/engine.mjs +38 -19
  8. package/bin/factoidal.mjs +513 -67
  9. package/bin/pack.mjs +37 -0
  10. package/bin/serve-worker.mjs +32 -0
  11. package/bin/serve.mjs +423 -0
  12. package/bin/store-worker-host.mjs +1 -1
  13. package/bin/store.mjs +1 -1
  14. package/browser.d.ts +110 -507
  15. package/browser.js +151 -2257
  16. package/docs/fuseki-replacement.md +108 -0
  17. package/fn.d.ts +24 -95
  18. package/fn.js +104 -150
  19. package/index.d.ts +598 -259
  20. package/index.js +4 -234
  21. package/index.mjs +176 -84
  22. package/l4-assets/l4factoidal.js +47 -7
  23. package/l4-assets/l4factoidal.mjs +1 -1
  24. package/l4-assets/l4factoidal.wasm +0 -0
  25. package/l4-assets/version.json +11 -7
  26. package/lib/api.mjs +2300 -0
  27. package/lib/browser-engine.mjs +69 -0
  28. package/lib/dataset.mjs +377 -0
  29. package/lib/delta-log-hosts.mjs +191 -0
  30. package/lib/lean-entry.mjs +204 -0
  31. package/lib/pure.mjs +98 -0
  32. package/lib/store.mjs +115 -0
  33. package/lib/version.mjs +5 -0
  34. package/lws/README.md +97 -0
  35. package/lws/server.mjs +331 -0
  36. package/package.json +59 -61
  37. package/rdfjs.js +5 -536
  38. package/rdfjs.mjs +594 -0
  39. package/sample-store.d.ts +8 -16
  40. package/sample-store.mjs +39 -29
  41. package/solid/client/README.md +100 -0
  42. package/solid/client/index.mjs +277 -0
  43. package/solid/server/README.md +96 -0
  44. package/solid/server/index.mjs +317 -0
  45. package/store-host/deno.mjs +8 -0
  46. package/store-host/index.mjs +12 -0
  47. package/store-host/node.mjs +8 -0
  48. package/xmpp/README.md +107 -0
  49. package/xmpp/index.mjs +229 -0
  50. package/browser-wasm.js +0 -872
  51. package/factoidal-npm-entry.js +0 -15477
  52. package/factoidal-npm-entry.wasm.assets/code-5a7fc68f2ab1323718b8.wasm +0 -0
  53. package/factoidal-npm-entry.wasm.js +0 -455
  54. package/factoidal.js +0 -13118
  55. package/factoidal.wasm.assets/code-bbe6099bfb5b10c4c3ab.wasm +0 -0
  56. package/factoidal.wasm.js +0 -457
  57. package/hacl-init.js +0 -92
  58. package/hacl-wasm/FStar.wasm +0 -0
  59. package/hacl-wasm/Hacl_Bignum.wasm +0 -0
  60. package/hacl-wasm/Hacl_Bignum25519_51.wasm +0 -0
  61. package/hacl-wasm/Hacl_Bignum_Base.wasm +0 -0
  62. package/hacl-wasm/Hacl_Curve25519_51.wasm +0 -0
  63. package/hacl-wasm/Hacl_Ed25519.wasm +0 -0
  64. package/hacl-wasm/Hacl_Ed25519_PrecompTable.wasm +0 -0
  65. package/hacl-wasm/Hacl_Hash_Base.wasm +0 -0
  66. package/hacl-wasm/Hacl_Hash_SHA2.wasm +0 -0
  67. package/hacl-wasm/Hacl_IntTypes_Intrinsics.wasm +0 -0
  68. package/hacl-wasm/LowStar_Endianness.wasm +0 -0
  69. package/hacl-wasm/WasmSupport.wasm +0 -0
  70. package/hacl-wasm/api.js +0 -775
  71. package/hacl-wasm/api.json +0 -3787
  72. package/hacl-wasm/layouts.json +0 -1
  73. package/hacl-wasm/loader.js +0 -568
  74. package/hacl-wasm/shell.js +0 -12
  75. package/l4-core.d.ts +0 -159
  76. package/l4-core.js +0 -188
  77. package/l4.d.ts +0 -50
  78. package/l4.js +0 -104
  79. package/lib/api.js +0 -2300
  80. package/lib/engine-js.js +0 -165
  81. package/lib/engine-wasm.js +0 -300
  82. package/sample-store/CURRENT +0 -1
  83. package/sample-store/gen-1/manifest.sbm2 +0 -0
  84. package/sample-store/gen-1/manifest.tsv +0 -14
  85. package/sample-store/gen-1/predicate-0.ibk3 +0 -0
  86. package/sample-store/gen-1/predicate-0.ibk3.merkle +0 -1
  87. package/sample-store/gen-1/predicate-0.ibk3.oli2 +0 -0
  88. package/sample-store/gen-1/predicate-0.ibk3.oli2.merkle +0 -1
  89. package/sample-store/gen-1/predicate-0.ibk3.sri2 +0 -0
  90. package/sample-store/gen-1/predicate-0.ibk3.sri2.merkle +0 -1
  91. package/sample-store/gen-1/predicate-0.ibk3.tli1 +0 -0
  92. package/sample-store/gen-1/predicate-0.ibk3.tli1.merkle +0 -0
  93. package/sample-store/gen-1/predicate-1.ibk3 +0 -0
  94. package/sample-store/gen-1/predicate-1.ibk3.merkle +0 -0
  95. package/sample-store/gen-1/predicate-1.ibk3.oli2 +0 -0
  96. package/sample-store/gen-1/predicate-1.ibk3.oli2.merkle +0 -0
  97. package/sample-store/gen-1/predicate-1.ibk3.sri2 +0 -0
  98. package/sample-store/gen-1/predicate-1.ibk3.sri2.merkle +0 -1
  99. package/sample-store/gen-1/predicate-1.ibk3.tli1 +0 -0
  100. package/sample-store/gen-1/predicate-1.ibk3.tli1.merkle +0 -1
  101. package/sample-store/gen-1/predicate-10.ibk3 +0 -0
  102. package/sample-store/gen-1/predicate-10.ibk3.merkle +0 -1
  103. package/sample-store/gen-1/predicate-10.ibk3.oli2 +0 -0
  104. package/sample-store/gen-1/predicate-10.ibk3.oli2.merkle +0 -0
  105. package/sample-store/gen-1/predicate-10.ibk3.sri2 +0 -0
  106. package/sample-store/gen-1/predicate-10.ibk3.sri2.merkle +0 -1
  107. package/sample-store/gen-1/predicate-10.ibk3.tli1 +0 -0
  108. package/sample-store/gen-1/predicate-10.ibk3.tli1.merkle +0 -1
  109. package/sample-store/gen-1/predicate-11.ibk3 +0 -0
  110. package/sample-store/gen-1/predicate-11.ibk3.merkle +0 -0
  111. package/sample-store/gen-1/predicate-11.ibk3.oli2 +0 -0
  112. package/sample-store/gen-1/predicate-11.ibk3.oli2.merkle +0 -1
  113. package/sample-store/gen-1/predicate-11.ibk3.sri2 +0 -0
  114. package/sample-store/gen-1/predicate-11.ibk3.sri2.merkle +0 -1
  115. package/sample-store/gen-1/predicate-11.ibk3.tli1 +0 -0
  116. package/sample-store/gen-1/predicate-11.ibk3.tli1.merkle +0 -1
  117. package/sample-store/gen-1/predicate-12.ibk3 +0 -0
  118. package/sample-store/gen-1/predicate-12.ibk3.merkle +0 -1
  119. package/sample-store/gen-1/predicate-12.ibk3.oli2 +0 -0
  120. package/sample-store/gen-1/predicate-12.ibk3.oli2.merkle +0 -1
  121. package/sample-store/gen-1/predicate-12.ibk3.sri2 +0 -0
  122. package/sample-store/gen-1/predicate-12.ibk3.sri2.merkle +0 -1
  123. package/sample-store/gen-1/predicate-12.ibk3.tli1 +0 -0
  124. package/sample-store/gen-1/predicate-12.ibk3.tli1.merkle +0 -1
  125. package/sample-store/gen-1/predicate-2.ibk3 +0 -0
  126. package/sample-store/gen-1/predicate-2.ibk3.merkle +0 -1
  127. package/sample-store/gen-1/predicate-2.ibk3.oli2 +0 -0
  128. package/sample-store/gen-1/predicate-2.ibk3.oli2.merkle +0 -1
  129. package/sample-store/gen-1/predicate-2.ibk3.sri2 +0 -0
  130. package/sample-store/gen-1/predicate-2.ibk3.sri2.merkle +0 -0
  131. package/sample-store/gen-1/predicate-2.ibk3.tli1 +0 -0
  132. package/sample-store/gen-1/predicate-2.ibk3.tli1.merkle +0 -0
  133. package/sample-store/gen-1/predicate-3.ibk3 +0 -0
  134. package/sample-store/gen-1/predicate-3.ibk3.merkle +0 -1
  135. package/sample-store/gen-1/predicate-3.ibk3.oli2 +0 -0
  136. package/sample-store/gen-1/predicate-3.ibk3.oli2.merkle +0 -1
  137. package/sample-store/gen-1/predicate-3.ibk3.sri2 +0 -0
  138. package/sample-store/gen-1/predicate-3.ibk3.sri2.merkle +0 -1
  139. package/sample-store/gen-1/predicate-3.ibk3.tli1 +0 -0
  140. package/sample-store/gen-1/predicate-3.ibk3.tli1.merkle +0 -1
  141. package/sample-store/gen-1/predicate-4.ibk3 +0 -0
  142. package/sample-store/gen-1/predicate-4.ibk3.merkle +0 -0
  143. package/sample-store/gen-1/predicate-4.ibk3.oli2 +0 -0
  144. package/sample-store/gen-1/predicate-4.ibk3.oli2.merkle +0 -1
  145. package/sample-store/gen-1/predicate-4.ibk3.sri2 +0 -0
  146. package/sample-store/gen-1/predicate-4.ibk3.sri2.merkle +0 -1
  147. package/sample-store/gen-1/predicate-4.ibk3.tli1 +0 -0
  148. package/sample-store/gen-1/predicate-4.ibk3.tli1.merkle +0 -2
  149. package/sample-store/gen-1/predicate-5.ibk3 +0 -0
  150. package/sample-store/gen-1/predicate-5.ibk3.merkle +0 -1
  151. package/sample-store/gen-1/predicate-5.ibk3.oli2 +0 -0
  152. package/sample-store/gen-1/predicate-5.ibk3.oli2.merkle +0 -1
  153. package/sample-store/gen-1/predicate-5.ibk3.sri2 +0 -0
  154. package/sample-store/gen-1/predicate-5.ibk3.sri2.merkle +0 -1
  155. package/sample-store/gen-1/predicate-5.ibk3.tli1 +0 -0
  156. package/sample-store/gen-1/predicate-5.ibk3.tli1.merkle +0 -1
  157. package/sample-store/gen-1/predicate-6.ibk3 +0 -0
  158. package/sample-store/gen-1/predicate-6.ibk3.merkle +0 -1
  159. package/sample-store/gen-1/predicate-6.ibk3.oli2 +0 -0
  160. package/sample-store/gen-1/predicate-6.ibk3.oli2.merkle +0 -1
  161. package/sample-store/gen-1/predicate-6.ibk3.sri2 +0 -0
  162. package/sample-store/gen-1/predicate-6.ibk3.sri2.merkle +0 -1
  163. package/sample-store/gen-1/predicate-6.ibk3.tli1 +0 -0
  164. package/sample-store/gen-1/predicate-6.ibk3.tli1.merkle +0 -0
  165. package/sample-store/gen-1/predicate-7.ibk3 +0 -0
  166. package/sample-store/gen-1/predicate-7.ibk3.merkle +0 -0
  167. package/sample-store/gen-1/predicate-7.ibk3.oli2 +0 -0
  168. package/sample-store/gen-1/predicate-7.ibk3.oli2.merkle +0 -2
  169. package/sample-store/gen-1/predicate-7.ibk3.sri2 +0 -0
  170. package/sample-store/gen-1/predicate-7.ibk3.sri2.merkle +0 -1
  171. package/sample-store/gen-1/predicate-7.ibk3.tli1 +0 -0
  172. package/sample-store/gen-1/predicate-7.ibk3.tli1.merkle +0 -1
  173. package/sample-store/gen-1/predicate-8.ibk3 +0 -0
  174. package/sample-store/gen-1/predicate-8.ibk3.merkle +0 -1
  175. package/sample-store/gen-1/predicate-8.ibk3.oli2 +0 -0
  176. package/sample-store/gen-1/predicate-8.ibk3.oli2.merkle +0 -1
  177. package/sample-store/gen-1/predicate-8.ibk3.sri2 +0 -0
  178. package/sample-store/gen-1/predicate-8.ibk3.sri2.merkle +0 -1
  179. package/sample-store/gen-1/predicate-8.ibk3.tli1 +0 -0
  180. package/sample-store/gen-1/predicate-8.ibk3.tli1.merkle +0 -1
  181. package/sample-store/gen-1/predicate-9.ibk3 +0 -0
  182. package/sample-store/gen-1/predicate-9.ibk3.merkle +0 -1
  183. package/sample-store/gen-1/predicate-9.ibk3.oli2 +0 -0
  184. package/sample-store/gen-1/predicate-9.ibk3.oli2.merkle +0 -1
  185. package/sample-store/gen-1/predicate-9.ibk3.sri2 +0 -0
  186. package/sample-store/gen-1/predicate-9.ibk3.sri2.merkle +0 -3
  187. package/sample-store/gen-1/predicate-9.ibk3.tli1 +0 -0
  188. package/sample-store/gen-1/predicate-9.ibk3.tli1.merkle +0 -2
  189. package/select.d.ts +0 -116
  190. package/select.js +0 -492
  191. package/version.json +0 -102
  192. package/wasm.d.ts +0 -130
  193. package/wasm.js +0 -158
package/README.md CHANGED
@@ -1,1012 +1,136 @@
1
1
  # @factoidal/core
2
2
 
3
- > Release 0.3.0. The API surface is early and may
4
- > change before 1.0. This package was previously developed in-tree
5
- > under the placeholder names `factoidal` and `@danbri/foafos`; it was
6
- > never published under those names. See [CHANGELOG.md](CHANGELOG.md).
7
-
8
- A formally verified RDF/SPARQL engine for JavaScript and WebAssembly.
9
- The semantics live in [F\*](https://www.fstar-lang.org/) and are
10
- compiled via OCaml and js_of_ocaml / wasm_of_ocaml into the engine
11
- bundles shipped here. There is no hand-written SPARQL evaluator — the
12
- JavaScript you run was extracted from the same `.fst` specifications
13
- that verify under Z3 and pass the W3C suites.
14
-
15
- Verification status qualifier: parser and algebra spec verified in
16
- F\*; the on-disk backend has unverified OCaml-side optimization layers
17
- being migrated back to F\* (the npm build does not include the
18
- on-disk backend).
19
-
20
- ## Why this instead of N3.js / rdflib.js / Comunica?
21
-
22
- - **Full SPARQL 1.1 client-side** — query and update evaluated in
23
- your process by conformance-tested code (631 of 631 W3C SPARQL
24
- tests, 1031 of 1031 RDF parsing tests on the native build of the
25
- same source); no server, no endpoint.
26
- - **RDFC-1.0 canonicalization built in**: `canonicalize()` yields
27
- standard canonical N-Quads (stable under blank-node renaming — for
28
- content addressing, dataset diffing, cache keys) without a separate
29
- library.
30
- - **RDFS / OWL-RL entailment** as a query option (`entail: "RDFS"`).
31
- - **RDF/JS data model** (`DataFactory`, terms with `.equals`,
32
- `DatasetCore`) so it composes with the existing ecosystem.
33
- - Provenance: every release corresponds to a gates-green commit of
34
- [danbri/factoidal](https://github.com/danbri/factoidal).
35
-
36
- ## Install
3
+ RDF, SPARQL, reasoning, validation and Linked Data servers in one package.
4
+ Runs in Node 22.12+, Deno 2 and browsers. No dependencies.
37
5
 
38
6
  ```bash
39
7
  npm install @factoidal/core
40
8
  ```
41
9
 
42
- ## Quickstart
43
-
44
- The API is **async throughout** — every engine call returns a Promise,
45
- so `await` each one (the engine bundle loads lazily on first use).
46
-
47
- ```js
48
- import { parse, query, serialize, canonicalize, dataFactory }
49
- from "@factoidal/core";
50
-
51
- // Parse any supported syntax into an RDF/JS DatasetCore.
52
- // If your data contains RELATIVE IRIs, pass { baseIRI: ... } —
53
- // see the note below this example.
54
- const ds = await parse(`
55
- @prefix foaf: <http://xmlns.com/foaf/0.1/> .
56
- _:a foaf:name "Alice" ; foaf:knows _:b .
57
- _:b foaf:name "Bob" .
58
- `, { format: "turtle" });
59
-
60
- // SELECT — bindings are Map<variableName, RDF/JS Term>.
61
- const rows = await query(ds, `
62
- PREFIX foaf: <http://xmlns.com/foaf/0.1/>
63
- SELECT ?name WHERE { ?p foaf:name ?name } ORDER BY ?name
64
- `);
65
- for (const b of rows) console.log(b.get("name").value); // Alice, Bob
66
-
67
- // ASK — plain boolean.
68
- const yes = await query(ds, "ASK { ?s ?p ?o }"); // true
69
-
70
- // Canonical N-Quads (RDFC-1.0): identical output for isomorphic
71
- // inputs regardless of blank-node labels.
72
- const c14n = await canonicalize(ds);
73
-
74
- // Round-trip.
75
- const nq = await serialize(ds, { format: "nquads" });
76
- ```
77
-
78
- > ⚠️ **Pass `baseIRI` when your input uses relative IRIs.** In the
79
- > current build, statements whose relative IRIs cannot be resolved are
80
- > **dropped without an error**, so a document can parse to fewer
81
- > triples than it contains (`{ baseIRI: "https://example.org/doc" }`
82
- > fixes it). A count check after parsing is a cheap guard. Surfacing
83
- > these drops as a throw or a warnings channel is tracked in the
84
- > repository issues.
85
-
86
- CommonJS: `const factoidal = require("@factoidal/core")`.
87
-
88
- ### WebAssembly entry
89
-
90
- ```js
91
- import { parse, query } from "factoidal/wasm";
92
- // Same API over the wasm_of_ocaml bundle; needs a Wasm-GC engine
93
- // (Chrome >= 119, Node >= 22). The unit suite asserts byte-parity
94
- // between the JS and Wasm entries on parse/SELECT/ASK.
95
- ```
96
-
97
- ### Load in the browser without npm
98
-
99
- You don't need `npm install` (or a bundler) to run this in a browser.
100
- Two options, in preference order:
101
-
102
- 1. **This site's own mirror (recommended, same-origin, no build step
103
- for you).** Every push regenerates `docs/npm/factoidal/` from this
104
- package (`formal/fstar/build-ocaml.sh npm`'s Pages-mirror step), so
105
- it's always in sync with what's published here:
106
-
107
- ```html
108
- <script type="module">
109
- import { query } from 'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
110
- const r = await query(dataTtl, 'SELECT * WHERE { ?s ?p ?o }');
111
- console.log(r.results.bindings);
112
- </script>
113
- ```
114
-
115
- See [`demo-jsonld-playground.html`](../../docs/fstar-extracted/demo-jsonld-playground.html)
116
- for a working page built exactly this way (loads `browser.js` from
117
- `../npm/factoidal/browser.js`, a relative same-origin path).
118
-
119
- 2. **jsDelivr's GitHub proxy** (serves the correct
120
- `text/javascript` MIME type; no Pages dependency, works from any
121
- branch/tag):
122
-
123
- ```html
124
- <script type="module">
125
- import { query } from 'https://cdn.jsdelivr.net/gh/danbri/factoidal@claude/main/npm/factoidal/browser.js';
126
- </script>
127
- ```
128
-
129
- **Do not** link `browser.js` (or `factoidal.js`) via
130
- `raw.githubusercontent.com`. Raw GitHub serves every file as
131
- `text/plain`, and browsers refuse to execute a `text/plain` response
132
- as an ES module (`<script type="module">` fails with a MIME-type
133
- error) — this bites people who copy a "raw" link expecting it to just
134
- work. Use the Pages mirror or jsDelivr above instead.
135
-
136
- `browser.js` also exports `toRdf(text, options)` (parse to sorted
137
- N-Quads — `--dump-nq` under the hood) and `canonicalize(text,
138
- options)` (RDFC-1.0 — `--canonicalize`) alongside `query()`; both
139
- default `options.format` to `'jsonld'` since that's the playground's
140
- use case, but accept the same formats as `query()`'s `dataFormat`.
141
-
142
- For multi-file, multi-named-graph datasets (several documents, each
143
- loaded into its own named graph or the default graph) and a choice of
144
- extraction target, use `queryDataset(files, queryString, options)`:
145
-
146
- ```js
147
- import { queryDataset } from 'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
148
-
149
- const r = await queryDataset(
150
- [
151
- { content: defaultGraphTtl },
152
- { content: peopleTtl, graph: 'urn:x:people' },
153
- ],
154
- 'SELECT * WHERE { GRAPH ?g { ?s ?p ?o } }',
155
- { entail: 'RDFS', engine: 'js' } // or engine: 'wasm'
156
- );
157
- ```
158
-
159
- On `engine: 'wasm'` it merges the named graphs into one TriG document
160
- (wasm_of_ocaml's `query()` only takes a single data string) via
161
- `browser-wasm.js`, loaded on demand. Both `query()` and `queryDataset()`
162
- attach a non-enumerable `engineMs` (bundle-eval wall-clock time) to the
163
- returned results object — invisible to `JSON.stringify()`/`Object.keys()`
164
- so it never perturbs a diff against a W3C `.srx`-derived fixture, but
165
- readable as `result.engineMs` for timing/observability UIs. This is
166
- what `docs/fstar-extracted/factoidal-sparql-client.js`'s web component
167
- is built on, rather than duplicating the engine-invocation logic itself.
168
-
169
- ### Durable browser persistence (delta log)
170
-
171
- `browser.js` also exports a small IndexedDB-backed durable-UPDATE log —
172
- see [`docs/designissues/2026-07-06-browser-persistence.md`](../../docs/designissues/2026-07-06-browser-persistence.md)
173
- for the full design (why IndexedDB and not OPFS for v1, the tab-close/
174
- crash guarantee mapping, and the quota/eviction honesty section). Every
175
- byte moved through these functions is produced/consumed by the same
176
- F\*-verified `RDF_Store_Columnar_DeltaLog`/`RDF_Store_Columnar_DeltaMerge`
177
- modules the native on-disk delta log uses (`factoidal serve --rw
178
- --delta-log`); this is a browser-native persistence path, not a mock:
179
-
180
- ```js
181
- import { deltaLogOpen, deltaLogAppend, deltaLogMerge } from
182
- 'https://danbri.github.io/factoidal/npm/factoidal/browser.js';
183
-
184
- const handle = await deltaLogOpen(); // opens/creates an IndexedDB database
185
- await deltaLogAppend(handle, 'INSERT DATA { <urn:x:a> <urn:x:p> "1" }');
186
- await deltaLogAppend(handle, 'INSERT DATA { <urn:x:b> <urn:x:p> "2" }');
187
-
188
- // ... reload the page, or close and reopen the browser ...
189
-
190
- const merged = await deltaLogMerge(handle, ''); // '' = empty base dataset
191
- console.log(merged); // the two INSERT DATA ops, replayed from IndexedDB
192
- ```
193
-
194
- Supported update ops: `INSERT DATA`, `DELETE DATA`, `CLEAR`, `DROP`,
195
- `CREATE` — the same subset the native `--rw` commit path accepts;
196
- anything else (`DELETE/INSERT WHERE`, `COPY`, `MOVE`, `ADD`) rejects
197
- with `ok:false` rather than silently no-op'ing. `deltaLogReadAllHex`,
198
- `deltaLogDestroy`, and the test-only `_deltaLogCorruptLastForTest` round
199
- out the surface (see `browser.js`'s own JSDoc for each). This is a
200
- prototype: no compaction and no `navigator.storage.persist()` wiring
201
- yet (both named as staged next steps in the design doc), and it does
202
- not yet back a hub demo page — that is separate, follow-on work.
203
-
204
- ### RDF/JS interop
205
-
206
10
  ```js
207
- import { dataFactory } from "@factoidal/core";
208
- const { namedNode, literal, quad, blankNode } = dataFactory;
209
- const q = quad(blankNode("x"),
210
- namedNode("http://xmlns.com/foaf/0.1/name"),
211
- literal("Alice", "en"));
212
- // Terms implement termType/value/language/datatype/.equals per the
213
- // RDF/JS data-model spec; Dataset implements DatasetCore
214
- // (add/delete/has/match/size/iteration).
215
- ```
216
-
217
- ## Custom extension functions (SPARQL 1.1 §17.6)
218
-
219
- Register your own functions by IRI (the
220
- [Comunica model](https://comunica.dev/docs/query/advanced/extension_functions/));
221
- sync or async both work. The dispatch semantics are F\*-specified:
222
- built-in function families always win, and a call to an IRI with no
223
- registered function is the spec-required error (unbound in
224
- SELECT/BIND position, row dropped in FILTER position). Issue:
225
- [#463](https://github.com/danbri/factoidal/issues/463).
226
-
227
- ```js
228
- import { query, registerExtensionFunction } from "@factoidal/core";
229
-
230
- await registerExtensionFunction(
231
- "http://example.org/fn#isAdult",
232
- async ([age]) => Number(age.value) >= 18 // args are SRJ-style terms
233
- );
234
-
235
- const rows = await query(ds, `
236
- PREFIX fn: <http://example.org/fn#>
237
- SELECT ?s WHERE { ?s <http://example.org/age> ?a
238
- FILTER(fn:isAdult(?a)) }`);
239
- ```
240
-
241
- Return a JS primitive (`boolean`/`number`/`string`), an SRJ-style term
242
- object (`{type:'uri'|'literal'|'bnode', value, datatype?, 'xml:lang'?}`),
243
- or a Promise of either; `null`/`undefined`/a thrown error is the §17.6
244
- error. Async functions run over the synchronous verified engine
245
- through a bounded, memoised re-evaluation loop — within one query
246
- every call with the same arguments sees one stable answer.
247
-
248
- ### The same functions against the Lean engine and a persisted store
249
-
250
- The registration above serves the F\* engine's in-memory `query()`. The
251
- Lean engine has its own registry, and it reaches every Lean query path,
252
- including a store handle — so a registered function can filter rows read
253
- off disk:
254
-
255
- ```js
256
- import { loadEngine } from '@factoidal/core/bin/engine.mjs'
257
- import { openStore, openStoreHandle } from '@factoidal/core/bin/store.mjs'
258
- import { registerExtensionFunction, withExtensionFunctions }
259
- from '@factoidal/core/bin/ext.mjs'
260
-
261
- const engine = await loadEngine()
262
- const handle = openStoreHandle(engine, openStore('/path/to/store'))
263
-
264
- registerExtensionFunction(engine, 'http://example.org/fn/endsWithZed',
265
- ([label]) => label.value.endsWith('z'))
266
-
267
- const answer = handle.query(`
268
- PREFIX ex: <http://example.org/fn/>
269
- PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
270
- SELECT ?c ?l WHERE { ?c skos:prefLabel ?l FILTER(ex:endsWithZed(?l)) }`)
271
- ```
272
-
273
- The argument and result encoding, the §17.6 error rules and the
274
- async re-evaluation loop are the same as above, so one function serves
275
- both engines. Two differences to plan around:
276
-
277
- - **Registration is per engine instance**, not per handle or per query.
278
- A server that answers for more than one caller uses
279
- `withExtensionFunctions(engine, map, body)`, which registers, runs and
280
- clears in a `finally`, or loads one engine per caller.
281
- - **Every call crosses into JavaScript**, so a function in a FILTER runs
282
- once per row. Measure before putting one on a large scan.
283
-
284
- Design, with the determinism and scope rules in full:
285
- [`docs/designissues/2026-09-04-lean-extension-functions.md`](../../docs/designissues/2026-09-04-lean-extension-functions.md).
286
-
287
- ## Functional API (fn)
288
-
289
- `@factoidal/core/fn` is a strictly functional variant of the API above:
290
- every extracted engine operation is already a value-to-value
291
- function — `FnDataset` makes the JS surface match that instead of
292
- papering over it with RDF/JS's mutable `add`/`delete`. Frozen
293
- snapshots, free functions instead of methods (so they compose), and
294
- RDFC-1.0 canonical hashes as a first-class, memoized identity — the
295
- piece that makes dataflow-style recompute-skipping possible. Full
296
- design rationale (cost model, backend/streaming extension points):
297
- [`docs/designissues/2026-07-05-functional-dataset-api.md`](../../docs/designissues/2026-07-05-functional-dataset-api.md).
298
-
299
- ```js
300
- const { parse, filter, hash, cell, derive } = require('@factoidal/core/fn');
301
-
302
- const ds = await parse(`
303
- @prefix foaf: <http://xmlns.com/foaf/0.1/> .
304
- _:a foaf:name "Alice" ; a foaf:Person .
305
- _:b foaf:name "Bob".
306
- `);
307
-
308
- // Every op returns a new FnDataset; ds is never touched.
309
- const people = filter(ds, (q) => q.predicate.value.endsWith('/type'));
310
-
311
- // Dataflow: a derive() node recomputes only when its input's content
312
- // hash actually changes -- not when a new (but equal-content)
313
- // FnDataset object is set into the cell.
314
- const source = cell(ds);
315
- const derived = derive((d) => filter(d, (q) => q.predicate.value.endsWith('name')), source);
316
- await derived.get(); // computes
317
- source.set(await parse(sameTextAsBefore));
318
- await derived.get(); // memoized hit -- same content, new object
319
-
320
- await hash(ds); // sha256 hex of RDFC-1.0 canonical N-Quads
321
- ```
322
-
323
- `fromDataset(dataset)` / `toDataset(fnDataset)` convert to and from
324
- the mutable RDF/JS `Dataset` above, so the two styles compose freely.
325
- `union`/`difference`/`filter`/`mapQuads` give set algebra without
326
- methods; `query`/`entail`/`canonicalize`/`graphs` mirror the plain
327
- API's capability gating (`capabilities()`) exactly. `validate`/`shex`/
328
- `fromMapping`/`fromCsvw`/`rif` wrap the SHACL/ShEx/RML/CSVW/RIF
329
- surface below the same way — FnDataset in, FnDataset (or a plain
330
- verdict) out:
331
-
332
- ```js
333
- const { parse, validate, shex, fromMapping, fromCsvw, rif } = require('@factoidal/core/fn');
334
-
335
- const data = await parse(personTtl);
336
- const shapes = await parse(shapesTtl);
337
- const { conforms, report } = await validate(data, shapes); // SHACL
338
-
339
- const ok = await shex(data, shexSchemaJson, 'http://example.org/alice'); // true|false|null
340
-
341
- const mapped = await fromMapping(await parse(rmlMappingTtl), csvOrJsonText, 'csv'); // RML
342
-
343
- const tabular = await fromCsvw(csvText, csvwMetadataJson, { mode: 'minimal' }); // CSVW csv2rdf
344
-
345
- const saturated = await rif(data, rifRulesXml); // RIF Core forward chaining
346
- ```
347
-
348
- ## Lean engine (new in 0.2.0)
349
-
350
- Two engines now ship in one package. `factoidal` and `factoidal/wasm`
351
- are the F\*-extracted engine, unchanged. The subpaths below are the
352
- Lean 4 engine (`L4Factoidal`, compiled to wasm).
353
-
354
- `factoidal/l4` also exposes two deliberately narrow physical helpers:
355
- `scanIBK2Predicate(ibk2Hex, predicateIri)` for the predecessor format and
356
- `scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` for the current
357
- predicate-local format. They validate one canonical RDF block and scan its
358
- named predicate, returning N-Triples and a row count. The IBK3 source scope
359
- must be shared across blocks partitioned from one RDF import unit and differ
360
- across unrelated units; this preserves document-scoped blank-node identity
361
- when fragments are composed. The hexadecimal argument is a portable
362
- diagnostic ABI, not the intended high-throughput buffer interface.
363
-
364
- ```js
365
- const l4 = require('factoidal/l4-core'); // Lean engine, same API shape
366
- const { select } = require('factoidal/select'); // choose an engine per call
367
- ```
368
-
369
- ### `factoidal/l4-core`
370
-
371
- Same call shape as the main API — `parse`, `query`, `update`,
372
- `serialize`, `canonicalize`, `graphs`, `canonicalHash`, `owlClosure`,
373
- `coreRdfsClosure`, `coreRdfsCheck`, `rhoDfClosure`,
374
- `rhoDfFragmentCheck`, `rdfsPlusClosure`, `owlIsConsistent` — plus four
375
- Common Logic / IKL operations that exist only here, because the F\* tree
376
- has no CL parser.
377
-
378
- | Function | In → out | What the answer is worth |
379
- |---|---|---|
380
- | `clParse(clifText)` | CLIF text → shape report | sentence count, CL-vs-IKL dialect, canonical re-serialisation. Reads CLIF; never produces RDF |
381
- | `clSerialize(clifText)` | CLIF → CLIF | canonical writer. Returns `roundTripProved: false` — `clif_roundTrip` is an open lemma and the fragment boundary is measured, not proved |
382
- | `clAlphaNorm(clifText)` | CLIF → CLIF | alpha-equivalence canonical form (IKL Appendix B condition 1) |
383
- | `clNormalize(clifText)` | CLIF → CLIF | Hayes's IKL-to-CL reduction. Returns `preserves: "satisfiability"` — **not** equivalence — and `noIntrusion`, the proof hypothesis decided rather than assumed |
384
-
385
- A fifth op, `clFiniteSat(interpJson, clifText)`, is reachable only
386
- through the raw dispatch ABI: `l4.call('clFiniteSat', [interpJson,
387
- clifText])`. It is not in the typed layer.
388
-
389
- ### `factoidal/select`
390
-
391
- Per-instance backend choice with a per-call override.
392
-
393
- | Value | Behaviour |
394
- |---|---|
395
- | `lean` / `fstar` | that engine only; **throws** if it does not implement the function |
396
- | `lean1st` / `fstar1st` | prefer that engine, fall through to the other for unimplemented functions |
397
- | `slowcompareboth` | run both and **report** disagreement rather than throwing |
398
-
399
- Every result names the engine that answered. `capabilityTable()`
400
- returns which functions each engine implements.
401
-
402
- ### What this surface is, honestly
403
-
404
- Four `String → String` operations plus one through raw dispatch. The
405
- Lean tree behind them is larger than that — the CL/IKL and unified
406
- model-theory modules run to about 22,000 lines — but only these reach
407
- JavaScript today. Everything else in the Lean tree is used through
408
- `parse`/`query`/`closure`, or not exposed at all.
409
-
410
- ## The `factoidal` command: querying a persisted store
411
-
412
- Installing this package puts a `factoidal` command on PATH. It reads a
413
- **Shardborough** store — the on-disk format the Lean `l4block-*` tools
414
- write — with no native binary: JavaScript reads the files and moves the
415
- bytes, and the Lean engine running as WebAssembly makes every format
416
- decision (parsing the manifest, choosing the blocks, verifying their
417
- SHA-256, evaluating the SPARQL).
418
-
419
- > This command is not the native F\* `factoidal` binary that the API
420
- > table below refers to. That one is `bin/<platform>/factoidal` in the
421
- > repository and takes subcommands such as `shex` and `compact`. This
422
- > one takes `version`, `sample-store`, `inspect` and `query`.
423
-
424
- ### First query, with nothing else to download
425
-
426
- The package carries an activated store, so a fresh install answers a
427
- SPARQL query at once:
11
+ import { parse, csvwToRdf } from "@factoidal/core";
12
+
13
+ const ds = await parse('<http://x/a> <http://x/n> "A" .', { format: "ntriples" });
14
+ const rows = await ds.query("SELECT ?n WHERE { ?s <http://x/n> ?n }");
15
+ console.log(rows[0].get("n").value); // A
16
+ console.log(ds.toString("turtle"));
17
+ console.log(await ds.serialize({ format: "rdfxml" }));
18
+ console.log(await ds.canonicalize()); // RDFC-1.0
19
+
20
+ const table = await csvwToRdf("id,name\n1,a\n", "", { mode: "minimal", url: "http://x/t.csv" });
21
+ console.log(table.size); // 2
22
+
23
+ ds.dispose();
24
+ table.dispose();
25
+ ```
26
+
27
+ A `Dataset` keeps its data inside the engine. Every producer returns one and
28
+ every consumer accepts one, so a pipeline does not re-parse between steps.
29
+ An unknown option key throws. With no `format`, the format is inferred and
30
+ reported in `dataset.format`; an explicit `format` is never overridden.
31
+ RDF 1.2 input uses `turtle12`, `ntriples12`, `nquads12` or `trig12`.
32
+
33
+ ## What it does
34
+
35
+ - Syntaxes: N-Triples, N-Quads, Turtle, TriG, RDF/XML, JSON-LD; RDF 1.1 and 1.2
36
+ - SPARQL 1.1 and 1.2 query and update; GeoSPARQL functions, text search,
37
+ JavaScript extension functions, SERVICE
38
+ - Reasoning: RDFS, OWL 2 RL closure, OWL 2 DL consistency and entailment, RIF Core
39
+ - Validation: SHACL, ShEx, JSON Schema, Schematron
40
+ - Mapping: CSVW, RML, JSON-LD
41
+ - Canonicalization: RDFC-1.0; Verifiable Credentials Data Integrity (eddsa-rdfc-2022), did:key
42
+ - XML: XML 1.0, XPath 1.0, XSLT 1.0, XForms
43
+ - Servers: SPARQL 1.1 Protocol and Graph Store (Fuseki-compatible paths,
44
+ Jena TDB2 read only), Linked Web Storage 1.0, Solid
45
+ - A persisted store with a delta log for updates (early stage)
46
+ - RDF/JS DataFactory and DatasetCore (`/rdfjs`), a functional API (`/fn`)
47
+
48
+ Formats (generated from `formats()`):
49
+
50
+ <!-- formats-table:start (generated by test/formats-table.test.mjs) -->
51
+ | format | read RDF 1.1 | read RDF 1.2 | write RDF 1.1 | write RDF 1.2 | engine |
52
+ |---|---|---|---|---|---|
53
+ | nquads | yes | yes | yes | yes | core |
54
+ | ntriples | yes | yes | yes | yes | core |
55
+ | turtle | yes | yes | yes | yes | core |
56
+ | trig | yes | yes | yes | yes | core |
57
+ | rdfxml | yes | no | yes | no | core |
58
+ | jsonld | yes | no | yes | no | full |
59
+ | rdfc | no | no | yes | no | core |
60
+ | sparql-results-json | no | no | yes | yes | core |
61
+ | sparql-results-xml | no | no | yes | yes | core |
62
+ | sparql-results-csv | no | no | yes | yes | core |
63
+ | sparql-results-tsv | no | no | yes | yes | core |
64
+ <!-- formats-table:end -->
65
+
66
+ Missing writers (HDT, N3, RDFa): https://github.com/danbri/factoidal/issues/883.
67
+ Full API: [API.md](https://github.com/danbri/factoidal-core/blob/main/npm/factoidal/API.md).
68
+ Changes: [CHANGELOG.md](CHANGELOG.md).
69
+
70
+ ## Command line
428
71
 
429
72
  ```console
430
- $ npm install @factoidal/core
431
- $ npx factoidal query "$(npx factoidal sample-store)" \
432
- 'SELECT ?c ?l
433
- WHERE { ?c <http://www.w3.org/2004/02/skos/core#inScheme>
434
- <http://cv.iptc.org/newscodes/videocodec/> ;
435
- <http://www.w3.org/2004/02/skos/core#prefLabel> ?l .
436
- FILTER(langMatches(lang(?l), "en")) }
437
- LIMIT 4'
438
- c l
439
- <http://cv.iptc.org/newscodes/videocodec/c001> "Analogue Black and White"@en-gb
440
- <http://cv.iptc.org/newscodes/videocodec/c002> "PAL"@en-gb
441
- <http://cv.iptc.org/newscodes/videocodec/c003> "NTSC"@en-gb
442
- <http://cv.iptc.org/newscodes/videocodec/c004> "SECAM"@en-gb
73
+ $ npx factoidal serve --file data.ttl /ds # SPARQL endpoint at /ds/sparql
74
+ $ npx factoidal serve --tdb2 DIR /ds # serve a Jena TDB2 database
75
+ $ npx factoidal pack data.ttl store/gen-1 && npx factoidal activate store gen-1
76
+ $ npx factoidal query store 'SELECT * WHERE { ?s ?p ?o } LIMIT 5'
77
+ $ npx factoidal query "$(npx factoidal sample-store)" 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
78
+ $ npx factoidal lws-serve DIR # Linked Web Storage
79
+ $ npx factoidal solid-serve DIR # Solid Protocol
443
80
  ```
444
81
 
445
- `factoidal sample-store` prints the path; `--json` adds what was
446
- recorded when the store was packed. From JavaScript:
82
+ `npx factoidal --help` lists every command. Replacing Apache Jena Fuseki:
83
+ [docs/fuseki-replacement.md](https://github.com/danbri/factoidal-core/blob/main/npm/factoidal/docs/fuseki-replacement.md).
447
84
 
448
- ```js
449
- import { sampleStorePath, sampleStoreFacts } from '@factoidal/core/sample-store'
450
- ```
451
-
452
- The store holds 4,434 triples in 13 predicate blocks: five IPTC
453
- NewsCodes vocabularies, published by the IPTC under CC BY 4.0 and taken
454
- from [danbri/skosdex](https://github.com/danbri/skosdex). See `NOTICE`.
85
+ ## Test results
455
86
 
456
- ### Any other store
87
+ Official test suites, pass / fail / total, 2026-10-09 [B] unless marked.
457
88
 
458
- ```console
459
- $ factoidal inspect ./mystore
460
- store ./mystore
461
- generation gen-1 (activated through CURRENT)
462
- manifest manifest.sbm2, 2372 bytes, wire version 6
463
- layout predicate-ibk3-ptd1-sri2-tli1-oli2-merkle-v0
464
- blank-node profile (none recorded)
465
- term registry local-ibk3-ptd1-v0
466
- fixed-chunk Merkle commitment yes
467
- 5 entries, 393775 bytes, 6455 rows
468
- generation directory holds 42 files, 846592 bytes
469
-
470
- # rows bytes kind graphs predicate
471
- 0 1800 110085 IBK3 - http://www.wikidata.org/prop/direct/P31
472
- 1 719 35535 IBK3 - http://www.wikidata.org/prop/direct/P361
473
- ...
474
-
475
- $ factoidal query ./mystore 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
476
- mode ibk3-paged-merkle-full-manifest(5), 5 artifacts, 393775 bytes read, plan declares 6455 block rows
477
- n
478
- "6455"^^<http://www.w3.org/2001/XMLSchema#integer>
479
- 1 row
480
- ```
481
-
482
- `STORE` is a collection root: the directory holding `CURRENT`. The plan
483
- line goes to stderr, so stdout carries only the result; `--quiet`
484
- removes it.
485
-
486
- | Option | What it does |
89
+ | Suite | Result |
487
90
  |---|---|
488
- | `--format table` | default; a human display of the results |
489
- | `--format json` | SELECT prints the engine's SPARQL 1.1 Query Results JSON; ASK and CONSTRUCT print the operation's envelope |
490
- | `--format nquads` | CONSTRUCT only: the graph the engine serialized |
491
- | `--format turtle` | CONSTRUCT only: that graph through the engine's own Turtle writer |
492
- | `--explain` | print the artifacts the query needs and the open mode, and stop |
493
- | `--limit N` | print at most N table rows; the total is always named |
494
- | `--file PATH` | read the query text from a file |
495
- | `--generation NAME` | read that generation rather than the activated one |
496
-
497
- Under Deno, run the file directly; `inspect` and `query` need only
498
- `--allow-read`:
499
-
500
- ```console
501
- $ deno run --allow-read node_modules/@factoidal/core/bin/factoidal.mjs query ./mystore 'ASK { ?s ?p ?o }'
502
- ```
503
-
504
- ### What the command answers for, and what it does not
505
-
506
- * **Every artifact is verified.** The engine refuses the whole query
507
- when a block's bytes do not hash to the SHA-256 the manifest commits,
508
- and names the artifact.
509
- * **Three caps.** One call reads at most 64 artifacts, 8388608 artifact
510
- bytes and 100000 rows. A query over any of them is refused before a
511
- single file is read, with the cap and the value named. Nothing is
512
- truncated.
513
- * **Committed artifacts only.** A store carrying uncompacted delta-log
514
- updates is not served by this path; use the native `l4block-*` tools.
515
- * **`pack`, `activate`, `update` and `compact` exit 3.** They need
516
- WebAssembly operations that do not exist yet
91
+ | SPARQL 1.1 Query, Update, Protocol | 631 / 0 / 631 |
92
+ | SPARQL 1.2 | 254 / 0 / 254 |
93
+ | SPARQL Protocol, Graph Store, Service Description against `serve` | 50 / 0 / 50 [P] |
94
+ | RDF 1.1 N-Triples, N-Quads, Turtle, TriG, RDF/XML | 992 / 0 / 992 |
95
+ | RDF 1.2 syntax suites | all pass |
96
+ | RDF 1.1 Semantics | 39 / 0 / 39 |
97
+ | RDF 1.2 Semantics | 40 / 7 / 47 |
98
+ | JSON-LD 1.1 toRdf, expand, flatten | 910 / 0 / 910 |
99
+ | JSON-LD 1.1 compact, fromRdf | 298 / 0 / 300 (2 not run) |
100
+ | JSON-LD 1.1 Framing | 29 / 63 / 92 |
101
+ | RDFC-1.0 | 86 / 0 / 86 |
102
+ | OWL 2 RL | 126 / 0 / 126 |
103
+ | OWL 2 DL type-inconsistency | 116 / 11 / 128 (1 other) |
104
+ | OWL 2 DL positive entailment | 169 / 35 / 206 (2 other) |
105
+ | SHACL 1.0 Core, SHACL-SPARQL | 120 / 0 / 120 |
106
+ | SHACL 1.2 Core | 103 / 30 / 133 |
107
+ | ShEx 2 validation, ShExC | 1624 / 0 / 1624 |
108
+ | CSVW | 821 / 1 / 822 |
109
+ | RIF Core | 43 / 0 / 46 (3 undecided) |
110
+ | RML Core | 60 / 0 / 76 (16 not attempted) |
111
+ | XML 1.0 (xmlconf) | 1861 / 0 (516 not run) |
112
+ | XSD 1.1 datatypes | 18951 / 5 / 18956 |
113
+ | XSLT 1.0 | 84 / 3 / 88 (1 refused) |
114
+ | JSON Schema draft-07 | 770 / 0 / 770 |
115
+ | VC Data Integrity, did:key, Ed25519 | 58 / 0 / 58 |
116
+ | XPath 1.0 (own battery) | 100 / 0 / 100 [X] |
117
+
118
+ [B] https://github.com/danbri/factoidal-core/blob/03e2e6deee8146ce7e6fdc3f8f34b8245af5ab54/docs/designissues/2026-10-09-lean-conformance-baseline.md
119
+ [P] https://github.com/danbri/factoidal-core/blob/9a4045f1572a032b53ccb867f8395fe0c8d97bb9/docs/designissues/2026-10-10-store-serve.md
120
+ [X] https://github.com/danbri/factoidal-core/blob/cb871c2de8354362ae6226eb57fb6c549624ea1a/docs/designissues/2026-10-10-xml-profile-widened.md
121
+
122
+ ## Limits
123
+
124
+ - Size: 2.5 MB packed, 12.4 MB unpacked; one 11.7 MB wasm module.
125
+ - `factoidal update` and `compact` exit 3; updates work through `serve`
517
126
  (https://github.com/danbri/factoidal/issues/641).
518
- * **Node's WebAssembly frame budget.** Some evaluator paths recurse once
519
- per row. Measured 2026-09-03 on a 6455-row store, `SELECT ?s ?p ?o
520
- WHERE { ?s ?p ?o }` overflows the stack under Node's default while
521
- `SELECT *`, or the same query with a `LIMIT`, does not, and Deno
522
- clears all of them. The command reports it and exits 1 rather than
523
- crashing; `node --stack-size=4000` clears it.
524
-
525
- Measured 2026-09-03 on macOS arm64, the 6455-triple `sequence_variant`
526
- store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
527
- including start-up: 220 ms through this command, 33 ms through the
528
- native `l4block-id-v3-query`.
529
-
530
- ### Holding a store open: verify once, answer many
531
-
532
- `queryStore` is stateless. Every call transfers the artifacts again, and
533
- the engine hashes, decodes and indexes them again. A process that answers
534
- many questions against one generation — a chat bot, an MCP server, a
535
- SPARQL endpoint — pays all of that for every question, and none of it
536
- depends on the question.
537
-
538
- A **store handle** holds that work. `openStoreHandle` verifies each
539
- artifact against the SHA-256 the manifest commits, decodes each block and
540
- indexes the rows, once. `handle.query()` then answers from what it
541
- retained. Several stores can be open at the same time.
542
-
543
- ```js
544
- import { loadEngine } from '@factoidal/core/bin/engine.mjs'
545
- import { openStore, openStoreHandle, listStoreHandles } from '@factoidal/core/store'
546
-
547
- const engine = await loadEngine()
548
-
549
- // Two stores, open at once, held for the life of the process.
550
- const skos = openStoreHandle(engine, openStore('./skos-store'))
551
- const docs = openStoreHandle(engine, openStore('./docs-store'))
552
-
553
- console.log(listStoreHandles(engine))
554
- // { ok: true, handles: [ {handle:'s1', bytes:…, rows:…}, {handle:'s2', …} ],
555
- // bytes: …, rows: …, handleCap: 8, bytesCap: 134217728 }
556
-
557
- const PREFIX = 'PREFIX skos: <http://www.w3.org/2004/02/skos/core#>'
558
-
559
- function labelsMatching (handle, needle) {
560
- const answer = handle.query(`${PREFIX}
561
- SELECT ?c ?l WHERE {
562
- GRAPH ?g { ?c skos:prefLabel ?l }
563
- FILTER(CONTAINS(LCASE(STR(?l)), "${needle.toLowerCase()}"))
564
- } LIMIT 10`)
565
- return answer.srj.results.bindings
566
- }
567
-
568
- // Many questions, each a new search string. None of them re-reads a block.
569
- for (const needle of ['water', 'forest', 'railway', 'volcano']) {
570
- console.log(needle, labelsMatching(skos, needle).length)
571
- }
572
- console.log(docs.query(`${PREFIX} SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`))
573
-
574
- skos.close()
575
- docs.close()
576
- ```
577
-
578
- `openStoreHandle(engine, store, options)` takes:
579
-
580
- | option | effect |
581
- |---|---|
582
- | (none) | open every artifact the manifest declares |
583
- | `{ sparql }` | open only the artifacts that query's plan names |
584
- | `{ keys }` | open exactly these artifact keys |
585
-
586
- `queryStoreHandle(engine, handle, sparql)`, `closeStoreHandle(engine,
587
- handle)` and `listStoreHandles(engine)` are the same operations for a
588
- caller that holds only the handle id.
589
-
590
- **Measured 2026-09-04**, macOS arm64, Node 22, the 141-graph SKOS store,
591
- `skos:prefLabel` block of 5,571,302 bytes and 45,806 rows, one
592
- `CONTAINS` query per row, a DIFFERENT search string every time:
593
-
594
- | | stateless `queryStore` | handle |
595
- |---|---|---|
596
- | first query (open + query) | 1376 ms | 1363 ms |
597
- | second query, different search string | 1376 ms | 95 ms |
598
- | tenth query, all different | 1382 ms | 103 ms |
599
- | ten queries, total | 13962 ms | 2277 ms |
600
-
601
- **What a handle buys.** It removes the per-query digest check, block
602
- decode and index build. With the LGI1 literal index (below) a search is
603
- also sub-linear rather than a scan, so a handle plus the index is what
604
- makes repeated search on a large store interactive.
605
-
606
- **Residency.** Retaining a decoded block costs memory. Measured on the
607
- same store: 76 MiB resident with the engine loaded and no handle, 170 MiB
608
- with the handle open — about 94 MiB for a 5.5 MB packed block, and
609
- evaluation peaks higher again (346 MiB during the queries above). At
610
- corpus scale the marginal figure is smaller, because that one is carrying
611
- the fixed cost of the process: measured 2026-09-05 on a 7,315,251-quad
612
- store, one handle over 257 blocks and 103,341,302 retained artifact bytes
613
- peaked at 1,675,345,920 bytes resident — 16.2 bytes resident per retained
614
- byte.
615
-
616
- The caps are on ARTIFACT bytes, which is what the manifest declares: 8
617
- open handles, and 134217728 (128 MiB) retained artifact bytes across all
618
- of them. There is NO cap on the number of artifacts a handle retains; a
619
- count bounds nothing that the bytes do not
620
- (https://github.com/danbri/factoidal/issues/657). 128 MiB is half the
621
- wasm32 address space divided by that measured multiplier, and it admits
622
- the 257-block corpus-wide set above. A cap is a refusal naming the cap
623
- and the value that tripped it; no handle is ever evicted to make room for
624
- another. `listStoreHandles` is how a server sees its own residency.
625
-
626
- **One call at a time.** The WebAssembly module is single-threaded. Two
627
- `query()` calls cannot overlap; a server queues them.
628
-
629
- ### A handle on a large store: the worker route
630
-
631
- Several engine paths recurse once per manifest entry and once per row.
632
- Against a large collection that exceeds the default call stack of Node
633
- and of Deno, and the failure is `Maximum call stack size exceeded`.
634
- **Measured 2026-09-05**, macOS arm64, Node 22.22.2, on a 7,315,251-quad
635
- collection of 3,286 blocks in 204 graphs: `storeQueryPlan` alone
636
- overflows on plain `node`, before one artifact byte is read.
637
- `node --stack-size=60000` clears the plan, the open and every query.
638
-
639
- `openStoreHandleOnWorker` removes the flag. It holds the engine and the
640
- handle on a `worker_threads` thread with `resourceLimits.stackSizeMb`,
641
- the route `factoidal pack` already takes
642
- ([issue 649](https://github.com/danbri/factoidal/issues/649)). A handle
643
- is state inside the wasm instance and an instance does not cross a
644
- thread boundary, so the handle lives where the raised stack is, and
645
- `query()` and `close()` are messages to it.
646
-
647
- ```js
648
- import { openStoreHandleOnWorker, closeSharedStoreWorkerSession }
649
- from '@factoidal/core/store-worker'
650
-
651
- // No runtime flag. One worker thread, shared by every handle opened
652
- // this way, so a caller that opens several stores pays for one thread
653
- // and one copy of the module.
654
- const handle = await openStoreHandleOnWorker('/path/to/store', {
655
- sparql: 'SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c ?p ?l } }'
656
- })
657
- const answer = await handle.query(`${PREFIX}
658
- SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c skos:prefLabel ?l }
659
- FILTER(CONTAINS(LCASE(STR(?l)), "volcan")) } LIMIT 8`)
660
- await handle.close()
661
- await closeSharedStoreWorkerSession()
662
- ```
663
-
664
- | option | effect |
665
- |---|---|
666
- | `{sparql}`, `{keys}` | the same artifact choice `openStoreHandle` takes |
667
- | `{generation}` | open a generation that has not been activated |
668
- | `{session}` | open into a session you started with `openStoreWorkerSession()` |
669
- | `{ownWorker: true}` | give this handle its own thread and its own copy of the module |
670
- | `{worker: false}` | open in this process, with no thread |
671
-
672
- What it costs, all **measured 2026-09-05** on the same machine, against
673
- the bundled sample store so the overhead is not lost in the query:
674
-
675
- | | in process | on a worker |
676
- |---|---|---|
677
- | one-shot query (thread start + engine load + query) | 163 ms | 220 ms |
678
- | handle open (thread start + engine load + open) | 124 ms | 236 ms |
679
- | every query after the open | 1 ms | 1 ms |
680
-
681
- So the thread and its second copy of the engine cost about 110 ms once,
682
- and the message round trip is under a millisecond. Three further costs:
683
- every call is asynchronous where the in-process handle is synchronous;
684
- an extension function registered on the main thread's engine
685
- (`bin/ext.mjs`) is not visible to the worker's engine; and the worker
686
- keeps the process alive until `close()`.
687
-
688
- **Deno takes a different route.** Deno's `node:worker_threads` shim
689
- accepts `resourceLimits.stackSizeMb` and raises almost nothing with it —
690
- measured by counting frames to the overflow inside the worker, Node
691
- reaches 41,195 frames by default and 696,555 at `stackSizeMb` 64, where
692
- Deno reaches 10,835 and 13,837. `openStoreHandleOnWorker` therefore
693
- gives a Deno caller an in-process handle behind the same asynchronous
694
- interface, and the process supplies the stack:
695
-
696
- ```
697
- deno run --allow-read --v8-flags=--stack-size=65536 your-program.mjs
698
- ```
699
-
700
- **A one-shot `factoidal query` pays none of this.** It builds no handle,
701
- so it runs in process and only retries on a worker if the runtime runs
702
- out of frames; under Deno it re-executes itself once with a raised V8
703
- stack, which needs `--allow-run` and `--allow-env`. `--no-worker` turns
704
- the retry off. See
705
- [issue 653](https://github.com/danbri/factoidal/issues/653).
706
-
707
- ## API (draft)
708
-
709
- The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to
710
- `bin/<platform>/factoidal`) calls the exact same F\*-extracted
711
- functions as this npm surface — see `tests/local/cli_api_parity.sh`
712
- for the test that diffs CLI output against the npm API on shared
713
- fixtures.
714
-
715
- | Function | Signature (informal) | CLI equivalent | Notes |
716
- |---|---|---|---|
717
- | `parse` | `(text, {format?, baseIRI?}) => Dataset` | `factoidal dump-nq -d FILE` (or `dump`/`dump-turtle`) | formats: `turtle`, `ntriples`, `nquads`, `trig`, `rdfxml`, `jsonld`\* — auto-detected where possible. Each call is one document: blank-node labels are scoped per RDF 1.1 |
718
- | `query` | `(Dataset \| string, sparql, {entail?}) => Bindings[] \| boolean \| Dataset` | `factoidal query -d FILE -e 'SPARQL' [--entail RDFS\|OWL-RL]` | SELECT → array of `Map<var, Term>`; ASK → boolean; CONSTRUCT → Dataset\*\*; `entail: "RDFS" \| "OWL-RL"` |
719
- | `update` | `(Dataset, sparqlUpdate) => Dataset` | `factoidal update -d FILE -e 'SPARQL update'` | \*\* in-memory; no persistence. (Durable UPDATE against a COTTAS store is a separate path: `factoidal serve --rw --delta-log ...` / `factoidal compact`.) |
720
- | `serialize` | `(Dataset, {format}) => string` | `factoidal dump-nq FILE` (nquads) / `factoidal dump FILE` (ntriples) / `factoidal dump-turtle FILE` | `nquads`, `ntriples` (sorted); `turtle`\*\* (prefix-compacted, subject-grouped — needs the entry bundle, flattens named graphs into the default graph) |
721
- | `canonicalize` | `(Dataset \| string) => string` | `factoidal canonicalize FILE` | RDFC-1.0 canonical N-Quads\*\* |
722
- | `graphs` | `(Dataset) => Array<[iri, Dataset]>` | `factoidal graphs list FILE` | enumerate named graphs (default graph excluded); pure enumeration, no engine round-trip |
723
- | `canonicalHash` | `(Dataset) => string` | `factoidal graphs hash FILE IRI` | RDFC-1.0 canonical hash of one graph\*\*; graph-scoped sibling of `canonicalize` — typically called with one entry of `graphs()`'s output |
724
- | `shaclValidate` | `(data, shapes) => {conforms, report: Dataset}` | `factoidal shacl --data FILE --shapes FILE [--json]` (alias: `factoidal validate --shapes FILE FILE`) | \*\* SHACL Core validation; `report` is the `sh:ValidationReport` graph; exit code 0 iff `sh:conforms` |
725
- | `shexValidate` | `(data, schemaText, focus, shape?) => boolean \| null` | `factoidal shex --data FILE --schema FILE.{json,shex} --node N [--shape S]` | \*\* ShEx (Shape Expressions) validation of one focus node; `schemaText` accepts ShExJ or ShExC — dispatched by the first non-whitespace character (`{` ⇒ ShExJ, else ShExC; the CLI additionally honors a `.shex` file extension); `null` = outside this engine's decidable ShEx fragment, never a guessed answer |
726
- | `owlClosure` | `(data, mode) => Dataset` | `factoidal entail --data FILE --regime RDFS\|OWL-RL` | \*\* materializes the entailment closure (input + derived triples), default graph only. (`query --entail` applies the same closure internally before evaluating a query, but does not dump it on its own.) |
727
- | `rmlMap` | `(mapping, sourceData, sourceKind) => Dataset` | `factoidal rml --mapping FILE --source FILE --kind json\|csv` | \*\* evaluates an RML mapping graph against one logical source (`sourceKind: "json" \| "csv"`); every triples map reads the SAME source — cross-source joins are out of scope for this entry point |
728
- | `csvwToRdf` | `(csvText, metadataJson?, {mode?, base?, url?}) => Dataset` | `factoidal csvw --csv FILE [--metadata FILE] [--minimal] [--base IRI] [--url URL]` | \*\* CSVW csv2rdf conversion; metadata omitted = schema inferred from the CSV header row; `mode: "standard" \| "minimal"` (default standard); every table in a multi-table group reads the SAME csvText |
729
- | `jsonldToRdf` | `(jsonldText, {base?, rdfDirection?, expandContext?, processingMode?}) => Dataset` | `factoidal jsonld --in FILE [--base IRI]` (or `factoidal dump-nq FILE.jsonld`, format auto-detected) | \*\* JSON-LD parsing with options `parse()` has no room for; plain `parse(text, {format:'jsonld'})` also works for the common case |
730
- | `jsonldFromRdf` | `(data, {useNativeTypes?, useRdfType?}) => JSON-LD` | `factoidal dump-nq` ← (reverse) | \*\* serializes an RDF dataset as expanded-form JSON-LD (the reverse of `jsonldToRdf`); returns the parsed JSON-LD document (array of node objects), JCS-canonical |
731
- | `didKeyResolve` | `(didString) => Dataset` | N/A (`did_runner`) | \*\* resolves a `did:key:z6Mk...` (Ed25519) to its DID Document as RDF; non-Ed25519 / malformed inputs reject rather than guess |
732
- | `xmlWellformed` | `(xmlText) => boolean` | N/A (`xml_runner`) | \*\* XML 1.0 well-formedness check (byte-oriented, no DOCTYPE/DTD production — a document with a DOCTYPE reports `false`) |
733
- | `xpathEval` | `(xmlText, xpathExpr) => {resultType, value \| nodes, ...}` | N/A | \*\* XPath 1.0 evaluation over an XML document; `resultType` is `'nodeset' \| 'string' \| 'number' \| 'boolean'` |
734
- | `rifEval` | `(data, rifRulesXml) => Dataset` | `factoidal rif --rules FILE --data FILE` | \*\* RIF Core forward-chaining saturation (materializes input + derived triples); accepts real vendored RIF-XML (`<!DOCTYPE>` + `&rif;`/`&xs;`/`&rdf;` entities) unmodified |
735
- | `toCottas` | `(data, {format?}) => Uint8Array` | `factoidal compact --native-writer` | \*\* serializes a dataset to COTTAS/Parquet bytes via the native writer; round-trips into `openCottas()` and into the native `--data-cottas`/`--data-cottas-mem` CLI flags byte-for-byte |
736
- | `openCottas` | `(bytes: string \| Uint8Array \| ArrayBuffer) => handle` | `factoidal query --data-cottas-mem FILE` | \*\* opens a whole `.cottas` artifact's bytes as a queryable, read-only, in-memory store; rows decode lazily as `queryCottas()` touches them (no heap `Dataset`, no full parse) |
737
- | `queryCottas` | `(handle, sparql) => Bindings[] \| boolean \| Dataset` | `factoidal query --data-cottas-mem FILE -e 'SPARQL'` | \*\* SPARQL over a store opened by `openCottas()`; no `entail` option, no write overlay (read-only), no DESCRIBE |
738
- | `closeCottas` | `(handle) => void` | N/A | releases a handle from this process's registry; does not evict the underlying byte cache |
739
- | `queryRaw` | `(input, sparql) => string` | `factoidal query -d FILE -e 'SPARQL' -o json` | SPARQL-Results-JSON string, for callers that want the wire form |
740
- | `capabilities` | `() => {construct, update, canonicalize, graphs, canonicalHash, shacl, shex, owlClosure, rml, csvw, jsonld, jsonldFromRdf, didKey, xml, xpath, rif, cottasBytesStore, ...}` | N/A | runtime feature probe; the CLI is one fixed native binary, not a runtime bundle whose feature set varies |
741
- | `dataFactory` | RDF/JS DataFactory | N/A | data-model class, not an engine operation |
742
- | `Dataset` | RDF/JS DatasetCore | N/A | returned by `parse`; accepted everywhere |
743
-
744
- The `fn.js` functional layer's own combinators — `union`, `difference`,
745
- `filter`, `mapQuads`, `equals`, `hash`, `builder`/`fromChunks`,
746
- `cell`/`derive` — are pure client-side set algebra and dataflow
747
- plumbing over already-materialized `Dataset`s. They have no CLI
748
- equivalent by design: there is no engine operation to wrap, only JS
749
- composition on top of the operations already listed above.
750
-
751
- \* JSON-LD parsing (expanded form, inline `@context`, `@base`
752
- resolution, `@reverse`, container maps) works through both `parse()`
753
- and `jsonldToRdf()` when the npm-entry bundle is loaded. Remote
754
- `@context` URLs need a `documentLoader`, which this package's entries
755
- don't register (an honest failure, not a silent wrong answer) —
756
- tracked against the vendored W3C json-ld-api suite.
757
- \*\* CONSTRUCT, UPDATE, `canonicalize`, `canonicalHash`,
758
- `shaclValidate`, `shexValidate`, `owlClosure`, `rmlMap`, `csvwToRdf`,
759
- `jsonldToRdf`, `jsonldFromRdf`, `didKeyResolve`, `xmlWellformed`, `xpathEval`, `rifEval`, `toCottas`, `openCottas`, `queryCottas`, and
760
- `closeCottas` are probed via `capabilities()`: they activate
761
- automatically when the dedicated npm-entry engine bundle is present,
762
- and the package reports their absence honestly against older bundles
763
- instead of guessing.
764
- `canonicalHash` rides the same engine support as `canonicalize` (it
765
- computes `canonicalize()` over one graph's triples); `graphs` is pure
766
- JS enumeration and is always available.
767
-
768
- ### The db API (openCottas/queryCottas/closeCottas/toCottas)
769
-
770
- The in-memory COTTAS/Parquet bytes store works identically on both
771
- engines: `require('@factoidal/core')` (js_of_ocaml) and
772
- `require('factoidal/wasm')` (wasm_of_ocaml, Node ≥ 22) expose the same
773
- `toCottas`/`openCottas`/`queryCottas`/`closeCottas` functions, backed
774
- by the same F\*-verified reader
775
- (`RDF.CottasStore`/`Parquet.Footer`/`SPARQL11_Store`). A store is
776
- read-only and holds no `entail` option — see the divergence list on
777
- `queryCottas`'s doc comment in `lib/api.js`. Zstd-compressed COTTAS
778
- pages are supported under js_of_ocaml (via the vendored `fzstd`
779
- decompressor baked into `factoidal.js`); under wasm_of_ocaml, Zstd
780
- decompression is still an identity-stubbed primitive (a documented
781
- gap, tracked separately) — write bytes with `toCottas()` (which never
782
- reaches for Zstd on small in-memory writes) rather than a
783
- Zstd-compressed on-disk fixture if you need a wasm-portable test case.
784
- The browser entries mirror this: `browser.js`'s `openCottas`/
785
- `queryCottas`/`closeCottas`/`toCottas` drive the js_of_ocaml npm-entry
786
- ABI over `fetch()`; `browser-wasm.js` exposes the same four functions
787
- against the wasm_of_ocaml npm-entry ABI (`factoidal-npm-entry.wasm.js`).
788
-
789
- ## Capability matrix
790
-
791
- Every public function, where it runs, and what it needs. "Node" is the
792
- `require('@factoidal/core')` / `import '@factoidal/core'` entry (`index.js` /
793
- `index.mjs`); "Browser" is `import 'factoidal/browser'` (`browser.js`);
794
- "Wasm" is `require('factoidal/wasm')`. **Needs** legend: *CLI bundle* =
795
- the fresh-eval `factoidal.js` bundle (always present); *entry* = the
796
- persistent `factoidal-npm-entry.js` ABI bundle (probe with
797
- `capabilities()`); *HACL\* init* = the wasm crypto backend must be
798
- initialised (auto on Node, explicit in the browser — see below);
799
- *IndexedDB* = a browser storage layer.
800
-
801
- | Function(s) | Node | Browser | Wasm | Needs |
802
- |---|---|---|---|---|
803
- | `parse`, `query` (SELECT/ASK), `serialize` (nquads/ntriples), `canonicalize`, `graphs`, `canonicalHash`, `queryHdt`, `queryRaw` | ✓ | ✓\* | ✓ | CLI bundle |
804
- | `query` (CONSTRUCT), `update`, `serialize` (turtle) | ✓ | ✓\* | ✓ | entry |
805
- | `shaclValidate`, `shexValidate`, `owlClosure`, `rmlMap`, `csvwToRdf`, `jsonldToRdf`, `jsonldFromRdf`, `didKeyResolve`, `xmlWellformed`, `xpathEval`, `rifEval` | ✓ | ✓ | partial† | entry |
806
- | `coreRdfsClosure`/`rhoDfClosure`, `coreRdfsCheck`/`rhoDfFragmentCheck`, `rdfsPlusClosure`, `tableauMaterialise`, `tableauDlInconsistent`, `owlIsConsistent`, `owlEntails` | ✓ | ✓ | wrapper only‡ | entry |
807
- | `xsltTransform`, `mathmlEval`, `xformsRecalc`, `jsonSchemaValidate`, `schematronValidate`, `toan*`, `matrix*` | ✓ | ✓ | partial† | entry |
808
- | `openCottas`, `queryCottas`, `closeCottas`, `toCottas` | ✓ | ✓ | ✓ | entry |
809
- | `vcSha256Hex`, `vcEd25519SecretToPublic`, `vcEd25519Sign`, `vcEd25519Verify`, `vcEddsaCreateFromCanonical`, `vcEddsaVerifyFromCanonical` | ✓ | ✓ | ✓ | entry + HACL\* init |
810
- | `deltaLogOpen`/`Append`/`ReadAllHex`/`Merge`/`Destroy` | ✗ | ✓ | ✗ | entry + IndexedDB |
811
- | `dataFactory`, `Dataset`, `fn.*` combinators (`union`/`difference`/`filter`/`mapQuads`/`equals`/`hash`/`builder`/`cell`/`derive`/`pipe`) | ✓ | — | ✓ | none (pure JS) |
812
- | `capabilities` | ✓ | — | ✓ | none (probe) |
813
-
814
- \* In the browser the same capability is reached through `browser.js`'s
815
- own function names, which differ from the Node API: SELECT/ASK is
816
- `query(dataString, sparql, {output})` (returns SPARQL-Results JSON, not
817
- `Bindings[]`); parse-to-N-Quads is `toRdf()`; multi-graph queries use
818
- `queryDataset()`. CONSTRUCT/UPDATE/Turtle ride the same `entry` bundle,
819
- fetched over the network on first use.
820
- † Wasm (`require('factoidal/wasm')`) today re-exports the core +
821
- validation/inference/COTTAS set (`parse`/`query`/`update`/`serialize`/
822
- `canonicalize`/`shaclValidate`/`shexValidate`/`owlClosure`/`rmlMap`/
823
- `csvwToRdf`/`jsonldToRdf`/`rifEval`/`openCottas`…/`capabilities`); the
824
- typed-engine `#74` functions and the `vc*`/`did`/`xml`/`xpath` wrappers
825
- are exposed on the js and browser entries. The underlying wasm ABI
826
- carries them — the `/wasm` re-export surface is being brought to parity.
827
- ‡ `coreRdfsClosure`/`rhoDfClosure`/`coreRdfsCheck`/`rhoDfFragmentCheck`/
828
- `rdfsPlusClosure`/`tableauMaterialise`/`tableauDlInconsistent`/
829
- `owlIsConsistent`/`owlEntails` are exported from `factoidal/wasm`
830
- (wrapper wiring — the same `buildApi()` `index.js` uses), but the
831
- committed `factoidal-npm-entry.wasm.js` ABI bundle predates these
832
- functions (built before they landed on the ABI) — calling them on the
833
- wasm engine throws the existing "pending npm-entry bundle" error until
834
- that bundle is rebuilt via a real `wasm_of_ocaml` build (not a copy).
835
- `capabilities()` on the wasm engine reports this honestly (`tableau:
836
- false` etc.) rather than guessing.
837
-
838
- ### VC crypto: the init story
839
-
840
- The `vc*` functions run the F\*-extracted VC Data Integrity pipeline
841
- over HACL\*'s official WebAssembly build, so the wasm backend has to be
842
- initialised before the first call (a verify against an uninitialised
843
- backend **throws** — it never silently returns `true`; issue #286).
844
-
845
- - **Node** (`index.js`/`index.mjs`, `/wasm`, `fn`): the typed wrappers
846
- **auto-await `initHacl()` on first call** — a caller writes
847
- `await vcEd25519Verify(pk, msg, sig)` and never touches init.
848
- - **Browser** (`browser.js`): **explicit init required** — the
849
- `hacl-wasm` URL is page-specific. Serve `hacl-wasm/` next to the page
850
- and `await initHacl({ apiUrl })` from `hacl-init.js` once, then call
851
- the `vc*` wrappers. (Auto-init isn't possible without a URL to fetch.)
852
-
853
- ### Native-CLI-only capabilities (not on the npm surface, by design)
854
-
855
- These live only in `bin/factoidal-cli` (the native `factoidal` binary),
856
- not in the JS package — they are I/O- or process-shaped, not pure
857
- value transforms:
858
-
859
- - **SPARQL endpoint / federation.** `factoidal serve` (HTTP endpoint),
860
- `SERVICE` federation and `--data-cottas`/`--data-hdt` *file-path*
861
- backends read the filesystem/network at query time. The npm
862
- `queryHdt`/`openCottas` take *bytes* in-process instead.
863
- - **Durable UPDATE against on-disk COTTAS.** `factoidal serve --rw
864
- --delta-log …` + `factoidal compact` — the npm `update()` is
865
- in-memory only; the browser `deltaLog*` family is the closest
866
- in-package durable path (IndexedDB, browser-only).
867
- - **Internal primitives.** `runFactoidalCli` / `loadNpmEntry` (browser)
868
- are exported but are the low-level bundle drivers, not the intended
869
- API; the `_*` functions (e.g. `_deltaLogCorruptLastForTest`) are
870
- test-only and intentionally left untyped.
871
-
872
- ### GeoSPARQL — six topological functions
873
-
874
- The `geof:` functions below are built into the SPARQL engine and need no
875
- import. They work through `query()` / `fn.query()` AND against a
876
- persisted store through `factoidal query`, because both paths evaluate
877
- in the same environment.
878
-
879
- geof:sfEquals geof:sfDisjoint geof:sfIntersects
880
- geof:sfTouches geof:sfWithin geof:sfContains
881
-
882
- ```sparql
883
- PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
884
- PREFIX geo: <http://www.opengis.net/ont/geosparql#>
885
- SELECT ?a WHERE {
886
- ?a :footprint ?w
887
- FILTER(geof:sfWithin(?w, "POLYGON((0 0,0 2,2 2,2 0,0 0))"^^geo:wktLiteral))
888
- }
889
- ```
890
-
891
- **What is NOT there**, stated so nobody plans around it: no
892
- `geof:distance`, `geof:buffer`, `geof:envelope`, `geof:boundary`,
893
- `geof:convexHull` or any other non-topological measure; no
894
- `geof:relate` with a DE-9IM matrix; no coordinate reference system
895
- handling beyond what the WKT literal carries; no GML literals. Geometry
896
- comes from a WKT parser, so a shapefile, GeoJSON or GML source must be
897
- converted to `geo:wktLiteral` before it is loaded.
898
-
899
- ### Full text: SPARQL's own functions, over a character-gram index
900
-
901
- `CONTAINS`, `STRSTARTS`, `STRENDS` and `REGEX` (SPARQL 1.1 §17.4.3) are
902
- the way to search text. There is no `text:query`-style extension and you
903
- do not ask for the index: where the query shape allows it, the planner
904
- uses the **LGI1** literal index the packer writes beside each block.
905
-
906
- LGI1 holds character 3-grams of the case-folded lexical form, and it is a
907
- CANDIDATE FILTER rather than a decider — it answers a superset and the
908
- engine re-evaluates your original `FILTER` on those rows, so the answer
909
- is exactly the answer a scan gives. Tokens would not do: `CONTAINS` is a
910
- substring test, and "underwater" contains "water" without being the
911
- token "water".
912
-
913
- Measured 2026-09-05, full skosdex corpus (7,315,251 quads, 3,286 blocks,
914
- 204 named graphs, 1.0 GB), through a store handle on plain `node`:
915
-
916
- | search | rows | time |
917
- |---|---|---|
918
- | `water` | 5 | 645 ms |
919
- | `glacier` | 0 | 670 ms |
920
- | `bicycle` | 2 | 617 ms |
921
-
922
- A miss costs what a hit costs, because both are index lookups rather than
923
- scans. On a 45,806-row block the same search was about 180 ms of scan
924
- before the index and 1.8 ms after it.
925
-
926
- **It falls back to a scan**, silently and correctly, for: a needle under
927
- 3 characters, a variable needle, `REGEX`, `UCASE`, `!CONTAINS`,
928
- `CONTAINS` under `||`, a filter on a variable not bound in object
929
- position, and any block whose generation has no `.lgi1` sidecar. The
930
- index costs about 55% of the block bytes.
931
-
932
- ### Geometry: the GBI1 bounding-box index
933
-
934
- The same construction for `geof:`. Each `geo:wktLiteral` object's
935
- bounding box is indexed, and five of the six topological functions are
936
- filtered by it — `sfIntersects`, `sfWithin`, `sfContains`, `sfTouches`,
937
- `sfEquals`. Measured 74x to 94x against a scan, with a miss at 82x.
938
-
939
- **`sfDisjoint` is refused and falls back to a scan**, deliberately: it
940
- accepts exactly the rows a box can exclude, so a box test inverts and
941
- would drop answers. A non-overlapping pair of boxes proves geometries
942
- disjoint, but that saves work inside a scan rather than reducing the
943
- candidate set.
944
-
945
- ### Wire version 10: large literals, RDF 1.2 terms, zone maps
946
-
947
- A generation packed at wire version 10 (`--layout ibk5`) carries three
948
- things earlier versions do not. Everything below is read by the command
949
- and by `bin/store.mjs` without a flag; wire versions 9 and earlier are
950
- read exactly as before.
951
-
952
- **Out-of-line literals.** A literal whose lexical form is at most 65,536
953
- UTF-8 bytes is stored inside the block. A longer one is stored as one
954
- file beside it, `blob-<sha256 hex>.lit`, holding exactly those bytes; the
955
- block keeps only the byte length and the SHA-256, and the manifest blob
956
- table commits the file. The name is the content address, so the same
957
- literal in twenty blocks is one file. `factoidal inspect` prints how many
958
- such files a generation holds and their total size, and the plan a query
959
- produces lists them under `blobKeys` beside `keys` and `sidecarKeys` — a
960
- host reading a store itself must fetch all three. A blob that is missing
961
- or whose bytes hash differently REFUSES the query; it is never answered
962
- with a shortened literal.
963
-
964
- Above 4,294,967,295 bytes the packer refuses the literal and names its
965
- subject and predicate. That ceiling, and every other one, is stated in
966
- `docs/designissues/2026-09-05-wire-version-10-scale.md` section 2.
967
-
968
- **RDF 1.2 terms.** A triple term (`<<( :a :p :b )>>`) and a directional
969
- language literal (`"..."@ar--rtl`) are stored and read back. Wire version
970
- 9 refuses both.
971
-
972
- **Zone maps.** Each manifest entry carries the smallest and the largest
973
- subject key and object key of its block, truncated to 64 bytes. A query
974
- with a constant subject or object skips every block whose range cannot
975
- hold it, from the manifest alone, with no block read. The plan reports
976
- how many entries were dropped that way as `zoneExcluded`. Selectivity
977
- depends on the source order: a subject-grouped or graph-grouped file
978
- gives disjoint ranges per block, a shuffled one gives overlapping ranges
979
- and a scan, which is correct and no worse than wire version 9.
980
-
981
- ## Limits (deliberate, documented)
982
-
983
- - **In-memory only.** ~1.2 KB RAM per quad (measured); 1M quads ≈
984
- 1.2 GB. No streaming parse yet — inputs are whole strings.
985
- - Lenient Turtle parsing: `parse()` cannot yet reject syntax errors
986
- (bad input can yield an empty dataset).
987
- - No *write* persistence in the npm build (SPARQL Update stays
988
- in-memory; durable UPDATE against a COTTAS store on disk is
989
- native-only today). *Reading* a COTTAS artifact's bytes is available
990
- in-process via `openCottas`/`queryCottas`/`closeCottas`/`toCottas`
991
- (both engines, both npm and browser entries) — see "The db API"
992
- above.
993
- - Bundle sizes (measured 2026-07-04): JS engine 554 KB, npm entry
994
- 461 KB, Wasm 43 KB loader + 1.3 MB assets. The Wasm entry trades
995
- startup cost for throughput.
996
-
997
- ## Testing
998
-
999
- `npm test` — unit suite covering the parse/query/serialize surface,
1000
- the RDF/JS contract, canonicalization stability under blank-node
1001
- renaming, and JS↔Wasm byte-parity. The engine underneath is gated on
1002
- every landing by the full W3C suites, a cross-backend parity harness,
1003
- and comparison probes against Apache Jena ARQ — live scores with
1004
- dates and commit links at
1005
- [danbri.github.io/factoidal/test-results](https://danbri.github.io/factoidal/test-results/).
127
+ - Fuseki: read-only administration API, one dataset per process, no
128
+ configuration file, no authentication, no JSON-LD from Graph Store GET
129
+ (https://github.com/danbri/factoidal/issues/829 to 834).
130
+ - Large result sets can exceed Node's default stack; use
131
+ `node --stack-size=4000` or Deno.
132
+ - The API can change before 1.0.
1006
133
 
1007
- ## Provenance & license
134
+ ## Licence
1008
135
 
1009
- Built from [danbri/factoidal](https://github.com/danbri/factoidal)
1010
- (Apache-2.0). This package's JavaScript is a thin consumer layer (API
1011
- shaping + RDF/JS conversion) containing no RDF/SPARQL semantics of
1012
- its own — the semantics are F\*-extracted.
136
+ Apache-2.0. See `LICENSE` and `NOTICE`.