@factoidal/core 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -1,5 +1,93 @@
 # Changelog
 
+## 0.5.0 — 2026-09-04
+
+**Queries against a persisted store are about six times faster.** Measured
+end to end the way a caller runs them — process start, engine load,
+digest verification, block decode and scan — on a 141-graph store with a
+5,571,302-byte `skos:prefLabel` block of 45,806 rows, at the same machine
+load:
+
+| query | 0.4.0 | 0.5.0 |
+|---|---|---|
+| `CONTAINS` over labels, `LIMIT 8` | 12.03 s | **2.06 s** |
+| the same for a second word | 12.18 s | **2.04 s** |
+| the same for a third | 11.96 s | **2.02 s** |
+
+Two causes, both fixed.
+
+- **A quadratic byte copy in SHA-256.** `Crypto.processBlocks256` copied
+  the whole remaining message once per 64-byte block, so verifying an
+  artifact was quadratic in its size. Fitting `t = c*n^k` to the
+  admission step gave k = 2.08 before and **k = 0.98 after**. On the
+  three blocks of that store, admission went 1,179 / 2,150 / 11,069 ms to
+  303 / 413 / 872 ms. Decode and evaluation were linear throughout.
+- **`LIMIT` was not pushed down through `GRAPH`.** A `LIMIT 8` cost what
+  a full count cost, because `GRAPH ?g { ... }` fell through to the
+  reference evaluator over the whole materialised dataset. The push-down
+  now takes one `GRAPH` layer with a constant IRI or a variable.
+  `ORDER BY`, `OFFSET`, `DISTINCT`, `REDUCED`, `GROUP BY`, `HAVING`,
+  `VALUES` and aggregates still reject, two of them pinned by theorems.
+- A `RangeError: Maximum call stack size exceeded` on `SELECT ... LIMIT 8`
+  is gone with it, because the query no longer materialises 45,806 rows
+  to return eight.
+
+**A correctness fix found while measuring.** The pre-existing bare-BGP
+`LIMIT` push-down could answer SHORT: a repeated variable (`?x ?p ?x`) or
+an RDF-star triple term let the backend stop early on rows the match then
+rejected. It now refuses both shapes.
+
+**New exports.** `@factoidal/core/store`, `/pack` and `/engine`. A caller
+can drive the store in process instead of spawning the command:
+
+```js
+import { openStore, queryStore } from '@factoidal/core/store'
+import { loadEngine } from '@factoidal/core/engine'
+const engine = await loadEngine()
+const store  = openStore('/path/to/store', null)
+const { result } = queryStore(engine, store, 'PREFIX skos: ... SELECT ...')
+```
+
+**The engine carries a day of OWL work**: OWL RL 1,181 pass, 266 fail
+(out of 1,457) and OWL DL about 1,326 pass, 121 fail (out of 1,457), both
+against a conclusion check corrected to require one functional blank-node
+mapping (RDF 1.1 Semantics §1.5 and the interpolation lemma). A false
+clash was removed — the materialiser had minted one existential witness
+for several obligations, so the engine denied three consistent
+ontologies; ConsistencyTest went 758 pass, 4 fail to 761 pass, 1 fail.
+
+**Named graphs now pack at scale.** The IBK4 quad path read the whole
+source file and a 553 MB, 194-graph corpus could not be packed at all. It
+streams now, and a quadratic term that only named graphs paid — a hash
+map copied per quad in `addQuadFast` — is gone. Peak memory per source
+byte fell from 37 and 20 to between 7.4 and 11.4; a 316,816,934-byte,
+194-graph input that used to fail now packs in 750 s at 2.34 GB. Byte
+identity with the previous packer holds by theorem, not only by diff.
+
+**Documentation corrected.** The GeoSPARQL section named functions that
+do not exist. Six topological functions are implemented — `geof:sfEquals`,
+`sfDisjoint`, `sfIntersects`, `sfTouches`, `sfWithin`, `sfContains` — and
+they work against a persisted store, verified. There is no
+`geof:distance`, `buffer`, `envelope`, `boundary`, `convexHull`, no
+`relate` with a DE-9IM matrix, no CRS handling beyond the WKT literal and
+no GML. Full text is SPARQL 1.1's own `CONTAINS` / `STRSTARTS` / `REGEX`,
+evaluated per row after a block decodes: **there is no inverted index**.
+
+Known limits, measured:
+
+- A query is still O(rows) per search string, and nothing is retained
+  between queries: `storeQuery` re-reads, re-verifies and re-decodes the
+  block every call. A store handle that decodes once is the next step.
+- `ORDER BY ... LIMIT n` still overflows the call stack above about
+  14,576 materialised rows.
+  https://github.com/danbri/factoidal/issues/653
+- `update` and `compact` still exit 3.
+  https://github.com/danbri/factoidal/issues/641
+- A query plan is refused above 64 artifacts, 8,388,608 blob bytes or
+  100,000 rows. https://github.com/danbri/factoidal/issues/648
+- The two SHA-256 folds are checked equal by the FIPS 180-4 build-time
+  guards and the HACL* differential, not proved.
+
 ## 0.4.0 — 2026-09-04
 
 The package builds a store of its own. `pack` and `activate` join

package/README.md

@@ -653,14 +653,42 @@ value transforms:
   API; the `_*` functions (e.g. `_deltaLogCorruptLastForTest`) are
   test-only and intentionally left untyped.
 
-### GeoSPARQL
-
-There is no separate GeoSPARQL function: the `geof:` functions
-(`geof:sfWithin`, `geof:sfDisjoint`, `geof:distance`, `geof:envelope`,
-…) are built into the SPARQL engine and work through ordinary
-`query()` / `fn.query()` — e.g.
-`query(data, 'PREFIX geof: <http://www.opengis.net/def/function/geosparql/> SELECT ?a ?b WHERE { … FILTER(geof:sfWithin(?a, ?b)) }')`.
-Nothing to import; nothing "missing".
+### GeoSPARQL — six topological functions
+
+The `geof:` functions below are built into the SPARQL engine and need no
+import. They work through `query()` / `fn.query()` AND against a
+persisted store through `factoidal query`, because both paths evaluate
+in the same environment.
+
+    geof:sfEquals   geof:sfDisjoint   geof:sfIntersects
+    geof:sfTouches  geof:sfWithin     geof:sfContains
+
+```sparql
+PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
+PREFIX geo:  <http://www.opengis.net/ont/geosparql#>
+SELECT ?a WHERE {
+  ?a :footprint ?w
+  FILTER(geof:sfWithin(?w, "POLYGON((0 0,0 2,2 2,2 0,0 0))"^^geo:wktLiteral))
+}
+```
+
+**What is NOT there**, stated so nobody plans around it: no
+`geof:distance`, `geof:buffer`, `geof:envelope`, `geof:boundary`,
+`geof:convexHull` or any other non-topological measure; no
+`geof:relate` with a DE-9IM matrix; no coordinate reference system
+handling beyond what the WKT literal carries; no GML literals. Geometry
+comes from a WKT parser, so a shapefile, GeoJSON or GML source must be
+converted to `geo:wktLiteral` before it is loaded.
+
+### Full text: SPARQL's own functions, no index
+
+`CONTAINS`, `STRSTARTS`, `STRENDS` and `REGEX` (SPARQL 1.1 §17.4.3) are
+implemented and are the way to search text. They are evaluated per row
+after a block is decoded — **there is no inverted index and no
+`text:query`-style extension**. Measured 2026-09-04: a `CONTAINS` over
+45,806 `skos:prefLabel` values in one block answers in about 6 seconds.
+That is fine for a vocabulary and will not scale to a large literal
+corpus.
 
 ## Limits (deliberate, documented)
 

package/l4-assets/l4factoidal.js

@@ -37,7 +37,7 @@
 // bytes change.
 
 // Stamped by formal/lean4/Wasm/build-wasm.sh step 9 -- do not hand-edit.
-const WASM_VERSION = "125d391e0ccc";
+const WASM_VERSION = "20551e4329bb";
 
 import createModule from './l4factoidal.mjs';
 

package/l4-assets/version.json

@@ -2,13 +2,13 @@
   "engine": "lean4",
   "note": "The Lean 4 (L4Factoidal) engine's own build provenance and claims -- distinct from ../version.json, which is the F* engine's. Do not merge the two: they come from different toolchains, different git SHAs in general, and carry different proof obligations (issue #618).",
   "version": "0.1.0",
-  "gitSha": "6b9f12003805f5bc1136949b12dc6b1f41885d00",
-  "builtAt": "2026-09-03T23:23:57+00:00",
+  "gitSha": "4295b2917e700fb3ead2d74fad05c9d2f592b7d9",
+  "builtAt": "2026-09-04T20:28:04+00:00",
   "leanToolchain": "leanprover/lean4:v4.33.1",
   "emscripten": "emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 6.0.8-git",
   "abiVersion": "1",
-  "wasmSha256": "125d391e0ccc7be1a673d748d40b8443fd66bb5f0a6e3736223fd6ed19307d03",
-  "wasmBytes": 5296153,
+  "wasmSha256": "20551e4329bb3fa363e4c64b9f9d62cc1371a2c563161d938226c0d2e102c87a",
+  "wasmBytes": 5384150,
   "claims": {
     "source": "formal/lean4 (L4Factoidal): no sorry, no user axioms, no native_decide; W3C behaviour pinned by build-time #guard",
     "suitesAtBuildSha": "see docs/test-results and formal/lean4/PORT_NOTES.md at gitSha"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@factoidal/core",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Factoidal: a linked information system with graph data and the Web at its heart - RDF parsing, SPARQL 1.1 query, canonicalization and entailment for Node and the browser (JS + Wasm), grounded in F* and Lean 4. Parser and algebra spec verified in F*; on-disk backend has unverified OCaml-side optimization layers being migrated back to F*.",
   "keywords": [
     "sparql",
@@ -79,6 +79,15 @@
     "./sample-store": {
       "types": "./sample-store.d.ts",
       "default": "./sample-store.mjs"
+    },
+    "./store": {
+      "default": "./bin/store.mjs"
+    },
+    "./pack": {
+      "default": "./bin/pack.mjs"
+    },
+    "./engine": {
+      "default": "./bin/engine.mjs"
     }
   },
   "files": [

package/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "0.4.0",
+  "version": "0.5.0",
   "gitSha": "49f8ca4d70bf57c12fea445611e2b01b067bf6ba",
   "builtAt": "2026-08-26T21:45:40Z",
   "claims": {