@factoidal/core 0.4.0 → 0.5.1

package/CHANGELOG.md

@@ -1,5 +1,144 @@
 # Changelog
 
+## 0.5.1 — 2026-09-04
+
+**A store can be opened once and queried many times.** `storeQuery` is
+stateless: it re-reads, re-verifies and re-decodes the block on every
+call, so a chat bot or a server paid the full cost for every question.
+The new handle pays it once.
+
+Measured on a 141-graph store, `skos:prefLabel` block of 5,571,302 bytes
+and 45,806 rows, a DIFFERENT search string every query, load 10:
+
+| | stateless | handle |
+|---|---|---|
+| first query (open + query) | 1,376 ms | 1,363 ms |
+| second query, different string | 1,376 ms | **95 ms** |
+| tenth query, all different | 1,382 ms | 103 ms |
+| ten queries, total | 13,962 ms | **2,277 ms** |
+
+```js
+import { openStoreHandle } from '@factoidal/core/store'
+import { loadEngine } from '@factoidal/core/engine'
+const engine = await loadEngine()
+const store  = await openStoreHandle(engine, '/path/to/store')
+const a = store.query('PREFIX skos: … SELECT … ')   //  95 ms
+const b = store.query('PREFIX skos: … SELECT … ')   //  95 ms
+store.close()
+```
+
+**What it buys**: the per-query SHA-256 verification, block decode,
+dataset build and index build all happen once. **What it does not buy**:
+`CONTAINS` still scans every retained row, so cost stays proportional to
+row count. This is not a search fix. There is still no inverted index.
+
+⚠️ **Memory.** A held-open handle is about 170 MiB resident, of which
+roughly 94 MiB is the decoded form of that 5.5 MB packed block — about
+17 times the packed size. Query evaluation peaks at 346 MiB. A process
+holding several stores should watch this; `storeHandleList` reports each
+handle's retained bytes and rows.
+
+**For a long-lived process**: several stores may be open at once, keyed
+independently. `storeOpen` REFUSES at its cap and never evicts another
+caller's handle — an eviction policy is a host decision and is not
+implemented. The WebAssembly module is single-threaded, so a host must
+queue overlapping calls; use one module instance per worker thread for
+concurrency. A server would still need a re-open path when a generation
+is replaced on disk, and delta-log overlay support: this path serves the
+manifest's committed artifacts only.
+
+Handle answers are gated against the stateless path by comparing ROWS,
+not row counts, in `tools/wasm-store-query-smoke.sh` and
+`Wasm/native-smoke.sh` (85 pass, 0 fail, out of 85; was 79).
+
+## 0.5.0 — 2026-09-04
+
+**Queries against a persisted store are about six times faster.** Measured
+end to end the way a caller runs them — process start, engine load,
+digest verification, block decode and scan — on a 141-graph store with a
+5,571,302-byte `skos:prefLabel` block of 45,806 rows, at the same machine
+load:
+
+| query | 0.4.0 | 0.5.0 |
+|---|---|---|
+| `CONTAINS` over labels, `LIMIT 8` | 12.03 s | **2.06 s** |
+| the same for a second word | 12.18 s | **2.04 s** |
+| the same for a third | 11.96 s | **2.02 s** |
+
+Two causes, both fixed.
+
+- **A quadratic byte copy in SHA-256.** `Crypto.processBlocks256` copied
+  the whole remaining message once per 64-byte block, so verifying an
+  artifact was quadratic in its size. Fitting `t = c*n^k` to the
+  admission step gave k = 2.08 before and **k = 0.98 after**. On the
+  three blocks of that store, admission went 1,179 / 2,150 / 11,069 ms to
+  303 / 413 / 872 ms. Decode and evaluation were linear throughout.
+- **`LIMIT` was not pushed down through `GRAPH`.** A `LIMIT 8` cost what
+  a full count cost, because `GRAPH ?g { ... }` fell through to the
+  reference evaluator over the whole materialised dataset. The push-down
+  now takes one `GRAPH` layer with a constant IRI or a variable.
+  `ORDER BY`, `OFFSET`, `DISTINCT`, `REDUCED`, `GROUP BY`, `HAVING`,
+  `VALUES` and aggregates still reject, two of them pinned by theorems.
+- A `RangeError: Maximum call stack size exceeded` on `SELECT ... LIMIT 8`
+  is gone with it, because the query no longer materialises 45,806 rows
+  to return eight.
+
+**A correctness fix found while measuring.** The pre-existing bare-BGP
+`LIMIT` push-down could answer SHORT: a repeated variable (`?x ?p ?x`) or
+an RDF-star triple term let the backend stop early on rows the match then
+rejected. It now refuses both shapes.
+
+**New exports.** `@factoidal/core/store`, `/pack` and `/engine`. A caller
+can drive the store in process instead of spawning the command:
+
+```js
+import { openStore, queryStore } from '@factoidal/core/store'
+import { loadEngine } from '@factoidal/core/engine'
+const engine = await loadEngine()
+const store  = openStore('/path/to/store', null)
+const { result } = queryStore(engine, store, 'PREFIX skos: ... SELECT ...')
+```
+
+**The engine carries a day of OWL work**: OWL RL 1,181 pass, 266 fail
+(out of 1,457) and OWL DL about 1,326 pass, 121 fail (out of 1,457), both
+against a conclusion check corrected to require one functional blank-node
+mapping (RDF 1.1 Semantics §1.5 and the interpolation lemma). A false
+clash was removed — the materialiser had minted one existential witness
+for several obligations, so the engine denied three consistent
+ontologies; ConsistencyTest went 758 pass, 4 fail to 761 pass, 1 fail.
+
+**Named graphs now pack at scale.** The IBK4 quad path read the whole
+source file and a 553 MB, 194-graph corpus could not be packed at all. It
+streams now, and a quadratic term that only named graphs paid — a hash
+map copied per quad in `addQuadFast` — is gone. Peak memory per source
+byte fell from 37 and 20 to between 7.4 and 11.4; a 316,816,934-byte,
+194-graph input that used to fail now packs in 750 s at 2.34 GB. Byte
+identity with the previous packer holds by theorem, not only by diff.
+
+**Documentation corrected.** The GeoSPARQL section named functions that
+do not exist. Six topological functions are implemented — `geof:sfEquals`,
+`sfDisjoint`, `sfIntersects`, `sfTouches`, `sfWithin`, `sfContains` — and
+they work against a persisted store, verified. There is no
+`geof:distance`, `buffer`, `envelope`, `boundary`, `convexHull`, no
+`relate` with a DE-9IM matrix, no CRS handling beyond the WKT literal and
+no GML. Full text is SPARQL 1.1's own `CONTAINS` / `STRSTARTS` / `REGEX`,
+evaluated per row after a block decodes: **there is no inverted index**.
+
+Known limits, measured:
+
+- A query is still O(rows) per search string, and nothing is retained
+  between queries: `storeQuery` re-reads, re-verifies and re-decodes the
+  block every call. A store handle that decodes once is the next step.
+- `ORDER BY ... LIMIT n` still overflows the call stack above about
+  14,576 materialised rows.
+  https://github.com/danbri/factoidal/issues/653
+- `update` and `compact` still exit 3.
+  https://github.com/danbri/factoidal/issues/641
+- A query plan is refused above 64 artifacts, 8,388,608 blob bytes or
+  100,000 rows. https://github.com/danbri/factoidal/issues/648
+- The two SHA-256 folds are checked equal by the FIPS 180-4 build-time
+  guards and the HACL* differential, not proved.
+
 ## 0.4.0 — 2026-09-04
 
 The package builds a store of its own. `pack` and `activate` join

package/README.md

@@ -488,6 +488,94 @@ store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
 including start-up: 220 ms through this command, 33 ms through the
 native `l4block-id-v3-query`.
 
+### Holding a store open: verify once, answer many
+
+`queryStore` is stateless. Every call transfers the artifacts again, and
+the engine hashes, decodes and indexes them again. A process that answers
+many questions against one generation — a chat bot, an MCP server, a
+SPARQL endpoint — pays all of that for every question, and none of it
+depends on the question.
+
+A **store handle** holds that work. `openStoreHandle` verifies each
+artifact against the SHA-256 the manifest commits, decodes each block and
+indexes the rows, once. `handle.query()` then answers from what it
+retained. Several stores can be open at the same time.
+
+```js
+import { loadEngine } from '@factoidal/core/bin/engine.mjs'
+import { openStore, openStoreHandle, listStoreHandles } from '@factoidal/core/store'
+
+const engine = await loadEngine()
+
+// Two stores, open at once, held for the life of the process.
+const skos = openStoreHandle(engine, openStore('./skos-store'))
+const docs = openStoreHandle(engine, openStore('./docs-store'))
+
+console.log(listStoreHandles(engine))
+// { ok: true, handles: [ {handle:'s1', bytes:…, rows:…}, {handle:'s2', …} ],
+//   bytes: …, rows: …, handleCap: 8, bytesCap: 67108864 }
+
+const PREFIX = 'PREFIX skos: <http://www.w3.org/2004/02/skos/core#>'
+
+function labelsMatching (handle, needle) {
+  const answer = handle.query(`${PREFIX}
+    SELECT ?c ?l WHERE {
+      GRAPH ?g { ?c skos:prefLabel ?l }
+      FILTER(CONTAINS(LCASE(STR(?l)), "${needle.toLowerCase()}"))
+    } LIMIT 10`)
+  return answer.srj.results.bindings
+}
+
+// Many questions, each a new search string. None of them re-reads a block.
+for (const needle of ['water', 'forest', 'railway', 'volcano']) {
+  console.log(needle, labelsMatching(skos, needle).length)
+}
+console.log(docs.query(`${PREFIX} SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`))
+
+skos.close()
+docs.close()
+```
+
+`openStoreHandle(engine, store, options)` takes:
+
+| option | effect |
+|---|---|
+| (none) | open every artifact the manifest declares |
+| `{ sparql }` | open only the artifacts that query's plan names |
+| `{ keys }` | open exactly these artifact keys |
+
+`queryStoreHandle(engine, handle, sparql)`, `closeStoreHandle(engine,
+handle)` and `listStoreHandles(engine)` are the same operations for a
+caller that holds only the handle id.
+
+**Measured 2026-09-04**, macOS arm64, Node 22, the 141-graph SKOS store,
+`skos:prefLabel` block of 5,571,302 bytes and 45,806 rows, one
+`CONTAINS` query per row, a DIFFERENT search string every time:
+
+| | stateless `queryStore` | handle |
+|---|---|---|
+| first query (open + query) | 1376 ms | 1363 ms |
+| second query, different search string | 1376 ms | 95 ms |
+| tenth query, all different | 1382 ms | 103 ms |
+| ten queries, total | 13962 ms | 2277 ms |
+
+**What a handle buys and what it does not.** It removes the per-query
+digest check, block decode and index build. It does NOT make search
+sub-linear: `CONTAINS` still scans every retained row, so a query still
+costs time proportional to the row count. A text index is separate work.
+
+**Residency.** Retaining a decoded block costs memory. Measured on the
+same store: 76 MiB resident with the engine loaded and no handle, 170 MiB
+with the handle open — about 94 MiB for a 5.5 MB packed block, and
+evaluation peaks higher again (346 MiB during the queries above). The
+caps are on ARTIFACT bytes, which is what the manifest declares: 8 open
+handles, and 67108864 retained artifact bytes across all of them. A cap
+is a refusal naming the cap; no handle is ever evicted to make room for
+another. `listStoreHandles` is how a server sees its own residency.
+
+**One call at a time.** The WebAssembly module is single-threaded. Two
+`query()` calls cannot overlap; a server queues them.
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to
@@ -653,14 +741,42 @@ value transforms:
   API; the `_*` functions (e.g. `_deltaLogCorruptLastForTest`) are
   test-only and intentionally left untyped.
 
-### GeoSPARQL
+### GeoSPARQL — six topological functions
+
+The `geof:` functions below are built into the SPARQL engine and need no
+import. They work through `query()` / `fn.query()` AND against a
+persisted store through `factoidal query`, because both paths evaluate
+in the same environment.
+
+    geof:sfEquals   geof:sfDisjoint   geof:sfIntersects
+    geof:sfTouches  geof:sfWithin     geof:sfContains
+
+```sparql
+PREFIX geof: <http://www.opengis.net/def/function/geosparql/>
+PREFIX geo:  <http://www.opengis.net/ont/geosparql#>
+SELECT ?a WHERE {
+  ?a :footprint ?w
+  FILTER(geof:sfWithin(?w, "POLYGON((0 0,0 2,2 2,2 0,0 0))"^^geo:wktLiteral))
+}
+```
 
-There is no separate GeoSPARQL function: the `geof:` functions
-(`geof:sfWithin`, `geof:sfDisjoint`, `geof:distance`, `geof:envelope`,
-…) are built into the SPARQL engine and work through ordinary
-`query()` / `fn.query()` — e.g.
-`query(data, 'PREFIX geof: <http://www.opengis.net/def/function/geosparql/> SELECT ?a ?b WHERE { … FILTER(geof:sfWithin(?a, ?b)) }')`.
-Nothing to import; nothing "missing".
+**What is NOT there**, stated so nobody plans around it: no
+`geof:distance`, `geof:buffer`, `geof:envelope`, `geof:boundary`,
+`geof:convexHull` or any other non-topological measure; no
+`geof:relate` with a DE-9IM matrix; no coordinate reference system
+handling beyond what the WKT literal carries; no GML literals. Geometry
+comes from a WKT parser, so a shapefile, GeoJSON or GML source must be
+converted to `geo:wktLiteral` before it is loaded.
+
+### Full text: SPARQL's own functions, no index
+
+`CONTAINS`, `STRSTARTS`, `STRENDS` and `REGEX` (SPARQL 1.1 §17.4.3) are
+implemented and are the way to search text. They are evaluated per row
+after a block is decoded — **there is no inverted index and no
+`text:query`-style extension**. Measured 2026-09-04: a `CONTAINS` over
+45,806 `skos:prefLabel` values in one block answers in about 6 seconds.
+That is fine for a vocabulary and will not scale to a large literal
+corpus.
 
 ## Limits (deliberate, documented)
 

package/bin/store.mjs

@@ -236,3 +236,160 @@ export function turtleOfNQuads (engine, nquads) {
     throw asStoreError(error)
   }
 }
+
+// ---------------------------------------------------------------- handles
+//
+// `queryStore` above is stateless: every call transfers the artifacts again,
+// and the engine hashes, decodes and indexes them again. A process that
+// answers many questions against one generation — a chat bot, an MCP server,
+// a SPARQL endpoint — pays that per question, and none of it depends on the
+// question.
+//
+// A handle holds the verified, decoded, indexed form inside the engine.
+// `storeOpen` verifies every artifact against the SHA-256 the manifest
+// commits, once; `storeHandleQuery` then answers from what it retained. The
+// engine refuses a query whose plan needs a block the handle does not retain,
+// so a handle answer is never a partial generation.
+//
+// What a handle does NOT change: a query still scans every retained row, so
+// its cost is still proportional to the row count. A handle removes the
+// per-query decode; it is not a text index.
+//
+// The stateless path above stays exactly as it was. A one-shot CLI query
+// should not pay to build a handle it will drop.
+
+/** The artifact keys a manifest declares, in manifest order. */
+function manifestKeys (engine, store) {
+  return inspectManifest(engine, store).entries.map((entry) => entry.key)
+}
+
+/** Read the named artifacts and concatenate them into one region. */
+function regionOf (store, keys) {
+  const chunks = keys.map((key) => readWhole(joinPath(store.generationDir, key)))
+  let total = 0
+  for (const chunk of chunks) total += chunk.length
+  const blob = new Uint8Array(total)
+  const artifacts = []
+  let offset = 0
+  for (let index = 0; index < chunks.length; index += 1) {
+    blob.set(chunks[index], offset)
+    artifacts.push({ key: keys[index], offset, len: chunks[index].length })
+    offset += chunks[index].length
+  }
+  return { blob, artifacts }
+}
+
+/**
+ * An open store held inside the engine. Hold this object for as long as the
+ * process answers questions about the generation, then `close()` it.
+ *
+ * The wasm module is single-threaded: two `query()` calls cannot overlap. A
+ * server awaits one before it starts the next.
+ */
+export class StoreHandle {
+  constructor (engine, store, envelope) {
+    this.engine = engine
+    this.store = store
+    this.handle = envelope.handle
+    this.identity = envelope.identity
+    this.layout = envelope.layout
+    this.wireVersion = envelope.wireVersion
+    this.artifacts = envelope.artifacts
+    this.bytes = envelope.bytes
+    this.rows = envelope.rows
+    this.closed = false
+  }
+
+  /**
+   * One SPARQL query against the retained blocks. The envelope is the one
+   * `queryStore` answers with — `kind`, `srj`/`boolean`/`nquads`, plus
+   * `shards` and `mode` — so a caller handles both paths alike.
+   */
+  query (sparql) {
+    if (this.closed) {
+      throw new StoreOperationError(`store handle ${this.handle} is closed`)
+    }
+    try {
+      return this.engine.call('storeHandleQuery', [this.handle, sparql])
+    } catch (error) {
+      throw asStoreError(error)
+    }
+  }
+
+  /** Drop the handle and everything it retained. Idempotent on this object. */
+  close () {
+    if (this.closed) return
+    this.closed = true
+    try {
+      this.engine.call('storeHandleClose', [this.handle])
+    } catch (error) {
+      throw asStoreError(error)
+    }
+  }
+}
+
+/**
+ * `storeOpen` — verify, decode and index a generation once.
+ *
+ * With no options every artifact the manifest declares is opened, so any
+ * query the generation can answer works. With `sparql` only the artifacts
+ * that query's plan names are opened, which is what a process asking one
+ * query shape wants. With `keys` exactly those artifacts are opened.
+ *
+ * @param {object} engine the loaded engine (bin/engine.mjs)
+ * @param {object} store  the result of `openStore`
+ * @param {{keys?: string[], sparql?: string}} options
+ * @returns {StoreHandle}
+ */
+export function openStoreHandle (engine, store, options = {}) {
+  let keys
+  if (Array.isArray(options.keys)) {
+    keys = options.keys
+  } else if (typeof options.sparql === 'string') {
+    keys = planQuery(engine, store, options.sparql).keys
+  } else {
+    keys = manifestKeys(engine, store)
+  }
+  const { blob, artifacts } = regionOf(store, keys)
+  let envelope
+  try {
+    envelope = engine.callBlobIO('storeOpen',
+      [store.manifestHex, JSON.stringify(artifacts)], blob).envelope
+  } catch (error) {
+    throw asStoreError(error)
+  }
+  return new StoreHandle(engine, store, envelope)
+}
+
+/** `storeHandleQuery` — the loose form, for a caller holding only the id. */
+export function queryStoreHandle (engine, handle, sparql) {
+  const id = typeof handle === 'string' ? handle : handle.handle
+  try {
+    return engine.call('storeHandleQuery', [id, sparql])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}
+
+/** `storeHandleClose` — the loose form. */
+export function closeStoreHandle (engine, handle) {
+  if (handle instanceof StoreHandle) return handle.close()
+  try {
+    return engine.call('storeHandleClose', [handle])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}
+
+/**
+ * `storeHandleList` — what this process holds open, with the retained bytes
+ * and rows and the two residency caps. A server that cannot see its own
+ * residency cannot be operated.
+ */
+export function listStoreHandles (engine) {
+  try {
+    return engine.call('storeHandleList', [])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}

package/l4-assets/l4factoidal.js

@@ -37,7 +37,7 @@
 // bytes change.
 
 // Stamped by formal/lean4/Wasm/build-wasm.sh step 9 -- do not hand-edit.
-const WASM_VERSION = "125d391e0ccc";
+const WASM_VERSION = "ffbb3a035705";
 
 import createModule from './l4factoidal.mjs';
 

package/l4-assets/version.json

@@ -2,13 +2,13 @@
   "engine": "lean4",
   "note": "The Lean 4 (L4Factoidal) engine's own build provenance and claims -- distinct from ../version.json, which is the F* engine's. Do not merge the two: they come from different toolchains, different git SHAs in general, and carry different proof obligations (issue #618).",
   "version": "0.1.0",
-  "gitSha": "6b9f12003805f5bc1136949b12dc6b1f41885d00",
-  "builtAt": "2026-09-03T23:23:57+00:00",
+  "gitSha": "6613042b8acf3be78541f862f3ad0caf057cbab3",
+  "builtAt": "2026-09-04T20:56:42+00:00",
   "leanToolchain": "leanprover/lean4:v4.33.1",
   "emscripten": "emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 6.0.8-git",
   "abiVersion": "1",
-  "wasmSha256": "125d391e0ccc7be1a673d748d40b8443fd66bb5f0a6e3736223fd6ed19307d03",
-  "wasmBytes": 5296153,
+  "wasmSha256": "ffbb3a0357051f78c2e9135ae629fc04a46037d540501f7fadd3d3e0e96c96b4",
+  "wasmBytes": 5412682,
   "claims": {
     "source": "formal/lean4 (L4Factoidal): no sorry, no user axioms, no native_decide; W3C behaviour pinned by build-time #guard",
     "suitesAtBuildSha": "see docs/test-results and formal/lean4/PORT_NOTES.md at gitSha"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@factoidal/core",
-  "version": "0.4.0",
+  "version": "0.5.1",
   "description": "Factoidal: a linked information system with graph data and the Web at its heart - RDF parsing, SPARQL 1.1 query, canonicalization and entailment for Node and the browser (JS + Wasm), grounded in F* and Lean 4. Parser and algebra spec verified in F*; on-disk backend has unverified OCaml-side optimization layers being migrated back to F*.",
   "keywords": [
     "sparql",
@@ -79,6 +79,15 @@
     "./sample-store": {
       "types": "./sample-store.d.ts",
       "default": "./sample-store.mjs"
+    },
+    "./store": {
+      "default": "./bin/store.mjs"
+    },
+    "./pack": {
+      "default": "./bin/pack.mjs"
+    },
+    "./engine": {
+      "default": "./bin/engine.mjs"
     }
   },
   "files": [

package/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "0.4.0",
+  "version": "0.5.1",
   "gitSha": "49f8ca4d70bf57c12fea445611e2b01b067bf6ba",
   "builtAt": "2026-08-26T21:45:40Z",
   "claims": {