@factoidal/core 0.5.0 → 0.5.1

package/CHANGELOG.md

@@ -1,5 +1,56 @@
 # Changelog
 
+## 0.5.1 — 2026-09-04
+
+**A store can be opened once and queried many times.** `storeQuery` is
+stateless: it re-reads, re-verifies and re-decodes the block on every
+call, so a chat bot or a server paid the full cost for every question.
+The new handle pays it once.
+
+Measured on a 141-graph store, `skos:prefLabel` block of 5,571,302 bytes
+and 45,806 rows, a DIFFERENT search string every query, load 10:
+
+| | stateless | handle |
+|---|---|---|
+| first query (open + query) | 1,376 ms | 1,363 ms |
+| second query, different string | 1,376 ms | **95 ms** |
+| tenth query, all different | 1,382 ms | 103 ms |
+| ten queries, total | 13,962 ms | **2,277 ms** |
+
+```js
+import { openStoreHandle } from '@factoidal/core/store'
+import { loadEngine } from '@factoidal/core/engine'
+const engine = await loadEngine()
+const store  = await openStoreHandle(engine, '/path/to/store')
+const a = store.query('PREFIX skos: … SELECT … ')   //  95 ms
+const b = store.query('PREFIX skos: … SELECT … ')   //  95 ms
+store.close()
+```
+
+**What it buys**: the per-query SHA-256 verification, block decode,
+dataset build and index build all happen once. **What it does not buy**:
+`CONTAINS` still scans every retained row, so cost stays proportional to
+row count. This is not a search fix. There is still no inverted index.
+
+⚠️ **Memory.** A held-open handle is about 170 MiB resident, of which
+roughly 94 MiB is the decoded form of that 5.5 MB packed block — about
+17 times the packed size. Query evaluation peaks at 346 MiB. A process
+holding several stores should watch this; `storeHandleList` reports each
+handle's retained bytes and rows.
+
+**For a long-lived process**: several stores may be open at once, keyed
+independently. `storeOpen` REFUSES at its cap and never evicts another
+caller's handle — an eviction policy is a host decision and is not
+implemented. The WebAssembly module is single-threaded, so a host must
+queue overlapping calls; use one module instance per worker thread for
+concurrency. A server would still need a re-open path when a generation
+is replaced on disk, and delta-log overlay support: this path serves the
+manifest's committed artifacts only.
+
+Handle answers are gated against the stateless path by comparing ROWS,
+not row counts, in `tools/wasm-store-query-smoke.sh` and
+`Wasm/native-smoke.sh` (85 pass, 0 fail, out of 85; was 79).
+
 ## 0.5.0 — 2026-09-04
 
 **Queries against a persisted store are about six times faster.** Measured

package/README.md

@@ -488,6 +488,94 @@ store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
 including start-up: 220 ms through this command, 33 ms through the
 native `l4block-id-v3-query`.
 
+### Holding a store open: verify once, answer many
+
+`queryStore` is stateless. Every call transfers the artifacts again, and
+the engine hashes, decodes and indexes them again. A process that answers
+many questions against one generation — a chat bot, an MCP server, a
+SPARQL endpoint — pays all of that for every question, and none of it
+depends on the question.
+
+A **store handle** holds that work. `openStoreHandle` verifies each
+artifact against the SHA-256 the manifest commits, decodes each block and
+indexes the rows, once. `handle.query()` then answers from what it
+retained. Several stores can be open at the same time.
+
+```js
+import { loadEngine } from '@factoidal/core/bin/engine.mjs'
+import { openStore, openStoreHandle, listStoreHandles } from '@factoidal/core/store'
+
+const engine = await loadEngine()
+
+// Two stores, open at once, held for the life of the process.
+const skos = openStoreHandle(engine, openStore('./skos-store'))
+const docs = openStoreHandle(engine, openStore('./docs-store'))
+
+console.log(listStoreHandles(engine))
+// { ok: true, handles: [ {handle:'s1', bytes:…, rows:…}, {handle:'s2', …} ],
+//   bytes: …, rows: …, handleCap: 8, bytesCap: 67108864 }
+
+const PREFIX = 'PREFIX skos: <http://www.w3.org/2004/02/skos/core#>'
+
+function labelsMatching (handle, needle) {
+  const answer = handle.query(`${PREFIX}
+    SELECT ?c ?l WHERE {
+      GRAPH ?g { ?c skos:prefLabel ?l }
+      FILTER(CONTAINS(LCASE(STR(?l)), "${needle.toLowerCase()}"))
+    } LIMIT 10`)
+  return answer.srj.results.bindings
+}
+
+// Many questions, each a new search string. None of them re-reads a block.
+for (const needle of ['water', 'forest', 'railway', 'volcano']) {
+  console.log(needle, labelsMatching(skos, needle).length)
+}
+console.log(docs.query(`${PREFIX} SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`))
+
+skos.close()
+docs.close()
+```
+
+`openStoreHandle(engine, store, options)` takes:
+
+| option | effect |
+|---|---|
+| (none) | open every artifact the manifest declares |
+| `{ sparql }` | open only the artifacts that query's plan names |
+| `{ keys }` | open exactly these artifact keys |
+
+`queryStoreHandle(engine, handle, sparql)`, `closeStoreHandle(engine,
+handle)` and `listStoreHandles(engine)` are the same operations for a
+caller that holds only the handle id.
+
+**Measured 2026-09-04**, macOS arm64, Node 22, the 141-graph SKOS store,
+`skos:prefLabel` block of 5,571,302 bytes and 45,806 rows, one
+`CONTAINS` query per row, a DIFFERENT search string every time:
+
+| | stateless `queryStore` | handle |
+|---|---|---|
+| first query (open + query) | 1376 ms | 1363 ms |
+| second query, different search string | 1376 ms | 95 ms |
+| tenth query, all different | 1382 ms | 103 ms |
+| ten queries, total | 13962 ms | 2277 ms |
+
+**What a handle buys and what it does not.** It removes the per-query
+digest check, block decode and index build. It does NOT make search
+sub-linear: `CONTAINS` still scans every retained row, so a query still
+costs time proportional to the row count. A text index is separate work.
+
+**Residency.** Retaining a decoded block costs memory. Measured on the
+same store: 76 MiB resident with the engine loaded and no handle, 170 MiB
+with the handle open — about 94 MiB for a 5.5 MB packed block, and
+evaluation peaks higher again (346 MiB during the queries above). The
+caps are on ARTIFACT bytes, which is what the manifest declares: 8 open
+handles, and 67108864 retained artifact bytes across all of them. A cap
+is a refusal naming the cap; no handle is ever evicted to make room for
+another. `listStoreHandles` is how a server sees its own residency.
+
+**One call at a time.** The WebAssembly module is single-threaded. Two
+`query()` calls cannot overlap; a server queues them.
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to

package/bin/store.mjs

@@ -236,3 +236,160 @@ export function turtleOfNQuads (engine, nquads) {
     throw asStoreError(error)
   }
 }
+
+// ---------------------------------------------------------------- handles
+//
+// `queryStore` above is stateless: every call transfers the artifacts again,
+// and the engine hashes, decodes and indexes them again. A process that
+// answers many questions against one generation — a chat bot, an MCP server,
+// a SPARQL endpoint — pays that per question, and none of it depends on the
+// question.
+//
+// A handle holds the verified, decoded, indexed form inside the engine.
+// `storeOpen` verifies every artifact against the SHA-256 the manifest
+// commits, once; `storeHandleQuery` then answers from what it retained. The
+// engine refuses a query whose plan needs a block the handle does not retain,
+// so a handle answer is never a partial generation.
+//
+// What a handle does NOT change: a query still scans every retained row, so
+// its cost is still proportional to the row count. A handle removes the
+// per-query decode; it is not a text index.
+//
+// The stateless path above stays exactly as it was. A one-shot CLI query
+// should not pay to build a handle it will drop.
+
+/** The artifact keys a manifest declares, in manifest order. */
+function manifestKeys (engine, store) {
+  return inspectManifest(engine, store).entries.map((entry) => entry.key)
+}
+
+/** Read the named artifacts and concatenate them into one region. */
+function regionOf (store, keys) {
+  const chunks = keys.map((key) => readWhole(joinPath(store.generationDir, key)))
+  let total = 0
+  for (const chunk of chunks) total += chunk.length
+  const blob = new Uint8Array(total)
+  const artifacts = []
+  let offset = 0
+  for (let index = 0; index < chunks.length; index += 1) {
+    blob.set(chunks[index], offset)
+    artifacts.push({ key: keys[index], offset, len: chunks[index].length })
+    offset += chunks[index].length
+  }
+  return { blob, artifacts }
+}
+
+/**
+ * An open store held inside the engine. Hold this object for as long as the
+ * process answers questions about the generation, then `close()` it.
+ *
+ * The wasm module is single-threaded: two `query()` calls cannot overlap. A
+ * server awaits one before it starts the next.
+ */
+export class StoreHandle {
+  constructor (engine, store, envelope) {
+    this.engine = engine
+    this.store = store
+    this.handle = envelope.handle
+    this.identity = envelope.identity
+    this.layout = envelope.layout
+    this.wireVersion = envelope.wireVersion
+    this.artifacts = envelope.artifacts
+    this.bytes = envelope.bytes
+    this.rows = envelope.rows
+    this.closed = false
+  }
+
+  /**
+   * One SPARQL query against the retained blocks. The envelope is the one
+   * `queryStore` answers with — `kind`, `srj`/`boolean`/`nquads`, plus
+   * `shards` and `mode` — so a caller handles both paths alike.
+   */
+  query (sparql) {
+    if (this.closed) {
+      throw new StoreOperationError(`store handle ${this.handle} is closed`)
+    }
+    try {
+      return this.engine.call('storeHandleQuery', [this.handle, sparql])
+    } catch (error) {
+      throw asStoreError(error)
+    }
+  }
+
+  /** Drop the handle and everything it retained. Idempotent on this object. */
+  close () {
+    if (this.closed) return
+    this.closed = true
+    try {
+      this.engine.call('storeHandleClose', [this.handle])
+    } catch (error) {
+      throw asStoreError(error)
+    }
+  }
+}
+
+/**
+ * `storeOpen` — verify, decode and index a generation once.
+ *
+ * With no options every artifact the manifest declares is opened, so any
+ * query the generation can answer works. With `sparql` only the artifacts
+ * that query's plan names are opened, which is what a process asking one
+ * query shape wants. With `keys` exactly those artifacts are opened.
+ *
+ * @param {object} engine the loaded engine (bin/engine.mjs)
+ * @param {object} store  the result of `openStore`
+ * @param {{keys?: string[], sparql?: string}} options
+ * @returns {StoreHandle}
+ */
+export function openStoreHandle (engine, store, options = {}) {
+  let keys
+  if (Array.isArray(options.keys)) {
+    keys = options.keys
+  } else if (typeof options.sparql === 'string') {
+    keys = planQuery(engine, store, options.sparql).keys
+  } else {
+    keys = manifestKeys(engine, store)
+  }
+  const { blob, artifacts } = regionOf(store, keys)
+  let envelope
+  try {
+    envelope = engine.callBlobIO('storeOpen',
+      [store.manifestHex, JSON.stringify(artifacts)], blob).envelope
+  } catch (error) {
+    throw asStoreError(error)
+  }
+  return new StoreHandle(engine, store, envelope)
+}
+
+/** `storeHandleQuery` — the loose form, for a caller holding only the id. */
+export function queryStoreHandle (engine, handle, sparql) {
+  const id = typeof handle === 'string' ? handle : handle.handle
+  try {
+    return engine.call('storeHandleQuery', [id, sparql])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}
+
+/** `storeHandleClose` — the loose form. */
+export function closeStoreHandle (engine, handle) {
+  if (handle instanceof StoreHandle) return handle.close()
+  try {
+    return engine.call('storeHandleClose', [handle])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}
+
+/**
+ * `storeHandleList` — what this process holds open, with the retained bytes
+ * and rows and the two residency caps. A server that cannot see its own
+ * residency cannot be operated.
+ */
+export function listStoreHandles (engine) {
+  try {
+    return engine.call('storeHandleList', [])
+  } catch (error) {
+    throw asStoreError(error)
+  }
+}

package/l4-assets/l4factoidal.js

@@ -37,7 +37,7 @@
 // bytes change.
 
 // Stamped by formal/lean4/Wasm/build-wasm.sh step 9 -- do not hand-edit.
-const WASM_VERSION = "20551e4329bb";
+const WASM_VERSION = "ffbb3a035705";
 
 import createModule from './l4factoidal.mjs';
 

package/l4-assets/version.json

@@ -2,13 +2,13 @@
   "engine": "lean4",
   "note": "The Lean 4 (L4Factoidal) engine's own build provenance and claims -- distinct from ../version.json, which is the F* engine's. Do not merge the two: they come from different toolchains, different git SHAs in general, and carry different proof obligations (issue #618).",
   "version": "0.1.0",
-  "gitSha": "4295b2917e700fb3ead2d74fad05c9d2f592b7d9",
-  "builtAt": "2026-09-04T20:28:04+00:00",
+  "gitSha": "6613042b8acf3be78541f862f3ad0caf057cbab3",
+  "builtAt": "2026-09-04T20:56:42+00:00",
   "leanToolchain": "leanprover/lean4:v4.33.1",
   "emscripten": "emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 6.0.8-git",
   "abiVersion": "1",
-  "wasmSha256": "20551e4329bb3fa363e4c64b9f9d62cc1371a2c563161d938226c0d2e102c87a",
-  "wasmBytes": 5384150,
+  "wasmSha256": "ffbb3a0357051f78c2e9135ae629fc04a46037d540501f7fadd3d3e0e96c96b4",
+  "wasmBytes": 5412682,
   "claims": {
     "source": "formal/lean4 (L4Factoidal): no sorry, no user axioms, no native_decide; W3C behaviour pinned by build-time #guard",
     "suitesAtBuildSha": "see docs/test-results and formal/lean4/PORT_NOTES.md at gitSha"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@factoidal/core",
-  "version": "0.5.0",
+  "version": "0.5.1",
   "description": "Factoidal: a linked information system with graph data and the Web at its heart - RDF parsing, SPARQL 1.1 query, canonicalization and entailment for Node and the browser (JS + Wasm), grounded in F* and Lean 4. Parser and algebra spec verified in F*; on-disk backend has unverified OCaml-side optimization layers being migrated back to F*.",
   "keywords": [
     "sparql",

package/version.json

@@ -1,5 +1,5 @@
 {
-  "version": "0.5.0",
+  "version": "0.5.1",
   "gitSha": "49f8ca4d70bf57c12fea445611e2b01b067bf6ba",
   "builtAt": "2026-08-26T21:45:40Z",
   "claims": {