@factoidal/core 0.5.1 → 0.6.0

package/CHANGELOG.md

@@ -1,5 +1,137 @@
 # Changelog
 
+## 0.6.0 — 2026-09-05
+
+**A full-text index, a geometry index, extension functions from
+JavaScript, and a planner that no longer reads the whole store for a
+`REGEX`.** Measured end to end on the largest corpus we have: 7,315,251
+quads over 204 named graphs, 3,286 blocks, a 1.0 GB generation, queried
+through a store handle on plain `node` with no flags.
+
+| search | rows | time |
+|---|---|---|
+| `water` | 5 | 645 ms |
+| `glacier` | **0** | 670 ms |
+| `bicycle` | 2 | 617 ms |
+
+A miss costs what a hit costs, because both are index lookups.
+
+### The planner stopped reading the whole store
+
+`REGEX`, `REPLACE`, `IRI()`, `NOW()`, the digest functions, aggregates in
+a filter position, the triple-term accessors and every SPARQL 1.1
+section 17.6 extension call each made the planner abandon predicate
+selection and take every block in the manifest. On a 119-block store a
+`REGEX` filter selected 119 artifacts where the same query with
+`CONTAINS` selected 1.
+
+The test was `Expr.backendLocal`, which answers a different question: may
+the backend evaluate this expression, or must it materialise and
+delegate. The right test is `Expr.existsFree` — `Expr.existsPat` and
+`Expr.notExistsPat` are the only constructors carrying a pattern, so an
+exists-free expression reads no triple whatever functions it calls. A
+FILTER may narrow a plan or leave it alone; it must never widen it.
+`EXISTS` and `NOT EXISTS` still widen, correctly.
+
+### LGI1, a character-gram literal index
+
+The packer writes a `.lgi1` sidecar beside each block and the planner
+uses it where the query shape allows. It holds character 3-grams of the
+case-folded lexical form and is a CANDIDATE FILTER, never a decider: it
+answers a superset and the engine re-evaluates the original `FILTER`, so
+the rows are the scan's rows. Tokens would not do, because `CONTAINS` is
+a substring test and "underwater" contains "water" without being the
+token "water".
+
+Falls back to a scan, silently and correctly: a needle under 3
+characters, a variable needle, `REGEX`, `UCASE`, `!CONTAINS`, `CONTAINS`
+under `||`, a filter on a variable not bound in object position, and any
+generation without the sidecar. Costs about 55% of the block bytes.
+
+### GBI1, a geometry bounding-box index
+
+The same construction for `geof:`. Five of the six topological functions
+are filtered by a box — `sfIntersects`, `sfWithin`, `sfContains`,
+`sfTouches`, `sfEquals` — at 74 to 94 times a scan, with a miss at 82
+times. **`sfDisjoint` is refused** and falls back: it accepts exactly the
+rows a box can exclude, so a box test inverts and would drop answers.
+
+### Extension functions, from JavaScript, on the Lean engine
+
+`registerExtensionFunction` reaches the Lean engine and the persisted
+store, not only the in-memory F* path. Section 17.6 semantics are gated:
+an unregistered IRI is unbound in SELECT and drops the row in FILTER, a
+registration never overrides a built-in family, and `geof:` still
+answers from the built-in table. Synchronous only; async is designed and
+deferred.
+
+### A handle is bounded by retained bytes
+
+`storeOpen` was capped at 64 artifacts, inherited from the stateless path
+where every query re-read and re-decoded its blocks. A handle pays that
+once, so a count was the wrong shape — 257 small blocks may cost less
+than 4 large ones. The cap is now 134,217,728 retained bytes, derived
+from a measured 16.2 bytes resident per retained byte against half the
+wasm32 address space, and the check runs BEFORE the read rather than
+after hashing and decoding the whole set. A corpus-wide handle over 257
+blocks and 741,179 rows now opens.
+
+### Named graphs pack at scale
+
+Blocks cut at graph boundaries, so `GRAPH <iri>` with a constant
+predicate reads one block and the graph filter is exact rather than
+conservative. Packer peak memory fell 32.6% and dictionary duplication
+is 2.57%. No new wire version was needed: SBM7 already admitted several
+entries per predicate.
+
+**A large store no longer needs `node --stack-size`.** Several engine
+paths recurse once per manifest entry and once per row, and on a
+7,315,251-quad collection of 3,286 blocks `storeQueryPlan` alone
+overflowed the default call stack of Node, before an artifact byte was
+read ([issue 653](https://github.com/danbri/factoidal/issues/653)).
+
+- New `@factoidal/core/store-worker`: `openStoreHandleOnWorker` holds the
+  engine and the handle on a `worker_threads` thread with a raised stack.
+  A handle is state inside the wasm instance and an instance does not
+  cross a thread boundary, so the handle lives there and `query()` and
+  `close()` are messages to it. One thread per session, many handles per
+  thread. Every call is asynchronous; the synchronous `openStoreHandle`
+  is unchanged.
+- `factoidal query` still runs IN PROCESS, so a one-shot query pays
+  nothing for a worker it would drop. It runs again on a worker only when
+  the runtime runs out of frames — under Deno by re-executing itself with
+  a raised V8 stack, which needs `--allow-run` and `--allow-env`.
+  `--no-worker` turns the retry off.
+- Measured: the worker costs about 110 ms once (thread start plus a
+  second engine load) and under a millisecond per query after that.
+  Deno's `worker_threads` shim does not raise the stack (10,835 frames
+  by default, 13,837 at `stackSizeMb` 64, against Node's 41,195 and
+  696,555), so a Deno library caller gets an in-process handle behind the
+  same interface and starts its process with
+  `--v8-flags=--stack-size=65536`.
+
+### Fixed
+
+- `openStoreHandle` with no options failed on any IBK3 generation with
+  `artifact 'predicate-0.ibk3.sri2' is not declared by this manifest`.
+  Two engine operations disagreed: `storeManifestInspect` reports
+  `subjectIndex`, `termIndex` and `objectIndex`, while the handle's
+  admission accepted only the block, `literalIndex` and `geoIndex`. One
+  list now serves both.
+- The README named GeoSPARQL functions that do not exist and said a text
+  index was "separate work". Both corrected, with the shapes that fall
+  back to a scan listed.
+
+### Known limits, measured
+
+- Ingest is about 2,180 quads per second and peak memory is 2.07 times
+  the source, so a 142 GB corpus is not importable at that ratio. Making
+  memory flat is a wire change and is not done.
+- The quad packer streams N-Quads only; Turtle and TriG are buffered.
+- `ORDER BY ... LIMIT n` still overflows above about 14,576 materialised
+  rows. https://github.com/danbri/factoidal/issues/653
+- The manifest is at version 9. Older generations still read.
+
 ## 0.5.1 — 2026-09-04
 
 **A store can be opened once and queried many times.** `storeQuery` is

package/README.md

@@ -245,6 +245,45 @@ error. Async functions run over the synchronous verified engine
 through a bounded, memoised re-evaluation loop — within one query
 every call with the same arguments sees one stable answer.
 
+### The same functions against the Lean engine and a persisted store
+
+The registration above serves the F\* engine's in-memory `query()`. The
+Lean engine has its own registry, and it reaches every Lean query path,
+including a store handle — so a registered function can filter rows read
+off disk:
+
+```js
+import { loadEngine } from '@factoidal/core/bin/engine.mjs'
+import { openStore, openStoreHandle } from '@factoidal/core/bin/store.mjs'
+import { registerExtensionFunction, withExtensionFunctions }
+  from '@factoidal/core/bin/ext.mjs'
+
+const engine = await loadEngine()
+const handle = openStoreHandle(engine, openStore('/path/to/store'))
+
+registerExtensionFunction(engine, 'http://example.org/fn/endsWithZed',
+  ([label]) => label.value.endsWith('z'))
+
+const answer = handle.query(`
+  PREFIX ex:   <http://example.org/fn/>
+  PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+  SELECT ?c ?l WHERE { ?c skos:prefLabel ?l FILTER(ex:endsWithZed(?l)) }`)
+```
+
+The argument and result encoding, the §17.6 error rules and the
+async re-evaluation loop are the same as above, so one function serves
+both engines. Two differences to plan around:
+
+- **Registration is per engine instance**, not per handle or per query.
+  A server that answers for more than one caller uses
+  `withExtensionFunctions(engine, map, body)`, which registers, runs and
+  clears in a `finally`, or loads one engine per caller.
+- **Every call crosses into JavaScript**, so a function in a FILTER runs
+  once per row. Measure before putting one on a large scan.
+
+Design, with the determinism and scope rules in full:
+[`docs/designissues/2026-09-04-lean-extension-functions.md`](../../docs/designissues/2026-09-04-lean-extension-functions.md).
+
 ## Functional API (fn)
 
 `@factoidal/core/fn` is a strictly functional variant of the API above:
@@ -513,7 +552,7 @@ const docs = openStoreHandle(engine, openStore('./docs-store'))
 
 console.log(listStoreHandles(engine))
 // { ok: true, handles: [ {handle:'s1', bytes:…, rows:…}, {handle:'s2', …} ],
-//   bytes: …, rows: …, handleCap: 8, bytesCap: 67108864 }
+//   bytes: …, rows: …, handleCap: 8, bytesCap: 134217728 }
 
 const PREFIX = 'PREFIX skos: <http://www.w3.org/2004/02/skos/core#>'
 
@@ -559,23 +598,112 @@ caller that holds only the handle id.
 | tenth query, all different | 1382 ms | 103 ms |
 | ten queries, total | 13962 ms | 2277 ms |
 
-**What a handle buys and what it does not.** It removes the per-query
-digest check, block decode and index build. It does NOT make search
-sub-linear: `CONTAINS` still scans every retained row, so a query still
-costs time proportional to the row count. A text index is separate work.
+**What a handle buys.** It removes the per-query digest check, block
+decode and index build. With the LGI1 literal index (below) a search is
+also sub-linear rather than a scan, so a handle plus the index is what
+makes repeated search on a large store interactive.
 
 **Residency.** Retaining a decoded block costs memory. Measured on the
 same store: 76 MiB resident with the engine loaded and no handle, 170 MiB
 with the handle open — about 94 MiB for a 5.5 MB packed block, and
-evaluation peaks higher again (346 MiB during the queries above). The
-caps are on ARTIFACT bytes, which is what the manifest declares: 8 open
-handles, and 67108864 retained artifact bytes across all of them. A cap
-is a refusal naming the cap; no handle is ever evicted to make room for
+evaluation peaks higher again (346 MiB during the queries above). At
+corpus scale the marginal figure is smaller, because that one is carrying
+the fixed cost of the process: measured 2026-09-05 on a 7,315,251-quad
+store, one handle over 257 blocks and 103,341,302 retained artifact bytes
+peaked at 1,675,345,920 bytes resident — 16.2 bytes resident per retained
+byte.
+
+The caps are on ARTIFACT bytes, which is what the manifest declares: 8
+open handles, and 134217728 (128 MiB) retained artifact bytes across all
+of them. There is NO cap on the number of artifacts a handle retains; a
+count bounds nothing that the bytes do not
+(https://github.com/danbri/factoidal/issues/657). 128 MiB is half the
+wasm32 address space divided by that measured multiplier, and it admits
+the 257-block corpus-wide set above. A cap is a refusal naming the cap
+and the value that tripped it; no handle is ever evicted to make room for
 another. `listStoreHandles` is how a server sees its own residency.
 
 **One call at a time.** The WebAssembly module is single-threaded. Two
 `query()` calls cannot overlap; a server queues them.
 
+### A handle on a large store: the worker route
+
+Several engine paths recurse once per manifest entry and once per row.
+Against a large collection that exceeds the default call stack of Node
+and of Deno, and the failure is `Maximum call stack size exceeded`.
+**Measured 2026-09-05**, macOS arm64, Node 22.22.2, on a 7,315,251-quad
+collection of 3,286 blocks in 204 graphs: `storeQueryPlan` alone
+overflows on plain `node`, before one artifact byte is read.
+`node --stack-size=60000` clears the plan, the open and every query.
+
+`openStoreHandleOnWorker` removes the flag. It holds the engine and the
+handle on a `worker_threads` thread with `resourceLimits.stackSizeMb`,
+the route `factoidal pack` already takes
+([issue 649](https://github.com/danbri/factoidal/issues/649)). A handle
+is state inside the wasm instance and an instance does not cross a
+thread boundary, so the handle lives where the raised stack is, and
+`query()` and `close()` are messages to it.
+
+```js
+import { openStoreHandleOnWorker, closeSharedStoreWorkerSession }
+  from '@factoidal/core/store-worker'
+
+// No runtime flag. One worker thread, shared by every handle opened
+// this way, so a caller that opens several stores pays for one thread
+// and one copy of the module.
+const handle = await openStoreHandleOnWorker('/path/to/store', {
+  sparql: 'SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c ?p ?l } }'
+})
+const answer = await handle.query(`${PREFIX}
+  SELECT ?c ?l WHERE { GRAPH <urn:g> { ?c skos:prefLabel ?l }
+    FILTER(CONTAINS(LCASE(STR(?l)), "volcan")) } LIMIT 8`)
+await handle.close()
+await closeSharedStoreWorkerSession()
+```
+
+| option | effect |
+|---|---|
+| `{sparql}`, `{keys}` | the same artifact choice `openStoreHandle` takes |
+| `{generation}` | open a generation that has not been activated |
+| `{session}` | open into a session you started with `openStoreWorkerSession()` |
+| `{ownWorker: true}` | give this handle its own thread and its own copy of the module |
+| `{worker: false}` | open in this process, with no thread |
+
+What it costs, all **measured 2026-09-05** on the same machine, against
+the bundled sample store so the overhead is not lost in the query:
+
+| | in process | on a worker |
+|---|---|---|
+| one-shot query (thread start + engine load + query) | 163 ms | 220 ms |
+| handle open (thread start + engine load + open) | 124 ms | 236 ms |
+| every query after the open | 1 ms | 1 ms |
+
+So the thread and its second copy of the engine cost about 110 ms once,
+and the message round trip is under a millisecond. Three further costs:
+every call is asynchronous where the in-process handle is synchronous;
+an extension function registered on the main thread's engine
+(`bin/ext.mjs`) is not visible to the worker's engine; and the worker
+keeps the process alive until `close()`.
+
+**Deno takes a different route.** Deno's `node:worker_threads` shim
+accepts `resourceLimits.stackSizeMb` and raises almost nothing with it —
+measured by counting frames to the overflow inside the worker, Node
+reaches 41,195 frames by default and 696,555 at `stackSizeMb` 64, where
+Deno reaches 10,835 and 13,837. `openStoreHandleOnWorker` therefore
+gives a Deno caller an in-process handle behind the same asynchronous
+interface, and the process supplies the stack:
+
+```
+deno run --allow-read --v8-flags=--stack-size=65536 your-program.mjs
+```
+
+**A one-shot `factoidal query` pays none of this.** It builds no handle,
+so it runs in process and only retries on a worker if the runtime runs
+out of frames; under Deno it re-executes itself once with a raised V8
+stack, which needs `--allow-run` and `--allow-env`. `--no-worker` turns
+the retry off. See
+[issue 653](https://github.com/danbri/factoidal/issues/653).
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to
@@ -768,15 +896,51 @@ handling beyond what the WKT literal carries; no GML literals. Geometry
 comes from a WKT parser, so a shapefile, GeoJSON or GML source must be
 converted to `geo:wktLiteral` before it is loaded.
 
-### Full text: SPARQL's own functions, no index
+### Full text: SPARQL's own functions, over a character-gram index
 
 `CONTAINS`, `STRSTARTS`, `STRENDS` and `REGEX` (SPARQL 1.1 §17.4.3) are
-implemented and are the way to search text. They are evaluated per row
-after a block is decoded — **there is no inverted index and no
-`text:query`-style extension**. Measured 2026-09-04: a `CONTAINS` over
-45,806 `skos:prefLabel` values in one block answers in about 6 seconds.
-That is fine for a vocabulary and will not scale to a large literal
-corpus.
+the way to search text. There is no `text:query`-style extension and you
+do not ask for the index: where the query shape allows it, the planner
+uses the **LGI1** literal index the packer writes beside each block.
+
+LGI1 holds character 3-grams of the case-folded lexical form, and it is a
+CANDIDATE FILTER rather than a decider — it answers a superset and the
+engine re-evaluates your original `FILTER` on those rows, so the answer
+is exactly the answer a scan gives. Tokens would not do: `CONTAINS` is a
+substring test, and "underwater" contains "water" without being the
+token "water".
+
+Measured 2026-09-05, full skosdex corpus (7,315,251 quads, 3,286 blocks,
+204 named graphs, 1.0 GB), through a store handle on plain `node`:
+
+| search | rows | time |
+|---|---|---|
+| `water` | 5 | 645 ms |
+| `glacier` | 0 | 670 ms |
+| `bicycle` | 2 | 617 ms |
+
+A miss costs what a hit costs, because both are index lookups rather than
+scans. On a 45,806-row block the same search was about 180 ms of scan
+before the index and 1.8 ms after it.
+
+**It falls back to a scan**, silently and correctly, for: a needle under
+3 characters, a variable needle, `REGEX`, `UCASE`, `!CONTAINS`,
+`CONTAINS` under `||`, a filter on a variable not bound in object
+position, and any block whose generation has no `.lgi1` sidecar. The
+index costs about 55% of the block bytes.
+
+### Geometry: the GBI1 bounding-box index
+
+The same construction for `geof:`. Each `geo:wktLiteral` object's
+bounding box is indexed, and five of the six topological functions are
+filtered by it — `sfIntersects`, `sfWithin`, `sfContains`, `sfTouches`,
+`sfEquals`. Measured 74x to 94x against a scan, with a miss at 82x.
+
+**`sfDisjoint` is refused and falls back to a scan**, deliberately: it
+accepts exactly the rows a box can exclude, so a box test inverts and
+would drop answers. A non-overlapping pair of boxes proves geometries
+disjoint, but that saves work inside a scan rather than reducing the
+candidate set.
 
 ## Limits (deliberate, documented)
 

package/bin/ext.mjs

@@ -0,0 +1,198 @@
+// Caller-registered SPARQL 1.1 §17.6 extension functions for the Lean
+// engine. https://github.com/danbri/factoidal/issues/463 (the F* side),
+// docs/designissues/2026-09-04-lean-extension-functions.md (this side).
+//
+// WHAT THIS FILE IS ALLOWED TO DO
+// Hold the caller's functions, carry values across the boundary, and
+// memoise. It makes NO semantic decision: which IRIs may reach a
+// registered function, what an absent answer means, and how a §17.6
+// error behaves are all decided in Lean (Wasm/Ops/ExtFns.lean and the
+// evaluator). A JavaScript answer this file cannot encode becomes the
+// empty string, which Lean reads as the §17.6 error — the same outcome
+// as a function that is not registered at all.
+//
+// THE BRIDGE
+// One function, globalThis.__factoidalExtCall(iri, argsJson), which the
+// EM_JS body in formal/lean4/ffi/l4_ext.c calls SYNCHRONOUSLY from
+// inside the evaluator. `argsJson` is a JSON array of SPARQL Query
+// Results JSON binding-value objects; the answer is one such object as
+// a JSON string, or the empty string for "no value".
+//
+// ASYNC
+// The evaluator is synchronous and inside WebAssembly, so it cannot wait
+// for a promise. The F* engine's answer is reused verbatim
+// (npm/factoidal/browser.js, withExtensionRounds): the bridge starts the
+// promise, records it, and answers "no value" for this round; the host
+// awaits every pending promise, writes the answers into the memo table,
+// and runs the WHOLE query again, until a round adds no new pending
+// call.
+//
+// DETERMINISM
+// The memo table is keyed on iri + ' ' + argsJson and lives for one
+// top-level query, so the same call answers the same way however many
+// times the physical plan evaluates the expression, and across
+// re-evaluation rounds.
+
+const XSD = 'http://www.w3.org/2001/XMLSchema#'
+
+/** How many evaluation rounds an async resolution may take. */
+export const EXT_MAX_ROUNDS = 25
+
+const extFunctions = new Map()
+const extRegistered = new Set()
+let extCache = new Map()
+let extPending = []
+let bridgeInstalled = false
+
+/**
+ * Encode one JavaScript answer as an SRJ binding-value object, as a JSON
+ * string. `''` means "no value" — the §17.6 error.
+ */
+function encodeAnswer (out) {
+  if (out === null || out === undefined) return ''
+  if (typeof out === 'boolean') {
+    return JSON.stringify({ type: 'literal', value: out ? 'true' : 'false', datatype: XSD + 'boolean' })
+  }
+  if (typeof out === 'number') {
+    if (!Number.isFinite(out)) return ''
+    return Number.isInteger(out)
+      ? JSON.stringify({ type: 'literal', value: String(out), datatype: XSD + 'integer' })
+      : JSON.stringify({ type: 'literal', value: String(out), datatype: XSD + 'double' })
+  }
+  if (typeof out === 'string') {
+    return JSON.stringify({ type: 'literal', value: out })
+  }
+  if (typeof out === 'object' && typeof out.type === 'string') {
+    // Already an SRJ binding-value object (or an RDF/JS-shaped term the
+    // caller built): pass it through and let the Lean decoder judge it.
+    return JSON.stringify(out)
+  }
+  return ''
+}
+
+/** The synchronous bridge the wasm module calls. */
+function extBridge (iri, argsJson) {
+  const key = iri + ' ' + argsJson
+  if (extCache.has(key)) return extCache.get(key)
+  const fn = extFunctions.get(iri)
+  if (typeof fn !== 'function') return ''
+  let out
+  try {
+    out = fn(JSON.parse(argsJson))
+  } catch (_error) {
+    extCache.set(key, '')
+    return ''
+  }
+  if (out !== null && typeof out === 'object' && typeof out.then === 'function') {
+    extPending.push({ key, promise: out })
+    return ''
+  }
+  const encoded = encodeAnswer(out)
+  extCache.set(key, encoded)
+  return encoded
+}
+
+function installBridge () {
+  if (bridgeInstalled) return
+  globalThis.__factoidalExtCall = extBridge
+  bridgeInstalled = true
+}
+
+/**
+ * Register one function under an absolute IRI.
+ *
+ * `fn` receives the evaluated arguments as an array of SRJ
+ * binding-value objects and returns a term object, a JavaScript
+ * string/number/boolean, `null`/`undefined` (the §17.6 error), or a
+ * promise of any of those.
+ *
+ * Scope: per loaded engine (one wasm module instance). A server that
+ * serves more than one caller uses `withExtensionFunctions` instead, or
+ * loads one engine per caller.
+ */
+export function registerExtensionFunction (engine, iri, fn) {
+  if (typeof iri !== 'string' || !/^[A-Za-z][A-Za-z0-9+.-]*:/.test(iri)) {
+    throw new TypeError('registerExtensionFunction: iri must be an absolute IRI string')
+  }
+  if (typeof fn !== 'function') {
+    throw new TypeError('registerExtensionFunction: fn must be a function')
+  }
+  installBridge()
+  extFunctions.set(iri, fn)
+  engine.call('extRegister', [iri])
+  extRegistered.add(iri)
+}
+
+/** Remove one registered function. */
+export function unregisterExtensionFunction (engine, iri) {
+  extFunctions.delete(iri)
+  extRegistered.delete(iri)
+  engine.call('extUnregister', [iri])
+}
+
+/** Return the engine to the built-in table (`geof:` and the built-ins). */
+export function clearExtensionFunctions (engine) {
+  extFunctions.clear()
+  extRegistered.clear()
+  engine.call('extClear', [])
+}
+
+/** The IRIs the engine will ask the host about. */
+export function listExtensionFunctions (engine) {
+  return engine.call('extList', []).iris
+}
+
+/**
+ * Run `body` with a fresh per-query memo table, resolving async answers
+ * by re-evaluation. `body` is called once per round and must be the
+ * WHOLE query, because a later round re-evaluates it with more answers
+ * known.
+ */
+export async function withExtensionRounds (body) {
+  extCache = new Map()
+  for (let round = 0; ; round += 1) {
+    extPending = []
+    const result = body()
+    if (extPending.length === 0) return result
+    if (round >= EXT_MAX_ROUNDS) {
+      throw new Error(
+        'extension functions: async resolution did not converge within ' +
+        `${EXT_MAX_ROUNDS} evaluation rounds`)
+    }
+    const pending = extPending
+    extPending = []
+    await Promise.all(pending.map(async ({ key, promise }) => {
+      try {
+        extCache.set(key, encodeAnswer(await promise))
+      } catch (_error) {
+        extCache.set(key, '')
+      }
+    }))
+  }
+}
+
+/**
+ * Run one query with a fresh memo table. Use this for a synchronous
+ * function set; use `withExtensionRounds` when any function is async.
+ */
+export function withFreshMemo (body) {
+  extCache = new Map()
+  extPending = []
+  return body()
+}
+
+/**
+ * Register `map` (IRI -> function), run `body`, and clear in a
+ * `finally`. This is the scope a long-lived server wants: one caller's
+ * registrations are never visible to the next.
+ */
+export async function withExtensionFunctions (engine, map, body) {
+  for (const [iri, fn] of Object.entries(map)) {
+    registerExtensionFunction(engine, iri, fn)
+  }
+  try {
+    return await withExtensionRounds(body)
+  } finally {
+    clearExtensionFunctions(engine)
+  }
+}

package/bin/factoidal.mjs

@@ -28,6 +28,9 @@ import { loadEngine } from './engine.mjs'
 import { sampleStoreFacts, sampleStorePath } from '../sample-store.mjs'
 import { PackError, packSupported, verifyGeneration } from './pack.mjs'
 import { denoReexec, isStackOverflow, runPack } from './pack-host.mjs'
+import {
+  planQueryOnWorker, queryStoreOnWorker, workerRouteAvailable
+} from './store-worker-host.mjs'
 import {
   STACK_REMEDY, StoreOperationError, inspectManifest, openStore, planQuery,
   queryStore, stackLimitAdvice, turtleOfNQuads
@@ -179,6 +182,21 @@ options:
   --generation NAME  query this generation instead of the activated one
   --json             shorthand for --format json
   --quiet            print the result only, no plan line on stderr
+  --no-worker        never retry on a worker thread (see below)
+
+Several engine paths recurse once per manifest entry or once per row, and
+against a large collection that exceeds the runtime's default call stack.
+A one-shot query builds no store handle, so it runs IN PROCESS and pays
+nothing for a worker it would drop; only when the runtime runs out of
+frames does it run again on a worker thread with a raised stack, or under
+Deno re-execute itself once with a raised V8 stack, which needs
+--allow-run and --allow-env (https://github.com/danbri/factoidal/issues/653).
+--no-worker turns the retry off, and the frame budget is then reported.
+
+A process that asks MANY questions about one store should hold a store
+handle instead: openStoreHandleOnWorker in bin/store-worker-host.mjs
+verifies and decodes once, on a thread with the raised stack, and answers
+every later query from what it retained.
 
 formats:
   table    a human display of the engine's SPARQL Query Results JSON;
@@ -532,6 +550,36 @@ function reportStoreFailure (error) {
   return EXIT_FAILURE
 }
 
+/**
+ * Run one store call again on a bigger call stack, after the in-process
+ * attempt ran out of frames.
+ *
+ * A one-shot query builds no handle and drops what it loads, so it runs
+ * IN PROCESS first and pays nothing for a worker it would throw away.
+ * This is the retry, and it is reached only by the overflow.
+ *
+ * Node takes the worker route; Deno has no `worker_threads` route with a
+ * stack size, so it re-executes this command once with a raised V8 stack
+ * (`denoReexec`), which needs --allow-run and --allow-env.
+ *
+ * @returns the retried answer, or a number to exit with (the Deno child's
+ *   code), or null when no bigger stack is available here
+ */
+async function onBiggerStack (options, quiet, retry) {
+  const host = { worker: options['no-worker'] !== true }
+  if (host.worker === false) return null
+  if (isDeno) {
+    const code = await denoReexec(host)
+    return typeof code === 'number' ? code : null
+  }
+  if (!workerRouteAvailable()) return null
+  if (!quiet) {
+    err('The runtime ran out of call stack; running again on a worker thread ' +
+      'with a bigger one.')
+  }
+  return await retry()
+}
+
 async function commandQuery (positional, options) {
   if (positional.length < 1) throw new UsageError('query needs a STORE')
   if (typeof options.base === 'string') {
@@ -554,8 +602,13 @@ async function commandQuery (positional, options) {
     try {
       plan = planQuery(engine, store, sparql)
     } catch (error) {
-      if (error instanceof StoreOperationError) return reportStoreFailure(error)
-      throw error
+      if (!(error instanceof StoreOperationError)) throw error
+      if (!error.stackLimit) return reportStoreFailure(error)
+      const retried = await onBiggerStack(options, quiet,
+        () => planQueryOnWorker(root, named, sparql))
+      if (retried === null) return reportStoreFailure(error)
+      if (typeof retried === 'number') return retried
+      plan = retried
     }
     if (format === 'json' || options.json === true) {
       out(JSON.stringify(plan, null, 2))
@@ -572,8 +625,13 @@ async function commandQuery (positional, options) {
   try {
     answer = queryStore(engine, store, sparql)
   } catch (error) {
-    if (error instanceof StoreOperationError) return reportStoreFailure(error)
-    throw error
+    if (!(error instanceof StoreOperationError)) throw error
+    if (!error.stackLimit) return reportStoreFailure(error)
+    const retried = await onBiggerStack(options, quiet,
+      () => queryStoreOnWorker(root, named, sparql))
+    if (retried === null) return reportStoreFailure(error)
+    if (typeof retried === 'number') return retried
+    answer = retried
   }
   const { plan, result, blobBytes } = answer
   if (!quiet) {