@factoidal/core 0.2.0 → 0.3.0

package/CHANGELOG.md

@@ -1,5 +1,48 @@
 # Changelog
 
+## 0.3.0 — 2026-09-03
+
+- The `factoidal` command answers SPARQL against a persisted
+  Shardborough store with no native binary. `factoidal inspect STORE`
+  decodes the manifest through the engine's `storeManifestInspect`
+  operation and prints the wire version, layout, blank-node profile and
+  the entry table; `factoidal query STORE 'SELECT ...'` asks the engine
+  which artifacts the query needs, reads exactly those, and hands their
+  bytes to `storeQuery` through one WebAssembly heap buffer with no
+  encoding. Every artifact is verified against the SHA-256 the manifest
+  commits before an answer is given. Formats: `table`, `json`,
+  `nquads`, `turtle`; `--explain` prints the artifact plan. Runs under
+  Node and under Deno (`--allow-read`).
+- The three caps of the store query operation — 64 artifacts, 8388608
+  artifact bytes, 100000 rows — are reported with the cap, the value
+  and a next step, and are decided before a single file is read.
+  Nothing is truncated.
+- `pack`, `activate`, `update` and `compact` still exit 3; they need
+  operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+- A large piped result is no longer truncated: the command sets
+  `process.exitCode` instead of calling `process.exit()`, which dropped
+  everything past the 64 KiB pipe boundary.
+
+## 0.3.0 — Lean block-worker preview
+
+- The bundled Lean-derived WASM artifact exports
+  `scanIBK2Predicate(ibk2Hex, predicateIri)` through `factoidal/l4`.
+  It validates a canonical IBK2 block and executes its selective predicate
+  scan, returning N-Triples plus a row count.  This is a narrow physical
+  helper for the Shardborough work, not full SPARQL execution inside a storage
+  backend and not a high-throughput buffer ABI (the current hex transport is
+  intentionally diagnostic).
+- The regenerated Lean artifact is synchronized across the package's
+  `l4-assets/` and the Hub build, and is exercised against the music IBK2
+  fixture by `tools/wasm-ibk2-smoke.mjs` in the repository.
+- The current-format companion
+  `scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` validates and
+  scans complete predicate-local IBK3 artifacts. Its mandatory source/dataset
+  scope preserves one blank node across blocks from the same RDF import while
+  keeping same-spelled labels in unrelated inputs apart. Hub post 51 composes
+  three real IBK3 scans into an editable Lean-WASM SPARQL query.
+
 ## 0.2.0 — both engines in one package, with a backend selector
 
 - The Lean 4 engine (`L4Factoidal`, wasm) now ships INSIDE

package/README.md

@@ -1,6 +1,6 @@
 # @factoidal/core
 
-> First published cut (0.1.0). The API surface is early and may
+> Release 0.3.0. The API surface is early and may
 > change before 1.0. This package was previously developed in-tree
 > under the placeholder names `factoidal` and `@danbri/foafos`; it was
 > never published under those names. See [CHANGELOG.md](CHANGELOG.md).
@@ -312,6 +312,16 @@ Two engines now ship in one package. `factoidal` and `factoidal/wasm`
 are the F\*-extracted engine, unchanged. The subpaths below are the
 Lean 4 engine (`L4Factoidal`, compiled to wasm).
 
+`factoidal/l4` also exposes two deliberately narrow physical helpers:
+`scanIBK2Predicate(ibk2Hex, predicateIri)` for the predecessor format and
+`scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` for the current
+predicate-local format. They validate one canonical RDF block and scan its
+named predicate, returning N-Triples and a row count. The IBK3 source scope
+must be shared across blocks partitioned from one RDF import unit and differ
+across unrelated units; this preserves document-scoped blank-node identity
+when fragments are composed. The hexadecimal argument is a portable
+diagnostic ABI, not the intended high-throughput buffer interface.
+
 ```js
 const l4 = require('factoidal/l4-core');       // Lean engine, same API shape
 const { select } = require('factoidal/select'); // choose an engine per call
@@ -358,6 +368,92 @@ model-theory modules run to about 22,000 lines — but only these reach
 JavaScript today. Everything else in the Lean tree is used through
 `parse`/`query`/`closure`, or not exposed at all.
 
+## The `factoidal` command: querying a persisted store
+
+Installing this package puts a `factoidal` command on PATH. It reads a
+**Shardborough** store — the on-disk format the Lean `l4block-*` tools
+write — with no native binary: JavaScript reads the files and moves the
+bytes, and the Lean engine running as WebAssembly makes every format
+decision (parsing the manifest, choosing the blocks, verifying their
+SHA-256, evaluating the SPARQL).
+
+> This command is not the native F\* `factoidal` binary that the API
+> table below refers to. That one is `bin/<platform>/factoidal` in the
+> repository and takes subcommands such as `shex` and `compact`. This
+> one takes `version`, `inspect` and `query`.
+
+```console
+$ factoidal inspect ./mystore
+store ./mystore
+generation gen-1 (activated through CURRENT)
+manifest manifest.sbm2, 2372 bytes, wire version 6
+layout predicate-ibk3-ptd1-sri2-tli1-oli2-merkle-v0
+blank-node profile (none recorded)
+term registry local-ibk3-ptd1-v0
+fixed-chunk Merkle commitment yes
+5 entries, 393775 bytes, 6455 rows
+generation directory holds 42 files, 846592 bytes
+
+#  rows  bytes   kind  graphs  predicate
+0  1800  110085  IBK3  -       http://www.wikidata.org/prop/direct/P31
+1  719   35535   IBK3  -       http://www.wikidata.org/prop/direct/P361
+...
+
+$ factoidal query ./mystore 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
+mode ibk3-paged-merkle-full-manifest(5), 5 artifacts, 393775 bytes read, plan declares 6455 block rows
+n
+"6455"^^<http://www.w3.org/2001/XMLSchema#integer>
+1 row
+```
+
+`STORE` is a collection root: the directory holding `CURRENT`. The plan
+line goes to stderr, so stdout carries only the result; `--quiet`
+removes it.
+
+| Option | What it does |
+|---|---|
+| `--format table` | default; a human display of the results |
+| `--format json` | SELECT prints the engine's SPARQL 1.1 Query Results JSON; ASK and CONSTRUCT print the operation's envelope |
+| `--format nquads` | CONSTRUCT only: the graph the engine serialized |
+| `--format turtle` | CONSTRUCT only: that graph through the engine's own Turtle writer |
+| `--explain` | print the artifacts the query needs and the open mode, and stop |
+| `--limit N` | print at most N table rows; the total is always named |
+| `--file PATH` | read the query text from a file |
+| `--generation NAME` | read that generation rather than the activated one |
+
+Under Deno, run the file directly; `inspect` and `query` need only
+`--allow-read`:
+
+```console
+$ deno run --allow-read node_modules/@factoidal/core/bin/factoidal.mjs query ./mystore 'ASK { ?s ?p ?o }'
+```
+
+### What the command answers for, and what it does not
+
+* **Every artifact is verified.** The engine refuses the whole query
+  when a block's bytes do not hash to the SHA-256 the manifest commits,
+  and names the artifact.
+* **Three caps.** One call reads at most 64 artifacts, 8388608 artifact
+  bytes and 100000 rows. A query over any of them is refused before a
+  single file is read, with the cap and the value named. Nothing is
+  truncated.
+* **Committed artifacts only.** A store carrying uncompacted delta-log
+  updates is not served by this path; use the native `l4block-*` tools.
+* **`pack`, `activate`, `update` and `compact` exit 3.** They need
+  WebAssembly operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+* **Node's WebAssembly frame budget.** Some evaluator paths recurse once
+  per row. Measured 2026-09-03 on a 6455-row store, `SELECT ?s ?p ?o
+  WHERE { ?s ?p ?o }` overflows the stack under Node's default while
+  `SELECT *`, or the same query with a `LIMIT`, does not, and Deno
+  clears all of them. The command reports it and exits 1 rather than
+  crashing; `node --stack-size=4000` clears it.
+
+Measured 2026-09-03 on macOS arm64, the 6455-triple `sequence_variant`
+store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
+including start-up: 220 ms through this command, 33 ms through the
+native `l4block-id-v3-query`.
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to

package/bin/engine.mjs

@@ -0,0 +1,124 @@
+// Loading the Lean engine (L4Factoidal compiled to WebAssembly) for the
+// `factoidal` command, under Node and under Deno.
+// https://github.com/danbri/factoidal/issues/641
+//
+// This file resolves three files that must stay together and keep their
+// names -- l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm -- and
+// imports the loader. The Emscripten glue resolves the .wasm sidecar from
+// its own basename, so a rename breaks the load on every runtime
+// (skills/lean4-wasm-export, "the naming trap").
+//
+// `npm/factoidal/l4.js` does the same resolution for the CommonJS API. It
+// is not reused here because it is CommonJS and this command must load
+// under Deno without the require() compatibility path.
+//
+// WHAT THIS FILE IS ALLOWED TO DO
+// Find the engine, load it, and turn bytes into the hexadecimal string
+// two of the three store operations take as their small argument. It
+// makes no format decision: it never parses a manifest, verifies a
+// digest, decodes a block, or chooses an artifact.
+
+import { readWhole } from '../store-host/index.mjs'
+import { fileUrlToPath, joinPath } from '../store-host/paths.mjs'
+
+/** Read one environment variable, or null where reading it is refused. */
+function environment (name) {
+  try {
+    if (typeof globalThis.Deno !== 'undefined' && globalThis.Deno.env) {
+      const value = globalThis.Deno.env.get(name)
+      return typeof value === 'string' && value.length > 0 ? value : null
+    }
+  } catch (_error) {
+    // Deno without --allow-env. An absent override is the normal case.
+    return null
+  }
+  const value = globalThis.process && globalThis.process.env
+    ? globalThis.process.env[name]
+    : undefined
+  return typeof value === 'string' && value.length > 0 ? value : null
+}
+
+function readable (path) {
+  try {
+    readWhole(path)
+    return true
+  } catch (_error) {
+    return false
+  }
+}
+
+/** The package directory that holds this command. */
+export function packageDirectory () {
+  return fileUrlToPath(new URL('..', import.meta.url).href).replace(/\/$/, '')
+}
+
+/**
+ * The loader file of the first engine source that exists, or null.
+ *
+ * Order, first hit wins:
+ *   1. this package's own l4-assets/ (what `npm install` gives);
+ *   2. $FACTOIDAL_L4_ASSETS (a custom deployment);
+ *   3. the repository checkout layout, docs/web/hub/assets/l4/.
+ */
+export function resolveEngine () {
+  const inPackage = joinPath(packageDirectory(), 'l4-assets/l4factoidal.js')
+  if (readable(inPackage)) return inPackage
+  const override = environment('FACTOIDAL_L4_ASSETS')
+  if (override !== null) {
+    const path = joinPath(override, 'l4factoidal.js')
+    if (readable(path)) return path
+    throw new Error(`FACTOIDAL_L4_ASSETS=${override} holds no l4factoidal.js`)
+  }
+  const inCheckout = joinPath(packageDirectory(),
+    '../../docs/web/hub/assets/l4/l4factoidal.js')
+  if (readable(inCheckout)) return inCheckout
+  return null
+}
+
+let enginePromise = null
+
+/**
+ * Load the engine once and return its handle: `call(op, args)` and
+ * `callBlob(op, args, bytes)`.
+ */
+export function loadEngine () {
+  if (enginePromise !== null) return enginePromise
+  enginePromise = (async () => {
+    const loaderPath = resolveEngine()
+    if (loaderPath === null) {
+      throw new Error(
+        'the Lean engine assets are missing. They normally ship in this ' +
+        "package's own l4-assets/ directory; if it is absent this install " +
+        'is incomplete. Otherwise set FACTOIDAL_L4_ASSETS to a directory ' +
+        'holding l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm.')
+    }
+    const url = loaderPath.startsWith('file://')
+      ? loaderPath
+      : 'file://' + encodeURI(loaderPath).replace(/#/g, '%23')
+    const module = await import(url)
+    return module.loadL4()
+  })()
+  return enginePromise
+}
+
+const HEX = (() => {
+  const table = new Array(256)
+  for (let byte = 0; byte < 256; byte += 1) {
+    table[byte] = byte.toString(16).padStart(2, '0')
+  }
+  return table
+})()
+
+/**
+ * Lowercase hexadecimal of a byte array. This is a transport encoding for
+ * the manifest argument of `storeManifestInspect` and `storeQueryPlan`,
+ * not an interpretation of the bytes. Artifact bytes never take this
+ * path: they cross raw through `callBlob`.
+ * @param {Uint8Array} bytes
+ * @returns {string}
+ */
+export function hexOfBytes (bytes) {
+  const parts = new Array(bytes.length)
+  for (let index = 0; index < bytes.length; index += 1) parts[index] = HEX[bytes[index]]
+  return parts.join('')
+}