@factoidal/core 0.1.0 → 0.3.0

package/CHANGELOG.md

@@ -1,5 +1,130 @@
 # Changelog
 
+## 0.3.0 — 2026-09-03
+
+- The `factoidal` command answers SPARQL against a persisted
+  Shardborough store with no native binary. `factoidal inspect STORE`
+  decodes the manifest through the engine's `storeManifestInspect`
+  operation and prints the wire version, layout, blank-node profile and
+  the entry table; `factoidal query STORE 'SELECT ...'` asks the engine
+  which artifacts the query needs, reads exactly those, and hands their
+  bytes to `storeQuery` through one WebAssembly heap buffer with no
+  encoding. Every artifact is verified against the SHA-256 the manifest
+  commits before an answer is given. Formats: `table`, `json`,
+  `nquads`, `turtle`; `--explain` prints the artifact plan. Runs under
+  Node and under Deno (`--allow-read`).
+- The three caps of the store query operation — 64 artifacts, 8388608
+  artifact bytes, 100000 rows — are reported with the cap, the value
+  and a next step, and are decided before a single file is read.
+  Nothing is truncated.
+- `pack`, `activate`, `update` and `compact` still exit 3; they need
+  operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+- A large piped result is no longer truncated: the command sets
+  `process.exitCode` instead of calling `process.exit()`, which dropped
+  everything past the 64 KiB pipe boundary.
+
+## 0.3.0 — Lean block-worker preview
+
+- The bundled Lean-derived WASM artifact exports
+  `scanIBK2Predicate(ibk2Hex, predicateIri)` through `factoidal/l4`.
+  It validates a canonical IBK2 block and executes its selective predicate
+  scan, returning N-Triples plus a row count.  This is a narrow physical
+  helper for the Shardborough work, not full SPARQL execution inside a storage
+  backend and not a high-throughput buffer ABI (the current hex transport is
+  intentionally diagnostic).
+- The regenerated Lean artifact is synchronized across the package's
+  `l4-assets/` and the Hub build, and is exercised against the music IBK2
+  fixture by `tools/wasm-ibk2-smoke.mjs` in the repository.
+- The current-format companion
+  `scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` validates and
+  scans complete predicate-local IBK3 artifacts. Its mandatory source/dataset
+  scope preserves one blank node across blocks from the same RDF import while
+  keeping same-spelled labels in unrelated inputs apart. Hub post 51 composes
+  three real IBK3 scans into an editable Lean-WASM SPARQL query.
+
+## 0.2.0 — both engines in one package, with a backend selector
+
+- The Lean 4 engine (`L4Factoidal`, wasm) now ships INSIDE
+  `@factoidal/core` as `l4-assets/`, alongside the F\*-extracted
+  engine. One install gets both. `require('factoidal/l4')` and
+  `factoidal/l4-core` are unchanged; the resolver checks in-package
+  assets first, then falls through to the old order (companion
+  package, `$FACTOIDAL_L4_ASSETS`, repo checkout).
+- `@factoidal/lean` is superseded and was never published.
+- New subpath `factoidal/select`: a backend selector, per-instance
+  with per-call override. Values `lean`, `fstar`, `lean1st`,
+  `fstar1st`, `slowcompareboth`. A request naming exactly one engine
+  never gets an answer from the other -- `lean` and `fstar` throw on a
+  function that engine does not implement; `lean1st` / `fstar1st` fall
+  through, and `lean1st` also takes a list of functions to route to
+  F\* regardless. Every result carries the answering engine.
+  `slowcompareboth` runs both and REPORTS disagreement rather than
+  throwing; comparison is RDFC-1.0 isomorphism for Dataset-shaped
+  results and bag equality (blank nodes relabelled per side) for
+  SELECT bindings.
+- Measured on the typed API surface at this release: 15 functions both
+  engines answer, 38 F\*-only, 4 Lean-only (out of 57). This measures
+  the typed wrapper surface, not the Lean engine's total capability --
+  `l4-core.js` wires 16 of the engine's 21 dispatch ops.
+- `clParse` (Common Logic Interchange Format text, ISO/IEC 24707:2018,
+  with the IKL `that`-operator extension) is now wired into the typed
+  API -- `l4-core.js`/`lib/api.js`/`select.js`, with `.d.ts` types and
+  tests. It reads CLIF text and reports its shape (sentence count,
+  `pureCL` dialect flag, canonical re-serialisation); it never produces
+  RDF. `pureCL` is a DIALECT flag, not a validity or quality signal:
+  true while the text stays inside ISO/IEC 24707 Common Logic, false
+  once it uses IKL's `that` operator. It is the first Lean-only entry
+  on the typed capability table -- formal/fstar has no CL/IKL parser at
+  all, so `index.js`/`wasm.js` never export it, `factoidal/select`'s
+  `backend:'fstar'` throws for it (never falls back to Lean), and
+  `backend:'slowcompareboth'` fails the capability precondition rather
+  than comparing one side against nothing.
+- `clSerialize`, `clAlphaNorm` and `clNormalize` join `clParse` on the
+  same Lean-only typed surface (owner instruction, 2026-08-26 -- "wire
+  into js functional api"), same `.d.ts`-typed, test-covered pattern:
+  `clSerialize` reads CLIF text and writes it back in canonical
+  spacing, surfacing `roundTripProved: false` unmodified (the
+  round-trip lemma `clif_roundTrip`, `CL/ClifAdequacy.lean`, is OPEN --
+  the fragment boundary `marksLexable` is measured, not proved).
+  `clAlphaNorm` gives each sentence's bound-variable-renaming canonical
+  form (IKL GUIDE Appendix B condition (1)). `clNormalize` is Hayes's
+  satisfiability-preserving reduction of IKL to Common Logic
+  ([#625](https://github.com/danbri/factoidal/issues/625)): it
+  surfaces `preserves: "satisfiability"` (not equivalence -- suited to
+  entailment/consistency testing, not to transforming data you intend
+  to keep) and `noIntrusion` (the proof hypothesis `CL.noIntrSs [] []`
+  decides, not a paraphrase of it). `clFiniteSat`, the fifth CL/IKL op,
+  is DEFERRED rather than wired: it takes a caller-supplied finite-
+  interpretation JSON encoding with no user yet, and a typed wrapper
+  would freeze that shape before it is known to be right; it stays
+  reachable through the raw dispatch ABI.
+- The IKL-to-RDF projection ops (`clToDataset`, `queryWithIklService`)
+  are DELETED from the engine source (danbri/factoidal#626), along with
+  the content-addressed proposition graph names they minted. They were
+  never exposed through the npm API. `x-ikl-*` entailment regimes are
+  still rejected at the JS layer, and the engine no longer defines the
+  family either. The two ops remain present in the compiled wasm
+  artifact until it is rebuilt (danbri/factoidal#627) -- the artifact
+  is ahead of its source. `clParse` is a different op (it never
+  produces RDF) and is unaffected.
+
+## 0.2.0 — Lean 4 engine subpath
+
+- New `factoidal/l4-core` subpath: the same typed API served by the
+  Lean 4-extracted wasm engine instead of the F\*-extracted one
+  ([#476](https://github.com/danbri/factoidal/issues/476)). Engine
+  assets resolve from the `@factoidal/lean` companion package, the
+  `FACTOIDAL_L4_ASSETS` environment variable, or the repository
+  checkout, in that order. `capabilities()` reports the Lean engine's
+  actual surface; `shaclValidate`/`owlIsConsistent`/`owlEntails`
+  raise pinned errors rather than returning wrong answers
+  ([#586](https://github.com/danbri/factoidal/issues/586) tracks the
+  OWL verdicts).
+- `serialize({ format: "nquads" })` now normalizes through the entry
+  ABI instead of the CLI path.
+
+
 ## 0.1.0 — First published release (as `@factoidal/core`)
 
 - SPARQL 1.1 §17.6 extension functions

package/README.md

@@ -1,6 +1,6 @@
 # @factoidal/core
 
-> First published cut (0.1.0). The API surface is early and may
+> Release 0.3.0. The API surface is early and may
 > change before 1.0. This package was previously developed in-tree
 > under the placeholder names `factoidal` and `@danbri/foafos`; it was
 > never published under those names. See [CHANGELOG.md](CHANGELOG.md).
@@ -306,6 +306,154 @@ const tabular = await fromCsvw(csvText, csvwMetadataJson, { mode: 'minimal' });
 const saturated = await rif(data, rifRulesXml); // RIF Core forward chaining
 ```
 
+## Lean engine (new in 0.2.0)
+
+Two engines now ship in one package. `factoidal` and `factoidal/wasm`
+are the F\*-extracted engine, unchanged. The subpaths below are the
+Lean 4 engine (`L4Factoidal`, compiled to wasm).
+
+`factoidal/l4` also exposes two deliberately narrow physical helpers:
+`scanIBK2Predicate(ibk2Hex, predicateIri)` for the predecessor format and
+`scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` for the current
+predicate-local format. They validate one canonical RDF block and scan its
+named predicate, returning N-Triples and a row count. The IBK3 source scope
+must be shared across blocks partitioned from one RDF import unit and differ
+across unrelated units; this preserves document-scoped blank-node identity
+when fragments are composed. The hexadecimal argument is a portable
+diagnostic ABI, not the intended high-throughput buffer interface.
+
+```js
+const l4 = require('factoidal/l4-core');       // Lean engine, same API shape
+const { select } = require('factoidal/select'); // choose an engine per call
+```
+
+### `factoidal/l4-core`
+
+Same call shape as the main API — `parse`, `query`, `update`,
+`serialize`, `canonicalize`, `graphs`, `canonicalHash`, `owlClosure`,
+`coreRdfsClosure`, `coreRdfsCheck`, `rhoDfClosure`,
+`rhoDfFragmentCheck`, `rdfsPlusClosure`, `owlIsConsistent` — plus four
+Common Logic / IKL operations that exist only here, because the F\* tree
+has no CL parser.
+
+| Function | In → out | What the answer is worth |
+|---|---|---|
+| `clParse(clifText)` | CLIF text → shape report | sentence count, CL-vs-IKL dialect, canonical re-serialisation. Reads CLIF; never produces RDF |
+| `clSerialize(clifText)` | CLIF → CLIF | canonical writer. Returns `roundTripProved: false` — `clif_roundTrip` is an open lemma and the fragment boundary is measured, not proved |
+| `clAlphaNorm(clifText)` | CLIF → CLIF | alpha-equivalence canonical form (IKL Appendix B condition 1) |
+| `clNormalize(clifText)` | CLIF → CLIF | Hayes's IKL-to-CL reduction. Returns `preserves: "satisfiability"` — **not** equivalence — and `noIntrusion`, the proof hypothesis decided rather than assumed |
+
+A fifth op, `clFiniteSat(interpJson, clifText)`, is reachable only
+through the raw dispatch ABI: `l4.call('clFiniteSat', [interpJson,
+clifText])`. It is not in the typed layer.
+
+### `factoidal/select`
+
+Per-instance backend choice with a per-call override.
+
+| Value | Behaviour |
+|---|---|
+| `lean` / `fstar` | that engine only; **throws** if it does not implement the function |
+| `lean1st` / `fstar1st` | prefer that engine, fall through to the other for unimplemented functions |
+| `slowcompareboth` | run both and **report** disagreement rather than throwing |
+
+Every result names the engine that answered. `capabilityTable()`
+returns which functions each engine implements.
+
+### What this surface is, honestly
+
+Four `String → String` operations plus one through raw dispatch. The
+Lean tree behind them is larger than that — the CL/IKL and unified
+model-theory modules run to about 22,000 lines — but only these reach
+JavaScript today. Everything else in the Lean tree is used through
+`parse`/`query`/`closure`, or not exposed at all.
+
+## The `factoidal` command: querying a persisted store
+
+Installing this package puts a `factoidal` command on PATH. It reads a
+**Shardborough** store — the on-disk format the Lean `l4block-*` tools
+write — with no native binary: JavaScript reads the files and moves the
+bytes, and the Lean engine running as WebAssembly makes every format
+decision (parsing the manifest, choosing the blocks, verifying their
+SHA-256, evaluating the SPARQL).
+
+> This command is not the native F\* `factoidal` binary that the API
+> table below refers to. That one is `bin/<platform>/factoidal` in the
+> repository and takes subcommands such as `shex` and `compact`. This
+> one takes `version`, `inspect` and `query`.
+
+```console
+$ factoidal inspect ./mystore
+store ./mystore
+generation gen-1 (activated through CURRENT)
+manifest manifest.sbm2, 2372 bytes, wire version 6
+layout predicate-ibk3-ptd1-sri2-tli1-oli2-merkle-v0
+blank-node profile (none recorded)
+term registry local-ibk3-ptd1-v0
+fixed-chunk Merkle commitment yes
+5 entries, 393775 bytes, 6455 rows
+generation directory holds 42 files, 846592 bytes
+
+#  rows  bytes   kind  graphs  predicate
+0  1800  110085  IBK3  -       http://www.wikidata.org/prop/direct/P31
+1  719   35535   IBK3  -       http://www.wikidata.org/prop/direct/P361
+...
+
+$ factoidal query ./mystore 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
+mode ibk3-paged-merkle-full-manifest(5), 5 artifacts, 393775 bytes read, plan declares 6455 block rows
+n
+"6455"^^<http://www.w3.org/2001/XMLSchema#integer>
+1 row
+```
+
+`STORE` is a collection root: the directory holding `CURRENT`. The plan
+line goes to stderr, so stdout carries only the result; `--quiet`
+removes it.
+
+| Option | What it does |
+|---|---|
+| `--format table` | default; a human display of the results |
+| `--format json` | SELECT prints the engine's SPARQL 1.1 Query Results JSON; ASK and CONSTRUCT print the operation's envelope |
+| `--format nquads` | CONSTRUCT only: the graph the engine serialized |
+| `--format turtle` | CONSTRUCT only: that graph through the engine's own Turtle writer |
+| `--explain` | print the artifacts the query needs and the open mode, and stop |
+| `--limit N` | print at most N table rows; the total is always named |
+| `--file PATH` | read the query text from a file |
+| `--generation NAME` | read that generation rather than the activated one |
+
+Under Deno, run the file directly; `inspect` and `query` need only
+`--allow-read`:
+
+```console
+$ deno run --allow-read node_modules/@factoidal/core/bin/factoidal.mjs query ./mystore 'ASK { ?s ?p ?o }'
+```
+
+### What the command answers for, and what it does not
+
+* **Every artifact is verified.** The engine refuses the whole query
+  when a block's bytes do not hash to the SHA-256 the manifest commits,
+  and names the artifact.
+* **Three caps.** One call reads at most 64 artifacts, 8388608 artifact
+  bytes and 100000 rows. A query over any of them is refused before a
+  single file is read, with the cap and the value named. Nothing is
+  truncated.
+* **Committed artifacts only.** A store carrying uncompacted delta-log
+  updates is not served by this path; use the native `l4block-*` tools.
+* **`pack`, `activate`, `update` and `compact` exit 3.** They need
+  WebAssembly operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+* **Node's WebAssembly frame budget.** Some evaluator paths recurse once
+  per row. Measured 2026-09-03 on a 6455-row store, `SELECT ?s ?p ?o
+  WHERE { ?s ?p ?o }` overflows the stack under Node's default while
+  `SELECT *`, or the same query with a `LIMIT`, does not, and Deno
+  clears all of them. The command reports it and exits 1 rather than
+  crashing; `node --stack-size=4000` clears it.
+
+Measured 2026-09-03 on macOS arm64, the 6455-triple `sequence_variant`
+store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
+including start-up: 220 ms through this command, 33 ms through the
+native `l4block-id-v3-query`.
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to

package/bin/engine.mjs

@@ -0,0 +1,124 @@
+// Loading the Lean engine (L4Factoidal compiled to WebAssembly) for the
+// `factoidal` command, under Node and under Deno.
+// https://github.com/danbri/factoidal/issues/641
+//
+// This file resolves three files that must stay together and keep their
+// names -- l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm -- and
+// imports the loader. The Emscripten glue resolves the .wasm sidecar from
+// its own basename, so a rename breaks the load on every runtime
+// (skills/lean4-wasm-export, "the naming trap").
+//
+// `npm/factoidal/l4.js` does the same resolution for the CommonJS API. It
+// is not reused here because it is CommonJS and this command must load
+// under Deno without the require() compatibility path.
+//
+// WHAT THIS FILE IS ALLOWED TO DO
+// Find the engine, load it, and turn bytes into the hexadecimal string
+// two of the three store operations take as their small argument. It
+// makes no format decision: it never parses a manifest, verifies a
+// digest, decodes a block, or chooses an artifact.
+
+import { readWhole } from '../store-host/index.mjs'
+import { fileUrlToPath, joinPath } from '../store-host/paths.mjs'
+
+/** Read one environment variable, or null where reading it is refused. */
+function environment (name) {
+  try {
+    if (typeof globalThis.Deno !== 'undefined' && globalThis.Deno.env) {
+      const value = globalThis.Deno.env.get(name)
+      return typeof value === 'string' && value.length > 0 ? value : null
+    }
+  } catch (_error) {
+    // Deno without --allow-env. An absent override is the normal case.
+    return null
+  }
+  const value = globalThis.process && globalThis.process.env
+    ? globalThis.process.env[name]
+    : undefined
+  return typeof value === 'string' && value.length > 0 ? value : null
+}
+
+function readable (path) {
+  try {
+    readWhole(path)
+    return true
+  } catch (_error) {
+    return false
+  }
+}
+
+/** The package directory that holds this command. */
+export function packageDirectory () {
+  return fileUrlToPath(new URL('..', import.meta.url).href).replace(/\/$/, '')
+}
+
+/**
+ * The loader file of the first engine source that exists, or null.
+ *
+ * Order, first hit wins:
+ *   1. this package's own l4-assets/ (what `npm install` gives);
+ *   2. $FACTOIDAL_L4_ASSETS (a custom deployment);
+ *   3. the repository checkout layout, docs/web/hub/assets/l4/.
+ */
+export function resolveEngine () {
+  const inPackage = joinPath(packageDirectory(), 'l4-assets/l4factoidal.js')
+  if (readable(inPackage)) return inPackage
+  const override = environment('FACTOIDAL_L4_ASSETS')
+  if (override !== null) {
+    const path = joinPath(override, 'l4factoidal.js')
+    if (readable(path)) return path
+    throw new Error(`FACTOIDAL_L4_ASSETS=${override} holds no l4factoidal.js`)
+  }
+  const inCheckout = joinPath(packageDirectory(),
+    '../../docs/web/hub/assets/l4/l4factoidal.js')
+  if (readable(inCheckout)) return inCheckout
+  return null
+}
+
+let enginePromise = null
+
+/**
+ * Load the engine once and return its handle: `call(op, args)` and
+ * `callBlob(op, args, bytes)`.
+ */
+export function loadEngine () {
+  if (enginePromise !== null) return enginePromise
+  enginePromise = (async () => {
+    const loaderPath = resolveEngine()
+    if (loaderPath === null) {
+      throw new Error(
+        'the Lean engine assets are missing. They normally ship in this ' +
+        "package's own l4-assets/ directory; if it is absent this install " +
+        'is incomplete. Otherwise set FACTOIDAL_L4_ASSETS to a directory ' +
+        'holding l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm.')
+    }
+    const url = loaderPath.startsWith('file://')
+      ? loaderPath
+      : 'file://' + encodeURI(loaderPath).replace(/#/g, '%23')
+    const module = await import(url)
+    return module.loadL4()
+  })()
+  return enginePromise
+}
+
+const HEX = (() => {
+  const table = new Array(256)
+  for (let byte = 0; byte < 256; byte += 1) {
+    table[byte] = byte.toString(16).padStart(2, '0')
+  }
+  return table
+})()
+
+/**
+ * Lowercase hexadecimal of a byte array. This is a transport encoding for
+ * the manifest argument of `storeManifestInspect` and `storeQueryPlan`,
+ * not an interpretation of the bytes. Artifact bytes never take this
+ * path: they cross raw through `callBlob`.
+ * @param {Uint8Array} bytes
+ * @returns {string}
+ */
+export function hexOfBytes (bytes) {
+  const parts = new Array(bytes.length)
+  for (let index = 0; index < bytes.length; index += 1) parts[index] = HEX[bytes[index]]
+  return parts.join('')
+}