@factoidal/core 0.2.0 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,127 @@
 # Changelog
 
+## 0.4.0 — 2026-09-04
+
+The package builds a store of its own. `pack` and `activate` join
+`inspect` and `query`, so `npm install @factoidal/core` gives a complete
+RDF store — import, activate, query — with no native binary on the
+machine. https://github.com/danbri/factoidal/issues/641
+
+- **`factoidal pack INPUT OUTPUT`** builds one immutable Shardborough
+  generation from a Turtle, TriG, N-Triples or N-Quads file, streaming
+  it in 65,536-byte chunks through the Lean engine's WebAssembly module.
+  The generation is BYTE-IDENTICAL to what the native
+  `l4block-shard-pack` writes: `diff -r` is empty for the `ibk3` triple
+  layout and for the `ibk4` quad layout, on inputs up to 888,949
+  triples. `--layout ibk3|ibk4`, `--syntax`, `--base`.
+- **`factoidal activate STORE GENERATION`** verifies every artifact
+  against the SHA-256 the manifest commits, and every cross-artifact
+  relation, then replaces `CURRENT` atomically. A generation that fails
+  verification never becomes current.
+- **`factoidal sample-store`** prints the path of a store this package
+  now carries, so a fresh install answers a SPARQL query with nothing
+  else to download: 4,434 triples in 13 predicate blocks, five IPTC
+  NewsCodes vocabularies under CC BY 4.0. Also exported as
+  `@factoidal/core/sample-store`. See NOTICE.
+- The engine gained a raw byte path out of the module
+  (`l4_call_blob_io`), so artifact bytes cross the boundary with no
+  encoding. Hexadecimal doubled them; base64 was refused. Measured on
+  the read path: 242,416 bytes and 96 ms hexadecimal against 4,893 bytes
+  and 70 ms raw.
+- The pack hashes with HACL* SHA-256, the same primitive the native
+  packer uses. It hashed with the pure Lean SHA-256 in development,
+  which cost 3.3 times: 104 s against 31 s on 888,949 triples. Now at
+  parity, 29.24 s against the native packer's 29.38 s.
+- `packBegin` takes a base IRI, defaulted by the command to
+  `file://<input>` so relative IRIs resolve exactly as the native packer
+  resolves them. `--base ''` asks for no base, which makes a relative
+  IRI a parse error rather than a silently different term.
+- `pack` on a syntax the streaming fold cannot read now says so by name
+  rather than raising an unhandled error.
+- `store-host` gained `readChunk` (a short read means end of file, not
+  an error), `writeNew` (create and fsync, refusing an existing file so
+  a name collision in a generation is reported) and `makeDirectory`, on
+  Node and on Deno both.
+- `factoidal activate` runs on the raised stack too. Verification decodes
+  the same blocks the pack encoded, so it recurses as deep; the worker was
+  given to `pack` alone at first, and a 112,742-row generation packed
+  successfully and then failed to activate with `Maximum call stack size
+  exceeded`, leaving a store that could be built and not opened. Found by
+  installing the tarball and running the command, which is why that step
+  is in the release procedure. `tests/store-host/cli.mjs` now gates
+  pack-then-activate on both runtimes.
+- `factoidal pack` no longer needs a runtime flag. The pack fold recurses
+  deeper than either runtime's default call stack allows, so an input
+  above roughly 0.5 MB ended with `Maximum call stack size exceeded`
+  (https://github.com/danbri/factoidal/issues/649). Under Node the pack
+  now runs on a `worker_threads` worker with a 64 MiB stack; under Deno
+  the command re-executes itself once with
+  `--v8-flags=--stack-size=65536`, which needs `--allow-run` and
+  `--allow-env` in addition to `--allow-read` and `--allow-write`.
+  `gene.ttl`, 17,363,312 bytes and 888,949 triples, packs on the default
+  stack of both runtimes, byte-identical to `l4block-shard-pack`.
+  `--no-worker` forces the in-process path, which still reports the
+  frame budget and the flag that raises it rather than crashing. A
+  browser tab has a fixed frame budget and no flag, so this does not
+  make an in-page packer possible.
+
+Known limits, measured:
+
+- `update` and `compact` still exit 3. The delta-log operations are
+  stage 4 of https://github.com/danbri/factoidal/issues/641.
+- The `ibk4` quad layout reads the whole source rather than streaming,
+  because a quad block commits a graph-set summary over the entire
+  input. The wasm packer refuses a quad file above 128 MiB.
+  https://github.com/danbri/factoidal/issues/650
+- A query plan is refused above 64 artifacts, 8,388,608 blob bytes or
+  100,000 rows. https://github.com/danbri/factoidal/issues/648
+- Packing in a browser tab is limited to about 7,800 distinct terms in
+  one block, whatever the file size, and no host flag raises it.
+  https://github.com/danbri/factoidal/issues/647
+
+## 0.3.0 — 2026-09-03
+
+- The `factoidal` command answers SPARQL against a persisted
+  Shardborough store with no native binary. `factoidal inspect STORE`
+  decodes the manifest through the engine's `storeManifestInspect`
+  operation and prints the wire version, layout, blank-node profile and
+  the entry table; `factoidal query STORE 'SELECT ...'` asks the engine
+  which artifacts the query needs, reads exactly those, and hands their
+  bytes to `storeQuery` through one WebAssembly heap buffer with no
+  encoding. Every artifact is verified against the SHA-256 the manifest
+  commits before an answer is given. Formats: `table`, `json`,
+  `nquads`, `turtle`; `--explain` prints the artifact plan. Runs under
+  Node and under Deno (`--allow-read`).
+- The three caps of the store query operation — 64 artifacts, 8388608
+  artifact bytes, 100000 rows — are reported with the cap, the value
+  and a next step, and are decided before a single file is read.
+  Nothing is truncated.
+- `pack`, `activate`, `update` and `compact` still exit 3; they need
+  operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+- A large piped result is no longer truncated: the command sets
+  `process.exitCode` instead of calling `process.exit()`, which dropped
+  everything past the 64 KiB pipe boundary.
+
+## 0.3.0 — Lean block-worker preview
+
+- The bundled Lean-derived WASM artifact exports
+  `scanIBK2Predicate(ibk2Hex, predicateIri)` through `factoidal/l4`.
+  It validates a canonical IBK2 block and executes its selective predicate
+  scan, returning N-Triples plus a row count.  This is a narrow physical
+  helper for the Shardborough work, not full SPARQL execution inside a storage
+  backend and not a high-throughput buffer ABI (the current hex transport is
+  intentionally diagnostic).
+- The regenerated Lean artifact is synchronized across the package's
+  `l4-assets/` and the Hub build, and is exercised against the music IBK2
+  fixture by `tools/wasm-ibk2-smoke.mjs` in the repository.
+- The current-format companion
+  `scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` validates and
+  scans complete predicate-local IBK3 artifacts. Its mandatory source/dataset
+  scope preserves one blank node across blocks from the same RDF import while
+  keeping same-spelled labels in unrelated inputs apart. Hub post 51 composes
+  three real IBK3 scans into an editable Lean-WASM SPARQL query.
+
 ## 0.2.0 — both engines in one package, with a backend selector
 
 - The Lean 4 engine (`L4Factoidal`, wasm) now ships INSIDE

package/NOTICE

@@ -0,0 +1,31 @@
+@factoidal/core — third-party content notices
+=============================================
+
+The code in this package is Apache-2.0; see LICENSE.
+
+sample-store/
+-------------
+
+The bundled sample Shardborough store holds five IPTC NewsCodes
+vocabularies, converted from the publisher's RDF/XML and packed into
+IBK3 predicate blocks:
+
+  spamfstat, videoqualifier, subjectqualifier, videocodec, spct
+
+  Publisher: International Press Telecommunications Council
+             https://iptc.org/
+  Licence:   Creative Commons Attribution 4.0 International (CC BY 4.0)
+             https://creativecommons.org/licenses/by/4.0/
+  Obtained through: https://github.com/danbri/skosdex third_party/skos
+
+The IRIs, labels and structure are the IPTC's. The block layout, the
+dictionary, the indexes and the manifest are this package's encoding of
+them. Redistribution of the vocabulary content stays under CC BY 4.0.
+
+hacl-wasm/
+----------
+
+HACL* verified cryptographic primitives, from
+https://github.com/hacl-star/hacl-star, Apache-2.0 / MIT. See the
+provenance recorded in the Factoidal repository under
+skills/node-crypto-haclstar-vc-wasm-build/.

package/README.md

@@ -1,6 +1,6 @@
 # @factoidal/core
 
-> First published cut (0.1.0). The API surface is early and may
+> Release 0.3.0. The API surface is early and may
 > change before 1.0. This package was previously developed in-tree
 > under the placeholder names `factoidal` and `@danbri/foafos`; it was
 > never published under those names. See [CHANGELOG.md](CHANGELOG.md).
@@ -312,6 +312,16 @@ Two engines now ship in one package. `factoidal` and `factoidal/wasm`
 are the F\*-extracted engine, unchanged. The subpaths below are the
 Lean 4 engine (`L4Factoidal`, compiled to wasm).
 
+`factoidal/l4` also exposes two deliberately narrow physical helpers:
+`scanIBK2Predicate(ibk2Hex, predicateIri)` for the predecessor format and
+`scanIBK3Predicate(ibk3Hex, predicateIri, blankNodeScope)` for the current
+predicate-local format. They validate one canonical RDF block and scan its
+named predicate, returning N-Triples and a row count. The IBK3 source scope
+must be shared across blocks partitioned from one RDF import unit and differ
+across unrelated units; this preserves document-scoped blank-node identity
+when fragments are composed. The hexadecimal argument is a portable
+diagnostic ABI, not the intended high-throughput buffer interface.
+
 ```js
 const l4 = require('factoidal/l4-core');       // Lean engine, same API shape
 const { select } = require('factoidal/select'); // choose an engine per call
@@ -358,6 +368,126 @@ model-theory modules run to about 22,000 lines — but only these reach
 JavaScript today. Everything else in the Lean tree is used through
 `parse`/`query`/`closure`, or not exposed at all.
 
+## The `factoidal` command: querying a persisted store
+
+Installing this package puts a `factoidal` command on PATH. It reads a
+**Shardborough** store — the on-disk format the Lean `l4block-*` tools
+write — with no native binary: JavaScript reads the files and moves the
+bytes, and the Lean engine running as WebAssembly makes every format
+decision (parsing the manifest, choosing the blocks, verifying their
+SHA-256, evaluating the SPARQL).
+
+> This command is not the native F\* `factoidal` binary that the API
+> table below refers to. That one is `bin/<platform>/factoidal` in the
+> repository and takes subcommands such as `shex` and `compact`. This
+> one takes `version`, `sample-store`, `inspect` and `query`.
+
+### First query, with nothing else to download
+
+The package carries an activated store, so a fresh install answers a
+SPARQL query at once:
+
+```console
+$ npm install @factoidal/core
+$ npx factoidal query "$(npx factoidal sample-store)" \
+    'SELECT ?c ?l
+     WHERE { ?c <http://www.w3.org/2004/02/skos/core#inScheme>
+                <http://cv.iptc.org/newscodes/videocodec/> ;
+                <http://www.w3.org/2004/02/skos/core#prefLabel> ?l .
+             FILTER(langMatches(lang(?l), "en")) }
+     LIMIT 4'
+c                                               l
+<http://cv.iptc.org/newscodes/videocodec/c001>  "Analogue Black and White"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c002>  "PAL"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c003>  "NTSC"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c004>  "SECAM"@en-gb
+```
+
+`factoidal sample-store` prints the path; `--json` adds what was
+recorded when the store was packed. From JavaScript:
+
+```js
+import { sampleStorePath, sampleStoreFacts } from '@factoidal/core/sample-store'
+```
+
+The store holds 4,434 triples in 13 predicate blocks: five IPTC
+NewsCodes vocabularies, published by the IPTC under CC BY 4.0 and taken
+from [danbri/skosdex](https://github.com/danbri/skosdex). See `NOTICE`.
+
+### Any other store
+
+```console
+$ factoidal inspect ./mystore
+store ./mystore
+generation gen-1 (activated through CURRENT)
+manifest manifest.sbm2, 2372 bytes, wire version 6
+layout predicate-ibk3-ptd1-sri2-tli1-oli2-merkle-v0
+blank-node profile (none recorded)
+term registry local-ibk3-ptd1-v0
+fixed-chunk Merkle commitment yes
+5 entries, 393775 bytes, 6455 rows
+generation directory holds 42 files, 846592 bytes
+
+#  rows  bytes   kind  graphs  predicate
+0  1800  110085  IBK3  -       http://www.wikidata.org/prop/direct/P31
+1  719   35535   IBK3  -       http://www.wikidata.org/prop/direct/P361
+...
+
+$ factoidal query ./mystore 'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
+mode ibk3-paged-merkle-full-manifest(5), 5 artifacts, 393775 bytes read, plan declares 6455 block rows
+n
+"6455"^^<http://www.w3.org/2001/XMLSchema#integer>
+1 row
+```
+
+`STORE` is a collection root: the directory holding `CURRENT`. The plan
+line goes to stderr, so stdout carries only the result; `--quiet`
+removes it.
+
+| Option | What it does |
+|---|---|
+| `--format table` | default; a human display of the results |
+| `--format json` | SELECT prints the engine's SPARQL 1.1 Query Results JSON; ASK and CONSTRUCT print the operation's envelope |
+| `--format nquads` | CONSTRUCT only: the graph the engine serialized |
+| `--format turtle` | CONSTRUCT only: that graph through the engine's own Turtle writer |
+| `--explain` | print the artifacts the query needs and the open mode, and stop |
+| `--limit N` | print at most N table rows; the total is always named |
+| `--file PATH` | read the query text from a file |
+| `--generation NAME` | read that generation rather than the activated one |
+
+Under Deno, run the file directly; `inspect` and `query` need only
+`--allow-read`:
+
+```console
+$ deno run --allow-read node_modules/@factoidal/core/bin/factoidal.mjs query ./mystore 'ASK { ?s ?p ?o }'
+```
+
+### What the command answers for, and what it does not
+
+* **Every artifact is verified.** The engine refuses the whole query
+  when a block's bytes do not hash to the SHA-256 the manifest commits,
+  and names the artifact.
+* **Three caps.** One call reads at most 64 artifacts, 8388608 artifact
+  bytes and 100000 rows. A query over any of them is refused before a
+  single file is read, with the cap and the value named. Nothing is
+  truncated.
+* **Committed artifacts only.** A store carrying uncompacted delta-log
+  updates is not served by this path; use the native `l4block-*` tools.
+* **`pack`, `activate`, `update` and `compact` exit 3.** They need
+  WebAssembly operations that do not exist yet
+  (https://github.com/danbri/factoidal/issues/641).
+* **Node's WebAssembly frame budget.** Some evaluator paths recurse once
+  per row. Measured 2026-09-03 on a 6455-row store, `SELECT ?s ?p ?o
+  WHERE { ?s ?p ?o }` overflows the stack under Node's default while
+  `SELECT *`, or the same query with a `LIMIT`, does not, and Deno
+  clears all of them. The command reports it and exits 1 rather than
+  crashing; `node --stack-size=4000` clears it.
+
+Measured 2026-09-03 on macOS arm64, the 6455-triple `sequence_variant`
+store, `SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }`, whole process
+including start-up: 220 ms through this command, 33 ms through the
+native `l4block-id-v3-query`.
+
 ## API (draft)
 
 The `factoidal` CLI (`bin/factoidal-cli/factoidal_cli.ml`, built to

package/bin/engine.mjs

@@ -0,0 +1,124 @@
+// Loading the Lean engine (L4Factoidal compiled to WebAssembly) for the
+// `factoidal` command, under Node and under Deno.
+// https://github.com/danbri/factoidal/issues/641
+//
+// This file resolves three files that must stay together and keep their
+// names -- l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm -- and
+// imports the loader. The Emscripten glue resolves the .wasm sidecar from
+// its own basename, so a rename breaks the load on every runtime
+// (skills/lean4-wasm-export, "the naming trap").
+//
+// `npm/factoidal/l4.js` does the same resolution for the CommonJS API. It
+// is not reused here because it is CommonJS and this command must load
+// under Deno without the require() compatibility path.
+//
+// WHAT THIS FILE IS ALLOWED TO DO
+// Find the engine, load it, and turn bytes into the hexadecimal string
+// two of the three store operations take as their small argument. It
+// makes no format decision: it never parses a manifest, verifies a
+// digest, decodes a block, or chooses an artifact.
+
+import { readWhole } from '../store-host/index.mjs'
+import { fileUrlToPath, joinPath } from '../store-host/paths.mjs'
+
+/** Read one environment variable, or null where reading it is refused. */
+function environment (name) {
+  try {
+    if (typeof globalThis.Deno !== 'undefined' && globalThis.Deno.env) {
+      const value = globalThis.Deno.env.get(name)
+      return typeof value === 'string' && value.length > 0 ? value : null
+    }
+  } catch (_error) {
+    // Deno without --allow-env. An absent override is the normal case.
+    return null
+  }
+  const value = globalThis.process && globalThis.process.env
+    ? globalThis.process.env[name]
+    : undefined
+  return typeof value === 'string' && value.length > 0 ? value : null
+}
+
+function readable (path) {
+  try {
+    readWhole(path)
+    return true
+  } catch (_error) {
+    return false
+  }
+}
+
+/** The package directory that holds this command. */
+export function packageDirectory () {
+  return fileUrlToPath(new URL('..', import.meta.url).href).replace(/\/$/, '')
+}
+
+/**
+ * The loader file of the first engine source that exists, or null.
+ *
+ * Order, first hit wins:
+ *   1. this package's own l4-assets/ (what `npm install` gives);
+ *   2. $FACTOIDAL_L4_ASSETS (a custom deployment);
+ *   3. the repository checkout layout, docs/web/hub/assets/l4/.
+ */
+export function resolveEngine () {
+  const inPackage = joinPath(packageDirectory(), 'l4-assets/l4factoidal.js')
+  if (readable(inPackage)) return inPackage
+  const override = environment('FACTOIDAL_L4_ASSETS')
+  if (override !== null) {
+    const path = joinPath(override, 'l4factoidal.js')
+    if (readable(path)) return path
+    throw new Error(`FACTOIDAL_L4_ASSETS=${override} holds no l4factoidal.js`)
+  }
+  const inCheckout = joinPath(packageDirectory(),
+    '../../docs/web/hub/assets/l4/l4factoidal.js')
+  if (readable(inCheckout)) return inCheckout
+  return null
+}
+
+let enginePromise = null
+
+/**
+ * Load the engine once and return its handle: `call(op, args)` and
+ * `callBlob(op, args, bytes)`.
+ */
+export function loadEngine () {
+  if (enginePromise !== null) return enginePromise
+  enginePromise = (async () => {
+    const loaderPath = resolveEngine()
+    if (loaderPath === null) {
+      throw new Error(
+        'the Lean engine assets are missing. They normally ship in this ' +
+        "package's own l4-assets/ directory; if it is absent this install " +
+        'is incomplete. Otherwise set FACTOIDAL_L4_ASSETS to a directory ' +
+        'holding l4factoidal.js, l4factoidal.mjs and l4factoidal.wasm.')
+    }
+    const url = loaderPath.startsWith('file://')
+      ? loaderPath
+      : 'file://' + encodeURI(loaderPath).replace(/#/g, '%23')
+    const module = await import(url)
+    return module.loadL4()
+  })()
+  return enginePromise
+}
+
+const HEX = (() => {
+  const table = new Array(256)
+  for (let byte = 0; byte < 256; byte += 1) {
+    table[byte] = byte.toString(16).padStart(2, '0')
+  }
+  return table
+})()
+
+/**
+ * Lowercase hexadecimal of a byte array. This is a transport encoding for
+ * the manifest argument of `storeManifestInspect` and `storeQueryPlan`,
+ * not an interpretation of the bytes. Artifact bytes never take this
+ * path: they cross raw through `callBlob`.
+ * @param {Uint8Array} bytes
+ * @returns {string}
+ */
+export function hexOfBytes (bytes) {
+  const parts = new Array(bytes.length)
+  for (let index = 0; index < bytes.length; index += 1) parts[index] = HEX[bytes[index]]
+  return parts.join('')
+}