@factoidal/core 0.3.0 → 0.4.0

package/CHANGELOG.md

@@ -1,5 +1,84 @@
 # Changelog
 
+## 0.4.0 — 2026-09-04
+
+The package builds a store of its own. `pack` and `activate` join
+`inspect` and `query`, so `npm install @factoidal/core` gives a complete
+RDF store — import, activate, query — with no native binary on the
+machine. https://github.com/danbri/factoidal/issues/641
+
+- **`factoidal pack INPUT OUTPUT`** builds one immutable Shardborough
+  generation from a Turtle, TriG, N-Triples or N-Quads file, streaming
+  it in 65,536-byte chunks through the Lean engine's WebAssembly module.
+  The generation is BYTE-IDENTICAL to what the native
+  `l4block-shard-pack` writes: `diff -r` is empty for the `ibk3` triple
+  layout and for the `ibk4` quad layout, on inputs up to 888,949
+  triples. `--layout ibk3|ibk4`, `--syntax`, `--base`.
+- **`factoidal activate STORE GENERATION`** verifies every artifact
+  against the SHA-256 the manifest commits, and every cross-artifact
+  relation, then replaces `CURRENT` atomically. A generation that fails
+  verification never becomes current.
+- **`factoidal sample-store`** prints the path of a store this package
+  now carries, so a fresh install answers a SPARQL query with nothing
+  else to download: 4,434 triples in 13 predicate blocks, five IPTC
+  NewsCodes vocabularies under CC BY 4.0. Also exported as
+  `@factoidal/core/sample-store`. See NOTICE.
+- The engine gained a raw byte path out of the module
+  (`l4_call_blob_io`), so artifact bytes cross the boundary with no
+  encoding. Hexadecimal doubled them; base64 was refused. Measured on
+  the read path: 242,416 bytes and 96 ms hexadecimal against 4,893 bytes
+  and 70 ms raw.
+- The pack hashes with HACL* SHA-256, the same primitive the native
+  packer uses. It hashed with the pure Lean SHA-256 in development,
+  which cost 3.3 times: 104 s against 31 s on 888,949 triples. Now at
+  parity, 29.24 s against the native packer's 29.38 s.
+- `packBegin` takes a base IRI, defaulted by the command to
+  `file://<input>` so relative IRIs resolve exactly as the native packer
+  resolves them. `--base ''` asks for no base, which makes a relative
+  IRI a parse error rather than a silently different term.
+- `pack` on a syntax the streaming fold cannot read now says so by name
+  rather than raising an unhandled error.
+- `store-host` gained `readChunk` (a short read means end of file, not
+  an error), `writeNew` (create and fsync, refusing an existing file so
+  a name collision in a generation is reported) and `makeDirectory`, on
+  Node and on Deno both.
+- `factoidal activate` runs on the raised stack too. Verification decodes
+  the same blocks the pack encoded, so it recurses as deep; the worker was
+  given to `pack` alone at first, and a 112,742-row generation packed
+  successfully and then failed to activate with `Maximum call stack size
+  exceeded`, leaving a store that could be built and not opened. Found by
+  installing the tarball and running the command, which is why that step
+  is in the release procedure. `tests/store-host/cli.mjs` now gates
+  pack-then-activate on both runtimes.
+- `factoidal pack` no longer needs a runtime flag. The pack fold recurses
+  deeper than either runtime's default call stack allows, so an input
+  above roughly 0.5 MB ended with `Maximum call stack size exceeded`
+  (https://github.com/danbri/factoidal/issues/649). Under Node the pack
+  now runs on a `worker_threads` worker with a 64 MiB stack; under Deno
+  the command re-executes itself once with
+  `--v8-flags=--stack-size=65536`, which needs `--allow-run` and
+  `--allow-env` in addition to `--allow-read` and `--allow-write`.
+  `gene.ttl`, 17,363,312 bytes and 888,949 triples, packs on the default
+  stack of both runtimes, byte-identical to `l4block-shard-pack`.
+  `--no-worker` forces the in-process path, which still reports the
+  frame budget and the flag that raises it rather than crashing. A
+  browser tab has a fixed frame budget and no flag, so this does not
+  make an in-page packer possible.
+
+Known limits, measured:
+
+- `update` and `compact` still exit 3. The delta-log operations are
+  stage 4 of https://github.com/danbri/factoidal/issues/641.
+- The `ibk4` quad layout reads the whole source rather than streaming,
+  because a quad block commits a graph-set summary over the entire
+  input. The wasm packer refuses a quad file above 128 MiB.
+  https://github.com/danbri/factoidal/issues/650
+- A query plan is refused above 64 artifacts, 8,388,608 blob bytes or
+  100,000 rows. https://github.com/danbri/factoidal/issues/648
+- Packing in a browser tab is limited to about 7,800 distinct terms in
+  one block, whatever the file size, and no host flag raises it.
+  https://github.com/danbri/factoidal/issues/647
+
 ## 0.3.0 — 2026-09-03
 
 - The `factoidal` command answers SPARQL against a persisted

package/NOTICE

@@ -0,0 +1,31 @@
+@factoidal/core — third-party content notices
+=============================================
+
+The code in this package is Apache-2.0; see LICENSE.
+
+sample-store/
+-------------
+
+The bundled sample Shardborough store holds five IPTC NewsCodes
+vocabularies, converted from the publisher's RDF/XML and packed into
+IBK3 predicate blocks:
+
+  spamfstat, videoqualifier, subjectqualifier, videocodec, spct
+
+  Publisher: International Press Telecommunications Council
+             https://iptc.org/
+  Licence:   Creative Commons Attribution 4.0 International (CC BY 4.0)
+             https://creativecommons.org/licenses/by/4.0/
+  Obtained through: https://github.com/danbri/skosdex third_party/skos
+
+The IRIs, labels and structure are the IPTC's. The block layout, the
+dictionary, the indexes and the manifest are this package's encoding of
+them. Redistribution of the vocabulary content stays under CC BY 4.0.
+
+hacl-wasm/
+----------
+
+HACL* verified cryptographic primitives, from
+https://github.com/hacl-star/hacl-star, Apache-2.0 / MIT. See the
+provenance recorded in the Factoidal repository under
+skills/node-crypto-haclstar-vc-wasm-build/.

package/README.md

@@ -380,7 +380,41 @@ SHA-256, evaluating the SPARQL).
 > This command is not the native F\* `factoidal` binary that the API
 > table below refers to. That one is `bin/<platform>/factoidal` in the
 > repository and takes subcommands such as `shex` and `compact`. This
-> one takes `version`, `inspect` and `query`.
+> one takes `version`, `sample-store`, `inspect` and `query`.
+
+### First query, with nothing else to download
+
+The package carries an activated store, so a fresh install answers a
+SPARQL query at once:
+
+```console
+$ npm install @factoidal/core
+$ npx factoidal query "$(npx factoidal sample-store)" \
+    'SELECT ?c ?l
+     WHERE { ?c <http://www.w3.org/2004/02/skos/core#inScheme>
+                <http://cv.iptc.org/newscodes/videocodec/> ;
+                <http://www.w3.org/2004/02/skos/core#prefLabel> ?l .
+             FILTER(langMatches(lang(?l), "en")) }
+     LIMIT 4'
+c                                               l
+<http://cv.iptc.org/newscodes/videocodec/c001>  "Analogue Black and White"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c002>  "PAL"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c003>  "NTSC"@en-gb
+<http://cv.iptc.org/newscodes/videocodec/c004>  "SECAM"@en-gb
+```
+
+`factoidal sample-store` prints the path; `--json` adds what was
+recorded when the store was packed. From JavaScript:
+
+```js
+import { sampleStorePath, sampleStoreFacts } from '@factoidal/core/sample-store'
+```
+
+The store holds 4,434 triples in 13 predicate blocks: five IPTC
+NewsCodes vocabularies, published by the IPTC under CC BY 4.0 and taken
+from [danbri/skosdex](https://github.com/danbri/skosdex). See `NOTICE`.
+
+### Any other store
 
 ```console
 $ factoidal inspect ./mystore

package/bin/factoidal.mjs

@@ -20,13 +20,17 @@
 // `--format turtle` all print documents the engine produced.
 
 import {
-  StoreHostError, listGeneration, readWhole, runtime
+  StoreHostError, atomicReplace, listGeneration, makeDirectory, readWhole,
+  runtime
 } from '../store-host/index.mjs'
 import { fileUrlToPath, joinPath } from '../store-host/paths.mjs'
 import { loadEngine } from './engine.mjs'
+import { sampleStoreFacts, sampleStorePath } from '../sample-store.mjs'
+import { PackError, packSupported, verifyGeneration } from './pack.mjs'
+import { denoReexec, isStackOverflow, runPack } from './pack-host.mjs'
 import {
-  StoreOperationError, inspectManifest, openStore, planQuery, queryStore,
-  turtleOfNQuads
+  STACK_REMEDY, StoreOperationError, inspectManifest, openStore, planQuery,
+  queryStore, stackLimitAdvice, turtleOfNQuads
 } from './store.mjs'
 
 const EXIT_OK = 0
@@ -34,6 +38,41 @@ const EXIT_FAILURE = 1
 const EXIT_USAGE = 2
 const EXIT_NOT_WIRED = 3
 
+// Progress is reported about every 16 MiB; the packer feeds 65,536 bytes
+// a time, so this is the window that catches one feed per report.
+const FEED_PROGRESS = 65536
+
+const PACK_LAYOUTS = ['ibk3', 'ibk4']
+const PACK_SYNTAXES = ['turtle', 'trig', 'nquads', 'ntriples']
+const PACK_SUFFIXES = [
+  ['.ttl', 'turtle'], ['.turtle', 'turtle'],
+  ['.trig', 'trig'],
+  ['.nq', 'nquads'], ['.nquads', 'nquads'],
+  ['.nt', 'ntriples'], ['.ntriples', 'ntriples']
+]
+
+// Suffixes the engine parses elsewhere but the packer's streaming fold does
+// not read. Naming them is better than "cannot tell the syntax from its
+// name", which sends the reader looking for a --syntax value that does not
+// exist.
+const PACK_UNSUPPORTED_SUFFIXES = [
+  ['.rdf', 'RDF/XML'], ['.owl', 'RDF/XML'], ['.xml', 'RDF/XML'],
+  ['.jsonld', 'JSON-LD'], ['.json', 'JSON-LD'], ['.n3', 'Notation3'],
+  ['.csv', 'CSV'], ['.tsv', 'TSV'], ['.hdt', 'HDT']
+]
+
+/** The parent of a path, and its last component. The `activate` hint
+ *  printed after a pack needs both; neither is a format decision. */
+function dirOf (path) {
+  const cut = path.replace(/[/\\]+$/, '').lastIndexOf('/')
+  return cut <= 0 ? '.' : path.slice(0, cut)
+}
+function nameOf (path) {
+  const trimmed = path.replace(/[/\\]+$/, '')
+  const cut = trimmed.lastIndexOf('/')
+  return cut < 0 ? trimmed : trimmed.slice(cut + 1)
+}
+
 const ISSUE = 'https://github.com/danbri/factoidal/issues/641'
 
 const isDeno = typeof globalThis.Deno !== 'undefined'
@@ -60,6 +99,7 @@ usage: factoidal <command> [options]
 
 commands:
   version                     print the package and engine versions
+  sample-store                print the path of the bundled sample store
   inspect  STORE              report what the activated manifest commits
   query    STORE [QUERY]      evaluate a SPARQL query against a store
   pack     INPUT OUTPUT       build one immutable generation from an RDF file
@@ -75,7 +115,10 @@ global options:
 exit codes:
   0 success   1 failure   2 usage error   3 not yet wired (${ISSUE})
 
-STORE is a collection root: the directory that holds CURRENT.`
+STORE is a collection root: the directory that holds CURRENT. This package
+carries one, so the first query needs no other download:
+
+  factoidal query "$(factoidal sample-store)" 'SELECT * WHERE { ?s ?p ?o } LIMIT 5'`
 
 const COMMAND_USAGE = {
   version: `factoidal version - print the package and engine versions
@@ -85,6 +128,21 @@ usage: factoidal version [--json]
 Prints the npm package version, the Lean engine's WebAssembly digest as
 recorded by its build, and which host-I/O implementation is loaded.`,
 
+  'sample-store': `factoidal sample-store - print the bundled store's path
+
+usage: factoidal sample-store [--json]
+
+Prints the collection root of the Shardborough store this package
+carries, so a fresh install can query something at once:
+
+  factoidal inspect "$(factoidal sample-store)"
+  factoidal query "$(factoidal sample-store)" \\
+    'SELECT (COUNT(*) AS ?n) WHERE { ?s ?p ?o }'
+
+The store holds five IPTC NewsCodes vocabularies (CC BY 4.0; see NOTICE)
+packed into IBK3 predicate blocks. --json adds what was recorded when it
+was packed.`,
+
   inspect: `factoidal inspect - report what a store's manifest commits
 
 usage: factoidal inspect STORE [--json] [--generation NAME]
@@ -157,7 +215,14 @@ options:
   --layout LAYOUT    ibk3 (triples, default) or ibk4 (quads)
   --syntax SYNTAX    turtle, trig or nquads; default from the file extension
   --chunk-bytes N    Merkle chunk size; default is the engine's
-  --json             emit one JSON object`,
+  --json             emit one JSON object
+  --no-worker        pack in this process instead of on a worker thread
+
+The pack fold recurses deeper than either runtime's default call stack
+allows, so it runs on a worker thread with a raised stack under Node, and
+under Deno the command re-executes itself once with a raised V8 stack
+(https://github.com/danbri/factoidal/issues/649). --no-worker turns both
+off; a pack above about half a megabyte of input then overflows.`,
 
   activate: `factoidal activate - make one generation the activated generation
 
@@ -239,9 +304,10 @@ class UsageError extends Error {}
 
 const VALUE_OPTIONS = {
   version: new Set([]),
+  'sample-store': new Set([]),
   inspect: new Set(['generation']),
   query: new Set(['query', 'file', 'format', 'limit', 'base', 'generation']),
-  pack: new Set(['layout', 'syntax', 'chunk-bytes']),
+  pack: new Set(['layout', 'syntax', 'chunk-bytes', 'base']),
   activate: new Set([]),
   update: new Set(['update', 'file']),
   compact: new Set([])
@@ -297,6 +363,16 @@ function commandVersion (options) {
   return EXIT_OK
 }
 
+function commandSampleStore (options) {
+  const path = sampleStorePath()
+  if (options.json === true) {
+    out(JSON.stringify({ path, ...sampleStoreFacts }, null, 2))
+    return EXIT_OK
+  }
+  out(path)
+  return EXIT_OK
+}
+
 // ------------------------------------------------------------ rendering
 
 /**
@@ -448,10 +524,7 @@ function reportStoreFailure (error) {
     err(`This query needs more of the store than one WebAssembly call may read: ${error.capValue} against a cap of ${error.capLimit}.`)
     err('Narrow the query - bind a predicate, or restrict the graph - or use the native l4block-* tools.')
   } else if (error.stackLimit) {
-    err('The runtime ran out of call stack inside the engine, not the store.')
-    err('Some evaluator paths recurse once per row, and a few thousand rows can')
-    err("exceed Node's default WebAssembly frame budget. Raise it with")
-    err('node --stack-size=4000, add a LIMIT, or run the query under Deno.')
+    for (const line of stackLimitAdvice(STACK_REMEDY.query)) err(line)
   } else if (error.digestKey !== null) {
     err(`The bytes of '${error.digestKey}' in the generation directory are not the bytes the manifest commits.`)
     err('The generation is damaged or was edited after it was packed; repack or restore it.')
@@ -561,14 +634,176 @@ function renderQueryResult (engine, result, format, limit, quiet) {
   return EXIT_FAILURE
 }
 
-function commandPack (positional, _options) {
+/** The syntax tag for an input, from --syntax or from the file name. The
+ *  engine is what actually decides how to read the bytes; this only picks
+ *  which of its parsers to name. */
+function packSyntax (input, options) {
+  if (typeof options.syntax === 'string') {
+    const syntax = options.syntax.toLowerCase()
+    if (PACK_SYNTAXES.indexOf(syntax) >= 0) return syntax
+    throw new UsageError(`--syntax ${options.syntax} is not one of ${PACK_SYNTAXES.join(', ')}`)
+  }
+  const lower = input.toLowerCase()
+  for (const [suffix, syntax] of PACK_SUFFIXES) {
+    if (lower.endsWith(suffix)) return syntax
+  }
+  for (const [suffix, name] of PACK_UNSUPPORTED_SUFFIXES) {
+    if (lower.endsWith(suffix)) {
+      throw new UsageError(
+        `pack does not read ${name}. The packer's streaming fold reads ` +
+        `${PACK_SYNTAXES.join(', ')} only. Convert the file first, for ` +
+        "example with: factoidal parse FILE --out nquads")
+    }
+  }
+  throw new UsageError(
+    `cannot tell the syntax of ${input} from its name; give --syntax ` +
+    `(${PACK_SYNTAXES.join(', ')})`)
+}
+
+/**
+ * The base IRI relative IRIs in the source resolve against.
+ *
+ * The native packer uses `file://<input>`, so this matches it by default
+ * and byte-identical output needs no flag. `--base` overrides it, and
+ * `--base ''` asks for no base, which turns a relative IRI into a parse
+ * error rather than a silently different term.
+ */
+function packBase (input, options) {
+  if (typeof options.base === 'string') return options.base
+  const absolute = input.startsWith('/') ? input : joinPath(currentDirectory(), input)
+  return 'file://' + absolute
+}
+
+/** The process's working directory, on Node and on Deno. */
+function currentDirectory () {
+  if (isDeno) return globalThis.Deno.cwd()
+  return process.cwd()
+}
+
+function packLayout (options) {
+  if (typeof options.layout !== 'string') return 'ibk3'
+  const layout = options.layout.toLowerCase()
+  if (PACK_LAYOUTS.indexOf(layout) >= 0) return layout
+  throw new UsageError(`--layout ${options.layout} is not one of ${PACK_LAYOUTS.join(', ')}`)
+}
+
+async function commandPack (positional, options) {
   if (positional.length !== 2) throw new UsageError('pack needs INPUT and OUTPUT')
-  return notWired('pack', 'The streaming pack operations are stage 3 of the milestone.')
+  const [input, output] = positional
+  const syntax = packSyntax(input, options)
+  const layout = packLayout(options)
+  // The pack fold needs a bigger call stack than either runtime gives by
+  // default (https://github.com/danbri/factoidal/issues/649). Under Node
+  // the work runs on a worker thread with a raised stack; under Deno the
+  // command re-executes itself once with --v8-flags=--stack-size, and
+  // this is where that happens, before any file is opened. --no-worker
+  // keeps the in-process path testable.
+  const host = { worker: options['no-worker'] !== true }
+  const reexec = await denoReexec(host)
+  if (reexec !== null) return reexec
+  makeDirectory(output)
+  const quiet = options.quiet === true
+  let answer
+  try {
+    answer = await runPack(
+      { kind: 'pack', input, output, syntax, layout, base: packBase(input, options) },
+      quiet
+        ? undefined
+        : (progress) => {
+            if (progress.bytesRead % (16 * 1024 * 1024) < FEED_PROGRESS) {
+              err(`${progress.pass}: ${progress.bytesRead} bytes read, ${progress.artifacts} artifacts written`)
+            }
+          },
+      host)
+  } catch (error) {
+    if (error instanceof PackError || error instanceof StoreHostError) {
+      err(`factoidal pack: ${error.message}`)
+      return EXIT_FAILURE
+    }
+    // Everything the engine refuses -- an unknown grammar tag, a parse
+    // error, a cap -- arrives as a plain Error carrying the engine's own
+    // words. A stack trace here would hide them.
+    if (error instanceof Error && typeof error.message === 'string') {
+      err(`factoidal pack: ${error.message.replace(/^l4factoidal:\s*/, '')}`)
+      // The raised stack was refused, unavailable, or still not enough.
+      if (isStackOverflow(error)) {
+        for (const line of stackLimitAdvice(STACK_REMEDY.pack)) err(line)
+      }
+      return EXIT_FAILURE
+    }
+    throw error
+  }
+  if (answer.notWired === true) {
+    return notWired('pack',
+      'This install carries an engine built before the streaming pack ' +
+      'operations. Update @factoidal/core, or set FACTOIDAL_L4_ASSETS to ' +
+      'a newer build.')
+  }
+  const report = answer.report
+  if (options.json === true) {
+    out(JSON.stringify(report, null, 2))
+    return EXIT_OK
+  }
+  out(`packed ${report.bytesRead} bytes of ${syntax} into ${output}`)
+  out(`${report.written.length} artifacts, ${report.bytesWritten} bytes, layout ${layout}`)
+  if (typeof report.rows === 'number') out(`${plural(report.rows, 'row')}`)
+  out(`activate it with: factoidal activate ${dirOf(output)} ${nameOf(output)}`)
+  return EXIT_OK
 }
 
-function commandActivate (positional, _options) {
+async function commandActivate (positional, options) {
   if (positional.length !== 2) throw new UsageError('activate needs STORE and GENERATION')
-  return notWired('activate', 'Activation must verify every artifact before it replaces CURRENT.')
+  const [root, generation] = positional
+  // Verification decodes the same blocks the pack encoded, so it recurses
+  // as deep and needs the same raised stack. Measured 2026-09-04: a
+  // 112,742-row generation packed successfully and then failed to
+  // activate with `Maximum call stack size exceeded`, leaving a store
+  // that could be built and not opened
+  // (https://github.com/danbri/factoidal/issues/649).
+  const host = { worker: options['no-worker'] !== true }
+  const reexec = await denoReexec(host)
+  if (reexec !== null) return reexec
+  let answer
+  try {
+    answer = await runPack({ kind: 'activate', root, generation }, undefined, host)
+  } catch (error) {
+    if (error instanceof PackError || error instanceof StoreHostError) {
+      err(`factoidal activate: ${error.code ? error.code + ': ' : ''}${error.message}`)
+      return EXIT_FAILURE
+    }
+    if (error instanceof Error && typeof error.message === 'string') {
+      err(`factoidal activate: ${error.message.replace(/^l4factoidal:\s*/, '')}`)
+      if (isStackOverflow(error)) {
+        for (const line of stackLimitAdvice(STACK_REMEDY.pack)) err(line)
+      }
+      return EXIT_FAILURE
+    }
+    throw error
+  }
+  if (answer.notWired === true) {
+    return notWired('activate',
+      'This install carries an engine built before the activation ' +
+      'verification operation. Update @factoidal/core.')
+  }
+  const verdict = answer.report
+  if (verdict.ok !== true) {
+    err(`factoidal activate: ${verdict.error}`)
+    err('The generation is NOT activated; CURRENT is unchanged.')
+    return EXIT_FAILURE
+  }
+  // Only now does the pointer move, and it moves atomically.
+  const pointer = new TextEncoder().encode(generation)
+  const synced = atomicReplace(joinPath(root, 'CURRENT'), pointer)
+  if (options.json === true) {
+    out(JSON.stringify({ ...verdict, generation, directorySynced: synced }, null, 2))
+    return EXIT_OK
+  }
+  out(`activated ${generation}: ${verdict.artifacts} artifacts verified, ${verdict.bytes} bytes`)
+  if (!synced) {
+    err('CURRENT was replaced, but the directory entry was not synced; a ' +
+        'crash now could lose the pointer update.')
+  }
+  return EXIT_OK
 }
 
 function commandUpdate (positional, options) {
@@ -589,6 +824,7 @@ function commandCompact (positional, _options) {
 
 const COMMANDS = {
   version: (positional, options) => commandVersion(options),
+  'sample-store': (positional, options) => commandSampleStore(options),
   inspect: commandInspect,
   query: commandQuery,
   pack: commandPack,