@factoidal/core 0.6.0 → 0.7.0

package/CHANGELOG.md

@@ -1,5 +1,64 @@
 # Changelog
 
+## 0.7.0 — 2026-09-05
+
+**Wire version 10.** The command and `bin/store.mjs` read a generation
+packed at `--layout ibk5`: out-of-line literals, RDF 1.2 triple terms and
+directional literals, and per-entry zone maps.
+
+### The packer
+
+`factoidal pack --layout ibk5` writes wire version 10. Blocks are now
+published DURING the ingest pass, every `--batch-bytes` of source (default
+64 MiB in the module, 256 MiB in the native tool), so peak memory is bounded
+by a batch and not by the source: the native packer's peak on the
+1,543,478,120-byte skosdex corpus fell from 5,951,730,560 to 396,869,632
+bytes, and the module's 128 MiB source cap is gone for N-Quads, Turtle and
+N-Triples (TriG still buffers). Rows are bucketed by predicate AND graph, so
+an N-Quads file whose graphs interleave line by line packs to the same
+block set as the same statements in graph order (measured: 11,636 blocks
+became 483). The literal search index is LGI2, whose LEB128 gaps are 23% of
+a `skos:prefLabel` block against LGI1's 54%; a version-10 generation of the
+210 MB skosdex prefix is 14.5% smaller than version 9.
+
+### Out-of-line literals
+
+A literal above 65,536 UTF-8 bytes is stored as one `blob-<sha256
+hex>.lit` file beside the block, committed by the manifest blob table.
+The block holds only the byte extent and the digest, so a host must fetch
+the blob as well as the block:
+
+- `storeQueryPlan` lists them under `blobKeys`, after `keys` and
+  `sidecarKeys`, and counts their bytes in `bytes`;
+- `queryStore` and `openStoreHandle` fetch them without a flag;
+- a blob that is absent or whose bytes hash differently REFUSES the
+  query by name. Nothing is answered with a shortened literal.
+
+A store handle resolves every blob ONCE, at `storeOpen`, and the open
+envelope reports how many. The two candidate-filter indexes keep an
+out-of-line literal in every candidate set, so `CONTAINS` over a
+70,000-byte literal and `geof:sfIntersects` over a polygon above the
+ceiling both answer their rows.
+
+### RDF 1.2 terms
+
+A triple term and a directional language literal are stored and read
+back. Wire version 9 refuses both.
+
+### Zone maps
+
+Each manifest entry carries the first 64 bytes of the smallest and the
+largest subject key and object key of its block. A query with a constant
+subject or object skips every block whose range cannot hold it, from the
+manifest alone. `storeQueryPlan` reports the count as `zoneExcluded`, and
+`factoidal inspect` prints the bounds beside each entry.
+
+### Unchanged
+
+Wire version 9 and earlier answer exactly what they answered before: no
+manifest below version 10 carries a blob table or a zone map, so
+`blobKeys` is empty and `zoneExcluded` is zero for them.
+
 ## 0.6.0 — 2026-09-05
 
 **A full-text index, a geometry index, extension functions from

package/README.md

@@ -942,6 +942,42 @@ would drop answers. A non-overlapping pair of boxes proves geometries
 disjoint, but that saves work inside a scan rather than reducing the
 candidate set.
 
+### Wire version 10: large literals, RDF 1.2 terms, zone maps
+
+A generation packed at wire version 10 (`--layout ibk5`) carries three
+things earlier versions do not. Everything below is read by the command
+and by `bin/store.mjs` without a flag; wire versions 9 and earlier are
+read exactly as before.
+
+**Out-of-line literals.** A literal whose lexical form is at most 65,536
+UTF-8 bytes is stored inside the block. A longer one is stored as one
+file beside it, `blob-<sha256 hex>.lit`, holding exactly those bytes; the
+block keeps only the byte length and the SHA-256, and the manifest blob
+table commits the file. The name is the content address, so the same
+literal in twenty blocks is one file. `factoidal inspect` prints how many
+such files a generation holds and their total size, and the plan a query
+produces lists them under `blobKeys` beside `keys` and `sidecarKeys` — a
+host reading a store itself must fetch all three. A blob that is missing
+or whose bytes hash differently REFUSES the query; it is never answered
+with a shortened literal.
+
+Above 4,294,967,295 bytes the packer refuses the literal and names its
+subject and predicate. That ceiling, and every other one, is stated in
+`docs/designissues/2026-09-05-wire-version-10-scale.md` section 2.
+
+**RDF 1.2 terms.** A triple term (`<<( :a :p :b )>>`) and a directional
+language literal (`"..."@ar--rtl`) are stored and read back. Wire version
+9 refuses both.
+
+**Zone maps.** Each manifest entry carries the smallest and the largest
+subject key and object key of its block, truncated to 64 bytes. A query
+with a constant subject or object skips every block whose range cannot
+hold it, from the manifest alone, with no block read. The plan reports
+how many entries were dropped that way as `zoneExcluded`. Selectivity
+depends on the source order: a subject-grouped or graph-grouped file
+gives disjoint ranges per block, a shuffled one gives overlapping ranges
+and a scan, which is correct and no worse than wire version 9.
+
 ## Limits (deliberate, documented)
 
 - **In-memory only.** ~1.2 KB RAM per quad (measured); 1M quads ≈

package/bin/factoidal.mjs

@@ -45,7 +45,7 @@ const EXIT_NOT_WIRED = 3
 // a time, so this is the window that catches one feed per report.
 const FEED_PROGRESS = 65536
 
-const PACK_LAYOUTS = ['ibk3', 'ibk4']
+const PACK_LAYOUTS = ['ibk3', 'ibk4', 'ibk5']
 const PACK_SYNTAXES = ['turtle', 'trig', 'nquads', 'ntriples']
 const PACK_SUFFIXES = [
   ['.ttl', 'turtle'], ['.turtle', 'turtle'],
@@ -230,9 +230,15 @@ INPUT is an RDF file. OUTPUT is the generation directory to create; it is
 normally STORE/gen-N. Packing does not activate: run activate next.
 
 options:
-  --layout LAYOUT    ibk3 (triples, default) or ibk4 (quads)
+  --layout LAYOUT    ibk3 (triples, default), ibk4 (quads, wire version 9)
+                     or ibk5 (quads, wire version 10: RDF 1.2 terms,
+                     out-of-line literals, zone maps)
   --syntax SYNTAX    turtle, trig or nquads; default from the file extension
   --chunk-bytes N    Merkle chunk size; default is the engine's
+  --batch-bytes N    ibk4 only: source bytes between publication batches;
+                     default is the engine's. Blocks are published during
+                     the ingest pass, so this bounds the packer's memory.
+                     Match l4block-shard-pack --batch-bytes for its blocks.
   --json             emit one JSON object
   --no-worker        pack in this process instead of on a worker thread
 
@@ -470,10 +476,27 @@ async function commandInspect (positional, options) {
   out(`term registry ${envelope.termRegistryVersion === '' ? '(none recorded)' : envelope.termRegistryVersion}`)
   out(`fixed-chunk Merkle commitment ${envelope.rangeCommitted ? 'yes' : 'no'}`)
   out(`${envelope.entries.length} ${envelope.entries.length === 1 ? 'entry' : 'entries'}, ${envelope.totalBytes} bytes, ${plural(envelope.totalRows, 'row')}`)
+  // Wire version 10: the manifest blob table. Each `blob-<hex>.lit` holds the
+  // lexical form of one literal above the 65,536-byte inline ceiling, and is
+  // shared by every block whose dictionary names it.
+  const blobs = envelope.blobs ?? []
+  if (blobs.length > 0) {
+    let blobBytes = 0
+    for (const blob of blobs) blobBytes += blob.bytes
+    out(`${plural(blobs.length, 'out-of-line literal')}, ${blobBytes} bytes`)
+  }
   out(`generation directory holds ${files.length} files, ${directoryBytes} bytes`)
   out('')
+  // The zone maps are per entry, so the columns appear only for a manifest
+  // that carries them. A bound is the first 64 bytes of the smallest (or
+  // largest) term key of the block; the first eight are enough to see that
+  // two blocks hold different ranges.
+  const zoned = envelope.entries.some((entry) => entry.subjectZone !== undefined)
+  const zoneCell = (zone) =>
+    zone === undefined ? '-' : `${zone.min.slice(0, 16)}..${zone.max.slice(0, 16)}`
   printTable(
-    ['#', 'rows', 'bytes', 'kind', 'graphs', 'predicate'],
+    ['#', 'rows', 'bytes', 'kind', 'graphs', 'predicate']
+      .concat(zoned ? ['blobs', 'subject zone'] : []),
     envelope.entries.map((entry) => [
       String(entry.ordinal),
       String(entry.rows),
@@ -481,7 +504,9 @@ async function commandInspect (positional, options) {
       entry.blockKind,
       entry.graphs.length === 0 ? '-' : entry.graphs.map(graphLabel).join(' '),
       entry.predicate
-    ]))
+    ].concat(zoned
+      ? [String((entry.blobRefs ?? []).length), zoneCell(entry.subjectZone)]
+      : [])))
   return EXIT_OK
 }
 
@@ -764,7 +789,8 @@ async function commandPack (positional, options) {
   let answer
   try {
     answer = await runPack(
-      { kind: 'pack', input, output, syntax, layout, base: packBase(input, options) },
+      { kind: 'pack', input, output, syntax, layout, base: packBase(input, options),
+        batchBytes: options['batch-bytes'] },
       quiet
         ? undefined
         : (progress) => {

package/bin/pack-host.mjs

@@ -215,6 +215,7 @@ export async function packHere (task, onProgress) {
     syntax: task.syntax,
     layout: task.layout,
     base: task.base,
+    batchBytes: task.batchBytes,
     onProgress
   })
   return { report }
@@ -298,7 +299,7 @@ async function packInWorker (task, onProgress) {
  * already re-executed (see `denoReexec`), so this runs in process.
  *
  * @param {object} task {kind, ...}: a pack takes
- *   {input, output, syntax, layout, base}, an activate takes
+ *   {input, output, syntax, layout, base, batchBytes}, an activate takes
  *   {root, generation}
  * @param {(progress: object) => void} [onProgress]
  * @param {object} [options] {worker: false} forces the in-process path

package/bin/pack.mjs

@@ -93,16 +93,22 @@ function drain (engine, handle, output) {
  * @param {string} input the RDF file to read
  * @param {string} output the generation directory to fill; it must exist
  * @param {object} options
- * @param {string} options.layout `ibk3` or `ibk4`
+ * @param {string} options.layout `ibk3`, `ibk4` or `ibk5`
  * @param {string} options.syntax `turtle`, `trig` or `nquads`
  * @param {string} [options.base] the base IRI relative IRIs resolve against;
  *   the empty string means no base
+ * @param {number|string} [options.batchBytes] source bytes per publication
+ *   batch for the IBK4 streamed grammars; omitted means the engine's own
+ *   default. Match the native packer's `--batch-bytes` to get its block set.
  * @param {(progress: object) => void} [options.onProgress]
  * @returns {object} the engine's own finish envelope, plus what was written
  */
 export function packFile (engine, input, output, options) {
   const begun = engine.call('packBegin',
-    [options.syntax, options.layout, typeof options.base === 'string' ? options.base : ''])
+    [options.syntax, options.layout,
+      typeof options.base === 'string' ? options.base : '',
+      options.batchBytes === undefined || options.batchBytes === null
+        ? '' : String(options.batchBytes)])
   const handle = begun.handle
   if (typeof handle !== 'string') {
     throw new PackError('packBegin answered no handle')

package/bin/store.mjs

@@ -3,7 +3,9 @@
 //
 // WHAT THIS FILE IS ALLOWED TO DO
 // Read CURRENT, read a manifest file by name, read the artifact files the
-// engine's plan named, concatenate their bytes, and hand them over. It
+// engine's plan named -- blocks, index sidecars, and the `blob-<hex>.lit`
+// out-of-line literals of wire version 10 -- concatenate their bytes, and
+// hand them over. It
 // never parses a manifest, never verifies a digest, never decodes a block
 // and never decides which artifact answers a query. Every one of those is
 // a format decision and it lives in `formal/lean4/Wasm/Ops/Store.lean`
@@ -192,6 +194,22 @@ function capDecision (engine, store, sparql) {
   }
 }
 
+/** Read the named artifacts and concatenate them into one region. */
+function regionOf (store, keys) {
+  const chunks = keys.map((key) => readWhole(joinPath(store.generationDir, key)))
+  let total = 0
+  for (const chunk of chunks) total += chunk.length
+  const blob = new Uint8Array(total)
+  const artifacts = []
+  let offset = 0
+  for (let index = 0; index < chunks.length; index += 1) {
+    blob.set(chunks[index], offset)
+    artifacts.push({ key: keys[index], offset, len: chunks[index].length })
+    offset += chunks[index].length
+  }
+  return { blob, artifacts, total }
+}
+
 /**
  * Evaluate one SPARQL query against one generation.
  *
@@ -210,17 +228,12 @@ export function queryStore (engine, store, sparql) {
   if (empty !== null) {
     return { plan, result: empty, blobBytes: 0, artifacts: [] }
   }
-  const chunks = plan.keys.map((key) => readWhole(joinPath(store.generationDir, key)))
-  let total = 0
-  for (const chunk of chunks) total += chunk.length
-  const blob = new Uint8Array(total)
-  const artifacts = []
-  let offset = 0
-  for (let index = 0; index < chunks.length; index += 1) {
-    blob.set(chunks[index], offset)
-    artifacts.push({ key: plan.keys[index], offset, len: chunks[index].length })
-    offset += chunks[index].length
-  }
+  // `blobKeys` are the out-of-line literals the selected blocks name (wire
+  // version 10). A block holds only their byte extent and SHA-256, so a
+  // query that touches one needs its file as much as it needs the block.
+  // The engine chose these keys; this host reads the files and nothing more.
+  const keys = plan.keys.concat(plan.blobKeys ?? [])
+  const { blob, artifacts, total } = regionOf(store, keys)
   let result
   try {
     result = engine.callBlobIO('storeQuery',
@@ -274,30 +287,18 @@ export function turtleOfNQuads (engine, nquads) {
  * need no change here: the engine names them under `entry.sidecars`.
  */
 function manifestKeys (engine, store) {
+  const envelope = inspectManifest(engine, store)
   const keys = []
-  for (const entry of inspectManifest(engine, store).entries) {
+  for (const entry of envelope.entries) {
     keys.push(entry.key)
     for (const key of Object.values(entry.sidecars ?? {})) keys.push(key)
   }
+  // The manifest blob table of wire version 10: one `blob-<hex>.lit` file per
+  // out-of-line literal, shared by every block whose dictionary names it.
+  for (const blob of envelope.blobs ?? []) keys.push(blob.key)
   return keys
 }
 
-/** Read the named artifacts and concatenate them into one region. */
-function regionOf (store, keys) {
-  const chunks = keys.map((key) => readWhole(joinPath(store.generationDir, key)))
-  let total = 0
-  for (const chunk of chunks) total += chunk.length
-  const blob = new Uint8Array(total)
-  const artifacts = []
-  let offset = 0
-  for (let index = 0; index < chunks.length; index += 1) {
-    blob.set(chunks[index], offset)
-    artifacts.push({ key: keys[index], offset, len: chunks[index].length })
-    offset += chunks[index].length
-  }
-  return { blob, artifacts }
-}
-
 /**
  * An open store held inside the engine. Hold this object for as long as the
  * process answers questions about the generation, then `close()` it.
@@ -316,6 +317,9 @@ export class StoreHandle {
     this.artifacts = envelope.artifacts
     this.bytes = envelope.bytes
     this.rows = envelope.rows
+    // Wire version 10 only: how many out-of-line literals the engine
+    // resolved when it opened the handle. Zero elsewhere.
+    this.blobs = envelope.blobs ?? 0
     this.closed = false
   }
 
@@ -366,7 +370,7 @@ export function openStoreHandle (engine, store, options = {}) {
     keys = options.keys
   } else if (typeof options.sparql === 'string') {
     const plan = planQuery(engine, store, options.sparql)
-    keys = plan.keys.concat(plan.sidecarKeys ?? [])
+    keys = plan.keys.concat(plan.sidecarKeys ?? [], plan.blobKeys ?? [])
   } else {
     keys = manifestKeys(engine, store)
   }

package/l4-assets/l4factoidal.js

@@ -37,7 +37,7 @@
 // bytes change.
 
 // Stamped by formal/lean4/Wasm/build-wasm.sh step 9 -- do not hand-edit.
-const WASM_VERSION = "0de098a8aad0";
+const WASM_VERSION = "a7720d5d5630";
 
 import createModule from './l4factoidal.mjs';
 

package/l4-assets/version.json

@@ -2,13 +2,13 @@
   "engine": "lean4",
   "note": "The Lean 4 (L4Factoidal) engine's own build provenance and claims -- distinct from ../version.json, which is the F* engine's. Do not merge the two: they come from different toolchains, different git SHAs in general, and carry different proof obligations (issue #618).",
   "version": "0.1.0",
-  "gitSha": "178d7ce330ab01f4e5814801e9fda483d4478f34",
-  "builtAt": "2026-09-05T13:03:17+00:00",
+  "gitSha": "e32b58ac2ec41dca4f86ef6e6b6f1a5cc6f46726",
+  "builtAt": "2026-09-05T20:46:27+00:00",
   "leanToolchain": "leanprover/lean4:v4.33.1",
   "emscripten": "emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 6.0.8-git",
   "abiVersion": "1",
-  "wasmSha256": "0de098a8aad0255eb575b4af236592eacb628da70a0784d22267624d2eaa6507",
-  "wasmBytes": 5606105,
+  "wasmSha256": "a7720d5d563014d875d26f63576993166c45b8f3eef7a6be9b75adfebd626500",
+  "wasmBytes": 5857701,
   "claims": {
     "source": "formal/lean4 (L4Factoidal): no sorry, no user axioms, no native_decide; W3C behaviour pinned by build-time #guard",
     "suitesAtBuildSha": "see docs/test-results and formal/lean4/PORT_NOTES.md at gitSha"

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@factoidal/core",
-  "version": "0.6.0",
+  "version": "0.7.0",
   "description": "Factoidal: a linked information system with graph data and the Web at its heart - RDF parsing, SPARQL 1.1 query, canonicalization and entailment for Node and the browser (JS + Wasm), grounded in F* and Lean 4. Parser and algebra spec verified in F*; on-disk backend has unverified OCaml-side optimization layers being migrated back to F*.",
   "keywords": [
     "sparql",