@xmbl/state-machine 0.1.5 → 0.1.12

package/CHANGELOG.md

@@ -0,0 +1,85 @@
+# @xmbl/state-machine
+
+## 0.1.12
+
+## 0.1.11
+
+### Patch Changes
+
+- Return every `@xmbl/*` protocol package to ONE version line (0.1.11) after a night of per-package
+  hotfix publishes (cubic-ledger reached 0.1.10 while zero-knowledge sat at 0.1.1, crates at 0.1.0).
+
+  Changes since the last unified line (0.1.0), carried as one PATCH line — 0.1.x stays the pre-mainnet line until the ⛔ AUDIT gates close:
+
+  - cubic-ledger: block ids address CONSENSUS CONTENT (`consensusBody`) instead of the whole tx envelope; an
+    anchor's `hash` must be a sha-256 digest; invalid txs are evicted for good (`evict`, `evicted:` keyspace);
+    legacy envelope-keyed rows converge to content ids on the next boot; a canonical rebuild preserves every
+    non-anchor block and reports what it wiped; `ready()` is the boot join point. `capabilities()` (hardcoded
+    flags) is gone; `census.mjs` (an operator one-off) no longer ships in the tarball. A FORGERY CAN NO LONGER
+    DELETE THE TRANSACTION IT IMPERSONATES: an invalid typed datum is evicted under a digest of its own bytes,
+    never under the xid it claims (a datum fails `validateXid` precisely when that xid is somebody else's), and
+    the anchor dedup key claimed before validation is released on every failure. Both doors are closed on BOTH
+    entry points — `addTransaction` and `addSealedBatch`, the path a finalized transaction actually takes — and
+    one invalid entry in a batch no longer discards the genuine transactions behind it.
+  - consensus: a content-addressed type-6 value tx is admitted on its content address (it carries no in-body
+    signature by design); a malformed anchor is refused at the door by the ledger's own `validateTransaction`;
+    `finalizeTransaction` preserves the signed `id`.
+  - identity: one canonical `signingMessage`; `signingStatus()` / `verifySigning()`; `Identity.fromPrivateKey`
+    (a stub that only threw) is removed.
+  - state-machine: the verkle trie is rebuilt on load; diffs are keyed by content identity; `ready()`.
+  - networking: self-elected circuit-relay server; NO_FATAL transport tolerance; throttled bootstrap warnings.
+  - storage-compute: `CoordinateDelivery` (broken: `require` in ESM, keyed on the public key) is removed.
+  - core: ships the node daemon as the `xmbl-node` bin; the control socket implements the coordinator
+    contract (`xsc`, `submit_batch`, `ledger_capabilities`, `identity_status`, a signed `chain` claim, genuine
+    `submit_tx` rejections) and reports the RUNNING versions; boot waits for the stores.
+  - every package exports a load-time `VERSION`.
+  - PROTOCOL (operator, 2026-09-16): every transaction is typed by its xid (tokens.json type codes; `micromineTx`/
+    `validateXid`; untyped rows deleted on boot, skipped by a canonical rebuild; an anchor's wire tx carries `prior`);
+    consensus validates in order — can it happen, is the xid correct, is the placement right (`validate.js`,
+    `verifyPlacement`); block hashes are content-only so every node seals the same cubes.
+  - ROLLOUT (operator, 2026-09-16): a node proves its version (`build` digest of the loaded code, signed into the
+    `chain` claim; `release` op), suspends itself when behind the npm `latest` of @xmbl/core (no submits, validations
+    or seals), installs the latest over the air and restarts (exit 75 under a supervisor).
+  - lng: ships its BROWSER build — `dist/lng.browser.js` (`@xmbl/lng/browser`), one dependency-free ES module
+    generated from the same `src/*.js` the node runs and byte-checked by the gate (same surface, same bytes, runs
+    with no Node globals); the sources no longer assume `process`/`Buffer`.
+  - lng: the `~bytes` BYTE-STRING TYPE reaches the WASM backend — a (pointer, length) pair on the operand
+    stack, never a 256-bit word: a literal's bytes live in a data segment with a compile-time length, a runtime
+    value (a UTXO id from `xmbl_input_id`) is host-written into fresh memory with its length in a local. A
+    `~bytes` FIELD or PARAM is now REFUSED — committed state and the call ABI are both 32-byte words with
+    nowhere to put a length, and until now both compiled SILENTLY as words. New opt-in host modes
+    `compile(src, { crypto: true })` (`xmbl.mayo.verify` / `xmbl.cubic.verify`) and `{ utxo: true }` (the
+    five-entry value ABI), so a contract written in LNG verifies a signature and spends a UTXO the caller
+    presented. The value ABI's `-1` sentinel TRAPS instead of widening to 2^256-1. The typechecker refuses
+    arithmetic/bitwise/ordering on `~bytes` and now walks `~contract` method bodies at all, which it never did.
+    A contract with no byte literals emits a byte-identical, import-free module.
+  - storage-compute: THE ERASURE CODER RETURNED CORRUPTED DATA WITHOUT SAYING SO. `StorageShard.decode`
+    recovers a lost data shard by XOR-ing its parity group, and XOR parity recovers AT MOST ONE loss per
+    group — but when two members of one group were missing it filled the hole with zeros and returned the
+    buffer as if decoding had succeeded, with no error, no flag and no short read. MEASURED on k=4, m=2:
+    losing data shards 0 and 2 handed back a buffer that differed from the original and nothing downstream
+    could tell. Two independent causes are fixed: the group is now checked for completeness before the XOR
+    is trusted, and `m` — the encoding's PARITY DEGREE — is carried on every shard as the new optional
+    `parityCount` field instead of being inferred from however many parity shards happened to survive (that
+    inference was wrong exactly when a parity shard was among the losses: given data 0,1,2 and parity 4 only,
+    the inferred m was 1, the recovery group became {0,1,2,3} instead of {0,2}, and decode returned wrong
+    bytes). An unrecoverable decode now THROWS and names every missing shard. Proven exhaustively over all
+    63 non-empty subsets of a k=4/m=2 encoding: every subset either decodes to the exact original or throws,
+    and none returns wrong bytes. COMPAT: `parityCount` is additive and optional, so a shard written by an
+    older node reads back fine — a new node treats it as legacy and REFUSES parity recovery rather than
+    guessing, which fails loudly where the old code failed silently. Shard metadata persists as JSON
+    (`meta:<id>`), so an old reader ignores the extra field.
+  - core: a boot crash. `Config._applyEnvOverrides()` assigned into `config.network` / `config.logging`
+    without creating them, so a node started with `XN_PORT` or `LOG_LEVEL` set against a config that omitted
+    those sections died on `undefined.port` before it could log why. The sections are created on demand.
+  - the protocol gate is 75 suites (was 67), and line coverage across the twelve protocol packages is 85.5%
+    (was 80.6%). `scripts/coverage-report.mjs` is the instrument: `NODE_V8_COVERAGE` + a V8-range reducer,
+    since nothing in the tree measured coverage at all. Ten protocol files that no suite had ever loaded now
+    have one; four remain, all process entry points.
+  - NODE 22 IS NOW DECLARED, because it was already REQUIRED. Every published package gains
+    `engines: { node: ">=22" }`; the workspace root's `">=20"` was simply false. `@xmbl/identity` imports
+    `node:sqlite` (Node 22.5+) for the durable nonce registry, `@xmbl/storage-compute` meters jobs with
+    `process.threadCpuUsage` (22.10+), and libp2p's own dependency chain calls `Promise.withResolvers`
+    (22.0). On Node 20 a consumer installed cleanly and crashed at import instead of being told at install
+    time. MEASURED: the protocol gate scores 54/75 on Node 20.20.2 and 75/75 on Node 22 — the twenty-one
+    failures were the runtime, not the code. CI and the release workflow now run Node 22 as well.

package/index.js

@@ -7,8 +7,9 @@ export { StateShard } from './src/sharding.js';
 export { StateAssembler } from './src/state-assembly.js';
 export { StateMachine } from './src/state-machine.js';
 
-const port = process.env.PORT || 3002;
-console.log(`XVSM (XMBL Virtual State Machine) starting on port ${port}`);
-
-
 
+// THE VERSION OF THE CODE THIS PROCESS LOADED. Read once at import time from this package's own manifest, so a
+// running node can report what it is actually executing — an install that lands on disk after this module was
+// loaded changes the file, not this constant. Consumed by @xmbl/core's control socket (`status`.versions).
+import { readFileSync as __readPkg } from 'node:fs';
+export const VERSION = JSON.parse(__readPkg(new URL('./package.json', import.meta.url), 'utf8')).version;

package/package.json

@@ -1,12 +1,15 @@
 {
   "name": "@xmbl/state-machine",
-  "version": "0.1.5",
+  "version": "0.1.12",
   "type": "module",
   "main": "index.js",
   "exports": {
     ".": "./index.js"
   },
   "license": "MIT",
+  "engines": {
+    "node": ">=22"
+  },
   "publishConfig": {
     "access": "public"
   },

package/src/applied-count.test.mjs

@@ -1,4 +1,4 @@
-// OUTCOME TEST: applied_tx_count after the convergence primitive. The metric the whole fleet reads.
+// OUTCOME TEST: applied_tx_count after the convergence primitive. The metric every node reads.
 const { StateMachine } = await import("../index.js");
 const sm = new StateMachine({ dbPath: null });
 sm._dbOpen = false;

package/src/apply-path.test.mjs

@@ -11,6 +11,8 @@
 //   node vendor/xmbl-node/state-machine/src/apply-path.test.mjs
 import { StateMachine } from './state-machine.js';
 import { Ledger } from '../../cubic-ledger/src/ledger.js';
+import { micromineTx } from '../../cubic-ledger/src/transaction-validator.js';
+import { createHash } from 'node:crypto';
 import { rm, mkdtemp } from 'node:fs/promises';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
@@ -23,10 +25,12 @@ const check = async (name, fn) => {
 };
 const eq = (a, b, what) => { if (a !== b) throw new Error(`${what}: expected ${b}, got ${a}`); };
 
-// Nine anchor txs seal exactly one face under the hash-sorted partition (9 blocks per face).
-const anchors = (n, tag) => Array.from({ length: n }, (_, i) => ({
-  type: 'anchor', event: 'task.created', hash: `${tag}-${i}`.padEnd(64, '0'), ts: 1_700_000_000_000 + i,
-}));
+// Nine anchor txs seal exactly one face under the hash-sorted partition (9 blocks per face). An anchor's
+// `hash` is a sha-256 digest by contract (cubic-ledger validateTransaction refuses anything else), so the
+// fixture mines a real digest per tag rather than a padded label.
+const anchors = (n, tag) => Array.from({ length: n }, (_, i) => micromineTx({
+  type: 'anchor', event: 'task.created', hash: createHash('sha256').update(`${tag}-${i}`).digest('hex'), ts: 1_700_000_000_000 + i,
+}));   // typed: every tx carries its xmbl type as a micromined xid
 
 const fresh = async () => {
   const dir = await mkdtemp(join(tmpdir(), 'xvsm-apply-'));
@@ -46,7 +50,7 @@ const close = async ({ dir, xclt, xvsm }) => {
 console.log('\n1. a SEALED face reaches the state tree (the missing block:added doorbell)');
 {
   const env = await fresh();
-  await check('the tree starts empty — the honest baseline, not an assumption', async () => {
+  await check('the tree starts empty — the true baseline, not an assumption', async () => {
     eq(env.xvsm.getStateRoot(), ZERO, 'initial root');
   });
   await check('sealing 9 txs moves state_root off zero and applies 9 diffs', async () => {

package/src/rehydration.test.mjs

@@ -1,4 +1,4 @@
-// THE THREE DEFECTS THAT KEPT NODES OFF A SHARED ROOT, each asserted by the symptom the fleet showed,
+// THE THREE DEFECTS THAT KEPT NODES OFF A SHARED ROOT, each asserted by the symptom the nodes showed,
 // not by the mechanism. All three were measured live on 2026-09-15 before being fixed here.
 import { test } from 'node:test';
 import assert from 'node:assert';
@@ -25,7 +25,7 @@ test('a restarted node publishes a REAL root, not 64 zeros, from the state: keys
   await sm.db.close();
 
   // Reopen. The diff replay is NOT the thing under test — delete every diff row first, so the only
-  // surviving source for the trie is `state:`. 39 of 44 reporting fleet nodes published ZERO here.
+  // surviving source for the trie is `state:`. 39 of 44 reporting nodes published ZERO here.
   sm = await open(dir);
   await sm.db.clear({ gte: 'diff:', lt: 'diff:\xFF' });
   await sm.db.close();
@@ -73,7 +73,7 @@ test('legacy rows keyed by block id are re-keyed to content identity on the next
   const dir = `/tmp/xvsm-rehydrate-${process.pid}-d`;
   rmSync(dir, { recursive: true, force: true });
   let sm = await open(dir);
-  // Write rows the way every node on the fleet already has them: one per submission, keyed by block id.
+  // Write rows the way every node on the nodes already has them: one per submission, keyed by block id.
   const set = anchors(40);
   let written = 0;
   for (let pass = 0; pass < 3; pass++) for (const a of set) {

package/src/state-machine.js

@@ -34,7 +34,7 @@ export class StateMachine extends EventEmitter {
     
     // Initialize database. KEEP THE PROMISE: `_dbOpen` flips true the moment Level opens, long before the
     // diff sweep and the tree rehydration have finished, so anything that polls `_dbOpen` and then reads a
-    // count is reading a store mid-rebuild. `ready()` is the only honest join point.
+    // count is reading a store mid-rebuild. `ready()` is the only genuine join point.
     this._ready = this._initDb().catch(() => {});
     
     // Listen to ledger events if available
@@ -235,6 +235,14 @@ export class StateMachine extends EventEmitter {
   }
 
   async _handleLedgerBlock(block) {
+    // JOIN THE STORE BEFORE TOUCHING IT. This handler is wired to `block:added` in the CONSTRUCTOR, so the
+    // ledger can deliver a block while `_initDb` is still opening the db and rehydrating the tree. Both
+    // writes below swallow their errors (by design — an in-memory fallback must not take the node down),
+    // which means an early block was applied to the in-memory tree and persisted NOWHERE: the root looked
+    // right for the life of the process and came back 64 zeros on the next boot, with nothing logged.
+    // MEASURED: applying two blocks before `ready()` resolved gave root ad778ca2… in-process and
+    // 0000000000… after a restart; awaiting `ready()` first, the same root survives.
+    await this.ready();
     const changes = this._stateChangesFor(block);
     if (!changes || !Object.keys(changes).length) return;
     try {
@@ -327,9 +335,9 @@ export class StateMachine extends EventEmitter {
     // ⛔ RECORD A DIFF FOR EVERY ANCHOR APPLIED, or applied_tx_count IS ZERO BY CONSTRUCTION. This loop wrote
     // straight into the tree and never touched `this.diffs`, which it had just emptied — and
     // getStatistics().totalTransactions is transactionLog.length + diffs.length. So the moment a node runs the
-    // convergence primitive, the number the whole fleet reads as "is this node applying anything" resets to 0
+    // convergence primitive, the number every node reads as "is this node applying anything" resets to 0
     // and STAYS 0 no matter how many anchors it applied. MEASURED 2026-09-15: this node held a correct root
-    // over 3,941 applied anchors and published applied_tx_count 0; across the fleet 42 of 46 reporting nodes
+    // over 3,941 applied anchors and published applied_tx_count 0; across the nodes 42 of 46 reporting nodes
     // read 0, and a design ruling was written on the premise that 41 of them had "applied nothing". They had.
     // The diff is not bookkeeping — it is the same StateDiff the block path records, keyed by the anchor's own
     // content, so a rebuild and a live apply of the same anchor upsert to ONE row rather than two.

package/src/verkle-independent-verify.test.mjs

@@ -9,7 +9,7 @@
 //   leaf hash            = sha256(value canonicalised)            (value node at depth 32)
 //   internal node hash   = sha256( concat of 256 child slots, each 32 bytes; empty slot = 32 zeros )
 //   a key's nibble at depth d = sha256(key)[d]
-// A proof carries, per level, the sibling hashes at that node; the honest verifier fills the key's own
+// A proof carries, per level, the sibling hashes at that node; the correct verifier fills the key's own
 // slot with the hash carried up from below and every other slot from the siblings (or zeros), hashes,
 // and repeats to the root. The proof is SOUND iff the reconstructed root equals the tree's real root —
 // which the verifier learns independently (tree.getRoot()), never from the attacker-supplied proof.root.

package/src/verkle-integration.test.mjs

@@ -10,10 +10,16 @@ import { join } from 'path';
 let pass = 0, fail = 0;
 const check = async (n, f) => { try { await f(); console.log(`  ok   ${n}`); pass++; } catch (e) { console.log(`  FAIL ${n}\n       ${e.message}`); fail++; } };
 const EMPTY = '0'.repeat(64);
-// Portable: this suite runs on the laptop AND on every Linux box in the fleet.
+// Portable: this suite runs on the laptop AND on every Linux box among the nodes — so it JOINS on `ready()`
+// rather than sleeping. A fixed timeout is a guess about someone else's scheduler: this suite waited
+// 120/150/250/500ms for the store to open and the tree to rehydrate, which held on this laptop and LOST on
+// ubuntu-latest, where the restart check read 64 zeros — the 150ms wait expired before `_initDb` had opened
+// the db, so `_handleLedgerBlock` persisted no diff rows and the restart had nothing to replay.
+// `StateMachine.ready()` resolves once the store is open, the tree is rehydrated from `state:` and the diff
+// rows are swept; every write below it is already awaited. There is nothing left to sleep for.
 const dir = join(tmpdir(), 'xvsm-verkle-test');
 const fresh = async () => { rmSync(dir, { recursive: true, force: true });
-  const sm = new StateMachine({ dbPath: dir }); await new Promise(r => setTimeout(r, 120)); return sm; };
+  const sm = new StateMachine({ dbPath: dir }); await sm.ready(); return sm; };
 
 const blk = (id, tx) => ({ id, tx });
 const REAL_TRAFFIC = [
@@ -107,18 +113,40 @@ console.log('\n4. restart survival');
 await check('verkle root survives a restart (diffs are REPLAYED, not just collected)', async () => {
   const d = dir + '-restart';
   rmSync(d, { recursive: true, force: true });
-  let sm = new StateMachine({ dbPath: d }); await new Promise(r => setTimeout(r, 150));
+  let sm = new StateMachine({ dbPath: d }); await sm.ready();
   for (const b of REAL_TRAFFIC) await sm._handleLedgerBlock(b);
   const before = sm.stateTree.getRoot();
   assert.notStrictEqual(before, EMPTY);
-  await new Promise(r => setTimeout(r, 250)); await sm.db.close();
-  sm = new StateMachine({ dbPath: d }); await new Promise(r => setTimeout(r, 500));
+  // No sleep before the close: _handleLedgerBlock awaits both the tree insert (which awaits
+  // _saveState) and the diff `put`, so when the loop above returns, the bytes are already down.
+  await sm.db.close();
+  sm = new StateMachine({ dbPath: d }); await sm.ready();
   const after = sm.stateTree.getRoot();
   await sm.db.close(); rmSync(d, { recursive: true, force: true });
   assert.strictEqual(after, before, 'root changed across restart');
   assert.notStrictEqual(after, EMPTY, 'root reset to zeros on restart');
 });
 
+// The regression the sleeps above were hiding. `_handleLedgerBlock` is wired to `block:added` in the
+// CONSTRUCTOR, so in production a block can arrive while `_initDb` is still opening the store — and both of
+// that handler's writes swallow their errors, so the block landed in the in-memory tree and nowhere on disk.
+// The root read correctly for the life of the process and came back 64 zeros on the next boot, silently.
+// This check applies traffic with NO join at all, the way the ledger's event does.
+await check('a block applied BEFORE the store finished opening still survives a restart', async () => {
+  const d = dir + '-race';
+  rmSync(d, { recursive: true, force: true });
+  let sm = new StateMachine({ dbPath: d });          // deliberately NOT awaiting ready()
+  for (const b of REAL_TRAFFIC) await sm._handleLedgerBlock(b);
+  const before = sm.stateTree.getRoot();
+  assert.notStrictEqual(before, EMPTY, 'control invalid: nothing was applied');
+  await sm.db.close();
+  sm = new StateMachine({ dbPath: d }); await sm.ready();
+  const after = sm.stateTree.getRoot();
+  await sm.db.close(); rmSync(d, { recursive: true, force: true });
+  assert.notStrictEqual(after, EMPTY, 'root came back as 64 zeros — the early block was never persisted');
+  assert.strictEqual(after, before, 'root changed across restart');
+});
+
 rmSync(dir, { recursive: true, force: true });
 console.log(`\n${fail === 0 ? 'PASS' : 'FAIL'} — ${pass} passed, ${fail} failed\n`);
 process.exit(fail === 0 ? 0 : 1);

package/src/verkle-tree.js

@@ -53,7 +53,7 @@ export class VerkleStateTree {
     // `this.root` a fresh empty VerkleNode. So a node that had loaded EVERY key off disk answered getRoot()
     // with 64 zeros: a tree that holds the whole state and commits to nothing. MEASURED 2026-09-15: after a
     // canonical rebuild of 10 anchors, restart loaded all 10 keys and published root 0000…0000; across the
-    // fleet 39 of 44 reporting nodes published exactly that, including one with 11,691 blocks persisted. The
+    // nodes 39 of 44 reporting nodes published exactly that, including one with 11,691 blocks persisted. The
     // only boxes showing a real root were showing it by accident, via XVSM's diff replay re-inserting keys
     // through insert(). The state map is not the commitment — the hashed trie over it is, and it has to be
     // reconstructed on open or every restart silently un-commits the node's entire state.
@@ -201,7 +201,7 @@ export class VerkleStateTree {
   // transactions instead of the structure that actually commits them. This returns the real nodes and the
   // real parent->child edges, bounded, with the counts needed to say what was left out.
   //
-  // Bounded on BOTH axes and honest about it: `maxDepth` limits how far down, `maxNodes` caps the total, and
+  // Bounded on BOTH axes and genuine about it: `maxDepth` limits how far down, `maxNodes` caps the total, and
   // the result carries `truncated` plus each node's own `descendants` so a cut branch reports its real size
   // rather than looking like a leaf. A viewer that silently stopped at 500 nodes would draw a tree that is
   // simply the wrong shape.
@@ -217,7 +217,7 @@ export class VerkleStateTree {
     }
     if (!start.hash) this._updateHash(start);
 
-    // Subtree size is what makes a truncated branch honest, so it is computed for every node we emit.
+    // Subtree size is what makes a truncated branch genuine, so it is computed for every node we emit.
     const sizeOf = (node) => {
       let n = 1;
       for (const c of node.children.values()) n += sizeOf(c);