@xmbl/state-machine 0.1.4 → 0.1.11

package/CHANGELOG.md

@@ -0,0 +1,83 @@
+# @xmbl/state-machine
+
+## 0.1.11
+
+### Patch Changes
+
+- Return every `@xmbl/*` protocol package to ONE version line (0.1.11) after a night of per-package
+  hotfix publishes (cubic-ledger reached 0.1.10 while zero-knowledge sat at 0.1.1, crates at 0.1.0).
+
+  Changes since the last unified line (0.1.0), carried as one PATCH line — 0.1.x stays the pre-mainnet line until the ⛔ AUDIT gates close:
+
+  - cubic-ledger: block ids address CONSENSUS CONTENT (`consensusBody`) instead of the whole tx envelope; an
+    anchor's `hash` must be a sha-256 digest; invalid txs are evicted for good (`evict`, `evicted:` keyspace);
+    legacy envelope-keyed rows converge to content ids on the next boot; a canonical rebuild preserves every
+    non-anchor block and reports what it wiped; `ready()` is the boot join point. `capabilities()` (hardcoded
+    flags) is gone; `census.mjs` (an operator one-off) no longer ships in the tarball. A FORGERY CAN NO LONGER
+    DELETE THE TRANSACTION IT IMPERSONATES: an invalid typed datum is evicted under a digest of its own bytes,
+    never under the xid it claims (a datum fails `validateXid` precisely when that xid is somebody else's), and
+    the anchor dedup key claimed before validation is released on every failure. Both doors are closed on BOTH
+    entry points — `addTransaction` and `addSealedBatch`, the path a finalized transaction actually takes — and
+    one invalid entry in a batch no longer discards the honest transactions behind it.
+  - consensus: a content-addressed type-6 value tx is admitted on its content address (it carries no in-body
+    signature by design); a malformed anchor is refused at the door by the ledger's own `validateTransaction`;
+    `finalizeTransaction` preserves the signed `id`.
+  - identity: one canonical `signingMessage`; `signingStatus()` / `verifySigning()`; `Identity.fromPrivateKey`
+    (a stub that only threw) is removed.
+  - state-machine: the verkle trie is rebuilt on load; diffs are keyed by content identity; `ready()`.
+  - networking: self-elected circuit-relay server; NO_FATAL transport tolerance; throttled bootstrap warnings.
+  - storage-compute: `CoordinateDelivery` (broken: `require` in ESM, keyed on the public key) is removed.
+  - core: ships the node daemon as the `xmbl-node` bin; the control socket implements the coordinator
+    contract (`xsc`, `submit_batch`, `ledger_capabilities`, `identity_status`, a signed `chain` claim, honest
+    `submit_tx` rejections) and reports the RUNNING versions; boot waits for the stores.
+  - every package exports a load-time `VERSION`.
+  - PROTOCOL (operator, 2026-09-16): every transaction is typed by its xid (tokens.json type codes; `micromineTx`/
+    `validateXid`; untyped rows deleted on boot, skipped by a canonical rebuild; an anchor's wire tx carries `prior`);
+    consensus validates in order — can it happen, is the xid correct, is the placement right (`validate.js`,
+    `verifyPlacement`); block hashes are content-only so every node seals the same cubes.
+  - ROLLOUT (operator, 2026-09-16): a node proves its version (`build` digest of the loaded code, signed into the
+    `chain` claim; `release` op), suspends itself when behind the npm `latest` of @xmbl/core (no submits, validations
+    or seals), installs the latest over the air and restarts (exit 75 under a supervisor).
+  - lng: ships its BROWSER build — `dist/lng.browser.js` (`@xmbl/lng/browser`), one dependency-free ES module
+    generated from the same `src/*.js` the node runs and byte-checked by the gate (same surface, same bytes, runs
+    with no Node globals); the sources no longer assume `process`/`Buffer`.
+  - lng: the `~bytes` BYTE-STRING TYPE reaches the WASM backend — a (pointer, length) pair on the operand
+    stack, never a 256-bit word: a literal's bytes live in a data segment with a compile-time length, a runtime
+    value (a UTXO id from `xmbl_input_id`) is host-written into fresh memory with its length in a local. A
+    `~bytes` FIELD or PARAM is now REFUSED — committed state and the call ABI are both 32-byte words with
+    nowhere to put a length, and until now both compiled SILENTLY as words. New opt-in host modes
+    `compile(src, { crypto: true })` (`xmbl.mayo.verify` / `xmbl.cubic.verify`) and `{ utxo: true }` (the
+    five-entry value ABI), so a contract written in LNG verifies a signature and spends a UTXO the caller
+    presented. The value ABI's `-1` sentinel TRAPS instead of widening to 2^256-1. The typechecker refuses
+    arithmetic/bitwise/ordering on `~bytes` and now walks `~contract` method bodies at all, which it never did.
+    A contract with no byte literals emits a byte-identical, import-free module.
+  - storage-compute: THE ERASURE CODER RETURNED CORRUPTED DATA WITHOUT SAYING SO. `StorageShard.decode`
+    recovers a lost data shard by XOR-ing its parity group, and XOR parity recovers AT MOST ONE loss per
+    group — but when two members of one group were missing it filled the hole with zeros and returned the
+    buffer as if decoding had succeeded, with no error, no flag and no short read. MEASURED on k=4, m=2:
+    losing data shards 0 and 2 handed back a buffer that differed from the original and nothing downstream
+    could tell. Two independent causes are fixed: the group is now checked for completeness before the XOR
+    is trusted, and `m` — the encoding's PARITY DEGREE — is carried on every shard as the new optional
+    `parityCount` field instead of being inferred from however many parity shards happened to survive (that
+    inference was wrong exactly when a parity shard was among the losses: given data 0,1,2 and parity 4 only,
+    the inferred m was 1, the recovery group became {0,1,2,3} instead of {0,2}, and decode returned wrong
+    bytes). An unrecoverable decode now THROWS and names every missing shard. Proven exhaustively over all
+    63 non-empty subsets of a k=4/m=2 encoding: every subset either decodes to the exact original or throws,
+    and none returns wrong bytes. COMPAT: `parityCount` is additive and optional, so a shard written by an
+    older node reads back fine — a new node treats it as legacy and REFUSES parity recovery rather than
+    guessing, which fails loudly where the old code failed silently. Shard metadata persists as JSON
+    (`meta:<id>`), so an old reader ignores the extra field.
+  - core: a boot crash. `Config._applyEnvOverrides()` assigned into `config.network` / `config.logging`
+    without creating them, so a node started with `XN_PORT` or `LOG_LEVEL` set against a config that omitted
+    those sections died on `undefined.port` before it could log why. The sections are created on demand.
+  - the protocol gate is 75 suites (was 67), and line coverage across the twelve protocol packages is 85.5%
+    (was 80.6%). `scripts/coverage-report.mjs` is the instrument: `NODE_V8_COVERAGE` + a V8-range reducer,
+    since nothing in the tree measured coverage at all. Ten protocol files that no suite had ever loaded now
+    have one; four remain, all process entry points.
+  - NODE 22 IS NOW DECLARED, because it was already REQUIRED. Every published package gains
+    `engines: { node: ">=22" }`; the workspace root's `">=20"` was simply false. `@xmbl/identity` imports
+    `node:sqlite` (Node 22.5+) for the durable nonce registry, `@xmbl/storage-compute` meters jobs with
+    `process.threadCpuUsage` (22.10+), and libp2p's own dependency chain calls `Promise.withResolvers`
+    (22.0). On Node 20 a consumer installed cleanly and crashed at import instead of being told at install
+    time. MEASURED: the protocol gate scores 54/75 on Node 20.20.2 and 75/75 on Node 22 — the twenty-one
+    failures were the runtime, not the code. CI and the release workflow now run Node 22 as well.

package/index.js

@@ -7,8 +7,9 @@ export { StateShard } from './src/sharding.js';
 export { StateAssembler } from './src/state-assembly.js';
 export { StateMachine } from './src/state-machine.js';
 
-const port = process.env.PORT || 3002;
-console.log(`XVSM (XMBL Virtual State Machine) starting on port ${port}`);
-
-
 
+// THE VERSION OF THE CODE THIS PROCESS LOADED. Read once at import time from this package's own manifest, so a
+// running node can report what it is actually executing — an install that lands on disk after this module was
+// loaded changes the file, not this constant. Consumed by @xmbl/core's control socket (`status`.versions).
+import { readFileSync as __readPkg } from 'node:fs';
+export const VERSION = JSON.parse(__readPkg(new URL('./package.json', import.meta.url), 'utf8')).version;

package/package.json

@@ -1,12 +1,15 @@
 {
   "name": "@xmbl/state-machine",
-  "version": "0.1.4",
+  "version": "0.1.11",
   "type": "module",
   "main": "index.js",
   "exports": {
     ".": "./index.js"
   },
   "license": "MIT",
+  "engines": {
+    "node": ">=22"
+  },
   "publishConfig": {
     "access": "public"
   },

package/src/apply-path.test.mjs

@@ -11,6 +11,8 @@
 //   node vendor/xmbl-node/state-machine/src/apply-path.test.mjs
 import { StateMachine } from './state-machine.js';
 import { Ledger } from '../../cubic-ledger/src/ledger.js';
+import { micromineTx } from '../../cubic-ledger/src/transaction-validator.js';
+import { createHash } from 'node:crypto';
 import { rm, mkdtemp } from 'node:fs/promises';
 import { tmpdir } from 'node:os';
 import { join } from 'node:path';
@@ -23,10 +25,12 @@ const check = async (name, fn) => {
 };
 const eq = (a, b, what) => { if (a !== b) throw new Error(`${what}: expected ${b}, got ${a}`); };
 
-// Nine anchor txs seal exactly one face under the hash-sorted partition (9 blocks per face).
-const anchors = (n, tag) => Array.from({ length: n }, (_, i) => ({
-  type: 'anchor', event: 'task.created', hash: `${tag}-${i}`.padEnd(64, '0'), ts: 1_700_000_000_000 + i,
-}));
+// Nine anchor txs seal exactly one face under the hash-sorted partition (9 blocks per face). An anchor's
+// `hash` is a sha-256 digest by contract (cubic-ledger validateTransaction refuses anything else), so the
+// fixture mines a real digest per tag rather than a padded label.
+const anchors = (n, tag) => Array.from({ length: n }, (_, i) => micromineTx({
+  type: 'anchor', event: 'task.created', hash: createHash('sha256').update(`${tag}-${i}`).digest('hex'), ts: 1_700_000_000_000 + i,
+}));   // typed: every tx carries its xmbl type as a micromined xid
 
 const fresh = async () => {
   const dir = await mkdtemp(join(tmpdir(), 'xvsm-apply-'));

package/src/rehydration.test.mjs

@@ -0,0 +1,94 @@
+// THE THREE DEFECTS THAT KEPT NODES OFF A SHARED ROOT, each asserted by the symptom the fleet showed,
+// not by the mechanism. All three were measured live on 2026-09-15 before being fixed here.
+import { test } from 'node:test';
+import assert from 'node:assert';
+import { rmSync } from 'node:fs';
+import { StateMachine } from './state-machine.js';
+
+const ZERO = '0'.repeat(64);
+const anchors = (n) => Array.from({ length: n }, (_, i) => ({
+  event: 'task.created', hash: `h${String(i).padStart(4, '0')}`.padEnd(12, '0'), ts: 1000 + i,
+}));
+const blockFor = (a, tag) => ({ id: `blk-${tag}-${Math.random().toString(16).slice(2)}`,
+  tx: { type: 'anchor', event: a.event, hash: a.hash, ts: a.ts } });
+const countDiffRows = async (sm) => { let n = 0; for await (const [] of sm.db.iterator({ gt: 'diff:', lt: 'diff:\xFF' })) n++; return n; };
+
+async function open(dir) { const sm = new StateMachine({ dbPath: dir }); await sm.ready(); return sm; }
+
+test('a restarted node publishes a REAL root, not 64 zeros, from the state: keyspace alone', async () => {
+  const dir = `/tmp/xvsm-rehydrate-${process.pid}-a`;
+  rmSync(dir, { recursive: true, force: true });
+  let sm = await open(dir);
+  for (const a of anchors(30)) await sm._handleLedgerBlock(blockFor(a, 'x'));
+  const before = sm.stateTree.getRoot();
+  assert.notStrictEqual(before, ZERO);
+  await sm.db.close();
+
+  // Reopen. The diff replay is NOT the thing under test — delete every diff row first, so the only
+  // surviving source for the trie is `state:`. 39 of 44 reporting fleet nodes published ZERO here.
+  sm = await open(dir);
+  await sm.db.clear({ gte: 'diff:', lt: 'diff:\xFF' });
+  await sm.db.close();
+  sm = await open(dir);
+  assert.strictEqual(sm.stateTree.state.size, 30, 'keys must come back');
+  assert.notStrictEqual(sm.stateTree.getRoot(), ZERO, 'a tree holding 30 keys must not commit to zeros');
+  assert.strictEqual(sm.stateTree.getRoot(), before, 'and it must be the SAME root it had before the restart');
+  await sm.db.close();
+  rmSync(dir, { recursive: true, force: true });
+});
+
+test('re-submitting the same anchor under fresh tx ids writes ONE durable row, not one per submission', async () => {
+  const dir = `/tmp/xvsm-rehydrate-${process.pid}-b`;
+  rmSync(dir, { recursive: true, force: true });
+  const sm = await open(dir);
+  const set = anchors(50);
+  for (let pass = 0; pass < 4; pass++) for (const a of set) await sm._handleLedgerBlock(blockFor(a, pass));
+  assert.strictEqual(await countDiffRows(sm), 50, '200 submissions of 50 anchors is 50 state changes');
+  assert.strictEqual(sm.getStatistics().totalTransactions, 50, 'and the published count must agree with disk');
+  await sm.db.close();
+  rmSync(dir, { recursive: true, force: true });
+});
+
+test('the canonical set a node adopts SURVIVES a restart', async () => {
+  const dir = `/tmp/xvsm-rehydrate-${process.pid}-c`;
+  rmSync(dir, { recursive: true, force: true });
+  let sm = await open(dir);
+  const local = anchors(50);
+  for (const a of local) await sm._handleLedgerBlock(blockFor(a, 'local'));
+  const canonical = local.slice(0, 10);
+  const reb = await sm.rebuildFromCanonical(canonical);
+  assert.strictEqual(reb.applied, 10);
+  const canonicalRoot = reb.state_root;
+  await sm.db.close();
+
+  sm = await open(dir);
+  assert.strictEqual(sm.stateTree.state.size, 10, 'the 40 non-canonical keys must not come back');
+  assert.strictEqual(sm.stateTree.getRoot(), canonicalRoot, 'the node must still be on the canonical root');
+  assert.strictEqual(sm.getStatistics().totalTransactions, 10);
+  await sm.db.close();
+  rmSync(dir, { recursive: true, force: true });
+});
+
+test('legacy rows keyed by block id are re-keyed to content identity on the next boot', async () => {
+  const dir = `/tmp/xvsm-rehydrate-${process.pid}-d`;
+  rmSync(dir, { recursive: true, force: true });
+  let sm = await open(dir);
+  // Write rows the way every node on the fleet already has them: one per submission, keyed by block id.
+  const set = anchors(40);
+  let written = 0;
+  for (let pass = 0; pass < 3; pass++) for (const a of set) {
+    const key = `anchor:${a.event}:${a.hash}`;
+    await sm.db.put(`diff:blk-${pass}-${a.hash}`, JSON.stringify({
+      txId: `blk-${pass}-${a.hash}`, timestamp: 1000 + pass, changes: { [key]: { ts: a.ts } },
+    }));
+    written++;
+  }
+  assert.strictEqual(await countDiffRows(sm), written);
+  await sm.db.close();
+
+  sm = await open(dir);
+  assert.strictEqual(await countDiffRows(sm), 40, `${written} legacy rows collapse to 40 distinct changes`);
+  assert.strictEqual(sm.getStatistics().totalTransactions, 40);
+  await sm.db.close();
+  rmSync(dir, { recursive: true, force: true });
+});

package/src/state-diff.js

@@ -5,6 +5,17 @@ export class StateDiff {
     this.changes = changes; // key -> new value
   }
 
+  // THE IDENTITY OF A STATE CHANGE IS THE KEY IT WRITES, not the id of whatever block happened to carry it.
+  // A node mints a fresh tx id every time it re-submits an anchor, so `diff:<txId>` made one durable row per
+  // SUBMISSION: measured on a live node 2026-09-15, 92,505 rows over 50,497 distinct anchors — 42,008 of them
+  // redundant, still growing by ~34 every seven hours, and every one of them replayed into the tree on boot.
+  // Keying by the change itself makes a re-submission an upsert, which is what it always was semantically.
+  // Multi-key changes (explicit `state_diff` txs) have no single content key, so they keep the tx id.
+  identity() {
+    const keys = Object.keys(this.changes || {});
+    return keys.length === 1 ? keys[0] : this.txId;
+  }
+
   apply(state) {
     const newState = { ...state };
     for (const [key, value] of Object.entries(this.changes)) {

package/src/state-machine.js

@@ -32,8 +32,10 @@ export class StateMachine extends EventEmitter {
     // Integration: xclt for state commitments from ledger
     this.xclt = options.xclt || null;
     
-    // Initialize database
-    this._initDb().catch(() => {});
+    // Initialize database. KEEP THE PROMISE: `_dbOpen` flips true the moment Level opens, long before the
+    // diff sweep and the tree rehydration have finished, so anything that polls `_dbOpen` and then reads a
+    // count is reading a store mid-rebuild. `ready()` is the only honest join point.
+    this._ready = this._initDb().catch(() => {});
     
     // Listen to ledger events if available
     if (this.xclt) {
@@ -51,10 +53,16 @@ export class StateMachine extends EventEmitter {
     }
   }
   
+  // Resolves once the store is open, the tree rehydrated from `state:`, and the diff rows swept and loaded.
+  ready() { return this._ready || Promise.resolve(); }
+
   async _initDb() {
     try {
       await this.db.open();
       this._dbOpen = true;
+      // The tree rehydrates itself from the `state:` keyspace; wait for it before deciding whether the diff
+      // rows still need to be replayed, or an unfinished load reads as an empty tree.
+      await this.stateTree.ready();
       await this._loadDiffs();
       await this._loadTransactionLog();
     } catch (error) {
@@ -72,24 +80,46 @@ export class StateMachine extends EventEmitter {
       // Replay is safe to do in iterator order because the tree is a key->value map: the root is a function
       // of the final key set, not of application order (proven in verkle-integration.test.mjs, "same set in
       // any order yields the SAME root"). Later diffs for the same key legitimately overwrite earlier ones.
+      // RE-KEY SWEEP. Rows written before content identity existed are keyed `diff:<block.id>`, so the same
+      // anchor re-submitted under a fresh tx id left one row per submission and every one of them came back
+      // here on boot. Rewriting each row under `diff:<identity>` collapses them: the identity-keyed row is
+      // PUT first and the old key deleted only after, so nothing is lost if this is interrupted. Measured on
+      // a live store 2026-09-15: 92,505 rows over 50,497 distinct anchors, 42,008 of them redundant.
       const loaded = [];
+      let rekeyed = 0;
       for await (const [key, value] of this.db.iterator({ gt: 'diff:', lt: 'diff:\xFF' })) {
         const diffData = JSON.parse(value.toString());
         const diff = new StateDiff(diffData.txId, diffData.changes);
         diff.timestamp = diffData.timestamp;
-        this._recordDiff(diff);
-        loaded.push(diff);
+        const want = this._diffKey(diff);
+        const have = key.toString();
+        if (have !== want) {
+          try { await this.db.put(want, value); await this.db.del(have); rekeyed++; } catch { /* leave it */ }
+        }
+        if (this._recordDiff(diff)) loaded.push(diff);
       }
-      loaded.sort((a, b) => (a.timestamp ?? 0) - (b.timestamp ?? 0) || String(a.txId).localeCompare(String(b.txId)));
+      if (rekeyed) console.log(`[XVSM] diff rows re-keyed to content identity: ${rekeyed}, distinct now ${this.diffs.length}`);
+
+      // ⛔ REPLAY ONLY AS RECOVERY. This loop replayed every diff into the tree unconditionally, which is how
+      // it papered over the real defect — VerkleStateTree._loadState restored the key map but never rebuilt
+      // the trie, so the root read 64 zeros and the replay was the only thing putting keys back through
+      // insert(). With the trie rebuilt on open, an unconditional replay is actively harmful: it reinstates
+      // every key that rebuildFromCanonical deliberately dropped, so the canonical root a node adopts SURVIVES
+      // until its next restart and no further. MEASURED 2026-09-15: rebuild to root 02eaf3e76c over 10
+      // canonical anchors, restart, root 4bc7dd2813 over 50 keys — the node left the canonical set by booting.
+      // Replay stays for the one case it is still needed: a store whose `state:` keyspace is genuinely empty.
       let replayed = 0;
-      for (const diff of loaded) {
-        for (const [k, v] of Object.entries(diff.changes || {})) {
-          if (v === null) { await this.stateTree.delete?.(k); continue; }
-          await this.stateTree.insert(k, v);
-          replayed++;
+      if (this.stateTree.state.size === 0 && loaded.length) {
+        loaded.sort((a, b) => (a.timestamp ?? 0) - (b.timestamp ?? 0) || String(a.txId).localeCompare(String(b.txId)));
+        for (const diff of loaded) {
+          for (const [k, v] of Object.entries(diff.changes || {})) {
+            if (v === null) { await this.stateTree.delete?.(k); continue; }
+            await this.stateTree.insert(k, v);
+            replayed++;
+          }
         }
       }
-      if (replayed) console.log(`[XVSM] verkle tree rehydrated: ${replayed} change(s) from ${loaded.length} diff(s), root=${this.stateTree.getRoot().slice(0, 16)}…`);
+      if (replayed) console.log(`[XVSM] verkle tree recovered from diffs (state: keyspace was empty): ${replayed} change(s) from ${loaded.length} diff(s), root=${this.stateTree.getRoot().slice(0, 16)}…`);
     } catch (error) {
       // Ignore load errors
     }
@@ -112,7 +142,7 @@ export class StateMachine extends EventEmitter {
     if (!this._dbOpen) return;
     
     try {
-      await this.db.put(`diff:${diff.txId}`, JSON.stringify({
+      await this.db.put(this._diffKey(diff), JSON.stringify({
         txId: diff.txId,
         changes: diff.changes,
         timestamp: diff.timestamp
@@ -153,13 +183,22 @@ export class StateMachine extends EventEmitter {
   //
   // REPLACE, don't skip: re-applying a txId with different changes is a legitimate later state for that key
   // and the disk already resolves it that way. Skipping would leave the array disagreeing with the tree.
+  //
+  // INDEXED BY CONTENT IDENTITY, NOT BY TX ID, so the in-memory set matches the durable one key-for-key. Two
+  // submissions of the same anchor carry different tx ids and are the SAME state change; keying either side
+  // by the tx id makes them two of everything. See StateDiff.identity().
   _recordDiff(diff) {
-    const at = this._diffIndex.get(diff.txId);
-    if (at === undefined) { this._diffIndex.set(diff.txId, this.diffs.length); this.diffs.push(diff); return true; }
+    const id = diff.identity();
+    const at = this._diffIndex.get(id);
+    if (at === undefined) { this._diffIndex.set(id, this.diffs.length); this.diffs.push(diff); return true; }
     this.diffs[at] = diff;
     return false;
   }
 
+  // The one place a diff's durable key is spelled. Every writer goes through it, or the re-key sweep in
+  // _loadDiffs is fighting a writer that still uses the block id.
+  _diffKey(diff) { return `diff:${diff.identity()}`; }
+
   _stateChangesFor(block) {
     const tx = block?.tx;
     if (!tx || typeof tx !== 'object') return null;
@@ -196,6 +235,14 @@ export class StateMachine extends EventEmitter {
   }
 
   async _handleLedgerBlock(block) {
+    // JOIN THE STORE BEFORE TOUCHING IT. This handler is wired to `block:added` in the CONSTRUCTOR, so the
+    // ledger can deliver a block while `_initDb` is still opening the db and rehydrating the tree. Both
+    // writes below swallow their errors (by design — an in-memory fallback must not take the node down),
+    // which means an early block was applied to the in-memory tree and persisted NOWHERE: the root looked
+    // right for the life of the process and came back 64 zeros on the next boot, with nothing logged.
+    // MEASURED: applying two blocks before `ready()` resolved gave root ad778ca2… in-process and
+    // 0000000000… after a restart; awaiting `ready()` first, the same root survives.
+    await this.ready();
     const changes = this._stateChangesFor(block);
     if (!changes || !Object.keys(changes).length) return;
     try {
@@ -208,7 +255,7 @@ export class StateMachine extends EventEmitter {
         // StateDiff.serialize() ALREADY returns a JSON string — wrapping it in JSON.stringify again
         // double-encodes, so _loadDiffs parses back a string instead of an object and `changes` comes out
         // undefined, silently replaying nothing. Store the serialized form directly.
-        try { await this.db.put(`diff:${block.id}`, diff.serialize()); }
+        try { await this.db.put(this._diffKey(diff), diff.serialize()); }
         catch { /* in-memory fallback */ }
       }
     } catch (error) {
@@ -252,7 +299,7 @@ export class StateMachine extends EventEmitter {
         this._recordDiff(diff);
         for (const [key, val] of Object.entries(changes)) await this.stateTree.insert(key, val);
         if (this._dbOpen !== false) {
-          try { await this.db.put(`diff:${block.id}`, diff.serialize()); } catch { /* in-memory fallback */ }
+          try { await this.db.put(this._diffKey(diff), diff.serialize()); } catch { /* in-memory fallback */ }
         }
         out.applied++;
       } catch { out.failed++; }
@@ -273,6 +320,18 @@ export class StateMachine extends EventEmitter {
     await this.stateTree.clear();
     this.diffs = [];
     this._diffIndex.clear();
+    // ⛔ THE DURABLE ROWS COME TOO. Clearing `this.diffs` in memory while 92,505 `diff:` rows stay on disk is
+    // not an authoritative rebuild — it is one that lasts until the next boot reads them back. The tree's
+    // `state:` keyspace is cleared above for exactly this reason; the diff keyspace is the same commitment
+    // written twice, and leaving half of it behind is what made the canonical set un-adoptable across a
+    // restart. The blocks themselves are untouched in the ledger: backfillFromLedger regenerates local
+    // history on demand, and the canonical rows are re-persisted below.
+    if (this._dbOpen) {
+      try {
+        if (typeof this.db.clear === 'function') await this.db.clear({ gte: 'diff:', lt: 'diff:\xFF' });
+        else for await (const [k] of this.db.iterator({ gte: 'diff:', lt: 'diff:\xFF' })) { try { await this.db.del(k); } catch { /* */ } }
+      } catch { /* best-effort */ }
+    }
     // ⛔ RECORD A DIFF FOR EVERY ANCHOR APPLIED, or applied_tx_count IS ZERO BY CONSTRUCTION. This loop wrote
     // straight into the tree and never touched `this.diffs`, which it had just emptied — and
     // getStatistics().totalTransactions is transactionLog.length + diffs.length. So the moment a node runs the
@@ -288,7 +347,9 @@ export class StateMachine extends EventEmitter {
         const key = `anchor:${a.event}:${a.hash}`;
         const value = { ts: a.ts ?? null };
         await this.stateTree.insert(key, value);
-        this._recordDiff(new StateDiff(key, { [key]: value }));
+        const diff = new StateDiff(key, { [key]: value });
+        this._recordDiff(diff);
+        if (this._dbOpen) { try { await this.db.put(this._diffKey(diff), diff.serialize()); } catch { /* in-memory fallback */ } }
         out.applied++;
       } catch { out.skipped++; }
     }

package/src/verkle-integration.test.mjs

@@ -10,10 +10,16 @@ import { join } from 'path';
 let pass = 0, fail = 0;
 const check = async (n, f) => { try { await f(); console.log(`  ok   ${n}`); pass++; } catch (e) { console.log(`  FAIL ${n}\n       ${e.message}`); fail++; } };
 const EMPTY = '0'.repeat(64);
-// Portable: this suite runs on the laptop AND on every Linux box in the fleet.
+// Portable: this suite runs on the laptop AND on every Linux box in the fleet — so it JOINS on `ready()`
+// rather than sleeping. A fixed timeout is a guess about someone else's scheduler: this suite waited
+// 120/150/250/500ms for the store to open and the tree to rehydrate, which held on this laptop and LOST on
+// ubuntu-latest, where the restart check read 64 zeros — the 150ms wait expired before `_initDb` had opened
+// the db, so `_handleLedgerBlock` persisted no diff rows and the restart had nothing to replay.
+// `StateMachine.ready()` resolves once the store is open, the tree is rehydrated from `state:` and the diff
+// rows are swept; every write below it is already awaited. There is nothing left to sleep for.
 const dir = join(tmpdir(), 'xvsm-verkle-test');
 const fresh = async () => { rmSync(dir, { recursive: true, force: true });
-  const sm = new StateMachine({ dbPath: dir }); await new Promise(r => setTimeout(r, 120)); return sm; };
+  const sm = new StateMachine({ dbPath: dir }); await sm.ready(); return sm; };
 
 const blk = (id, tx) => ({ id, tx });
 const REAL_TRAFFIC = [
@@ -107,18 +113,40 @@ console.log('\n4. restart survival');
 await check('verkle root survives a restart (diffs are REPLAYED, not just collected)', async () => {
   const d = dir + '-restart';
   rmSync(d, { recursive: true, force: true });
-  let sm = new StateMachine({ dbPath: d }); await new Promise(r => setTimeout(r, 150));
+  let sm = new StateMachine({ dbPath: d }); await sm.ready();
   for (const b of REAL_TRAFFIC) await sm._handleLedgerBlock(b);
   const before = sm.stateTree.getRoot();
   assert.notStrictEqual(before, EMPTY);
-  await new Promise(r => setTimeout(r, 250)); await sm.db.close();
-  sm = new StateMachine({ dbPath: d }); await new Promise(r => setTimeout(r, 500));
+  // No sleep before the close: _handleLedgerBlock awaits both the tree insert (which awaits
+  // _saveState) and the diff `put`, so when the loop above returns, the bytes are already down.
+  await sm.db.close();
+  sm = new StateMachine({ dbPath: d }); await sm.ready();
   const after = sm.stateTree.getRoot();
   await sm.db.close(); rmSync(d, { recursive: true, force: true });
   assert.strictEqual(after, before, 'root changed across restart');
   assert.notStrictEqual(after, EMPTY, 'root reset to zeros on restart');
 });
 
+// The regression the sleeps above were hiding. `_handleLedgerBlock` is wired to `block:added` in the
+// CONSTRUCTOR, so in production a block can arrive while `_initDb` is still opening the store — and both of
+// that handler's writes swallow their errors, so the block landed in the in-memory tree and nowhere on disk.
+// The root read correctly for the life of the process and came back 64 zeros on the next boot, silently.
+// This check applies traffic with NO join at all, the way the ledger's event does.
+await check('a block applied BEFORE the store finished opening still survives a restart', async () => {
+  const d = dir + '-race';
+  rmSync(d, { recursive: true, force: true });
+  let sm = new StateMachine({ dbPath: d });          // deliberately NOT awaiting ready()
+  for (const b of REAL_TRAFFIC) await sm._handleLedgerBlock(b);
+  const before = sm.stateTree.getRoot();
+  assert.notStrictEqual(before, EMPTY, 'control invalid: nothing was applied');
+  await sm.db.close();
+  sm = new StateMachine({ dbPath: d }); await sm.ready();
+  const after = sm.stateTree.getRoot();
+  await sm.db.close(); rmSync(d, { recursive: true, force: true });
+  assert.notStrictEqual(after, EMPTY, 'root came back as 64 zeros — the early block was never persisted');
+  assert.strictEqual(after, before, 'root changed across restart');
+});
+
 rmSync(dir, { recursive: true, force: true });
 console.log(`\n${fail === 0 ? 'PASS' : 'FAIL'} — ${pass} passed, ${fail} failed\n`);
 process.exit(fail === 0 ? 0 : 1);

package/src/verkle-tree.js

@@ -16,9 +16,10 @@ export class VerkleStateTree {
     this.db = options.db || null;
     this._dbOpen = false;
     
-    if (this.db) {
-      this._initDb().catch(() => {});
-    }
+    // KEEP THE PROMISE. This was fire-and-forget, so XVSM's own _loadDiffs raced the tree's _loadState and
+    // could not tell "the tree is empty" from "the tree has not finished loading" — the exact distinction the
+    // replay decision below now turns on. `ready()` is the join point.
+    this._ready = this.db ? this._initDb().catch(() => {}) : Promise.resolve();
   }
   
   async _initDb() {
@@ -33,6 +34,9 @@ export class VerkleStateTree {
     }
   }
   
+  // Resolves once the `state:` keyspace has been read and the trie rebuilt from it.
+  ready() { return this._ready || Promise.resolve(); }
+
   async _loadState() {
     if (!this.db || !this._dbOpen) return;
     
@@ -45,6 +49,30 @@ export class VerkleStateTree {
     } catch (error) {
       // Ignore errors during load
     }
+    // ⛔ REBUILD THE TRIE. This loop restored `this.state` — the key->value map — and stopped there, leaving
+    // `this.root` a fresh empty VerkleNode. So a node that had loaded EVERY key off disk answered getRoot()
+    // with 64 zeros: a tree that holds the whole state and commits to nothing. MEASURED 2026-09-15: after a
+    // canonical rebuild of 10 anchors, restart loaded all 10 keys and published root 0000…0000; across the
+    // fleet 39 of 44 reporting nodes published exactly that, including one with 11,691 blocks persisted. The
+    // only boxes showing a real root were showing it by accident, via XVSM's diff replay re-inserting keys
+    // through insert(). The state map is not the commitment — the hashed trie over it is, and it has to be
+    // reconstructed on open or every restart silently un-commits the node's entire state.
+    this._rebuildTrie();
+  }
+
+  // Rebuild the hashed trie from `this.state` in ONE bottom-up pass. Per-key _updateHashPath would re-hash
+  // every ancestor once per key (32 levels x 8KB of child hashes each); this inserts every key first and
+  // hashes each node exactly once, so a 50k-key store rehydrates in a single walk instead of 1.6M of them.
+  _rebuildTrie() {
+    this.root = new VerkleNode();
+    for (const [k, v] of this.state) {
+      this._insertNode(this.root, this._hashKey(k), this._hashValue(v), 0, []);
+    }
+    const hashSubtree = (node) => {
+      for (const child of node.children.values()) hashSubtree(child);
+      this._updateHash(node);
+    };
+    hashSubtree(this.root);
   }
   
   async _saveState(key, value) {