@xmbl/simulator 0.1.0 → 0.1.12

package/CHANGELOG.md

@@ -1,5 +1,89 @@
 # Changelog
 
+## 0.1.12
+
+## 0.1.11
+
+### Patch Changes
+
+- Return every `@xmbl/*` protocol package to ONE version line (0.1.11) after a night of per-package
+  hotfix publishes (cubic-ledger reached 0.1.10 while zero-knowledge sat at 0.1.1, crates at 0.1.0).
+
+  Changes since the last unified line (0.1.0), carried as one PATCH line — 0.1.x stays the pre-mainnet line until the ⛔ AUDIT gates close:
+
+  - cubic-ledger: block ids address CONSENSUS CONTENT (`consensusBody`) instead of the whole tx envelope; an
+    anchor's `hash` must be a sha-256 digest; invalid txs are evicted for good (`evict`, `evicted:` keyspace);
+    legacy envelope-keyed rows converge to content ids on the next boot; a canonical rebuild preserves every
+    non-anchor block and reports what it wiped; `ready()` is the boot join point. `capabilities()` (hardcoded
+    flags) is gone; `census.mjs` (an operator one-off) no longer ships in the tarball. A FORGERY CAN NO LONGER
+    DELETE THE TRANSACTION IT IMPERSONATES: an invalid typed datum is evicted under a digest of its own bytes,
+    never under the xid it claims (a datum fails `validateXid` precisely when that xid is somebody else's), and
+    the anchor dedup key claimed before validation is released on every failure. Both doors are closed on BOTH
+    entry points — `addTransaction` and `addSealedBatch`, the path a finalized transaction actually takes — and
+    one invalid entry in a batch no longer discards the genuine transactions behind it.
+  - consensus: a content-addressed type-6 value tx is admitted on its content address (it carries no in-body
+    signature by design); a malformed anchor is refused at the door by the ledger's own `validateTransaction`;
+    `finalizeTransaction` preserves the signed `id`.
+  - identity: one canonical `signingMessage`; `signingStatus()` / `verifySigning()`; `Identity.fromPrivateKey`
+    (a stub that only threw) is removed.
+  - state-machine: the verkle trie is rebuilt on load; diffs are keyed by content identity; `ready()`.
+  - networking: self-elected circuit-relay server; NO_FATAL transport tolerance; throttled bootstrap warnings.
+  - storage-compute: `CoordinateDelivery` (broken: `require` in ESM, keyed on the public key) is removed.
+  - core: ships the node daemon as the `xmbl-node` bin; the control socket implements the coordinator
+    contract (`xsc`, `submit_batch`, `ledger_capabilities`, `identity_status`, a signed `chain` claim, genuine
+    `submit_tx` rejections) and reports the RUNNING versions; boot waits for the stores.
+  - every package exports a load-time `VERSION`.
+  - PROTOCOL (operator, 2026-09-16): every transaction is typed by its xid (tokens.json type codes; `micromineTx`/
+    `validateXid`; untyped rows deleted on boot, skipped by a canonical rebuild; an anchor's wire tx carries `prior`);
+    consensus validates in order — can it happen, is the xid correct, is the placement right (`validate.js`,
+    `verifyPlacement`); block hashes are content-only so every node seals the same cubes.
+  - ROLLOUT (operator, 2026-09-16): a node proves its version (`build` digest of the loaded code, signed into the
+    `chain` claim; `release` op), suspends itself when behind the npm `latest` of @xmbl/core (no submits, validations
+    or seals), installs the latest over the air and restarts (exit 75 under a supervisor).
+  - lng: ships its BROWSER build — `dist/lng.browser.js` (`@xmbl/lng/browser`), one dependency-free ES module
+    generated from the same `src/*.js` the node runs and byte-checked by the gate (same surface, same bytes, runs
+    with no Node globals); the sources no longer assume `process`/`Buffer`.
+  - lng: the `~bytes` BYTE-STRING TYPE reaches the WASM backend — a (pointer, length) pair on the operand
+    stack, never a 256-bit word: a literal's bytes live in a data segment with a compile-time length, a runtime
+    value (a UTXO id from `xmbl_input_id`) is host-written into fresh memory with its length in a local. A
+    `~bytes` FIELD or PARAM is now REFUSED — committed state and the call ABI are both 32-byte words with
+    nowhere to put a length, and until now both compiled SILENTLY as words. New opt-in host modes
+    `compile(src, { crypto: true })` (`xmbl.mayo.verify` / `xmbl.cubic.verify`) and `{ utxo: true }` (the
+    five-entry value ABI), so a contract written in LNG verifies a signature and spends a UTXO the caller
+    presented. The value ABI's `-1` sentinel TRAPS instead of widening to 2^256-1. The typechecker refuses
+    arithmetic/bitwise/ordering on `~bytes` and now walks `~contract` method bodies at all, which it never did.
+    A contract with no byte literals emits a byte-identical, import-free module.
+  - storage-compute: THE ERASURE CODER RETURNED CORRUPTED DATA WITHOUT SAYING SO. `StorageShard.decode`
+    recovers a lost data shard by XOR-ing its parity group, and XOR parity recovers AT MOST ONE loss per
+    group — but when two members of one group were missing it filled the hole with zeros and returned the
+    buffer as if decoding had succeeded, with no error, no flag and no short read. MEASURED on k=4, m=2:
+    losing data shards 0 and 2 handed back a buffer that differed from the original and nothing downstream
+    could tell. Two independent causes are fixed: the group is now checked for completeness before the XOR
+    is trusted, and `m` — the encoding's PARITY DEGREE — is carried on every shard as the new optional
+    `parityCount` field instead of being inferred from however many parity shards happened to survive (that
+    inference was wrong exactly when a parity shard was among the losses: given data 0,1,2 and parity 4 only,
+    the inferred m was 1, the recovery group became {0,1,2,3} instead of {0,2}, and decode returned wrong
+    bytes). An unrecoverable decode now THROWS and names every missing shard. Proven exhaustively over all
+    63 non-empty subsets of a k=4/m=2 encoding: every subset either decodes to the exact original or throws,
+    and none returns wrong bytes. COMPAT: `parityCount` is additive and optional, so a shard written by an
+    older node reads back fine — a new node treats it as legacy and REFUSES parity recovery rather than
+    guessing, which fails loudly where the old code failed silently. Shard metadata persists as JSON
+    (`meta:<id>`), so an old reader ignores the extra field.
+  - core: a boot crash. `Config._applyEnvOverrides()` assigned into `config.network` / `config.logging`
+    without creating them, so a node started with `XN_PORT` or `LOG_LEVEL` set against a config that omitted
+    those sections died on `undefined.port` before it could log why. The sections are created on demand.
+  - the protocol gate is 75 suites (was 67), and line coverage across the twelve protocol packages is 85.5%
+    (was 80.6%). `scripts/coverage-report.mjs` is the instrument: `NODE_V8_COVERAGE` + a V8-range reducer,
+    since nothing in the tree measured coverage at all. Ten protocol files that no suite had ever loaded now
+    have one; four remain, all process entry points.
+  - NODE 22 IS NOW DECLARED, because it was already REQUIRED. Every published package gains
+    `engines: { node: ">=22" }`; the workspace root's `">=20"` was simply false. `@xmbl/identity` imports
+    `node:sqlite` (Node 22.5+) for the durable nonce registry, `@xmbl/storage-compute` meters jobs with
+    `process.threadCpuUsage` (22.10+), and libp2p's own dependency chain calls `Promise.withResolvers`
+    (22.0). On Node 20 a consumer installed cleanly and crashed at import instead of being told at install
+    time. MEASURED: the protocol gate scores 54/75 on Node 20.20.2 and 75/75 on Node 22 — the twenty-one
+    failures were the runtime, not the code. CI and the release workflow now run Node 22 as well.
+
 All notable changes to the `xsim` module will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),

package/DEVNET-SEAM-FINDING.md

@@ -5,7 +5,7 @@ opts into ledger-side signature verification — it wires **both** `xid` and
 `getPublicKeyByAddress` into the `Ledger` — which is precisely the caller configuration that
 exercises defects no production code path currently reaches. Two of the three defects below are
 now **FIXED**; the third is a genuine signature-domain decision left for the audit and documented
-honestly rather than papered over. All are covered by `src/devnet.test.mjs`.
+accurately rather than papered over. All are covered by `src/devnet.test.mjs`.
 
 ## Where ledger-side verification runs
 
@@ -92,7 +92,7 @@ architectural choices, **which is an audit-level signature-domain / block-identi
 Both touch `block.js`, ledger dedup, and cross-node determinism, so neither rides in on a tooling
 commit. Until one is made and audited, **ledger-side re-verification stays OFF in production**
 (the `getPublicKeyByAddress` lookup is deliberately not wired into the `Ledger`), and consensus
-remains the single verification point. This is the honest, current posture — not a silent gap.
+remains the single verification point. This is the genuine, current posture — not a silent gap.
 
 ## Why the devnet drives the direct path
 

package/index.js

@@ -46,5 +46,8 @@ if (isMainModule || process.argv.includes('--run')) {
 
 export { SystemSimulator, StructuredLogger, LocalDevnet, DevnetRpc };
 
-
-
+// THE VERSION OF THE CODE THIS PROCESS LOADED. Read once at import time from this package's own manifest, so a
+// running node can report what it is actually executing — an install that lands on disk after this module was
+// loaded changes the file, not this constant. Consumed by @xmbl/core's control socket (`status`.versions).
+import { readFileSync as __readPkg } from 'node:fs';
+export const VERSION = JSON.parse(__readPkg(new URL('./package.json', import.meta.url), 'utf8')).version;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@xmbl/simulator",
-  "version": "0.1.0",
+  "version": "0.1.12",
   "type": "module",
   "main": "index.js",
   "description": "XMBL Simulator",
@@ -14,6 +14,9 @@
     "chance": "^1.1.13"
   },
   "license": "MIT",
+  "engines": {
+    "node": ">=22"
+  },
   "publishConfig": {
     "access": "public"
   },

package/src/capabilities.js

@@ -2,7 +2,7 @@
 // zk / homomorphic-encryption / signature primitives in-page (they need node:crypto; an MV3 page
 // CSP and WebCrypto's async sha256 make a faithful in-page port impossible), so the devnet node
 // process — which already loads the REAL modules — runs each primitive end-to-end and returns a
-// JSON-safe verdict. Each runner does the honest thing and its NEGATIVE control in one call: it
+// JSON-safe verdict. Each runner does the right thing and its NEGATIVE control in one call: it
 // proves the capability verifies true input AND rejects tampered input, so a green card on screen
 // is backed by a real refusal, not a fabricated pass. Every flow mirrors the headless
 // reproductions in @xmbl/contracts (reproductions/contract-zk.mjs, contract-he.mjs,
@@ -25,7 +25,7 @@ const CUBE_CONTEXT = {
 
 // ── Coordinate / curve zero-knowledge (@xmbl/zero-knowledge, FRI, ⛔ UNAUDITED) ──
 // A prover shows a coordinate (derivedX, derivedY) lies on a curve through secret points, and a
-// verifier checks it WITHOUT the secret points. We prove the honest coordinate and reject a
+// verifier checks it WITHOUT the secret points. We prove the genuine coordinate and reject a
 // tampered y (derivedY+1) — the exact pair reproductions/contract-zk.mjs gates a Verkle write on.
 export function zkProof({ derivedX = 99 } = {}) {
   const ctx = setup();
@@ -34,14 +34,14 @@ export function zkProof({ derivedX = 99 } = {}) {
   const dX = bi(derivedX);
   const { Pt, derivedY } = blindedCurve(ctx, { publicPoints, secretPoints, derivedX: dX });
   const proof = prove(ctx, { Pt, publicPoints, derivedX: dX, derivedY });
-  const honest = zkVerify(ctx, { proof, publicPoints, derivedX: dX, derivedY });
+  const genuine = zkVerify(ctx, { proof, publicPoints, derivedX: dX, derivedY });
   const tampered = zkVerify(ctx, { proof, publicPoints, derivedX: dX, derivedY: derivedY + 1n });
   return {
-    ok: honest === true && tampered === false,
+    ok: genuine === true && tampered === false,
     scheme: 'xzk (FRI coordinate/curve proof, UNAUDITED)',
     derivedX: str(dX), derivedY: str(derivedY),
     publicAnchors: publicPoints.length,
-    honestVerifies: honest, tamperedRejected: tampered === false,
+    genuineVerifies: genuine, tamperedRejected: tampered === false,
     note: 'proof checked without the secret points; a coordinate off the curve is rejected',
   };
 }

package/src/devnet-rpc.js

@@ -17,7 +17,7 @@
 // It ALSO serves the node-side capability surface the extension cannot run in-page — each a REAL
 // primitive run end-to-end in this node process, verdict returned JSON-safe (see capabilities.js):
 //   getStateRoot    {type}                  → { root, pooled, landed }   // live ledger state root
-//   zkProof         {type, derivedX?}       → { ok, derivedX, derivedY, honestVerifies, tamperedRejected, … }
+//   zkProof         {type, derivedX?}       → { ok, derivedX, derivedY, genuineVerifies, tamperedRejected, … }
 //   heAdd           {type, a?, b?}          → { ok, a, b, sum, expected, … }  // homomorphic add
 //   sigVerify       {type, message?}        → { ok, signedVerifies, tamperedRejected, … }  // Cubic-SIG
 //   seal            {type, secret?}         → { ok, roundTrip, … }             // PQ KEM seal
@@ -69,7 +69,7 @@ export class DevnetRpc {
       }
       default: {
         // Node-side capability surface (zk / HE / signature / seal): a real primitive run to a
-        // verdict in this process. Unknown types still fall through to an honest error.
+        // verdict in this process. Unknown types still fall through to an genuine error.
         const cap = CAPABILITIES[message && message.type];
         if (cap) return cap(message || {});
         return { error: `Unknown message type: ${message && message.type}` };

package/src/devnet-rpc.test.mjs

@@ -54,7 +54,7 @@ try {
 
   // ── node-side capability surface: each a REAL primitive verified + negative-controlled ──
   const zk = await call({ type: 'zkProof', derivedX: 99 });
-  ok('zkProof verifies the honest coordinate and rejects the tampered one', zk.ok === true && zk.honestVerifies === true && zk.tamperedRejected === true, JSON.stringify(zk));
+  ok('zkProof verifies the genuine coordinate and rejects the tampered one', zk.ok === true && zk.genuineVerifies === true && zk.tamperedRejected === true, JSON.stringify(zk));
 
   const he1 = await call({ type: 'heAdd', a: 1, b: 0 });
   ok('heAdd: ENC(1) ⊞ ENC(0) decrypts to 1 (blind add)', he1.ok === true && he1.sum === 1, JSON.stringify(he1));

package/src/devnet.js

@@ -24,7 +24,7 @@
 //   getPublicKeyByAddress); the devnet opts it in, which is why the direct path is what it drives.
 import { EventEmitter } from 'node:events';
 import { Identity } from '../../identity/index.js';
-import { Ledger } from '../../cubic-ledger/index.js';
+import { Ledger, micromineTx } from '../../cubic-ledger/index.js';
 
 export class LocalDevnet extends EventEmitter {
   constructor(options = {}) {
@@ -120,7 +120,8 @@ export class LocalDevnet extends EventEmitter {
       amount,
       timestamp: Date.now(),
     };
-    const signed = await from.signTransaction(tx); // sets from=from.address, adds MAYO sig
+    const typed = micromineTx(tx);                  // the xmbl type: xid micromined over the body, from final
+    const signed = await from.signTransaction(typed); // adds the MAYO sig over everything incl. the xid
     this.metrics.submitted++;
     try {
       const result = await this.ledger.addTransaction(signed);

package/src/devnet.test.mjs

@@ -9,7 +9,9 @@
 // control so a regression fails loudly.
 import assert from 'node:assert/strict';
 import { LocalDevnet } from './devnet.js';
+import { micromineTx } from '../../cubic-ledger/index.js';
 import { Identity } from '../../identity/index.js';
+import { Ledger } from '@xmbl/cubic-ledger';
 import { ConsensusWorkflow } from '../../consensus/index.js';
 
 let pass = 0;
@@ -47,7 +49,7 @@ const ok = (name, cond, detail = '') => {
 
   // NEGATIVE CONTROL — verification is actually ON: a tx tampered after signing is rejected.
   const from = net.identities[0];
-  const signed = await from.signTransaction({ id: 'tamper_1', type: 'utxo', from: from.address, to: net.addressOf(1), amount: 5, timestamp: Date.now() });
+  const signed = await from.signTransaction(micromineTx({ id: 'tamper_1', type: 'utxo', from: from.address, to: net.addressOf(1), amount: 5, timestamp: Date.now() }));
   const tampered = { ...signed, amount: 9999 }; // mutate a signed field
   let threw = '';
   try { await net.ledger.addTransaction(tampered); } catch (e) { threw = e.message; }
@@ -60,7 +62,7 @@ const ok = (name, cond, detail = '') => {
 {
   const net = await new LocalDevnet({ identities: 2 }).start();
   const from = net.identities[0];
-  const signed = await from.signTransaction({ id: 'seam_client_id', type: 'utxo', from: from.address, to: net.addressOf(1), amount: 7, timestamp: Date.now() });
+  const signed = await from.signTransaction(micromineTx({ id: 'seam_client_id', type: 'utxo', from: from.address, to: net.addressOf(1), amount: 7, timestamp: Date.now() }));
 
   // FIX(a): consensus finalizeTransaction now PRESERVES the originator's signed `id`. It used to
   // overwrite it with validatedHash (workflow.js), corrupting the signed message so the ledger's
@@ -77,18 +79,62 @@ const ok = (name, cond, detail = '') => {
     ok('FIX(a): finalize PRESERVES the signed id (not validatedHash)', !!emitted && emitted.txData.id === 'seam_client_id', `id=${emitted?.txData?.id}`);
     ok('FIX(a): the finalized tx STILL verifies against the signer key', (await Identity.verifyTransaction(emitted.txData, from.publicKey)) === true);
     ok('FIX(a): the consensus hash is carried separately as the event txId', emitted.txId === validatedHash);
-    // OPEN defect (c), PINNED here (not only in DEVNET-SEAM-FINDING.md prose): moveToProcessing
-    // injects a `validationTimestamp` INTO the signed tx body, which is outside the signed domain,
-    // so a real finalized tx does NOT re-verify against the signer key. This is why ledger-side
-    // re-verification stays OFF in production. If someone wires getPublicKeyByAddress into the
-    // Ledger without first resolving (c) (block-id-from-signed-body OR carrying validationTimestamp
-    // as a sibling), THIS assertion flips and the gate fails — which is the intended tripwire.
+    // WHY A TX MUST NOT CARRY THE CONSENSUS CLOCK: identity's signingMessage covers every field except
+    // sig/publicKey, so a `validationTimestamp` written into the body is a value the signer never saw.
     const withVt = { ...signed, validationTimestamp: '123' };
-    ok('FIX(a) scope: a tx carrying consensus-injected validationTimestamp does NOT re-verify (open defect (c))',
+    ok('a tx carrying a validationTimestamp INSIDE the signed body does NOT re-verify (why A5 moved it out)',
        (await Identity.verifyTransaction(withVt, from.publicKey)) === false);
     try { await w.mempool?.db?.close?.(); } catch { /* in-memory / already closed */ }
   }
 
+  // FIX(c) / A5 — DRIVEN AT THE REAL SITE, not asserted on a hand-built object. The previous version of this
+  // block pinned the DEFECT by checking a property of verifyTransaction itself, which no amount of fixing
+  // moveToProcessing could ever flip — a tripwire that could not trip. This runs the actual pipeline:
+  // addRawTransaction → quorum timestamps → moveToProcessing → finalizeTransaction → the ledger, with a key
+  // resolver wired exactly as the daemon wires it (B7). If consensus ever writes its clock back into the
+  // signed body, the finalized tx stops verifying and the ledger refuses it — here, in the gate.
+  {
+    const w = new ConsensusWorkflow({});
+    const rawTxId = await w.mempool.addRawTransaction('leaderA', { ...signed });
+    w.rawTxToId.set(rawTxId, 'leaderA');
+    const entry = w.mempool.rawTx.get('leaderA').get(rawTxId);
+    entry.validationTimestamps = [
+      { nodeId: 'v1', timestamp: 1789470971116000000n },
+      { nodeId: 'v2', timestamp: 1789470971117000000n },
+      { nodeId: 'v3', timestamp: 1789470971118000000n },
+    ];
+    await w.moveToProcessing(rawTxId);
+    const [key, processing] = [...w.mempool.processingTx.entries()][0];
+    ok('A5: the processing tx body carries NO validationTimestamp (the clock is a sibling)',
+       processing.txData.validationTimestamp === undefined, `body=${JSON.stringify(Object.keys(processing.txData))}`);
+    ok('A5: the clock travels BESIDE the tx and is non-null', processing.validationTimestamp != null);
+    ok('A5: the processing tx STILL verifies against the signer key (the whole point)',
+       (await Identity.verifyTransaction(processing.txData, from.publicKey)) === true);
+
+    let fin = null;
+    w.on('tx:finalized', (d) => { fin = d; });
+    await w.finalizeTransaction(key);
+    ok('A5: the FINALIZED tx verifies against the signer key', !!fin && (await Identity.verifyTransaction(fin.txData, from.publicKey)) === true);
+    ok('A5: the finalized event carries the clock as its own field', fin.validationTimestamp != null && fin.txData.validationTimestamp === undefined);
+
+    // B7 — the ledger, with the resolver the daemon now wires, verifies the same tx a SECOND time.
+    const led = new Ledger({});
+    led.xid = from;                                   // enables the ledger's signature check
+    led.getPublicKeyByAddress = (addr) => (addr === from.address ? from.publicKey : null);
+    const res = await led.addTransaction(fin.txData, { validationTimestamp: fin.validationTimestamp });
+    ok('B7: a signed finalized tx is ACCEPTED at the ledger with re-verification ON', !!res && res.duplicate !== true);
+    const block = [...led._membershipPool, ...led.blocks.values()].find((b) => b.tx && b.tx.id === 'seam_client_id');
+    ok('B7: the block carries the consensus clock as its own field, and its tx does not',
+       !!block && block.validationTimestamp != null && block.tx.validationTimestamp === undefined);
+    let refused = '';
+    try { await led.addTransaction({ ...fin.txData, amount: 424242 }, { validationTimestamp: fin.validationTimestamp }); }
+    catch (e) { refused = e.message; }
+    ok('B7: a tx tampered AFTER consensus is REFUSED at the ledger (the defense-in-depth layer is live)',
+       /Invalid transaction signature|address mismatch/.test(refused), `threw="${refused}"`);
+    try { await w.mempool?.db?.close?.(); } catch { /* in-memory / already closed */ }
+    try { await led.close?.(); } catch { /* in-memory */ }
+  }
+
   // FIX(a) negative control: mutating `id` AFTER signing (what the old overwrite effectively did)
   // is REJECTED by the ledger — `id` is inside the signed domain, which is exactly why consensus
   // must not overwrite it.

package/src/simulator.js

@@ -613,7 +613,7 @@ export class SystemSimulator extends EventEmitter {
     // Compute operations can trigger state changes. This used to call a fake
     // xvsm.executeTransaction with a comment string as "WASM"; that path is gone (real
     // contract execution is @xmbl/contracts' ContractHost over @xmbl/storage-compute). The
-    // simulator's honest job here is just to drive a state change, so it uses the real
+    // simulator's genuine job here is just to drive a state change, so it uses the real
     // state-diff path directly.
     if (this.modules.xvsm && chance.bool({ likelihood: 70 })) {
       setTimeout(() => {