@coda-console/kernel 0.1.0 → 0.2.1

package/docs/design.md

@@ -345,7 +345,25 @@ kernel.pending({ address });               // { calls, runs, timers, letters }
 
 测试是确定性的模拟：几个实例跑在一个进程里，送信的口、时钟、`schedule` 都由测试控制；时钟只由测试推进，`tick()` 由测试随时调用。测试辅助代码放 `sim/`。
 
-模拟分两层。`sim/world.mjs` 是世界：几个实例、受控的网络与时钟、提交计数、崩溃与提交失败的注入，以及下面的检查。它不认识任何宿主，经 `@coda-console/kernel/sim` 导出，宿主可以把自己真的处理函数接进来撞故障：每一代启动时，世界在迁移的那个事务里调用宿主的 `setup`；`setup` 在事务里，只能在事务外调用的函数放进 `node.recovered(fn)`：世界在这一代的 `recover()` 成功之后立即在事务外调用它，每一代一次，宿主在这里做第 10 节说的重扫。宿主经世界给的连接读写自己的表，这一代死了之后这条连接上的读写一律抛错；经 `node.schedule` 排的事也不再执行，排的事里撞上崩溃或提交失败，由世界接住。世界在宿主登记处理函数时包一层，在处理函数自己的事务里往 `sim_` 开头的表记一行，再照登记的那个对象的方法调用宿主的函数；按处理函数计的检查（S2、S3、S8、S9，取消函数与状态处理函数的次数）读这些表，其余的读内核的表。记账不取决于宿主给了哪些函数：一个地址在这一代登记过任何东西（`handle` 或 `results`），它听到的状态、结局、晚到的结局世界都记一行，宿主给没给对应的函数都一样；取消函数、回答函数会改变内核的回执，给了才包。宿主可以加自己的每步检查和静止点检查。`sim/host.mjs` 是内核自己测试用的宿主。
+模拟分两层。`sim/world.mjs` 是世界：几个实例、受控的网络与时钟、提交计数、崩溃与提交失败的注入，以及下面的检查。它不认识任何宿主，经 `@coda-console/kernel/sim` 导出，宿主可以把自己真的处理函数接进来撞故障：每一代启动时，世界在迁移的那个事务里调用宿主的 `setup`；`setup` 在事务里，只能在事务外调用的函数放进 `node.recovered(fn)`：世界在这一代的 `recover()` 成功之后立即在事务外调用它，每一代一次，宿主在这里做第 10 节说的重扫。宿主经世界给的连接读写自己的表，这一代死了之后这条连接上的读写一律抛错；经 `node.schedule` 排的事也不再执行，排的事里撞上崩溃或提交失败，由世界接住。世界在宿主登记处理函数时包一层，在处理函数自己的事务里往 `sim_` 开头的表记一行，再照登记的那个对象的方法调用宿主的函数；按处理函数计的检查（S2、S3、S8、S9，取消函数与状态处理函数的次数）读这些表，其余的读内核的表。记账不取决于宿主给了哪些函数：一个地址在这一代登记过任何东西（`handle` 或 `results`），它听到的状态、结局、晚到的结局世界都记一行，宿主给没给对应的函数都一样；取消函数、回答函数会改变内核的回执，给了才包。宿主可以加自己的每步检查和静止点检查。
+
+宿主在内核外面有自己一层（收信时核对收发双方、发信时选路、恢复时起自己的重送）时，在 `setup` 里用 `const post = node.ports({ receive, send, recover })` 把这一层接在世界和内核之间。每一代一次，随新的一代重置；哪一样不给，那一样就照世界自己的走法：
+
+- `receive(envelope, from)`：世界把一封信交给这台时调它，代替直接调 `kernel.receive`，返回回执。`from` 是把这封信放进网络的那一台，由世界作保（P3），不取自信封；测试直接调 `world.receive(to, envelope)` 时取信封的 `from` 所属的那台。这台没起来时不调。
+- `send(to, envelope)`：这一代的内核交出一封信时调它，返回回执的 Promise。S6 仍在内核交出的那一刻检查。宿主要把信放进网络，调 `node.ports()` 返回的 `post(to, envelope)`；不给 `send` 时内核交出的信就直接 `post`。`post` 绑在返回它的那一代上：这一代死了之后再调，哪怕新的一代已经起来，也抛崩溃，不入网，不占新一代的额度。
+- `recover()`：世界恢复这一代时调它，代替直接调 `kernel.recover()`。它正常返回，这台才算恢复好，再调 `node.recovered` 登记的函数；它撞上崩溃或提交失败，同 `recover()` 撞上。
+
+世界对接进来的这一层查四条，违反了记进 `world.violations`：**P-in** 世界的一次转交里，`kernel.receive` 至多被调一次。按来源认，不看信的文本，也不看在宿主的第几层回调里：转交期间，宿主手里的内核（`setup` 拿到的 `kernel`、`node.kernel`）的 `receive` 每被调一次算一次，世界自己走默认路径的那一次也算；宿主把信重建一遍（换字段顺序、增删字段）再交，处理函数提交之后把同一封信再交，都算。内核自己内部的交付不经宿主手里的这个句柄，一律不算，比如 `answer()` 回答本机地址时内核经自己的 `receive` 交付；**P-out** 按信的文本、在一代之内累计：同一封信 `post` 的次数不多于这一代内核到此为止交出它的次数。内核重送同一封信是又交出一次；宿主没 `post` 的那次交出，额度一直留在这一代，之后可以多 `post` 一次。没交出过的信一 `post` 就报；**P-rec** `recover` 正常返回时，这一代的 `kernel.recover()` 恰好被调了一次；**P-receipt** 宿主的 `receive` 交回的回执，世界照网上的样子转一遍（JSON 序列化再解析，什么都没交回的当作 `null`），查的是转出来、交给发信一方的那份回执，它带着发信一方的内核要读的那几样：`ok` 是 `true` 或 `false`；`ok: false` 的带字符串 `code`；`ok: true` 的调用、取消的回执带 `view` 对象，`view` 里要么有结局 `outcome`，要么有整数的 `rev` 和 `asked`（内核拿它们比版本、写进调用的行里不许为空的整数列）。取消信的发信方不是调用方（`from` 不等于 `call.from`）时，内核不读也不带 `view`，回 `ok: true` 就够。`view` 的其余字段、状态与结局的回执，内核都不读，世界也不查。回答的回执原样交给回答的宿主，内核不读，但宿主要读、世界的 C8 也读它的 `applied`，所以 `ok: true` 的回答回执要带布尔的 `applied`。缺了的，世界记一条违反，这张回执当作在路上丢了，不交给发信的一方，世界按回执计数的地方也不读它。宿主写自己「恰好一次」的检查，可以用 `node.taken`（世界转交期间宿主手里的内核的 `receive` 被调的次数，按来源认，同 P-in）和 `node.posted`（这台经 `post` 放进网络的封数），两者都跨代累计。
+
+`quiesce` 在第一次静止之后、查各调用的状态之前，先断言 `world.violations` 为空：宿主的错（比如收尾的 drain 里才第一次出现的坏回执）先作为它自己的违反报出来，不报成它留下的状态不一致。
+
+收尾时世界默认每一轮 `pump` 每台，这会替一个从不重送的宿主补上。宿主自己负责重送时，用 `world.quiesce(advance, { resend })`：`resend()` 每一轮调一次，让各台把留着的信再送一遍（比如把宿主的钟推过一个重送周期），世界不再 `pump`；推进时钟、`tick()` 仍由世界做。静不下来时抛出的错写明每台发件箱里还有几封信、还有几条调用没结束。
+
+世界的连接活过每一代，而真的进程一死，连接上的临时表、视图、触发器就没了。所以每一代启动、建内核之前，世界把这条连接上宿主建的临时对象全部删掉，第一代也删，宿主在 `open()` 里建的临时对象留不下来；SQLite 自己管的（名字以 `sqlite_` 开头，如 `AUTOINCREMENT` 带出的 `sqlite_sequence`）删不掉，不删；虚拟表的影子表（`PRAGMA temp.table_list` 里 `type` 为 `shadow` 的）随虚拟表一起删，不单独删。注册的 SQL 函数删不掉，宿主每一代重新注册。
+
+宿主经 `world.node(name, addresses, setup, { open })` 交自己的库时，库里可能已经有内核的表（比如宿主自己的内核建的）。所以一台「迁移过」，世界看自己的 `sim_` 表在不在，不看内核的表：`sim_` 表和内核的迁移、宿主的 `setup` 在同一个事务里建，这台第一代的这个事务提交过才有。没迁移过的台（比如第一代的这次提交被注入失败），每步检查和按台读表的计数都跳过它；之后哪一代起来，迁移照常再跑。
+
+`sim/host.mjs` 是内核自己测试用的宿主，走世界自己的路；`test/ports.test.mjs` 是一个经 `node.ports` 接进来、只靠自己的钟重送的宿主。
 
 故障模型：
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@coda-console/kernel",
-  "version": "0.1.0",
+  "version": "0.2.1",
   "description": "Coda 的调用内核：嵌在宿主进程里的库，只管幂等调用与责任转移",
   "type": "module",
   "license": "UNLICENSED",

package/sim/world.mjs

@@ -7,7 +7,8 @@ import { createKernel } from '../kernel.mjs';
 //
 // The world knows no host. Each instance runs whatever handlers its `setup` registers; the world wraps them to record,
 // in its own `sim_` tables and in the handler's own transaction, what committed. Its checks read those tables and the
-// kernel's, so they hold for any host; a host adds its own with onStep() and onQuiet().
+// kernel's, so they hold for any host; a host adds its own with onStep() and onQuiet(). A host with a layer of its own
+// around the kernel puts it between the world and the kernel with node.ports(), and the world checks that layer too.
 
 /** The instance died right after a commit: the commit stands, everything in memory is gone. */
 export class Crash extends Error {
@@ -62,6 +63,23 @@ const lateEnding = row => {
 };
 const statusOf = row => JSON.stringify([row.status, row.rev, row.asked, row.asking, row.progress]);
 
+/**
+ * Whether the kernel that sent the letter can read `receipt`, as far as it reads one: a refusal's code, which it
+ * compares; and for a call or cancel taken, the view's outcome, and without one the status's rev and asked, integers it
+ * compares and writes into its table. It reads nothing else of a view. An answer's receipt goes to its host, and the
+ * world reads its applied (C8), so that is a boolean.
+ */
+function fits(envelope, receipt) {
+  if (receipt?.ok === false) return typeof receipt.code === 'string';
+  if (receipt?.ok !== true) return false;
+  if (envelope.kind === 'answer') return typeof receipt.applied === 'boolean';
+  if (envelope.kind !== 'call' && envelope.kind !== 'cancel') return true;
+  // A cancel from someone but the caller is taken without a view (the kernel's #revoke).
+  if (envelope.kind === 'cancel' && envelope.from !== envelope.call.from) return true;
+  const { view } = receipt;
+  return typeof view === 'object' && view !== null && (Boolean(view.outcome) || Number.isInteger(view.rev) && Number.isInteger(view.asked));
+}
+
 // The connection one generation of an instance gets, over the database it runs on: it counts commits and fails them
 // on purpose, and once that generation dies every call through it, or through a statement it prepared, throws Crash.
 // Everything else passes through. The kernel's copy also records what it prepares.
@@ -94,14 +112,53 @@ function connection(node, generation, record) {
   });
 }
 
+/**
+ * A generation's host puts a letter its kernel handed over into the network, for `to`, an address of some instance of
+ * the world. P-out counts by the letter's text, over the whole generation: a letter may be posted no more often than
+ * this generation's kernel has handed it over so far. A handing over the host did not post stays to its credit, so a
+ * letter handed over twice and posted once may be posted again later. Once that generation has died it throws Crash,
+ * whatever generation has come since.
+ * @param {SimNode} node
+ * @param {object} generation
+ * @param {string} to
+ * @param {object} envelope
+ * @returns {Promise<object>} the receipt; rejected when the letter or its receipt is lost
+ */
+function post(node, generation, to, envelope) {
+  if (generation.dead) throw new Crash(`${node.name} is down`);
+  const { handed } = generation;
+  const text = JSON.stringify(envelope);
+  const owed = handed.get(text) ?? 0;
+  if (owed === 0) node.world.violations.push(`P-out: ${node.name} posted a letter more often than its kernel handed it over: ${text}`);
+  else if (owed === 1) handed.delete(text);
+  else handed.set(text, owed - 1);
+  node.posted += 1;
+  return new Promise((resolve, reject) => node.world.net.push({ request: true, from: node, to, envelope: text, resolve, reject }));
+}
+
 /**
  * What a host gives an instance: called on every generation inside its first transaction, with migrate(), so what it
  * writes commits with it. It creates the host's own tables and registers its handlers with `kernel.handle` and
  * `kernel.results`. Being inside a transaction, it cannot call what runs only outside one (recover(), tick(), pump(),
- * inquire(), answer(), when()); it hands that to `node.recovered(fn)`.
+ * inquire(), answer(), when()); it hands that to `node.recovered(fn)`. A host with a layer of its own around the kernel
+ * puts it between the world and the kernel with `node.ports()`.
  * @typedef {(context: { kernel: object, node: SimNode, world: World }) => void} Setup
  */
 
+/**
+ * Where a host stands between the world and its kernel, for one generation; each one left out is the world's own way.
+ * - receive: the world hands the host a letter for this instance; `from` is the instance that put it into the network,
+ *   which the world vouches for (P3). Returns the receipt, as its sending kernel reads one (P-receipt).
+ * - send: this generation's kernel hands over a letter; the host puts it into the network with the `post` that
+ *   `node.ports()` returned, or not.
+ * - recover: the world recovers this generation; the host calls `kernel.recover()`.
+ * @typedef {{
+ *   receive?: (envelope: object, from: SimNode) => object,
+ *   send?: (to: string, envelope: object) => Promise<object>,
+ *   recover?: () => void,
+ * }} Ports
+ */
+
 /** One instance: a database that outlives crashes, and the kernel generation running on it. */
 export class SimNode {
   up = false;
@@ -129,6 +186,16 @@ export class SimNode {
   host = undefined;
   /** @type {(() => void) | undefined} what this generation's host does once its recover() has succeeded */
   afterRecover = undefined;
+  /**
+   * calls of receive() on the host's handle while the world was handing this instance a letter, over all generations:
+   * the world's own call on its way, and every one of the host's, whatever the letter and at whatever depth. The kernel
+   * delivering to itself, as answer() does to a local address, does not go through the handle and does not count.
+   */
+  taken = 0;
+  /** letters put into the network with a generation's post, over all generations */
+  posted = 0;
+  /** the world is handing this instance a letter */
+  delivering = false;
 
   /**
    * @param {World} world
@@ -152,10 +219,25 @@ export class SimNode {
    * @param {{ recover?: boolean }} [options]
    */
   boot({ recover = true } = {}) {
-    this.generation = { dead: false };
+    // A process that dies takes its connection's temporary tables, views and triggers with it; this connection outlives
+    // every generation, so the world drops them itself. The first generation too: nothing a host's open() made in temp
+    // is left for it. Triggers go first, as dropping a table drops its own. What SQLite keeps itself, such as the
+    // sqlite_sequence of an AUTOINCREMENT table, cannot be dropped and is no host's. Nor are a virtual table's shadow
+    // tables, which go with it: SQLite itself tells them apart, asking the table's module, in table_list. SQL functions
+    // cannot be taken back: a host registers its own again on every generation.
+    for (const { type, name } of this.q(`SELECT type, name FROM sqlite_temp_master AS m
+      WHERE type IN ('trigger', 'view', 'table') AND name NOT GLOB 'sqlite_*'
+        AND NOT EXISTS (SELECT 1 FROM pragma_table_list AS l WHERE l.schema = 'temp' AND l.name = m.name AND l.type = 'shadow')
+      ORDER BY CASE type WHEN 'trigger' THEN 0 WHEN 'view' THEN 1 ELSE 2 END`).all())
+      this.raw.exec(`DROP ${type.toUpperCase()} temp."${name.replaceAll('"', '""')}"`);
+    // handed: per letter text, how often this generation's kernel handed it over and its host has not posted it yet.
+    const generation = { dead: false, /** @type {Ports} */ ports: {}, handed: new Map(), recovers: 0 };
+    // Bound to this generation, so that a host of a generation that died cannot post through the next one.
+    generation.post = (to, envelope) => post(this, generation, to, envelope);
+    this.generation = generation;
     this.db = connection(this, this.generation, false);
     this.woke = null;
-    this.kernel = createKernel({
+    const kernel = createKernel({
       db: connection(this, this.generation, true),
       now: () => this.world.time,
       send: (to, envelope) => this.world.send(this, to, envelope),
@@ -168,13 +250,33 @@ export class SimNode {
     // What the host asks for, once it commits: a run asked to cancel, and a timer gone from its table without coming
     // due, must be traceable to one of these (C7, 7).
     for (const name of ['cancel', 'abandon', 'stop', 'unschedule']) {
-      const fn = this.kernel[name].bind(this.kernel);
-      this.kernel[name] = (ref, ...rest) => {
+      const fn = kernel[name].bind(kernel);
+      kernel[name] = (ref, ...rest) => {
         this.intents.add(`${name} ${key(ref.from, ref.id)}`);
         return fn(ref, ...rest);
       };
     }
-    this.wrap(this.kernel);
+    // Counted for P-rec.
+    const recovering = kernel.recover.bind(kernel);
+    kernel.recover = () => {
+      generation.recovers += 1;
+      return recovering();
+    };
+    this.wrap(kernel);
+    // The handle the host and the world hold, counted for P-in by where a call of receive() comes from, not by the letter
+    // it carries: one through the handle during a delivery is taken. Every other function is the kernel's own, bound to
+    // it, so the kernel calling its own receive() never comes back through the handle.
+    const receive = envelope => {
+      if (this.delivering) this.taken += 1;
+      return kernel.receive(envelope);
+    };
+    this.kernel = new Proxy(kernel, {
+      get(target, name) {
+        if (name === 'receive') return receive;
+        const value = Reflect.get(target, name);
+        return typeof value === 'function' ? value.bind(target) : value;
+      },
+    });
     this.afterRecover = undefined;
     this.up = true;
     this.ready = false;
@@ -235,8 +337,14 @@ export class SimNode {
   }
 
   recover() {
+    const { generation } = this;
     try {
-      this.kernel.recover();
+      if (generation.ports.recover) {
+        this.world.count('ports: a recovery through the host');
+        generation.ports.recover();
+        if (generation.recovers !== 1)
+          this.world.violations.push(`P-rec: the host of ${this.name} recovered, its kernel's recover() called ${generation.recovers} times, not once`);
+      } else this.kernel.recover();
       this.ready = true;
       this.afterRecover?.();
     } catch (error) {
@@ -262,6 +370,17 @@ export class SimNode {
     this.afterRecover = fn;
   }
 
+  /**
+   * Puts the host between the world and this generation's kernel (see Ports). Called from the setup, once per
+   * generation; the next generation starts without them.
+   * @param {Ports} ports
+   * @returns {(to: string, envelope: object) => Promise<object>} this generation's post (see post())
+   */
+  ports(ports) {
+    this.generation.ports = ports;
+    return this.generation.post;
+  }
+
   /**
    * The host's way to do something later: `fn` runs when the world runs its tasks, unless this generation has died by
    * then. A crash or a failed commit in it is the world's business, as in act().
@@ -303,8 +422,10 @@ export class SimNode {
     if (this.called.size > 1) this.world.violations.push(`S9: one transaction of ${this.name} called handlers of ${[...this.called].join(', ')}`);
   }
 
+  // A first generation's migration committed. The world's own tables tell, as they commit with the kernel's: a database
+  // the host opened may come with the kernel's tables already made, and without the world's.
   migrated() {
-    return this.q("SELECT 1 FROM sqlite_master WHERE name = 'kernel_runs'").get() !== undefined;
+    return this.q("SELECT 1 FROM sqlite_master WHERE name = 'sim_accepted'").get() !== undefined;
   }
 
   // The checks' own statements, kept per database; they bypass the generation's connection.
@@ -438,7 +559,10 @@ export class World {
     node.timers = timers;
   }
 
-  /** The `send` every instance gets. It checks S6 at the moment of handing over; an answer is no letter of the outbox. */
+  /**
+   * The `send` every instance gets. It checks S6 at the moment of handing over; an answer is no letter of the outbox.
+   * The letter then goes to the host's send, or straight into the network.
+   */
   send(node, to, envelope) {
     const call = envelope.kind === 'call' ? envelope : envelope.call;
     const text = JSON.stringify(envelope);
@@ -448,7 +572,11 @@ export class World {
         .get(call.from, call.id, to, envelope.digest) !== undefined
       : node.q('SELECT envelope FROM kernel_outbox WHERE call_from = ? AND call_id = ?').all(call.from, call.id).some(row => row.envelope === text));
     if (!owed) this.violations.push(`S6: ${node.name} sent a letter that is no committed outbox row nor an inquiry about an open call: ${text}`);
-    return new Promise((resolve, reject) => this.net.push({ request: true, to, envelope: text, resolve, reject }));
+    const { handed, ports } = node.generation;
+    handed.set(text, (handed.get(text) ?? 0) + 1);
+    if (!ports.send) return node.generation.post(to, envelope);
+    this.count('ports: a letter out through the host');
+    return ports.send(to, envelope);
   }
 
   /**
@@ -472,7 +600,7 @@ export class World {
     if (this.chance(faults.duplicate)) this.net.push({ ...event, resolve() {}, reject() {} });
     const envelope = JSON.parse(event.envelope);
     this.notice('letter', envelope);
-    let receipt = this.receive(event.to, envelope);
+    let receipt = this.receive(event.to, envelope, event.from);
     if (envelope.kind === 'answer') this.checkAnswer(envelope, receipt);
     this.notice('receipt', envelope, receipt);
     if (this.chance(faults.dropReceipt)) receipt = undefined;
@@ -545,14 +673,37 @@ export class World {
       this.violations.push(`C8: an answer ${key(answer.from, answer.id)} got applied: ${receipt.applied}, but its number was ${taken ? '' : 'not '}taken`);
   }
 
-  receive(to, envelope) {
+  /**
+   * Hands a letter to the instance that owns `to`, through its host's receive when it gave one. The receipt comes back
+   * as it would over a wire; undefined when that instance is down or crashed meanwhile.
+   * @param {string} to
+   * @param {object} envelope
+   * @param {SimNode} [from] the instance that put the letter into the network; called directly, the owner of its `from`
+   */
+  receive(to, envelope, from = this.owners.get(envelope.from)) {
     const node = this.owners.get(to);
     if (!node?.up) return undefined;
+    const { ports } = node.generation;
+    const taken = node.taken;
+    node.delivering = true;
     try {
-      return JSON.parse(JSON.stringify(node.kernel.receive(envelope)));
+      if (!ports.receive) return JSON.parse(JSON.stringify(node.kernel.receive(envelope)));
+      this.count('ports: a letter in through the host');
+      // What goes over the wire is what the sending kernel gets, and what is checked; nothing at all comes as null.
+      const receipt = JSON.parse(JSON.stringify(ports.receive(envelope, from)) ?? 'null');
+      // The sending kernel could not read it: the world tells of it, and the receipt is lost on its way.
+      if (!fits(envelope, receipt)) {
+        this.violations.push(`P-receipt: the host of ${node.name} answered a ${envelope.kind} letter with no receipt of the kernel's: ${JSON.stringify(receipt)}`);
+        return undefined;
+      }
+      return receipt;
     } catch (error) {
       if (!lost(error)) throw error;
       return undefined;
+    } finally {
+      node.delivering = false;
+      if (node.taken - taken > 1)
+        this.violations.push(`P-in: the host of ${node.name} handed one ${envelope.kind} letter to its kernel ${node.taken - taken} times`);
     }
   }
 
@@ -617,19 +768,23 @@ export class World {
    * checks their statuses there; then it rests again, calling `advance` every round, until a round finds nothing to
    * advance. Otherwise it looks at none of the kernel's tables; what they hold at rest is for checkQuiet() to judge.
    * @param {() => boolean} [advance] the hosts move on what they hold by a step; true while there was something to move
+   * @param {{ resend?: () => void }} [options] resend: how the hosts have every instance send what it keeps again, once a
+   *   round; without it the world pumps every instance itself, which would make up for a host that never does
    */
-  async quiesce(advance = () => false) {
+  async quiesce(advance = () => false, { resend } = {}) {
     this.calm = true;
     // Letters already in flight may still come back refused by faults chosen earlier; the pumps that count come after.
     await this.drain();
-    await this.rest(() => false);
+    await this.rest(() => false, resend);
+    // A host's fault met on the way is told of first, not as the kernel's invariant it leaves broken.
+    assert.deepEqual(this.violations, []);
     this.checkStatus();
     this.checkHalted();
-    await this.rest(advance);
+    await this.rest(advance, resend);
   }
 
-  /** @param {() => boolean} advance */
-  async rest(advance) {
+  /** @param {() => boolean} advance @param {(() => void) | undefined} resend */
+  async rest(advance, resend) {
     for (let round = 0; round < 50; round++) {
       const crashed = this.crashed;
       for (const node of this.nodes) {
@@ -637,7 +792,8 @@ export class World {
         else if (!node.ready) node.recover();
       }
       const advanced = advance();
-      for (const node of this.nodes) this.pump(node);
+      if (resend) resend();
+      else for (const node of this.nodes) this.pump(node);
       // The clock passes every timer, and every instance ticks.
       this.time = Math.max(this.time, ...this.nodes.filter(node => node.migrated()).map(node => node.q('SELECT max(due) AS due FROM kernel_timers').get().due ?? 0));
       for (const node of this.nodes) this.tick(node);
@@ -646,7 +802,8 @@ export class World {
       if (this.crashed === crashed && !advanced && this.net.length === 0 && this.tasks.length === 0
         && this.nodes.every(node => node.ready && node.count('SELECT count(*) AS n FROM kernel_timers') === 0)) return;
     }
-    throw new Error('the world did not come to rest');
+    throw new Error(`the world did not come to rest: ${this.nodes.map(node => `${node.name} has ${node.count('SELECT count(*) AS n FROM kernel_outbox')} `
+      + `letters in its outbox and ${node.count(`SELECT count(*) AS n FROM kernel_calls WHERE state IN ${UNENDED}`)} calls not ended`).join(', ')}`);
   }
 
   /** At rest, before the held runs finish: a call that has not ended knows the status its run has now (C6). */