@effectstream/runtime 0.100.22 → 0.100.24

package/README.md

@@ -69,6 +69,40 @@ exercised when this loop is running. The `StartConfig` shape determines
 which DB migrations apply, which state machines fire, and whether the
 runtime exposes a Fastify router.
 
+## Empty-block coalescing (catch-up)
+
+The runtime replays one Effectstream block per main-clock tick, each in its
+own DB transaction. During a deep catch-up over an idle gap (e.g. an NTP main
+chain at one block/second across days), most of those blocks are **empty** — no
+on-chain content, no scheduled input, no migration — yet still cost a full
+`BEGIN…COMMIT` round-trip.
+Empty-block coalescing folds a run of consecutive empty blocks into a **single
+transaction** that writes only the run's endpoint. It is **opt-in** and off by
+default:
+
+```bash
+EFFECTSTREAM_COALESCE_EMPTY_BLOCKS=true
+```
+
+Behavior and guarantees:
+
+- **Catch-up only.** Coalescing engages only while behind the chain tip
+  (`now - block.timestamp` greater than 10× the main clock's block time) and
+  disengages at the tip, so steady-state stays one block per transaction.
+- **Identical database.** A coalesced sync produces a byte-for-byte identical
+  database to a non-coalesced one. Empty blocks already never advance the block
+  hash or RNG seed, so only the endpoint's block row and the per-protocol resume
+  watermark are written — exactly as the normal path would leave them.
+- **Never skips work.** A run is flushed before any block that has on-chain
+  content, a migration at its height, or a due block-height/timestamp-scheduled
+  input — so nothing that mutates state is ever folded away.
+- **Tip stays visible.** If the producer goes quiet on an empty tail, the run is
+  flushed on a short idle timeout so the latest height still commits.
+- A run is also capped (`MAX_EMPTY_RUN`) to bound transaction size and recovery.
+
+Implementation: [`src/coalesce.ts`](./src/coalesce.ts) (the `createEmptyBlockCoalescer`
+coordinator the main loop drives).
+
 ## Key exports
 
 - `init()` - `Operation<void>`. One-shot setup: OpenTelemetry, config validation, version pinning. Call before `start`.

package/conciseinputhash-determinism.md

@@ -0,0 +1,114 @@
+# Bug Report: non-deterministic `effectstream_tx_hash` (`conciseInputHash`)
+
+**Package:** `@effectstream/node-sdk` — runtime
+**Location:** `runtime/src/process-blocks.ts` (scheduled-input loop, STEP 5)
+**Severity:** Medium — silent cross-node state divergence in a derived column; not (currently) consensus-relevant
+**Status:** Open
+
+---
+
+## Summary
+
+When the runtime records the result of a **scheduled** STF input, it fabricates the
+`effectstream_tx_hash` with `Math.random()`. The value is therefore different on every run
+and on every node, so two nodes that replay the exact same source blocks produce **different
+`rollup_input_result.effectstream_tx_hash` values** for the same input. For a framework whose
+core promise is "the database is a pure function of the source blocks," this is a determinism
+hole.
+
+## The code
+
+`runtime/src/process-blocks.ts`, inside the loop over `scheduledData` (STEP 5):
+
+```ts
+const conciseInputHash = `0x${
+  Array(64).fill(0).map(() => Math.floor(Math.random() * 16).toString(16)).join("")
+}`;
+yield* until(
+  insertGameInputResult.run({
+    id: data.id,
+    success,
+    effectstream_tx_hash: Buffer.from(conciseInputHash),
+    index_in_block,
+    block_height: value.blockNumber,
+  }, dbConn),
+);
+```
+
+This runs only for **scheduled** inputs (engine-queued timers / derived inputs that have no
+real on-chain transaction), so the engine has to synthesize a hash. It does so randomly.
+
+## Why it matters / why it's currently masked
+
+- **Block hash is unaffected.** `generateEffectstreamBlockHash` (`@effectstream/crypto`) hashes
+  `blockInfo[].blockHash` (source-chain hashes) + the previous block hash — never
+  `effectstream_tx_hash`. So the block-hash chain stays deterministic; the divergence is
+  confined to the `rollup_input_result.effectstream_tx_hash` column and anything that reads it.
+- **Masked in tests.** The reproduction tests use a no-op STF (`stateMachinePayload: null`),
+  which schedules no STF inputs, so this branch never executes. That is exactly why
+  `consistency-snapshot.ts` lists `effectstream_tx_hash` in `VOLATILE_COLUMNS` and excludes it
+  — the snapshot would otherwise diff on every run once real STFs are involved.
+
+## Two existing quirks to fix alongside
+
+1. **Encoding bug.** `Buffer.from(conciseInputHash)` is given the *string* `"0x" + 64 hex`
+   with no encoding, so it stores 66 **UTF-8** bytes, not the 32 decoded bytes. Whatever
+   derivation replaces `Math.random()`, encode it deliberately (`Buffer.from(hex, "hex")`, or
+   commit to the `0x`-string form consistently).
+
+2. **Ordering prerequisite (the real root).** The hash inputs that give uniqueness
+   (`index_in_block`, and the STF execution order itself) are only deterministic if the
+   `scheduledData` order is deterministic. Today it is not fully:
+   ```ts
+   // TODO What should be the order of the scheduled data - per id?
+   const scheduledData = [...scheduledData1, ...scheduledData2];
+   ```
+   - `scheduledData1` (block-scheduled) is `ORDER BY id ASC` → deterministic given a
+     deterministic scheduling history.
+   - `scheduledData2` (timestamp-scheduled) is `ORDER BY future_ms_timestamp ASC` **only** —
+     **same-timestamp rows have no tiebreak**, so their order is arbitrary.
+
+   This is bigger than the hash: the same order drives **STF execution** and the
+   **`randomGenerator` (Prando) draw sequence**. If it isn't pinned, game state itself can
+   diverge; the random tx-hash is just the most visible symptom.
+
+## Suggested fixes (in order)
+
+### 1. Pin the scheduled-input order (prerequisite, do first)
+Give the queries a total deterministic sort and a deterministic merge:
+- timestamp list → `ORDER BY future_ms_timestamp ASC, id ASC` (break ties by `id`);
+- keep block-scheduled before timestamp-scheduled (or merge both by a single deterministic
+  key). This is required for deterministic STFs regardless of the hash, and it makes
+  `index_in_block` deterministic.
+
+### 2. Derive the hash deterministically (recommended)
+Replace `Math.random()` with a content-addressed hash over the input's identity + position,
+reusing the `sha512` already used for the block hash:
+
+```
+effectstream_tx_hash =
+  sha512(block_height ‖ index_in_block ‖ input_data ‖ from_address ‖ from_address_type)
+```
+
+- `block_height` + `index_in_block` give **uniqueness** (one result per position; needs fix #1).
+- `input_data` + `from_address` + `from_address_type` make it **content-addressed**
+  (tx-hash-like, collision-resistant). All are non-null columns already on the scheduled row.
+
+### Alternatives considered
+- **Reuse the seeded `Prando` (`new Prando(blockHash)`):** smallest change and deterministic,
+  but the shared `randomGenerator` is also consumed by the STFs — drawing hash bytes from it
+  would shift the RNG stream every later STF sees. Would require a *separate*
+  `new Prando(blockHash + ":txhash")`. Produces opaque (non-content) hashes.
+- **`H(data.id, block_height)`:** `id` is a global SERIAL counter — a weaker determinism
+  guarantee than the block-scoped `index_in_block`, and `block_height` adds nothing once `id`
+  is unique. Prefer `index_in_block`.
+- **Provenance: `H(origin_tx_hash, …)`:** ties to the originating real tx, but
+  `origin_tx_hash` is `null` for timestamp-scheduled inputs, so it needs a fallback.
+
+## Verification
+- Add a determinism test with a **real** STF (one that schedules a derived input) and assert
+  two from-scratch syncs produce identical `rollup_input_result.effectstream_tx_hash` —
+  i.e. drop `effectstream_tx_hash` from `VOLATILE_COLUMNS` once fixed and confirm
+  `consistency.test.ts` Test A still passes.
+- Cover the timestamp-tie case explicitly: schedule ≥2 inputs at the same `future_ms_timestamp`
+  and assert identical ordering / hashes across runs.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@effectstream/runtime",
-  "version": "0.100.22",
+  "version": "0.100.24",
   "license": "MIT",
   "homepage": "https://effectstream.github.io/docs/",
   "repository": {
@@ -13,17 +13,17 @@
   },
   "exports": "./src/mod.ts",
   "dependencies": {
-    "@effectstream/concise": "0.100.22",
-    "@effectstream/config": "0.100.22",
-    "@effectstream/coroutine": "0.100.22",
-    "@effectstream/crypto": "0.100.22",
-    "@effectstream/db": "0.100.22",
-    "@effectstream/event-client": "0.100.22",
-    "@effectstream/event-server": "0.100.22",
-    "@effectstream/log": "0.100.22",
-    "@effectstream/sm": "0.100.22",
-    "@effectstream/sync": "0.100.22",
-    "@effectstream/utils": "0.100.22",
+    "@effectstream/concise": "0.100.24",
+    "@effectstream/config": "0.100.24",
+    "@effectstream/coroutine": "0.100.24",
+    "@effectstream/crypto": "0.100.24",
+    "@effectstream/db": "0.100.24",
+    "@effectstream/event-client": "0.100.24",
+    "@effectstream/event-server": "0.100.24",
+    "@effectstream/log": "0.100.24",
+    "@effectstream/sm": "0.100.24",
+    "@effectstream/sync": "0.100.24",
+    "@effectstream/utils": "0.100.24",
     "@fastify/cors": "^11.0.1",
     "@metamask/json-rpc-engine": "^10.0.3",
     "pg": "8.21.0",

package/src/api/apply-status.ts

@@ -0,0 +1,30 @@
+/**
+ * Apply-stage liveness for /debug/metrics.
+ *
+ * The serial finalized-block loop in main.ts calls recordAppliedBlock() right
+ * after each block COMMITs; the /debug/metrics handler reads `appliedBlockStatus`
+ * to report true lag (now − applied block timestamp). Unlike a protocol's `buf`
+ * (fetch backlog), this stays high when the node is write/apply-bound.
+ *
+ * Process-global: one node runs per process, so a module-level singleton keeps
+ * main.ts to a single call instead of threading a ref through startHttpServer.
+ */
+export type AppliedBlockStatus = {
+  blockNumber: number | null;
+  timestamp: number | null;
+  appliedAtMs: number | null;
+};
+
+export const appliedBlockStatus: AppliedBlockStatus = {
+  blockNumber: null,
+  timestamp: null,
+  appliedAtMs: null,
+};
+
+export function recordAppliedBlock(
+  block: { blockNumber: number | bigint; timestamp: number | bigint },
+): void {
+  appliedBlockStatus.blockNumber = Number(block.blockNumber);
+  appliedBlockStatus.timestamp = Number(block.timestamp);
+  appliedBlockStatus.appliedAtMs = Date.now();
+}

package/src/api/http-server.ts

@@ -1,5 +1,7 @@
 import fastify, { type FastifyInstance, type FastifyReply, type FastifyRequest } from "fastify";
 import { evmRpcEngine } from "./rpc-evm/eip1193.ts";
+import { appliedBlockStatus } from "./apply-status.ts";
+import { finalizedStreamStatus } from "./stream-status.ts";
 import type { Pool } from "pg";
 import cors from "@fastify/cors";
 import { run, until } from "effection";
@@ -13,6 +15,7 @@ import {
   getTableSchema,
   type IGetAllAddressesResult,
   type IGetAllTableNamesResult,
+  poolErrors,
   releaseDBMutex,
   runPreparedQuery,
   waitUntilFree,
@@ -364,19 +367,29 @@ export const startHttpServer = function* (
     yield* until(apiRouter(server, dbConn));
   }
 
+  const HealthDbSchema = Type.Object({
+    consecutive: Type.Number(),
+    firstFailureAt: Type.Number(),
+    sustainedDurationMs: Type.Number(),
+    sustained: Type.Boolean(),
+  });
+  const HealthResponseSchema = Type.Object({
+    status: Type.String(),
+    db: HealthDbSchema,
+  });
   server.get("/health", {
     schema: {
       tags: ["status"],
       response: {
-        200: Type.Object({
-          status: Type.String(),
-        }),
+        200: HealthResponseSchema,
+        503: HealthResponseSchema,
       },
     },
-  }, () => {
-    return {
-      status: "ok",
-    };
+  }, (_req, reply) => {
+    const db = poolErrors.state();
+    return reply
+      .status(db.sustained ? 503 : 200)
+      .send({ status: db.sustained ? "db-unreachable" : "ok", db });
   });
 
   server.get("/addresses", {
@@ -469,6 +482,54 @@ export const startHttpServer = function* (
   });
 
   if (ENV.ENABLE_DEV_AND_DEBUG_ENDPOINTS) {
+    server.get("/debug/metrics", {
+      schema: {
+        tags: ["developer"],
+        response: {
+          200: Type.Object({}, { additionalProperties: true }),
+        },
+      },
+    }, () => {
+      const appliedTs = appliedBlockStatus.timestamp;
+      return {
+        timestamp: Date.now(),
+        uptimeSeconds: process.uptime(),
+        memory: process.memoryUsage(),
+        protocols: syncProtocols.map((p) => ({
+          name: p.name,
+          buf: p.bufferedData.size(),
+          ownBlockNumber: p.lastPage?.ownBlockNumber ?? null,
+          // Backpressure observability
+          cap: p.bufferCap,
+          bufHighWater: p.bufferHighWater,
+          pausedNow: p.pausedNow,
+          pauses: p.backpressurePauses,
+          pausedMs: p.backpressurePausedMs,
+          // Merge-demand exemption: when true the merge is gated on this chain's
+          // page and the cap is lifted.
+          mergeWaiting: p.mergeWaitingForPage,
+          mergeDemandRoot: p.mergeDemandRoot ?? null,
+        })),
+        // Finalized-block stream backlog: blocks the merge has produced but the
+        // apply loop hasn't drained. `inFlight` is the otherwise-unobservable
+        // subscriber-queue depth, bounded by backpressure (sync/CLAUDE.md Finding #1).
+        // `coalesced` = empty blocks folded away during catch-up.
+        finalizedStream: {
+          inFlight: finalizedStreamStatus.produced - finalizedStreamStatus.consumed,
+          coalesced: finalizedStreamStatus.coalesced,
+        },
+        // Apply-stage lag: how old the last-applied block is. Unlike `buf` (fetch
+        // backlog) this stays high when the node is write/apply-bound.
+        applied: {
+          blockNumber: appliedBlockStatus.blockNumber,
+          timestamp: appliedTs,
+          lagSeconds: appliedTs != null
+            ? Math.round((Date.now() - appliedTs) / 100) / 10
+            : null,
+        },
+      };
+    });
+
     server.get("/debug/sync-protocols", {
       schema: {
         tags: ["developer"],

package/src/api/stream-status.ts

@@ -0,0 +1,35 @@
+/**
+ * Finalized-block stream backlog for /debug/metrics.
+ *
+ * `finalized-stream.ts` bumps `produced` when the merge sends a block and `consumed`
+ * when the apply loop drains one; their difference is the live subscriber-queue
+ * depth that drives backpressure and is surfaced as `inFlight` on /debug/metrics
+ * (the one structure flagged unbounded in sync/CLAUDE.md Finding #1). `coalesced`
+ * counts empty blocks folded away, so a catch-up test can prove coalescing fired.
+ *
+ * Process-global: one node runs per process, so a module-level singleton avoids
+ * threading a ref through startMerge and the coalescer.
+ */
+export type FinalizedStreamStatus = {
+  produced: number;
+  consumed: number;
+  coalesced: number;
+};
+
+export const finalizedStreamStatus: FinalizedStreamStatus = {
+  produced: 0,
+  consumed: 0,
+  coalesced: 0,
+};
+
+export function recordProduced(): void {
+  finalizedStreamStatus.produced++;
+}
+
+export function recordConsumed(n: number): void {
+  finalizedStreamStatus.consumed += n;
+}
+
+export function recordCoalesced(n: number): void {
+  finalizedStreamStatus.coalesced += n;
+}

package/src/coalesce.ts

@@ -0,0 +1,291 @@
+/** Empty-block coalescing during deep catch-up: fold consecutive no-op blocks into one transaction. */
+import type { ChainBlock } from "@effectstream/sync";
+import { call, type Operation, race, sleep, until } from "effection";
+import type { Pool, PoolClient } from "pg";
+import {
+  acquireDBMutex,
+  blockHeightDone,
+  deleteEmptyBlocks,
+  getEarliestScheduledBlockHeight,
+  getEarliestScheduledTimestamp,
+  getMigrationsForBlockHeight,
+  isTransientPgError,
+  poolErrors,
+  releaseDBMutex,
+  removePages,
+  saveLastBlock,
+  upsertPage,
+} from "@effectstream/db";
+import { Buffer } from "node:buffer";
+import { ComponentNames, log, SeverityNumber } from "@effectstream/log";
+import { generateEffectstreamBlockHash, Prando } from "@effectstream/crypto";
+import type { EffectstreamBlockHash } from "@effectstream/utils";
+import type { DBMigrations } from "@effectstream/runtime";
+
+/** Scheduled-input boundaries sampled once at run start (stable until the next non-empty block). */
+export type EmptyRunBoundaries = {
+  minScheduledBlockHeight: number | null;
+  minScheduledTimestampMs: number | null;
+};
+
+export const MAX_EMPTY_RUN = 5_000;
+
+const MAX_RETRY_BACKOFF_MS = 5_000;
+
+export function* loadEmptyRunBoundaries(
+  dbConn: Pool,
+): Operation<EmptyRunBoundaries> {
+  const [bh] = yield* until(getEarliestScheduledBlockHeight.run(undefined, dbConn));
+  const [ts] = yield* until(getEarliestScheduledTimestamp.run(undefined, dbConn));
+  return {
+    minScheduledBlockHeight: bh?.min_block_height ?? null,
+    minScheduledTimestampMs: ts?.min_timestamp != null
+      ? new Date(ts.min_timestamp).getTime()
+      : null,
+  };
+}
+
+/** Mirrors the empty-block checks in process-blocks.ts without DB/STF work. */
+export function isCoalescableEmptyBlock(
+  value: ChainBlock,
+  boundaries: EmptyRunBoundaries,
+  migrations: DBMigrations[] | undefined,
+): boolean {
+  if (value.primitives.length > 0) return false;
+  if (
+    getMigrationsForBlockHeight(migrations, value.blockNumber, value.blockNumber)
+      .length > 0
+  ) {
+    return false;
+  }
+  if (
+    boundaries.minScheduledBlockHeight != null &&
+    value.blockNumber >= boundaries.minScheduledBlockHeight
+  ) {
+    return false;
+  }
+  if (
+    boundaries.minScheduledTimestampMs != null &&
+    boundaries.minScheduledTimestampMs <= value.timestamp
+  ) {
+    return false;
+  }
+  return true;
+}
+
+export function* processCoalescedEmptyRun(
+  endpoint: ChainBlock,
+  pool: Pool,
+  previousBlockHash: EffectstreamBlockHash | null,
+): Operation<void> {
+  const lockName = `processing-blocks:${endpoint.blockNumber}`;
+  let attempt = 0;
+  while (true) {
+    let dbClient: PoolClient | undefined;
+    let transientErr: unknown;
+    try {
+      yield* acquireDBMutex(lockName);
+      dbClient = yield* until(pool.connect());
+      yield* writeEndpoint(
+        endpoint,
+        dbClient as unknown as Pool,
+        previousBlockHash,
+      );
+      return;
+    } catch (err) {
+      if (!isTransientPgError(err)) throw err;
+      transientErr = err;
+    } finally {
+      releaseDBMutex(lockName);
+      if (dbClient) {
+        try {
+          dbClient.release(transientErr as Error | undefined);
+        } catch {
+          /* already released/destroyed */
+        }
+      }
+    }
+
+    attempt++;
+    poolErrors.log(transientErr, ["sync-write"]);
+    const backoffMs = Math.min(500 * 2 ** (attempt - 1), MAX_RETRY_BACKOFF_MS);
+    log.local(
+      ComponentNames.EFFECTSTREAM_SYNC,
+      "block-processing",
+      SeverityNumber.WARN,
+      (l) =>
+        l(
+          `transient db error on coalesced run → block ${endpoint.blockNumber}; retry #${attempt} in ${backoffMs}ms: ${
+            String(transientErr)
+          }`,
+        ),
+    );
+    yield* sleep(backoffMs);
+  }
+}
+
+/** One transaction for the run endpoint; matches process-blocks.ts empty-block writes. */
+function* writeEndpoint(
+  endpoint: ChainBlock,
+  dbConn: Pool,
+  previousBlockHash: EffectstreamBlockHash | null,
+): Operation<void> {
+  const blockHash = generateEffectstreamBlockHash(endpoint, previousBlockHash);
+  const seed = new Prando(blockHash).seed.toString();
+  try {
+    yield* until(dbConn.query("BEGIN"));
+
+    yield* call(() =>
+      saveLastBlock.run({
+        block_height: endpoint.blockNumber,
+        ver: 0,
+        main_chain_block_hash: Buffer.from(endpoint.blockNumber.toString()),
+        seed,
+        ms_timestamp: new Date(endpoint.timestamp),
+      }, dbConn)
+    );
+
+    yield* call(() => deleteEmptyBlocks.run(undefined, dbConn));
+    yield* call(() =>
+      blockHeightDone.run({
+        block_hash: Buffer.from("0x0"),
+        block_height: endpoint.blockNumber,
+      }, dbConn)
+    );
+
+    for (const { protocol_name, lastPage } of endpoint.resumePages) {
+      yield* until(
+        upsertPage.run({
+          protocol_name,
+          page_number: lastPage.ownBlockNumber,
+          page: JSON.stringify(lastPage),
+        }, dbConn),
+      );
+      yield* until(
+        removePages.run({
+          protocol_name,
+          page_number: lastPage.ownBlockNumber,
+        }, dbConn),
+      );
+    }
+
+    yield* until(dbConn.query("COMMIT"));
+    poolErrors.markHealthy();
+  } catch (err) {
+    try {
+      yield* until(dbConn.query("ROLLBACK"));
+    } catch (rollbackErr) {
+      log.remote(
+        ComponentNames.EFFECTSTREAM_SYNC,
+        "block-processing",
+        SeverityNumber.WARN,
+        (l) =>
+          l(
+            `ROLLBACK failed for coalesced run → block ${endpoint.blockNumber} (connection likely dead): ${
+              String(rollbackErr)
+            }`,
+          ),
+      );
+    }
+    throw err;
+  }
+}
+
+export type FinalizedBlockSubscription = {
+  next(): Operation<IteratorResult<ChainBlock, void>>;
+};
+
+function* readFinalizedBlock(
+  subscription: FinalizedBlockSubscription,
+): Operation<{ tag: "item"; value: ChainBlock } | { tag: "done" }> {
+  const n = yield* subscription.next();
+  if (n.done) return { tag: "done" };
+  // IteratorResult value type does not narrow on `done` under non-strict tsconfig.
+  return { tag: "item", value: n.value as ChainBlock };
+}
+
+export type EmptyBlockCoalescerOptions = {
+  enabled: boolean;
+  subscription: FinalizedBlockSubscription;
+  pool: Pool;
+  migrations: DBMigrations[] | undefined;
+  /** Coalesce only while `now - block.timestamp` exceeds this. Undefined disables lag gating. */
+  lagThresholdMs: number | undefined;
+  getPreviousBlockHash: () => EffectstreamBlockHash | null;
+  onFlush: (endpoint: ChainBlock, length: number) => void;
+  idleFlushMs?: number;
+};
+
+export type EmptyBlockCoalescer = {
+  advance(): Operation<ChainBlock | undefined>;
+};
+
+export function createEmptyBlockCoalescer(
+  opts: EmptyBlockCoalescerOptions,
+): EmptyBlockCoalescer {
+  const idleFlushMs = opts.idleFlushMs ?? 250;
+  let pendingRun:
+    | { endpoint: ChainBlock; boundaries: EmptyRunBoundaries; length: number }
+    | null = null;
+
+  function* flush(): Operation<void> {
+    if (pendingRun == null) return;
+    const { endpoint, length } = pendingRun;
+    yield* processCoalescedEmptyRun(
+      endpoint,
+      opts.pool,
+      opts.getPreviousBlockHash(),
+    );
+    pendingRun = null;
+    opts.onFlush(endpoint, length);
+  }
+
+  function* nextItem(): Operation<
+    { tag: "item"; value: ChainBlock } | { tag: "done" } | { tag: "idle" }
+  > {
+    if (pendingRun == null) return yield* readFinalizedBlock(opts.subscription);
+    return yield* race([
+      readFinalizedBlock(opts.subscription),
+      (function* () {
+        yield* sleep(idleFlushMs);
+        return { tag: "idle" as const };
+      })(),
+    ]);
+  }
+
+  function* advance(): Operation<ChainBlock | undefined> {
+    if (!opts.enabled) {
+      const n = yield* readFinalizedBlock(opts.subscription);
+      return n.tag === "done" ? undefined : n.value;
+    }
+    while (true) {
+      const next = yield* nextItem();
+      if (next.tag === "idle") {
+        yield* flush();
+        continue;
+      }
+      if (next.tag === "done") {
+        yield* flush();
+        return undefined;
+      }
+      const value = next.value;
+      const behindTip = opts.lagThresholdMs != null &&
+        Date.now() - value.timestamp > opts.lagThresholdMs;
+      if (behindTip && value.primitives.length === 0) {
+        const boundaries = pendingRun?.boundaries ??
+          (yield* loadEmptyRunBoundaries(opts.pool));
+        if (isCoalescableEmptyBlock(value, boundaries, opts.migrations)) {
+          pendingRun = pendingRun
+            ? { boundaries, endpoint: value, length: pendingRun.length + 1 }
+            : { boundaries, endpoint: value, length: 1 };
+          if (pendingRun.length >= MAX_EMPTY_RUN) yield* flush();
+          continue;
+        }
+      }
+      yield* flush();
+      return value;
+    }
+  }
+
+  return { advance };
+}