@effectstream/runtime 0.100.18 → 0.100.19

package/conciseinputhash-determinism.md

@@ -0,0 +1,114 @@
+# Bug Report: non-deterministic `effectstream_tx_hash` (`conciseInputHash`)
+
+**Package:** `@effectstream/node-sdk` — runtime
+**Location:** `runtime/src/process-blocks.ts` (scheduled-input loop, STEP 5)
+**Severity:** Medium — silent cross-node state divergence in a derived column; not (currently) consensus-relevant
+**Status:** Open
+
+---
+
+## Summary
+
+When the runtime records the result of a **scheduled** STF input, it fabricates the
+`effectstream_tx_hash` with `Math.random()`. The value is therefore different on every run
+and on every node, so two nodes that replay the exact same source blocks produce **different
+`rollup_input_result.effectstream_tx_hash` values** for the same input. For a framework whose
+core promise is "the database is a pure function of the source blocks," this is a determinism
+hole.
+
+## The code
+
+`runtime/src/process-blocks.ts`, inside the loop over `scheduledData` (STEP 5):
+
+```ts
+const conciseInputHash = `0x${
+  Array(64).fill(0).map(() => Math.floor(Math.random() * 16).toString(16)).join("")
+}`;
+yield* until(
+  insertGameInputResult.run({
+    id: data.id,
+    success,
+    effectstream_tx_hash: Buffer.from(conciseInputHash),
+    index_in_block,
+    block_height: value.blockNumber,
+  }, dbConn),
+);
+```
+
+This runs only for **scheduled** inputs (engine-queued timers / derived inputs that have no
+real on-chain transaction), so the engine has to synthesize a hash. It does so randomly.
+
+## Why it matters / why it's currently masked
+
+- **Block hash is unaffected.** `generateEffectstreamBlockHash` (`@effectstream/crypto`) hashes
+  `blockInfo[].blockHash` (source-chain hashes) + the previous block hash — never
+  `effectstream_tx_hash`. So the block-hash chain stays deterministic; the divergence is
+  confined to the `rollup_input_result.effectstream_tx_hash` column and anything that reads it.
+- **Masked in tests.** The reproduction tests use a no-op STF (`stateMachinePayload: null`),
+  which schedules no STF inputs, so this branch never executes. That is exactly why
+  `consistency-snapshot.ts` lists `effectstream_tx_hash` in `VOLATILE_COLUMNS` and excludes it
+  — the snapshot would otherwise diff on every run once real STFs are involved.
+
+## Two existing quirks to fix alongside
+
+1. **Encoding bug.** `Buffer.from(conciseInputHash)` is given the *string* `"0x" + 64 hex`
+   with no encoding, so it stores 66 **UTF-8** bytes, not the 32 decoded bytes. Whatever
+   derivation replaces `Math.random()`, encode it deliberately (`Buffer.from(hex, "hex")`, or
+   commit to the `0x`-string form consistently).
+
+2. **Ordering prerequisite (the real root).** The hash inputs that give uniqueness
+   (`index_in_block`, and the STF execution order itself) are only deterministic if the
+   `scheduledData` order is deterministic. Today it is not fully:
+   ```ts
+   // TODO What should be the order of the scheduled data - per id?
+   const scheduledData = [...scheduledData1, ...scheduledData2];
+   ```
+   - `scheduledData1` (block-scheduled) is `ORDER BY id ASC` → deterministic given a
+     deterministic scheduling history.
+   - `scheduledData2` (timestamp-scheduled) is `ORDER BY future_ms_timestamp ASC` **only** —
+     **same-timestamp rows have no tiebreak**, so their order is arbitrary.
+
+   This is bigger than the hash: the same order drives **STF execution** and the
+   **`randomGenerator` (Prando) draw sequence**. If it isn't pinned, game state itself can
+   diverge; the random tx-hash is just the most visible symptom.
+
+## Suggested fixes (in order)
+
+### 1. Pin the scheduled-input order (prerequisite, do first)
+Give the queries a total deterministic sort and a deterministic merge:
+- timestamp list → `ORDER BY future_ms_timestamp ASC, id ASC` (break ties by `id`);
+- keep block-scheduled before timestamp-scheduled (or merge both by a single deterministic
+  key). This is required for deterministic STFs regardless of the hash, and it makes
+  `index_in_block` deterministic.
+
+### 2. Derive the hash deterministically (recommended)
+Replace `Math.random()` with a content-addressed hash over the input's identity + position,
+reusing the `sha512` already used for the block hash:
+
+```
+effectstream_tx_hash =
+  sha512(block_height ‖ index_in_block ‖ input_data ‖ from_address ‖ from_address_type)
+```
+
+- `block_height` + `index_in_block` give **uniqueness** (one result per position; needs fix #1).
+- `input_data` + `from_address` + `from_address_type` make it **content-addressed**
+  (tx-hash-like, collision-resistant). All are non-null columns already on the scheduled row.
+
+### Alternatives considered
+- **Reuse the seeded `Prando` (`new Prando(blockHash)`):** smallest change and deterministic,
+  but the shared `randomGenerator` is also consumed by the STFs — drawing hash bytes from it
+  would shift the RNG stream every later STF sees. Would require a *separate*
+  `new Prando(blockHash + ":txhash")`. Produces opaque (non-content) hashes.
+- **`H(data.id, block_height)`:** `id` is a global SERIAL counter — a weaker determinism
+  guarantee than the block-scoped `index_in_block`, and `block_height` adds nothing once `id`
+  is unique. Prefer `index_in_block`.
+- **Provenance: `H(origin_tx_hash, …)`:** ties to the originating real tx, but
+  `origin_tx_hash` is `null` for timestamp-scheduled inputs, so it needs a fallback.
+
+## Verification
+- Add a determinism test with a **real** STF (one that schedules a derived input) and assert
+  two from-scratch syncs produce identical `rollup_input_result.effectstream_tx_hash` —
+  i.e. drop `effectstream_tx_hash` from `VOLATILE_COLUMNS` once fixed and confirm
+  `consistency.test.ts` Test A still passes.
+- Cover the timestamp-tie case explicitly: schedule ≥2 inputs at the same `future_ms_timestamp`
+  and assert identical ordering / hashes across runs.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@effectstream/runtime",
-  "version": "0.100.18",
+  "version": "0.100.19",
   "license": "MIT",
   "homepage": "https://effectstream.github.io/docs/",
   "repository": {
@@ -13,17 +13,17 @@
   },
   "exports": "./src/mod.ts",
   "dependencies": {
-    "@effectstream/concise": "0.100.18",
-    "@effectstream/config": "0.100.18",
-    "@effectstream/coroutine": "0.100.18",
-    "@effectstream/crypto": "0.100.18",
-    "@effectstream/db": "0.100.18",
-    "@effectstream/event-client": "0.100.18",
-    "@effectstream/event-server": "0.100.18",
-    "@effectstream/log": "0.100.18",
-    "@effectstream/sm": "0.100.18",
-    "@effectstream/sync": "0.100.18",
-    "@effectstream/utils": "0.100.18",
+    "@effectstream/concise": "0.100.19",
+    "@effectstream/config": "0.100.19",
+    "@effectstream/coroutine": "0.100.19",
+    "@effectstream/crypto": "0.100.19",
+    "@effectstream/db": "0.100.19",
+    "@effectstream/event-client": "0.100.19",
+    "@effectstream/event-server": "0.100.19",
+    "@effectstream/log": "0.100.19",
+    "@effectstream/sm": "0.100.19",
+    "@effectstream/sync": "0.100.19",
+    "@effectstream/utils": "0.100.19",
     "@fastify/cors": "^11.0.1",
     "@metamask/json-rpc-engine": "^10.0.3",
     "pg": "8.17.2",

package/src/api/apply-status.ts

@@ -0,0 +1,30 @@
+/**
+ * Apply-stage liveness for /debug/metrics.
+ *
+ * The serial finalized-block loop in main.ts calls recordAppliedBlock() right
+ * after each block COMMITs; the /debug/metrics handler reads `appliedBlockStatus`
+ * to report true lag (now − applied block timestamp). Unlike a protocol's `buf`
+ * (fetch backlog), this stays high when the node is write/apply-bound.
+ *
+ * Process-global: one node runs per process, so a module-level singleton keeps
+ * main.ts to a single call instead of threading a ref through startHttpServer.
+ */
+export type AppliedBlockStatus = {
+  blockNumber: number | null;
+  timestamp: number | null;
+  appliedAtMs: number | null;
+};
+
+export const appliedBlockStatus: AppliedBlockStatus = {
+  blockNumber: null,
+  timestamp: null,
+  appliedAtMs: null,
+};
+
+export function recordAppliedBlock(
+  block: { blockNumber: number | bigint; timestamp: number | bigint },
+): void {
+  appliedBlockStatus.blockNumber = Number(block.blockNumber);
+  appliedBlockStatus.timestamp = Number(block.timestamp);
+  appliedBlockStatus.appliedAtMs = Date.now();
+}

package/src/api/http-server.ts

@@ -1,5 +1,6 @@
 import fastify, { type FastifyInstance, type FastifyReply, type FastifyRequest } from "fastify";
 import { evmRpcEngine } from "./rpc-evm/eip1193.ts";
+import { appliedBlockStatus } from "./apply-status.ts";
 import type { Pool } from "pg";
 import cors from "@fastify/cors";
 import { run, until } from "effection";
@@ -13,6 +14,7 @@ import {
   getTableSchema,
   type IGetAllAddressesResult,
   type IGetAllTableNamesResult,
+  poolErrors,
   releaseDBMutex,
   runPreparedQuery,
   waitUntilFree,
@@ -25,6 +27,7 @@ import type {
   EvmFetcher,
   MidnightFetcher,
   NtpFetcher,
+  TestFetcher,
   UtxoRpcFetcher,
 } from "@effectstream/sync";
 import fastifySwagger, {
@@ -297,6 +300,24 @@ export const startHttpServer = function* (
               }
             }
             break;
+          case ConfigNetworkType.TEST:
+            {
+              const testFetcher = fetcher as TestFetcher;
+              const cfg = testFetcher.config.network;
+              if (!cfg?.startTime || !cfg?.blockTimeMS) {
+                return reply.status(500).send({ error: "TEST config missing" });
+              }
+              for (let n = from; n <= to; n++) {
+                const timestamp = BigInt(cfg.startTime) +
+                  BigInt(cfg.blockTimeMS) * BigInt(n);
+                blocks.push({
+                  blockNumber: n,
+                  timestamp,
+                  hash: `0x${timestamp.toString(16)}`,
+                });
+              }
+            }
+            break;
           case ConfigNetworkType.CARDANO: {
             const cardanoFetcher = fetcher as UtxoRpcFetcher;
             const res = cardanoFetcher.client.fetchBlocks(from, to);
@@ -345,19 +366,29 @@ export const startHttpServer = function* (
     yield* until(apiRouter(server, dbConn));
   }
 
+  const HealthDbSchema = Type.Object({
+    consecutive: Type.Number(),
+    firstFailureAt: Type.Number(),
+    sustainedDurationMs: Type.Number(),
+    sustained: Type.Boolean(),
+  });
+  const HealthResponseSchema = Type.Object({
+    status: Type.String(),
+    db: HealthDbSchema,
+  });
   server.get("/health", {
     schema: {
       tags: ["status"],
       response: {
-        200: Type.Object({
-          status: Type.String(),
-        }),
+        200: HealthResponseSchema,
+        503: HealthResponseSchema,
       },
     },
-  }, () => {
-    return {
-      status: "ok",
-    };
+  }, (_req, reply) => {
+    const db = poolErrors.state();
+    return reply
+      .status(db.sustained ? 503 : 200)
+      .send({ status: db.sustained ? "db-unreachable" : "ok", db });
   });
 
   server.get("/addresses", {
@@ -450,6 +481,42 @@ export const startHttpServer = function* (
   });
 
   if (ENV.ENABLE_DEV_AND_DEBUG_ENDPOINTS) {
+    server.get("/debug/metrics", {
+      schema: {
+        tags: ["developer"],
+        response: {
+          200: Type.Object({}, { additionalProperties: true }),
+        },
+      },
+    }, () => {
+      const appliedTs = appliedBlockStatus.timestamp;
+      return {
+        timestamp: Date.now(),
+        uptimeSeconds: process.uptime(),
+        memory: process.memoryUsage(),
+        protocols: syncProtocols.map((p) => ({
+          name: p.name,
+          buf: p.bufferedData.size(),
+          ownBlockNumber: p.lastPage?.ownBlockNumber ?? null,
+          // Backpressure observability
+          cap: p.bufferCap,
+          bufHighWater: p.bufferHighWater,
+          pausedNow: p.pausedNow,
+          pauses: p.backpressurePauses,
+          pausedMs: p.backpressurePausedMs,
+        })),
+        // Apply-stage lag: how old the last-applied block is. Unlike `buf` (fetch
+        // backlog) this stays high when the node is write/apply-bound.
+        applied: {
+          blockNumber: appliedBlockStatus.blockNumber,
+          timestamp: appliedTs,
+          lagSeconds: appliedTs != null
+            ? Math.round((Date.now() - appliedTs) / 100) / 10
+            : null,
+        },
+      };
+    });
+
     server.get("/debug/sync-protocols", {
       schema: {
         tags: ["developer"],

package/src/config-snapshot.ts

@@ -23,11 +23,19 @@ import { getEnv } from "@effectstream/utils/runtime";
  *   Bitcoin, Celestia          → syncProtocol.startBlockHeight
  */
 function extractImmutableConfig(protocol: SyncProtocolWithNetwork): Record<string, unknown> {
-  if (protocol.networkType === ConfigNetworkType.NTP) {
-    const ntpNetwork = protocol.network as { startTime: number; blockTimeMS: number };
+  // NTP and the synthetic TEST chain both derive blocks arithmetically from
+  // network.startTime + network.blockTimeMS, so those are the immutable fields.
+  if (
+    protocol.networkType === ConfigNetworkType.NTP ||
+    protocol.networkType === ConfigNetworkType.TEST
+  ) {
+    const arithmeticNetwork = protocol.network as {
+      startTime: number;
+      blockTimeMS: number;
+    };
     return {
-      startTime: ntpNetwork.startTime,
-      blockTimeMS: ntpNetwork.blockTimeMS,
+      startTime: arithmeticNetwork.startTime,
+      blockTimeMS: arithmeticNetwork.blockTimeMS,
     };
   }
 

package/src/main.ts

@@ -22,15 +22,18 @@ import { startMerge, startSync } from "@effectstream/sync";
 import { ComponentNames, log, SeverityNumber } from "@effectstream/log";
 import {
   createChannel,
-  each,
   type Operation,
   sleep,
   spawn,
   until,
 } from "effection";
 import { initTelemetry } from "./telemetry.ts";
-import { type PendingEvent, processFinalizedBlock } from "./process-blocks.ts";
+import {
+  type PendingEvent,
+  processFinalizedBlockWithRetry,
+} from "./process-blocks.ts";
 import { startHttpServer } from "./api/http-server.ts";
+import { recordAppliedBlock } from "./api/apply-status.ts";
 import type { StartConfig } from "./types.ts";
 import type { Client } from "pg";
 import type { EffectstreamBlockHash } from "@effectstream/utils";
@@ -62,6 +65,9 @@ export function* start(config: StartConfig): Operation<void> {
   const syncProtocols = yield* startup(dbConn as any, // Client,
     syncInfo, config);
 
+  // Test-only: surface live sync protocols (e.g. for buffer-size assertions).
+  config.dev?.onStarted?.({ syncProtocols });
+
   log.remote(
     ComponentNames.EFFECTSTREAM_RUNTIME,
     [],
@@ -92,6 +98,14 @@ export function* start(config: StartConfig): Operation<void> {
 
   const finalizedBlockStream = createChannel<ChainBlock>();
 
+  // Subscribe BEFORE spawning the merge: effection channels drop sends with no
+  // active subscriber, so a fast restart (in-memory state, no RPC wait) could
+  // emit blocks before we subscribe and silently stall at the pre-restart height.
+  const finalizedBlocks = yield* finalizedBlockStream;
+  // NOTE: this subscriber's queue is unbounded. During deep catch-up the merge
+  // can outpace the per-block processing below and grow it without bound — the
+  // downstream side of the Fix C backpressure issue (see sync/CLAUDE.md).
+
   yield* spawn(() => startMerge(syncProtocols, finalizedBlockStream));
 
   const heartbeatIntervalMs = 60_000;
@@ -134,37 +148,26 @@ export function* start(config: StartConfig): Operation<void> {
     yield* spawn(() => runSnapshotLoop(config.snapshotConfig!));
   }
 
-  for (const value of yield* each(finalizedBlockStream)) {
-    // We request a dbClient for a non-shared dbConn object.
-    // For PGLite, this is not enough, as the can only be one connection at a time.
-    // So we request a DBMutex as well.
-    let dbClient: Client | undefined;
-    // Custom app events emitted by STFs during this block. Collected by
-    // processFinalizedBlock and flushed below ONLY after it returns — i.e.
-    // strictly after the block's COMMIT. See plan invariant I1.
-    let blockAppEvents: PendingEvent[] = [];
-    try {
-      yield* acquireDBMutex(`processing-blocks:${value.blockNumber}`);
-      dbClient = yield* until((dbConn as any).connect()); // Client,
-
-      const result = yield* processFinalizedBlock(
-        value,
-        config,
-        dbClient as any, // Client,
-        blockHash,
-      );
-      const resultHash = result.blockHash;
-      blockAppEvents = result.events;
-      if (resultHash !== "0x0") {
-        blockHash = resultHash;
-      }
-    } finally {
-      releaseDBMutex(`processing-blocks:${value.blockNumber}`);
-      if (dbClient) {
-        (dbClient as any).release(); // Client,
-      }
+  let nextBlock = yield* finalizedBlocks.next();
+  for (; !nextBlock.done; nextBlock = yield* finalizedBlocks.next()) {
+    const value = nextBlock.value;
+    // processFinalizedBlockWithRetry owns connection checkout, the per-block
+    // DB mutex (PGLite), and transient-pg retry/backoff. App events it
+    // collects are flushed below ONLY after it returns — i.e. strictly after
+    // the block's COMMIT.
+    const result = yield* processFinalizedBlockWithRetry(
+      value,
+      config,
+      dbConn as any, // Pool,
+      blockHash,
+    );
+    const blockAppEvents: PendingEvent[] = result.events;
+    if (result.blockHash !== "0x0") {
+      blockHash = result.blockHash;
     }
 
+    recordAppliedBlock(value); // apply-stage liveness for /debug/metrics
+
     // Used to emit & log the block range for each protocol.
     const contentBlocksForProtocol = getRangesForSyncProtocols(value);
 
@@ -215,7 +218,7 @@ export function* start(config: StartConfig): Operation<void> {
           }...${lagSuffix} | ${JSON.stringify(contentBlocksForProtocol)}`,
         ),
     );
-    yield* each.next();
+    if (config.dev?.applyDelayMs) yield* sleep(config.dev.applyDelayMs);
   }
 }
 

package/src/process-blocks.ts

@@ -1,6 +1,6 @@
 import type { ChainBlock } from "@effectstream/sync";
-import { call, type Operation, until } from "effection";
-import type { Pool } from "pg";
+import { call, type Operation, sleep, until } from "effection";
+import type { Pool, PoolClient } from "pg";
 import { type BaseStfInput, type EmitFn, primitiveTransitionFunction } from "@effectstream/sm";
 import { PreparedQuery } from "@pgtyped/runtime";
 import type {
@@ -9,19 +9,24 @@ import type {
   SyncStateUpdateStream,
 } from "@effectstream/coroutine";
 import {
+  acquireDBMutex,
   blockHeightDone,
   deleteEmptyBlocks,
   deleteScheduled,
   getFutureGameInputByBlockHeight,
   getFutureGameInputByMaxTimestamp,
   insertGameInputResult,
+  isTransientPgError,
+  poolErrors,
+  releaseDBMutex,
   removePages,
   saveLastBlock,
+  upsertPage,
 } from "@effectstream/db";
 import { Buffer } from "node:buffer";
 import { ComponentNames, log, SeverityNumber } from "@effectstream/log";
 import type { StartConfig } from "./types.ts";
-import type { BlockNumber, EffectstreamBlockHash } from "@effectstream/utils";
+import type { EffectstreamBlockHash } from "@effectstream/utils";
 import { generateEffectstreamBlockHash, Prando } from "@effectstream/crypto";
 import { applyUserMigrations } from "./version-migrations.ts";
 import type { EventPathAndDef } from "@effectstream/event-client";
@@ -308,36 +313,53 @@ export function* processFinalizedBlock(
       }, dbConn)
     );
 
-    /* STEP 7: Update page state for all sync protocols consumed
-    *          This is to mark what blocks where consumed by this specific
-    *          block, to avoid re-processing the same blocks more than once
-    *          if the sync protocol is restarted.
+    /* STEP 7: Persist the block-accurate resume marker per protocol, inside this
+    *          block's transaction (the runtime is the sole writer of the resume
+    *          position). Upsert at page_number = ownBlockNumber, then prune
+    *          everything below → one row per protocol = the committed watermark.
+    *          Full mechanism: @effectstream/sync CLAUDE.md, design idea #5.
     */
-
-    // Get latest block for each sync protocol.
-    const latestBlocks: Record<string, BlockNumber> = {};
-    for (const blockInfo of value.blockInfo) {
-      if (
-        !latestBlocks[blockInfo.protocol_name] ||
-        latestBlocks[blockInfo.protocol_name] < blockInfo.block_number
-      ) {
-        latestBlocks[blockInfo.protocol_name] = blockInfo.block_number;
-      }
-    }
-
-    for (const [protocolName, blockNumber] of Object.entries(latestBlocks)) {
+    for (const { protocol_name, lastPage } of value.resumePages) {
+      yield* until(
+        upsertPage.run({
+          protocol_name,
+          page_number: lastPage.ownBlockNumber,
+          page: JSON.stringify(lastPage),
+        }, dbConn),
+      );
       yield* until(
         removePages.run({
-          protocol_name: protocolName,
-          page_number: blockNumber,
+          protocol_name,
+          page_number: lastPage.ownBlockNumber,
         }, dbConn),
       );
     }
 
     /* STEP 8: Commit the transaction. */
     yield* until(dbConn.query("COMMIT"));
+    // A successful COMMIT proves the DB is reachable end-to-end — reset the
+    // pool failure clock so the write path also feeds the health tracker
+    // (it otherwise only sees floating pool errors and the read-side API).
+    poolErrors.markHealthy();
   } catch (err) {
-    yield* until(dbConn.query("ROLLBACK"));
+    // Best-effort rollback. If the socket is dead the ROLLBACK throws too;
+    // swallow that so it doesn't mask the original error — the caller's retry
+    // loop keys off the original (transient) error, not a rollback failure.
+    try {
+      yield* until(dbConn.query("ROLLBACK"));
+    } catch (rollbackErr) {
+      log.remote(
+        ComponentNames.EFFECTSTREAM_SYNC,
+        "block-processing",
+        SeverityNumber.WARN,
+        (log) =>
+          log(
+            `ROLLBACK failed for block ${value.blockNumber} (connection likely dead): ${
+              String(rollbackErr)
+            }`,
+          ),
+      );
+    }
     log.remote(
       ComponentNames.EFFECTSTREAM_SYNC,
       "block-processing",
@@ -345,12 +367,89 @@ export function* processFinalizedBlock(
       (log) =>
         log(`Error processing block ${value.blockNumber}: ${String(err)}`),
     );
-    // We cannot recover from this error.
+    // Re-throw: `processFinalizedBlockWithRetry` decides whether this is a
+    // retryable transient pg failure or a fatal bug.
     throw err;
   }
 
-  return { 
-    blockHash: hasOperations ? blockHash : "0x0" as PaimaBlockHash, 
-    events: blockEvents 
+  return {
+    blockHash: hasOperations ? blockHash : "0x0" as PaimaBlockHash,
+    events: blockEvents
   };
 }
+
+/** Caps the exponential backoff between block-retry attempts. */
+const MAX_RETRY_BACKOFF_MS = 5_000;
+
+/**
+ * Run {@link processFinalizedBlock} against a freshly checked-out pool client,
+ * retrying the whole block on transient pg failures (Neon cold-starts,
+ * idle-socket evictions, TLS blips).
+ *
+ * The previous socket is dead after such a failure, so each attempt
+ * re-acquires a client and reprocesses the block from `BEGIN` — safe because
+ * the failed tx never committed and the caller only advances the block stream
+ * after this returns.
+ *
+ * `poolErrors` governs escalation: it logs WARN→ERROR as the outage persists
+ * and calls `process.exit(1)` once it crosses the fatal threshold, letting the
+ * orchestrator restart us. A non-transient error (a real SQL/logic bug) is
+ * re-thrown and crashes, as before.
+ *
+ * Owns the per-block DB mutex (for PGLite) and client lifecycle so the caller
+ * doesn't have to.
+ */
+export function* processFinalizedBlockWithRetry(
+  value: ChainBlock,
+  config: StartConfig,
+  pool: Pool,
+  previousBlockHash: EffectstreamBlockHash | null,
+): Operation<ProcessFinalizedBlockResult> {
+  const lockName = `processing-blocks:${value.blockNumber}`;
+  let attempt = 0;
+  while (true) {
+    let dbClient: PoolClient | undefined;
+    let transientErr: unknown;
+    try {
+      yield* acquireDBMutex(lockName);
+      dbClient = yield* until(pool.connect());
+      return yield* processFinalizedBlock(
+        value,
+        config,
+        dbClient as unknown as Pool, // a checked-out client, used as the tx conn
+        previousBlockHash,
+      );
+    } catch (err) {
+      if (!isTransientPgError(err)) throw err;
+      transientErr = err;
+    } finally {
+      releaseDBMutex(lockName);
+      if (dbClient) {
+        // Passing the error destroys the client instead of recycling it, so
+        // the pool never hands the dead socket back out. On success
+        // `transientErr` is undefined and the client returns to the pool.
+        try {
+          dbClient.release(transientErr as Error | undefined);
+        } catch {
+          /* already released/destroyed */
+        }
+      }
+    }
+
+    attempt++;
+    poolErrors.log(transientErr, ["sync-write"]);
+    const backoffMs = Math.min(500 * 2 ** (attempt - 1), MAX_RETRY_BACKOFF_MS);
+    log.local(
+      ComponentNames.EFFECTSTREAM_SYNC,
+      "block-processing",
+      SeverityNumber.WARN,
+      (l) =>
+        l(
+          `transient db error on block ${value.blockNumber}; retry #${attempt} in ${backoffMs}ms: ${
+            String(transientErr)
+          }`,
+        ),
+    );
+    yield* sleep(backoffMs);
+  }
+}

package/src/types.ts

@@ -1,4 +1,5 @@
 import type { Operation } from "effection";
+import type { AllSyncProtocols } from "@effectstream/sync";
 import type { SyncProtocolWithNetwork } from "@effectstream/config";
 import type { AppEvents, BaseStfInput, BaseStfOutput, Primitive } from "@effectstream/sm";
 import type { FastifyInstance } from "fastify";
@@ -72,5 +73,18 @@ export type StartConfig = {
   dev?: {
     /** Reset public-schema tables on each sync reset. For local testing only. */
     resetPublicData?: boolean;
+    /**
+     * Test-only hook invoked once the sync protocols are instantiated (after
+     * `genSyncProtocols`, before the merge starts). Lets tests observe live
+     * state such as `bufferedData.size()`. No effect in production.
+     */
+    onStarted?: (ctx: { syncProtocols: AllSyncProtocols[] }) => void;
+    /**
+     * Test/diagnostic only: sleep this many ms after each applied block, slowing
+     * the drain so the per-chain fetch buffers fill to their cap and the
+     * backpressure fix becomes observable (otherwise drain ≈ fetch and the buffer
+     * never grows). No effect in production (leave unset / 0).
+     */
+    applyDelayMs?: number;
   };
 };