@effectstream/sync 0.100.17 → 0.100.19

package/CLAUDE.md

@@ -0,0 +1,206 @@
+# `@effectstream/sync` — how the sync service works
+
+This package fetches data from every configured chain, interleaves it into a single
+deterministic stream of **effectstream blocks** ordered by time, and hands that stream
+to the runtime for state-machine execution.
+
+> **Naming gotcha:** despite the project name and an `effect` (Effect-TS) entry in the
+> root `package.json`, the sync service uses **[`effection`](https://frontside.com/effection)**,
+> a generator-based structured-concurrency library. Effect-TS is **not imported anywhere**
+> in source. Everywhere below, an "effect" is an `effection` `Operation<T>` — a lazy,
+> cancellable computation written as a `function*` and composed with `yield*`.
+
+## effection in 30 seconds
+
+| effection | rough analogy |
+|---|---|
+| `Operation<T>` | a lazy, cancellable `async` function |
+| `function*` / `yield*` | `async` / `await` |
+| `spawn(op)` | fork a child task (dies with its parent scope) |
+| `call(() => promise)` / `until(promise)` | `await` a Promise inside an Operation |
+| `all([...ops])` | `Promise.all` |
+| `createChannel()` + `each(ch)` | an async stream / `for await` |
+| `conditionVariable()` (`@effectstream/utils`, `concurrency/condVar.ts`) | a monitor wait/wake |
+
+Because every loop is `spawn`ed under the runtime's `start()` scope, cancelling the parent
+tears down every fetcher, the merge, the heartbeat, and the HTTP server cleanly.
+
+## The pipeline at a glance
+
+```
+config (syncProtocols.main + syncProtocols.parallel)
+        │  genSyncProtocols()  → one SyncState per chain, main at index 0
+        ▼
+┌──────────────────────────────────────────────────────────────────────┐
+│  PER-CHAIN FETCH LOOPS              one spawn per protocol             │
+│  startSync(state):  stateToInput → readData → updateState             │
+│    → push Output to bufferedData (Deque)  + wake newData/newPageCondVar│
+│    → lastPage kept IN-MEMORY only (no DB write; runtime owns resume)   │
+│    → producerChannel.send(...)  ← NOTE: nothing consumes this today    │
+└────────────────────────────────────────┬─────────────────────────────┘
+                                          ▼
+┌──────────────────────────────────────────────────────────────────────┐
+│  MERGE LOOP   startMerge()    one spawn, produces ONE ChainBlock/iter  │
+│   i=0 main:   toRootOutput(front datum)  → new root block @ timestamp T│
+│   i>0 parallel: wait page>T, drain buffered data ≤T via mergeDatum()   │
+│   → finalizedBlockStream.send(block); then run cleanups (pop Deques)   │
+└────────────────────────────────────────┬─────────────────────────────┘
+                                          ▼
+   runtime/src/main.ts:  for (block of each(finalizedBlockStream))
+     acquire DB mutex → processFinalizedBlock (STF)
+     → upsertPage(block.resumePages) + removePages(< committed) [in txn] → COMMIT
+     → fire-and-forget MQTT events → each.next()
+```
+
+## The three per-chain abstractions (`src/sync-protocols/<chain>/`)
+
+1. **Fetcher** — `BaseDataFetcher` subclass (`base/fetcher.ts`). `readData(input)` fetches a
+   page-range of raw chain data → `DataFetched { output[], lastPage }`. Implements
+   `PaginatedFetcher` (`getLatestPage`, `nextInterval`, `previousInterval`,
+   `intervalFromStart`) for chunked pagination by `stepSize`, and optionally
+   `PrimitiveFetcher` (`base/primitive.ts`: `readPrimitives` / `groupByPage`) to extract
+   app-relevant primitives from raw blocks.
+
+2. **State** — `SyncState` subclass (`base/state.ts`). Holds `bufferedData` (a `denque`
+   Deque of fetched-not-yet-merged data, `state.ts:50`), `lastPage` (high-water mark,
+   `state.ts:70`), two CondVars (`newDataCondVar` / `newPageCondVar`, `state.ts:57-63`),
+   and error/health counters. Defines the chain↔root contract:
+   - `stateToInput()` — compute the next page-range to fetch (via `common/page-helpers.ts:genInputRange`)
+   - `toRootPage()` — map a block → millisecond timestamp (the **merge key**)
+   - `toRootOutput()` — **main only** — produce a root `ChainBlock`
+   - `mergeDatum()` — **parallel only** — fold this chain's data into the root
+   - `updateState()` (`state.ts:109`) — push outputs to the Deque, wake CondVars, and
+     persist the page (`upsertPage`, `state.ts:132`).
+
+3. **types.ts** — `Input` / `Output` / `Page` aliases (+ a `Client.ts` per chain where an
+   RPC client is needed).
+
+**main vs parallel** is structural in config — `toSyncProtocolWithNetwork`
+(`@effectstream/config`, `config/utils.ts`) builds the array **main-first**, then parallels —
+but enforced at runtime only by thrown errors: e.g. `evm/state.ts` `toRootOutput` throws
+"Only main chains create root outputs"; `ntp/state.ts` `mergeDatum` throws "Only parallel
+chains merge into root". **The main chain must be at index 0.**
+
+## The two driving loops
+
+**`startSync(state)`** (`src/sync-protocols/orchestration/sync.ts`) spawns:
+- `startAsync()` — optional background producer (websocket subs for utxorpc/midnight;
+  no-op for NTP/EVM).
+- the **polling loop**: `stateToInput()` → `readData()` → `updateState()` →
+  `producerChannel.send()` (`sync.ts:70`). Errors are swallowed via `tryYield`, bump
+  `consecutiveErrors`, log, sleep `pollingInterval`, continue. The loop only sleeps when
+  `stateToInput` returns `undefined` (caught up); while behind it fetches chunks
+  back-to-back.
+
+**`startMerge(syncProtocols, finalizedBlockStream)`** (`orchestration/merge.ts`) loops; per
+iteration it walks protocols in order:
+- **main (i=0, root not yet set):** wait for first page, peek the front buffered datum,
+  `toRootOutput` → a fresh `ChainBlock` whose `timestamp` becomes the slot boundary `T`.
+- **parallel (i>0):** block on `newPageCondVar` until *that chain's page has advanced past
+  `T`* (proving it scanned all data up to `T`), then drain every buffered datum with
+  timestamp ≤ `T` into the root via `mergeDatum`.
+- send the block downstream, **then** run cleanups that pop consumed data from each Deque.
+
+## Key design ideas
+
+1. **Time is the universal merge key.** Every chain maps blocks → ms timestamp. The main
+   chain (usually NTP, a wall-clock heartbeat at `blockTimeMS`) emits empty root blocks at a
+   fixed cadence; parallel chains' data is bucketed into those slots.
+2. **Page ≠ Data.** `lastPage` advances even on an empty query, so a parallel chain can
+   signal "scanned up to `T`, found nothing" and let the merge proceed. `newPageCondVar`
+   = progress; `newDataCondVar` = content.
+3. **Confirmation depth as a time shift.** `common/utils.ts:applyDelay` *adds* `delayMs` to
+   a parallel chain's timestamp so older blocks merge into more-recent root slots.
+4. **Mutate-after-commit.** `OutputAndCleanup` keeps data in the Deque until the block is
+   safely produced; the consumer commits to Postgres *before* emitting events.
+5. **Durable, block-accurate resume queue.** *(Single source of truth for how restart/resume
+   works — code comments point here; see also Finding #2 for the bug this fixes.)*
+
+   `effectstream.sync_protocol_pagination` is keyed `(protocol_name, page_number)` and the
+   **runtime is the sole writer** of the resume position (Fix D). End to end:
+   - **Fetch loop** (`base/state.ts:updateState`) keeps `lastPage` in memory only — it does
+     **not** persist (it runs ahead of commits, so its chunk-end page is not a safe resume
+     point).
+   - **Merge** (`orchestration/merge.ts`) tags every `ChainBlock` with `resumePages`: one
+     marker per protocol = `outputToLastPage(lastConsumed)`, the `LastPage`
+     (`{ own, ownBlockNumber, root }`) of the **highest datum merged into the block**. Each
+     protocol implements `outputToLastPage` (`base/state.ts` + per-chain `state.ts`) because
+     `own` is a structured page for object-paged chains and can't be rebuilt from the block
+     number. Pending data above the committed timestamp is **not** marked, so it is always
+     re-fetched, never skipped.
+   - **Commit** (`runtime/process-blocks.ts`, STEP 7) writes each marker **inside the block's
+     transaction**: `upsertPage(page_number = ownBlockNumber)` then
+     `removePages(page_number < ownBlockNumber)` — leaving exactly one row per protocol = the
+     committed watermark, atomic with the block.
+   - **Boot** (`restoreState`) reads `getPage` = `ORDER BY page_number ASC LIMIT 1` and
+     resumes from `nextInterval(own).from` = the next uncommitted block.
+
+   **Re-scan on restart is bounded by `stepSize`, not a chain's whole quiet tail.** Every
+   fetched chunk's boundary page (`data.to`) is buffered and consumed even when empty, so the
+   marker advances to within one `stepSize` of the committed frontier — and reaches the
+   fetched tip exactly at full catch-up. So a long-quiet chain re-fetches at most ~`stepSize`
+   blocks after a restart, regardless of how long ago its last data was. Trade-off:
+   `getSyncAndLastPage` now reports `synced_page == fetched_page`.
+
+---
+
+## Findings (open issues, as of this investigation)
+
+Both bite hardest during **deep catch-up** — e.g. DB synced 3 days ago with a 0.25s chain
+(~1.04M blocks behind) and a 6s chain (~43k behind), NTP main at `blockTimeMS: 1000`.
+
+### 1. Unbounded buffering + head-of-line blocking (performance / OOM)
+
+The fetch loop has **no backpressure** and **no clamp** relative to the merge or other
+chains: `genInputRange` (`common/page-helpers.ts`) clamps each fetch only to the chain's own
+tip and `nextPage + stepSize`. A `grep` confirms the only reads of `bufferedData.size()` are
+the merge's first-datum wait (`merge.ts:110`) and the heartbeat log (`runtime/main.ts:106`) —
+nothing pauses fetching when the Deque grows.
+
+Consequences during catch-up:
+- Each chain races to its own tip, filling its in-memory Deque toward the **entire backlog**
+  (~1.04M block objects for the 0.25s chain; NTP fills ~259k near-instantly since it's pure
+  arithmetic). Drain happens only at serial-replay speed (one DB txn + STF per block in
+  `processFinalizedBlock`), which is far slower than fetch → peak memory ≈ whole backlog →
+  **OOM risk**. No cap exists.
+- The merge is gated per timestamp `T` on the slowest chain's *page* to advance
+  (`mergeIntoRoot`, `merge.ts`), so a slow/stalled chain stalls all block production while
+  the others keep ballooning (head-of-line blocking).
+
+→ Fix: **(C) backpressure** — cap `bufferedData` and pause the fetch loop until the merge drains.
+
+### 2. Silent data gap on restart (correctness) — ✅ FIXED (D)
+
+**The bug (historical).** The page queue was **chunk-granular** but commits are
+**single-block-granular**: the fetch loop persisted `upsertPage(page_number = chunk_end)`,
+commit deleted `removePages(page_number < merged_block_N)`, and restart resumed from
+`getPage` = oldest remaining chunk at `nextInterval(own).from = chunk_end + 1`. During
+catch-up chunks are full `stepSize`, so the committed watermark `N` sat **inside** the
+oldest retained chunk `[start, end]` (`end ≥ N ⇒ not deleted`); resume jumped to `end + 1`,
+**silently skipping `(N, end]`** → missing primitives → divergent app state, with no error.
+
+**The fix (D) — block-accurate resume, runtime is the sole writer.** The runtime persists a
+block-accurate marker per protocol on commit instead of the fetch loop persisting chunk-end
+pages. Full mechanism in **design idea #5** above. Regression test: the
+`consistency.test.ts` "mid-chunk restart" case (Postgres-only).
+
+### 3. Serial replay is the catch-up bottleneck
+
+Block cadence is the main chain's granularity (1/s for NTP@1000), so 3 days ≈ 259,200
+effectstream blocks, each a separate DB transaction + STF run. Most are empty during
+catch-up.
+
+→ Fix: **(E) faster replay** (conservative, flag-gated) — coalesce consecutive empty
+catch-up blocks into fewer transactions while preserving steady-state semantics and the
+block-hash chain.
+
+> This PR adds the synthetic `test` chain (`src/sync-protocols/test/`) and an
+> in-process harness (`packages/node-sdk/runtime/test/reproduction/`, in the
+> runtime package because it boots the full `start()`, which sync must not depend
+> on). `sanity.test.ts` proves the chain syncs, processes a configured event, and
+> survives a restart. See `docs/ADDING-A-SYNC-PROTOCOL.md` for how to add a chain
+> (with `test` as the worked example).
+>
+> Deterministic reproductions of #1/#2/#3 and their fixes land alongside each fix:
+> **#2 → block-accurate resume is DONE** (`consistency.test.ts`). Still in follow-up
+> PRs: #1 → fetch backpressure; #3 → opt-in empty-block commit batching during catch-up.

package/README.md

@@ -46,6 +46,55 @@ const protocols = await genSyncProtocols(config);
 // protocols.parallelEvmRPC_fast.runOne()  // poll one block
 ```
 
+## Backpressure (`maxBufferedPages`)
+
+During deep catch-up a chain's fetch loop races to its tip far faster than the
+merge can drain (the merge applies one block per DB transaction). Without a bound
+the in-memory buffer (`SyncState.bufferedData`) grows toward the **entire backlog**
+— hundreds of thousands of block objects — which is an OOM risk.
+
+**The cap.** Every chain's `stateToInput` calls `bufferAtCap(state, syncProtocol)`
+first (`sync-protocols/common/page-helpers.ts`): when
+`bufferedData.size() >= maxBufferedPages` it returns `undefined`, so the chain stops
+fetching — exactly as if it had caught up to the tip — and the polling loop sleeps
+`pollingInterval` and retries. The merge keeps draining the buffer; once it drops
+below the cap the next poll resumes. Peak in-memory buffering is therefore bounded
+to **≈ `maxBufferedPages + stepSize`** per chain (one in-flight chunk can overshoot
+the cap), instead of the whole backlog.
+
+**Config.** `maxBufferedPages` is an optional field on every sync-protocol config
+(declared once on the shared `PollingSyncProtocol` schema). When unset it defaults to
+**`4 × stepSize`** (the `MAX_BUFFER_MULTIPLE` constant in `page-helpers.ts`), always
+clamped to **`≥ stepSize + 1`** so a chain can always fetch at least one chunk to feed
+the merge. ~4 chunks of look-ahead never starves the merge while keeping memory
+bounded; raise it to trade memory for more fetch look-ahead. The cap only bites during
+catch-up — in steady state the buffer sits near zero and the cap is never reached.
+
+**Deadlock-safety.** Pausing a full chain cannot deadlock: a chain's page only
+advances via a fetch (`updateState`), and the merge drains a chain's buffered data
+*before* it waits on that chain's page — so the paused (full) chain is never the one
+the merge is blocked on without a path to resume.
+
+**Scope.** The guard runs in **every** chain's `stateToInput`, so all sync chains are
+covered: EVM, NTP, Bitcoin, Avail, Celestia, NEAR, Midnight, Cardano (UTXO-RPC), and
+the synthetic `test` chain. Two notes:
+- **Cardano (UTXO-RPC)** has no `stepSize` (it streams one block per pass), so the cap
+  falls back to a default chunk size of 1000 (⇒ default cap 4000); set `maxBufferedPages`
+  explicitly to tune it.
+- The cap bounds `SyncState.bufferedData` (the merge-facing Deque the backpressure
+  feature protects). The UTXO-RPC fetcher additionally keeps its own internal FollowTip stream
+  buffer; pausing `stateToInput` stops draining it into `bufferedData`, but bounding
+  that lower-level stream is a separate, fetcher-specific concern.
+
+**Observability.** Each `SyncState` tracks, and the runtime's `/debug/metrics`
+endpoint reports per protocol: `cap` (resolved `maxBufferedPages`), `buf` (current
+size), `bufHighWater` (peak since boot — catches spikes between samples), `pausedNow`,
+`pauses` (rising-edge count — **the "backpressure engaged" signal**), and `pausedMs`
+(total time paused). `pauses > 0` means the cap actively bounded memory; `0` means it
+was never needed in that run. Steady-state these sit at `0`; during a real deep
+catch-up (or under the perf harness's `PERF_APPLY_DELAY_MS` drain throttle) they climb
+as `buf` pins to `cap`.
+
 ## Key exports
 
 - `genSyncProtocols(dbConn, syncInfo)` - Effection generator that instantiates a runtime fetcher + state pair for every protocol in `syncInfo` (from `config.syncProtocols`). Called from the runtime's process-blocks loop.

package/docs/ADDING-A-SYNC-PROTOCOL.md

@@ -0,0 +1,164 @@
+# Adding a new chain / sync protocol
+
+This is the end-to-end checklist for registering a new chain, distilled from
+adding the synthetic **`test`** chain (its files are the worked example — search
+for `TEST_MAIN` / `TEST_PARALLEL` / `ConfigNetworkType.TEST`). Read
+`packages/node-sdk/sync/CLAUDE.md` first for how the sync service works.
+
+A chain needs a **network type** (`ConfigNetworkType`) and one or more
+**sync-protocol types** (`ConfigSyncProtocolType`). A network may have several
+sync protocols (e.g. Cardano has CARP + UTXORPC; the `test` chain has a `main`
+clock variant and a `parallel` variant). Each sync protocol is either **main**
+(produces root blocks via `toRootOutput`) or **parallel** (folds data into the
+root via `mergeDatum`).
+
+> TypeScript is your friend here: several registries are
+> `satisfies Record<ConfigSyncProtocolType, …>`, so the compiler forces you to
+> add entries. But note this repo does **not** typecheck cleanly under
+> `bunx tsc` (it runs via bun's loaders); validate by running
+> `bun test ./packages` and a config that uses the new chain.
+
+## Pick an archetype to copy
+
+Start from the closest existing chain rather than from scratch:
+
+| Your chain | Copy | Notes |
+|---|---|---|
+| Synthetic / arithmetic clock (no RPC) | `ntp/`, `test/` | blocks computed from `startTime`+`blockTimeMS`; main role |
+| RPC, polled, emits primitives (parallel) | `evm/` | `getLatestPage` from RPC; `readPrimitives` filters logs; needs a viem client |
+| Bitcoin-style RPC | `bitcoin/` | custom `Client.ts` (`BitcoinRpcClient`), small `stepSize` (25) |
+| Streaming / subscription (push) | `utxorpc/`, `midnight/` | override **`startAsync`**; a `Client.ts` + a multiplexer/buffer feed the channel |
+| Substrate-based | `avail/`, `midnight/` | substrate network schema + genesis hash |
+| NEAR-style | `near/` | `Client.ts`, event-standard filters |
+
+`stepSize` (chunk size) defaults vary by chain: 1000 (EVM/NTP/Avail), 25
+(Bitcoin), 10 (Midnight/Celestia/NEAR) — pick what the RPC can serve per call.
+
+## 1. Config (`@effectstream/config`, `src/schema/`)
+
+- **`network/types.ts`** — add to the `ConfigNetworkType` enum.
+- **`network/<chain>.ts`** (new) — a `new ConfigSchema({ required, optional })`
+  describing the network (mirror `network/ntp.ts`). Optional fields **must** have
+  a `default`.
+- **`network/all.ts`** — import the schema and add it to `networkTypes`. (This is
+  all that's needed for the network to appear in `NetworkConfig` — it's derived
+  from `networkTypes`.)
+- **`network/utils.ts`** — add a `case` to `caip2PrefixFor` (the trailing
+  `throw` means an un-handled type fails at runtime).
+- **`sync-protocols/types.ts`** —
+  - add to the `ConfigSyncProtocolType` enum;
+  - add to `SyncProtocolToNetwork` (**exhaustive** `satisfies Record`);
+  - add a `…Primitive` type and an entry in `ProtocolPrimitiveMap`.
+- **`sync-protocols/<chain>/rpc.ts`** (new) — `ConfigSchema`s for each variant +
+  a `CommonResponse…` per variant (mirror `ntp/rpc.ts` for main,
+  `evm/rpc.ts` for parallel). Shared fields come from `NameField`,
+  `PollingSyncProtocol`, `StartStopBlockheight` via `.cloneMerge(...)`.
+- **`sync-protocols/all.ts`** — register each variant in `mainSyncProtocolTypes`
+  / `parallelSyncProtocolTypes`, and its `CommonResponse` in
+  `syncProtocolCommonResponse` (**exhaustive** `satisfies Record`).
+- **`primitive/types.ts`** — add a payload type + entry in `ProtocolPayloadMap`
+  (used by `FlattenSyncProtocolIOFor`).
+
+`mod.ts` re-exports are optional for the rpc files (NTP/NEAR/test rpc are *not*
+re-exported; `all.ts` imports them directly). `CommonResponse` has **no
+node-sdk runtime consumer** — it's config-side type derivation — so you have
+latitude on its exact shape.
+
+## 2. Sync (`@effectstream/sync`, `src/`)
+
+- **`sync-protocols/<chain>/types.ts`** — `Page`, `Input` (`PageSyncRange<Page>`),
+  `Output` (must include `primitives: PrimitiveType[]` if the chain emits any),
+  `PrimitiveType = FlattenSyncProtocolIOFor<…>`, `toMsTimestamp`.
+- **`sync-protocols/<chain>/fetcher.ts`** — extend `BaseDataFetcher`; implement
+  `readData` (+ `PaginatedFetcher`: `getLatestPage`/`nextInterval`/
+  `previousInterval`/`intervalFromStart`, and `PrimitiveFetcher` if it emits
+  primitives). `lastPage.own`/`ownBlockNumber` = the **chunk end** (`data.to`).
+  Page-request helpers live in `base/page.ts`
+  (`genImmediatePageRequests`/`genOnDemandPageRequests`).
+- **`sync-protocols/<chain>/Client.ts`** (new, only if it talks to a node) — the
+  RPC/stream client (e.g. `bitcoin/fetcher.ts:BitcoinRpcClient`,
+  `near/NearClient.ts`). EVM-family chains instead use a viem client built with
+  `createViemPublicClient` + `getViemNetwork` (declare the network via the
+  builder's `addViemNetwork`).
+- **`sync-protocols/<chain>/state.ts`** — extend `SyncState`; implement
+  `stateToInput` (via `common/page-helpers.ts:genInputRange`), `toRootPage`
+  (→ ms timestamp; use `applyDelay` for confirmation depth), `toRootOutput`
+  (main), `mergeDatum` (parallel), `getNamespace`, and a static `restoreState`
+  that reads `getPage`. For **streaming** chains, override **`startAsync()`** to
+  run the subscription/producer in the background (it feeds `bufferedData`);
+  polled chains leave it as the no-op default. (A `maxBufferedData` backpressure
+  cap on the constructor is a planned follow-up — see CLAUDE.md finding #1.)
+- **`syncProtocolFactory.ts`** — add an `else if (entry.networkType === …)`
+  branch constructing the client + fetcher + `restoreState`.
+- **`sync-protocols/types.ts`** — add the state class to the `AllSyncProtocols`
+  union.
+- **`sync-protocols/mod.ts`** — export the new fetcher/state.
+
+## 3. Runtime (`@effectstream/runtime`, `src/`)
+
+- **`api/http-server.ts`** — add a `case` to the `switch (networkType)` in the
+  blocks endpoint (import the fetcher).
+- **`config-snapshot.ts`** — `extractImmutableConfig` **throws** for un-handled
+  protocol types. Add the chain to the right branch (NTP/TEST share
+  `startTime`+`blockTimeMS`; most parallel chains use `startBlockHeight`).
+- **`main.ts` clock detection** (only if your chain can be the **main** clock) —
+  the lag threshold is derived from the main clock's `blockTimeMS` via a
+  `syncInfo.find(...)` that today keys on `NTP_MAIN`. If your chain is a new main
+  clock and you want lag logging (and the planned catch-up batching) to engage,
+  add its `*_MAIN` type to that predicate.
+
+## 4. Primitives (only if the chain emits data)
+
+A fetched primitive's `primitive` field must equal a **registered primitive's
+instanceName**, which is the config primitive's `id` (= its `name`). The fetcher
+typically tags emitted primitives with `this.config.primitives[i].id`. The
+primitive class (built-in in `@effectstream/sm` or supplied via
+`StartConfig.userDefinedPrimitives`) turns raw data into `primitive_accounting`
+rows and optional scheduled STF inputs.
+
+## Gotchas / learnings
+
+- **One network → many sync protocols** is allowed (Cardano, `test`). `FlipObject`
+  unions them, so `addMain`/`addParallel` narrow correctly.
+- **Resume is currently chunk-granular** (a known issue — CLAUDE.md finding #2):
+  on restart a chain resumes from the *end* of its oldest retained page chunk,
+  which can silently skip an un-committed tail. A block-accurate-resume fix is
+  planned; until then, avoid relying on exact restart positions mid-chunk.
+- **Merge boundary gate**: a parallel chain only merges into a root block at
+  timestamp `T` once its own page is *strictly* past `T`. If the main clock runs
+  to the same height as a parallel chain's fetched tip, the merge stalls at that
+  boundary — keep the main clock at/behind the parallel tips.
+- **`finalizedBlockStream` is unbuffered**: a message sent with no active
+  subscriber is dropped. The merge is spawned before the consumer loop
+  subscribes, so a very fast producer could in principle drop early blocks
+  (RPC-paced chains don't hit this); a subscribe-before-spawn hardening is a
+  planned follow-up.
+- **`isPresync`** is currently ~always false (`genInputRange` TODO) — the
+  historical-presync path may not trigger as intended.
+
+## Verify
+
+Build a `ConfigBuilder` config that declares the chain (see
+`packages/node-sdk/runtime/test/reproduction/sanity.test.ts` for the `test`-chain
+config), then `toSyncProtocolWithNetwork(config)` + `start({...})`, or run
+`bun test packages/node-sdk/runtime/test/reproduction/`.
+
+## Testing a new chain (in-process harness)
+
+`packages/node-sdk/runtime/test/reproduction/harness.ts` is the reusable
+template — it boots the real `start()` in-process against an in-process PGLite
+server and the synthetic `test` chain. Reuse its patterns for any chain:
+
+- **Env is lazy** (`ENV.*` reads `process.env` at access), so set
+  `PGLITE`/`DB_*`/`EFFECTSTREAM_API_PORT` per run before `start()`. With multiple
+  harnesses in one file, re-assert DB env in `runNode` (shared `process.env`).
+- **Determinism**: drive the synthetic chain's tip via the in-memory
+  `TestChainControl` registry (no real clock); poll with
+  `waitForHeight`/`waitUntilStable` (fixed tips → a well-defined stable state).
+- **Observe live state** via the test-only `StartConfig.dev.onStarted` hook
+  (exposes the `AllSyncProtocols[]`, e.g. for `bufferedData.size()` assertions)
+  and assert effects against DB tables (`primitive_accounting`,
+  `rollup_input_result`, `effectstream_blocks`, `sync_protocol_pagination`).
+- **Simulate a restart** by halting the node task while keeping PGLite alive
+  (state persists), then booting again.
+- Set `MQTT_BROKER=false` (it binds fixed ports that aren't freed on halt).

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@effectstream/sync",
-  "version": "0.100.17",
+  "version": "0.100.19",
   "license": "MIT",
   "homepage": "https://effectstream.github.io/docs/",
   "repository": {
@@ -13,10 +13,10 @@
   },
   "exports": "./src/mod.ts",
   "dependencies": {
-    "@effectstream/config": "0.100.17",
-    "@effectstream/db": "0.100.17",
-    "@effectstream/log": "0.100.17",
-    "@effectstream/utils": "0.100.17",
+    "@effectstream/config": "0.100.19",
+    "@effectstream/db": "0.100.19",
+    "@effectstream/log": "0.100.19",
+    "@effectstream/utils": "0.100.19",
     "@midnight-ntwrk/ledger-v8": "8.0.3",
     "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "4.0.2",
     "@midnight-ntwrk/onchain-runtime": "npm:@midnight-ntwrk/onchain-runtime-v3@3.0.0",

package/src/sync-protocols/avail/state.ts

@@ -10,6 +10,7 @@ import type { ConfigNetworkType, SyncProtocolWithNetwork } from "@effectstream/c
 import { getPage } from "@effectstream/db";
 import { AvailClient } from "./AvailClient.ts";
 import { applyDelay } from "../common/utils.ts";
+import { bufferAtCap } from "../common/page-helpers.ts";
 
 export class AvailSyncState extends SyncState<
   Input,
@@ -63,6 +64,7 @@ export class AvailSyncState extends SyncState<
 
   @bound
   override *stateToInput(): Operation<Input | undefined> {
+    if (bufferAtCap(this, this.config.syncProtocol)) return undefined;
     const latestHeight = yield* call(() => this.client.getLatestBlockHeight());
     const startHeight = this.lastPage?.own.height ??
       this.config.syncProtocol.startBlockHeight - 1;
@@ -99,6 +101,15 @@ export class AvailSyncState extends SyncState<
     rootOutput.primitives.push(...primitives);
   }
 
+  @bound
+  override outputToLastPage(data: Output): LastPage<Page, RootPage> {
+    return {
+      own: { height: data.raw.number, hash: data.raw.hash },
+      ownBlockNumber: data.raw.number,
+      root: this.toRootPage(data),
+    };
+  }
+
   @bound
   override getNamespace(): string[] {
     return [this.config.network.name, this.config.syncProtocol.name];

package/src/sync-protocols/base/state.ts

@@ -1,4 +1,4 @@
-import { type Operation, until } from "effection";
+import { type Operation } from "effection";
 import Deque from "denque";
 import {
   type BlockNumber,
@@ -11,7 +11,6 @@ import { ComponentNames, log, SeverityNumber } from "@effectstream/log";
 import type { BaseDataFetcher, DataFetched } from "./fetcher.ts";
 import type { PageRelation } from "./page.ts";
 import type { PoolClient } from "pg";
-import { acquireDBMutex, releaseDBMutex, upsertPage } from "@effectstream/db";
 
 export type LastPage<Page, RootPage> = {
   own: Page;
@@ -75,6 +74,20 @@ export abstract class SyncState<
   /** Timestamp of the most recent error, or 0 if no error has occurred. */
   public lastErrorTimestamp: number = 0;
 
+  // ── Backpressure observability (see common/page-helpers.ts + README) ──
+  /** Resolved fetch cap (`maxBufferedPages`); set on each backpressure check, 0 until the first. */
+  public bufferCap: number = 0;
+  /** Peak `bufferedData.size()` seen since boot — catches spikes the periodic sampler misses. */
+  public bufferHighWater: number = 0;
+  /** Whether fetching is currently paused because the buffer is at/over the cap. */
+  public pausedNow: boolean = false;
+  /** How many times the cap engaged (rising edges) — the "backpressure fired" counter. */
+  public backpressurePauses: number = 0;
+  /** Total time fetching has been paused by the cap, in ms. */
+  public backpressurePausedMs: number = 0;
+  /** Start of the current pause, for accumulating `backpressurePausedMs`. */
+  private pauseStartMs: number = 0;
+
   constructor(
     public readonly name: string,
     lastPage: undefined | LastPage<Page, RootPage>,
@@ -106,6 +119,24 @@ export abstract class SyncState<
    */
   abstract mergeDatum(ourOutput: Output, rootOutput: RootOutput): void;
 
+  recordBackpressure(atCap: boolean, cap: number): void {
+    this.bufferCap = cap;
+    const now = Date.now();
+    if (atCap && !this.pausedNow) {
+      this.pausedNow = true;
+      this.pauseStartMs = now;
+      this.backpressurePauses += 1;
+    } else if (!atCap && this.pausedNow) {
+      this.pausedNow = false;
+      this.backpressurePausedMs += now - this.pauseStartMs;
+    }
+  }
+  /**
+   * Build the resume marker for a single output: the {@link LastPage} to resume
+   * fetching *after* it. See CLAUDE.md design idea #5 (restart/resume).
+   */
+  abstract outputToLastPage(data: Output): LastPage<Page, RootPage>;
+
   @bound
   *updateState(
     _input: Input,
@@ -115,6 +146,8 @@ export abstract class SyncState<
       for (const datum of data.output) {
         this.bufferedData.push(datum);
       }
+      // Peak right after a push — the local max before the merge drains it.
+      this.bufferHighWater = Math.max(this.bufferHighWater, this.bufferedData.size());
       this.newDataCondVar.wake();
     }
 
@@ -129,13 +162,8 @@ export abstract class SyncState<
           ),
       );
 
-      yield* acquireDBMutex(`update-state-${this.name}`);
-      yield* until(upsertPage.run({
-        protocol_name: this.name,
-        page_number: data.lastPage.ownBlockNumber,
-        page: JSON.stringify(data.lastPage),
-      }, this.dbConn as any)); // Client,
-      releaseDBMutex(`update-state-${this.name}`);
+      // lastPage is in-memory only; the runtime is the sole writer of the
+      // persisted resume marker (see CLAUDE.md design idea #5).
       this.lastPage = data.lastPage;
       this.newPageCondVar.wake();
     }

package/src/sync-protocols/bitcoin/state.ts

@@ -16,7 +16,7 @@ import {
   type Output,
   type Page,
 } from "./types.ts";
-import { genInputRange } from "../common/page-helpers.ts";
+import { bufferAtCap, genInputRange } from "../common/page-helpers.ts";
 import type {
   ConfigNetworkType,
   SyncProtocolWithNetwork,
@@ -58,6 +58,7 @@ export class BitcoinSyncState extends SyncState<
         block_number: Number(data.raw.height),
         blockHash: hash,
       })),
+      resumePages: [],
       primitives: data.primitives.map((primitive) => ({
         ...primitive,
         source: this.config.syncProtocol.name,
@@ -75,6 +76,7 @@ export class BitcoinSyncState extends SyncState<
 
   @bound
   override *stateToInput(): Operation<Input | undefined> {
+    if (bufferAtCap(this, this.config.syncProtocol)) return undefined;
     return yield* genInputRange(
       this as BitcoinSyncState,
       this.config.syncProtocol.startBlockHeight as Page,
@@ -112,6 +114,16 @@ export class BitcoinSyncState extends SyncState<
     rootOutput.blockInfo.push(...blockInfo);
   }
 
+  @bound
+  override outputToLastPage(data: Output): LastPage<Page, RootPage> {
+    const block = Number(data.raw.height);
+    return {
+      own: block,
+      ownBlockNumber: block,
+      root: this.toRootPage(data),
+    };
+  }
+
   @bound
   override getNamespace(): string[] {
     return [this.config.network.name, this.config.syncProtocol.name];

package/src/sync-protocols/celestia/state.ts

@@ -10,6 +10,7 @@ import type { ConfigNetworkType, SyncProtocolWithNetwork } from "@effectstream/c
 import { getPage } from "@effectstream/db";
 import { CelestiaClient } from "./CelestiaClient.ts";
 import { applyDelay } from "../common/utils.ts";
+import { bufferAtCap } from "../common/page-helpers.ts";
 
 export class CelestiaSyncState extends SyncState<
   Input,
@@ -61,6 +62,7 @@ export class CelestiaSyncState extends SyncState<
 
   @bound
   override *stateToInput(): Operation<Input | undefined> {
+    if (bufferAtCap(this, this.config.syncProtocol)) return undefined;
     const latestHeight = yield* call(() =>
       this.client.getLatestBlockHeight()
     );
@@ -99,6 +101,15 @@ export class CelestiaSyncState extends SyncState<
     rootOutput.primitives.push(...primitives);
   }
 
+  @bound
+  override outputToLastPage(data: Output): LastPage<Page, RootPage> {
+    return {
+      own: { height: data.height, hash: data.hash },
+      ownBlockNumber: data.height,
+      root: this.toRootPage(data),
+    };
+  }
+
   @bound
   override getNamespace(): string[] {
     return [this.config.network.name, this.config.syncProtocol.name];