@effectstream/sync 0.101.0 → 0.102.0

package/CLAUDE.md

@@ -69,8 +69,10 @@ config (syncProtocols.main + syncProtocols.parallel)
    - `toRootPage()` — map a block → millisecond timestamp (the **merge key**)
    - `toRootOutput()` — **main only** — produce a root `ChainBlock`
    - `mergeDatum()` — **parallel only** — fold this chain's data into the root
-   - `updateState()` (`state.ts:109`) — push outputs to the Deque, wake CondVars, and
-     persist the page (`upsertPage`, `state.ts:132`).
+   - `updateState()` — push outputs to the Deque and wake the CondVars. It does **not**
+     persist anything: `lastPage` is in-memory only, because the fetch loop runs ahead of
+     commits and its chunk-end page is not a safe resume point. The runtime is the sole
+     writer of the persisted resume marker (design idea #5).
 
 3. **types.ts** — `Input` / `Output` / `Page` aliases (+ a `Client.ts` per chain where an
    RPC client is needed).
@@ -84,12 +86,16 @@ chains merge into root". **The main chain must be at index 0.**
 ## The two driving loops
 
 **`startSync(state)`** (`src/sync-protocols/orchestration/sync.ts`) spawns:
-- `startAsync()` — optional background producer (websocket subs for utxorpc/midnight;
-  no-op for NTP/EVM).
+- `startAsync()` — optional background producer (websocket/gRPC subs for utxorpc; no-op
+  for polled chains). **Supervised** when the state sets `hasAsyncProducer`: both a throw
+  and a clean return count as failure (a producer that returns has ended its stream) and
+  are restarted with capped backoff, counted in `producerRestarts`. Polled chains run
+  their no-op once.
 - the **polling loop**: `stateToInput()` → `readData()` → `updateState()` →
-  `producerChannel.send()` (`sync.ts:70`). Errors are swallowed via `tryYield`, bump
-  `consecutiveErrors`, log, sleep `pollingInterval`, continue. The loop only sleeps when
-  `stateToInput` returns `undefined` (caught up); while behind it fetches chunks
+  `producerChannel.send()`. Errors are swallowed via `tryYield`, bump
+  `consecutiveErrors`, log, sleep, continue. Every branch sleeps
+  `pollingInterval` (falling back to 1 s if a config somehow lacks one) — a pass that
+  neither fetches nor sleeps starves the whole event loop. While behind it fetches chunks
   back-to-back.
 
 **`startMerge(syncProtocols, finalizedBlockStream)`** (`orchestration/merge.ts`) loops; per
@@ -214,15 +220,73 @@ catch-up.
 catch-up blocks into fewer transactions while preserving steady-state semantics and the
 block-hash chain.
 
-> This PR adds the synthetic `test` chain (`src/sync-protocols/test/`) and an
-> in-process harness (`packages/node-sdk/runtime/test/reproduction/`, in the
-> runtime package because it boots the full `start()`, which sync must not depend
-> on). `sanity.test.ts` proves the chain syncs, processes a configured event, and
-> survives a restart. See `docs/ADDING-A-SYNC-PROTOCOL.md` for how to add a chain
-> (with `test` as the worked example).
+### 4. No RPC timeouts — a hung fetch stalled the node silently (correctness) — ✅ FIXED
+
+`fetch` has no default timeout. Bitcoin, NEAR and Avail used it bare, so a blackholed
+endpoint (dropped LB backend, half-open socket) hung `readData` forever. That is the worst
+shape of failure here: the fetch loop never reaches its `catch`, so `consecutiveErrors`
+stays 0 and `lastSuccessfulFetchMs` freezes while the merge blocks on that chain's page.
+Block production stopped and nothing reported it.
+
+→ Fix: `common/http.ts:fetchWithTimeout` (AbortSignal.timeout) on every client, configured
+per protocol by `requestTimeoutMs` on `PollingSyncProtocol` (default 15 s). Timeout-only,
+no internal retry — the fetch loop already retries the same page range idempotently, and
+retrying inside the client would hide the failure from the health endpoint. Regression
+test: `sync/test/rpc-timeout.test.ts` (each client vs a blackholed socket).
+
+### 5. An unpaced fetch loop starved the event loop (correctness) — ✅ FIXED
+
+`startSync` guarded its sleeps with `if ("pollingInterval" in config.syncProtocol)`, and
+the Cardano/utxorpc schema was the only sync protocol not merging `PollingSyncProtocol` —
+so that property was undeclared, untyped and undefaulted while the loop keyed its pacing
+off it. Cardano worked only because every config passed it anyway as an undeclared extra
+field; a config built strictly from the schema froze the node.
+
+Froze rather than span: a pass that neither fetches nor sleeps never yields to the
+macrotask queue. Measured 200k iterations in ~4 s while a 500 ms `setTimeout` never fired
+once — no HTTP server, no other chain's fetch loop, and for a streaming chain none of the
+callbacks that would let it escape.
+
+→ Fix: utxorpc merges `PollingSyncProtocol` like every other protocol, **and** the sleep is
+unconditional with a 1 s fallback. Regression test: `sync/test/poll-loop-spin.test.ts`.
+
+### 6. The streaming producer was unsupervised (correctness) — ✅ FIXED
+
+`startAsync` was spawned bare. A stream that **ended cleanly** left the chain receiving
+nothing forever with every health counter clean — a total, silent, unrecoverable stall. A
+stream that **threw** tore down the enclosing scope, i.e. `start()`, taking every unrelated
+chain with it.
+
+→ Fix: both treated as failure and restarted with capped backoff, counted in
+`producerRestarts`. Gated on `hasAsyncProducer` so polled chains still run their no-op
+once. Regression test: `sync/test/start-async-supervision.test.ts`.
+
+### 7. `/health` reported only database reachability (operability) — ✅ FIXED
+
+Every failure above leaves the database perfectly healthy, so a node that had not applied a
+block in an hour still answered `200 {"status":"ok"}`.
+
+→ Fix: `runtime/src/api/health.ts` reports per-protocol state — most of which `SyncState`
+already tracked and never exposed — keyed on wall-clock time since the last *applied*
+block rather than block-time lag (a node replaying history is legitimately far behind in
+block time while healthy). `blockingMerge` names the chain the merge is waiting on. 503 is
+reserved for `db-unreachable` / `stalled` / `starting`; a chain erroring while blocks still
+flow is `degraded` + 200. Regression test: `runtime/test/reproduction/health.test.ts`.
+
+---
+
+> The synthetic `test` chain (`src/sync-protocols/test/`) and the in-process harness
+> (`packages/node-sdk/runtime/test/reproduction/`, in the runtime package because it boots
+> the full `start()`, which sync must not depend on) back all of these. `sanity.test.ts`
+> proves the chain syncs, processes a configured event, and survives a restart. See
+> `docs/ADDING-A-SYNC-PROTOCOL.md` for how to add a chain (with `test` as the worked
+> example).
+>
+> Every finding above has a deterministic reproduction that was written to fail first.
+> Run them all, isolated from other suites, with:
+>
+> ```
+> bun run e2e/sync-repro/run-tests.ts --docker
+> ```
 >
-> Deterministic reproductions of #1/#2/#3 and their fixes land alongside each fix:
-> **#2 → block-accurate resume is DONE** (`consistency.test.ts`); **#1 → fetch
-> backpressure is DONE** (`buffering.test.ts`, incl. 1c/1d for the merge-demand
-> exemption). Still in follow-up PRs: #3 → opt-in empty-block commit batching during
-> catch-up.
+> Still open: #3 → opt-in empty-block commit batching during catch-up.

package/LICENSE-APACHE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2024-2026 Midnight Foundation
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/LICENSE-MIT

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024-2026 Midnight Foundation
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -2,12 +2,12 @@
 
 The blockchain-sync service inside an EffectStream node. Reads finalized
 blocks from every chain you've configured (EVM, Bitcoin, Cardano, Midnight,
-Avail, Celestia, NEAR…), normalizes them into a single rollup ordering,
+Avail, Celestia, NEAR, Solana…), normalizes them into a single rollup ordering,
 and stages the inputs the state machine consumes.
 
 - Blockchain-sync service: reads finalized blocks from every configured chain.
 - Normalises into a single rollup ordering and stages inputs for the state machine.
-- Drop-in fetchers: EVM, Bitcoin, Cardano UTXO-RPC, Midnight, Avail, Celestia, NEAR, NTP.
+- Drop-in fetchers: EVM, Bitcoin, Cardano UTXO-RPC, Midnight, Avail, Celestia, NEAR, Solana, NTP.
 - `genSyncProtocols(config)` is what the runtime calls during boot.
 
 ## Install
@@ -87,7 +87,7 @@ re-engages. A chain that is merely tip-limited (caught up, not gated by the merg
 `buffering.test.ts` 1c (skip-ahead) and 1d (density).
 
 **Scope.** The guard runs in **every** chain's `stateToInput`, so all sync chains are
-covered: EVM, NTP, Bitcoin, Avail, Celestia, NEAR, Midnight, Cardano (UTXO-RPC), and
+covered: EVM, NTP, Bitcoin, Avail, Celestia, NEAR, Solana, Midnight, Cardano (UTXO-RPC), and
 the synthetic `test` chain. Two notes:
 - **Cardano (UTXO-RPC)** has no `stepSize` (it streams one block per pass), so the cap
   falls back to a default chunk size of 1000 (⇒ default cap 4000); set `maxBufferedPages`
@@ -108,6 +108,65 @@ was never needed in that run. Steady-state these sit at `0`; during a real deep
 catch-up (or under the perf harness's `PERF_APPLY_DELAY_MS` drain throttle) they climb
 as `buf` pins to `cap`.
 
+## Request timeouts (`requestTimeoutMs`)
+
+`fetch` has no default timeout, so an RPC endpoint that accepts the connection and
+then never answers — a load balancer that dropped its backend, a socket left
+half-open by a NAT rebind — would hang the fetch indefinitely. That is worse than
+an error: the fetch loop never reaches its `catch`, so no error is counted, while
+the merge blocks on that chain's page and block production stops.
+
+**Config.** `requestTimeoutMs` is an optional field on every polling sync-protocol
+config, defaulting to **15 s**. It bounds a single request; retry is the fetch
+loop's job (it re-runs the same page range and counts the failure, which is what
+`/health` reports on).
+
+**Coverage.** Honoured by EVM (as the viem transport timeout), Bitcoin, NEAR,
+Avail (light-client HTTP), Midnight and Celestia. Two exceptions:
+
+- **Celestia** retries internally, so the value bounds each *attempt* rather than
+  the whole call. It still falls back to `CELESTIA_RPC_TIMEOUT_MS` when no
+  protocol-level value is set.
+- **Cardano / utxorpc** reads its data from a gRPC stream rather than
+  request/response RPCs, so the field is inert there. That path is instead
+  protected by producer supervision — a stream that dies or ends is restarted
+  with backoff (see below).
+
+NTP and the synthetic `test` chain make no network calls at all.
+
+```ts
+.addParallel((n) => n.myChain, () => ({
+  name: "myChain",
+  type: ConfigSyncProtocolType.EVM_RPC_PARALLEL,
+  pollingInterval: 1000,
+  requestTimeoutMs: 30_000, // slow archive node
+  // ...
+}))
+```
+
+## Producer supervision (streaming chains)
+
+Chains whose data arrives over a subscription rather than by polling (utxorpc
+today) set `hasAsyncProducer` on their `SyncState`, and `startSync` supervises
+that producer instead of spawning it bare.
+
+Both of its exit modes count as failure and trigger a restart with capped
+exponential backoff (1s → 30s), including a **clean return** — a producer that
+returns has ended its stream, and the observable effect is identical to a throw:
+no more data. Left unsupervised, a stream that simply ended left the chain silent
+forever with every health counter clean, and a stream that threw tore down the
+whole node.
+
+The backoff de-escalates: a producer that ran longer than the maximum backoff
+before dying resets the delay, so an occasional incident over a long-lived stream
+does not permanently penalise it.
+
+Restarts are counted in `producerRestarts` and failures in `producerErrors`, both
+reported per protocol on `/health`. They are tracked separately from the fetch
+loop's `consecutiveErrors` on purpose: that counter is only cleared by a
+successful `readData`, so sharing it left an idle streaming chain reporting
+errors long after its producer recovered.
+
 ## Key exports
 
 - `genSyncProtocols(dbConn, syncInfo)` - Effection generator that instantiates a runtime fetcher + state pair for every protocol in `syncInfo` (from `config.syncProtocols`). Called from the runtime's process-blocks loop.
@@ -116,7 +175,7 @@ as `buf` pins to `cap`.
 
 Per-chain `Fetcher` / `SyncState` classes (`EvmFetcher`,
 `BitcoinFetcher`, `MidnightFetcher`, `AvailFetcher`, `UtxoRpcFetcher`,
-`NtpFetcher`, `CelestiaFetcher`, `NearFetcher`, and matching `*SyncState`
+`NtpFetcher`, `CelestiaFetcher`, `NearFetcher`, `SolanaFetcher`, and matching `*SyncState`
 classes) are exported but are internal to the factory wiring -
 application code drives them through `genSyncProtocols` rather than
 instantiating them directly. Reach for them only if you're writing a

package/docs/ADDING-A-SYNC-PROTOCOL.md

@@ -73,6 +73,9 @@ latitude on its exact shape.
   `readData` (+ `PaginatedFetcher`: `getLatestPage`/`nextInterval`/
   `previousInterval`/`intervalFromStart`, and `PrimitiveFetcher` if it emits
   primitives). `lastPage.own`/`ownBlockNumber` = the **chunk end** (`data.to`).
+  **Bound every RPC call** with `common/http.ts:fetchWithTimeout` (or the client
+  library's own timeout) — a bare `fetch` against a blackholed endpoint hangs
+  `readData` forever, which silently stalls the whole node (CLAUDE.md #4).
   Page-request helpers live in `base/page.ts`
   (`genImmediatePageRequests`/`genOnDemandPageRequests`).
 - **`sync-protocols/<chain>/Client.ts`** (new, only if it talks to a node) — the
@@ -120,19 +123,23 @@ rows and optional scheduled STF inputs.
 
 - **One network → many sync protocols** is allowed (Cardano, `test`). `FlipObject`
   unions them, so `addMain`/`addParallel` narrow correctly.
-- **Resume is currently chunk-granular** (a known issue — CLAUDE.md finding #2):
-  on restart a chain resumes from the *end* of its oldest retained page chunk,
-  which can silently skip an un-committed tail. A block-accurate-resume fix is
-  planned; until then, avoid relying on exact restart positions mid-chunk.
+- **Resume is block-accurate** (CLAUDE.md finding #2, fixed): the runtime is the
+  sole writer of the persisted marker, written inside the block's transaction.
+  Your `state.ts` must implement `outputToLastPage`; `updateState` must not
+  persist anything.
+- **Declare `pollingInterval`** by merging `PollingSyncProtocol` into your schema.
+  It is not optional in practice — the fetch loop paces itself with it, and a
+  protocol without one used to starve the entire event loop (CLAUDE.md #5).
+- **Set `hasAsyncProducer = true`** on the state if your chain's data arrives via
+  `startAsync` rather than polling, so the producer is supervised and restarted
+  (CLAUDE.md #6).
 - **Merge boundary gate**: a parallel chain only merges into a root block at
   timestamp `T` once its own page is *strictly* past `T`. If the main clock runs
   to the same height as a parallel chain's fetched tip, the merge stalls at that
   boundary — keep the main clock at/behind the parallel tips.
-- **`finalizedBlockStream` is unbuffered**: a message sent with no active
-  subscriber is dropped. The merge is spawned before the consumer loop
-  subscribes, so a very fast producer could in principle drop early blocks
-  (RPC-paced chains don't hit this); a subscribe-before-spawn hardening is a
-  planned follow-up.
+- **`finalizedBlockStream` subscribes before the merge is spawned**
+  (`runtime/src/finalized-stream.ts`), because an effection channel drops sends
+  with no active subscriber.
 - **`isPresync`** is currently ~always false (`genInputRange` TODO) — the
   historical-presync path may not trigger as intended.
 

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@effectstream/sync",
-  "version": "0.101.0",
-  "license": "MIT",
+  "version": "0.102.0",
+  "license": "(MIT OR Apache-2.0)",
   "homepage": "https://effectstream.github.io/docs/",
   "repository": {
     "type": "git",
@@ -13,12 +13,12 @@
   },
   "exports": "./src/mod.ts",
   "dependencies": {
-    "@effectstream/config": "0.101.0",
-    "@effectstream/db": "0.101.0",
-    "@effectstream/log": "0.101.0",
-    "@effectstream/utils": "0.101.0",
-    "@midnight-ntwrk/ledger-v8": "8.0.3",
-    "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "4.0.4",
+    "@effectstream/config": "0.102.0",
+    "@effectstream/db": "0.102.0",
+    "@effectstream/log": "0.102.0",
+    "@effectstream/utils": "0.102.0",
+    "@midnight-ntwrk/ledger-v8": "8.1.0",
+    "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "4.1.1",
     "@midnight-ntwrk/onchain-runtime": "npm:@midnight-ntwrk/onchain-runtime-v3@3.0.0",
     "effection": "3.5.0",
     "graphql-ws": "^6.0.6",

package/src/sync-protocols/avail/AvailClient.ts

@@ -1,4 +1,5 @@
 import { SDK } from "avail-js-sdk";
+import { DEFAULT_REQUEST_TIMEOUT_MS, fetchWithTimeout } from "../common/http.ts";
 import type {
   AvailBlock,
   AvailBlockDataItem,
@@ -12,13 +13,25 @@ const AVAIL_NODE_DEFAULT_URL = "ws://localhost:9955/ws";
 export class AvailClient {
   private readonly url: string;
   private readonly sdk: Promise<SDK>;
-  constructor(nodeUrl: string, lightClientUrl: string) {
+  /** Per-request deadline for light-client HTTP; see `common/http.ts`. */
+  private readonly requestTimeoutMs: number;
+  constructor(
+    nodeUrl: string,
+    lightClientUrl: string,
+    requestTimeoutMs = DEFAULT_REQUEST_TIMEOUT_MS,
+  ) {
     this.url = lightClientUrl ?? AVAIL_CLIENT_DEFAULT_URL;
     this.sdk = SDK.New(nodeUrl ?? AVAIL_NODE_DEFAULT_URL);
+    this.requestTimeoutMs = requestTimeoutMs;
   }
 
   async getStatus(): Promise<AvailStatus> {
-    const response = await fetch(`${this.url}/v2/status`);
+    const response = await fetchWithTimeout(
+      `${this.url}/v2/status`,
+      {},
+      "Avail v2/status",
+      this.requestTimeoutMs,
+    );
     if (!response.ok) {
       throw new Error(
         `Failed to get status from light client, status: ${response.status}`,
@@ -32,9 +45,14 @@ export class AvailClient {
   }
 
   async getLatestBlockHeight(): Promise<number> {
-    const api = (await this.sdk).client.api;
+    // Deliberately does NOT await `this.sdk`. The height comes from the light
+    // client below; the websocket SDK was only needed by the commented-out
+    // `api.rpc.chain.getHeader()` call, and awaiting it here reintroduced an
+    // unbounded wait on the sync path — if the node's websocket never connects,
+    // this never returns and the chain stalls silently. That is the exact
+    // failure `requestTimeoutMs` exists to prevent.
     // Block from avail node is slightly above the light client
-    // const header = await api.rpc.chain.getHeader();
+    // const header = await (await this.sdk).client.api.rpc.chain.getHeader();
     const status: AvailStatus = await this.getStatus();
 
     // If the available field is present, this is the real latest block 
@@ -80,7 +98,12 @@ export class AvailClient {
       }
     }
 
-    const response = await fetch(`${this.url}/v2/blocks/${height}/header`);
+    const response = await fetchWithTimeout(
+      `${this.url}/v2/blocks/${height}/header`,
+      {},
+      `Avail v2/blocks/${height}/header`,
+      this.requestTimeoutMs,
+    );
     if (!response.ok) {
       throw new Error(
         `Failed to get block header from height ${height}, status: ${response.status}`,
@@ -103,7 +126,12 @@ export class AvailClient {
       return { block_number: height, data_transactions: [] }
     }
 
-    const response = await fetch(`${this.url}/v2/blocks/${height}/data`);
+    const response = await fetchWithTimeout(
+      `${this.url}/v2/blocks/${height}/data`,
+      {},
+      `Avail v2/blocks/${height}/data`,
+      this.requestTimeoutMs,
+    );
     if (!response.ok) {
       throw new Error(
         `Failed to get block data from height ${height}, status: ${response.status}`,

package/src/sync-protocols/avail/fetcher.ts

@@ -5,6 +5,7 @@ import {
 } from "@effectstream/config";
 import { BaseDataFetcher } from "../base/fetcher.ts";
 import type { DataFetched } from "../base/fetcher.ts";
+import { requestTimeoutOf } from "../common/http.ts";
 import type {
   LastPage,
   OutputAndCleanup,
@@ -38,6 +39,7 @@ export class AvailFetcher extends BaseDataFetcher<
     this.client = new AvailClient(
       config.syncProtocol.rpc,
       config.syncProtocol.lightClient,
+      requestTimeoutOf(config.syncProtocol),
     );
   }
 

package/src/sync-protocols/avail/state.ts

@@ -11,6 +11,7 @@ import { getPage } from "@effectstream/db";
 import { AvailClient } from "./AvailClient.ts";
 import { applyDelay } from "../common/utils.ts";
 import { bufferAtCap } from "../common/page-helpers.ts";
+import { requestTimeoutOf } from "../common/http.ts";
 
 export class AvailSyncState extends SyncState<
   Input,
@@ -138,6 +139,7 @@ export class AvailSyncState extends SyncState<
       new AvailClient(
         config.syncProtocol.rpc,
         config.syncProtocol.lightClient,
+        requestTimeoutOf(config.syncProtocol),
       ),
       dbConn,
     );