@effectstream/sync 0.200.4 → 0.200.6

package/README.md

@@ -72,7 +72,7 @@ catch-up — in steady state the buffer sits near zero and the cap is never reac
 
 **Deadlock-safety (merge-demand exemption).** A naive cap can deadlock the merge: it
 produces a root block at timestamp `τ` only once a parallel chain's page passes `τ`, and
-drains that chain's buffer only *after* the page passes `τ`. If the cap pauses the
+drains that chain's buffer only _after_ the page passes `τ`. If the cap pauses the
 fetcher while its page is still `≤ τ` (the data up to `τ` exceeds the cap — e.g. a far
 skip-ahead or a parallel chain finer-grained than the cap), neither side can proceed: a
 circular wait, not a lost wakeup.
@@ -83,12 +83,13 @@ page (`mergeIntoRoot` sets `SyncState.mergeWaitingForPage`), `bufferAtCap` retur
 **bounded by necessity** — only `(lastPage.root, τ]` is buffered above the cap, exactly
 what the merge must hold to build block `τ`; once the page passes `τ` the normal cap
 re-engages. A chain that is merely tip-limited (caught up, not gated by the merge) is
-*not* exempted, so steady-state and head-of-line bounds are unaffected. Reproductions:
+_not_ exempted, so steady-state and head-of-line bounds are unaffected. Reproductions:
 `buffering.test.ts` 1c (skip-ahead) and 1d (density).
 
 **Scope.** The guard runs in **every** chain's `stateToInput`, so all sync chains are
 covered: EVM, NTP, Bitcoin, Avail, Celestia, NEAR, Solana, Midnight, Cardano (UTXO-RPC), and
 the synthetic `test` chain. Two notes:
+
 - **Cardano (UTXO-RPC)** has no `stepSize` (it streams one block per pass), so the cap
   falls back to a default chunk size of 1000 (⇒ default cap 4000); set `maxBufferedPages`
   explicitly to tune it.
@@ -124,7 +125,7 @@ loop's job (it re-runs the same page range and counts the failure, which is what
 **Coverage.** Honoured by EVM (as the viem transport timeout), Bitcoin, NEAR,
 Avail (light-client HTTP), Midnight and Celestia. Two exceptions:
 
-- **Celestia** retries internally, so the value bounds each *attempt* rather than
+- **Celestia** retries internally, so the value bounds each _attempt_ rather than
   the whole call. It still falls back to `CELESTIA_RPC_TIMEOUT_MS` when no
   protocol-level value is set.
 - **Cardano / utxorpc** reads its data from a gRPC stream rather than
@@ -132,7 +133,8 @@ Avail (light-client HTTP), Midnight and Celestia. Two exceptions:
   protected by producer supervision — a stream that dies or ends is restarted
   with backoff (see below).
 
-NTP and the synthetic `test` chain make no network calls at all.
+The synthetic `test` chain makes no network calls. NTP performs bounded UDP I/O;
+its one-shot boundary is described below.
 
 ```ts
 .addParallel((n) => n.myChain, () => ({
@@ -144,6 +146,42 @@ NTP and the synthetic `test` chain make no network calls at all.
 }))
 ```
 
+## NTP clock ownership and one-shot tip
+
+Every `NtpFetcher` owns its own `ntp-time-sync` client and cache, including when
+`servers` is absent or empty. A non-empty server list is defensively copied into
+that client. This prevents a default or configured fetcher from reusing another
+fetcher's server selection or cached offset.
+
+For startup boundaries, `getNtpTip` builds a private `ntp-time-sync` client
+per call, samples the selected servers concurrently, and returns the inclusive
+page from the first validated round:
+
+```ts
+import { getNtpTip } from "@effectstream/sync";
+
+const { height } = await getNtpTip({
+  startTime: network.startTime,
+  blockTimeMS: network.blockTimeMS,
+  servers: network.servers,
+  signal,
+  // requestTimeoutMs: 15_000, // default
+});
+```
+
+The operation has one deadline (`requestTimeoutMs`), which also bounds the
+client's own socket wait; on timeout or caller abort it clears its timer and
+abort listener and rejects, while the client's in-flight sockets close at that
+same deadline. `NtpTipError.code` distinguishes invalid options, abort,
+timeout, network, and invalid time; abort preserves the caller's reason as
+`cause`, while timeout exposes the effective `timeoutMs`.
+
+Compatibility warning: the exact `ntp-time-sync` 0.6.0 upgrade drops Node 18
+support. Its dependency graph requires Node 20.19+ or Node 22.12+ (the repository's
+supported Bun runtime also works). Version 0.6.0 also corrects offsets to their
+signed RFC 5905 meaning, so both the runtime floor and clock-direction behavior
+are breaking compatibility changes.
+
 ## Producer supervision (streaming chains)
 
 Chains whose data arrives over a subscription rather than by polling (utxorpc
@@ -181,6 +219,34 @@ application code drives them through `genSyncProtocols` rather than
 instantiating them directly. Reach for them only if you're writing a
 custom orchestration layer.
 
+## One-shot Midnight tip
+
+`getMidnightTip()` performs one bounded query for the Midnight indexer's
+current block height. The caller must supply the absolute HTTP(S) `indexer`;
+the helper has no network or endpoint default.
+
+```typescript
+import { getMidnightTip } from "@effectstream/sync";
+
+const { height } = await getMidnightTip({
+  indexer: midnight.indexer,
+  requestTimeoutMs: 15_000, // optional; 15 seconds by default
+  signal,
+});
+```
+
+The operation sends exactly one `query { block { height } }` POST and returns
+one validated non-negative safe integer. It never retries, polls, caches, or
+persists a partial or successful result.
+
+Failures are `MidnightTipError` instances with stable codes:
+`INVALID_OPTIONS`, `ABORTED`, `TIMEOUT`, `NETWORK`, `HTTP`, `GRAPHQL`, and
+`INVALID_RESPONSE`. Caller abort preserves `signal.reason`; network and JSON
+parse failures preserve their original cause; HTTP failures expose
+`status`/`statusText`; and GraphQL failures expose a frozen `graphqlErrors`
+copy. Caller abort and the operation deadline use first-winner semantics and
+release their timer, listener, and request on every settlement.
+
 ## Examples
 
 End-to-end sync test (boots a node, reads blocks, asserts the DB):

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@effectstream/sync",
-  "version": "0.200.4",
+  "version": "0.200.6",
   "license": "(MIT OR Apache-2.0)",
   "homepage": "https://effectstream.github.io/docs/",
   "repository": {
@@ -13,16 +13,16 @@
   },
   "exports": "./src/mod.ts",
   "dependencies": {
-    "@effectstream/config": "0.200.4",
-    "@effectstream/db": "0.200.4",
-    "@effectstream/log": "0.200.4",
-    "@effectstream/utils": "0.200.4",
+    "@effectstream/config": "0.200.6",
+    "@effectstream/db": "0.200.6",
+    "@effectstream/log": "0.200.6",
+    "@effectstream/utils": "0.200.6",
     "@midnightntwrk/ledger-v9": "1.0.0-rc.3",
     "@midnight-ntwrk/midnight-js-indexer-public-data-provider": "5.0.0-beta.6",
     "@midnight-ntwrk/onchain-runtime": "npm:@midnightntwrk/onchain-runtime-v4@4.0.0-rc.3",
     "effection": "3.5.0",
     "graphql-ws": "^6.0.6",
-    "ntp-time-sync": "^0.5.0",
+    "ntp-time-sync": "0.6.0",
     "viem": "2.37.3",
     "denque": "^2.1.0",
     "pg": "^8.14.0",

package/src/sync-protocols/midnight/tip.ts

@@ -0,0 +1,295 @@
+import { Type } from "@sinclair/typebox";
+import { Value } from "@sinclair/typebox/value";
+
+const DEFAULT_REQUEST_TIMEOUT_MS = 15_000;
+const MAX_TIMER_DELAY_MS = 2_147_483_647;
+
+export type GetMidnightTipOptions = {
+  indexer: string;
+  requestTimeoutMs?: number;
+  signal?: AbortSignal;
+};
+
+export type MidnightTip = { height: number };
+
+export type MidnightTipErrorCode =
+  | "INVALID_OPTIONS"
+  | "ABORTED"
+  | "TIMEOUT"
+  | "NETWORK"
+  | "HTTP"
+  | "GRAPHQL"
+  | "INVALID_RESPONSE";
+
+export class MidnightTipError extends Error {
+  readonly code: MidnightTipErrorCode;
+  override readonly cause?: unknown;
+  readonly status?: number;
+  readonly statusText?: string;
+  readonly graphqlErrors?: readonly unknown[];
+
+  constructor(
+    code: MidnightTipErrorCode,
+    message: string,
+    options: {
+      cause?: unknown;
+      status?: number;
+      statusText?: string;
+      graphqlErrors?: readonly unknown[];
+    } = {},
+  ) {
+    super(message, options.cause === undefined ? undefined : { cause: options.cause });
+    this.name = "MidnightTipError";
+    this.code = code;
+    this.cause = options.cause;
+    this.status = options.status;
+    this.statusText = options.statusText;
+    this.graphqlErrors = options.graphqlErrors;
+  }
+}
+
+function validateOptions(options: GetMidnightTipOptions): {
+  indexer: string;
+  requestTimeoutMs: number;
+  signal?: AbortSignal;
+} {
+  if (options == null || typeof options !== "object") {
+    throw new MidnightTipError("INVALID_OPTIONS", "Options must be an object");
+  }
+  if (typeof options.indexer !== "string") {
+    throw new MidnightTipError(
+      "INVALID_OPTIONS",
+      "indexer must be an absolute HTTP(S) URL",
+    );
+  }
+
+  let url: URL;
+  try {
+    url = new URL(options.indexer);
+  } catch {
+    throw new MidnightTipError(
+      "INVALID_OPTIONS",
+      "indexer must be an absolute HTTP(S) URL",
+    );
+  }
+  if (url.protocol !== "http:" && url.protocol !== "https:") {
+    throw new MidnightTipError("INVALID_OPTIONS", "indexer must use HTTP or HTTPS");
+  }
+
+  const requestTimeoutMs =
+    options.requestTimeoutMs === undefined
+      ? DEFAULT_REQUEST_TIMEOUT_MS
+      : options.requestTimeoutMs;
+  if (
+    !Number.isSafeInteger(requestTimeoutMs) ||
+    !Number.isFinite(requestTimeoutMs) ||
+    requestTimeoutMs <= 0
+  ) {
+    throw new MidnightTipError(
+      "INVALID_OPTIONS",
+      "requestTimeoutMs must be a positive finite safe integer",
+    );
+  }
+
+  const signal = options.signal;
+  if (
+    signal !== undefined &&
+    (signal === null ||
+      typeof signal !== "object" ||
+      typeof signal.addEventListener !== "function" ||
+      typeof signal.removeEventListener !== "function" ||
+      typeof signal.aborted !== "boolean")
+  ) {
+    throw new MidnightTipError("INVALID_OPTIONS", "signal must be an AbortSignal");
+  }
+
+  return { indexer: url.href, requestTimeoutMs, signal };
+}
+
+function invalidResponse(message: string, cause?: unknown): MidnightTipError {
+  return new MidnightTipError("INVALID_RESPONSE", message, { cause });
+}
+
+/**
+ * Split deadlines at the maximum delay supported consistently by Node and Bun.
+ * Larger delays are otherwise clamped and can fire years too early.
+ */
+function scheduleDeadline(delayMs: number, onDeadline: () => void): () => void {
+  let remainingMs = delayMs;
+  let active = true;
+  let timeout: ReturnType<typeof setTimeout> | undefined;
+
+  const scheduleNext = (): void => {
+    const chunkMs = Math.min(remainingMs, MAX_TIMER_DELAY_MS);
+    remainingMs -= chunkMs;
+    timeout = setTimeout(() => {
+      if (!active) return;
+      timeout = undefined;
+      if (remainingMs > 0) {
+        scheduleNext();
+        return;
+      }
+      active = false;
+      onDeadline();
+    }, chunkMs);
+  };
+
+  scheduleNext();
+  return () => {
+    if (!active) return;
+    active = false;
+    if (timeout !== undefined) clearTimeout(timeout);
+    timeout = undefined;
+  };
+}
+
+function aborted(reason: unknown): MidnightTipError {
+  return new MidnightTipError("ABORTED", "Midnight tip query was aborted", {
+    cause: reason,
+  });
+}
+
+function timedOut(): MidnightTipError {
+  return new MidnightTipError("TIMEOUT", "Midnight tip query timed out");
+}
+
+/**
+ * POST a JSON body and return the parsed JSON response. Owns all the
+ * network plumbing — caller abort, request timeout, HTTP-status failures,
+ * unread-body cleanup, JSON parsing — and maps every failure to a
+ * MidnightTipError so callers only deal with the decoded payload.
+ */
+async function postJsonRequest(
+  url: string,
+  requestBody: unknown,
+  requestTimeoutMs: number,
+  signal?: AbortSignal,
+): Promise<unknown> {
+  if (signal?.aborted) throw aborted(signal.reason);
+
+  const requestController = new AbortController();
+  let winner: "ABORTED" | "TIMEOUT" | undefined;
+  let callerReason: unknown;
+
+  const onCallerAbort = (): void => {
+    if (winner !== undefined) return;
+    winner = "ABORTED";
+    callerReason = signal?.reason;
+    requestController.abort(callerReason);
+  };
+
+  signal?.addEventListener("abort", onCallerAbort, { once: true });
+  // Close the listener-registration race before allocating a timer or doing I/O.
+  if (signal?.aborted) onCallerAbort();
+  if (winner === "ABORTED") {
+    signal?.removeEventListener("abort", onCallerAbort);
+    throw aborted(callerReason);
+  }
+
+  const clearDeadline = scheduleDeadline(requestTimeoutMs, () => {
+    if (winner !== undefined) return;
+    winner = "TIMEOUT";
+    requestController.abort();
+  });
+
+  const throwIfCancelled = (): void => {
+    if (winner === "ABORTED") throw aborted(callerReason);
+    if (winner === "TIMEOUT") throw timedOut();
+  };
+
+  try {
+    let response: Response;
+    try {
+      response = await fetch(url, {
+        method: "POST",
+        body: JSON.stringify(requestBody),
+        headers: { "Content-Type": "application/json" },
+        redirect: "manual",
+        signal: requestController.signal,
+      });
+    } catch (error) {
+      throwIfCancelled();
+      throw new MidnightTipError("NETWORK", "Midnight tip query failed", {
+        cause: error,
+      });
+    }
+    throwIfCancelled();
+
+    if (!response.ok) {
+      // A fetch resolves after response headers. Explicitly cancel an unread
+      // body so an HTTP failure cannot leave its streaming request/socket live.
+      try {
+        await response.body?.cancel();
+      } catch {
+        // Classification is based on the already-received HTTP response.
+      }
+      throwIfCancelled();
+      throw new MidnightTipError(
+        "HTTP",
+        `Midnight indexer returned ${response.status} ${response.statusText}`,
+        { status: response.status, statusText: response.statusText },
+      );
+    }
+
+    let body: unknown;
+    try {
+      body = await response.json();
+    } catch (error) {
+      throwIfCancelled();
+      throw invalidResponse("Midnight indexer returned malformed JSON", error);
+    }
+    throwIfCancelled();
+
+    return body;
+  } finally {
+    clearDeadline();
+    signal?.removeEventListener("abort", onCallerAbort);
+  }
+}
+
+/**
+ * Checked before the tip shape so a non-empty errors array classifies as
+ * GRAPHQL even when data is absent, and a malformed errors field (or a
+ * non-object body) classifies as INVALID_RESPONSE.
+ */
+const GraphQLEnvelopeSchema = Type.Object({
+  errors: Type.Optional(Type.Array(Type.Unknown())),
+});
+
+const TipResponseSchema = Type.Object({
+  data: Type.Object({
+    block: Type.Object({
+      // minimum + maximum encode Number.isSafeInteger(height) && height >= 0.
+      height: Type.Integer({ minimum: 0, maximum: Number.MAX_SAFE_INTEGER }),
+    }),
+  }),
+});
+
+/** Query one validated numeric block height from an explicit Midnight indexer. */
+export async function getMidnightTip(
+  options: GetMidnightTipOptions,
+): Promise<MidnightTip> {
+  const validated = validateOptions(options);
+  const body = await postJsonRequest(
+    validated.indexer,
+    { query: "query { block { height } }" },
+    validated.requestTimeoutMs,
+    validated.signal,
+  );
+
+  if (!Value.Check(GraphQLEnvelopeSchema, body)) {
+    throw invalidResponse("Midnight indexer returned a malformed GraphQL response");
+  }
+
+  if (body.errors !== undefined && body.errors.length > 0) {
+    throw new MidnightTipError("GRAPHQL", "Midnight indexer returned GraphQL errors", {
+      graphqlErrors: Object.freeze(body.errors.slice()),
+    });
+  }
+
+  if (!Value.Check(TipResponseSchema, body)) {
+    throw invalidResponse("Midnight indexer returned an invalid tip response");
+  }
+
+  return { height: body.data.block.height };
+}

package/src/sync-protocols/mod.ts

@@ -3,6 +3,7 @@ export * from "./evm/state.ts";
 
 export * from "./midnight/fetcher.ts";
 export * from "./midnight/state.ts";
+export * from "./midnight/tip.ts";
 
 export * from "./avail/fetcher.ts";
 export * from "./avail/state.ts";
@@ -12,6 +13,8 @@ export * from "./bitcoin/state.ts";
 
 export * from "./ntp/fetcher.ts";
 export * from "./ntp/state.ts";
+export { getNtpTip, NtpTipError } from "./ntp/tip.ts";
+export type { GetNtpTipOptions, NtpTip, NtpTipErrorCode } from "./ntp/tip.ts";
 
 export * from "./utxorpc/fetcher.ts";
 export * from "./utxorpc/state.ts";

package/src/sync-protocols/ntp/fetcher.ts

@@ -6,7 +6,11 @@ import {
 import type { RootOutput, RootPage } from "../types.ts";
 import type { Operation } from "effection";
 import { all, call, until } from "effection";
-import { bound, type NtpBlockHash, type NtpPageJson } from "@effectstream/utils";
+import {
+  bound,
+  type NtpBlockHash,
+  type NtpPageJson,
+} from "@effectstream/utils";
 import { blockNumberRelation } from "../common/utils.ts";
 import type { Input, Output, Page, PrimitiveType } from "./types.ts";
 import {
@@ -24,11 +28,7 @@ import type {
   SyncProtocolWithNetwork,
 } from "@effectstream/config";
 import type { ConfigSyncProtocolType } from "@effectstream/config";
-import { NtpTimeSync, type NtpTimeSyncDefaultOptions } from "ntp-time-sync";
-
-type RecursivePartial<T> = {
-  [P in keyof T]?: RecursivePartial<T[P]>;
-};
+import { NtpTimeSync } from "ntp-time-sync";
 
 export class NtpFetcher
   extends BaseDataFetcher<Input, Output, RootOutput, Page, RootPage>
@@ -40,7 +40,8 @@ export class NtpFetcher
       PrimitiveType,
       ConfigSyncProtocolType.NTP_MAIN
     >,
-    PaginatedFetcher<Page> {
+    PaginatedFetcher<Page>
+{
   readonly ntpTimeSync: NtpTimeSync;
   constructor(
     readonly config: Extract<
@@ -50,17 +51,15 @@ export class NtpFetcher
   ) {
     super(config.syncProtocol.name);
 
-    const options: RecursivePartial<typeof NtpTimeSyncDefaultOptions> = {
+    const options = {
       ntpDefaults: {
         minPoll: 10,
       },
+      ...(config.network.servers?.length
+        ? { servers: [...config.network.servers] }
+        : {}),
     };
-    if (config.network.servers && config.network.servers.length) {
-      options.servers = config.network.servers;
-    }
-    // Cannot select options
-    // https://github.com/buffcode/ntp-time-sync/issues/104
-    this.ntpTimeSync = NtpTimeSync.getInstance();
+    this.ntpTimeSync = new NtpTimeSync(options);
   }
 
   @bound
@@ -125,7 +124,7 @@ export class NtpFetcher
         own: Number(data.to),
         root: rootConversion.toRootPage({
           blockHashes: [] as NtpBlockHash[],
-          raw: (yield* call(() => pageFetcher(Number(data.to)))),
+          raw: yield* call(() => pageFetcher(Number(data.to))),
         }),
       },
     };
@@ -152,15 +151,13 @@ export class NtpFetcher
   }
 
   @bound
-  *getLatestPage(
-    knownLastPage: undefined | Page,
-  ): Operation<Page> {
+  *getLatestPage(knownLastPage: undefined | Page): Operation<Page> {
     const self = this;
     return yield* fetchNewestPage<Page>(
       blockNumberRelation,
       function* (): Operation<Page> {
         const result = yield* until(self.ntpTimeSync.getTime());
-        if (result.offset > 5000) {
+        if (Math.abs(result.offset) > 5000) {
           console.error(
             `NTP offset is higher than 5[s]. Please check your NTP server & system time configuration.
 NTP time: ${result.now}