@secondlayer/sdk 9.0.2 → 10.0.0

package/AGENTS.md

@@ -4,20 +4,24 @@ Five facts that are load-bearing. Everything else you can infer from the types;
 these you cannot, and getting them wrong produces code that looks correct and
 silently loses or corrupts data.
 
-## 1. Reads need no key
+## 1. `/v1` reads are open on loopback; `/api` needs the token
 
-`/v1/index/*` and the public subgraph reads are open. Construct the client with
-no credentials and start reading:
+`sl.index.*`, `sl.streams.*`, `sl.subgraphs.rows`, and the typed `subscribe`
+read `/v1`, open on loopback. Construct the client with no credentials and start reading:
 
 ```ts
-import { Index } from "@secondlayer/sdk";
+import { SecondLayer } from "@secondlayer/sdk";
 
-const sl = new Index();
-const { events } = await sl.events.list({ eventType: "ft_transfer" });
+const sl = new SecondLayer();
+const { events } = await sl.index.events.list({ eventType: "ft_transfer" });
 ```
 
-Keys are for Streams, writes, and higher rate limits. Do not invent an
-`apiKey` requirement or send a placeholder — an empty Bearer is worse than none.
+Everything else under `sl.subgraphs.*` and all of `sl.subscriptions.*` calls
+`/api`, which needs `INSTANCE_TOKEN` as soon as one is configured, loopback
+included, and `secondlayer init` always configures one. `/v1` needs it too once
+the API is bound beyond loopback. Every client reads `INSTANCE_TOKEN` from the
+env when `apiKey` is omitted, so the golden path is: omit `apiKey`, export the
+token. Do not send a placeholder; an empty Bearer is worse than none.
 
 ## 2. Cursors are opaque
 
@@ -28,7 +32,7 @@ arithmetically. Pass back exactly what you were handed:
 ```ts
 let cursor: string | null = null;
 for (;;) {
-  const page = await sl.events.list({ eventType: "ft_transfer", cursor });
+  const page = await sl.index.events.list({ eventType: "ft_transfer", cursor });
   if (page.events.length === 0) break;
   cursor = page.next_cursor;      // ✓
   // cursor = `${lastHeight}:0`;  // ✗ — skips events, silently
@@ -49,10 +53,30 @@ consistent.
 ```ts
 onReorg: async (reorg, ctx) => {
   await tx.deleteFrom("my_rows").where("height", ">=", reorg.fork_point_height).execute();
-  await saveCursor(tx, ctx.cursor);
+  await saveCursor(tx, ctx.cursor); // string, or null for a fork at genesis
 },
 ```
 
+Three more reorg facts the loop enforces so you do not have to:
+
+- **Never rewinds forward.** A page read from below a fork still reports that
+  fork. Nothing past it has been written, so the loop skips the rollback and
+  delivers the page; `onReorg` fires only for forks at or below the checkpoint.
+- **Idle tip is covered (Streams).** A page only reports reorgs overlapping its
+  own span, so on every empty page the Streams loop also calls
+  `reorgs.list` (one extra request per idle poll) and rolls back anything it
+  finds. Index has no reorg list; a fork while idling is seen once a page
+  overlaps it.
+- **Rollback depth is capped.** The fork point is server supplied and drives a
+  `DELETE ... >= fork` on every declared table, so a rewind more than
+  `maxRollbackDepth` blocks (default 1000) below the checkpoint throws a
+  `ValidationError` before the sink runs. Raise it only for a source you trust.
+- **Rollback is at-least-once across restarts.** Applied reorgs are deduped in
+  memory only. A consumer restarted at a checkpoint below a recent fork's
+  orphaned tip sees that fork again, fires `onReorg` again, and re-reads.
+  Correct (the rollback is idempotent, the re-read is a no-op on keyed rows)
+  but not free: persist a checkpoint above the fork to stop it.
+
 ## 4. Rows and the cursor commit in ONE transaction
 
 If they commit separately, a crash between them either replays a batch (double
@@ -75,12 +99,34 @@ await sl.streams.events.consume({
 Hand-rolling it is allowed, but the checkpoint write must be inside the same
 transaction as the row writes.
 
-## 5. `walk()` is not reorg-safe
+## 5. `walk()` and `events.subscribe()` are not reorg-safe
 
 `walk()` iterates a fixed range for backfills and analysis. It does not surface
-reorgs and does not rewind. Anything writing durable state off the tip belongs
-in `consume()` with `onReorg` (or a sink). Use `walk()` for history that is
-already final.
+reorgs and does not rewind. `streams.events.subscribe()` pushes events over
+SSE as they land and never retracts one that a reorg orphans. Anything writing
+durable state off the tip belongs in `consume()` with `onReorg` (or a sink).
+Use `walk()` for history that is already final and `subscribe()` for displays
+and notifications. `subscribe()` is at-least-once: the cursor advances after
+`onEvent` resolves, so a throwing handler sees the event again on reconnect;
+401/4xx/signature errors end the loop and reject `subscription.done`, transport
+errors reconnect with backoff. `batchSize` tops out at 1000 (the largest page the Index
+serves); above that `walk()` throws `ValidationError` instead of paging in
+silently smaller steps. A walk ends only when the server stops advancing
+`next_cursor`, never on a short page.
+
+## 6. `replay()` dump delivery is file-granular, at-least-once
+
+`events.replay({ from })` hands `onDumpFile` every dump file ending above
+`from`, whole. The file that straddles `from` includes rows at or below it, so
+skip those with `ctx.from` (the same cursor) or key rows by `cursor` so a
+re-run is a no-op. Files ending at or below `from` are not delivered.
+
+```ts
+onDumpFile: async (file, { from }) => {
+  const rows = await readParquet(await streams.dumps.download(file));
+  await insert(rows.filter((r) => from === null || isAfter(r.cursor, from)));
+},
+```
 
 ---
 
@@ -96,9 +142,24 @@ already final.
   the SELECT server-side; unrequested columns are absent from the row *and* the
   type. `cursor`, `block_height`, and `event_type` always come back.
 - **Errors**: everything derives from `SecondLayerError` — `code`, `retryable`,
-  `retryAfterSeconds`, `docsUrl`, and `walk(predicate)` to find a cause. Retries
-  are built in (`retryCount`/`retryDelay`); `onError` is a void observer, not a
-  retry decision.
+  `retryAfterSeconds`, `docsUrl`, and `walk(predicate)` to find a cause. Every
+  failure status keeps the server's `{error, code}` envelope, Streams and Index
+  alike; a 409 `OPERATION_IN_PROGRESS` on deploy/reindex/backfill is a plain
+  `ApiError`. `context()` never throws: each field is `{ value, error? }`.
+- **Verification**: `verify: true` fetches the signing key once and caches
+  only a successful fetch (a 5xx there is a retryable `StreamsServerError`).
+  Over plain http off loopback it requires `verify: { publicKey }`.
+- **Retries**: `consume()` and `walk()` retry each page fetch on 429/5xx/network
+  (`retryCount` default 3, `retryDelay` default 1000 ms, `Retry-After` honored);
+  `onError` is a void observer, not a retry decision. One-shot reads (`list`,
+  `get`, `discover`) do not retry; wrap them yourself if a blip must not fail
+  the call. Index and REST requests (`Index`, `Contracts`, `Subgraphs`) time
+  out after `requestTimeoutMs` (default 30 s, `0` disables) as a retryable
+  `ApiError` with code `REQUEST_TIMEOUT`, so a hung socket trips the retry
+  policy instead of stalling a loop. Streams requests do not time out yet.
+  `signal` on `walk()` cancels the in-flight request and the walk rejects
+  with the signal's reason at every boundary, so a walk that returns without
+  throwing reached the end of the feed.
 
 ## More
 

package/README.md

@@ -14,14 +14,19 @@ bun add @secondlayer/sdk
 import { SecondLayer } from "@secondlayer/sdk";
 
 const sl = new SecondLayer({
-  apiKey: process.env.INSTANCE_TOKEN, // from secondlayer init; omit on loopback
+  apiKey: process.env.INSTANCE_TOKEN, // from secondlayer init; read from the env when omitted
   // default baseUrl: http://127.0.0.1:3800  (or SL_API_URL)
 });
 ```
 
-Reads hit your instance. Writes use `INSTANCE_TOKEN` from `secondlayer init` when the
-API is bound beyond loopback. Public archive dumps (`client.dumps`) need no
-instance key.
+Auth, once: `sl.index`, `sl.streams`, `sl.subgraphs.rows`, and the typed
+`subscribe` read `/v1`, which is open on loopback and needs `INSTANCE_TOKEN`
+once the API is bound beyond it. Everything under `sl.subgraphs.*` and `sl.subscriptions.*` calls
+`/api`, which needs `INSTANCE_TOKEN` as soon as one is configured, loopback
+included, and `secondlayer init` always configures one. Every client, including
+`createStreamsClient`, reads `INSTANCE_TOKEN` from the env when `apiKey` is
+omitted; pass `apiKey: ""` to force a keyless call. Public archive dumps
+(`sl.streams.dumps`) need no instance key.
 
 ## Mental model
 
@@ -39,7 +44,7 @@ Everything is indexing — the question is how much of the indexer you run:
 
 ## Streams
 
-Typed HTTP client for the raw event firehose. Loopback reads need no key.
+Typed HTTP client for the raw event firehose. `/v1` reads need no key on loopback.
 
 ```typescript
 const tip = await sl.streams.tip();
@@ -62,7 +67,7 @@ console.log({ tip, firstCursor: page.events[0]?.cursor });
 import { createStreamsClient } from "@secondlayer/sdk";
 
 const streams = createStreamsClient({
-  // apiKey: process.env.INSTANCE_TOKEN, // when the API is bound beyond loopback
+  // apiKey: process.env.INSTANCE_TOKEN, // read from the env when omitted
   // verify: true,                 // verify ed25519 X-Signature on every read
   //                               // (auto-fetches the public key; { publicKey } pins a PEM)
   // dumpsBaseUrl: process.env.SL_STREAMS_DUMPS_URL, // required to use client.dumps
@@ -73,7 +78,16 @@ Verified responses: every Streams read is signed (ed25519 `X-Signature` +
 `X-Signature-KeyId`). Pass `verify: true` to check it on every read (or
 `{ publicKey }` to pin a PEM); a missing/bad signature throws
 `StreamsSignatureError`. The public key is at
-`GET /public/streams/signing-key`.
+`GET /public/streams/signing-key`. `verify` and `verifyDumpsManifest` are
+accepted by `new SecondLayer({...})` too and reach `sl.streams`.
+
+What `verify: true` proves depends on where the key came from. Over https, or
+on loopback, the key is fetched once from the instance and cached; a 5xx on
+that fetch surfaces as a retryable `StreamsServerError` and the next read
+fetches again. Over plain http to any other host the key travels the same
+unprotected path as the data, so the client refuses `verify: true` there and
+asks for `verify: { publicKey }` instead. The tradeoff: you copy the PEM once,
+and a server-side rotation then fails closed until you update it.
 
 Convenience reads:
 
@@ -125,28 +139,39 @@ for await (const event of streams.events.stream({
 
 Real-time push (`events.subscribe`).
 
-Use `client.events.subscribe` for callback-style live delivery — it pushes each
+Use `client.events.subscribe` for callback-style live delivery: it pushes each
 event to `onEvent` as it lands. It's fetch-based (so it carries the Bearer key)
-and works in browsers and Node 18+. It auto-reconnects from the last delivered
-cursor on a dropped connection, and returns an unsubscribe function.
+and works in browsers and Node 18+. It reconnects from the last handled cursor
+after a dropped socket, a clean server close, or `staleAfterMs` (60 s) with no
+frame, backing off from `reconnectDelayMs` (1 s) up to 30 s with jitter and
+honoring `Retry-After`. Errors a retry cannot fix (401, other 4xx, a bad
+signature) end the loop: `onError` sees them and the handle's `done` rejects.
 
 ```typescript
-const unsubscribe = streams.events.subscribe({
+const subscription = streams.events.subscribe({
   types: ["ft_transfer"],          // notTypes / contractId / sender / recipient / assetIdentifier also filter
   // fromCursor: lastCursor,       // resume strictly after this cursor; omit to tail from the tip
   onEvent: async (event) => {
     console.log(event.cursor, event.tx_id);
   },
-  onError: (err) => console.error("reconnecting…", err),
+  onError: (err) => console.error("subscribe", err),
 });
 
+await subscription.done;          // rejects when the loop stopped on an unfixable error
 // later
-unsubscribe(); // or pass `signal` and abort it
+subscription();                   // unsubscribe, or pass `signal` and abort it
 ```
 
+The cursor advances only after `onEvent` resolves, so a handler that throws
+sees the same event again on reconnect (at-least-once). Key durable writes by
+`cursor`. `subscribe` is not reorg-aware: an event delivered then orphaned is
+never retracted, so anything writing durable state belongs in `consume()` with
+`onReorg` or a sink.
+
 Each pushed frame is `{ event, sig, key_id }`. When the client was created with
 `verify` (or `{ publicKey }`), the per-frame ed25519 signature is checked before
-`onEvent` runs; a bad/missing signature throws `StreamsSignatureError`.
+`onEvent` runs; a bad/missing signature ends the subscription with
+`StreamsSignatureError`.
 
 Bulk parquet dumps.
 
@@ -163,7 +188,6 @@ with `verifyDumpsManifest: false`.
 
 ```typescript
 const streams = createStreamsClient({
-  apiKey: process.env.SL_API_KEY!,
   dumpsBaseUrl: process.env.SL_STREAMS_DUMPS_URL!,
 });
 
@@ -174,18 +198,34 @@ for (const file of manifest.files) {
 }
 ```
 
+`download` hashes the body as it streams in and retries a dropped socket or
+5xx with the same policy `consume()` uses for pages, so a multi-hundred-MB
+file costs its own size in memory, not double, and one flaky fetch does not
+end a replay. The bytes are still returned whole; write them out per file.
+
+```typescript
+for (const file of (await streams.dumps.list()).files) {
+  await Bun.write(file.path.split("/").pop()!, await streams.dumps.download(file));
+}
+```
+
 Backfill then tail (`events.replay`).
 
 Backfills from bulk dumps, then tails live from the manifest's
-`latest_finalized_cursor` — no gap or dupe at the seam. `onDumpFile` hands you
+`latest_finalized_cursor`, with no gap or dupe at the seam. `onDumpFile` hands you
 each finalized file; `onBatch` receives live events after the seam.
 
+Dump delivery is file-granular and at-least-once: the file straddling `from`
+arrives whole, with `ctx.from` so you can skip rows at or below the checkpoint
+(or key rows by `cursor` and let the upsert dedupe). Files ending at or below
+`from` are not delivered.
+
 ```typescript
 await streams.events.replay({
   from: lastCheckpoint,
-  async onDumpFile(file) {
+  async onDumpFile(file, { from }) {
     const bytes = await streams.dumps.download(file);
-    await ingestParquet(bytes); // your tooling
+    await ingestParquet(bytes, { skipAtOrBelow: from }); // your tooling
   },
   async onBatch(events, envelope) {
     for (const event of events) await handle(event);
@@ -245,9 +285,12 @@ Checkpointed consumer — build your app index.
 
 `index.events.consume` / `index.contractCalls.consume` is the same contract as
 the Streams consumer: write your rows inside `onBatch`, return the cursor you
-committed, and reorgs rewind automatically to the fork point. `finalizedOnly`
-holds delivery to rows at or below `tip.finalized_height` (Index rows carry no
-per-event flag). Full runnable example: `examples/sales-index/`.
+committed, and reorgs rewind automatically to the fork point. Only forks at or
+below your checkpoint roll back (a fork above it has nothing to undo), and a
+rewind deeper than `maxRollbackDepth` (default 1000 blocks) is refused before
+anything is deleted. `finalizedOnly` holds delivery to rows at or below
+`tip.finalized_height` (Index rows carry no per-event flag). Walkthrough:
+[docs/index](https://www.secondlayer.tools/docs/index#build-your-index-on-it).
 
 ```typescript
 await sl.index.contractCalls.consume({
@@ -338,7 +381,7 @@ Exported types: `TransactionProof`, `TransactionProofVerifyResult`, `RewardSet`.
 
 Deploy and query app-specific tables.
 
-Subgraphs and subscriptions live on the platform API alongside Streams and Index. Deploying and managing them needs your `sk-sl_` key — no extra setup, no tenant URL.
+Subgraphs and subscriptions live on the instance API alongside Streams and Index. Everything here except `rows` and the typed `subscribe` calls `/api`, which needs `INSTANCE_TOKEN` once one is configured (init always configures one), loopback included.
 
 ```typescript
 // List
@@ -347,7 +390,7 @@ const { data } = await sl.subgraphs.list();
 // Get
 const subgraph = await sl.subgraphs.status("my-subgraph");
 
-// Open read (/v1) — keyless for public subgraphs; pass apiKey for your private ones
+// Open read (/v1): keyless on loopback; needs the token once the API is bound beyond it
 const { rows, next_cursor, tip } = await sl.subgraphs.rows("my-subgraph", "transfers", {
   order: "desc",
   limit: 50,
@@ -374,6 +417,12 @@ const gaps = await sl.subgraphs.gaps("my-subgraph");
 const result = await sl.subgraphs.deploy({ name, sources, schema, handlerCode });
 ```
 
+Filters and `orderBy` on the typed client name the system columns
+`_id`, `_blockHeight`, `_txId`, `_createdAt` (the canonical row shape). The
+unprefixed `id` / `blockHeight` / `txId` / `createdAt` shorthands mean the
+system column only when your table declares no column of that name; a declared
+`id` column is always your `id`.
+
 Stream rows live with the typed client — each table exposes `subscribe`
 alongside `findMany`/`count`:
 
@@ -393,10 +442,14 @@ const unsubscribe = subgraph.transfers.subscribe(
 unsubscribe();
 ```
 
-`subscribe` is an SSE stream over the global `EventSource` (available in
-browsers and Node ≥ 22; it throws if no `EventSource` is present). Frames are
-unsigned rows. `since: <block_height>` replays matching rows from that height,
-then tails the live edge; omit it to tail only.
+`subscribe` reads `/v1` like `rows`, so it is keyless on loopback; it is a
+fetch-based SSE stream, so it carries the client's bearer token once the API is
+bound past loopback, in browsers and Node 18+. Frames are unsigned rows in the
+wire shape (`_block_height`, bigint columns as strings). `since: <block_height>`
+replays matching rows from that height, then tails the live edge; omit it to
+tail only. A dropped connection reconnects from the last delivered row's
+`_block_height`, so rows at that height can arrive twice: key durable writes
+by `_id`.
 
 ## Subscriptions
 
@@ -435,7 +488,7 @@ to a SIP/trait (e.g. `"sip-010"`); amounts are non-negative integer strings
 ```typescript
 import { SecondLayer, trigger } from "@secondlayer/sdk";
 
-const sl = new SecondLayer({ apiKey: "sk-sl_..." });
+const sl = new SecondLayer({ apiKey: process.env.INSTANCE_TOKEN });
 
 const { subscription, signingSecret } = await sl.subscriptions.create({
   name: "amm-swaps",
@@ -572,7 +625,21 @@ try {
 }
 ```
 
-Tenant-resolution failures surface as `ApiError` with distinctive codes:
+Every failure status keeps the server's envelope: a 401 with
+`code: "TOKEN_REVOKED"` reads differently from a missing token, a 503 carries
+its reason and `retryAfterSeconds`, and Streams 4xx errors carry `code` the
+same way Index errors do (`err.code === "CURSOR_INVALID"`).
+
+Codes you will branch on:
+
+- `code: "OPERATION_IN_PROGRESS"` (409) from `subgraphs.deploy`, `reindex`, or
+  `backfill`: a reindex or backfill is already running for that subgraph. Poll
+  `subgraphs.operations(name)` and retry when it finishes. There is no
+  dedicated error class for this; it is an `ApiError` with `status === 409`.
+- `code: "REQUEST_TIMEOUT"` (status 0): the request outlived `requestTimeoutMs`.
+  Retryable.
 
-- `code: "TENANT_SUSPENDED"` — your tenant is suspended (see `err.message` for the limit reason)
-- `code: "NO_TENANT"` — your account has no provisioned tenant yet
+`sl.context()` never throws. Each field is `{ value, error? }`, so a `null`
+says why: `error.status === 0` means the API was unreachable,
+`error.code === "UNAUTHORIZED"` means the token was rejected, and no `error`
+at all means the read succeeded and found nothing.