@saga-sync/producer 0.1.3 → 0.2.0

package/README.md

@@ -0,0 +1,473 @@
+# @saga-sync/producer
+
+The producer side of **saga-sync** — the reference implementation of
+[privacy-protocol-state-distribution](../../README.md). It scrapes a privacy
+protocol's on-chain events, packages them into immutable compressed chunks with
+integrity digests, and publishes an `index.json` manifest that any number of
+[`@saga-sync/client`](../client) consumers can download and verify.
+
+Three stages, each **both a standalone CLI and an importable library**:
+
+```
+  RPC ──▶ scraper ──NDJSON──▶ chunk-builder ──▶ Store (disk | gs://…)
+              ▲                     ▲                     │ index.json + *.jsonl.gz
+              └──────── orchestrator (cron entry) ────────┘
+```
+
+- **scraper** — connects to an Ethereum JSON-RPC, fetches event logs for a
+  protocol over a block range, normalizes them, emits them as NDJSON.
+- **chunk-builder** — consumes that NDJSON, partitions it into size-bounded
+  immutable chunks (gzip + sha256), and maintains the append-only manifest. The
+  trailing partial chunk is kept as a mutable **hot head**.
+- **orchestrator** — the cron entry point. Loops every protocol in the config,
+  runs `scrape → chunk` for each **in-process** in block-range batches, and owns
+  the hot-head lifecycle. This is the normal path; the two lower CLIs are for
+  backfills, debugging, or composing by hand.
+
+Depends on [`@saga-sync/core`](../core) (schema, crypto, `Store`), `viem` (RPC),
+`zod` (config validation), and optionally `@google-cloud/storage` (lazy-loaded,
+only for `gs://` targets).
+
+## Install & build
+
+```bash
+pnpm install
+pnpm build          # tsc -b, builds core first
+```
+
+The examples below run the built JS from the repo root (`packages/producer/dist/…`).
+
+---
+
+## Orchestrator CLI — the normal path
+
+Loads every protocol from the config, computes each one's next-from-block from the
+manifest (sealed chunks + mutable hot head), and loops `scrape → chunk` in
+**batches** of `--batch-size` blocks (default 100K). Within a tick protocols run
+sequentially, a lockfile prevents overlapping runs, and the trailing partial after
+each batch is persisted as the protocol's **hot head** — subsequent ticks fold new
+events into it until it reaches the size limit and is promoted into the immutable
+list.
+
+```bash
+node packages/producer/dist/orchestrator/cli.js \
+  --config ./example-config.json \
+  --rpc https://ethereum-rpc.publicnode.com \
+  --output-dir ./chunks
+```
+
+stderr summary on a cold start with a forced-promotion size limit:
+
+```
+orchestrator: tornado-cash-1-eth-0.1 ran 3 batch(es), sealed 14 chunk(s) + hot head [0x17f76ce, 0x17f7803)
+orchestrator: 1 ran, 0 skipped, 0 failed [tip 0x17f7802]
+```
+
+What gets created:
+
+```
+./chunks/
+  tornado-cash-1-eth-0.1-[0x17f7000,0x17f71f3).jsonl.gz      # immutable sealed chunks
+  ...
+  tornado-cash-1-eth-0.1-[0x17f76ce,0x17f7803).hot.jsonl.gz  # mutable hot head (one per protocol)
+  index.json
+  .orchestrator.lock                                          # only present during a run
+```
+
+Re-running with no chain advance is a no-op: the start block is derived from
+`max(hotHead.toBlock, lastSealed.toBlock)` and the protocol is skipped when
+there's nothing new. The previous hot-head file is deleted after the manifest is
+updated (its URL changes each time the range advances, so every URL stays
+immutable and CDN-cacheable).
+
+### Cron entry (daily)
+
+One entry handles every protocol; a still-running tick makes the next exit quietly
+via the lockfile.
+
+```
+0 2 * * * cd /path/to/repo && node packages/producer/dist/orchestrator/cli.js \
+  --config ./example-config.json --rpc https://ethereum-rpc.publicnode.com \
+  --output-dir ./chunks >> /var/log/orchestrator.log 2>&1
+```
+
+For the cloud deployment (Cloud Run Job + Cloud Scheduler + Cloud CDN) see the
+repo-root [DEPLOY.md](../../DEPLOY.md).
+
+### Flags
+
+```
+--config <path>        required   config JSON (all protocols)
+--rpc <url>            required   Ethereum JSON-RPC URL (one chain per run)
+--output-dir <path>    required   chunks + index.json directory, or gs://bucket[/prefix]
+--lock-dir <path>      optional   directory for .orchestrator.lock (default = output-dir)
+--protocol-id <id>     optional   restrict to one protocol (backfills / reruns)
+--batch-size <n>       optional   blocks per batch (default 100000)
+--concurrency <n>      optional   protocols scraped in parallel (default 4)
+--confirmations <n>    optional   fallback reorg buffer (default 12)
+--window <n>           optional   scraper window (default 2000)
+--size-limit <n>       optional   chunk cap if config has none (default 10 MiB)
+--dry-run              optional   report what would run, touch nothing
+```
+
+- **`--protocol-id`** still derives the from-block from the manifest, so it
+  catches one protocol up to tip without affecting the others.
+- **`--batch-size`** bounds the crash blast radius (in-flight events lost on crash
+  ≤ one batch); larger batches reduce hot-head rewrite overhead during cold-start
+  backfills. Daily steady-state usually fits in one batch.
+- **Chunk size** comes from each protocol's `chunkSettings.maxSizeBytes` in the
+  config, falling back to `--size-limit`.
+- **`--dry-run`** prints per-protocol ranges without acquiring the lockfile — safe
+  alongside a live cron.
+
+### Exit codes
+
+- `0` — clean run (zero or more protocols ran, or another orchestrator held the lock)
+- `1` — config error, missing flags, or unknown `--protocol-id`
+- `2` — at least one protocol threw mid-run (others may still have succeeded)
+
+---
+
+## Scraper CLI
+
+Pure "input range → output events": connects to an RPC, fetches + normalizes logs,
+writes NDJSON to **stdout** and a one-line summary to **stderr**. Writes no chunks
+or manifest.
+
+```bash
+node packages/producer/dist/scraper/cli.js \
+  --config ./example-config.json \
+  --protocol-id tornado-cash-1-eth-0.1 \
+  --rpc https://ethereum-rpc.publicnode.com \
+  --from-block 0xC50101 --to-block 0xC50200 \
+  --dry-run
+# → NDJSON on stdout; stderr:
+# scraper: 2 event(s) for tornado-cash-1-eth-0.1 [0xc50101, 0xc50200] (dry-run: cursor not updated)
+```
+
+Useful variations:
+
+```bash
+# Persist a cursor (drop --dry-run) — next run with no --from-block resumes from cursor+1
+node .../scraper/cli.js --config ./example-config.json --protocol-id tornado-cash-1-eth-0.1 \
+  --rpc <url> --from-block 0xC50000 --to-block 0xC50100
+cat cursor.json   # → { "tornado-cash-1-eth-0.1": { "lastScrapedBlock": "0xc50100" } }
+
+# Scrape up to the chain's finalized block (omit --to-block)
+node .../scraper/cli.js --config ./example-config.json --protocol-id tornado-cash-1-eth-0.1 \
+  --rpc <url> --from-block 0xC50101 --dry-run
+
+# Inspect with jq
+node .../scraper/cli.js ... --dry-run | jq -c '{block: .blockNumber, topic: .eventTopic}'
+```
+
+### Flags
+
+```
+--config <path>        required   config JSON
+--protocol-id <id>     required   protocol key to scrape
+--rpc <url>            required   Ethereum JSON-RPC URL
+--from-block <hex>     optional   override cursor / config fromBlock
+--to-block <hex>       optional   override the resolved finalized block
+--confirmations <n>    optional   fallback reorg buffer (default 12)
+--window <n>           optional   blocks per eth_getLogs call (default 2000)
+--cursor-dir <path>    optional   directory for cursor.json (default = config dir)
+--dry-run              optional   do not persist the cursor
+```
+
+Exit 0 on success, 1 on error.
+
+---
+
+## Chunk-builder CLI
+
+Reads `CanonicalEvent` NDJSON on **stdin** (scraper output), partitions the scanned
+range into immutable `.jsonl.gz` chunks, and appends to a local `index.json`.
+
+```bash
+node packages/producer/dist/scraper/cli.js --config ./example-config.json \
+  --protocol-id tornado-cash-1-eth-0.1 --rpc <url> \
+  --from-block 0xC50101 --to-block 0xC50200 --dry-run \
+| node packages/producer/dist/chunk-builder/cli.js \
+    --protocol-id tornado-cash-1-eth-0.1 \
+    --from-block 0xC50101 --to-block 0xC50201 \
+    --output-dir ./chunks
+```
+
+**Mind the upper bound.** The scraper's `--to-block` is **inclusive**
+(`[from, to]`); the chunk-builder's is **exclusive** (`[from, to)`). To cover the
+same range, pass `scraper.to_block + 1` to the chunk-builder.
+
+The standalone CLI always **seals** the trailing partial (no hot heads — that path
+is orchestrator-only). An empty range still emits one zero-byte chunk covering the
+full `[from, to)`, so the manifest asserts the range was scanned.
+
+```bash
+# Force multiple chunks with a small size limit
+node .../scraper/cli.js ... | node .../chunk-builder/cli.js \
+  --protocol-id tornado-cash-1-eth-0.1 --from-block 0xC50000 --to-block 0xC50501 \
+  --output-dir ./chunks --size-limit 800
+
+# Inspect a chunk
+gunzip -c './chunks/tornado-cash-1-eth-0.1-[0xc50101,0xc50201).jsonl.gz' | jq -c .
+```
+
+### Flags
+
+```
+--protocol-id <id>     required   manifest key + filename prefix
+--from-block <hex>     required   inclusive start of the scanned range
+--to-block <hex>       required   exclusive end of the scanned range
+--output-dir <path>    required   chunks + index.json directory
+--size-limit <n>       optional   max uncompressed bytes per chunk (default 10 MiB)
+--dry-run              optional   compute metadata, write nothing
+```
+
+---
+
+## Module internals
+
+**`scraper/`** — pure "range → events"; writes no chunks or manifest.
+
+- **`config.ts`** — reads the config JSON, zod-validates **only** the fields the
+  pipeline uses. `loadConfig(path, id)` returns one `ScraperTarget`;
+  `loadAllProtocols(path)` returns all. Validated: `chainId`, `fromBlock`,
+  `events[]`; optional `chunkSettings.maxSizeBytes`, `protocol`, `protocolMetadata`.
+  Derives `trackedAddresses`/`trackedEventTopics` from the unique
+  `events[].contractAddress`/`.eventTopic` in first-seen order.
+- **`normalize.ts`** — `normalize(rpcLog)` → `CanonicalEvent`: lowercases every hex
+  field, sets `eventTopic = topics[0]`, keeps the full `topics` array (indexed args
+  like commitments/nullifiers), rejects pending logs, and drops only
+  `transactionHash`/`blockHash`.
+- **`scrape.ts`** — `scrape(client, opts)`, an async generator yielding raw RPC
+  logs. Slices `[fromBlock, toBlock]` into `window`-sized sub-ranges, issuing one
+  `eth_getLogs` per event filter; on a range/result-size error it **halves the
+  window and retries**. Each window is fully buffered and sorted by
+  `(blockNumber, logIndex)` before any log is yielded, so a retry never double-emits.
+- **`cursor.ts`** — `Cursor` over a `Store`, recording `lastScrapedBlock` per
+  protocol. **Only** the standalone scraper CLI uses it; the orchestrator derives
+  resume points from the manifest.
+- **`cli.ts`** — the scraper entry point. Also exports `finalizedBlock(client)`
+  and `assertChainId(client, expected)`, reused by the orchestrator.
+
+**`chunk-builder/`** — `CanonicalEvent` NDJSON → chunk files + drives core's `Manifest`.
+
+- **`accumulator.ts`** — `ChunkAccumulator`, a **pure** block-aligned partition
+  state machine. Buffers the in-progress block separately and commits it only once
+  the next block arrives — a chunk boundary always falls *between* blocks (a
+  multi-event block is never split). `add(event)` returns a completed chunk on a
+  size boundary; `finish()` returns the trailing accumulator.
+- **`archive.ts`** — `ChunkArchive` over a `Store`. `seal()` / `writeHotHead()`
+  build the JSONL, compute the sha256 of the **uncompressed** bytes, gzip, and
+  `put` under a range-derived filename. `readEvents()` is the inverse.
+  `buildJsonl()` is exported (the client's integration tests use it as a fixture
+  builder).
+- **`cli.ts`** — `processStream(lines, args)` drives the accumulator: optional seed
+  events (a prior hot head) → stream events, sealing each completed chunk → the
+  trailing accumulator, either **sealed** (`trailingMode: "seal"`, standalone
+  default) or **returned** (`trailingMode: "suspend"`, the orchestrator's hot-head
+  carry-over).
+
+**`orchestrator/`** — the cron entry point; composes the other two in-process.
+
+- **`pipeline.ts`** — `runProtocolOnce(opts)` builds the scrape → normalize →
+  NDJSON generator and hands it to `processStream`. No subprocess, no stdio: errors
+  propagate as plain exceptions.
+- **`cli.ts`** — per tick: acquire the lockfile, load config + manifest, query the
+  chain tip, then for each protocol run `processProtocol` (resolve start block,
+  load any prior hot head, loop `[start, tip]` in `--batch-size` steps, persist the
+  final trailing accumulator as the new hot head). Also exports `acquireLock(path)`.
+
+**`storage/`** — the producer-only stores + the factory.
+
+- **`GcsStore`** — write-side, backed by a GCS bucket (optional `prefix`). GCS
+  writes are atomic + strongly consistent (no temp-file+rename). Sets per-key
+  `Content-Type` + `Cache-Control` (sealed chunks immutable/long-lived; `index.json`
+  / hot head short TTL). The `@google-cloud/storage` SDK is lazy-loaded behind an
+  injectable provider, so the module compiles + unit-tests without the dependency.
+- **`DryRunStore`** — decorates another `Store`; `put`/`delete` become no-ops,
+  `get`/`list` pass through. How `--dry-run` is implemented without every other
+  class knowing about it.
+- **`createStore(cfg)`** / **`parseStoreTarget(target)`** — the factory mapping
+  `disk`→core's `DiskStore`, `http`→core's `HttpStore`, `gcs`→`GcsStore`
+  (`s3`/`ftp` throw until added); a `gs://bucket[/prefix]` `--output-dir` selects
+  `GcsStore`, anything else is a local disk path.
+
+**`sources/`** — where a stream's records come from. The orchestrator drives block
+ranges and chunking; a `ScraperSource` decides what to yield.
+
+- **`RpcLogSource`** — windowed `eth_getLogs`, the original path (adaptive window
+  halving, rate-limit backoff, `normalize()` to canonical form). Its
+  `latestCoveredBlock()` is the chain's finalized block.
+- **`SubsquidSource`** — mirrors a Squid GraphQL index, for state no log carries
+  (Railgun's per-transaction operations live only in `transact()` calldata).
+  Pagination, retries and termination match kohaku's own client so both fail the
+  same way. **Its tip is clamped to the chain's finalized block**: the index runs
+  ~75 blocks behind head, i.e. ahead of finality, and sealed chunks are immutable.
+
+Selected per stream by the config's `source` block, which defaults to
+`{"kind":"rpc"}` so configs written before sources existed are unchanged. `events`
+is required for an rpc source and rejected for any other. Adding a source kind is
+one `case` in `createSource()` plus one class — the same shape as `createStore()`.
+
+**`keygen.ts`** — CLI that mints a manifest-signing keypair (over core's
+`signing`). `keygen` alone gives Ed25519; `keygen --alg secp256k1` gives an
+Ethereum-shaped key. Set `MANIFEST_SIGNING_KEY` and/or
+`MANIFEST_SIGNING_KEY_SECP256K1` (32-byte hex seeds) on the orchestrator /
+chunk-builder and each configured key signs every manifest write; consumers pin
+any of the matching public keys via the client's `--public-key`.
+
+A consumer is accepted by **any** one signature, so every additional key is
+another way to forge a manifest. Add a second algorithm for reach (secp256k1
+works with Ethereum tooling and hardware wallets), not for strength — SPEC §9.1.
+
+---
+
+## Data formats
+
+### Config JSON (`--config`)
+
+*What* to scrape, not *where/when*. The store target is a per-run CLI argument and
+the schedule is external — all protocols in one config share one store + manifest.
+
+```json
+{
+  "protocols": {
+    "tornado-cash-1-eth-0.1": {
+      "chainId": "0x1",
+      "fromBlock": "0x8b1d26",
+      "chunkSettings": { "maxSizeBytes": 10485760 },
+      "protocol": "tornado-cash",
+      "protocolMetadata": { "denomination": "100000000000000000", "asset": "ETH" },
+      "events": [
+        { "contractAddress": "0x12d66f87…", "eventTopic": "0xa945e51e…" },
+        { "contractAddress": "0x12d66f87…", "eventTopic": "0xe9e508ba…",
+          "filter": ["0x000…indexed-arg-match"] }
+      ]
+    }
+  }
+}
+```
+
+| Field | Required | Meaning |
+|---|---|---|
+| `chainId` | yes | `0x`-hex chain id; verified against the RPC at startup |
+| `fromBlock` | yes | contract deploy block; the cold-start scrape origin |
+| `events[]` | yes | one entry per `(contractAddress, eventTopic)` to scrape |
+| `events[].filter` | no | extra indexed-topic matchers appended after `eventTopic` |
+| `chunkSettings.maxSizeBytes` | no | per-protocol chunk cap (number or `0x`-hex) |
+| `protocol` | no | family name copied into the manifest (e.g. `tornado-cash`) |
+| `protocolMetadata` | no | free-form passthrough; **immutable per stream** once first published |
+
+The protocol key `${protocol}-${chainId}-${instanceId}` is treated as an opaque id
+(chain identity comes from the explicit `chainId`, since the key itself isn't
+safely parseable — protocol names contain hyphens).
+
+### NDJSON (scraper stdout → chunk-builder stdin)
+
+One `CanonicalEvent` per line, all-lowercase `0x`-hex, globally ordered by
+`(blockNumber, logIndex)`:
+
+```json
+{"contractAddress":"0x12d6…","eventTopic":"0xa945…","topics":["0xa945…","0x1e8f…"],"data":"0x…","blockNumber":"0xc501f5","logIndex":"0x68"}
+```
+
+### Chunk file `${protocolId}-[${fromBlock},${toBlock}).jsonl.gz`
+
+Gzip-compressed JSONL of the NDJSON above, for the block range in the filename.
+Immutable once written. An empty chunk is a zero-byte payload. The **hot-head file**
+(`…hot.jsonl.gz`) is byte-identical in format — the only differences are the `.hot.`
+infix and that the manifest tracks it in the entry's `hotHead` field, not `chunks`.
+At most one hot head per protocol.
+
+### Manifest `index.json`
+
+Format **v1** — see the root [SPEC.md §3.1](../../SPEC.md) for the normative schema
+and the root [README.md](../../README.md) for the protocol-level walkthrough. The
+`Manifest` class that reads/writes it lives in [`@saga-sync/core`](../core).
+`digest.data` is the sha256 of the **uncompressed** JSONL (`gunzip -c <file> |
+shasum -a 256`); `size` is the compressed byte length. With a signing key set, a
+sibling `index.json.sigs` holds the detached signature envelope over the exact
+bytes — one entry per configured algorithm — and `index.json.sig` still holds the
+bare Ed25519 signature for consumers pinned to the original file.
+
+### State files
+
+- **`cursor.json`** — the standalone scraper's resume pointer
+  (`{ "<id>": { "lastScrapedBlock": "0x…" } }`). The orchestrator does not use it.
+- **`.orchestrator.lock`** — plain text pid, created `O_EXCL` at tick start, removed
+  on exit; a stale lock (dead pid) is reclaimed automatically.
+
+---
+
+## Invariants a replica must honor
+
+These are what make two independent scrapers produce byte-identical chunks (see
+[SPEC.md](../../SPEC.md) for the normative statements):
+
+1. **Windowed `eth_getLogs` with adaptive split** — slice the range; on a
+   "range too large" / "too many results" error, halve the window and retry.
+   Buffer + sort each window before yielding.
+2. **Normalization keeps the whole log** — lowercase all hex; `eventTopic` =
+   `topics[0]`; keep `topics[1..]`; drop only `transactionHash`/`blockHash`.
+3. **Block-aligned chunking** — a chunk covers a half-open `[from, to)`; boundaries
+   fall between blocks, so all events of a block land in exactly one chunk.
+4. **Size-bounded sealing** — seal when accumulated **uncompressed** bytes would
+   exceed the limit; the digest is sha256 of that uncompressed JSONL; `size` in the
+   manifest is the **compressed** count.
+5. **Hot heads** — the trailing partial is written mutable (`*.hot.jsonl.gz`,
+   tracked in the entry's `hotHead`). The next run loads it, appends, re-writes; on
+   crossing the size limit it is **promoted** (overflow sealed, fresh hot head holds
+   the remainder).
+6. **Batching** — process `[start, tip]` in `--batch-size` steps; each batch's
+   trailing accumulator seeds the next. The manifest advances only on a completed
+   seal, so a re-run is idempotent (`(blockNumber, logIndex)` is the dedup key).
+7. **Reorg safety** — `toBlock` defaults to the chain's own **finalized** block;
+   fallback for RPCs without the tag is `head − confirmations` (default 12).
+8. **Start-block resolution** — `hotHead.toBlock` → last sealed chunk's `toBlock` →
+   `config.fromBlock` (first run only). The manifest self-bootstraps after the first
+   chunk.
+9. **Range-derived, immutable URLs** — every chunk/hot-head file is named
+   `${protocolId}-[${fromBlock},${toBlock})…`, so a URL's bytes never change. When a
+   hot head advances, a new file is written and the old one deleted; only the
+   manifest pointer is mutable.
+10. **Single-writer lock** — the orchestrator holds `.orchestrator.lock` (`O_EXCL` +
+    pid, stale-pid recovery). A second instance exits quietly.
+
+### Block-range conventions
+
+- The **scraper** works in **inclusive** ranges `[fromBlock, toBlock]` (what
+  `eth_getLogs` takes).
+- **Chunks and the manifest** use **half-open** ranges `[fromBlock, toBlock)`, so
+  consecutive chunks compose with no gap or overlap.
+- The orchestrator bridges them: a scraper batch over inclusive `[X, Y]` becomes a
+  chunk-builder range of half-open `[X, Y+1)`.
+
+Invariant: a protocol's sealed chunks + its hot head partition
+`[firstSealed.fromBlock, hotHead.toBlock)` contiguously.
+
+---
+
+## Publishing to Google Cloud Storage
+
+Publishing to GCS is entirely a `Store`-seam concern — the scrape/chunk logic, the
+manifest/chunk format, and the client are all unchanged. Point `--output-dir` at a
+bucket; consumers read the same objects over plain HTTP.
+
+- **Producer write** — `--output-dir gs://bucket[/prefix]` routes through
+  `parseStoreTarget` → `GcsStore`, writing chunks + `index.json` with per-object
+  `Cache-Control` (sealed chunks `max-age=31536000, immutable`; `index.json` / hot
+  head `max-age=30`).
+- **Consumer read** — point the client's manifest URL at a CDN in front of the
+  bucket, or at the bucket directly. `HttpStore` does plain GETs; nothing in the
+  client changes.
+
+Operational notes: the lockfile is filesystem-only — for a `gs://` target it
+defaults to the cwd (`--lock-dir` to override), and in a stateless container
+single-execution scheduling is the real single-writer guard. Hot-head objects
+orphan as their range advances; a GCS Object Lifecycle rule (delete
+`*.hot.jsonl.gz` after N days) cleans them up. The bucket is public-read —
+integrity comes from the sha256 digests, not access control.
+
+The full cloud runbook (Cloud Run Job, Cloud Scheduler, Secret Manager, Cloud CDN)
+is in the repo-root **[DEPLOY.md](../../DEPLOY.md)**.

package/dist/chunk-builder/accumulator.d.ts

@@ -1,11 +1,11 @@
-import type { CanonicalEvent } from "../scraper/normalize.js";
+import type { CanonicalRecord } from "@saga-sync/core";
 export type CompletedChunk = {
-    events: CanonicalEvent[];
+    events: CanonicalRecord[];
     from: bigint;
     to: bigint;
 };
 export type Trailing = {
-    events: CanonicalEvent[];
+    events: CanonicalRecord[];
     fromBlock: bigint;
 };
 export declare class ChunkAccumulator {
@@ -17,7 +17,7 @@ export declare class ChunkAccumulator {
     private pendingBytes;
     private pendingBlock;
     constructor(sizeLimit: number, chunkFrom: bigint);
-    add(event: CanonicalEvent): CompletedChunk | null;
+    add(event: CanonicalRecord): CompletedChunk | null;
     finish(): {
         completed: CompletedChunk | null;
         trailing: Trailing;

package/dist/chunk-builder/accumulator.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"accumulator.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/accumulator.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,yBAAyB,CAAC;AAE9D,MAAM,MAAM,cAAc,GAAG;IAAE,MAAM,EAAE,cAAc,EAAE,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAAC;AACpF,MAAM,MAAM,QAAQ,GAAG;IAAE,MAAM,EAAE,cAAc,EAAE,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,CAAC;AAYvE,qBAAa,gBAAgB;IAQzB,OAAO,CAAC,QAAQ,CAAC,SAAS;IAC1B,OAAO,CAAC,SAAS;IARnB,OAAO,CAAC,WAAW,CAAwB;IAC3C,OAAO,CAAC,gBAAgB,CAAK;IAC7B,OAAO,CAAC,OAAO,CAAwB;IACvC,OAAO,CAAC,YAAY,CAAK;IACzB,OAAO,CAAC,YAAY,CAAuB;gBAGxB,SAAS,EAAE,MAAM,EAC1B,SAAS,EAAE,MAAM;IAG3B,GAAG,CAAC,KAAK,EAAE,cAAc,GAAG,cAAc,GAAG,IAAI;IAoBjD,MAAM,IAAI;QAAE,SAAS,EAAE,cAAc,GAAG,IAAI,CAAC;QAAC,QAAQ,EAAE,QAAQ,CAAA;KAAE;IAclE,OAAO,CAAC,aAAa;IAOrB,OAAO,CAAC,aAAa;IAQrB,OAAO,CAAC,GAAG;CAOZ"}
+{"version":3,"file":"accumulator.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/accumulator.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAC;AAEvD,MAAM,MAAM,cAAc,GAAG;IAAE,MAAM,EAAE,eAAe,EAAE,CAAC;IAAC,IAAI,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAAC;AACrF,MAAM,MAAM,QAAQ,GAAG;IAAE,MAAM,EAAE,eAAe,EAAE,CAAC;IAAC,SAAS,EAAE,MAAM,CAAA;CAAE,CAAC;AAYxE,qBAAa,gBAAgB;IAQzB,OAAO,CAAC,QAAQ,CAAC,SAAS;IAC1B,OAAO,CAAC,SAAS;IARnB,OAAO,CAAC,WAAW,CAAyB;IAC5C,OAAO,CAAC,gBAAgB,CAAK;IAC7B,OAAO,CAAC,OAAO,CAAyB;IACxC,OAAO,CAAC,YAAY,CAAK;IACzB,OAAO,CAAC,YAAY,CAAuB;gBAGxB,SAAS,EAAE,MAAM,EAC1B,SAAS,EAAE,MAAM;IAG3B,GAAG,CAAC,KAAK,EAAE,eAAe,GAAG,cAAc,GAAG,IAAI;IAoBlD,MAAM,IAAI;QAAE,SAAS,EAAE,cAAc,GAAG,IAAI,CAAC;QAAC,QAAQ,EAAE,QAAQ,CAAA;KAAE;IAclE,OAAO,CAAC,aAAa;IAOrB,OAAO,CAAC,aAAa;IAQrB,OAAO,CAAC,GAAG;CAOZ"}

package/dist/chunk-builder/accumulator.js.map

@@ -1 +1 @@
-{"version":3,"file":"accumulator.js","sourceRoot":"","sources":["../../src/chunk-builder/accumulator.ts"],"names":[],"mappings":"AAKA,qEAAqE;AACrE,+EAA+E;AAC/E,0EAA0E;AAC1E,+EAA+E;AAC/E,0DAA0D;AAC1D,EAAE;AACF,6EAA6E;AAC7E,0EAA0E;AAC1E,+EAA+E;AAC/E,2DAA2D;AAC3D,MAAM,OAAO,gBAAgB;IAQR;IACT;IARF,WAAW,GAAqB,EAAE,CAAC;IACnC,gBAAgB,GAAG,CAAC,CAAC;IACrB,OAAO,GAAqB,EAAE,CAAC;IAC/B,YAAY,GAAG,CAAC,CAAC;IACjB,YAAY,GAAkB,IAAI,CAAC;IAE3C,YACmB,SAAiB,EAC1B,SAAiB;QADR,cAAS,GAAT,SAAS,CAAQ;QAC1B,cAAS,GAAT,SAAS,CAAQ;IACxB,CAAC;IAEJ,GAAG,CAAC,KAAqB;QACvB,MAAM,UAAU,GAAG,MAAM,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC;QAC7C,MAAM,SAAS,GAAG,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,GAAG,IAAI,EAAE,MAAM,CAAC,CAAC;QAC1E,IAAI,SAAS,GAA0B,IAAI,CAAC;QAE5C,IAAI,IAAI,CAAC,YAAY,KAAK,IAAI,IAAI,UAAU,KAAK,IAAI,CAAC,YAAY,EAAE,CAAC;YACnE,kEAAkE;YAClE,2DAA2D;YAC3D,IAAI,IAAI,CAAC,aAAa,EAAE,EAAE,CAAC;gBACzB,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC1C,CAAC;YACD,IAAI,CAAC,aAAa,EAAE,CAAC;QACvB,CAAC;QAED,IAAI,CAAC,YAAY,GAAG,UAAU,CAAC;QAC/B,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACzB,IAAI,CAAC,YAAY,IAAI,SAAS,CAAC;QAC/B,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,MAAM;QACJ,IAAI,SAAS,GAA0B,IAAI,CAAC;QAC5C,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,YAAY,KAAK,IAAI,EAAE,CAAC;YAC1D,IAAI,IAAI,CAAC,aAAa,EAAE,EAAE,CAAC;gBACzB,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC1C,CAAC;YACD,IAAI,CAAC,aAAa,EAAE,CAAC;QACvB,CAAC;QACD,OAAO;YACL,SAAS;YACT,QAAQ,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,SAAS,EAAE,IAAI,CAAC,SAAS,EAAE;SAClE,CAAC;IACJ,CAAC;IAEO,aAAa;QACnB,OAAO,CACL,IAAI,CAAC,gBAAgB,GAAG,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,SAAS;YAC1D,IAAI,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAC5B,CAAC;IACJ,CAAC;IAEO,aAAa;QACnB,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,CAAC;QACvC,IAAI,CAAC,gBAAgB,IAAI,IAAI,CAAC,YAAY,CAAC;QAC3C,IAAI,CAAC,OAAO,GAAG,EAAE,CAAC;QAClB,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC;IACxB,CAAC;IAED,yDAAyD;IACjD,GAAG,CAAC,EAAU;QACpB,MAAM,KAAK,GAAmB,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,IAAI,EAAE,IAAI,CAAC,SAAS,EAAE,EAAE,EAAE,CAAC;QACrF,IAAI,CAAC,SAAS,GAAG,EAAE,CAAC;QACpB,IAAI,CAAC,WAAW,GAAG,EAAE,CAAC;QACtB,IAAI,CAAC,gBAAgB,GAAG,CAAC,CAAC;QAC1B,OAAO,KAAK,CAAC;IACf,CAAC;CACF"}
+{"version":3,"file":"accumulator.js","sourceRoot":"","sources":["../../src/chunk-builder/accumulator.ts"],"names":[],"mappings":"AAKA,qEAAqE;AACrE,+EAA+E;AAC/E,0EAA0E;AAC1E,+EAA+E;AAC/E,0DAA0D;AAC1D,EAAE;AACF,6EAA6E;AAC7E,0EAA0E;AAC1E,+EAA+E;AAC/E,2DAA2D;AAC3D,MAAM,OAAO,gBAAgB;IAQR;IACT;IARF,WAAW,GAAsB,EAAE,CAAC;IACpC,gBAAgB,GAAG,CAAC,CAAC;IACrB,OAAO,GAAsB,EAAE,CAAC;IAChC,YAAY,GAAG,CAAC,CAAC;IACjB,YAAY,GAAkB,IAAI,CAAC;IAE3C,YACmB,SAAiB,EAC1B,SAAiB;QADR,cAAS,GAAT,SAAS,CAAQ;QAC1B,cAAS,GAAT,SAAS,CAAQ;IACxB,CAAC;IAEJ,GAAG,CAAC,KAAsB;QACxB,MAAM,UAAU,GAAG,MAAM,CAAC,KAAK,CAAC,WAAW,CAAC,CAAC;QAC7C,MAAM,SAAS,GAAG,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,SAAS,CAAC,KAAK,CAAC,GAAG,IAAI,EAAE,MAAM,CAAC,CAAC;QAC1E,IAAI,SAAS,GAA0B,IAAI,CAAC;QAE5C,IAAI,IAAI,CAAC,YAAY,KAAK,IAAI,IAAI,UAAU,KAAK,IAAI,CAAC,YAAY,EAAE,CAAC;YACnE,kEAAkE;YAClE,2DAA2D;YAC3D,IAAI,IAAI,CAAC,aAAa,EAAE,EAAE,CAAC;gBACzB,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC1C,CAAC;YACD,IAAI,CAAC,aAAa,EAAE,CAAC;QACvB,CAAC;QAED,IAAI,CAAC,YAAY,GAAG,UAAU,CAAC;QAC/B,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC;QACzB,IAAI,CAAC,YAAY,IAAI,SAAS,CAAC;QAC/B,OAAO,SAAS,CAAC;IACnB,CAAC;IAED,MAAM;QACJ,IAAI,SAAS,GAA0B,IAAI,CAAC;QAC5C,IAAI,IAAI,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,IAAI,IAAI,CAAC,YAAY,KAAK,IAAI,EAAE,CAAC;YAC1D,IAAI,IAAI,CAAC,aAAa,EAAE,EAAE,CAAC;gBACzB,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC;YAC1C,CAAC;YACD,IAAI,CAAC,aAAa,EAAE,CAAC;QACvB,CAAC;QACD,OAAO;YACL,SAAS;YACT,QAAQ,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,SAAS,EAAE,IAAI,CAAC,SAAS,EAAE;SAClE,CAAC;IACJ,CAAC;IAEO,aAAa;QACnB,OAAO,CACL,IAAI,CAAC,gBAAgB,GAAG,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC,SAAS;YAC1D,IAAI,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,CAC5B,CAAC;IACJ,CAAC;IAEO,aAAa;QACnB,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,CAAC;QACvC,IAAI,CAAC,gBAAgB,IAAI,IAAI,CAAC,YAAY,CAAC;QAC3C,IAAI,CAAC,OAAO,GAAG,EAAE,CAAC;QAClB,IAAI,CAAC,YAAY,GAAG,CAAC,CAAC;IACxB,CAAC;IAED,yDAAyD;IACjD,GAAG,CAAC,EAAU;QACpB,MAAM,KAAK,GAAmB,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,IAAI,EAAE,IAAI,CAAC,SAAS,EAAE,EAAE,EAAE,CAAC;QACrF,IAAI,CAAC,SAAS,GAAG,EAAE,CAAC;QACpB,IAAI,CAAC,WAAW,GAAG,EAAE,CAAC;QACtB,IAAI,CAAC,gBAAgB,GAAG,CAAC,CAAC;QAC1B,OAAO,KAAK,CAAC;IACf,CAAC;CACF"}

package/dist/chunk-builder/archive.d.ts

@@ -1,18 +1,18 @@
-import type { CanonicalEvent } from "../scraper/normalize.js";
+import type { CanonicalRecord } from "@saga-sync/core";
 import type { Store } from "@saga-sync/core";
 import type { ChunkMeta } from "@saga-sync/core";
 export type Range = {
     from: bigint;
     to: bigint;
 };
-export declare function buildJsonl(events: CanonicalEvent[]): Buffer;
+export declare function buildJsonl(events: CanonicalRecord[]): Buffer;
 export declare class ChunkArchive {
     private readonly store;
     constructor(store: Store);
-    seal(protocolId: string, events: CanonicalEvent[], range: Range): Promise<ChunkMeta>;
-    writeHotHead(protocolId: string, events: CanonicalEvent[], range: Range): Promise<ChunkMeta>;
+    seal(protocolId: string, events: CanonicalRecord[], range: Range): Promise<ChunkMeta>;
+    writeHotHead(protocolId: string, events: CanonicalRecord[], range: Range): Promise<ChunkMeta>;
     private write;
-    readEvents(meta: ChunkMeta): Promise<CanonicalEvent[]>;
+    readEvents(meta: ChunkMeta): Promise<CanonicalRecord[]>;
     delete(meta: ChunkMeta): Promise<void>;
 }
 //# sourceMappingURL=archive.d.ts.map

package/dist/chunk-builder/archive.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"archive.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/archive.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,yBAAyB,CAAC;AAC9D,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,iBAAiB,CAAC;AAC7C,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAEjD,MAAM,MAAM,KAAK,GAAG;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAAC;AAKjD,wBAAgB,UAAU,CAAC,MAAM,EAAE,cAAc,EAAE,GAAG,MAAM,CAG3D;AAUD,qBAAa,YAAY;IACX,OAAO,CAAC,QAAQ,CAAC,KAAK;gBAAL,KAAK,EAAE,KAAK;IAGzC,IAAI,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,EAAE,cAAc,EAAE,EAAE,KAAK,EAAE,KAAK,GAAG,OAAO,CAAC,SAAS,CAAC;IAKpF,YAAY,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,EAAE,cAAc,EAAE,EAAE,KAAK,EAAE,KAAK,GAAG,OAAO,CAAC,SAAS,CAAC;YAI9E,KAAK;IA4Bb,UAAU,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,cAAc,EAAE,CAAC;IAWtD,MAAM,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;CAG7C"}
+{"version":3,"file":"archive.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/archive.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAC;AACvD,OAAO,KAAK,EAAE,KAAK,EAAE,MAAM,iBAAiB,CAAC;AAC7C,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAEjD,MAAM,MAAM,KAAK,GAAG;IAAE,IAAI,EAAE,MAAM,CAAC;IAAC,EAAE,EAAE,MAAM,CAAA;CAAE,CAAC;AAKjD,wBAAgB,UAAU,CAAC,MAAM,EAAE,eAAe,EAAE,GAAG,MAAM,CAG5D;AAUD,qBAAa,YAAY;IACX,OAAO,CAAC,QAAQ,CAAC,KAAK;gBAAL,KAAK,EAAE,KAAK;IAGzC,IAAI,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,KAAK,EAAE,KAAK,GAAG,OAAO,CAAC,SAAS,CAAC;IAKrF,YAAY,CAAC,UAAU,EAAE,MAAM,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,KAAK,EAAE,KAAK,GAAG,OAAO,CAAC,SAAS,CAAC;YAI/E,KAAK;IA4Bb,UAAU,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,eAAe,EAAE,CAAC;IAWvD,MAAM,CAAC,IAAI,EAAE,SAAS,GAAG,OAAO,CAAC,IAAI,CAAC;CAG7C"}

package/dist/chunk-builder/archive.js

@@ -1,7 +1,7 @@
 import { gzipSync, gunzipSync } from "node:zlib";
 import { numberToHex } from "viem";
 import { sha256Hex } from "@saga-sync/core";
-// Build the JSONL bytes for a chunk: one CanonicalEvent per line, trailing
+// Build the JSONL bytes for a chunk: one CanonicalRecord per line, trailing
 // newline. Empty events list produces zero bytes so an empty chunk file is
 // genuinely empty rather than a single blank line.
 export function buildJsonl(events) {

package/dist/chunk-builder/archive.js.map

@@ -1 +1 @@
-{"version":3,"file":"archive.js","sourceRoot":"","sources":["../../src/chunk-builder/archive.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AACjD,OAAO,EAAE,WAAW,EAAE,MAAM,MAAM,CAAC;AACnC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAO5C,2EAA2E;AAC3E,2EAA2E;AAC3E,mDAAmD;AACnD,MAAM,UAAU,UAAU,CAAC,MAAwB;IACjD,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IAChD,OAAO,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,IAAI,EAAE,MAAM,CAAC,CAAC;AACrF,CAAC;AAED,8EAA8E;AAC9E,gFAAgF;AAChF,iFAAiF;AACjF,oEAAoE;AACpE,2EAA2E;AAC3E,EAAE;AACF,6EAA6E;AAC7E,oCAAoC;AACpC,MAAM,OAAO,YAAY;IACM;IAA7B,YAA6B,KAAY;QAAZ,UAAK,GAAL,KAAK,CAAO;IAAG,CAAC;IAE7C,2BAA2B;IAC3B,IAAI,CAAC,UAAkB,EAAE,MAAwB,EAAE,KAAY;QAC7D,OAAO,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;IACtD,CAAC;IAED,yCAAyC;IACzC,YAAY,CAAC,UAAkB,EAAE,MAAwB,EAAE,KAAY;QACrE,OAAO,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,IAAI,CAAC,CAAC;IACrD,CAAC;IAEO,KAAK,CAAC,KAAK,CACjB,UAAkB,EAClB,MAAwB,EACxB,KAAY,EACZ,GAAY;QAEZ,MAAM,YAAY,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;QACxC,MAAM,MAAM,GAAG,SAAS,CAAC,YAAY,CAAC,CAAC;QACvC,MAAM,UAAU,GAAG,QAAQ,CAAC,YAAY,CAAC,CAAC;QAE1C,MAAM,OAAO,GAAG,WAAW,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACxC,MAAM,KAAK,GAAG,WAAW,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACpC,MAAM,IAAI,GAAG,GAAG,UAAU,KAAK,OAAO,IAAI,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC;QAEzF,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;QAEvC,OAAO;YACL,SAAS,EAAE,OAAO;YAClB,OAAO,EAAE,KAAK;YACd,IAAI;YACJ,IAAI,EAAE,WAAW,CAAC,UAAU,CAAC,MAAM,CAAC;YACpC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE;SACzC,CAAC;IACJ,CAAC;IAED,+EAA+E;IAC/E,2EAA2E;IAC3E,iDAAiD;IACjD,KAAK,CAAC,UAAU,CAAC,IAAe;QAC9B,MAAM,UAAU,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACnD,IAAI,CAAC,UAAU,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAC;QACtD,MAAM,YAAY,GAAG,UAAU,CAAC,UAAU,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;QAC7D,IAAI,YAAY,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAC;QACzC,OAAO,YAAY;aAChB,KAAK,CAAC,IAAI,CAAC;aACX,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC;aACjC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAmB,CAAC,CAAC;IACvD,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,IAAe;QAC1B,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACrC,CAAC;CACF"}
+{"version":3,"file":"archive.js","sourceRoot":"","sources":["../../src/chunk-builder/archive.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AACjD,OAAO,EAAE,WAAW,EAAE,MAAM,MAAM,CAAC;AACnC,OAAO,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAO5C,4EAA4E;AAC5E,2EAA2E;AAC3E,mDAAmD;AACnD,MAAM,UAAU,UAAU,CAAC,MAAyB;IAClD,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,MAAM,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC;IAChD,OAAO,MAAM,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,IAAI,EAAE,MAAM,CAAC,CAAC;AACrF,CAAC;AAED,8EAA8E;AAC9E,gFAAgF;AAChF,iFAAiF;AACjF,oEAAoE;AACpE,2EAA2E;AAC3E,EAAE;AACF,6EAA6E;AAC7E,oCAAoC;AACpC,MAAM,OAAO,YAAY;IACM;IAA7B,YAA6B,KAAY;QAAZ,UAAK,GAAL,KAAK,CAAO;IAAG,CAAC;IAE7C,2BAA2B;IAC3B,IAAI,CAAC,UAAkB,EAAE,MAAyB,EAAE,KAAY;QAC9D,OAAO,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;IACtD,CAAC;IAED,yCAAyC;IACzC,YAAY,CAAC,UAAkB,EAAE,MAAyB,EAAE,KAAY;QACtE,OAAO,IAAI,CAAC,KAAK,CAAC,UAAU,EAAE,MAAM,EAAE,KAAK,EAAE,IAAI,CAAC,CAAC;IACrD,CAAC;IAEO,KAAK,CAAC,KAAK,CACjB,UAAkB,EAClB,MAAyB,EACzB,KAAY,EACZ,GAAY;QAEZ,MAAM,YAAY,GAAG,UAAU,CAAC,MAAM,CAAC,CAAC;QACxC,MAAM,MAAM,GAAG,SAAS,CAAC,YAAY,CAAC,CAAC;QACvC,MAAM,UAAU,GAAG,QAAQ,CAAC,YAAY,CAAC,CAAC;QAE1C,MAAM,OAAO,GAAG,WAAW,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACxC,MAAM,KAAK,GAAG,WAAW,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;QACpC,MAAM,IAAI,GAAG,GAAG,UAAU,KAAK,OAAO,IAAI,KAAK,IAAI,GAAG,CAAC,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC;QAEzF,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,EAAE,UAAU,CAAC,CAAC;QAEvC,OAAO;YACL,SAAS,EAAE,OAAO;YAClB,OAAO,EAAE,KAAK;YACd,IAAI;YACJ,IAAI,EAAE,WAAW,CAAC,UAAU,CAAC,MAAM,CAAC;YACpC,MAAM,EAAE,EAAE,IAAI,EAAE,QAAQ,EAAE,IAAI,EAAE,MAAM,EAAE;SACzC,CAAC;IACJ,CAAC;IAED,+EAA+E;IAC/E,2EAA2E;IAC3E,iDAAiD;IACjD,KAAK,CAAC,UAAU,CAAC,IAAe;QAC9B,MAAM,UAAU,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACnD,IAAI,CAAC,UAAU,IAAI,UAAU,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAC;QACtD,MAAM,YAAY,GAAG,UAAU,CAAC,UAAU,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;QAC7D,IAAI,YAAY,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,EAAE,CAAC;QACzC,OAAO,YAAY;aAChB,KAAK,CAAC,IAAI,CAAC;aACX,MAAM,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,MAAM,GAAG,CAAC,CAAC;aACjC,GAAG,CAAC,CAAC,IAAI,EAAE,EAAE,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,CAAoB,CAAC,CAAC;IACxD,CAAC;IAED,KAAK,CAAC,MAAM,CAAC,IAAe;QAC1B,MAAM,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACrC,CAAC;CACF"}

package/dist/chunk-builder/cli.d.ts

@@ -1,7 +1,8 @@
 #!/usr/bin/env node
-import type { CanonicalEvent } from "../scraper/normalize.js";
+import type { CanonicalRecord } from "@saga-sync/core";
 import { ChunkArchive } from "./archive.js";
 import { Manifest } from "@saga-sync/core";
+import "@saga-sync/core/secp256k1";
 import type { ChunkMeta } from "@saga-sync/core";
 export type ProcessArgs = {
     protocolId: string;
@@ -11,7 +12,7 @@ export type ProcessArgs = {
     archive: ChunkArchive;
     manifest: Manifest;
     seed?: {
-        events: CanonicalEvent[];
+        events: CanonicalRecord[];
         chunkFrom: bigint;
     };
     trailingMode?: "seal" | "suspend";
@@ -19,7 +20,7 @@ export type ProcessArgs = {
 export type ProcessResult = {
     sealed: ChunkMeta[];
     trailing?: {
-        events: CanonicalEvent[];
+        events: CanonicalRecord[];
         fromBlock: bigint;
         toBlock: bigint;
     };

package/dist/chunk-builder/cli.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/cli.ts"],"names":[],"mappings":";AAMA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,yBAAyB,CAAC;AAE9D,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAG5C,OAAO,EAAE,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAE3C,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAuEjD,MAAM,MAAM,WAAW,GAAG;IACxB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,YAAY,CAAC;IACtB,QAAQ,EAAE,QAAQ,CAAC;IAKnB,IAAI,CAAC,EAAE;QAAE,MAAM,EAAE,cAAc,EAAE,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAE,CAAC;IAGvD,YAAY,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACnC,CAAC;AAEF,MAAM,MAAM,aAAa,GAAG;IAC1B,MAAM,EAAE,SAAS,EAAE,CAAC;IAEpB,QAAQ,CAAC,EAAE;QAAE,MAAM,EAAE,cAAc,EAAE,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC;CAC7E,CAAC;AAMF,wBAAsB,aAAa,CACjC,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,EAC5B,IAAI,EAAE,WAAW,GAChB,OAAO,CAAC,aAAa,CAAC,CA+DxB"}
+{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../../src/chunk-builder/cli.ts"],"names":[],"mappings":";AAMA,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,iBAAiB,CAAC;AAEvD,OAAO,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAG5C,OAAO,EAAE,QAAQ,EAAE,MAAM,iBAAiB,CAAC;AAG3C,OAAO,2BAA2B,CAAC;AACnC,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,iBAAiB,CAAC;AAuEjD,MAAM,MAAM,WAAW,GAAG;IACxB,UAAU,EAAE,MAAM,CAAC;IACnB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,MAAM,CAAC;IAChB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,EAAE,YAAY,CAAC;IACtB,QAAQ,EAAE,QAAQ,CAAC;IAKnB,IAAI,CAAC,EAAE;QAAE,MAAM,EAAE,eAAe,EAAE,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAE,CAAC;IAGxD,YAAY,CAAC,EAAE,MAAM,GAAG,SAAS,CAAC;CACnC,CAAC;AAEF,MAAM,MAAM,aAAa,GAAG;IAC1B,MAAM,EAAE,SAAS,EAAE,CAAC;IAEpB,QAAQ,CAAC,EAAE;QAAE,MAAM,EAAE,eAAe,EAAE,CAAC;QAAC,SAAS,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAA;KAAE,CAAC;CAC9E,CAAC;AAMF,wBAAsB,aAAa,CACjC,KAAK,EAAE,aAAa,CAAC,MAAM,CAAC,EAC5B,IAAI,EAAE,WAAW,GAChB,OAAO,CAAC,aAAa,CAAC,CA+DxB"}

package/dist/chunk-builder/cli.js

@@ -8,7 +8,9 @@ import { createStore, parseStoreTarget } from "../storage/index.js";
 import { ChunkArchive } from "./archive.js";
 import { ChunkAccumulator } from "./accumulator.js";
 import { Manifest } from "@saga-sync/core";
-import { signerFromEnv } from "@saga-sync/core";
+import { signersFromEnv } from "@saga-sync/core";
+// Registers secp256k1 so MANIFEST_SIGNING_KEY_SECP256K1 is honoured.
+import "@saga-sync/core/secp256k1";
 const DEFAULT_SIZE_LIMIT = 10 * 1024 * 1024; // 10 MiB
 const USAGE = `chunk-builder — partition scraper NDJSON into immutable .jsonl.gz chunks
 
@@ -70,7 +72,7 @@ function parseCliArgs() {
         dryRun: values["dry-run"] ?? false,
     };
 }
-// Reads CanonicalEvent NDJSON, partitions it into chunks via ChunkAccumulator,
+// Reads CanonicalRecord NDJSON, partitions it into chunks via ChunkAccumulator,
 // and seals each completed chunk through ChunkArchive + Manifest. The trailing
 // accumulator is either sealed (seal mode) or returned (suspend mode — the
 // hot-head carry-over path).
@@ -137,7 +139,7 @@ async function main() {
     const args = parseCliArgs();
     const store = createStore({ ...parseStoreTarget(args.output), dryRun: args.dryRun });
     const archive = new ChunkArchive(store);
-    const manifest = await Manifest.load(store, undefined, { signer: signerFromEnv() });
+    const manifest = await Manifest.load(store, undefined, { signers: signersFromEnv() });
     const rl = createInterface({ input: process.stdin, crlfDelay: Infinity });
     const { sealed } = await processStream(rl, {
         protocolId: args.protocolId,
@@ -147,6 +149,8 @@ async function main() {
         archive,
         manifest,
     });
+    // Manifest mutations only schedule throttled writes; flush to make them durable.
+    await manifest.flush();
     process.stderr.write(`chunk-builder: ${sealed.length} chunk(s) for ${args.protocolId} ` +
         `[${numberToHex(args.fromBlock)}, ${numberToHex(args.toBlock)})` +
         (args.dryRun ? " (dry-run: no files or manifest written)\n" : "\n"));