@aztec/prover-node 0.0.1-commit.993d52e → 0.0.1-commit.9a89641

package/README.md

@@ -1 +1,573 @@
 # Prover Node
+
+The prover node turns sequenced checkpoints into epoch proofs that get submitted to the L1
+rollup contract.  It runs alongside an Aztec validator/full-node and consumes the
+canonical chain view those nodes emit, proving epochs **optimistically** — sub-tree
+work begins the moment a checkpoint lands on L1, not when the epoch closes.
+
+This document describes the internal architecture: the state held by the prover-node,
+the events that drive it, and the data flow from a fresh `chain-checkpointed` event
+through to a `submitEpochRootProof` on L1.
+
+## Contents
+
+1. [Architecture](#architecture)
+2. [CheckpointProver lifecycle](#checkpointprover-lifecycle)
+3. [EpochSession lifecycle](#epochsession-lifecycle)
+4. [Event flow](#event-flow)
+5. [Walkthroughs](#walkthroughs)
+6. [Design rationale](#design-rationale)
+7. [Configuration](#configuration)
+8. [Failure handling and observability](#failure-handling-and-observability)
+
+## Architecture
+
+```mermaid
+flowchart TB
+  L2BlockStream -->|chain-checkpointed| ProverNode
+  L2BlockStream -->|chain-pruned| ProverNode
+  L2BlockStream -->|chain-proven| ProverNode
+  L2BlockStream -->|any event| ProverNode
+  ProverNode --> CheckpointStore
+  ProverNode --> ChonkCache
+  ProverNode --> SessionManager
+  ProverNode --> ProofPublishingService
+  SessionManager --> EpochTicker[(periodic tick)]
+  SessionManager --> FullSessions[(fullSessions)]
+  SessionManager --> PartialSessions[(partialSessions)]
+  FullSessions -.referenced checkpoints.-> CheckpointStore
+  PartialSessions -.referenced checkpoints.-> CheckpointStore
+  FullSessions --> TopTreeJob
+  PartialSessions --> TopTreeJob
+  TopTreeJob -->|PublishCandidate| ProofPublishingService
+  ProofPublishingService -->|fresh per publish| ProverNodePublisher
+  ProverNodePublisher --> L1[L1 Rollup]
+```
+
+The prover-node splits responsibility between four classes:
+
+- **`ProverNode`** — owns the long-lived collections, wires the L2BlockStream, and
+  translates each chain event into a single method call on the `SessionManager` or
+  `ProofPublishingService`. It also performs side effects that don't belong on an
+  `EpochSession`: registering new checkpoints with the store per event, and sweeping expired
+  epochs out of the cache and the store on a periodic ticker (see below). It uploads post-mortem
+  snapshots on failure at two levels: per checkpoint when a `CheckpointProver` fails (its `onFailed`
+  callback), and per session when a full session fails on its own account (`onSessionFailed` /
+  `EpochSession.hasFailed()`).
+- **`CheckpointStore`** — a registry of `CheckpointProver` instances keyed by
+  `(checkpointNumber, slot, archiveRoot)`. Each `CheckpointProver` runs its own sub-tree pipeline
+  (tx gather → block processing → block-rollup proofs), starting eagerly the moment a
+  checkpoint is registered. The store holds the canonical checkpoint content that
+  `EpochSession`s query when assembling their subsets; a prune cancels and removes the affected
+  provers, so every prover in the store is canonical.
+- **`SessionManager`** — owns every live `EpochSession`, the serial reconcile queue,
+  the periodic tick, and all `EpochSession` lifecycle decisions. `ProverNode` calls into it
+  via `onCheckpointAdded`, `onPrune`, and `startProof`. Every trigger it receives is
+  translated into a `reconcile(trigger)` call, a single idempotent function that walks
+  all `EpochSession`s, cancels any whose canonical content has shifted, re-creates them with
+  the new content, and opens fresh full `EpochSession`s for any epoch that has become provable.
+  It never builds a session over a **failed** `CheckpointProver` (one whose block proofs rejected —
+  a sub-tree fault or a prune-induced fork fault): a session over it could only fail, so the epoch is
+  cheaply skipped until a prune/re-add installs a fresh prover in its place, at which point a rebuilt
+  session reuses every already-completed sub-proof from the content-addressed broker. Reconcile runs on
+  a `SerialQueue` (from `@aztec/foundation/queue`), so two concurrent triggers can never interleave on an
+  `await` and race on the `EpochSession` maps.
+- **`ProofPublishingService`** — central owner of L1 proof submission. `EpochSession`s hand
+  their top-tree proofs to the service as `PublishCandidate`s; the service serialises
+  one publish at a time against a freshly-created `ProverNodePublisher`, gates eligibility
+  on the proven block tip, picks the longest candidate per epoch as the winner
+  (others resolve `'superseded'`), and enforces a per-candidate `deadline`. It runs its
+  own `drain()` on a separate `SerialQueue`: submits, withdrawals, chain-proven advances,
+  and per-candidate deadline expiries all enqueue a drain pass, so the eligibility
+  re-check and the L1 publish never interleave with each other.
+
+## CheckpointProver lifecycle
+
+A `CheckpointProver` is content-addressed by `(checkpoint.number, slot, archiveRoot)`,
+where `archiveRoot` is the checkpoint's own archive root (its post-state). Keying on the
+post-state makes the identity precise: two checkpoints are "the same" iff they produce
+the same archive — so a reorg branch, or a replacement built on the same predecessor but
+with different content, yields a different archive root and a distinct `CheckpointProver`. A prune
+cancels and removes a prover — its sub-tree work forks world-state per block and cannot survive the
+prune — so a re-add, even of identical content, constructs a fresh `CheckpointProver` (block-rollup
+jobs already completed in the proving broker are still reused, as they are content-addressed).
+
+```mermaid
+stateDiagram-v2
+  [*] --> Created
+  Created --> Proving: gather + execute
+  Proving --> Proven: sub-tree resolves blockProofs
+  Proving --> Cancelled: cancel() (prune / shutdown / epoch-session error)
+  Proven --> Cancelled: cancel() (prune / reap)
+  Proven --> Reaped: reapExpired(epoch)
+  Cancelled --> [*]
+```
+
+A prune **cancels and removes** the affected `CheckpointProver`s from the store: a prover's
+sub-tree work forks world-state per block, and an L1 prune of a base block faults those reads,
+so there is nothing to preserve across a reorg. A re-add — even of identical content — therefore
+constructs a fresh `CheckpointProver`. `EpochSession`s ask the store for the checkpoints in a slot
+range; every prover in the store is canonical.
+
+### Removal rules
+
+- **Pruned**: `CheckpointStore.cancelAndRemoveAboveBlock(target)` cancels and removes every
+  `CheckpointProver` whose last block is above the prune target, the moment a `chain-pruned`
+  event arrives.
+- **Expired**: `CheckpointStore.reapExpired(expiredEpoch)` drops any `CheckpointProver` whose
+  epoch is at or below the supplied expired epoch. Once an epoch's proof-submission window has
+  closed, its proof can no longer be accepted on L1, so the `CheckpointProver` is no longer needed.
+- **Shutdown**: `CheckpointStore.stop()` cancels every remaining prover and awaits its teardown
+  (including teardowns still in flight for provers already removed by a prune or reap).
+
+### Eager tx gathering
+
+A `CheckpointProver` starts its tx gather + sub-tree pipeline **in its constructor**.
+The tx provider is injected as a dependency,
+and the `CheckpointProver` pulls its own txs via `txProvider.getTxsForBlock(block)` for each
+block in its checkpoint.
+
+This means the moment a checkpoint lands on L1, sub-tree proving is already in flight.
+By the time the epoch closes (and the `EpochSession` is constructed), most or all of the
+block-rollup proofs are already done — the `EpochSession` only has to drive the top tree.
+
+## EpochSession lifecycle
+
+An `EpochSession` is identified by a slot-based **spec**:
+
+```ts
+interface SessionSpec {
+  kind: 'full' | 'partial';
+  epochNumber: EpochNumber;
+  fromSlot: SlotNumber;
+  toSlot: SlotNumber;
+}
+```
+
+The spec declares *what to prove* (a slot range).  The concrete checkpoint set the
+`EpochSession` holds is the *implementation* of the spec — frozen at construction time,
+derived from the canonical content for that slot range.
+
+```mermaid
+stateDiagram-v2
+  [*] --> initialized
+  initialized --> awaiting_checkpoints: start()
+  awaiting_checkpoints --> awaiting_root: sub-tree proofs ready, top-tree prove begins
+  awaiting_root --> publishing_proof: epoch proof ready, submit to L1
+  publishing_proof --> completed: publish succeeds
+  publishing_proof --> superseded: longer same-epoch candidate wins
+  publishing_proof --> failed: L1 submission errored (provers healthy)
+  awaiting_checkpoints --> stopped: a checkpoint prover failed
+  awaiting_root --> failed: top-tree prove errored (provers healthy)
+  initialized --> timed_out: deadline
+  awaiting_checkpoints --> timed_out: deadline (EpochSession or candidate)
+  awaiting_root --> timed_out: deadline (EpochSession or candidate)
+  awaiting_checkpoints --> cancelled: cancel()
+  awaiting_root --> cancelled: cancel()
+  completed --> [*]
+  superseded --> [*]
+  cancelled --> [*]
+  timed_out --> [*]
+  stopped --> [*]
+  failed --> [*]
+```
+
+A fault settles the `EpochSession` in one of two terminal states:
+
+- **`stopped`** — a `CheckpointProver` under the session failed (its block proofs rejected). This may
+  be a prune-induced fork fault, so it is *not* a verdict on the epoch and is *not* uploaded. The
+  reconciler drops the session and, guarded by `isFailed()`, does not rebuild over the failed prover;
+  a prune/re-add installs a fresh prover and the epoch is retried.
+- **`failed`** — the session's own work (top-tree prove or L1 submit) failed while *every* checkpoint
+  prover succeeded. Because healthy provers rule out a prune, this is a genuine, race-free failure
+  (`EpochSession.hasFailed()`). The reconciler retains such a full session (so the tick does not
+  re-prove a deterministically-failing epoch) and uploads a post-mortem exactly once. No prune/fault
+  classification is needed — the two states carry it.
+
+The non-terminal states track the window between `start()` and the L1 submission:
+
+- `awaiting-checkpoints` — a `TopTreeJob` is awaiting each checkpoint's sub-tree result
+  (`CheckpointProver.whenBlockProofsReady`) before the top-tree prove can begin.
+- `awaiting-root` — the sub-tree proofs are ready and the top-tree (root) prove is running,
+  assembling the epoch proof (set via the `TopTreeJob` `beforeProve` hook).
+- `publishing-proof` — the epoch proof is being submitted to L1 via `ProofPublishingService`.
+
+`cancel()` and the deadline can fire during any of these pre-submit phases; a terminal state
+set that way wins over the phase transitions (which are guarded by `isTerminal()`).
+
+The `EpochSession` does three sequential things: (1) run a `TopTreeJob` over the frozen
+checkpoint subset, (2) hand the resulting proof to `ProofPublishingService` as a
+`PublishCandidate`, (3) translate the service's outcome into a terminal state.
+Predecessor gating, same-epoch dedup, deadline enforcement, and the L1 tx are all
+the `ProofPublishingService`'s concern; the `EpochSession` is just the producer of one
+candidate and the observer of its outcome.
+
+Outcome → state mapping:
+
+| `PublishOutcome` | `EpochSession` state |
+|---|---|
+| `published` | `completed` |
+| `superseded` | `superseded` |
+| `failed` | `failed` |
+| `expired` | `timed-out` |
+| `withdrawn` | `cancelled` |
+
+There is a single deadline — the proof submission window — that applies across both
+proving and publishing. Before submission, the `EpochSession` arms its own timer against
+it: if proving doesn't finish in time, the `EpochSession` enters `timed-out` via
+`cancel('deadline')`. After submission, the publishing service enforces the same deadline
+on the candidate. It's the same instant throughout; only which component enforces it
+changes once the candidate has been handed off.
+
+### Full vs partial
+
+Every `EpochSession` — full or partial — has `fromSlot = firstSlotOfEpoch(N)`. The L1 rollup
+contract requires every proof to extend from the previous epoch's proven tip, so
+there's no value in starting later than the epoch boundary. The two kinds differ
+only in `toSlot` and in how the publishing service treats their candidate:
+
+- **Full** `EpochSession`s are opened by reconcile when the epoch is complete on L1 *and*
+  every archiver-reported checkpoint is present in the store. Their `toSlot` is
+  the epoch's last slot. The publishing service never auto-supersedes a `full`
+  candidate on proven-tip subsumption — the L1 contract records a `(epoch, prover-id)`
+  submission for every full-epoch proof, so even after another prover-node has
+  landed first, this prover's submission is still worthwhile.
+- **Partial** `EpochSession`s are constructed by an explicit `startProof(epochNumber)` API
+  call. Their `toSlot` is the last canonical slot present at request time, which may
+  be earlier than the epoch's last slot. Partial candidates are an early-finish
+  optimisation: if the proven chain has caught up to or past `endBlock` by the time
+  the publishing service picks the winner, the partial resolves `'superseded'`
+  without spending L1 gas. Dedup: if the partial's spec collapses to the full's spec
+  (canonical content already covers the whole epoch), `startProof` awaits the
+  existing full `EpochSession` instead of opening a duplicate.
+
+## ProofPublishingService
+
+The service is a single per-prover-node owner of L1 submission. `EpochSession`s call
+`submit(candidate)` and await one of five outcomes:
+
+| Outcome | Meaning |
+|---|---|
+| `published` | L1 accepted the proof. |
+| `superseded` | A longer same-epoch candidate won, or (for `partial` candidates) the proven tip has caught up to `endBlock`. |
+| `failed` | L1 submission errored. |
+| `expired` | The candidate's `deadline` elapsed before publishing started. |
+| `withdrawn` | An `EpochSession` called `withdraw(uuid)` on a still-queued candidate. |
+
+Key invariants:
+
+- **One publish at a time** via a `SerialQueue` drain.
+- **Fresh publisher per publish.** Each drain call constructs a new `ProverNodePublisher`
+  via the factory. There is no shared in-memory state across publishes.
+- **Once an L1 publish starts, it runs to completion.** `withdraw` is a queue-only
+  operation: it removes a candidate that hasn't started publishing. An in-flight
+  candidate is left alone and its outcome reports whatever L1 returned. The
+  originating `EpochSession` has already moved to a terminal state via `cancel()` and
+  ignores the late outcome.
+- **Drain reads the proven block number afresh** from `l2BlockSource` inside the
+  serial queue, so the eligibility
+  check is consistent with the publish that follows it on the same drain pass.
+- **Per-candidate `deadline`** arms a `setTimeout` (against the injected `DateProvider`).
+  When it fires, a still-queued candidate resolves `'expired'`. An in-flight publish
+  is left alone (its outcome reports the natural L1 result).
+- **Transient `publisherFactory.create()` failures are retried.** Instead of resolving
+  the candidate as `'failed'`, the service schedules another drain after a 1s backoff
+  and leaves the candidate in the queue. The candidate's `deadline` caps the total
+  retry window — persistent acquire failure resolves as `'expired'`.
+
+### Eligibility
+
+A candidate is eligible to publish when its **predecessor block is proven**
+(`startBlock - 1 <= proven`). Among eligible candidates for the same epoch, the
+one with the **highest `endBlock`** wins; the others resolve `'superseded'`.
+Partial candidates whose `endBlock <= proven` are dropped before this check
+(early-finish optimisation no longer helps); full candidates are never
+auto-superseded on the proven tip.
+
+## Event flow
+
+### chain-checkpointed
+
+```mermaid
+sequenceDiagram
+  participant L2 as L2BlockStream
+  participant PN as ProverNode
+  participant CS as CheckpointStore
+  participant CP as CheckpointProver
+  participant SM as SessionManager
+
+  L2->>PN: chain-checkpointed{checkpoint}
+  PN->>PN: collectRegisterData (prev-header, l1ToL2 messages, sibling path)
+  PN->>CS: addOrUpdate(checkpoint, data)
+  alt content key new
+    CS->>CP: new CheckpointProver(args)
+    CP->>CP: eager gather + sub-tree start
+  else content key already live
+    CS->>CP: reuse existing prover
+  end
+  PN->>SM: onCheckpointAdded(epoch)
+  SM->>SM: queue reconcile({kind:'checkpoint', epoch})
+  SM->>SM: walk EpochSessions, recreate invalid
+  SM->>SM: open full EpochSession if epoch ready
+```
+
+### chain-pruned
+
+```mermaid
+sequenceDiagram
+  participant L2 as L2BlockStream
+  participant PN as ProverNode
+  participant CS as CheckpointStore
+  participant SM as SessionManager
+
+  L2->>PN: chain-pruned{block}
+  PN->>CS: cancelAndRemoveAboveBlock(prunedToBlock)
+  CS->>CS: cancel + remove every CheckpointProver whose last block is above the target
+  PN->>SM: onPrune(affectedEpochs)
+  SM->>SM: queue reconcile({kind:'prune', affectedEpochs})
+  SM->>SM: walk EpochSessions, cancel-and-recreate those with shifted content
+```
+
+### chain-proven
+
+```mermaid
+sequenceDiagram
+  participant L2 as L2BlockStream
+  participant PN as ProverNode
+  participant PS as ProofPublishingService
+
+  L2->>PN: chain-proven{block}
+  PN->>PS: onChainProven(blockNumber)
+  PS->>PS: scheduleDrain (wake-up only, no state cached)
+  PS->>PS: drain reads proven afresh, re-checks eligibility
+```
+
+### Periodic expiry sweep
+
+```mermaid
+sequenceDiagram
+  participant T as expiryTicker (RunningPromise)
+  participant PN as ProverNode
+  participant L2 as L2BlockStream
+  participant CC as ChonkCache
+  participant CS as CheckpointStore
+
+  T->>PN: checkEpochExpiry() (every poll interval)
+  PN->>L2: getSyncedL2SlotNumber()
+  PN->>PN: latestEpoch = getEpochAtSlot(latestSlot)
+  PN->>PN: newlyExpiredUpTo = latestEpoch - (proofSubmissionEpochs + 1)
+  loop for each newly-expired epoch
+    PN->>L2: getBlocks({epoch, onlyCheckpointed})
+    PN->>CC: releaseForBlocks(blocks)
+    PN->>CS: reapExpired(epoch)
+  end
+```
+
+The sweep is driven solely by the `expiryTicker` (a `RunningPromise` armed in `start()`), which
+calls `checkEpochExpiry()` every poll interval whether or not block-stream events arrive. It is
+deliberately **not** run from `handleBlockStreamEvent` — expiry is a background sweep keyed off the
+archiver's synced slot, not part of event processing — and because a `RunningPromise` never overlaps
+its own runs, no two sweeps can race. An epoch `E` is expired once the chain reaches the start of epoch
+`E + proofSubmissionEpochs + 1` — the deadline beyond which an L1 submission for `E` would be rejected.
+Expiry only releases the chonk cache and reaps `E`'s provers.
+A monotonic high-water mark (`lastExpiredEpoch`) makes the sweep cheap: it only
+advances after a sweep completes and never revisits an epoch. It is seeded at `start()` from the last
+fully-proven epoch (`resolveLastFullyProvenEpoch`), so on a restart we never re-sweep epochs that already
+reached L1.
+
+### Periodic tick
+
+`SessionManager.start()` arms a `RunningPromise` that fires
+`reconcile({ kind: 'tick' })` every `tickIntervalMs`. The tick picks up epochs that
+became complete by time alone (no fresh checkpoint event) and advances to the
+next unproven epoch once the previous one lands on L1. The tick is not gated by any per-epoch
+bookkeeping; two things keep it from re-proving a doomed epoch:
+
+- `openFullSessionIfReady` refuses to build a session when any `CheckpointProver` in the set has
+  **failed** (`isFailed()`). So a prover fault (possibly a prune) skips the epoch cheaply each tick —
+  no session, no proving — until a prune/re-add installs a fresh prover.
+- A session that failed on its own account (`hasFailed()` — top-tree/submit failed with every prover
+  healthy) is **retained** by `recreateInvalidSessions` rather than deleted, so the tick's
+  `fullSessions.has(epoch)` check skips it. It is replaced only when its canonical content changes (a
+  re-add), so a deterministically-failing epoch is not re-proved every tick.
+
+Transient blockers (max-pending-jobs reached, archiver still indexing) create no session and no failed
+prover, so the next tick simply tries again.
+
+## Walkthroughs
+
+### checkpoint-added → prune → checkpoint-added (reorg resilience)
+
+State: epoch N has checkpoints c1..c4 (slots s1..s4).  `fullSessions[N]` holds `EpochSession`
+**A** with spec `{kind:'full', N, fromSlot:s1, toSlot:s4}`, referencing `[c1, c2, c3, c4]`.
+
+1. **chain-pruned arrives, target c3.**  The store cancels and removes c4 (its last block is
+   above the prune target).  Reconcile fires: `EpochSession` A's frozen set `[c1, c2, c3, c4]`
+   includes the now-cancelled c4, so it no longer matches the store's `[c1, c2, c3]` →
+   `A.cancel('canonical content changed')`.  Epoch N still complete on L1 → reconcile constructs
+   `EpochSession` **B** with the same spec `{full, N, s1, s4}` but checkpoints `[c1, c2, c3]`.
+
+2. **`EpochSession` B starts top-tree proving over [c1, c2, c3].**
+
+3. **chain-checkpointed arrives, target c4_re (same content key as old c4).**  The old c4
+   prover was removed on the prune, so the store constructs a **fresh** `CheckpointProver` for
+   c4_re and starts its sub-tree work from scratch.  (Its block-rollup jobs are content-addressed
+   in the proving broker, so any the old c4 already completed are reused rather than re-proved.)
+
+4. **Reconcile fires.**  `EpochSession` B's content for `(s1, s4)` is now `[c1, c2, c3, c4_re]`,
+   doesn't match its frozen `[c1, c2, c3]` → `B.cancel(...)`.  Construct `EpochSession` **C** with
+   the same spec but checkpoints `[c1, c2, c3, c4_re]`.
+
+5. **`EpochSession` C reuses the long-lived c1..c3 `CheckpointProver` instances and the fresh
+   c4_re.**  Only c4_re's witnesses are regenerated; the top-tree is recomputed.  The chonk cache
+   survived the reorg because no epoch in this range has expired yet.
+
+
+### Partial request dedups against a running full `EpochSession`
+
+The operator calls `startProof(N)` while the full `EpochSession` for epoch N is running with
+c1..c4.  Current canonical slot range is `(s1, s4)`, so the partial's computed spec is
+`{partial, N, s1, s4}` — its `fromSlot`/`toSlot` exactly match the running full `EpochSession`'s. `startProof`
+detects this and awaits the existing full instead of opening a duplicate: no partial
+`EpochSession` is created and no second `TopTreeJob` is built. The caller simply blocks on the
+full session's result and the epoch is proven once.
+
+### True partial proof
+
+The operator calls `startProof(N)` when only c1, c2 are canonical (epoch incomplete).
+`fromSlot` is the epoch's first slot; `toSlot` is `s2` (the last canonical slot).
+Partial `EpochSession` created with spec `{partial, N, firstSlotOfEpoch(N), s2}` and
+checkpoints `[c1, c2]`.
+
+When c3 later arrives in slot s3, the partial is **not** invalidated — c3's slot is
+outside its range. If c2 is then pruned, the partial **is** invalidated (canonical
+content for the same slot range is now just `[c1]`) and recreated with the same
+spec but checkpoints `[c1]`. If c2 re-adds, the partial is invalidated again and
+recreated with `[c1, c2]`.
+
+## Design rationale
+
+### Why slot-based specs (not checkpoint-based)?
+
+A spec like "prove checkpoints 7..10" is invalidated by any reorg that renumbers
+those checkpoints.  A spec like "prove slots 350..399" survives renumbering — the
+slot range is determined by epoch math and L1 constants, not by which checkpoints
+happen to be canonical at the moment.  Reconciliation preserves the slot range
+across cancel-and-recreate cycles.
+
+### Why does every `EpochSession` start at the epoch's first slot?
+
+The L1 rollup contract validates that every submitted proof extends from the previous
+proven tip — the `fromCheckpoint` of any submission must be the checkpoint immediately
+after the current L1 proven head. Starting a partial `EpochSession` at a later slot would
+mean the partial's `fromCheckpoint` lies past the proven tip, which the contract
+rejects. Fixing `fromSlot` to `firstSlotOfEpoch(N)` for both kinds means partials and
+fulls always share the same starting point; they differ only in `toSlot` and in the
+submission decision.
+
+### Why does a publishing service own L1 submission instead of the `EpochSession`?
+
+Concentrating L1 submission gives us three properties for free that were awkward
+or impossible when each `EpochSession` called the publisher directly:
+
+1. **Atomic same-epoch dedup.** Multiple candidates for the same epoch (full +
+   partial, or partial-then-full as canonical content extends) can be in flight
+   at once; the service picks the winner under the serial drain so only one L1
+   tx is ever sent for the longer candidate.
+2. **One source of truth for the proven tip.** Reading the proven block number
+   inside the drain means the eligibility check and the publish that follows are
+   guaranteed to use the same value. `EpochSession`s can't race each other on stale
+   reads.
+3. **Per-candidate deadline and retry.** The service owns expiry timers and the
+   `publisherFactory.create()` retry loop. `EpochSession`s don't need to know about
+   either — they just await the outcome.
+
+### Why is the chonk cache keyed by tx hash and released on finality?
+
+Chonk-verifier proofs are tx-scoped: they prove a transaction's chonk circuit is
+valid, independently of which block or epoch the tx lands in.  A tx that gets
+reorged out of one block and re-mined into another should not need to be re-proved.
+Keying by tx hash makes the cache survive any reorg up to finality; releasing on
+finality means we don't grow the cache indefinitely while still keeping every
+reorg-relevant proof.
+
+### Why is a pruned `CheckpointProver` removed rather than kept for a possible re-add?
+
+A prover's sub-tree work forks world-state per block, and an L1 prune of a base block faults
+those in-flight reads — so the work cannot survive the prune, and there is nothing to preserve
+by keeping the prover around.  Removing it on prune and rebuilding on re-add is therefore both
+correct and simpler; the expensive block-rollup proofs are still reused when content re-appears,
+because the proving broker is content-addressed independently of the prover's lifetime.
+
+### How is a genuine failure told apart from a prune, without a racy classification?
+
+Knowledge of an L1 reorg reaches the prover-node on two causally unordered channels: the control
+plane (`chain-pruned` → `onPrune`) and the data plane (world-state unwinds, and an in-flight fork
+read faults inside a `CheckpointProver` mid-proof). If the `EpochSession` reacted to a fault by asking
+"was this a prune or a genuine failure?", every answer would sample control-plane state that lags the
+data-plane unwind — racy by construction. We remove the question by pushing the fact down to where it
+is unambiguous: a `CheckpointProver` records `isFailed()` when *its* block proofs reject. The session
+then reads a clean signal: if any prover failed, the fault might be a prune (`stopped`) — don't upload,
+don't rebuild over it, recover on re-add; if *no* prover failed yet the session's own top-tree/submit
+work failed, that is definitively **not** a prune (`failed`) — a genuine, race-free failure that is
+retained (so a deterministic failure isn't re-proved every tick) and uploaded once. No control-plane
+sampling, no timing race — the two prover-level and session-level facts carry the whole decision.
+
+## Configuration
+
+| Env var | Description |
+|---|---|
+| `PROVER_NODE_POLLING_INTERVAL_MS` | Polling interval for the L2BlockStream and the SessionManager periodic tick.  Default 1000 ms. |
+| `PROVER_NODE_MAX_PENDING_JOBS` | Cap on the number of non-terminal `EpochSession`s (full + partial).  When at limit, reconcile defers opening new full `EpochSession`s; explicit `startProof` calls throw. |
+| `PROVER_NODE_EPOCH_PROVING_DELAY_MS` | Optional sleep at the start of each `EpochSession`, before the TopTreeJob is constructed.  Used in tests to give late events time to land. |
+| `TX_GATHERING_TIMEOUT_MS` | Per-block tx gather deadline used by each `CheckpointProver`. |
+| `PROVER_NODE_FAILED_EPOCH_STORE` | If set, a full session that fails on its own account (top-tree/submit, provers healthy) uploads its proving data (every `CheckpointProver`'s txs + register-time data, regardless of sub-tree completion) to this file store. |
+| `PROVER_NODE_DISABLE_PROOF_PUBLISH` | If true, the publishing service runs `analyzeEpochProofSubmission` (estimates L1 fees) instead of actually submitting. |
+
+## Failure handling and observability
+
+Loggers:
+
+- `prover-node` — `ProverNode` itself (event dispatch, lifecycle).
+- `prover-node:session-manager` — reconcile decisions, `EpochSession` opens / drops, tick.
+- `prover-node:epoch-session` — per-`EpochSession` lifecycle (`Created EpochSession`,
+  `Top-tree proof ready`, `Submitted proof for epoch N`, etc.).
+- `prover-node:proof-publishing-service` — candidate submit / withdraw / expire,
+  drain, publish attempts, transient acquire retries.
+- `prover-node:l1-tx-publisher` — the per-publish `ProverNodePublisher`'s L1 work.
+- `prover-node:checkpoint-store` — content-key collisions, reap decisions.
+- `prover-node:checkpoint-prover` — sub-tree pipeline (gather, block processing).
+- `prover-client:chonk-cache` — chonk-verifier cache enqueue / release events.
+
+Failed proving data is uploaded to the configured file store (`PROVER_NODE_FAILED_EPOCH_STORE`) at two
+levels, and re-proved offline by the matching downloader/rerunner: an epoch job via
+`downloadEpochProvingJob` + `rerunEpochProvingJob`, and a single checkpoint via the same download plus
+`rerunCheckpointProvingJob` (which rebuilds just that checkpoint's sub-tree prover and awaits its block
+proofs — no epoch top-tree or L1 submit). Both upload levels build an `EpochProvingJobData` snapshot with
+`SessionManager.buildProvingData(...)` — every `CheckpointProver`'s txs and register-time data, even if
+its sub-tree never reached `isCompleted()` — and ship it via `uploadEpochProofFailure` alongside a
+world-state + archiver backup:
+
+- **Per checkpoint.** When a `CheckpointProver`'s block proofs reject for a non-cancel reason (a sub-tree
+  fault or a prune-induced fork fault), it fires its `onFailed` callback and `ProverNode` uploads that one
+  checkpoint via `tryUploadCheckpointFailure(prover)`. This fires for prune-induced faults too — that is
+  intentional; a prune-caused checkpoint snapshot is harmless, and *not* trying to tell prune from genuine
+  failure is exactly what keeps this race-free. A cancelled prover (control-plane prune / shutdown) is not
+  a failure and does not upload.
+- **Per session.** When a full session ends in its own genuine failure (`EpochSession.hasFailed()` —
+  top-tree/submit failed with every prover healthy), the session manager fires `onSessionFailed` and
+  `ProverNode` uploads all the session's checkpoints via `tryUploadEpochFailure(...)`. This is race-free
+  (healthy provers rule out a prune) and fires once, since the failed session is retained and never re-run.
+
+A `stopped` session (a prover under it failed) does not upload at the session level — its failed checkpoint
+already uploaded itself. Each backup is a full world-state + archiver snapshot, so a prune that faults
+several in-flight provers produces several uploads; this is accepted for the diagnostic value.
+
+Metrics emitted by `EpochSession`s:
+
+- `aztec.prover_node.execution_duration` — wall-clock time from `EpochSession` start to terminal.
+- `aztec.prover_node.job_duration` — same, in seconds.
+- `aztec.prover_node.job_checkpoints` / `_blocks` / `_transactions` — sizes of the
+  proven range.
+- `aztec.prover_node.block_processing_duration` /
+  `aztec.prover_node.checkpoint_processing_duration` — sub-tree breakdown.

package/dest/actions/download-epoch-proving-job.js

@@ -25,7 +25,7 @@ import { deserializeEpochProvingJobData } from '../job/epoch-proving-job-data.js
     }, log, {
         ...config,
         ...metadata,
-        snapshotsUrl: location
+        fileStore
     });
     const dataPath = urlJoin(location, 'data.bin');
     const localPath = config.jobDataDownloadPath;

package/dest/actions/rerun-epoch-proving-job.d.ts

@@ -1,12 +1,22 @@
 import type { L1ContractsConfig } from '@aztec/ethereum/config';
 import type { Logger } from '@aztec/foundation/log';
-import type { DataStoreConfig } from '@aztec/kv-store/config';
 import { type ProverClientConfig } from '@aztec/prover-client';
 import { ProverBrokerConfig } from '@aztec/prover-client/broker';
+import type { DataStoreConfig } from '@aztec/stdlib/kv-store';
+import type { GenesisData } from '@aztec/stdlib/world-state';
+type RerunConfig = DataStoreConfig & ProverBrokerConfig & ProverClientConfig & Pick<L1ContractsConfig, 'aztecEpochDuration'>;
 /**
  * Given a local folder where `downloadEpochProvingJob` was called, creates a new archiver and world state
- * using the state snapshots, and creates a new epoch proving job to prove the downloaded proving job.
+ * using the state snapshots, and creates a new epoch proving session to prove the downloaded proving job.
  * Proving is done with a local proving broker and agents as specified by the config.
  */
-export declare function rerunEpochProvingJob(localPath: string, log: Logger, config: DataStoreConfig & ProverBrokerConfig & ProverClientConfig & Pick<L1ContractsConfig, 'aztecEpochDuration'>): Promise<"awaiting-prover" | "completed" | "failed" | "initialized" | "processing" | "publishing-proof" | "reorg" | "stopped" | "timed-out">;
-//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoicmVydW4tZXBvY2gtcHJvdmluZy1qb2IuZC50cyIsInNvdXJjZVJvb3QiOiIiLCJzb3VyY2VzIjpbIi4uLy4uL3NyYy9hY3Rpb25zL3JlcnVuLWVwb2NoLXByb3Zpbmctam9iLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiJBQUNBLE9BQU8sS0FBSyxFQUFFLGlCQUFpQixFQUFFLE1BQU0sd0JBQXdCLENBQUM7QUFDaEUsT0FBTyxLQUFLLEVBQUUsTUFBTSxFQUFFLE1BQU0sdUJBQXVCLENBQUM7QUFDcEQsT0FBTyxLQUFLLEVBQUUsZUFBZSxFQUFFLE1BQU0sd0JBQXdCLENBQUM7QUFDOUQsT0FBTyxFQUFFLEtBQUssa0JBQWtCLEVBQXNCLE1BQU0sc0JBQXNCLENBQUM7QUFDbkYsT0FBTyxFQUFFLGtCQUFrQixFQUErQixNQUFNLDZCQUE2QixDQUFDO0FBVzlGOzs7O0dBSUc7QUFDSCx3QkFBc0Isb0JBQW9CLENBQ3hDLFNBQVMsRUFBRSxNQUFNLEVBQ2pCLEdBQUcsRUFBRSxNQUFNLEVBQ1gsTUFBTSxFQUFFLGVBQWUsR0FBRyxrQkFBa0IsR0FBRyxrQkFBa0IsR0FBRyxJQUFJLENBQUMsaUJBQWlCLEVBQUUsb0JBQW9CLENBQUMsK0lBc0NsSCJ9
+export declare function rerunEpochProvingJob(localPath: string, log: Logger, config: RerunConfig, genesis?: GenesisData): Promise<"awaiting-checkpoints" | "awaiting-predecessor" | "awaiting-root" | "cancelled" | "completed" | "failed" | "initialized" | "publishing-proof" | "stopped" | "superseded" | "timed-out">;
+/**
+ * Re-proves a single downloaded checkpoint proving job (as uploaded by a `CheckpointProver` failure).
+ * Reconstructs just that checkpoint's sub-tree prover from the snapshot and awaits its block proofs — no
+ * epoch top-tree, no L1 submission — so a checkpoint-level failure can be reproduced offline in isolation.
+ * Returns the block-rollup proof outputs on success; throws if the checkpoint fails to prove again.
+ */
+export declare function rerunCheckpointProvingJob(localPath: string, log: Logger, config: RerunConfig, genesis?: GenesisData): Promise<import("@aztec/stdlib/interfaces/server").PublicInputsAndRecursiveProof<import("@aztec/stdlib/rollup").BlockRollupPublicInputs, 480>[]>;
+export {};
+//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoicmVydW4tZXBvY2gtcHJvdmluZy1qb2IuZC50cyIsInNvdXJjZVJvb3QiOiIiLCJzb3VyY2VzIjpbIi4uLy4uL3NyYy9hY3Rpb25zL3JlcnVuLWVwb2NoLXByb3Zpbmctam9iLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiJBQUNBLE9BQU8sS0FBSyxFQUFFLGlCQUFpQixFQUFFLE1BQU0sd0JBQXdCLENBQUM7QUFFaEUsT0FBTyxLQUFLLEVBQUUsTUFBTSxFQUFFLE1BQU0sdUJBQXVCLENBQUM7QUFFcEQsT0FBTyxFQUFFLEtBQUssa0JBQWtCLEVBQXNCLE1BQU0sc0JBQXNCLENBQUM7QUFDbkYsT0FBTyxFQUFFLGtCQUFrQixFQUErQixNQUFNLDZCQUE2QixDQUFDO0FBTzlGLE9BQU8sS0FBSyxFQUFFLGVBQWUsRUFBRSxNQUFNLHdCQUF3QixDQUFDO0FBRzlELE9BQU8sS0FBSyxFQUFFLFdBQVcsRUFBRSxNQUFNLDJCQUEyQixDQUFDO0FBVzdELEtBQUssV0FBVyxHQUFHLGVBQWUsR0FDaEMsa0JBQWtCLEdBQ2xCLGtCQUFrQixHQUNsQixJQUFJLENBQUMsaUJBQWlCLEVBQUUsb0JBQW9CLENBQUMsQ0FBQztBQUVoRDs7OztHQUlHO0FBQ0gsd0JBQXNCLG9CQUFvQixDQUFDLFNBQVMsRUFBRSxNQUFNLEVBQUUsR0FBRyxFQUFFLE1BQU0sRUFBRSxNQUFNLEVBQUUsV0FBVyxFQUFFLE9BQU8sQ0FBQyxFQUFFLFdBQVcsbU1Bc0NwSDtBQUVEOzs7OztHQUtHO0FBQ0gsd0JBQXNCLHlCQUF5QixDQUM3QyxTQUFTLEVBQUUsTUFBTSxFQUNqQixHQUFHLEVBQUUsTUFBTSxFQUNYLE1BQU0sRUFBRSxXQUFXLEVBQ25CLE9BQU8sQ0FBQyxFQUFFLFdBQVcsbUpBaUJ0QiJ9

package/dest/actions/rerun-epoch-proving-job.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"rerun-epoch-proving-job.d.ts","sourceRoot":"","sources":["../../src/actions/rerun-epoch-proving-job.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAChE,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,uBAAuB,CAAC;AACpD,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AAC9D,OAAO,EAAE,KAAK,kBAAkB,EAAsB,MAAM,sBAAsB,CAAC;AACnF,OAAO,EAAE,kBAAkB,EAA+B,MAAM,6BAA6B,CAAC;AAW9F;;;;GAIG;AACH,wBAAsB,oBAAoB,CACxC,SAAS,EAAE,MAAM,EACjB,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,eAAe,GAAG,kBAAkB,GAAG,kBAAkB,GAAG,IAAI,CAAC,iBAAiB,EAAE,oBAAoB,CAAC,+IAsClH"}
+{"version":3,"file":"rerun-epoch-proving-job.d.ts","sourceRoot":"","sources":["../../src/actions/rerun-epoch-proving-job.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,iBAAiB,EAAE,MAAM,wBAAwB,CAAC;AAEhE,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,uBAAuB,CAAC;AAEpD,OAAO,EAAE,KAAK,kBAAkB,EAAsB,MAAM,sBAAsB,CAAC;AACnF,OAAO,EAAE,kBAAkB,EAA+B,MAAM,6BAA6B,CAAC;AAO9F,OAAO,KAAK,EAAE,eAAe,EAAE,MAAM,wBAAwB,CAAC;AAG9D,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,2BAA2B,CAAC;AAW7D,KAAK,WAAW,GAAG,eAAe,GAChC,kBAAkB,GAClB,kBAAkB,GAClB,IAAI,CAAC,iBAAiB,EAAE,oBAAoB,CAAC,CAAC;AAEhD;;;;GAIG;AACH,wBAAsB,oBAAoB,CAAC,SAAS,EAAE,MAAM,EAAE,GAAG,EAAE,MAAM,EAAE,MAAM,EAAE,WAAW,EAAE,OAAO,CAAC,EAAE,WAAW,mMAsCpH;AAED;;;;;GAKG;AACH,wBAAsB,yBAAyB,CAC7C,SAAS,EAAE,MAAM,EACjB,GAAG,EAAE,MAAM,EACX,MAAM,EAAE,WAAW,EACnB,OAAO,CAAC,EAAE,WAAW,mJAiBtB"}