reeflog-client 0.0.0-stage → 0.7.21

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +1875 -2
  3. package/index.js +597 -0
  4. package/package.json +34 -4
package/README.md CHANGED
@@ -1,3 +1,1876 @@
1
- # Temporary Holding Version
1
+ # reeflog
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ A durable local mailbox for Solana events. A daemon journals logs and what
4
+ they did to account balances for selected programs into a cursor-addressed
5
+ append-only log, so a consumer resumes without gaps after a crash,
6
+ disconnect, or reorg.
7
+
8
+ **Status: v0.7.21 shipped (2026-11-13), v0 working.** Live against mainnet-beta: see *Measured results* below. Install the binary without a Rust toolchain: `npm install reeflog-client` then `import { bootstrap } from 'reeflog-client'; await bootstrap()` (downloads the prebuilt release archive and verifies its sha256; the archive digests are `6bfec083…` linux / `303d0836…` macos, both equal what the release API publishes). The same install from the tagged release: `npm install github:auroraxo/reeflog#v0.7.21`. Or from source: `cargo install --git https://github.com/auroraxo/reeflog --tag v0.7.21`. The full event envelope (logs, balances, instructions, account list with owners, the bytes each account held, and the authority each token account had at the moment of the event), the Node client (see *The Node client*), webhook fan-out, a dead-letter queue, and archive backfill ship in this tree; Yellowstone as a second transport does not.
9
+
10
+ ## The idea
11
+
12
+ Solana event consumption is normally a live firehose. `logsSubscribe` has no delivery guarantee, no durable cursor, and no catch-up after disconnection. Every project rebuilds reconnect, deduplication, gap-fill, and reorg handling, or it silently loses events.
13
+
14
+ reeflog treats events as a mailbox instead of a phone call:
15
+
16
+ - **ingest:** subscribe to configured programs over WebSocket.
17
+ - **seal:** record events as confirmed, promote them when finalized, mark failures as errored.
18
+ - **gap-fill:** catch up on signatures the journal missed before resuming. A gap must be reported; silent loss is a protocol error.
19
+ - **journal:** append-only SQLite with one monotone `seq` and a portable file.
20
+ - **deliver:** a cursor-based CLI iterator that resumes exactly where it stopped.
21
+
22
+ ## Shape
23
+
24
+ ```bash
25
+ reeflog init --db reeflog.db --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc
26
+ reeflog start --db reeflog.db --seconds 60 # ingest over WS
27
+ reeflog seal --db reeflog.db # promote confirmed -> finalized
28
+ reeflog tail --db reeflog.db --cursor ./c.txt # resume from a cursor
29
+ reeflog verify --db reeflog.db # find holes
30
+ reeflog status --db reeflog.db
31
+ ```
32
+
33
+ The planned core is a Rust daemon and a TypeScript client. Decoding stays pluggable; reeflog does not itself decide what bytes mean.
34
+
35
+ **First-touch contract (measured 2026-11-02, `tests/quickstart.rs` +
36
+ `tests/rpc_timeout.rs`):** `init` is the only command that creates a journal,
37
+ and it refuses a `--program` that is not a base58 32-byte address — a typo
38
+ here used to be accepted silently and produced a mailbox that watches
39
+ nothing while reporting perfect health (`0 events`, `0 gaps`, exit 0).
40
+ Every other command refuses a `--db` path that has no journal, with the
41
+ `reeflog init …` line as the fix, instead of conjuring an empty one.
42
+ Every RPC call is bounded (10 s per call, two attempts for a stalled peer; a
43
+ 429 still gets the full six-attempt backoff). Degraded is not the same as
44
+ hung: when the endpoint accepts connections and never answers, `start` can
45
+ exceed its `--seconds` while the shutdown passes pay their bounded timeouts,
46
+ but it always returns and names the failure on stderr (measured 2026-11-02:
47
+ pre-fix, 4 of 5 repetitions of `start --seconds 3` never returned at 45 s;
48
+ post-fix, worst observed run returned in 83 s with `rpc call timed out
49
+ twice` reported).
50
+
51
+ **A flag the CLI does not know is refused, not ignored** (measured 2026-11-06,
52
+ `tests/arg_parsing.rs` + `tests/tool_invocations.rs`,
53
+ `research/flag-surface-2026-11-06.json`). Until now a typo in a flag was
54
+ silently DROPPED and the command ran anyway — the most expensive kind of quiet
55
+ failure this product can make, because the tool reported doing something it
56
+ did not do:
57
+
58
+ | Before | What actually happened |
59
+ |---|---|
60
+ | `reeflog verify --db reeflog.db --pagesx 400` | scanned with the **default** page budget (40), not 400, and reported it as if it had |
61
+ | `reeflog status --dbx nope.db` | read the **default** `reeflog.db` and printed a healthy status for it |
62
+ | `reeflog status --db reeflog.db --bogus-flag` | printed a status, output identical to a correct run |
63
+ | `reeflog init --db probe.db --program <pk> --nope` | created the journal anyway |
64
+ | `reeflog init --db probe.db` (no `--program`) | created the journal FIRST, refused SECOND — a mailbox left behind |
65
+ | `reeflog status --db=other.db` | ignored the inline form and read the default journal |
66
+ | `reeflog status --help` | ran `status` and printed a health report instead of the usage |
67
+
68
+ Now every command declares the flags it accepts, and anything else is
69
+ refused **before** any journal is opened or created:
70
+
71
+ - an unknown flag is refused, and when the typo is close to a real flag the
72
+ likely intent is named (`unknown flag --pagesx (did you mean --pages?)`);
73
+ a far-away typo gets no guess, because a wrong suggestion is worse than none;
74
+ - a flag that lost its value is refused — including when the "value" is
75
+ another flag (`--db --program` must not become a path called `--program`);
76
+ - a flag given twice is refused rather than silently resolved;
77
+ - a numeric flag whose value is not a non-negative whole number is refused,
78
+ naming the flag and the value (`flag --limit expects a non-negative whole
79
+ number, got `abc``) — before any journal is opened. Until 2026-11-08 an
80
+ unparseable value silently became the DEFAULT: `tail --limit abc` delivered
81
+ every row, and `tail --limit -1` delivered every row *twice as unbounded*,
82
+ because SQLite reads `LIMIT -1` as "no limit". `0` stays legal (several
83
+ flags document it as "disable this pass"), and the bare `--anchor-slot`
84
+ keeps its documented meaning. Measured against the published v0.7.1 binary:
85
+ 4/10 cases behaved as specified, 2 "refusals" delivered 650 rows anyway —
86
+ `research/numeric-flag-values-2026-11-08.json`;
87
+ - a numeric flag whose value is a number the CONSUMER cannot hold is also
88
+ refused, naming the flag and the value (`flag --limit expects a
89
+ non-negative whole number no greater than 9223372036854775807, got
90
+ `18446744073709551615``) — and a cadence in seconds must be one this
91
+ clock can schedule. "A non-negative whole number" is not one number
92
+ space: until 2026-11-08 the checker accepted `u64` while consumers parse
93
+ `i64` or add to a clock, so `tail --limit 18446744073709551615` delivered
94
+ the DEFAULT limit (1000 rows, exit 0), `verify --since-slot
95
+ 9223372036854775808` dropped the window and scanned a different one, and
96
+ `start --seconds 18446744073709551615` panicked while `start
97
+ --repair-every …` hung. Measured against the published v0.7.2 binary:
98
+ 4/11 cases behaved as specified (2 silent defaults, 3 panics, 1 hang) →
99
+ 11/11 — `research/numeric-flag-domain-2026-11-08.json`;
100
+ - a URL flag whose value is not a URL the CONSUMER can speak is also
101
+ refused, naming the flag and the value (`flag --ws expects a ws or wss URL
102
+ — got `https://…``) — before any journal is opened. Until 2026-11-09
103
+ `--ws` and `--rpc` were the only value flags with no domain at all: on the
104
+ published v0.7.5 binary `start --ws ftp://x/y` exited 0 with a journal it
105
+ had just created (0/11 cases behaved as specified), `--ws https://…` ran
106
+ its full duration and recorded a gap for a scheme tungstenite can never
107
+ speak, and `verify --rpc not-a-url` failed only after opening the journal,
108
+ in ureq's words (`bad uri: …`) rather than reeflog's. The domains are the
109
+ consumers', not an opinion: `ureq` resolves `http`/`https` only,
110
+ `tokio-tungstenite` accepts `ws`/`wss` only — and both compare schemes
111
+ case-insensitively, so `WS://…`, which the consumer accepts, is accepted
112
+ here too. Measured against the published v0.7.5 binary: 0/11 refused →
113
+ 11/11, 6/6 guards green on both trees —
114
+ `research/url-flag-domain-2026-11-09.json`;
115
+ - an argument that is not a flag is refused;
116
+ - `--flag=value` is honoured instead of falling back to the default;
117
+ - `--anchor-slot` keeps its documented **optional** value (bare = anchor at
118
+ the journal's own window), and `--help` prints the usage instead of running
119
+ the command.
120
+
121
+ `reeflog status` now names the journal it reports on (`"db": …`), so a report
122
+ can be checked against what was actually asked for. The false-positive guard
123
+ matters as much: `tests/tool_invocations.rs` pins every invocation shape the
124
+ `tools/e2e-*.mjs` harnesses pass, so a strict parser cannot quietly break the
125
+ project's own measured runs.
126
+
127
+ ## Measured results
128
+
129
+ Checked live on 2026-10-27 against `api.mainnet-beta.solana.com`, program `whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc` (Whirlpool):
130
+
131
+ | Measurement | Result |
132
+ |---|---|
133
+ | WS ingest | 3122 events in 75 s, 0 disconnects |
134
+ | Coverage in the claimed slot window | 224 / 226 chain signatures = **99.12 %** |
135
+ | Tail gap after a restart | 9 / 9 deleted events refilled |
136
+ | Middle hole detection | 17 missing reported = 15 deleted + 2 baseline, exactly |
137
+ | Cursor resume | no replay, monotone advance, no overlap |
138
+ | Unit tests | 58 / 58 |
139
+
140
+ Kill test (600 s on Raydium AMM v4, SIGKILL every 60 s): see *The kill test*
141
+ below — the first run **falsified** the delivery claim; the fixes are now
142
+ re-verified in the **same 600 s configuration** that falsified it
143
+ (`full-after-fix`, verdict `contract_survived`).
144
+
145
+ ## The kill test
146
+
147
+ `tools/killtest.mjs` is a falsifying harness, not a demo. It runs three
148
+ consumers against the same program for 600 s:
149
+
150
+ - **reeflog** — SIGKILLed every 60 s, restarted after 4 s of downtime.
151
+ - **naive-killed** — a bare `logsSubscribe` consumer, killed the same way.
152
+ - **naive-live** — a bare `logsSubscribe` consumer, **never killed**: the
153
+ control that separates "the kill caused it" from "the endpoint drops events
154
+ anyway".
155
+
156
+ It then fetches the chain's own signature list for the window independently
157
+ and holds it against every consumer's seen-set: a truth signature that *some*
158
+ consumer received on the log stream is deliverable, so missing it is a real
159
+ loss; a signature nobody on the stream received is classified with
160
+ `getTransaction` instead of assumed — only a transaction that actually
161
+ invokes the program (`invoked`) can count as a loss, never a
162
+ `referenced-only` one.
163
+
164
+ Run `sync` (2026-10-28, same harness, 120 s / 4 kills on Raydium AMM v4)
165
+ re-tested the claim and it **survived**: every check passed, verdict
166
+ `contract_survived` (exit 0).
167
+
168
+ | Check | Result |
169
+ |---|---|
170
+ | reeflog real losses | **0** of 1,112 deliverable events |
171
+ | naive-killed (control) real losses | **169** — the kill alone loses events |
172
+ | naive-live (control, never killed) real losses | **3** — the public WS drops events without any kill |
173
+ | verify before repair | 14 missing, `complete: true`, exit 0 |
174
+ | repair | `found_missing: 14`, `filled: 14`, `complete: true` |
175
+ | verify after repair | `missing_count: 0`, `complete: true`, exit 0 |
176
+ | dedup | 1,215 rows = 1,215 distinct signatures |
177
+ | cursor resume | second iteration: 0 rows (no replay) |
178
+ | incomplete scan | never exits 0 |
179
+
180
+ Run `full-after-fix` (2026-10-28, 600 s / 9 kills on Raydium AMM v4 — the
181
+ exact configuration that falsified the claim, re-run against the fixed
182
+ binary) **survived**: verdict `contract_survived`, exit 0, every product check
183
+ passed. This is the confirmation run the falsification owed.
184
+
185
+ | Check | Result |
186
+ |---|---|
187
+ | reeflog real losses | **0** of 5,374 deliverable events |
188
+ | naive-killed (control) real losses | **455** across 9 kill windows |
189
+ | naive-live (control, never killed) real losses | **4** — the public WS drops events without any kill |
190
+ | verify before repair | 1 missing, `complete: true`, exit 0 |
191
+ | repair | `found_missing: 1`, `filled: 1`, `complete: true` |
192
+ | verify after repair | `missing_count: 0`, `complete: true`, exit 0 |
193
+ | dedup | 5,636 rows = 5,636 distinct signatures |
194
+ | cursor resume | second iteration: 0 rows (no replay) |
195
+ | live repair during the run | 19 gaps reported, `gaps_open: 0`, 380 backfilled |
196
+
197
+ Run `full` (2026-10-28, the earlier 600 s run) **falsified** the same claim
198
+ and is kept here because it is what exposed the defects:
199
+
200
+ | Check | Result |
201
+ |---|---|
202
+ | reeflog lost deliverable events | **32 real losses** of 11,275 deliverable |
203
+ | naive-live lost deliverable events | 158 — the public WS drops events even with no kill |
204
+ | `reeflog verify` | reported `missing_count: 0`, `truncated: true`, **exit 0** |
205
+
206
+ Two defects, both measured against the RPC and fixed:
207
+
208
+ 1. **Phantom anchor.** The journal's newest held signature was a
209
+ confirmed-then-forked-out transaction. `getSignatureStatuses` returns
210
+ `null` for it, and `before=<it>` makes `getSignaturesForAddress` answer
211
+ `-32020 Transaction not found` for the whole walk. verify therefore read
212
+ **zero pages** and printed `missing_count: 0` — a hole detector that goes
213
+ blind exactly when holes exist. Every walk now starts at the chain tip and
214
+ pages only with signatures the chain itself returned; the tail stop is on
215
+ the boundary **slot**, not the boundary signature.
216
+ 2. **Nothing repaired mid-window holes.** The tail backfill only reaches
217
+ forward from the newest held slot. Events the socket dropped *while
218
+ connected* sit behind that frontier — all 32 losses did — and nothing ever
219
+ filled them. `start` now runs a bounded repair pass over a rolling window
220
+ (default every 30 s, 1800 slots back) and once more at shutdown.
221
+
222
+ Both are covered by regression tests, and the honesty rule is explicit:
223
+ `Report::complete()` requires no error, no truncation, at least one page read,
224
+ **and** measured coverage of the claimed window (first row at or above the
225
+ ceiling, last row at or below the floor). An unfinished scan exits 3 and can
226
+ never be mistaken for a clean one.
227
+
228
+ ## RPC semantics, measured
229
+
230
+ `getSignaturesForAddress` does not behave the way the parameter names suggest. Measured with an anchor at the 4th of 10 tip signatures:
231
+
232
+ | Parameter | Observed behaviour |
233
+ |---|---|
234
+ | `before=<sig>` | returns signatures **older** than the anchor — the backwards-paging primitive |
235
+ | `until=<sig>` | returns signatures **newer** than the anchor |
236
+ | `since=<sig>` | behaves like `until` — **newer** |
237
+ | `before` + `until` together | rejected with `-32020 Transaction not found` for an inverted window |
238
+ | `before=<phantom>` | `-32020 Transaction not found` — a held signature can be a transaction the address index no longer knows |
239
+ | `until=<phantom>` | same `-32020`; neither parameter tolerates a phantom anchor |
240
+ | `since=<slot>` | silently **ignored** by the public endpoint (a typo'd parameter returns the same rows), so a slot-bounded walk cannot be expressed at all |
241
+ | `getSignatureStatuses` | returns `null` for a phantom, a status for a real signature — the reliable way to tell them apart |
242
+ | throughput | 19 of 40 consecutive `limit=1000` pages answered HTTP 429, ~0.63 pages/s sustained |
243
+
244
+ Consequence: a backfill written with `since=<newest signature we hold>` returns an empty window and reports no loss while filling nothing. reeflog pages backwards with `before` and stops at the boundary **slot** instead. The default walk starts from the chain tip so no held signature is ever an anchor; `--anchor-slot` starts it from a signature the CHAIN returned (the last signature of a finalized block), which is the same rule from the other side. This was found by measuring the endpoint, not by reading the docs, and it is recorded in `src/backfill.rs`.
245
+
246
+ ## Event content (payload hydration)
247
+
248
+ A signature alone is not an event — a mailbox that delivers "something
249
+ happened at slot N" without saying what happened forces every consumer back
250
+ onto the RPC, which is the fragility reeflog exists to remove.
251
+
252
+ The gap machinery (`backfill.rs`) recovers *signatures* from
253
+ `getSignaturesForAddress`, which carries slot and error but no `logs`.
254
+ Measured 2026-10-28 on a 60 s live run against mainnet: 13 rows filled by
255
+ `repair`, all 13 stored with `logs IS NULL`, while all 2,082 rows the socket
256
+ delivered held payload. That asymmetry meant the rows the journal had to
257
+ *recover* were exactly the rows a consumer could not read.
258
+
259
+ reeflog hydrates missing content (`src/payload.rs`) with `getTransaction` and
260
+ distinguishes four states:
261
+
262
+ - **`held`** — log lines stored as a JSON array. Both socket capture and
263
+ hydration store into this state; hydration never overwrites captured logs.
264
+ - **`unavailable`** — the chain does not know the transaction (forked out or
265
+ pruned). Keeps `logs = null` and records the reason; an empty array would
266
+ claim "no logs", which is a different statement.
267
+ - **`invalid`** — the endpoint refuses the key (-32602). A corrupted row,
268
+ recorded separately from a fork-out so an operator can tell them apart.
269
+ Never folded into `unavailable`.
270
+ - **`pending`** — still unknown. Any pending row makes a pass incomplete;
271
+ incomplete passes exit 3 (same honesty rule as `verify` and `repair`).
272
+
273
+ `reeflog tail` delivers the log array by default; `--no-logs` keeps the
274
+ compact identity-only shape. A background worker hydrates recovered rows
275
+ continuously, beside the ingest loop — not once at startup, because the
276
+ periodic repair pass keeps recovering rows while the daemon runs, and a
277
+ killed run never reaches a shutdown pass. A final pass runs at shutdown.
278
+ Verified live: `tools/e2e-payload.mjs` checks all 13 conditions against
279
+ mainnet, including independent byte-for-byte comparison of stored payloads
280
+ against freshly fetched transactions.
281
+
282
+ ## What the event did to accounts
283
+
284
+ Until 2026-10-31 the opening line of this README claimed the daemon journals
285
+ "account changes" — and `grep -ci account src/*.rs` returned 0 across every
286
+ source file. The claim was checked against the code for the first time in
287
+ the feature's own commit message, and the code was brought up to the claim
288
+ in the same commit.
289
+
290
+ The reason this is nearly free: hydration already pays for the whole
291
+ `getTransaction` response and kept only `meta.logMessages`. Measured with
292
+ `tools/probe-accounts.mjs` on live mainnet (Raydium AMM v4, 4 transactions,
293
+ 2026-10-31): the discarded remainder of that same response averages 11.9 KB
294
+ per transaction and carries ~33.5 account keys, ~5.5 lamport-delta accounts
295
+ and ~3.5 changed SPL token balances per transaction. `src/accounts.rs`
296
+ extracts that into an envelope stored alongside the logs — at no extra RPC
297
+ cost for rows the journal had to recover.
298
+
299
+ The envelope contains: `lamport_changes` (index, address, before, after,
300
+ delta), `token_changes` (index, address, mint, owner, before, after,
301
+ delta) — amounts as exact decimal strings, never floats — plus
302
+ `instructions` and `err`.
303
+
304
+ It also carries **`token_state`** (added 2026-11-02): the event-time
305
+ authority of every token account the transaction touched whose amount did
306
+ NOT change — `index`, `address`, `mint`, `authority`, `token_program`,
307
+ `source: "tx-time"`. Until then the envelope stated the authority only for
308
+ accounts whose balance moved, and dropped the rest. That is the majority
309
+ of what a transaction touches: measured on live mainnet (Raydium AMM v4,
310
+ 10 transactions, `tools/probe-tx-time-authority.mjs`) **103 of 125 token
311
+ accounts (82.4 %)** carry an unchanged amount, and all 103 carry an
312
+ event-time authority the previous shape discarded. Changed accounts are
313
+ not duplicated — they keep their authority in `token_changes[].owner`, and
314
+ the two lists never claim the same account (pinned by
315
+ `a_changed_token_account_belong_to_token_changes_not_token_state`).
316
+
317
+ It also carries **the account list itself** (`accounts`, added 2026-10-31):
318
+ every account the transaction touched, in the same canonical order, with
319
+ `signer` and `writable` permissions and a `source` of `static`,
320
+ `lookup_writable` or `lookup_readonly`. Until then the envelope said *what
321
+ changed* and *how many* accounts there were, but not *which ones* or *whether
322
+ the transaction could write to them* — a consumer that needed those had to
323
+ pay for a second round trip to the chain.
324
+
325
+ Where the permissions come from is stated, not hidden, in
326
+ `accounts_flags_source`:
327
+
328
+ - `stated` — the response already carries them (the object form of
329
+ `accountKeys` states signer/writable per static key);
330
+ - `header` — derived from the message header's three counts (writable signed,
331
+ readonly signed, writable unsigned, readonly unsigned) plus the two loaded
332
+ groups. Measured before landing against `jsonParsed`'s authoritative flags
333
+ over 346 keys across 8 live mainnet transactions including two with address
334
+ lookup tables: 346/346 agreement, 0 mismatches, static-prefix order
335
+ identical. A real lookup-table transaction is pinned as
336
+ `test/fixtures/account-list-lookuptable.json` and re-checked offline on
337
+ every run (`the_derived_account_list_equals_the_authoritative_flags_of_a_real_lookup_table_transaction`);
338
+ - `unavailable` — the header is missing or its counts cannot describe the
339
+ static keys, so `accounts` is `null` and the permissions are **refused**
340
+ rather than guessed. A guessed permission is worse than an absent one: a
341
+ consumer that trusts `writable` would act on it.
342
+
343
+ `instructions` is the transaction's instruction bytes: every outer
344
+ instruction in execution order, then the CPIs each one spawned (`inner:
345
+ true`, tagged with the parent's index in `outer` and their own position in
346
+ `inner_index`), each with the resolved `program_id`, resolved `accounts`,
347
+ and the verbatim base58 `data` exactly as the RPC returned it — reeflog
348
+ never re-encodes bytes it did not produce. Account indices resolve through
349
+ the same canonical list the balances use (so a lookup table cannot shift an
350
+ argument to the wrong address); an index that does not resolve becomes an
351
+ explicit `null` at its own position, because position is information and a
352
+ shifted list would put an argument under the wrong name downstream. `err`
353
+ is `meta.err` verbatim (`null` = success), so a consumer never has to guess
354
+ whether a failed transaction's envelope describes an effect that happened.
355
+
356
+ The envelope does NOT contain pre-event account state; see *Evidence and
357
+ limits*. What an account HELD at fetch time is the section below.
358
+
359
+ ### What an account held (the bytes)
360
+
361
+ Ownership answers WHO controls an account; the bytes answer WHAT it held —
362
+ and both are outside the `getTransaction` response, so the bytes are their
363
+ own explicitly-budgeted pass: `hydrate --account-data` (after
364
+ `--accounts`, like `--owners`). It batch-fetches `getMultipleAccounts`
365
+ (100 addresses per call) for every address the held envelopes enumerate and
366
+ stores `account_data` inside each envelope. Every address gets exactly one
367
+ of three answers:
368
+
369
+ - **`held`** — the account's data bytes (base64), capped at 16384 B per
370
+ address. The cap is a measured defense: on live mainnet (Raydium AMM v4
371
+ window, 2026-11-02, `tools/probe-account-data.mjs`) account sizes sit at
372
+ p50 165 B / p90 1024 B / p95 4483 B — but the tail is real (a
373
+ 1,048,588-byte Serum/OpenBook slab, a 262,156-byte one) — and the
374
+ endpoint's `space` field reports the true size under the slice, so the
375
+ row records `bytes` (the real size) and `truncated`/`truncation_verified`
376
+ instead of pretending the cut is the whole. The probe semantics were
377
+ pinned live first: `space` reports the full size even when sliced
378
+ (1,048,588 reported as 16384 delivered), and an at-cap probe
379
+ (offset=cap, length=1) returns the account when more data exists and
380
+ `null` when it is gone-or-exact — an ambiguity the envelope reports as
381
+ `truncation_verified: false` rather than guessed.
382
+ - **`missing`** — the chain does not have the account at fetch time (the
383
+ same rule as the owner map: absence is a fact, recorded, never guessed).
384
+ - **`skipped_executable`** — the account is program code (same window: the
385
+ second-largest accounts were 105 KB / 74 KB of SPL program binaries).
386
+ Journaling program bytes as event data would be wrong in a different way
387
+ than truncating state is, so they are delivered as a flag plus their
388
+ true size, byte-free.
389
+
390
+ Measured on live mainnet (Raydium AMM v4, 2026-11-02, 20 s window, 119
391
+ rows): **14 calls for 1,235 distinct addresses**; sizes agree with an
392
+ independent re-fetch on **252 / 252**; the 81 executable accounts were all
393
+ flagged byte-free with correct sizes; 43 recorded-missing addresses
394
+ confirmed absent on an independent read; mint/owner decoded from the
395
+ stored bytes matched the transaction's own token metadata **35 / 35**.
396
+
397
+ Storing bytes honestly means admitting what a re-read can and cannot
398
+ prove: these are LIVE accounts, and between reeflog's fetch and any later
399
+ read a hot SPL token account's `amount` legitimately moves (measured in
400
+ the second e2e window: 39 of 252 re-reads differed this way). The e2e harness
401
+ therefore classifies differences instead of demanding byte-frozen
402
+ equality: identical, live-moving (a third read also differs),
403
+ volatile-region of a classic 165-byte token account (mint/owner immutable,
404
+ amount/delegate/close-authority are not), and — for layouts that grant no
405
+ immutable claim (mints, program state) — **unattributed**, which is
406
+ reported as a count and never waved through as a pass. The product claims
407
+ are anchored where time does not matter: completeness (every address
408
+ answered), true size (`space`), tx-consensus (decoded immutable fields vs
409
+ the transaction's own metadata), and absence-not-guessed.
410
+
411
+ The bytes are read AT FETCH TIME, not at event time — measured on the
412
+ probe window, 16–18 % of enumerated addresses no longer existed by the
413
+ time the pass ran. The envelope's slot and the `source` tag say when the
414
+ facts were true; historical state (the bytes AS OF the event) remains
415
+ open. Its boundary was measured on 2026-11-02 (`tools/probe-pre-event-bytes.mjs`,
416
+ report `research/pre-event-bytes-probe-2026-11-02.json`, 60 touched
417
+ accounts over 10 live transactions):
418
+
419
+ - **A past-slot question does not reach the past.** `getAccountInfo` with
420
+ `minContextSlot` set to the event's slot answered with the current state
421
+ and no error (requested slot 464542126, response context 464542734) —
422
+ the parameter is a floor, not a time machine, and a caller who reads it
423
+ as one silently receives today's bytes.
424
+ - **The transaction history carries no account bytes either.** 6 nearest
425
+ predecessor transactions were fetched and walked: 0 account-state
426
+ payloads anywhere in the responses; the 56 `data` fields found are
427
+ instruction bytes, not account state. The "recurse over predecessor
428
+ transactions" route this item carried is refuted, measured — the
429
+ history cannot serve what it does not contain.
430
+ - **What the history does certify is the other direction.** For 7 of the
431
+ 60 touched accounts nothing touched them after the event, so the bytes a
432
+ fetch returns now ARE the post-event bytes; the event's own post-state
433
+ witnesses that claim — lamports 4/4, token mint/owner/amount decoded
434
+ from the fetched bytes 2/2 — and on the 3 witnessed accounts the event
435
+ had moved, the pre-side witness agreed 0/3, which is what keeps the
436
+ check from being vacuous (wrong-side guard). 3 more certified accounts
437
+ were already gone by fetch time and stay counted but unwitnessed.
438
+ Certification is checked AFTER the fetch for a reason: an earlier
439
+ classify-then-fetch run hit the race it implies (1 witness disagreement,
440
+ root-caused to an account touched between the two calls), so the probe
441
+ re-checks the boundary post-fetch and counts expiring certifications
442
+ instead of hiding them.
443
+
444
+ Pre-event bytes therefore need a data source this RPC class does not
445
+ have (geyser with history, snapshot replay, an indexer) — not a second
446
+ fetch and not the chain's history, both measured here rather than argued.
447
+ The certification half is measured and NOT built into the envelope: no
448
+ field claims it.
449
+
450
+ ### Which program owns what
451
+
452
+ Ownership answers a different question than the ones above — *what program
453
+ controls this account* — and unlike everything else in the envelope, `owner`
454
+ is **not** in the `getTransaction` response (measured 2026-11-01 with
455
+ `tools/probe-owners.mjs`: the response names a token account's `programId`
456
+ inside `pre/postTokenBalances`, but nothing states the owner of an arbitrary
457
+ account). So reeflog treats it as its own budget rather than pretending it is
458
+ free:
459
+
460
+ - **Free tx-time half:** every changed token account and lamport-change
461
+ entry of a token account carries the SPL token program the transaction
462
+ itself named (`token_program` / `owner_program` on the delta entry), read
463
+ from the same response hydration already pays for.
464
+ - **Paid owner map:** `hydrate --owners` collects every address the held
465
+ envelopes enumerate **across all owed rows first**, then batch-fetches
466
+ `getMultipleAccounts` (100 addresses per call, owner-only dataSlice) and
467
+ stores `account_owners` inside each envelope: an `owners` map for addresses
468
+ the chain still has, and an explicit `missing` list for addresses it does
469
+ not.
470
+
471
+ Measured on live mainnet (Raydium AMM v4, 2026-11-01):
472
+
473
+ | Measurement | Result |
474
+ |---|---|
475
+ | Probe: distinct addresses → batched calls | 406 addresses, 5 calls (291× vs 1 per address) |
476
+ | Probe: addresses gone at fetch time | 63 / 406 = **15.5 %**, recorded as missing |
477
+ | Probe: authority agreement (stated vs decoded from bytes vs map) | **12 / 12** token accounts, all three paths |
478
+ | E2E: stored owners vs an independent re-fetch | **412 / 412** agree |
479
+ | E2E: missing addresses re-fetch as null **and carry no owner** | **57 / 57** |
480
+ | E2E: batched cost | **12 calls for 1,111 distinct addresses** |
481
+ | E2E: tx-time token program vs chain | **20 / 20** statable entries |
482
+ | E2E: `owner_program` vs re-fetched account owner | **13 / 13** |
483
+ | Idempotence | second run: 0 rows, 0 calls |
484
+ | Unit tests | **97 / 97** |
485
+
486
+ Three honesty rules the implementation is built against, each pinned by a
487
+ test and by the live harness:
488
+
489
+ - **A gone address is a fact, never a guess.** ~15 % of addresses no longer
490
+ exist when owners are fetched (accounts closed after the transaction).
491
+ They land in `missing` with **no** owner entry — writing `null`, omitting
492
+ the key, or filling in a plausible program would all be indistinguishable
493
+ from "fetched" to a consumer.
494
+ - **No half-written rows.** A row is written only when *every* address it
495
+ enumerates has an answer; a failed batch leaves the row owed instead of
496
+ storing a map with a silent hole in it.
497
+ - **Owners are read at fetch time, not at event time.** An account can change
498
+ owner after the transaction; the map's `source` says which pass wrote it,
499
+ and nothing here claims historical state (see *Evidence and limits*).
500
+ The exception that is not an exception: for token accounts the
501
+ transaction itself touched, the envelope NOW carries the event-time
502
+ authority (`token_state[].authority` / `token_changes[].owner`, read from
503
+ the transaction's own `preTokenBalances` — the chain's statement at event
504
+ time, measured to agree with a later fetch-time read 124/124 in the same
505
+ probe window, and irreplaceable for the 1 of 125 accounts that no longer
506
+ exist at all). Historical state for accounts a transaction did NOT touch
507
+ remains open, unchanged.
508
+
509
+ The two pre-existing correctness traps above — lookup-table indexing and a
510
+ closed token account — apply to the owner map for the same reason: it is keyed
511
+ by the canonical address list, and the negative control in the probe proves an
512
+ absent account produces no entry rather than a fabricated one.
513
+
514
+ Two correctness traps the extractor is built against, each pinned by a test:
515
+
516
+ - **Lookup-table indexing.** `preBalances`/`postBalances` are indexed over
517
+ static keys ++ loaded writable ++ loaded readonly, while a versioned
518
+ transaction's `message.accountKeys` lists only the static keys. Indexing
519
+ balances by that field silently attaches every changed balance to the
520
+ wrong address. The account list is rebuilt in canonical order instead
521
+ (`a_lookup_table_does_not_shift_the_balances_onto_the_wrong_addresses`).
522
+ - **A closed token account.** An account present in `preTokenBalances` and
523
+ absent from `postTokenBalances` closed mid-transaction. The first live
524
+ cross-check caught exactly this: the delta was encoded as an empty string.
525
+ Absence on one side means zero on that side, so the delta is `-before`
526
+ (`a_closed_token_account_reports_the_full_negative_delta`).
527
+
528
+ Delivery and enrichment:
529
+
530
+ - `hydrate --accounts` fetches envelopes for rows whose logs came from the
531
+ WebSocket (the `logsNotification` carries no balances). Same honesty
532
+ contract as payload: a rate limit leaves the debt, exit 3 reports it; the
533
+ chain forgetting a transaction is recorded as `account_state`
534
+ `unavailable`, never a pending-forever row and never an empty envelope.
535
+ - `tail` delivers the envelope as `accounts` on every row; `account_state`
536
+ says why one is missing. `tail --with-accounts` delivers only rows whose
537
+ envelope is complete, and **holds** at the first row the debt queues still
538
+ owe: the cursor stays behind it, the hold and the filling command
539
+ (`hydrate --accounts`) are printed, and the command exits 3 — the same
540
+ honesty contract as `hydrate` itself. Only a row with a recorded outcome
541
+ (`unavailable`/`invalid`/`no_metadata`), which can never be filled, is
542
+ skipped, reported, and consumed.
543
+ - `webhook push` bodies carry the envelope in the same signed delivery.
544
+ - `status` counts `accounts_held` / `accounts_missing` /
545
+ `accounts_unreachable`.
546
+
547
+ Verified live (`tools/e2e-accounts.mjs`, mainnet, 22/22): envelopes from the
548
+ free path and the paid path, idempotence, delivery shape, cursor semantics,
549
+ and a field-by-field cross-check of every stored envelope — balances,
550
+ instructions and outcome — against an independent re-fetch and re-parse of
551
+ the same transactions from the chain. Two claims do not depend on live
552
+ traffic, because a sample of healthy transactions cannot show them: a local
553
+ stub serves a transaction with two CPIs (the parent tag and the position
554
+ inside it are pinned) and a failed transaction (`InstructionError [0,
555
+ {Custom: 42}]` must arrive verbatim; on successful transactions the live
556
+ comparison is `null === null` and would stay green if the field were
557
+ dropped). Against the pre-fix binary (741b622) the same harness reports
558
+ 16/19 — exactly the instruction checks go red, and the 16 checks that
559
+ existed before stay green on both trees.
560
+
561
+ `tools/e2e-accounts-mutations.mjs` mutates the extractor six ways (address
562
+ shift, dropped delta, empty delta, wrong mint, instruction data shifted, CPI
563
+ position dropped) and each mutant must fail the live harness — a green
564
+ harness that cannot go red proves nothing. The probe counts only exit 1 (a
565
+ named check failed) as a catch: exit 2 is the harness failing to set up
566
+ against the rate-limited public endpoint, and counting that as a catch
567
+ would report a mutant as caught by a verdict that never ran.
568
+
569
+ ## First external consumer: a fill watcher
570
+
571
+ Until this section existed, every claim was checked by reeflog's own
572
+ harness. `tools/fill-watcher.mjs` is a consumer in the sense a bot cares
573
+ about: it reads `reeflog tail --cursor` and turns rows into **fills** —
574
+ signature, slot, direction, `amount_in`, `out_amount`, pool reserves — and it
575
+ does that from local bytes, with no RPC call of its own.
576
+
577
+ The decode is not guessed. Raydium AMM v4 emits `ray_log:` as a
578
+ bincode-serialized struct (`SwapBaseInLog` / `SwapBaseOutLog` in
579
+ `raydium-amm/program/src/log.rs`), so the field order is the program's own.
580
+ The wire layout was then verified against the chain: for every sampled swap,
581
+ `pool_coin + amount_in` equals the pool's coin-vault balance after the swap
582
+ and `pool_pc − out_amount` equals its pc-vault balance, both read from the
583
+ transaction's pre/post token balances. A line the decoder cannot read is
584
+ reported as `undecodable`, never guessed at.
585
+
586
+ `tools/e2e-fillwatch.mjs` runs this watcher against the alternative — the
587
+ classic `getSignaturesForAddress` + `getTransaction` poller — over the **same
588
+ window on the same endpoint**, and fails if:
589
+
590
+ 1. `hydrate` leaves any row unreadable (`pending` must be 0, exit 0);
591
+ 2. any fill the poller decoded **inside the daemon's slot window** is
592
+ missing from reeflog's stream;
593
+ 3. a fill both consumers saw decodes to **different** amounts (a payload that
594
+ exists but disagrees with the chain is not delivery);
595
+ 4. the watcher's own decoder drops a held leg;
596
+ 5. the local decode phase makes any RPC call.
597
+
598
+ The comparison is **slot-aligned**, and the reason is worth stating: a naive
599
+ page fetch reaches back ~1000 signatures into history, while the daemon's
600
+ window starts when its WebSocket subscription delivers its first event. The
601
+ first run of this harness compared the raw sets and reported 21 "missing"
602
+ fills — every sampled one sat in a slot *before* the journal's first slot,
603
+ i.e. before the daemon had connected. That is not a loss, and treating it as
604
+ one would be exactly the kind of plausible-wrong-number this project refuses
605
+ to publish. Fills outside the window are now counted and reported
606
+ separately, never as losses.
607
+
608
+ Measured 2026-10-28, 60 s live window on mainnet, Raydium AMM v4:
609
+
610
+ | Measurement | reeflog | naive poller |
611
+ |---|---|---|
612
+ | fills decoded | **155** | 12 |
613
+ | RPC calls for that content | **0** | 44 (4 signature pages + 40 transactions) |
614
+ | in-window fills missing from reeflog | **0** of 7 | — |
615
+ | fills decoding to identical amounts | **7 / 7** | — |
616
+ | rows drained | 474 rows in 316 ms | — |
617
+
618
+ The naive consumer is budgeted on purpose: the public endpoint answers
619
+ roughly one `getTransaction` per second under load, so a poller that wants
620
+ content pays for it — every call above is counted, not estimated. reeflog
621
+ paid for its content once, at ingest, and the watcher then reads bytes it
622
+ already had.
623
+
624
+ `tools/test-decoder.mjs` is the offline half: three real mainnet fixtures
625
+ with every field asserted, the 57-byte rule enforced, a non-swap log type
626
+ refused, and a row without a swap proven not to become a fill. It runs in CI
627
+ and needs no network.
628
+
629
+ ## Reorgs: a retraction a consumer can actually receive
630
+
631
+ A mailbox that delivers an event and then loses it is only half a mailbox.
632
+ The other half is saying *that event no longer stands* — and saying it in a
633
+ way a consumer can act on.
634
+
635
+ reeflog marks a reorg only when **two independent sources refuse the same
636
+ transaction**. That rule exists because of what a single source can do:
637
+
638
+ - `getSignatureStatuses` answers `null` for a signature it cannot answer for,
639
+ and a transport hiccup looks exactly like a fork-out;
640
+ - a rate limit looks like silence;
641
+ - an endpoint that refuses the *key* (-32602) says nothing at all about the
642
+ chain — that is a corrupted row, not a retracted event.
643
+
644
+ So the states stay apart, the way `hydrate` keeps `unavailable`, `invalid` and
645
+ `pending` apart:
646
+
647
+ | Both sources | Verdict | What happens |
648
+ |---|---|---|
649
+ | at least one serves the transaction | known | nothing changes |
650
+ | both refuse it | **reverted** | recorded in `reorgs`, event's seal becomes `reverted` |
651
+ | one refuses the key | invalid | no chain verdict, the pass stays complete |
652
+ | one fails | undecided | the pass exits 3 — a failure is not a verdict |
653
+
654
+ ```bash
655
+ reeflog reorg check --db reeflog.db --limit 200 # detect + record
656
+ reeflog reorg tail --db reeflog.db --cursor ./reorgs.txt
657
+ ```
658
+
659
+ **The check runs by itself.** `reeflog start` seals pending events and
660
+ re-checks them on a schedule (`--seal-every`, default 15 s; `--reorg-every`,
661
+ default 60 s), and once more at shutdown. A retraction that only exists when
662
+ an operator remembers to run a command is not a product: the consumer who
663
+ acted on a fill at 14:02 has to find out at 14:02, not at 18:00. The daemon's
664
+ window is bounded in **slots** (`--reorg-lookback`, default 300 behind the
665
+ tip), because the chain — not the journal — decides how far a fork can reach.
666
+
667
+ The daemon's summary reports what the reorg work actually did
668
+ (`reorg_passes`, `reorg_considered`, `reorg_reverted`,
669
+ `reorg_incomplete_passes`), so "0 reverts" can be told apart from "the check
670
+ never ran" — the difference between evidence and silence.
671
+
672
+ Reorgs get **their own channel and their own cursor**, because a consumer
673
+ that already read an event cannot learn about its retraction from the event
674
+ stream: the event keeps its `seq` and is delivered exactly once. `tail`
675
+ reports the event's *current* seal, `reorg tail` reports the retraction
676
+ itself. A consumer that only reads events learns that a row now reads
677
+ `reverted`; one that reads the reorg channel learns it at the moment it
678
+ happened. Neither is optional if a consumer is to undo a fill.
679
+
680
+ `tools/e2e-reorg.mjs` proves both directions against live mainnet, because
681
+ the dangerous direction is the false positive — a detector that retracts a
682
+ live transaction is worse than no detector, since a consumer would undo a
683
+ real fill. Measured 2026-10-28:
684
+
685
+ - **negative:** real finalized transactions harvested from mainnet → **0
686
+ reverts**, twice: 179 in the first run and 189 in the confirming run (the
687
+ count moves with the harvest, the verdict does not);
688
+ - **positive:** a well-formed signature the chain does not know → detected,
689
+ recorded, delivered exactly once on the reorg channel, and the second read
690
+ delivered nothing (strict cursor);
691
+ - **honesty:** an unreachable endpoint → exit 3, `complete: false`, **0
692
+ reverts** — a transport failure never becomes a fact about the chain.
693
+
694
+ The check is batched: one `getSignatureStatuses` call covers 200 rows, and
695
+ `getTransaction` is spent only on rows the first source already refused. On a
696
+ healthy chain a pass costs one call.
697
+
698
+ The same two-source rule now also guards `seal`: a `null` from the status
699
+ endpoint no longer writes `missing` on its own. Before this, one silent
700
+ source could turn a transaction the chain still knows into a recorded
701
+ "missing" — the same false statement a reorg marker must never publish.
702
+ `missing` now means *both* sources refuse the transaction; if the second
703
+ source cannot answer either, the row stays pending and the pass reports
704
+ `undecided`.
705
+
706
+ ### A real fork, end to end
707
+
708
+ `tools/e2e-reorg.mjs`'s positive case plants a well-formed signature the
709
+ chain never saw. That proves the two-source rule fires on an *unknown* key —
710
+ not that it fires on a transaction that really was executed, really reached
711
+ `confirmationStatus=finalized` with the root past its slot, and was then
712
+ taken back. That is the only case a consumer can actually lose money on,
713
+ and it was the open gap.
714
+
715
+ `tools/e2e-fork.mjs` manufactures it honestly on a local agave chain
716
+ (`node tools/e2e-fork.mjs`; `AGAVE_BIN` to point at a release). It runs a real
717
+ validator, sends real system transfers, and rolls the ledger back to a
718
+ copy of the blockstore taken *after* a first transaction and *before* a
719
+ second one. Nothing is planted — the signature the detector retracts is one
720
+ the chain itself made and then unmade. From the measured run (slots and
721
+ hashes differ run to run; they are recorded per run in the report):
722
+
723
+ - the **survivor** (slot 45) sits inside the snapshot: after the rollback its
724
+ `getTransaction` still serves it and the block hash at its slot is
725
+ byte-identical (`Cs4JNVar…` before and after);
726
+ - the **forked transaction** (slot 108) is finalized, sealed, and promised as
727
+ final — then both sources refuse it, and the block at that same slot is a
728
+ different block (`CgbDQNLd…` → `FGJcD4zT…`);
729
+ - the **running daemon** — not a one-shot CLI call — reports
730
+ `1 event(s) REVERTED`, with `reorg_reverted=1`, `reorg_incomplete=0` over
731
+ 12 passes, and the journal ends with exactly one reorg record naming that
732
+ signature while the survivor and a post-fork transaction both stay
733
+ `finalized`;
734
+ - the retraction is delivered **exactly once**, the second read delivers
735
+ nothing, and the consumer's ledger undoes the acted-on event and remembers
736
+ it as retracted;
737
+ - controls: with the chain intact the detector finds **0** reverts (CLI and
738
+ daemon), and a new transaction on the forked chain is ingested, sealed, and
739
+ never retracted.
740
+
741
+ Measured 2026-10-29: **47/47 checks pass**, twice
742
+ (`research/fork-e2e-report.json`, verdict `passed`). The harness asserts the
743
+ toolchain gap as `skipped` (exit 3) rather than a pass when an agave binary
744
+ or the built binary is missing.
745
+
746
+ A real fork on **mainnet** has still not been observed end to end — that
747
+ requires watching long enough for mainnet to produce one.
748
+
749
+ ## Undoing a fill: the retraction reaches the consumer
750
+
751
+ Detecting a revert is only half the job. The other half is the question a
752
+ consumer actually has to answer: **which of my actions must be undone?**
753
+
754
+ `tools/fill-watcher.mjs` now keeps a ledger of the fills it read, and
755
+ `watchRetractions()` joins the reorg channel against that ledger:
756
+
757
+ - a retraction for a signature the consumer **acted on** comes back marked
758
+ `undoable`, carrying the original fill's data — so the consumer reverses
759
+ precisely what it did, amounts included;
760
+ - a retraction for a signature it **never read** comes back marked
761
+ `undoable: false` — reported, never silently dropped, and never turned into
762
+ an unfill of some *other* fill;
763
+ - the retracted fill leaves the open ledger and enters the undone ledger, so
764
+ the same reversal is never applied twice.
765
+
766
+ The ledger is written BigInt-safe: the amount fields are u64 and exceed
767
+ `Number` range, and a ledger that quietly loses digits would make every later
768
+ comparison meaningless. (The watcher's own CLI used to crash on its first
769
+ printed fill for exactly that reason — `JSON.stringify` has no answer for a
770
+ BigInt.)
771
+
772
+ `tools/e2e-retraction.mjs` proves the whole path against live mainnet: the
773
+ daemon's own check finds the revert, the retraction arrives on its own
774
+ channel, the ledger join marks it undoable and carries the original fill,
775
+ an unrelated retraction is reported but not unfilled, and a second read
776
+ delivers nothing.
777
+
778
+ One bug in the harnesses themselves is worth naming, because it hides in
779
+ every measurement made with them. The consumer helper resolved its child
780
+ process on Node's `'exit'` event, which fires when the process *dies* — not
781
+ when its piped stdout has been read. A `reorg tail` run could therefore
782
+ resolve with an empty output while the child had already written its own
783
+ cursor file, i.e. it had really delivered the row the harness could not see.
784
+ The consumer then reported **0 retractions for a retraction that was there**,
785
+ and the strict cursor made it unrecoverable on the next read. Every harness
786
+ now resolves on `'close'` (stdio drained) and fails loudly on a spawn error.
787
+ Measured 2026-10-29: this was the difference between 44/47 and 47/47 on the
788
+ same harness, same code.
789
+
790
+ ## Webhook fan-out: a consumer that is not on this machine
791
+
792
+ `reeflog tail` makes the journal a local tool: the consumer has to run next
793
+ to the daemon and read a file. A webhook makes it a mailbox in the actual
794
+ sense — an outside service gets the events pushed to it, and the delivery
795
+ cursor survives a crash on either side.
796
+
797
+ ```bash
798
+ cat > webhooks.json <<'EOF'
799
+ {
800
+ "targets": [
801
+ { "id": "fill-service", "url": "https://example.com/reeflog", "secret": "whsec_…", "timeout_ms": 5000 }
802
+ ]
803
+ }
804
+ EOF
805
+
806
+ **Config values are checked before the pass starts**, because a config file
807
+ is one layer above the flag parser and fails the same way the parser used
808
+ to: a value the consumer cannot honour must be refused, not accepted into a
809
+ behaviour the file does not describe.
810
+
811
+ | Key | Domain | What is refused |
812
+ |---|---|---|
813
+ | `url` | absolute URL of scheme `http` or `https` (case-insensitive) with a host, a port `1..=65535` when written, and bytes `http::Uri` can carry | `http://` or `https://` with no host (the consumer dies with "http: empty string" on first delivery), empty authority (`http:///x`), missing scheme, non-http scheme (`ftp://…`), a non-numeric or out-of-range port (`:abc`, `:99999`, a bare `:` — the consumer silently dials 80/443 instead), port `0` (nothing ever listens there), a space/control byte/`<` `>` backtick in the path (the parser refuses it as `http: invalid uri character`), one of `" < > \ ^ \` { \| }` in the host, raw non-ASCII in the path or query (the receiver answers 400 — percent-encode it instead) |
814
+ | `id` | non-empty, no leading ASCII whitespace, no control byte except HTAB | empty, a leading space or tab (the receiver strips it, so the consumer's dedup key stops matching the journal), any C0 control byte or DEL (the consumer's header parser refuses the field: every delivery dies with `http: failed to parse header value`) |
815
+ | `secret` | non-empty; and one secret per ENDPOINT — two targets that share a secret must point at the same url | empty — an empty secret signs with the empty key, which authenticates nothing. Two targets that share a secret while pointing at DIFFERENT urls are refused — the signed body carries no target id and no endpoint (the headers that do are not signed), so what one receiver verifies the other verifies too: measured, A's accepted bytes replayed at B under a forged event-id header and passed B's own HMAC check (`tools/measure-secret-domain.mjs`, 4/4 faces on the published v0.7.15 asset). One secret is one identity space; give each endpoint its own secret. The same secret on the SAME url stays legal — one consumer listed twice |
816
+ | `timeout_ms` | whole milliseconds in `1..=3600000` (1 hour) | `0` (the poster would silently re-read it as 1 ms), anything above the 1-hour ceiling (a timeout no process will live to see — an attempt that cannot finish within an hour is stuck, not slow), a fraction, a negative number, a string, an integer beyond `u64` |
817
+ | `dead_letter_after` | non-negative integer in `i64` | a fraction, a negative number, an integer outside `i64` (the count is compared against an `i64` refusal counter) |
818
+
819
+ The `url` and `id` rules are not opinions: they call the CONSUMER's own
820
+ parser — `http::Uri` and `http::HeaderValue::from_str`, the same functions
821
+ ureq and tokio-tungstenite link (one `http` version in Cargo.lock). What the
822
+ checker refuses, the consumer would have refused later, at delivery time,
823
+ in its own words.
824
+
825
+ Every number behind those rules was measured against a real receiver, and
826
+ the measurement is reproducible from this repository:
827
+
828
+ ```bash
829
+ # 1. the published baseline (the release asset's sha256 equals the
830
+ # Release-API digest) — the consumer's behaviour does not change
831
+ # between releases, but the config refusals do
832
+ gh release download v0.7.10 --repo auroraxo/reeflog -p "*linux*"
833
+ tar xzf reeflog-linux-amd64.tar.gz
834
+
835
+ # 2. the probe: one byte (or character) per delivery, against the binary
836
+ node tools/probe-webhook-value-domain.mjs ./reeflog /tmp/whprobe
837
+ node tools/make-webhook-value-domain-artifact.mjs /tmp/whprobe \
838
+ research/webhook-value-domain-<date>.json
839
+
840
+ # 3. the before/after: the same values against the published baseline
841
+ # and against this tree's build
842
+ node tools/measure-webhook-value-domain.mjs ./reeflog \
843
+ "$(pwd)/target/release/reeflog" /tmp/before-after.json
844
+ ```
845
+
846
+ The raw sweeps land in `research/webhook-value-domain-<date>.json` and are
847
+ pinned by `tests/webhook_value_measurement.rs`; the before/after counts
848
+ (24 cases, 11 defects in the published baseline, all refused up front by
849
+ this tree) are pinned by `tests/webhook_value_before_after.rs`.
850
+
851
+ Every refusal names **the key and the offending value** and happens before
852
+ any connection is attempted — a refusal with a report would look like a
853
+ delivery pass. `timeout_ms: 5000` (default) and `dead_letter_after: 5`
854
+ (default) are unaffected; `dead_letter_after: 0` keeps its documented
855
+ meaning of "disable the valve".
856
+
857
+ reeflog webhook push --db reeflog.db --config webhooks.json # one pass
858
+ reeflog webhook status --db reeflog.db --config webhooks.json # cursors + failures
859
+ reeflog webhook dlq list --db reeflog.db --config webhooks.json # quarantined events
860
+ reeflog webhook dlq replay --db reeflog.db --config webhooks.json # hand them over
861
+ reeflog start --db reeflog.db --webhook-config webhooks.json --webhook-every 5
862
+ ```
863
+
864
+ **The contract, in order of importance:**
865
+
866
+ 1. **A delivery is recorded only when the endpoint answered 2xx.** A 403,
867
+ a timeout or a rate limit is recorded as a *failed attempt* with the
868
+ status that caused it. `webhook push` exits 3 while a target is blocked,
869
+ exactly like `verify` and `hydrate`: silence is never health.
870
+ 2. **At-least-once, in order.** Per target the deliveries follow the
871
+ journal's `seq` and stop at the first failure, so a consumer sees events
872
+ in chain order and never with a hole in front of it. It may see one
873
+ twice; the stable id (`X-Reeflog-Event-Id: <target>:<kind>:<seq>`) is
874
+ what a consumer dedups on for exactly-once.
875
+ 3. **The payload is authenticated.** Every request carries
876
+ `X-Reeflog-Signature: sha256=<hex>` = HMAC-SHA256(secret, body) over the
877
+ exact bytes sent (canonical JSON — `serde_json`'s sorted keys, so the same
878
+ event always produces the same signature). The receiver checks it with
879
+ one line of its own language:
880
+
881
+ ```python
882
+ import hmac, hashlib
883
+ want = "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest()
884
+ ```
885
+ 4. **A retraction is delivered too**, on its own cursor, and it is delivered
886
+ *even when the event queue is blocked behind a refusal* — a retraction is
887
+ the one message whose lateness costs money.
888
+ 5. **Receipts make the crash window safe.** The state (per-target cursor)
889
+ and the receipts (which exact body was accepted, by sha256) live in the
890
+ journal. A daemon that dies between "POST succeeded" and "cursor advanced"
891
+ does not deliver the event twice on the next pass.
892
+ 6. **A refusal stops being a verdict only when it stops repeating.** One
893
+ 4xx can be an accident (mid-deploy, a config reload), so the refusal is
894
+ counted. After `dead_letter_after` refused attempts on the same event
895
+ (default 5, `0` disables it) the event moves to the **dead-letter queue**:
896
+ it is quarantined with the exact bytes it was owed, and the target's
897
+ cursor moves past it, so one consumer that will never accept one event
898
+ does not stop its own mailbox forever while the events behind it are
899
+ never delivered. A dead letter is **not** a delivery — no receipt is
900
+ written, `push` reports it as `dead_lettered`, and `webhook status` shows
901
+ it as `dead_letters` until it is replayed. Transient failures (5xx, 429,
902
+ timeout) are never quarantined: an outage is not a verdict. Retractions
903
+ are never quarantined either: a late retraction costs money, so its queue
904
+ stays blocked rather than being papered over.
905
+
906
+ ```bash
907
+ reeflog webhook dlq list --db reeflog.db --config webhooks.json
908
+ reeflog webhook dlq replay --db reeflog.db --config webhooks.json
909
+ ```
910
+
911
+ The replay sends the stored body byte for byte, so the signature over it
912
+ is the same signature it would have had — a receiver deduping on
913
+ `X-Reeflog-Event-Id` sees the same event it rejected (or missed), not a
914
+ new one. A replay that is refused again stays in the queue and exits 3;
915
+ nothing is ever dropped.
916
+
917
+ ### Measured, against an independent receiver
918
+
919
+ `tools/e2e-webhook.mjs` (2026-10-29, live mainnet, Raydium AMM v4): a
920
+ separate receiver process that reimplements the signature check with
921
+ `node:crypto`, decides its own answers, and logs what it actually received —
922
+ **14/14 checks passed** over 118 real events:
923
+
924
+ | Check | Result |
925
+ |---|---|
926
+ | Fan-out to two independent targets | 118/118 delivered to **each**, in seq order, 118 distinct |
927
+ | Signature verification at the receiver | 236 bodies, all valid (independent `node:crypto`) |
928
+ | Delivered bodies carry content | 6 of 118 with real non-empty `logs` |
929
+ | A **403** endpoint | 0 delivered, `blocked_seq: 1`, exit 3, cursor parked at 0, 0 receipts, attempt recorded as status 403 |
930
+ | A body signed with the **wrong secret** | receiver rejects, 0 delivered, blocked |
931
+ | A **429** endpoint | blocks the queue instead of dropping; recovery delivers 118 exactly once (118 accepted, 118 distinct) |
932
+ | Caught-up queue | delivers nothing on a repeat pass |
933
+ | Crash between POST and cursor (receipt without cursor) | 117 of 118 delivered, the owed event **not** re-sent, cursor catches up |
934
+ | Retraction while the event queue is blocked | reorg delivered and verified, event queue still blocked, exit 3 |
935
+ | Third implementation | Python `hmac` recomputes 476 bodies, **0 mismatches** against what the receiver accepted |
936
+
937
+ Report: `research/webhook-e2e-report.json`.
938
+
939
+ ### The daemon path, killed mid-run
940
+
941
+ `tools/e2e-webhook-daemon.mjs` (2026-10-29, live mainnet): `reeflog start
942
+ --webhook-config` ingests over a live WebSocket and pushes to the receiver on
943
+ its own cadence. The daemon is **SIGKILLed** after 35 s — a mailbox that only
944
+ survives a clean shutdown is not a mailbox. **7/7 checks passed**:
945
+
946
+ | Check | Result |
947
+ |---|---|
948
+ | The daemon delivers what it ingested | 968 of 1.069 journal events delivered live (the rest were ingested after the last pass before the kill) |
949
+ | Order and no repeats | 968 deliveries, 968 distinct, seqs 1..968 |
950
+ | Signatures at the receiver | 968/968 valid (independent `node:crypto`) |
951
+ | The cursor survives the SIGKILL | cursor 968 = last delivered seq, 968 receipts |
952
+ | The follow-up resumes only the owed remainder | 101 delivered — exactly the 101 that were still owed, **0 re-sends** |
953
+ | The consumer ends up holding the whole journal | 1.069 distinct of 1.069 journal events |
954
+
955
+ Report: `research/webhook-daemon-e2e-report.json`.
956
+
957
+ ### The dead-letter queue, measured against the same receiver
958
+
959
+ `tools/e2e-dlq.mjs` (2026-10-29, live mainnet, Raydium AMM v4): the receiver
960
+ permanently refuses ONE event (seq 1) and accepts everything behind it —
961
+ the exact case that used to stop that target's mailbox forever.
962
+ **13/13 checks passed**:
963
+
964
+ | Check | Result |
965
+ |---|---|
966
+ | A 503 is retried, not quarantined | blocked, 0 dead letters, 0 pending — an outage is not a verdict |
967
+ | The refused event lands in the dead-letter queue | 1 letter, status 403, `attempts: 3` (= `dead_letter_after`), state `pending` |
968
+ | The push reports the quarantine | `dead_lettered: 1`, `blocked: false` — the queue is no longer stuck |
969
+ | The events BEHIND the refused one are delivered | 54 of 54, in seq order — what the old design lost |
970
+ | The refused event is never counted as delivered | 0 receipts for seq 1, cursor at the tip |
971
+ | The quarantined body is the canonical event body | same type/seq/signature/slot as the journal row |
972
+ | The quarantined body carries its own signature | `stuck:event:1`, HMAC over exactly those bytes |
973
+ | Replay after recovery | `replayed: 1`, `failed: 0` |
974
+ | The replayed event verifies at the receiver | 1 accepted, signature valid |
975
+ | The dead letter is marked replayed | `state: replayed`, `replayed_at` set |
976
+ | **The replay sends byte-identical bytes** | 287 = 287 bytes — a replay is the same event, not a new one |
977
+ | A second replay has nothing left to send | `replayed: 0` |
978
+ | A wrong secret fails the receiver's own check | 55 requests, 0 with a valid signature |
979
+
980
+ Report: `research/dlq-e2e-report.json`.
981
+
982
+ ### The harness can fail
983
+
984
+ A green harness proves nothing if it cannot go red.
985
+ `tools/e2e-webhook-mutations.mjs` mutates the sender in four ways — sign the
986
+ wrong bytes, count a refusal as a delivery, drop the receipt guard, gate the
987
+ retraction behind the blocked queue — and requires the e2e harness to catch
988
+ each one with its named check. **4/4 caught.**
989
+
990
+ `tools/e2e-dlq-mutations.mjs` does the same for the dead-letter harness —
991
+ **4/4 caught**: disable the queue (the refused event never lands and the
992
+ queue dies), count a quarantined event as a delivery, reclassify a 5xx as a
993
+ verdict, and send different bytes on replay.
994
+
995
+ The third mutation earned its keep: the first version of the refusal counter
996
+ selected on a hard-coded `400..500` status range while the classifier
997
+ decides "refusal" independently, so a mutation that reclassified 5xx as a
998
+ refusal produced a dead letter the counter never counted — the harness went
999
+ green over a real defect. The counter now follows the classification (a
1000
+ recorded non-2xx status IS a refusal, by construction of where attempts are
1001
+ recorded), and the mutation turns the harness red as it must.
1002
+
1003
+ ### The dead-letter report must match the queue
1004
+
1005
+ The dead-letter queue is the one place where "empty" means "a consumer is not
1006
+ holding events it was promised". So the report about that queue has to be
1007
+ right, and it was not. Measured 2026-11-09 on the **published v0.7.6
1008
+ binary** (the sha256 of whose release archive equals the digest the release
1009
+ API publishes), against a journal holding planted pending letters and a
1010
+ config with two targets:
1011
+
1012
+ | command | v0.7.6 answered | the truth was |
1013
+ | --- | --- | --- |
1014
+ | `dlq list --target bogus` | exit 0, `{"targets": [], "pending_total": 0}` | 3 pending letters |
1015
+ | `dlq replay --target bogus` | exit 0, `{"replayed": 0, "failed": 0}` | 3 pending letters |
1016
+ | `dlq list --limit 2` | `pending: 2`, `pending_total: 2` | 7 pending letters |
1017
+ | `dlq list` (target removed from config) | `pending_total: 0` | 1 pending letter |
1018
+
1019
+ Four separate lies. A typo in a target id was indistinguishable from a clean
1020
+ queue. `pending` counted **the rows the listing showed**, not the letters that
1021
+ are pending — so `webhook status` said `dead_letters: 7` for the same queue
1022
+ that `dlq list` called 2. And a letter whose target had left the config was
1023
+ invisible: it can never be replayed (no url, no secret) and no command ever
1024
+ said so, which breaks the product's promise that nothing is ever lost.
1025
+
1026
+ The rule now:
1027
+
1028
+ * `--target <id>` must name a target the config knows — an unknown id is a
1029
+ **refusal** that names the flag, the value and the known ids, in both
1030
+ `list` and `replay`;
1031
+ * `pending` and `pending_total` count the letters **pending in the journal**,
1032
+ not the rows a `--limit` happened to show; a truncated listing says
1033
+ `truncated: true`;
1034
+ * a pending letter whose target is not in the config is **reported as an
1035
+ orphan** (`in_config: false`), counted in the total, and keeps `dlq replay`
1036
+ from exiting 0 — a letter nobody can replay is a fact the operator needs,
1037
+ not a number to round to zero.
1038
+
1039
+ Measured on both trees by `tools/measure-dlq-target-domain.mjs`
1040
+ (`research/dlq-target-domain-2026-11-09.json`): the published v0.7.6 asset
1041
+ violates **5 of 8** cases; this tree violates **0 of 8**, with the three
1042
+ guards — a known target is still listed with its letters, a listing without a
1043
+ filter still reports every config target, an empty queue still reads healthy
1044
+ — unchanged on both trees.
1045
+
1046
+ The fix first reached only the branch, so the release that carried it is
1047
+ itself measured: v0.7.7 (linux asset, archive sha256 equal to the digest the
1048
+ release API publishes) violates **0 of 8** against the same baseline
1049
+ (`research/dlq-target-domain-published-v0.7.7-2026-11-09.json`, pinned by
1050
+ `tests/dlq_target_published_v077.rs`), and the same two cases were driven by
1051
+ hand through the empty-app install path (`npm install
1052
+ github:auroraxo/reeflog#v0.7.7` → `bootstrap()` → `reeflog 0.7.7`): an
1053
+ unknown `--target` is a refusal naming the flag, the value and the known
1054
+ targets (exit 1), and `dlq list --limit 2` over a journal holding three
1055
+ letters reports `pending: 3, listed: 2, truncated: true` — the queue, not the
1056
+ page.
1057
+
1058
+ The ladder is `tests/dlq_target_domain.rs` (10 tests) and
1059
+ `tools/mutation-ladder-dlq-target-domain.sh` (**5/5 mutations caught**): the
1060
+ unknown target is accepted again, `pending` counts the rows shown again, an
1061
+ orphaned letter is dropped from the report again, a replay with an orphaned
1062
+ letter reads clean again, and the refusal no longer names the offending
1063
+ value.
1064
+
1065
+ ### The webhook URL domain is the consumer's domain, not a prefix
1066
+
1067
+ Measured 2026-11-09 against the **published v0.7.7 binary** (the sha256 of
1068
+ whose release archive equals the digest published by GitHub Releases):
1069
+ `load_targets` judged the target URL with a string prefix test
1070
+ (`url.starts_with("http://") || url.starts_with("https://")`) — the FORM of
1071
+ a URL, not its meaning. Two faces, both measured:
1072
+
1073
+ | Case | What v0.7.7 did | What the consumer does |
1074
+ |---|---|---|
1075
+ | `"url": "HTTP://127.0.0.1:8794/hook"` | **REFUSED** (`"url" must be http:// or https://`) | `ureq` (the poster's transport) lowercases the scheme: answers **HTTP 200** to `HTTP://` and `HtTp://` from a live receiver |
1076
+ | `"url": "http://"` | **ACCEPTED** (exit 0, report printed, queue configured) | every delivery dies in transport with `http: empty string` |
1077
+ | `"url": "http:///x"` | **ACCEPTED** | every delivery dies with `http: invalid format` |
1078
+ | `"url": "https://"` | **ACCEPTED** | every delivery dies with `http: empty string` |
1079
+
1080
+ The flag parser had already closed this exact defect for the very same
1081
+ consumer (`--rpc`, v0.7.6): scheme checked case-insensitively, host required,
1082
+ refusal naming the offending value. The config loader, one layer up, still
1083
+ used the prefix test.
1084
+
1085
+ The fix is ONE implementation for both layers (`check_url_domain` in
1086
+ `src/flags.rs`), shared by the flag parser and `load_targets`: scheme
1087
+ compared case-insensitively, authority required, and the refusal names the
1088
+ key, the offending value, the expected domain and an example before any
1089
+ delivery is attempted.
1090
+
1091
+ Measured on both trees by `tools/measure-webhook-url-domain.mjs`
1092
+ (`research/webhook-url-domain-2026-11-09.json`, pinned by
1093
+ `tests/webhook_url_measurement.rs`): the published v0.7.7 asset violates
1094
+ **5 of 8** cases; this tree violates **0 of 8**, with the three guards
1095
+ (lowercase `http://`, lowercase `https://`, non-http scheme refused)
1096
+ unchanged on both trees.
1097
+
1098
+ The fix first reached only the branch, so the release that carried it is
1099
+ itself measured: v0.7.8 (linux asset, archive sha256 equal to the digest
1100
+ the release API publishes) violates **0 of 8** against the same baseline
1101
+ (`research/webhook-url-domain-published-v0.7.8-2026-11-09.json`, pinned by
1102
+ `tests/webhook_url_published_v078.rs`, falsified five ways — the count, the
1103
+ verdict, a dropped case id, a source line pointing at the branch, an
1104
+ injected host path).
1105
+
1106
+ The ladder is `tests/webhook_url_domain.rs` (12 tests) and
1107
+ `tools/mutation-ladder-webhook-url.sh` (**5/5 mutations caught**): the host
1108
+ requirement dropped, the scheme check made case-sensitive again, the scheme
1109
+ check dropped entirely, the value no longer named, and `webhook.rs` reverted
1110
+ to the prefix test. The end-to-end case drives an uppercase-scheme target
1111
+ against a local receiver: the URL loads **and delivers** (`delivered: 1`,
1112
+ `blocked: false`, push exit 0), while the no-host URL is refused before any
1113
+ delivery runs.
1114
+
1115
+ ### The webhook config values are judged by the consumer that consumes them
1116
+
1117
+ Measured 2026-11-10 against the **published v0.7.8 binary** (the sha256 of
1118
+ whose release archive equals the digest published by GitHub Releases), one
1119
+ byte or character per delivery, against a receiver that records what
1120
+ actually arrives. The webhook config accepted values the consumer cannot
1121
+ carry:
1122
+
1123
+ * a target `id` with a C0 control byte — accepted (exit 0, report), then
1124
+ **every** delivery died with `http: failed to parse header value`: the id
1125
+ becomes the `X-Reeflog-Event-Id` field value, and ureq builds that field
1126
+ with `http::HeaderValue::from_str`. **32 of the 33** control bytes (all but
1127
+ HTAB) were fatal;
1128
+ * a target `id` with a leading space — accepted and delivered, but the
1129
+ receiver **strips** it: the journal stored ` t1` while the consumer saw
1130
+ `t1:event:1`, so the consumer's deduplication key — the field the README
1131
+ tells consumers to dedup on — stopped matching the sender's record;
1132
+ * a `url` with a raw space, `<`, `>` or a backtick in the path, or one of
1133
+ `" < > \ ^ \` { | }` in the host — accepted, then every delivery died with
1134
+ `http: invalid uri character`;
1135
+ * a `url` with raw non-ASCII in the path — accepted, sent raw, and the
1136
+ receiver answered **HTTP 400**;
1137
+ * a `url` with a non-numeric port (`:abc`, `:99999`, a bare `:`) — accepted,
1138
+ and the consumer **silently dialled 80/443** instead of the configured
1139
+ target.
1140
+
1141
+ The rules now call the consumer's own parser — `http::Uri` and
1142
+ `http::HeaderValue::from_str`, the functions ureq and tokio-tungstenite
1143
+ link (one `http` version in `Cargo.lock`) — so the checker cannot drift from
1144
+ the code that delivers. The `secret` stays a key: it is an HMAC key and
1145
+ never reaches the wire, so only emptiness is out of its domain (measured: a
1146
+ secret with a newline, a control byte or non-ASCII bytes delivers, and the
1147
+ receiver's own HMAC over the received body verifies every time).
1148
+
1149
+ Measured on both trees by `tools/measure-webhook-value-domain.mjs`
1150
+ (`research/webhook-value-domain-before-after-2026-11-09.json`, pinned by
1151
+ `tests/webhook_value_before_after.rs`): the published v0.7.8 asset accepts
1152
+ **11 of 24** sampled values and then fails every delivery with them; this
1153
+ tree refuses all 11 up front and still delivers the 9 that always worked.
1154
+
1155
+ The raw sweeps are in `research/webhook-value-domain-2026-11-09.json`
1156
+ (pinned by `tests/webhook_value_measurement.rs`) and are reproducible from
1157
+ this repository — `tools/probe-webhook-value-domain.mjs` is the probe that
1158
+ produced them. The ladder is `tests/webhook_value_domain.rs` (22 tests) and
1159
+ `tools/mutation-ladder-webhook-value.sh` (**6/6 mutations caught**): the
1160
+ leading-whitespace rule dropped, the header validator judging a constant
1161
+ instead of the operator's id, the internal-space rule widened into an
1162
+ over-refusal, the port-0 rule dropped, the non-ASCII rule dropped, the empty
1163
+ secret allowed.
1164
+
1165
+ ### The refusal voice must be one line of reeflog's own
1166
+
1167
+ Measured 2026-11-10 against the **published v0.7.9 asset** with
1168
+ `tools/probe-error-voice.mjs` (`research/error-voice-*.json`). Every refusal
1169
+ in the seven hostile cases was *correct* — exit 1, nothing on stdout — and
1170
+ the voice was still forgeable, because a refusal names the offending value
1171
+ and named it **raw**:
1172
+
1173
+ * a value carrying a newline split the refusal into **two** lines, the
1174
+ second one starting with `reeflog: error: ` exactly like a real message.
1175
+ The operator's own input — or a config file from anywhere — wrote a line
1176
+ into the log stream that no log reader can distinguish from reeflog's:
1177
+
1178
+ ```
1179
+ reeflog: error: flag --seconds expects a non-negative whole number, got `5
1180
+ reeflog: error: forged` — not a number (e.g. --seconds 10)
1181
+ ```
1182
+
1183
+ * ESC, CR, NUL and DEL reached stderr **raw** — into the terminal (ESC
1184
+ sequences redraw what the operator sees) and into every log file.
1185
+
1186
+ 5 of 7 cases forged a line, 14 control bytes crossed the stream. The fix is
1187
+ one choke point: `die()` escapes the message, so every refusal path gets it
1188
+ (`src/main.rs`). Control bytes become their escape (`\n`, `\r`, `\x1b`,
1189
+ `\x00`, `\x7f`); **HTAB stays literal** — it cannot end a line, so it
1190
+ cannot forge a message, and it is inside the id domain (an HTAB id
1191
+ delivers byte-identical, measured 2026-11-09).
1192
+
1193
+ The verdict never changed: all 8 cases are still refused, and a plain
1194
+ refusal is byte-identical to before. Measured by
1195
+ `tools/measure-error-voice.mjs`
1196
+ (`research/error-voice-before-after-2026-11-10.json`, pinned by
1197
+ `tests/error_voice_measurement.rs`): the published v0.7.9 asset forges **5**
1198
+ lines and leaks **7** raw control bytes over 8 cases; this tree forges **0**
1199
+ and leaks **0**, refusing the same 8. The ladder is `tests/error_voice.rs`
1200
+ (7 tests) and `tools/mutation-ladder-error-voice.sh` (**5/5 mutations
1201
+ caught**): escaping dropped, escaping narrowed to newlines only, ESC left
1202
+ raw, HTAB escaped too (the over-refusal direction), the value no longer
1203
+ named.
1204
+
1205
+ ### One journal, one program
1206
+
1207
+ Measured 2026-11-10 against the **published v0.7.10 asset** (the release
1208
+ archive's sha256 equals the digest the Release API publishes): the rule "a
1209
+ program recorded in the journal is the journal's identity" existed — in
1210
+ exactly ONE command of four. The other three disagreed with it, each in its
1211
+ own direction:
1212
+
1213
+ | command (journal watches `<A>`, caller passes `<B>`) | v0.7.10 answered |
1214
+ | --- | --- |
1215
+ | `init --program <B>` | exit 0, `initialized … for program B` — the mailbox **silently changed identity** |
1216
+ | `verify --program <B>` | exit 0 with a full report — **about A**; the flag was silently ignored |
1217
+ | `repair --program <B>` | the same silent ignore, on the command that **writes** to the journal |
1218
+ | `start --program <B>` | exit 1, `journal watches A, refusing --program B` — the rule defended itself here |
1219
+
1220
+ An operator who re-ran `init` with the wrong id walked away with a journal
1221
+ that reported `program: <B>` while holding A's events. An operator who
1222
+ asked `verify --program <B>` got a verdict about A, with nothing to show
1223
+ that the two differ — on the command whose whole job is to tell the truth
1224
+ about a journal. The rule was real; it was just local to `start`.
1225
+
1226
+ The rule now:
1227
+
1228
+ * a program recorded in the journal **is** the journal's identity;
1229
+ * a command given a different program is **refused**, naming the recorded
1230
+ program, the given program and the command — before anything is scanned,
1231
+ rewritten or opened for writing;
1232
+ * a command given the same program, or none, is served from the recorded
1233
+ one; `init` with the same program stays idempotent.
1234
+
1235
+ Measured on both trees by `tools/measure-program-identity.mjs`
1236
+ (`research/program-identity-2026-11-10.json`, pinned by
1237
+ `tests/program_identity_measurement.rs`): the published v0.7.10 asset
1238
+ violates **3 of 6** cases and changed the journal's identity **1** time;
1239
+ this tree violates **0 of 6**, identity changes **0**. The journal's
1240
+ recorded program is read back from the journal itself, never from the
1241
+ report of the command under test — the report is the thing that lied. The
1242
+ ladder is `tests/program_identity_domain.rs` (7 tests: the three defect
1243
+ faces are red on the pre-fix tree, guards green on both) and
1244
+ `tools/mutation-ladder-program-identity.sh` (**5/5 mutations caught**,
1245
+ tree restored byte-identically).
1246
+
1247
+ **And again on the published v0.7.11 asset**, so the refusal is what a
1248
+ stranger actually installs (`research/program-identity-published-v0.7.11-2026-11-10.json`,
1249
+ pinned by `tests/program_identity_published_v0711.rs`): the published
1250
+ v0.7.10 asset violates **3 of 6** and repoints the journal **1** time, the
1251
+ published v0.7.11 asset violates **0 of 6** and repoints it **0** times —
1252
+ both archives' sha256 equal the digests the Release API publishes
1253
+ (`2f6eded2…` / `f75898ff…`). The pin is falsified two ways
1254
+ (`tools/falsify-program-identity-pin-v0711.sh`: a wrong count and an
1255
+ injected machine path both turn it red, the restored artifact is
1256
+ byte-identical and green again). Installed from an empty directory
1257
+ (`npm install github:auroraxo/reeflog#v0.7.11`), the refusal speaks
1258
+ naming both programs and the command:
1259
+
1260
+ ```
1261
+ reeflog: error: journal …/j.db watches 11111111111111111111111111111111,
1262
+ refusing init --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc —
1263
+ one journal, one program; the recorded program is the journal's identity,
1264
+ pass --db <other-journal> to work on a different one
1265
+ ```
1266
+
1267
+ `init`, `verify` and `repair` all exit 1 on a disagreement;
1268
+ `init` with the recorded program stays idempotent (exit 0).
1269
+
1270
+ ### The dlq replay report describes the queue, not the page
1271
+
1272
+ Measured 2026-11-10 on the **published v0.7.11 binary** (archive sha256
1273
+ `f75898ff9121df1bd280399401893268e161ec95e1af38b0ccabbd656e588ac3` equal
1274
+ to the GitHub Releases digest), against a receiver that answers 200 to
1275
+ everything, on a journal holding 7 pending dead letters:
1276
+
1277
+ ```
1278
+ reeflog webhook dlq replay --db reeflog.db --config wh.json --limit 2
1279
+ ```
1280
+
1281
+ exited **0** with `{"replayed": 2, "failed": 0, "targets": [{"target": "alpha", "replayed": 2, "failed": 0}], "unreachable_orphans": 0}` —
1282
+ while the journal still held **5 pending letters**.
1283
+
1284
+ Nothing in the report named them, no exit code carried them. The report
1285
+ described the page and the command called that the job — the exact class
1286
+ v0.7.7 closed for `dlq list` ("`pending` counted the rows the listing
1287
+ showed, not the letters pending"): the listing got the fix, the command an
1288
+ operator automates did not. A cron job trusting exit 0 ("the command cannot
1289
+ report clean success while events remain un-deliverable in the mailbox" —
1290
+ the rule this very function documents) believed the queue drained while 5
1291
+ owed events sat un-attempted.
1292
+
1293
+ The rule now:
1294
+
1295
+ * `dlq replay` reports `pending_before` (the queue at the start of the pass),
1296
+ `remaining_pending` (what the journal still holds for the targets this pass
1297
+ was responsible for), and `truncated: true` per target;
1298
+ * a pass that leaves letters it never reached pending exits **3** (the same
1299
+ exit-3-means-incomplete contract `tail`/`verify`/`push` use), naming the
1300
+ unreached count and the repair (`run again, or raise --limit`);
1301
+ * delivery itself is unchanged: exactly `limit` letters per target are
1302
+ attempted, byte-identical, marked replayed;
1303
+ * a full drain, and an empty queue, still exit 0.
1304
+
1305
+ Measured on both trees by `tools/measure-dlq-replay-page.mjs`
1306
+ (`research/dlq-replay-page-2026-11-10.json`): the published v0.7.11 asset
1307
+ violates **5 of 5** cases; this tree violates **0 of 5**, with the guards —
1308
+ a full drain still exits clean, an empty queue still reads healthy, delivery
1309
+ itself is unchanged, and `--target` still scopes the pass — unchanged on both
1310
+ trees.
1311
+
1312
+ **And again on the published v0.7.12 asset**, so the rule is what a stranger
1313
+ actually installs (`research/dlq-replay-page-published-v0.7.12-2026-11-10.json`,
1314
+ pinned by `tests/dlq_replay_page_published_v0712.rs`): the published v0.7.11
1315
+ asset violates **5 of 5**, the published v0.7.12 asset violates **0 of 5** —
1316
+ both archives' sha256 equal the digests the Release API publishes
1317
+ (`f75898ff…` / `ab28aac2…`). An undrained pass on the published asset exits
1318
+ **3** and says so:
1319
+
1320
+ ```text
1321
+ reeflog: 5 pending dead letter(s) were not reached by this pass — the queue
1322
+ still holds them; run `reeflog webhook dlq replay` again (or raise --limit)
1323
+ to drain it
1324
+ ```
1325
+
1326
+ The pin is falsified two ways (a tampered count and an injected machine path
1327
+ both turn it red, the restored artifact is byte-identical and green again).
1328
+
1329
+ The ladder is `tests/dlq_replay_page_domain.rs` (8 tests) and
1330
+ `tools/mutation-ladder-dlq-replay-page.sh` (**5/5 mutations caught**): the
1331
+ truncated replay exits 0 again, the remaining queue is dropped from the
1332
+ report again, `pending_before` is dropped again, `truncated` is dropped
1333
+ again, and the refusal voice stops naming the unreached count.
1334
+
1335
+ The measurement is pinned by `tests/dlq_replay_page_measurement.rs` (5 tests,
1336
+ falsified two ways: count tampered → red, host path injected → red).
1337
+
1338
+ ### Limits
1339
+
1340
+ No TLS client authentication and no per-target payload filtering yet; the
1341
+ secret is a shared HMAC key, not a certificate. A target that stays refused
1342
+ used to hold its own queue forever; since the dead-letter queue it holds it
1343
+ for `dead_letter_after` refused attempts, then the event is quarantined and
1344
+ the queue moves on — `webhook status` shows `dead_letters`, `webhook dlq
1345
+ list` shows what is in them, and `webhook dlq replay` hands them over. Other
1346
+ targets always kept moving. Nothing here is an outside user yet: the
1347
+ receiver above is my own test harness, not a third party. The first consumer
1348
+ that is not reeflog lives in its own repository —
1349
+ [auroraxo/whirlpool-watcher](https://github.com/auroraxo/whirlpool-watcher),
1350
+ which reads this mailbox over `tail` and the signed webhook, decodes Whirlpool
1351
+ `Traded` events with a layout it verified against live mainnet itself, and
1352
+ keeps a fill ledger without spending an RPC call (see the qualification list).
1353
+
1354
+ A **second** consumer in a different repository now reads the envelope's
1355
+ instruction bytes: [auroraxo/sememe](https://github.com/auroraxo/sememe)
1356
+ harvests from this mailbox instead of paying `getSignaturesForAddress` /
1357
+ `getBlock` / `getTransaction`, with the zero-RPC claim *enforced* (its
1358
+ `fetch` is replaced by a recording rejector, and the run fails if any call
1359
+ was attempted). Measured 2026-10-31 on a fresh 45-second window (456 events,
1360
+ 452 envelopes held): 156 payloads from 137 transactions with **0 network
1361
+ attempts**; against independently fetched chain transactions, 10/10 sampled
1362
+ payload-bearing transactions **byte-identical** and 10/10 sampled
1363
+ no-target transactions corroborated, 0 different, 0 unavailable. Its own
1364
+ report of the first run was wrong (it labelled v1-transaction rejections as
1365
+ rate limiting) and says so — the retraction and the reader contract (what
1366
+ was read, what was written, which counters reconciled) are in
1367
+ [issue #1](https://github.com/auroraxo/reeflog/issues/1).
1368
+
1369
+ ## Three kinds of gap
1370
+
1371
+ They need different tools, and conflating them is how a "we never lose events" claim turns out to be false:
1372
+
1373
+ - **Tail gap** — events after the newest signature the journal holds (crash, disconnect, restart). Fixed by walking back from the chain tip with `before` until the boundary **slot**. Bounded and cheap. reeflog closes this automatically on `start`.
1374
+ - **Rolling-window hole** — events missing from the middle of the recent history the daemon saw. Repaired by `start` every 30s over a rolling window (default 1800 slots) so rows the WebSocket dropped do not age out unrescued.
1375
+ - **Archive hole** — events missing from history deeper than one tip-walk budget reaches (measured 2026-10-30: 40 pages of 1000 signatures = 40,000 signatures covers only ~17,490 slots / ~2.4h on Raydium AMM v4). A tip-started walk burns its budget crossing history the journal never claimed, reports `truncated: true` and exits 3. `reeflog verify --anchor-slot` and `reeflog repair --anchor-slot` anchor the walk at a finalized block's last signature at the window's ceiling, making 24h-deep (or deeper) holes reachable and verifiable with zero tip-crossing overhead.
1376
+
1377
+ A walk that is cut short is reported as truncated, the gap stays **open**, and the command exits non-zero. A partially filled gap is never presented as filled, and a scan that read nothing cannot report `missing_count: 0` as if it had checked.
1378
+
1379
+ ## The cursor file is read strictly
1380
+
1381
+ The cursor file is state the consumer owns, and it never passes through the
1382
+ flag parser — so the numeric-flag gate (v0.7.2, v0.7.3) never saw it.
1383
+ Measured 2026-11-08 against the published v0.7.3 binary with 20 planted rows,
1384
+ one cursor file per case (`research/cursor-state-2026-11-08.json`):
1385
+
1386
+ | What the existing cursor file held | v0.7.3 did | Now |
1387
+ |---|---|---|
1388
+ | `abc` | exit 0, **all 20 rows replayed**, file overwritten | refused, 0 rows, file untouched |
1389
+ | `9223372036854775808` (outside i64) | exit 0, all 20 rows replayed | refused, 0 rows |
1390
+ | `-1` | exit 0, all 20 rows replayed | refused, 0 rows |
1391
+ | empty (what a truncate-then-crash leaves) | exit 0, all 20 rows replayed | refused, 0 rows |
1392
+ | `15.0` | exit 0, all 20 rows replayed | refused, 0 rows |
1393
+ | **missing** (a first run) | exit 0, 20 rows | unchanged — that is the documented start |
1394
+ | `0` (a deliberate rewind) | exit 0, 20 rows | unchanged |
1395
+ | `15` (resume) | exit 0, 5 rows | unchanged |
1396
+
1397
+ `read_cursor` was `.ok().and_then(parse).unwrap_or(0)`: every way an
1398
+ existing cursor file can fail to carry a cursor collapsed into "start from
1399
+ the beginning", delivered in full, with an exit code that says the run was
1400
+ fine. The consumer's central promise — *resumes exactly where it stopped* —
1401
+ is not a promise a silent restart from 0 can keep. The rule now: a missing
1402
+ file is a first run; an existing file that does not hold a non-negative
1403
+ whole number inside `i64` is a refusal that names the file and its content,
1404
+ before the journal is opened, and the file is left untouched. `reorg tail`
1405
+ reads cursors the same way. Ladder: `tests/cursor_state.rs` (7 defect cases
1406
+ RED on v0.7.3, 6 guards green on both trees), artifact pinned by
1407
+ `tests/cursor_state_measurement.rs`.
1408
+
1409
+ ## A cursor knows where it came from
1410
+
1411
+ A strict read still left the cursor anonymous: one bare number, written by
1412
+ whichever command last ran, with no record of **which journal** or **which
1413
+ channel** it belonged to. reeflog keeps two cursors per journal — `tail`
1414
+ stores an event `seq`, `reorg tail` stores a reorg `id` — and each reader
1415
+ accepted any number from any source. Measured 2026-11-10 against the
1416
+ published v0.7.12 binary, 20 events + 3 reorgs per journal, one crossing per
1417
+ case (`research/cursor-provenance-2026-11-10.json`):
1418
+
1419
+ | The crossing | v0.7.12 did | Now |
1420
+ |---|---|---|
1421
+ | event cursor `20` handed to `reorg tail` | exit 0, `0 reorg(s)` — **3 reorgs silently lost** | refused, 0 rows, file untouched |
1422
+ | reorg cursor `3` handed to `tail` | exit 0, **17 delivered events replayed** | refused, 0 rows |
1423
+ | cursor copied to another journal | exit 0, `0 events after cursor 20` — a fresh mailbox read as drained | refused, 0 rows |
1424
+
1425
+ The rule: a cursor reeflog **wrote** now carries its provenance — one line,
1426
+ `{"v":1,"channel":"events","journal":"3b07e3569b86c5e2","value":20}` — and is
1427
+ honoured only by the same channel of the same journal. Any other use is a
1428
+ refusal that names the file, both sides of the crossing, and the repair; it
1429
+ delivers nothing, exits non-zero, and leaves the file byte-identical so the
1430
+ operator can still see what it said. A **bare** hand-written number stays
1431
+ legal — `0` to restart, `15` to resume, an ahead-of-tip value to wait: it is
1432
+ the documented deliberate rewind, makes no claim about its origin, and a
1433
+ reader that refuses the operator's own numbers is worse than the defect.
1434
+ The journal's id is seeded once at `init` and stored in the journal itself;
1435
+ the summary line still reports the plain number (`cursor now 20`), and the
1436
+ Node client's `cursorValue` reads the position out of either form.
1437
+ Ladder: `tests/cursor_provenance.rs` (4 crossing cases RED on the pre-fix
1438
+ tree, 7 guards green on both), mutations caught by
1439
+ `tools/mutation-ladder-cursor-provenance.sh` (7/7), both trees measured by
1440
+ `tools/measure-cursor-provenance.mjs`.
1441
+
1442
+ ## A failed cursor write never destroys the cursor
1443
+
1444
+ Reading the cursor strictly did not help if the **write** of the cursor
1445
+ could destroy it. `write_cursor` was `std::fs::write`: open with `O_TRUNC`,
1446
+ then write. When that write failed — `ENOSPC`, and this build host hit a
1447
+ genuinely full disk twice in the two days before the fix — the cursor file
1448
+ was left **zero bytes** *after* the rows had already been printed on stdout.
1449
+ Every later run then refused with the reader's own message about an empty
1450
+ file being what a crash between truncate and write leaves behind: the reader
1451
+ named the state correctly, and the **writer** had produced it. The consumer
1452
+ was wedged at exit 1, and the one documented repair (delete the file) means
1453
+ rewinding delivery to the very beginning.
1454
+
1455
+ Measured 2026-11-10 against the published v0.7.13 binary (archive sha256
1456
+ `4a0355f9536e30217b3b0ab2cb1510df1ab1583eafe0e0e8fa0c1cc577399eb6`, equal
1457
+ to the Release-API digest), with an `ENOSPC` fault injected only on the
1458
+ cursor write (`research/cursor-write-durability-2026-11-10.json`):
1459
+
1460
+ | The failed write | v0.7.13 did | Now |
1461
+ |---|---|---|
1462
+ | `tail` with a hand-written `0` cursor fails mid-run | exit 1, cursor file **0 bytes** — the old position gone | exit 1, cursor file still `0`, refusal names file + errno + the redeliver promise |
1463
+ | same face on `reorg tail` | exit 1, truncated | exit 1, unchanged |
1464
+ | first batch, no cursor file yet | exit 1, an **empty file appears** — the wedge | no file created at all |
1465
+ | run again after the failure | refused forever (`empty file … crash between truncate and write`) | **redelivers the same rows** and advances the cursor |
1466
+
1467
+ The rule: the cursor is staged in a sibling temp file, fsynced, and renamed
1468
+ over the cursor path — the file always holds the old full cursor or the new
1469
+ full cursor, never a hole. A failed stage refuses naming the file, the
1470
+ errno, and the contract that makes the refusal safe: *the cursor file is
1471
+ UNCHANGED … the rows printed above were delivered and will REDLIVER on the
1472
+ next run (delivery is at-least-once), so simply run again*. That is honest
1473
+ for the same reason the reader is: the delivery already happened on stdout,
1474
+ so at-least-once redelivery is the truth, and a redelivery the consumer has
1475
+ seen before is strictly better than a silent rewind to zero.
1476
+
1477
+ Ladder: `tests/cursor_write_durability.rs` (4 defect cases RED on the
1478
+ pre-fix tree, 4 guards green on both), mutations caught by
1479
+ `tools/mutation-ladder-cursor-write.sh` (**7/7**), both trees measured by
1480
+ `tools/measure-cursor-write-durability.mjs`; the artifact's verdict is
1481
+ pinned by `tests/cursor_write_durability_measurement.rs`, falsified three
1482
+ ways (`tools/falsify-cursor-write-pin.sh`).
1483
+
1484
+ **And again on the published v0.7.14 asset**, so the rule is what a stranger
1485
+ actually installs (`research/cursor-write-durability-published-v0.7.14-2026-11-10.json`,
1486
+ pinned by `tests/cursor_write_durability_published_v0714.rs`): the published
1487
+ v0.7.13 asset violates **4 of 4** defect cases, the published v0.7.14 asset
1488
+ violates **0 of 4** — both archives' sha256 equal the digests the Release
1489
+ API publishes (`4a0355f9…` / `9b6c3034…`). The pin is falsified three ways
1490
+ (`tools/falsify-cursor-write-published-v0714.sh`: a falsified count, an
1491
+ injected machine path and a fixed seat repointed at the build instead of the
1492
+ release each turn it red; the restored artifact is byte-identical and green
1493
+ again).
1494
+
1495
+ ## A webhook target belongs to its endpoint
1496
+
1497
+ A delivery cursor is a promise to ONE consumer. Until v0.7.15 the journal
1498
+ keyed that promise by the target id alone — measured 2026-11-11 against the
1499
+ PUBLISHED v0.7.14 asset (`tools/measure-endpoint-binding.mjs`,
1500
+ `research/endpoint-binding-published-v0.7.14-2026-11-11.json`): target `t1`
1501
+ delivered 3 events to receiver A; the SAME id repointed to receiver B
1502
+ (200-OK receiver, same secret) delivered **0**, `webhook push` exited **0**
1503
+ reporting `delivered: 0, blocked: false, next_seq: 3`, and `webhook status`
1504
+ printed `delivered: 3` next to B's url while `pending_events: 0`. A consumer
1505
+ that moves — a new port after a restart, a second environment behind the
1506
+ same id — starves behind a green report, and the old consumer's receipts
1507
+ silence every event the new one is owed. This is the same class v0.7.13
1508
+ closed for cursor *files*: a position that does not say where it applies
1509
+ reads as "already delivered" somewhere it never was.
1510
+
1511
+ The rule now: the state row and every delivery receipt carry the endpoint
1512
+ they were written for, and a config whose url differs from the recorded one
1513
+ is **refused before a single byte moves** — exit 1, naming the target id,
1514
+ BOTH endpoints and the repair (give the new endpoint its own target id), on
1515
+ `webhook push`, `webhook status` and `webhook dlq replay` alike. The journal
1516
+ is untouched: the old endpoint's cursors survive the refused pass exactly.
1517
+ A state row written before this rule (no endpoint recorded) is **adopted**
1518
+ by the url it meets first — same url, same cursors, no re-delivery — so an
1519
+ upgrading journal keeps working without ceremony.
1520
+
1521
+ Measured after the fix (`research/endpoint-binding-fixed-2026-11-11.json`,
1522
+ pinned by `tests/endpoint_binding_measurement.rs`, defect faces also pinned
1523
+ as CLI tests in `tests/endpoint_binding_domain.rs`): the published v0.7.14
1524
+ asset starves a repointed consumer (A: 3, B: 0, exit 0); this tree refuses
1525
+ the same pass with exit 1 naming both endpoints and leaves B at 0 — the
1526
+ refusal sends nothing, and nothing inherited was owed to B.
1527
+
1528
+ ### One secret is one consumer, not two
1529
+
1530
+ Measured 2026-11-11 against the **published v0.7.15 release asset**
1531
+ (sha256 `8f941253…`, the Release-API digest) with
1532
+ `tools/measure-secret-domain.mjs` — two VERIFYING live receivers, each
1533
+ answering 200 only when its own documented rule `HMAC(secret, body)`
1534
+ passed, else 401. Two targets (`ta`, `tb`) with DIFFERENT urls sharing ONE
1535
+ non-empty secret — a config the table above used to accept, because the
1536
+ secret is a key and never reaches the wire. Four faces, all violated
1537
+ (`research/secret-domain-published-v0.7.15-2026-11-11.json`, verdict
1538
+ `SIGNATURE-TARGET-AGNOSTIC`):
1539
+
1540
+ * the pair receiver A accepted verifies 3/3 — and it is exactly the pair
1541
+ receiver B computes, because B holds the same key: the signature attests
1542
+ "someone with this secret signed this body", never "signed it FOR me";
1543
+ * A's accepted bytes replayed at B under a **forged** `tb:` event-id header
1544
+ answered **200** from B's own rule, and B recorded the forged id as its
1545
+ own — the delivery-id header that names the consumer is transport
1546
+ metadata outside the signature;
1547
+ * the same replay in the other direction (B's body at A) also answered 200:
1548
+ one secret = *** identity space, both ways;
1549
+ * a body with a forged `seq: 999` header under a valid signature answered
1550
+ 200 — the seq header lies independently of the signature. (The body
1551
+ carries its own `seq`; a receiver that reads `body.seq` instead of the
1552
+ header is immune to this face — that limit stays with the receiver.)
1553
+
1554
+ The **guard** (`tools/measure-secret-domain-guard.mjs`, same published
1555
+ asset, distinct secrets) is 4/4 green: foreign pairs are refused 401 by
1556
+ the receivers' own rules, and the legitimate delivery still gets 200 — so
1557
+ the defect is the shared secret across endpoints, not the wire format.
1558
+
1559
+ The rule now: a config whose targets share a secret while pointing at
1560
+ **different** urls is refused before a byte moves — exit 1, one line,
1561
+ naming both target ids, both endpoints and the repair (`fix: give each
1562
+ endpoint its own secret, or use the same url on both rows`). The refusal
1563
+ never echoes the secret: a key that appears in a refusal lands in logs,
1564
+ terminals and CI output. The same secret on the same url stays legal —
1565
+ one consumer listed twice is nothing a signature can betray.
1566
+
1567
+ Measured after the fix (`research/secret-domain-fixed-2026-11-11.json`,
1568
+ pinned by `tests/secret_domain_measurement.rs`, defect faces also pinned
1569
+ as CLI tests in `tests/secret_domain.rs`): the published v0.7.15 asset
1570
+ violates 4/4 faces, this tree refuses the same config up front
1571
+ (`REFUSED-UP-FRONT`, 0 faces measurable — the door is closed, not the
1572
+ measurement skipped). Mutation ladder
1573
+ `tools/mutation-ladder-secret-domain.sh` catches 9/9, tree byte-identical
1574
+ afterwards.
1575
+
1576
+ ### The tag's own client asked for the PREVIOUS release
1577
+
1578
+ Measured 2026-11-11 on the **published v0.7.16 tag tree**
1579
+ (`research/client-pin-lag-published-v0716-2026-11-11.json`, verdict
1580
+ `defect-present-on-baseline`): the v0.7.16 release commit moved Cargo.toml,
1581
+ Cargo.lock and the install pins above — and forgot `package.json`, which
1582
+ still read `"version": "0.7.15"`. `bootstrap()` derives the release tag it
1583
+ requests from the package manifest (that coupling is
1584
+ `test/bootstrap.test.mjs` B8), so a stranger who followed this README
1585
+ (`npm install github:auroraxo/reeflog#v0.7.16`) installed a client that
1586
+ asked the release API for **v0.7.15** and downloaded the **v0.7.15**
1587
+ archive — the binary that still carries the shared-secret defect
1588
+ v0.7.16 fixed. The tag the README pinned and the binary a stranger got
1589
+ were two different releases; nothing on CI could see it, because the
1590
+ manifest was the lying side.
1591
+
1592
+ The measurement runs each tree's own `index.js` against a stand-in release
1593
+ API and records the request it actually makes (not what it should make):
1594
+ the published v0.7.16 tree requests `v0.7.15`
1595
+ (`servesThePinnedRelease: false`), this tree requests the release it names
1596
+ (`servesThePinnedRelease: true`); the tool fails if the baseline shows no
1597
+ defect or the fixed tree still does. Gate: `test/bootstrap.test.mjs` B10
1598
+ ties `package.json` to `Cargo.toml` (semver shape + equality) — green here,
1599
+ red on the published tag tree. Mutation ladder
1600
+ `tools/mutation-ladder-client-pin-lag.sh` catches 5/5 (lag, lead, bad
1601
+ shape, crate-bumped-manifest-forgotten, pre-release), tree byte-identical.
1602
+ The artifact is pinned by 5 tests, falsified by a tampered number and by a
1603
+ host path.
1604
+
1605
+ ## Repeated use
1606
+
1607
+ ```bash
1608
+ reeflog init --db reeflog.db --program <pubkey>
1609
+ reeflog start --db reeflog.db --seconds 60 [--repair-every 30] \
1610
+ [--repair-lookback 1800] [--repair-pages 40] \
1611
+ [--payload-limit 500]
1612
+ reeflog seal --db reeflog.db
1613
+ reeflog verify --db reeflog.db --pages 400 [--anchor-slot <slot>] # detect, exit 3 if unfinished
1614
+ reeflog repair --db reeflog.db --pages 400 [--anchor-slot <slot>] # fill signatures verify found
1615
+ reeflog hydrate --db reeflog.db [--limit 500] # fetch missing event content
1616
+ reeflog hydrate --db reeflog.db --accounts # + account envelopes for socket rows
1617
+ reeflog hydrate --db reeflog.db --owners # + owning program per address (batched)
1618
+ reeflog tail --db reeflog.db --cursor ./c.txt # deliver events (content included)
1619
+ reeflog webhook push --db reeflog.db --config webhooks.json # deliver to outside targets
1620
+ reeflog status --db reeflog.db
1621
+ ```
1622
+
1623
+ `tools/e2e-repair.mjs` runs detect → repair → prove against live mainnet on a
1624
+ journal with planted holes and fails if any step misreports; it passed on
1625
+ 2026-10-28 with 333 planted holes: verify saw 333, repair filled 333, the
1626
+ second verify returned `missing_count: 0, complete: true`, and a deliberately
1627
+ truncated scan exited 3.
1628
+
1629
+ `tools/e2e-archive.mjs` proves archive-anchored gap fill on live mainnet
1630
+ at 24h depth (~216,000 slots back, well past a tip-walk's 17.5k-slot budget):
1631
+ tip-started verify exits 3 refusing to claim completeness, anchored verify
1632
+ sees all 39 planted holes, anchored repair fills all 39, second verify
1633
+ returns `missing_count: 0, complete: true`, and `--anchor-slot` without value
1634
+ anchors at the journal's ceiling. Report: `research/archive-e2e-report.json`.
1635
+
1636
+ The raw report of the confirmation kill test is committed at
1637
+ `research/killtest-full-after-fix-2026-10-28.json` (compact form: the
1638
+ per-signature loss samples are truncated, every number and every check verdict
1639
+ is the harness's own).
1640
+
1641
+ ## Free and paid
1642
+
1643
+ - Free, MIT: daemon, journal, CLI, the Node client (below), webhook fan-out, dead-letter queue, archive backfill. No event or cursor limits.
1644
+ - Paid later: managed always-on feeds, replicated journals with delivery objectives, and an enterprise mailbox with access control and a delivery audit trail. Events are never sold individually.
1645
+
1646
+ ## The Node client: the exit-code contract, as a library
1647
+
1648
+ The CLI is a contract before it is a convenience: exit 0 delivers and
1649
+ advances the cursor, exit 3 (on `tail`) means the delivery **held** at a row
1650
+ whose account envelope is still owed — rows before it are delivered, nothing
1651
+ at or after it is consumed — and exit 1 is a refusal. A consumer that maps
1652
+ those three outcomes wrongly either replays rows forever or drops the owed
1653
+ row without ever seeing it. Until now every Node consumer had to derive that
1654
+ by hand.
1655
+
1656
+ `index.js` (npm package `reeflog-client`, MIT, zero dependencies, Node ≥ 18)
1657
+ ships that mapping as one file. It never opens the journal — every call runs
1658
+ the `reeflog` binary, so there is exactly one implementation of the reader
1659
+ contract, and a library's idea of the schema cannot drift from the daemon's.
1660
+
1661
+ ```bash
1662
+ npm install reeflog-client # the registry; this version, no Rust toolchain
1663
+ npm install github:auroraxo/reeflog#v0.7.21 # the same client from the tagged release
1664
+ ```
1665
+
1666
+ The client needs the `reeflog` binary. `bootstrap()` gets it **without a
1667
+ Rust toolchain** — it downloads the prebuilt archive the release already
1668
+ ships for your platform (linux-amd64, macos-arm64), verifies its bytes
1669
+ against the sha256 the GitHub release API stores for that asset, and only
1670
+ then installs it to `~/.reeflog`:
1671
+
1672
+ ```js
1673
+ import reeflog, { bootstrap } from 'reeflog-client';
1674
+
1675
+ const { bin, digest, binarySha256 } = await bootstrap();
1676
+ // digest = sha256 of the ARCHIVE (what the release API stores)
1677
+ // binarySha256 = sha256 of `bin` itself — check this against the file
1678
+ const box = reeflog({ db: 'reeflog.db', bin });
1679
+ ```
1680
+
1681
+ - **Refuses instead of trusting.** A download whose sha256 does not match
1682
+ the release digest is refused and nothing is written — no binary, no
1683
+ sidecar. A release asset with no digest is refused too.
1684
+ - **A cache hit is verified, not assumed.** The sidecar (`install.json`)
1685
+ records the extracted binary's own `binarySha256`; a second call re-hashes
1686
+ the file on disk and only reports `cached: true` when it matches. Before
1687
+ v0.7.1 the cache path compared only version + asset name, so a replaced or
1688
+ truncated binary came back as `cached: true` under a digest that named the
1689
+ ARCHIVE, not the bytes on disk (measured: claimed `f6b36c53…`, actual
1690
+ `6b9fd8b7…`). A sidecar without `binarySha256` — every install made before
1691
+ v0.7.1 — is re-fetched once to upgrade it, then caches again. The check
1692
+ defends against a stale or corrupted cache being reported as verified; an
1693
+ attacker who can write your install directory can simply re-run
1694
+ `bootstrap()` afterwards, so it is not a defence against a hostile local
1695
+ user. Live measurement: `research/bootstrap-cache-2026-11-07.json`.
1696
+ - **A platform without a published archive is refused by name** (the
1697
+ refusal lists what exists and gives the `cargo install` fallback), rather
1698
+ than fetching a file that is not there.
1699
+ - **The default tag comes from the package version**, never a hard-coded
1700
+ string, so the client can never fetch an older release than the package
1701
+ it ships in.
1702
+ - **A stalled or looping download gets a verdict, not silence.** Every
1703
+ request `bootstrap()` makes carries a 30 s inactivity budget (tune with
1704
+ `{ timeoutMs }`): no response headers in time, or no bytes between two
1705
+ chunks for that long, destroys the socket and rejects with a named refusal
1706
+ — an install must never hang without an answer. Redirects are followed at
1707
+ most 5 hops; a loop is refused by name. Before v0.7.17 a server that
1708
+ accepted the connection and never answered left `await bootstrap()`
1709
+ hanging forever, and a self-referential 302 re-requested itself 11,727
1710
+ times in 8 s (measured on the published v0.7.17 client:
1711
+ `research/bootstrap-hang-2026-11-11.json`).
1712
+ - **An answer that is not a release gets a verdict in the product's voice.**
1713
+ The release API can answer `200` with something that is not a release
1714
+ object: a captive portal's login page (HTML), a corporate proxy, a
1715
+ rate limiter's JSON. Before v0.7.19 the HTML case threw a raw
1716
+ `SyntaxError` out of `bootstrap()` — a crash no `ReeflogError` catch ever
1717
+ saw — and the rate-limit case was refused as `release v0.7.18 has no
1718
+ asset named …`, which sends the stranger to a release page that is fine.
1719
+ Now both refuse with the endpoint's OWN words: the URL that answered, the
1720
+ fact that it answered HTML, and the API's message when it sent one. A
1721
+ release object that genuinely lacks this platform's asset keeps its
1722
+ honest `has no asset named` refusal (measured on the published v0.7.18
1723
+ client: `research/captive-portal-2026-11-11.json`).
1724
+ - **Trust is the same anchor as `cargo install --git`**: this repository
1725
+ over TLS. It protects against a corrupted or substituted download, not
1726
+ against whoever can push tags and releases here. If you want to be
1727
+ independent of that, compare `digest` against the asset's published
1728
+ sha256 yourself.
1729
+ - Measured end to end against the published release —
1730
+ `research/bootstrap-2026-11-07.json` (install + cached re-run + real
1731
+ rows delivered through the installed binary) and
1732
+ `research/bootstrap-cache-2026-11-07.json` (fresh install, verified cache
1733
+ hit, tampered binary, pre-v0.7.1 sidecar upgrade, other-platform asset).
1734
+
1735
+ ```js
1736
+ import reeflog from 'reeflog-client';
1737
+
1738
+ const box = reeflog({ db: 'reeflog.db' }); // bin on PATH, or { bin: '/path/to/reeflog' }
1739
+ const s = await box.status(); // parsed health: events, envelope debt, gaps
1740
+
1741
+ const { events, held, cursorValue } = await box.tail({
1742
+ limit: 500,
1743
+ consumer: 'my-service', // separate default cursor per consumer
1744
+ withAccounts: true, // hold rather than deliver an unreadable row
1745
+ });
1746
+ if (held) {
1747
+ // exit 3: `held.seq` and `held.signature` name the owed row; the rows
1748
+ // above it are already delivered and the cursor already advanced past
1749
+ // them. Pay the debt (`reeflog hydrate --accounts`) and call again — the
1750
+ // held row and everything behind it arrive next, nothing was lost.
1751
+ }
1752
+
1753
+ const { retractions } = await box.reorgs({ consumer: 'my-service' }); // own cursor channel
1754
+ await box.verify({ pages: 400 }); // incomplete scan THROWS (exit 3): no clean-looking result
1755
+ await box.hydrate({ accounts: true });
1756
+ ```
1757
+
1758
+ The rules the client enforces instead of leaving to the caller:
1759
+
1760
+ - **exit 0 vs exit 3 are different answers.** `tail` returns
1761
+ `{ events, held, cursorValue }`; `held: null` is the only shape that means
1762
+ "the batch is fully delivered". `verify`/`repair`/`hydrate` throw on exit 3,
1763
+ because a pass that proves nothing must never arrive as a resolved value.
1764
+ - **exit 1 keeps the binary's words.** The thrown `ReeflogError` carries the
1765
+ CLI's own refusal (`no journal at … — run \`reeflog init …\``), plus
1766
+ `exitCode`, `stderr` and the `args`, so a caller can act on it rather than
1767
+ on a rephrased guess.
1768
+ - **Only the documented codes are verdicts.** The typed API accepts 0
1769
+ (delivered), 1 (refused) and 3 (held/incomplete). Every OTHER close is a
1770
+ refusal carrying the REAL code and the binary's last words: the
1771
+ journal-write failure's `2`, an old binary's usage `64` (client/binary
1772
+ version skew), a panic's `101`, and a kill by signal (the process died, so
1773
+ there is no exit code at all — the refusal names the signal instead of
1774
+ inventing one). Before v0.7.20 the client treated "not 1 and not 3" as
1775
+ success, so a run that died AFTER printing its rows resolved as a
1776
+ delivered batch (`held: null` = "fully delivered"), an old binary's usage
1777
+ error resolved as a clean empty batch — and on the JSON commands it
1778
+ crashed out of `JSON.parse` with a raw `SyntaxError` no `ReeflogError`
1779
+ catch ever saw — and a SIGKILL'd run was dressed up as a forged `exit 1`
1780
+ refusal naming no signal. Measured on the published v0.7.19 client:
1781
+ `research/client-exit-voice-2026-11-12.json` (6/6 defect faces on the
1782
+ baseline, 6/6 refused afterwards). `raw()` stays the escape hatch: there,
1783
+ any code comes back as `{ exitCode }`.
1784
+ - **Cursors are the binary's.** The client only chooses a default path per
1785
+ db + channel + consumer label, so two consumers of one journal never share
1786
+ a cursor by accident; pass `cursor` to name it yourself.
1787
+ - **A missing binary says how to install one** — naming `bootstrap()` (no
1788
+ Rust toolchain) and the `cargo install` fallback — instead of crashing
1789
+ with `ENOENT`.
1790
+
1791
+ The contract is tested against the real binary and a real journal in
1792
+ `test/client.test.mjs` (`npm test`, 8 tests): delivery order, no replay
1793
+ across a resumed cursor, the hold at an owed row (delivered before, not
1794
+ consumed at/after, delivered once the debt is paid), separate cursor
1795
+ channels, refusals with their fix line intact, and the incomplete-verify
1796
+ throw. CI runs them on every push.
1797
+
1798
+ Webhook fan-out ships in this tree: `reeflog webhook push` / `status`, signed
1799
+ bodies, per-target durable cursors, in-order at-least-once with receipts —
1800
+ plus the dead-letter queue (`reeflog webhook dlq list` / `replay`) for a
1801
+ target that refuses one event instead of holding its queue forever. Archive
1802
+ backfill (`verify --anchor-slot` / `repair --anchor-slot`) ships as well,
1803
+ anchored at finalized blocks for holes deeper than the tip-walk window.
1804
+
1805
+ ## Evidence and limits
1806
+
1807
+ Checked on 2026-10-23: Solana client errors already name the failure modes, including closed WebSockets, missing subscription IDs, and corrupt iterators. Yellowstone is a strong live transport but explicitly ephemeral. The closest general-chain project, Chainhook, is archived and does not cover Solana. One developer rebuilt reliable replay into Postgres as a four-star personal pipeline.
1808
+
1809
+ reeflog does not recover account bytes from before an event, and the account
1810
+ envelope is balance MOVEMENTS an event caused — not the state of the accounts
1811
+ involved. `tail --with-accounts` does not skip past a row whose envelope has
1812
+ not been fetched yet: it holds the cursor there and exits 3 until
1813
+ `hydrate --accounts` pays the debt, so an owed row can never be lost to the
1814
+ consumer. Rows the chain itself made unfillable are the only ones skipped,
1815
+ and each skip is reported. That is a different problem.
1816
+ Public WebSocket endpoints are also rate-limited, so provider independence
1817
+ has to be real rather than assumed.
1818
+
1819
+ Ownership is the partial exception: `hydrate --owners` fills an `account_owners`
1820
+ map (owning program per address), and changed token accounts name their token
1821
+ program for free. Two limits remain and are stated rather than smoothed over:
1822
+ owners are read **at fetch time**, not at event time (an account may have
1823
+ changed owner since), and account **bytes / pre-event state** are still not in
1824
+ the envelope at all — that stays open, see *Next qualification* item 4.
1825
+
1826
+ The account BYTES are now in the envelope too (`hydrate --account-data`),
1827
+ which closes one half of that sentence and not the other: the bytes say what
1828
+ an account held **at fetch time**, not what it held when the event happened.
1829
+ 16–18 % of the addresses a transaction enumerates no longer exist by the
1830
+ time the pass runs (measured on two live windows), and a hot token
1831
+ account's balance moves between two reads. Historical state — the bytes AS
1832
+ OF the event — remains open, and no envelope field claims otherwise.
1833
+ Measured 2026-11-02 (`tools/probe-pre-event-bytes.mjs`,
1834
+ `research/pre-event-bytes-probe-2026-11-02.json`): both free routes to
1835
+ pre-event bytes are dead — a past-slot `getAccountInfo` parameter answers
1836
+ anchored at the current context slot (requested 464542126, got 464542734),
1837
+ and predecessor transactions carry instruction bytes but no account-state
1838
+ payload (0 across 6 scans) — so the boundary is "no data source on this RPC
1839
+ class", not "a second fetch" and not "the chain's history". What history
1840
+ does prove, measured on the same window: 7/60 touched accounts had no
1841
+ touch after the event, for which fetch-time bytes ARE the post-event bytes
1842
+ (witness: lamports 4/4, token mint/owner/amount 2/2; wrong-side guard on
1843
+ moved accounts 0/3; 3 certified accounts already gone at fetch time are
1844
+ counted, never witnessed). That certification is checked after the fetch,
1845
+ not before: a certification can expire mid-run (an earlier
1846
+ classify-then-fetch run showed exactly one such disagreement,
1847
+ root-caused) — and it is measured, not shipped: no envelope field claims
1848
+ it, and it covers the post-event side only.
1849
+
1850
+ Still open: Yellowstone as a second transport, and a real consumer other
1851
+ than my own two repositories (the Whirlpool watcher and sememe). Reorg handling marks a
1852
+ retracted event, delivers the retraction, runs the check inside the daemon,
1853
+ and joins retractions against the fill watcher's ledger — and it has now been
1854
+ proven against a REAL fork of a real chain, not only against a planted
1855
+ signature. What remains unproven is the check's reach on mainnet: it still
1856
+ looks only at sealed rows near the tip, and no mainnet fork has been observed
1857
+ end to end.
1858
+
1859
+ ## Next qualification
1860
+
1861
+ 1. ~~Put my own RFQ fill polling and Whirlpool watcher on `reeflog tail`.~~ **User zero exists**: [auroraxo/whirlpool-watcher](https://github.com/auroraxo/whirlpool-watcher) (2026-10-31) is a real out-of-tree consumer — separate repository, no reeflog code linked, the `Traded` decode verified from live mainnet independently. Measured against a direct-RPC poller on the same window: 7/7 checks, including `window_superset` (0 in-window trades missing), `decoded_amounts_match` (two independently written decoders agree) and `chain_witness` (every sampled fill's amounts appear as real token-balance deltas); the watcher made **0 RPC calls** for 138 fills where the poller spent **85 calls** for 19 trade-bearing transactions. Its mutation probe goes red as required. What remains out of scope: this consumer is still the mailbox owner's own (user zero, not user one), and a third party adopting it is what would close this item.
1862
+ **Correction, measured 2026-10-31**: the second half of this item as originally written ("my own RFQ fill polling has not been moved over yet") named work that does not exist. A scan of all 28 repositories of mine on this machine plus the home directory for an RFQ fill poller found none — no such script, service or file exists. The premise behind that sentence was the falsified "21 own key-load sites" claim from 24.10, the same category of error: a sentence that repeated a plausible-sounding inventory without checking it against the disk. The honest remainder of this item is the one that was always the real gate: **a third party using reeflog**, not the migration of a poller nobody ever wrote.
1863
+ **A second out-of-tree consumer landed the same day** (2026-10-31): [auroraxo/sememe](https://github.com/auroraxo/sememe) harvests the envelope's `instructions[]` instead of paying RPC (0 enforced attempts; 10/10 sampled transactions byte-identical against independently fetched chain data, 10/10 no-target corroborated, mutation red at the named check), its reader contract and one retracted claim of its own in [issue #1](https://github.com/auroraxo/reeflog/issues/1). Two consumers, two different byte surfaces (logs/fills and instruction bytes), both in my own repositories — still no third party.
1864
+ **The install blocker is gone (2026-11-01, v0.1.0)**: before this release a third party could not adopt reeflog at all — there was no tag, no release, no install path, and the binary could not even say which version it was. `v0.1.0` is now a real release (`cargo install --git https://github.com/auroraxo/reeflog --tag v0.1.0`, per-platform archives attached to the [release](https://github.com/auroraxo/reeflog/releases/tag/v0.1.0)), the binary reports its own version, and the release workflow refuses to publish a tag whose version the binary disagrees with. Proven from a stranger's empty directory: installed from the tag, then journaled 535 live mainnet events with 0 gaps. That removed the *install* blocker; the *adoption* blocker (an unaffiliated third party) is untouched and remains the real remainder of this item.
1865
+ **Distribution opened, adoption instrumented (2026-11-02)**: the first earned placement is submitted as [StockpileLabs/awesome-solana-oss#93](https://github.com/StockpileLabs/awesome-solana-oss/pull/93) (Data section, per that list's CONTRIBUTING; the list merges about half of what outsiders send — 24 merged vs 22 closed at submission time, so a merge is not assumed). And "no third party" is no longer a sentence in a README but a measured verdict: `node tools/measure-adoption.mjs` classifies external use from non-owner stars/forks/issues/PRs, non-GitHub referrers and the live state of the placement PR. Baseline committed in `research/adoption-2026-11-02.json`: 0 non-owner stars, 0 forks, 0 external issues/PRs, 0 external referrers, 14-day views 0/0 — verdict `not-distributed-not-consumed` (the 356 clones in the same window are our own CI checkouts, explicitly not an adoption signal). Controls: the probe flips to `user-one-evidence` on a repo with real outsiders (100 non-owner forks, outsider issues), and prints `NO VERDICT` with exit 3 when a data source is partial — a merge alone will read as `distributed-not-yet-consumed`, never as adoption.
1866
+ 2. Yellowstone source as a second transport.
1867
+ 3. Observe a real **mainnet** fork end to end: the detector is now proven
1868
+ against a real fork of a real chain (`tools/e2e-fork.mjs`, 47/47 checks,
1869
+ 2026-10-29) and against a planted signature on live mainnet — but no
1870
+ mainnet fork has been caught while watching. The daemon's loop makes that
1871
+ plausible to catch; only an observed one closes it.
1872
+ 4. ~~An account envelope says what an event *did* to balances… The next envelope would carry the account list a transaction touched — addresses, owners, writable flags — so a consumer can decide which accounts it cares about without a second RPC round trip.~~ **Landed 2026-10-31**: `accounts[]` carries every account a transaction touched with `signer`, `writable` and `source`, derived from the message header and the two loaded groups (or stated by the response where it says so), with `accounts_flags_source` recording which. Falsified live against `jsonParsed`'s authoritative flags: 346/346 keys over 8 transactions including lookup tables, 0 mismatches; the loaded path is pinned offline by a real transaction fixture.
1873
+ **Ownership landed 2026-11-01**: `hydrate --owners` resolves the owning program of every address the envelope enumerates (batched `getMultipleAccounts`, 12 calls for 1,111 addresses in the live e2e), and changed token accounts name their token program for free inside the transaction response. Falsified live: 412/412 stored owners agree with an independent re-fetch, 57/57 addresses the chain no longer has are recorded in an explicit `missing` list with **no** owner entry (never guessed), and the probe pins the authority three ways (stated vs decoded from the account's own bytes vs the map): 12/12.
1874
+ **Account bytes landed 2026-11-02**: `hydrate --account-data` stores what each enumerated account HELD — the bytes, not just the owner — batched (11 calls for 964 distinct addresses in the live e2e), capped at 16 KiB per address with the real size and a verified truncation flag, program code flagged instead of stored, and absence recorded as `missing`. Premise measured before building (`tools/probe-account-data.mjs`, 428 distinct addresses over 30 live transactions): p50 165 B, p95 4483 B, 5 addresses over the cap, and a 1,048,588-byte slab — which is why the cap exists and why program binaries are not journalled as event data. Falsified live: 258/258 stored sizes equal the chain's byte length; 232 of 258 re-reads byte-identical, 17 live-moving, 4 in the mutable region of a classic token account, 5 unattributed (no immutability claimed — reported, not waved through); the mint/owner decoded from the stored bytes matched the transaction's own token metadata 12/12; 82/82 program accounts flagged byte-free with correct sizes; 47/47 recorded-missing addresses confirmed absent on an independent read; a second run stored nothing and cost 0 calls.
1875
+ **What remains open is the half that is genuinely different**: account bytes AS OF the event (historical state), not bytes at fetch time. That is a different problem, and it was last written here as "needs the chain's history, not a second fetch" — **measured 2026-11-02, that sentence was wrong in its remedy**: the history has no account bytes to serve (6 predecessor transactions, 0 account-state payloads), and a past-slot parameter on a public RPC answers with current state, not the requested slot. Pre-event bytes need a data source this RPC class does not offer; the two dead routes and the measured boundary are in *Evidence and limits* (`tools/probe-pre-event-bytes.mjs`). The same probe found the one thing history does prove — for 7/60 touched accounts with no later touch, fetch-time bytes ARE the post-event bytes (witness 4/4 lamports, 2/2 token fields, wrong-side guard 0/3; 3 certified accounts already gone stay counted but unwitnessed) — measured, deliberately not shipped as an envelope field, and it certifies the post-event side only.
1876
+ **Released as v0.2.0 (2026-11-02)**: this owners + account-bytes envelope is the release. The historical-state half above remains the open item it was the day before.