reeflog-client 0.7.24 → 0.7.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +167 -1856
  2. package/package.json +1 -4
package/README.md CHANGED
@@ -1,1649 +1,98 @@
1
1
  # reeflog
2
2
 
3
- A durable local mailbox for Solana events. A daemon journals logs and what
4
- they did to account balances for selected programs into a cursor-addressed
5
- append-only log, so a consumer resumes without gaps after a crash,
6
- disconnect, or reorg.
3
+ **A durable local mailbox for Solana events.** A daemon subscribes to the
4
+ programs you pick, journals every log (and what it did to account balances)
5
+ into an append-only SQLite log with one monotone `seq`, and lets a consumer
6
+ resume *exactly where it stopped* — after a crash, a disconnect, or a reorg.
7
7
 
8
- **Status: v0.7.24 shipped (2026-11-13), v0 working.** Live against mainnet-beta: see *Measured results* below. Install the binary without a Rust toolchain: `npm install reeflog-client` then `import { bootstrap } from 'reeflog-client'; await bootstrap()` (downloads the prebuilt release archive and verifies its sha256; the archive digests are `6cea050d…` linux / `d3cbe390…` macos, both equal what the release API publishes). The same install from the tagged release: `npm install github:auroraxo/reeflog#v0.7.24`. Or from source: `cargo install --git https://github.com/auroraxo/reeflog --tag v0.7.24`. The full event envelope (logs, balances, instructions, account list with owners, the bytes each account held, and the authority each token account had at the moment of the event), the Node client (see *The Node client*), webhook fan-out, a dead-letter queue, and archive backfill ship in this tree; Yellowstone as a second transport does not.
8
+ Solana's `logsSubscribe` is a live firehose: no delivery guarantee, no
9
+ durable cursor, no catch-up. Every project rebuilds reconnect, dedup,
10
+ gap-fill, and reorg handling, or silently loses events. reeflog makes that
11
+ one daemon, one file, one cursor.
9
12
 
10
- ## The idea
13
+ **Status: v0.7.25 (2026-11-13), live against mainnet-beta — no Rust toolchain needed: `npm install reeflog-client`, then `import { bootstrap } from 'reeflog-client'; await bootstrap()` (downloads the prebuilt release archive and verifies its sha256 against what the GitHub release API publishes). The client itself is zero-dependency: one package, no `node_modules` of its own, and it installs without `git` on the machine.**
11
14
 
12
- Solana event consumption is normally a live firehose. `logsSubscribe` has no delivery guarantee, no durable cursor, and no catch-up after disconnection. Every project rebuilds reconnect, deduplication, gap-fill, and reorg handling, or it silently loses events.
15
+ The same client from the tagged release:
16
+ `npm install github:auroraxo/reeflog#v0.7.25`. Or from source:
17
+ `cargo install --git https://github.com/auroraxo/reeflog --tag v0.7.25`.
13
18
 
14
- reeflog treats events as a mailbox instead of a phone call:
19
+ **Honest state:** it works, and it is measured — but so far every real
20
+ consumer belongs to the project itself. There is no unaffiliated third-party
21
+ user yet, and this README calls that what it is. The full measurement history
22
+ lives in [docs/measured-history.md](https://github.com/auroraxo/reeflog/blob/main/docs/measured-history.md).
15
23
 
16
- - **ingest:** subscribe to configured programs over WebSocket.
17
- - **seal:** record events as confirmed, promote them when finalized, mark failures as errored.
18
- - **gap-fill:** catch up on signatures the journal missed before resuming. A gap must be reported; silent loss is a protocol error.
19
- - **journal:** append-only SQLite with one monotone `seq` and a portable file.
20
- - **deliver:** a cursor-based CLI iterator that resumes exactly where it stopped.
24
+ ## Who this is for
21
25
 
22
- ## Shape
26
+ - You run a bot, indexer, or watcher against specific Solana programs and
27
+ cannot afford to silently miss a fill — or must know a fill was retracted.
28
+ - You are tired of rebuilding reconnect + gap-fill + dedup per project.
29
+ - You want the event record on *your* disk: a portable SQLite file you can
30
+ verify (`reeflog verify`) instead of trusting a hosted stream.
23
31
 
24
- ```bash
25
- reeflog init --db reeflog.db --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc
26
- reeflog start --db reeflog.db --seconds 60 # ingest over WS
27
- reeflog seal --db reeflog.db # promote confirmed -> finalized
28
- reeflog tail --db reeflog.db --cursor ./c.txt # resume from a cursor
29
- reeflog verify --db reeflog.db # find holes
30
- reeflog status --db reeflog.db
31
- ```
32
-
33
- The planned core is a Rust daemon and a TypeScript client. Decoding stays pluggable; reeflog does not itself decide what bytes mean.
34
-
35
- **First-touch contract (measured 2026-11-02, `tests/quickstart.rs` +
36
- `tests/rpc_timeout.rs`):** `init` is the only command that creates a journal,
37
- and it refuses a `--program` that is not a base58 32-byte address — a typo
38
- here used to be accepted silently and produced a mailbox that watches
39
- nothing while reporting perfect health (`0 events`, `0 gaps`, exit 0).
40
- Every other command refuses a `--db` path that has no journal, with the
41
- `reeflog init …` line as the fix, instead of conjuring an empty one.
42
- Every RPC call is bounded (10 s per call, two attempts for a stalled peer; a
43
- 429 still gets the full six-attempt backoff). Degraded is not the same as
44
- hung: when the endpoint accepts connections and never answers, `start` can
45
- exceed its `--seconds` while the shutdown passes pay their bounded timeouts,
46
- but it always returns and names the failure on stderr (measured 2026-11-02:
47
- pre-fix, 4 of 5 repetitions of `start --seconds 3` never returned at 45 s;
48
- post-fix, worst observed run returned in 83 s with `rpc call timed out
49
- twice` reported).
50
-
51
- **A flag the CLI does not know is refused, not ignored** (measured 2026-11-06,
52
- `tests/arg_parsing.rs` + `tests/tool_invocations.rs`,
53
- `research/flag-surface-2026-11-06.json`). Until now a typo in a flag was
54
- silently DROPPED and the command ran anyway — the most expensive kind of quiet
55
- failure this product can make, because the tool reported doing something it
56
- did not do:
57
-
58
- | Before | What actually happened |
59
- |---|---|
60
- | `reeflog verify --db reeflog.db --pagesx 400` | scanned with the **default** page budget (40), not 400, and reported it as if it had |
61
- | `reeflog status --dbx nope.db` | read the **default** `reeflog.db` and printed a healthy status for it |
62
- | `reeflog status --db reeflog.db --bogus-flag` | printed a status, output identical to a correct run |
63
- | `reeflog init --db probe.db --program <pk> --nope` | created the journal anyway |
64
- | `reeflog init --db probe.db` (no `--program`) | created the journal FIRST, refused SECOND — a mailbox left behind |
65
- | `reeflog status --db=other.db` | ignored the inline form and read the default journal |
66
- | `reeflog status --help` | ran `status` and printed a health report instead of the usage |
67
-
68
- Now every command declares the flags it accepts, and anything else is
69
- refused **before** any journal is opened or created:
70
-
71
- - an unknown flag is refused, and when the typo is close to a real flag the
72
- likely intent is named (`unknown flag --pagesx (did you mean --pages?)`);
73
- a far-away typo gets no guess, because a wrong suggestion is worse than none;
74
- - a flag that lost its value is refused — including when the "value" is
75
- another flag (`--db --program` must not become a path called `--program`);
76
- - a flag given twice is refused rather than silently resolved;
77
- - a numeric flag whose value is not a non-negative whole number is refused,
78
- naming the flag and the value (`flag --limit expects a non-negative whole
79
- number, got `abc``) — before any journal is opened. Until 2026-11-08 an
80
- unparseable value silently became the DEFAULT: `tail --limit abc` delivered
81
- every row, and `tail --limit -1` delivered every row *twice as unbounded*,
82
- because SQLite reads `LIMIT -1` as "no limit". `0` stays legal (several
83
- flags document it as "disable this pass"), and the bare `--anchor-slot`
84
- keeps its documented meaning. Measured against the published v0.7.1 binary:
85
- 4/10 cases behaved as specified, 2 "refusals" delivered 650 rows anyway —
86
- `research/numeric-flag-values-2026-11-08.json`;
87
- - a numeric flag whose value is a number the CONSUMER cannot hold is also
88
- refused, naming the flag and the value (`flag --limit expects a
89
- non-negative whole number no greater than 9223372036854775807, got
90
- `18446744073709551615``) — and a cadence in seconds must be one this
91
- clock can schedule. "A non-negative whole number" is not one number
92
- space: until 2026-11-08 the checker accepted `u64` while consumers parse
93
- `i64` or add to a clock, so `tail --limit 18446744073709551615` delivered
94
- the DEFAULT limit (1000 rows, exit 0), `verify --since-slot
95
- 9223372036854775808` dropped the window and scanned a different one, and
96
- `start --seconds 18446744073709551615` panicked while `start
97
- --repair-every …` hung. Measured against the published v0.7.2 binary:
98
- 4/11 cases behaved as specified (2 silent defaults, 3 panics, 1 hang) →
99
- 11/11 — `research/numeric-flag-domain-2026-11-08.json`;
100
- - a URL flag whose value is not a URL the CONSUMER can speak is also
101
- refused, naming the flag and the value (`flag --ws expects a ws or wss URL
102
- — got `https://…``) — before any journal is opened. Until 2026-11-09
103
- `--ws` and `--rpc` were the only value flags with no domain at all: on the
104
- published v0.7.5 binary `start --ws ftp://x/y` exited 0 with a journal it
105
- had just created (0/11 cases behaved as specified), `--ws https://…` ran
106
- its full duration and recorded a gap for a scheme tungstenite can never
107
- speak, and `verify --rpc not-a-url` failed only after opening the journal,
108
- in ureq's words (`bad uri: …`) rather than reeflog's. The domains are the
109
- consumers', not an opinion: `ureq` resolves `http`/`https` only,
110
- `tokio-tungstenite` accepts `ws`/`wss` only — and both compare schemes
111
- case-insensitively, so `WS://…`, which the consumer accepts, is accepted
112
- here too. Measured against the published v0.7.5 binary: 0/11 refused →
113
- 11/11, 6/6 guards green on both trees —
114
- `research/url-flag-domain-2026-11-09.json`;
115
- - an argument that is not a flag is refused;
116
- - `--flag=value` is honoured instead of falling back to the default;
117
- - `--anchor-slot` keeps its documented **optional** value (bare = anchor at
118
- the journal's own window), and `--help` prints the usage instead of running
119
- the command.
120
-
121
- `reeflog status` now names the journal it reports on (`"db": …`), so a report
122
- can be checked against what was actually asked for. The false-positive guard
123
- matters as much: `tests/tool_invocations.rs` pins every invocation shape the
124
- `tools/e2e-*.mjs` harnesses pass, so a strict parser cannot quietly break the
125
- project's own measured runs.
126
-
127
- ## Measured results
128
-
129
- Checked live on 2026-10-27 against `api.mainnet-beta.solana.com`, program `whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc` (Whirlpool):
130
-
131
- | Measurement | Result |
132
- |---|---|
133
- | WS ingest | 3122 events in 75 s, 0 disconnects |
134
- | Coverage in the claimed slot window | 224 / 226 chain signatures = **99.12 %** |
135
- | Tail gap after a restart | 9 / 9 deleted events refilled |
136
- | Middle hole detection | 17 missing reported = 15 deleted + 2 baseline, exactly |
137
- | Cursor resume | no replay, monotone advance, no overlap |
138
- | Unit tests | 58 / 58 |
139
-
140
- Kill test (600 s on Raydium AMM v4, SIGKILL every 60 s): see *The kill test*
141
- below — the first run **falsified** the delivery claim; the fixes are now
142
- re-verified in the **same 600 s configuration** that falsified it
143
- (`full-after-fix`, verdict `contract_survived`).
144
-
145
- ## The kill test
146
-
147
- `tools/killtest.mjs` is a falsifying harness, not a demo. It runs three
148
- consumers against the same program for 600 s:
149
-
150
- - **reeflog** — SIGKILLed every 60 s, restarted after 4 s of downtime.
151
- - **naive-killed** — a bare `logsSubscribe` consumer, killed the same way.
152
- - **naive-live** — a bare `logsSubscribe` consumer, **never killed**: the
153
- control that separates "the kill caused it" from "the endpoint drops events
154
- anyway".
155
-
156
- It then fetches the chain's own signature list for the window independently
157
- and holds it against every consumer's seen-set: a truth signature that *some*
158
- consumer received on the log stream is deliverable, so missing it is a real
159
- loss; a signature nobody on the stream received is classified with
160
- `getTransaction` instead of assumed — only a transaction that actually
161
- invokes the program (`invoked`) can count as a loss, never a
162
- `referenced-only` one.
163
-
164
- Run `sync` (2026-10-28, same harness, 120 s / 4 kills on Raydium AMM v4)
165
- re-tested the claim and it **survived**: every check passed, verdict
166
- `contract_survived` (exit 0).
167
-
168
- | Check | Result |
169
- |---|---|
170
- | reeflog real losses | **0** of 1,112 deliverable events |
171
- | naive-killed (control) real losses | **169** — the kill alone loses events |
172
- | naive-live (control, never killed) real losses | **3** — the public WS drops events without any kill |
173
- | verify before repair | 14 missing, `complete: true`, exit 0 |
174
- | repair | `found_missing: 14`, `filled: 14`, `complete: true` |
175
- | verify after repair | `missing_count: 0`, `complete: true`, exit 0 |
176
- | dedup | 1,215 rows = 1,215 distinct signatures |
177
- | cursor resume | second iteration: 0 rows (no replay) |
178
- | incomplete scan | never exits 0 |
179
-
180
- Run `full-after-fix` (2026-10-28, 600 s / 9 kills on Raydium AMM v4 — the
181
- exact configuration that falsified the claim, re-run against the fixed
182
- binary) **survived**: verdict `contract_survived`, exit 0, every product check
183
- passed. This is the confirmation run the falsification owed.
184
-
185
- | Check | Result |
186
- |---|---|
187
- | reeflog real losses | **0** of 5,374 deliverable events |
188
- | naive-killed (control) real losses | **455** across 9 kill windows |
189
- | naive-live (control, never killed) real losses | **4** — the public WS drops events without any kill |
190
- | verify before repair | 1 missing, `complete: true`, exit 0 |
191
- | repair | `found_missing: 1`, `filled: 1`, `complete: true` |
192
- | verify after repair | `missing_count: 0`, `complete: true`, exit 0 |
193
- | dedup | 5,636 rows = 5,636 distinct signatures |
194
- | cursor resume | second iteration: 0 rows (no replay) |
195
- | live repair during the run | 19 gaps reported, `gaps_open: 0`, 380 backfilled |
196
-
197
- Run `full` (2026-10-28, the earlier 600 s run) **falsified** the same claim
198
- and is kept here because it is what exposed the defects:
199
-
200
- | Check | Result |
201
- |---|---|
202
- | reeflog lost deliverable events | **32 real losses** of 11,275 deliverable |
203
- | naive-live lost deliverable events | 158 — the public WS drops events even with no kill |
204
- | `reeflog verify` | reported `missing_count: 0`, `truncated: true`, **exit 0** |
205
-
206
- Two defects, both measured against the RPC and fixed:
207
-
208
- 1. **Phantom anchor.** The journal's newest held signature was a
209
- confirmed-then-forked-out transaction. `getSignatureStatuses` returns
210
- `null` for it, and `before=<it>` makes `getSignaturesForAddress` answer
211
- `-32020 Transaction not found` for the whole walk. verify therefore read
212
- **zero pages** and printed `missing_count: 0` — a hole detector that goes
213
- blind exactly when holes exist. Every walk now starts at the chain tip and
214
- pages only with signatures the chain itself returned; the tail stop is on
215
- the boundary **slot**, not the boundary signature.
216
- 2. **Nothing repaired mid-window holes.** The tail backfill only reaches
217
- forward from the newest held slot. Events the socket dropped *while
218
- connected* sit behind that frontier — all 32 losses did — and nothing ever
219
- filled them. `start` now runs a bounded repair pass over a rolling window
220
- (default every 30 s, 1800 slots back) and once more at shutdown.
221
-
222
- Both are covered by regression tests, and the honesty rule is explicit:
223
- `Report::complete()` requires no error, no truncation, at least one page read,
224
- **and** measured coverage of the claimed window (first row at or above the
225
- ceiling, last row at or below the floor). An unfinished scan exits 3 and can
226
- never be mistaken for a clean one.
227
-
228
- ## RPC semantics, measured
229
-
230
- `getSignaturesForAddress` does not behave the way the parameter names suggest. Measured with an anchor at the 4th of 10 tip signatures:
231
-
232
- | Parameter | Observed behaviour |
233
- |---|---|
234
- | `before=<sig>` | returns signatures **older** than the anchor — the backwards-paging primitive |
235
- | `until=<sig>` | returns signatures **newer** than the anchor |
236
- | `since=<sig>` | behaves like `until` — **newer** |
237
- | `before` + `until` together | rejected with `-32020 Transaction not found` for an inverted window |
238
- | `before=<phantom>` | `-32020 Transaction not found` — a held signature can be a transaction the address index no longer knows |
239
- | `until=<phantom>` | same `-32020`; neither parameter tolerates a phantom anchor |
240
- | `since=<slot>` | silently **ignored** by the public endpoint (a typo'd parameter returns the same rows), so a slot-bounded walk cannot be expressed at all |
241
- | `getSignatureStatuses` | returns `null` for a phantom, a status for a real signature — the reliable way to tell them apart |
242
- | throughput | 19 of 40 consecutive `limit=1000` pages answered HTTP 429, ~0.63 pages/s sustained |
243
-
244
- Consequence: a backfill written with `since=<newest signature we hold>` returns an empty window and reports no loss while filling nothing. reeflog pages backwards with `before` and stops at the boundary **slot** instead. The default walk starts from the chain tip so no held signature is ever an anchor; `--anchor-slot` starts it from a signature the CHAIN returned (the last signature of a finalized block), which is the same rule from the other side. This was found by measuring the endpoint, not by reading the docs, and it is recorded in `src/backfill.rs`.
245
-
246
- ## Event content (payload hydration)
247
-
248
- A signature alone is not an event — a mailbox that delivers "something
249
- happened at slot N" without saying what happened forces every consumer back
250
- onto the RPC, which is the fragility reeflog exists to remove.
251
-
252
- The gap machinery (`backfill.rs`) recovers *signatures* from
253
- `getSignaturesForAddress`, which carries slot and error but no `logs`.
254
- Measured 2026-10-28 on a 60 s live run against mainnet: 13 rows filled by
255
- `repair`, all 13 stored with `logs IS NULL`, while all 2,082 rows the socket
256
- delivered held payload. That asymmetry meant the rows the journal had to
257
- *recover* were exactly the rows a consumer could not read.
258
-
259
- reeflog hydrates missing content (`src/payload.rs`) with `getTransaction` and
260
- distinguishes four states:
261
-
262
- - **`held`** — log lines stored as a JSON array. Both socket capture and
263
- hydration store into this state; hydration never overwrites captured logs.
264
- - **`unavailable`** — the chain does not know the transaction (forked out or
265
- pruned). Keeps `logs = null` and records the reason; an empty array would
266
- claim "no logs", which is a different statement.
267
- - **`invalid`** — the endpoint refuses the key (-32602). A corrupted row,
268
- recorded separately from a fork-out so an operator can tell them apart.
269
- Never folded into `unavailable`.
270
- - **`pending`** — still unknown. Any pending row makes a pass incomplete;
271
- incomplete passes exit 3 (same honesty rule as `verify` and `repair`).
272
-
273
- `reeflog tail` delivers the log array by default; `--no-logs` keeps the
274
- compact identity-only shape. Since v0.7.22 the default (payload-bearing)
275
- tail shares the delivery rule the accounts filter already had: a row the
276
- payload queue still owes (`logs IS NULL AND payload_state IS NULL`) HOLDS
277
- the delivery — rows before it are handed over, the cursor stays behind the
278
- owed row, tail exits 3 naming `hydrate`, and the same cursor reads the row
279
- complete once the debt is paid. Measured on the pre-fix tree (v0.7.21):
280
- tail consumed a pending row, hydrate filled it, and the next pass reported
281
- `0 events after cursor 2` — the payload existed in the journal and never
282
- reached the consumer that had already been walked past it. Recorded
283
- outcomes (`unavailable` / `invalid`) are still delivered in-line with their
284
- marker — they are answers, not debts — and `--no-logs` callers are never
285
- held (they asked for identities only). A background worker hydrates
286
- recovered rows continuously, beside the ingest loop — not once at startup,
287
- because the periodic repair pass keeps recovering rows while the daemon
288
- runs, and a killed run never reaches a shutdown pass. A final pass runs at
289
- shutdown. Verified live: `tools/e2e-payload.mjs` checks all 13 conditions
290
- against mainnet, including independent byte-for-byte comparison of stored
291
- payloads against freshly fetched transactions.
292
-
293
- ## What the event did to accounts
294
-
295
- Until 2026-10-31 the opening line of this README claimed the daemon journals
296
- "account changes" — and `grep -ci account src/*.rs` returned 0 across every
297
- source file. The claim was checked against the code for the first time in
298
- the feature's own commit message, and the code was brought up to the claim
299
- in the same commit.
300
-
301
- The reason this is nearly free: hydration already pays for the whole
302
- `getTransaction` response and kept only `meta.logMessages`. Measured with
303
- `tools/probe-accounts.mjs` on live mainnet (Raydium AMM v4, 4 transactions,
304
- 2026-10-31): the discarded remainder of that same response averages 11.9 KB
305
- per transaction and carries ~33.5 account keys, ~5.5 lamport-delta accounts
306
- and ~3.5 changed SPL token balances per transaction. `src/accounts.rs`
307
- extracts that into an envelope stored alongside the logs — at no extra RPC
308
- cost for rows the journal had to recover.
309
-
310
- The envelope contains: `lamport_changes` (index, address, before, after,
311
- delta), `token_changes` (index, address, mint, owner, before, after,
312
- delta) — amounts as exact decimal strings, never floats — plus
313
- `instructions` and `err`.
314
-
315
- It also carries **`token_state`** (added 2026-11-02): the event-time
316
- authority of every token account the transaction touched whose amount did
317
- NOT change — `index`, `address`, `mint`, `authority`, `token_program`,
318
- `source: "tx-time"`. Until then the envelope stated the authority only for
319
- accounts whose balance moved, and dropped the rest. That is the majority
320
- of what a transaction touches: measured on live mainnet (Raydium AMM v4,
321
- 10 transactions, `tools/probe-tx-time-authority.mjs`) **103 of 125 token
322
- accounts (82.4 %)** carry an unchanged amount, and all 103 carry an
323
- event-time authority the previous shape discarded. Changed accounts are
324
- not duplicated — they keep their authority in `token_changes[].owner`, and
325
- the two lists never claim the same account (pinned by
326
- `a_changed_token_account_belong_to_token_changes_not_token_state`).
327
-
328
- It also carries **the account list itself** (`accounts`, added 2026-10-31):
329
- every account the transaction touched, in the same canonical order, with
330
- `signer` and `writable` permissions and a `source` of `static`,
331
- `lookup_writable` or `lookup_readonly`. Until then the envelope said *what
332
- changed* and *how many* accounts there were, but not *which ones* or *whether
333
- the transaction could write to them* — a consumer that needed those had to
334
- pay for a second round trip to the chain.
335
-
336
- Where the permissions come from is stated, not hidden, in
337
- `accounts_flags_source`:
338
-
339
- - `stated` — the response already carries them (the object form of
340
- `accountKeys` states signer/writable per static key);
341
- - `header` — derived from the message header's three counts (writable signed,
342
- readonly signed, writable unsigned, readonly unsigned) plus the two loaded
343
- groups. Measured before landing against `jsonParsed`'s authoritative flags
344
- over 346 keys across 8 live mainnet transactions including two with address
345
- lookup tables: 346/346 agreement, 0 mismatches, static-prefix order
346
- identical. A real lookup-table transaction is pinned as
347
- `test/fixtures/account-list-lookuptable.json` and re-checked offline on
348
- every run (`the_derived_account_list_equals_the_authoritative_flags_of_a_real_lookup_table_transaction`);
349
- - `unavailable` — the header is missing or its counts cannot describe the
350
- static keys, so `accounts` is `null` and the permissions are **refused**
351
- rather than guessed. A guessed permission is worse than an absent one: a
352
- consumer that trusts `writable` would act on it.
353
-
354
- `instructions` is the transaction's instruction bytes: every outer
355
- instruction in execution order, then the CPIs each one spawned (`inner:
356
- true`, tagged with the parent's index in `outer` and their own position in
357
- `inner_index`), each with the resolved `program_id`, resolved `accounts`,
358
- and the verbatim base58 `data` exactly as the RPC returned it — reeflog
359
- never re-encodes bytes it did not produce. Account indices resolve through
360
- the same canonical list the balances use (so a lookup table cannot shift an
361
- argument to the wrong address); an index that does not resolve becomes an
362
- explicit `null` at its own position, because position is information and a
363
- shifted list would put an argument under the wrong name downstream. `err`
364
- is `meta.err` verbatim (`null` = success), so a consumer never has to guess
365
- whether a failed transaction's envelope describes an effect that happened.
366
-
367
- The envelope does NOT contain pre-event account state; see *Evidence and
368
- limits*. What an account HELD at fetch time is the section below.
369
-
370
- ### What an account held (the bytes)
371
-
372
- Ownership answers WHO controls an account; the bytes answer WHAT it held —
373
- and both are outside the `getTransaction` response, so the bytes are their
374
- own explicitly-budgeted pass: `hydrate --account-data` (after
375
- `--accounts`, like `--owners`). It batch-fetches `getMultipleAccounts`
376
- (100 addresses per call) for every address the held envelopes enumerate and
377
- stores `account_data` inside each envelope. Every address gets exactly one
378
- of three answers:
379
-
380
- - **`held`** — the account's data bytes (base64), capped at 16384 B per
381
- address. The cap is a measured defense: on live mainnet (Raydium AMM v4
382
- window, 2026-11-02, `tools/probe-account-data.mjs`) account sizes sit at
383
- p50 165 B / p90 1024 B / p95 4483 B — but the tail is real (a
384
- 1,048,588-byte Serum/OpenBook slab, a 262,156-byte one) — and the
385
- endpoint's `space` field reports the true size under the slice, so the
386
- row records `bytes` (the real size) and `truncated`/`truncation_verified`
387
- instead of pretending the cut is the whole. The probe semantics were
388
- pinned live first: `space` reports the full size even when sliced
389
- (1,048,588 reported as 16384 delivered), and an at-cap probe
390
- (offset=cap, length=1) returns the account when more data exists and
391
- `null` when it is gone-or-exact — an ambiguity the envelope reports as
392
- `truncation_verified: false` rather than guessed.
393
- - **`missing`** — the chain does not have the account at fetch time (the
394
- same rule as the owner map: absence is a fact, recorded, never guessed).
395
- - **`skipped_executable`** — the account is program code (same window: the
396
- second-largest accounts were 105 KB / 74 KB of SPL program binaries).
397
- Journaling program bytes as event data would be wrong in a different way
398
- than truncating state is, so they are delivered as a flag plus their
399
- true size, byte-free.
400
-
401
- Measured on live mainnet (Raydium AMM v4, 2026-11-02, 20 s window, 119
402
- rows): **14 calls for 1,235 distinct addresses**; sizes agree with an
403
- independent re-fetch on **252 / 252**; the 81 executable accounts were all
404
- flagged byte-free with correct sizes; 43 recorded-missing addresses
405
- confirmed absent on an independent read; mint/owner decoded from the
406
- stored bytes matched the transaction's own token metadata **35 / 35**.
407
-
408
- Storing bytes honestly means admitting what a re-read can and cannot
409
- prove: these are LIVE accounts, and between reeflog's fetch and any later
410
- read a hot SPL token account's `amount` legitimately moves (measured in
411
- the second e2e window: 39 of 252 re-reads differed this way). The e2e harness
412
- therefore classifies differences instead of demanding byte-frozen
413
- equality: identical, live-moving (a third read also differs),
414
- volatile-region of a classic 165-byte token account (mint/owner immutable,
415
- amount/delegate/close-authority are not), and — for layouts that grant no
416
- immutable claim (mints, program state) — **unattributed**, which is
417
- reported as a count and never waved through as a pass. The product claims
418
- are anchored where time does not matter: completeness (every address
419
- answered), true size (`space`), tx-consensus (decoded immutable fields vs
420
- the transaction's own metadata), and absence-not-guessed.
421
-
422
- The bytes are read AT FETCH TIME, not at event time — measured on the
423
- probe window, 16–18 % of enumerated addresses no longer existed by the
424
- time the pass ran. The envelope's slot and the `source` tag say when the
425
- facts were true; historical state (the bytes AS OF the event) remains
426
- open. Its boundary was measured on 2026-11-02 (`tools/probe-pre-event-bytes.mjs`,
427
- report `research/pre-event-bytes-probe-2026-11-02.json`, 60 touched
428
- accounts over 10 live transactions):
429
-
430
- - **A past-slot question does not reach the past.** `getAccountInfo` with
431
- `minContextSlot` set to the event's slot answered with the current state
432
- and no error (requested slot 464542126, response context 464542734) —
433
- the parameter is a floor, not a time machine, and a caller who reads it
434
- as one silently receives today's bytes.
435
- - **The transaction history carries no account bytes either.** 6 nearest
436
- predecessor transactions were fetched and walked: 0 account-state
437
- payloads anywhere in the responses; the 56 `data` fields found are
438
- instruction bytes, not account state. The "recurse over predecessor
439
- transactions" route this item carried is refuted, measured — the
440
- history cannot serve what it does not contain.
441
- - **What the history does certify is the other direction.** For 7 of the
442
- 60 touched accounts nothing touched them after the event, so the bytes a
443
- fetch returns now ARE the post-event bytes; the event's own post-state
444
- witnesses that claim — lamports 4/4, token mint/owner/amount decoded
445
- from the fetched bytes 2/2 — and on the 3 witnessed accounts the event
446
- had moved, the pre-side witness agreed 0/3, which is what keeps the
447
- check from being vacuous (wrong-side guard). 3 more certified accounts
448
- were already gone by fetch time and stay counted but unwitnessed.
449
- Certification is checked AFTER the fetch for a reason: an earlier
450
- classify-then-fetch run hit the race it implies (1 witness disagreement,
451
- root-caused to an account touched between the two calls), so the probe
452
- re-checks the boundary post-fetch and counts expiring certifications
453
- instead of hiding them.
454
-
455
- Pre-event bytes therefore need a data source this RPC class does not
456
- have (geyser with history, snapshot replay, an indexer) — not a second
457
- fetch and not the chain's history, both measured here rather than argued.
458
- The certification half is measured and NOT built into the envelope: no
459
- field claims it.
460
-
461
- ### Which program owns what
462
-
463
- Ownership answers a different question than the ones above — *what program
464
- controls this account* — and unlike everything else in the envelope, `owner`
465
- is **not** in the `getTransaction` response (measured 2026-11-01 with
466
- `tools/probe-owners.mjs`: the response names a token account's `programId`
467
- inside `pre/postTokenBalances`, but nothing states the owner of an arbitrary
468
- account). So reeflog treats it as its own budget rather than pretending it is
469
- free:
470
-
471
- - **Free tx-time half:** every changed token account and lamport-change
472
- entry of a token account carries the SPL token program the transaction
473
- itself named (`token_program` / `owner_program` on the delta entry), read
474
- from the same response hydration already pays for.
475
- - **Paid owner map:** `hydrate --owners` collects every address the held
476
- envelopes enumerate **across all owed rows first**, then batch-fetches
477
- `getMultipleAccounts` (100 addresses per call, owner-only dataSlice) and
478
- stores `account_owners` inside each envelope: an `owners` map for addresses
479
- the chain still has, and an explicit `missing` list for addresses it does
480
- not.
481
-
482
- Measured on live mainnet (Raydium AMM v4, 2026-11-01):
483
-
484
- | Measurement | Result |
485
- |---|---|
486
- | Probe: distinct addresses → batched calls | 406 addresses, 5 calls (291× vs 1 per address) |
487
- | Probe: addresses gone at fetch time | 63 / 406 = **15.5 %**, recorded as missing |
488
- | Probe: authority agreement (stated vs decoded from bytes vs map) | **12 / 12** token accounts, all three paths |
489
- | E2E: stored owners vs an independent re-fetch | **412 / 412** agree |
490
- | E2E: missing addresses re-fetch as null **and carry no owner** | **57 / 57** |
491
- | E2E: batched cost | **12 calls for 1,111 distinct addresses** |
492
- | E2E: tx-time token program vs chain | **20 / 20** statable entries |
493
- | E2E: `owner_program` vs re-fetched account owner | **13 / 13** |
494
- | Idempotence | second run: 0 rows, 0 calls |
495
- | Unit tests | **97 / 97** |
496
-
497
- Three honesty rules the implementation is built against, each pinned by a
498
- test and by the live harness:
499
-
500
- - **A gone address is a fact, never a guess.** ~15 % of addresses no longer
501
- exist when owners are fetched (accounts closed after the transaction).
502
- They land in `missing` with **no** owner entry — writing `null`, omitting
503
- the key, or filling in a plausible program would all be indistinguishable
504
- from "fetched" to a consumer.
505
- - **No half-written rows.** A row is written only when *every* address it
506
- enumerates has an answer; a failed batch leaves the row owed instead of
507
- storing a map with a silent hole in it.
508
- - **Owners are read at fetch time, not at event time.** An account can change
509
- owner after the transaction; the map's `source` says which pass wrote it,
510
- and nothing here claims historical state (see *Evidence and limits*).
511
- The exception that is not an exception: for token accounts the
512
- transaction itself touched, the envelope NOW carries the event-time
513
- authority (`token_state[].authority` / `token_changes[].owner`, read from
514
- the transaction's own `preTokenBalances` — the chain's statement at event
515
- time, measured to agree with a later fetch-time read 124/124 in the same
516
- probe window, and irreplaceable for the 1 of 125 accounts that no longer
517
- exist at all). Historical state for accounts a transaction did NOT touch
518
- remains open, unchanged.
519
-
520
- The two pre-existing correctness traps above — lookup-table indexing and a
521
- closed token account — apply to the owner map for the same reason: it is keyed
522
- by the canonical address list, and the negative control in the probe proves an
523
- absent account produces no entry rather than a fabricated one.
524
-
525
- Two correctness traps the extractor is built against, each pinned by a test:
526
-
527
- - **Lookup-table indexing.** `preBalances`/`postBalances` are indexed over
528
- static keys ++ loaded writable ++ loaded readonly, while a versioned
529
- transaction's `message.accountKeys` lists only the static keys. Indexing
530
- balances by that field silently attaches every changed balance to the
531
- wrong address. The account list is rebuilt in canonical order instead
532
- (`a_lookup_table_does_not_shift_the_balances_onto_the_wrong_addresses`).
533
- - **A closed token account.** An account present in `preTokenBalances` and
534
- absent from `postTokenBalances` closed mid-transaction. The first live
535
- cross-check caught exactly this: the delta was encoded as an empty string.
536
- Absence on one side means zero on that side, so the delta is `-before`
537
- (`a_closed_token_account_reports_the_full_negative_delta`).
538
-
539
- Delivery and enrichment:
540
-
541
- - `hydrate --accounts` fetches envelopes for rows whose logs came from the
542
- WebSocket (the `logsNotification` carries no balances). Same honesty
543
- contract as payload: a rate limit leaves the debt, exit 3 reports it; the
544
- chain forgetting a transaction is recorded as `account_state`
545
- `unavailable`, never a pending-forever row and never an empty envelope.
546
- - `tail` delivers the envelope as `accounts` on every row; `account_state`
547
- says why one is missing. `tail --with-accounts` delivers only rows whose
548
- envelope is complete, and **holds** at the first row the debt queues still
549
- owe: the cursor stays behind it, the hold and the filling command
550
- (`hydrate --accounts`) are printed, and the command exits 3 — the same
551
- honesty contract as `hydrate` itself. Only a row with a recorded outcome
552
- (`unavailable`/`invalid`/`no_metadata`), which can never be filled, is
553
- skipped, reported, and consumed.
554
- - `webhook push` bodies carry the envelope in the same signed delivery.
555
- - `status` counts `accounts_held` / `accounts_missing` /
556
- `accounts_unreachable`.
557
-
558
- Verified live (`tools/e2e-accounts.mjs`, mainnet, 22/22): envelopes from the
559
- free path and the paid path, idempotence, delivery shape, cursor semantics,
560
- and a field-by-field cross-check of every stored envelope — balances,
561
- instructions and outcome — against an independent re-fetch and re-parse of
562
- the same transactions from the chain. Two claims do not depend on live
563
- traffic, because a sample of healthy transactions cannot show them: a local
564
- stub serves a transaction with two CPIs (the parent tag and the position
565
- inside it are pinned) and a failed transaction (`InstructionError [0,
566
- {Custom: 42}]` must arrive verbatim; on successful transactions the live
567
- comparison is `null === null` and would stay green if the field were
568
- dropped). Against the pre-fix binary (741b622) the same harness reports
569
- 16/19 — exactly the instruction checks go red, and the 16 checks that
570
- existed before stay green on both trees.
571
-
572
- `tools/e2e-accounts-mutations.mjs` mutates the extractor six ways (address
573
- shift, dropped delta, empty delta, wrong mint, instruction data shifted, CPI
574
- position dropped) and each mutant must fail the live harness — a green
575
- harness that cannot go red proves nothing. The probe counts only exit 1 (a
576
- named check failed) as a catch: exit 2 is the harness failing to set up
577
- against the rate-limited public endpoint, and counting that as a catch
578
- would report a mutant as caught by a verdict that never ran.
579
-
580
- ## First external consumer: a fill watcher
581
-
582
- Until this section existed, every claim was checked by reeflog's own
583
- harness. `tools/fill-watcher.mjs` is a consumer in the sense a bot cares
584
- about: it reads `reeflog tail --cursor` and turns rows into **fills** —
585
- signature, slot, direction, `amount_in`, `out_amount`, pool reserves — and it
586
- does that from local bytes, with no RPC call of its own.
587
-
588
- The decode is not guessed. Raydium AMM v4 emits `ray_log:` as a
589
- bincode-serialized struct (`SwapBaseInLog` / `SwapBaseOutLog` in
590
- `raydium-amm/program/src/log.rs`), so the field order is the program's own.
591
- The wire layout was then verified against the chain: for every sampled swap,
592
- `pool_coin + amount_in` equals the pool's coin-vault balance after the swap
593
- and `pool_pc − out_amount` equals its pc-vault balance, both read from the
594
- transaction's pre/post token balances. A line the decoder cannot read is
595
- reported as `undecodable`, never guessed at.
596
-
597
- `tools/e2e-fillwatch.mjs` runs this watcher against the alternative — the
598
- classic `getSignaturesForAddress` + `getTransaction` poller — over the **same
599
- window on the same endpoint**, and fails if:
600
-
601
- 1. `hydrate` leaves any row unreadable (`pending` must be 0, exit 0);
602
- 2. any fill the poller decoded **inside the daemon's slot window** is
603
- missing from reeflog's stream;
604
- 3. a fill both consumers saw decodes to **different** amounts (a payload that
605
- exists but disagrees with the chain is not delivery);
606
- 4. the watcher's own decoder drops a held leg;
607
- 5. the local decode phase makes any RPC call.
608
-
609
- The comparison is **slot-aligned**, and the reason is worth stating: a naive
610
- page fetch reaches back ~1000 signatures into history, while the daemon's
611
- window starts when its WebSocket subscription delivers its first event. The
612
- first run of this harness compared the raw sets and reported 21 "missing"
613
- fills — every sampled one sat in a slot *before* the journal's first slot,
614
- i.e. before the daemon had connected. That is not a loss, and treating it as
615
- one would be exactly the kind of plausible-wrong-number this project refuses
616
- to publish. Fills outside the window are now counted and reported
617
- separately, never as losses.
618
-
619
- Measured 2026-10-28, 60 s live window on mainnet, Raydium AMM v4:
620
-
621
- | Measurement | reeflog | naive poller |
622
- |---|---|---|
623
- | fills decoded | **155** | 12 |
624
- | RPC calls for that content | **0** | 44 (4 signature pages + 40 transactions) |
625
- | in-window fills missing from reeflog | **0** of 7 | — |
626
- | fills decoding to identical amounts | **7 / 7** | — |
627
- | rows drained | 474 rows in 316 ms | — |
32
+ Not for you, if: you need pre-event account state (the envelope holds
33
+ balance *movements* and fetch-time bytes, not the bytes as of the event —
34
+ no data source on public RPC serves that), or a managed hosted feed
35
+ (none exists; see *Free and paid*).
628
36
 
629
- The naive consumer is budgeted on purpose: the public endpoint answers
630
- roughly one `getTransaction` per second under load, so a poller that wants
631
- content pays for it — every call above is counted, not estimated. reeflog
632
- paid for its content once, at ingest, and the watcher then reads bytes it
633
- already had.
37
+ ## Quickstart (verified run, ~1 minute)
634
38
 
635
- `tools/test-decoder.mjs` is the offline half: three real mainnet fixtures
636
- with every field asserted, the 57-byte rule enforced, a non-swap log type
637
- refused, and a row without a swap proven not to become a fill. It runs in CI
638
- and needs no network.
639
-
640
- ## Reorgs: a retraction a consumer can actually receive
641
-
642
- A mailbox that delivers an event and then loses it is only half a mailbox.
643
- The other half is saying *that event no longer stands* — and saying it in a
644
- way a consumer can act on.
645
-
646
- reeflog marks a reorg only when **two independent sources refuse the same
647
- transaction**. That rule exists because of what a single source can do:
648
-
649
- - `getSignatureStatuses` answers `null` for a signature it cannot answer for,
650
- and a transport hiccup looks exactly like a fork-out;
651
- - a rate limit looks like silence;
652
- - an endpoint that refuses the *key* (-32602) says nothing at all about the
653
- chain — that is a corrupted row, not a retracted event.
654
-
655
- So the states stay apart, the way `hydrate` keeps `unavailable`, `invalid` and
656
- `pending` apart:
657
-
658
- | Both sources | Verdict | What happens |
659
- |---|---|---|
660
- | at least one serves the transaction | known | nothing changes |
661
- | both refuse it | **reverted** | recorded in `reorgs`, event's seal becomes `reverted` |
662
- | one refuses the key | invalid | no chain verdict, the pass stays complete |
663
- | one fails | undecided | the pass exits 3 — a failure is not a verdict |
39
+ Everything below was executed against `api.mainnet-beta.solana.com` on
40
+ 2026-11-13 with the published v0.7.24 binary; your numbers will differ with
41
+ chain traffic.
664
42
 
665
43
  ```bash
666
- reeflog reorg check --db reeflog.db --limit 200 # detect + record
667
- reeflog reorg tail --db reeflog.db --cursor ./reorgs.txt
44
+ npm install reeflog-client
45
+ npx --no-install node -e "
46
+ import('reeflog-client').then(async (m) => {
47
+ const { bin } = await m.bootstrap(); // verified prebuilt binary
48
+ console.log(bin);
49
+ });"
50
+ # the printed path is the binary; use it as BIN below
51
+
52
+ BIN=~/.reeflog/reeflog
53
+
54
+ $BIN init --db reeflog.db --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc
55
+ $BIN start --db reeflog.db --seconds 60 # ingest over WebSocket
56
+ $BIN seal --db reeflog.db # promote confirmed -> finalized
57
+ $BIN tail --db reeflog.db --cursor ./c.txt --limit 5 # resume from cursor
58
+ $BIN verify --db reeflog.db # find holes (exit 3 if unfinished)
59
+ $BIN status --db reeflog.db # journal health as JSON
668
60
  ```
669
61
 
670
- **The check runs by itself.** `reeflog start` seals pending events and
671
- re-checks them on a schedule (`--seal-every`, default 15 s; `--reorg-every`,
672
- default 60 s), and once more at shutdown. A retraction that only exists when
673
- an operator remembers to run a command is not a product: the consumer who
674
- acted on a fill at 14:02 has to find out at 14:02, not at 18:00. The daemon's
675
- window is bounded in **slots** (`--reorg-lookback`, default 300 behind the
676
- tip), because the chain — not the journal — decides how far a fork can reach.
677
-
678
- The daemon's summary reports what the reorg work actually did
679
- (`reorg_passes`, `reorg_considered`, `reorg_reverted`,
680
- `reorg_incomplete_passes`), so "0 reverts" can be told apart from "the check
681
- never ran" — the difference between evidence and silence.
682
-
683
- Reorgs get **their own channel and their own cursor**, because a consumer
684
- that already read an event cannot learn about its retraction from the event
685
- stream: the event keeps its `seq` and is delivered exactly once. `tail`
686
- reports the event's *current* seal, `reorg tail` reports the retraction
687
- itself. A consumer that only reads events learns that a row now reads
688
- `reverted`; one that reads the reorg channel learns it at the moment it
689
- happened. Neither is optional if a consumer is to undo a fill.
690
-
691
- `tools/e2e-reorg.mjs` proves both directions against live mainnet, because
692
- the dangerous direction is the false positive — a detector that retracts a
693
- live transaction is worse than no detector, since a consumer would undo a
694
- real fill. Measured 2026-10-28:
695
-
696
- - **negative:** real finalized transactions harvested from mainnet → **0
697
- reverts**, twice: 179 in the first run and 189 in the confirming run (the
698
- count moves with the harvest, the verdict does not);
699
- - **positive:** a well-formed signature the chain does not know → detected,
700
- recorded, delivered exactly once on the reorg channel, and the second read
701
- delivered nothing (strict cursor);
702
- - **honesty:** an unreachable endpoint → exit 3, `complete: false`, **0
703
- reverts** — a transport failure never becomes a fact about the chain.
704
-
705
- The check is batched: one `getSignatureStatuses` call covers 200 rows, and
706
- `getTransaction` is spent only on rows the first source already refused. On a
707
- healthy chain a pass costs one call.
708
-
709
- The same two-source rule now also guards `seal`: a `null` from the status
710
- endpoint no longer writes `missing` on its own. Before this, one silent
711
- source could turn a transaction the chain still knows into a recorded
712
- "missing" — the same false statement a reorg marker must never publish.
713
- `missing` now means *both* sources refuse the transaction; if the second
714
- source cannot answer either, the row stays pending and the pass reports
715
- `undecided`.
716
-
717
- ### A real fork, end to end
718
-
719
- `tools/e2e-reorg.mjs`'s positive case plants a well-formed signature the
720
- chain never saw. That proves the two-source rule fires on an *unknown* key —
721
- not that it fires on a transaction that really was executed, really reached
722
- `confirmationStatus=finalized` with the root past its slot, and was then
723
- taken back. That is the only case a consumer can actually lose money on,
724
- and it was the open gap.
725
-
726
- `tools/e2e-fork.mjs` manufactures it honestly on a local agave chain
727
- (`node tools/e2e-fork.mjs`; `AGAVE_BIN` to point at a release). It runs a real
728
- validator, sends real system transfers, and rolls the ledger back to a
729
- copy of the blockstore taken *after* a first transaction and *before* a
730
- second one. Nothing is planted — the signature the detector retracts is one
731
- the chain itself made and then unmade. From the measured run (slots and
732
- hashes differ run to run; they are recorded per run in the report):
733
-
734
- - the **survivor** (slot 45) sits inside the snapshot: after the rollback its
735
- `getTransaction` still serves it and the block hash at its slot is
736
- byte-identical (`Cs4JNVar…` before and after);
737
- - the **forked transaction** (slot 108) is finalized, sealed, and promised as
738
- final — then both sources refuse it, and the block at that same slot is a
739
- different block (`CgbDQNLd…` → `FGJcD4zT…`);
740
- - the **running daemon** — not a one-shot CLI call — reports
741
- `1 event(s) REVERTED`, with `reorg_reverted=1`, `reorg_incomplete=0` over
742
- 12 passes, and the journal ends with exactly one reorg record naming that
743
- signature while the survivor and a post-fork transaction both stay
744
- `finalized`;
745
- - the retraction is delivered **exactly once**, the second read delivers
746
- nothing, and the consumer's ledger undoes the acted-on event and remembers
747
- it as retracted;
748
- - controls: with the chain intact the detector finds **0** reverts (CLI and
749
- daemon), and a new transaction on the forked chain is ingested, sealed, and
750
- never retracted.
751
-
752
- Measured 2026-10-29: **47/47 checks pass**, twice
753
- (`research/fork-e2e-report.json`, verdict `passed`). The harness asserts the
754
- toolchain gap as `skipped` (exit 3) rather than a pass when an agave binary
755
- or the built binary is missing.
756
-
757
- A real fork on **mainnet** has still not been observed end to end — that
758
- requires watching long enough for mainnet to produce one.
759
-
760
- ## Undoing a fill: the retraction reaches the consumer
761
-
762
- Detecting a revert is only half the job. The other half is the question a
763
- consumer actually has to answer: **which of my actions must be undone?**
764
-
765
- `tools/fill-watcher.mjs` now keeps a ledger of the fills it read, and
766
- `watchRetractions()` joins the reorg channel against that ledger:
767
-
768
- - a retraction for a signature the consumer **acted on** comes back marked
769
- `undoable`, carrying the original fill's data — so the consumer reverses
770
- precisely what it did, amounts included;
771
- - a retraction for a signature it **never read** comes back marked
772
- `undoable: false` — reported, never silently dropped, and never turned into
773
- an unfill of some *other* fill;
774
- - the retracted fill leaves the open ledger and enters the undone ledger, so
775
- the same reversal is never applied twice.
776
-
777
- The ledger is written BigInt-safe: the amount fields are u64 and exceed
778
- `Number` range, and a ledger that quietly loses digits would make every later
779
- comparison meaningless. (The watcher's own CLI used to crash on its first
780
- printed fill for exactly that reason — `JSON.stringify` has no answer for a
781
- BigInt.)
782
-
783
- `tools/e2e-retraction.mjs` proves the whole path against live mainnet: the
784
- daemon's own check finds the revert, the retraction arrives on its own
785
- channel, the ledger join marks it undoable and carries the original fill,
786
- an unrelated retraction is reported but not unfilled, and a second read
787
- delivers nothing.
788
-
789
- One bug in the harnesses themselves is worth naming, because it hides in
790
- every measurement made with them. The consumer helper resolved its child
791
- process on Node's `'exit'` event, which fires when the process *dies* — not
792
- when its piped stdout has been read. A `reorg tail` run could therefore
793
- resolve with an empty output while the child had already written its own
794
- cursor file, i.e. it had really delivered the row the harness could not see.
795
- The consumer then reported **0 retractions for a retraction that was there**,
796
- and the strict cursor made it unrecoverable on the next read. Every harness
797
- now resolves on `'close'` (stdio drained) and fails loudly on a spawn error.
798
- Measured 2026-10-29: this was the difference between 44/47 and 47/47 on the
799
- same harness, same code.
800
-
801
- ## Webhook fan-out: a consumer that is not on this machine
802
-
803
- `reeflog tail` makes the journal a local tool: the consumer has to run next
804
- to the daemon and read a file. A webhook makes it a mailbox in the actual
805
- sense — an outside service gets the events pushed to it, and the delivery
806
- cursor survives a crash on either side.
62
+ The same commands as the installed CLI (on `PATH` after `cargo install`):
807
63
 
808
64
  ```bash
809
- cat > webhooks.json <<'EOF'
810
- {
811
- "targets": [
812
- { "id": "fill-service", "url": "https://example.com/reeflog", "secret": "whsec_…", "timeout_ms": 5000 }
813
- ]
814
- }
815
- EOF
816
-
817
- **Config values are checked before the pass starts**, because a config file
818
- is one layer above the flag parser and fails the same way the parser used
819
- to: a value the consumer cannot honour must be refused, not accepted into a
820
- behaviour the file does not describe.
821
-
822
- | Key | Domain | What is refused |
823
- |---|---|---|
824
- | `url` | absolute URL of scheme `http` or `https` (case-insensitive) with a host, a port `1..=65535` when written, and bytes `http::Uri` can carry | `http://` or `https://` with no host (the consumer dies with "http: empty string" on first delivery), empty authority (`http:///x`), missing scheme, non-http scheme (`ftp://…`), a non-numeric or out-of-range port (`:abc`, `:99999`, a bare `:` — the consumer silently dials 80/443 instead), port `0` (nothing ever listens there), a space/control byte/`<` `>` backtick in the path (the parser refuses it as `http: invalid uri character`), one of `" < > \ ^ \` { \| }` in the host, raw non-ASCII in the path or query (the receiver answers 400 — percent-encode it instead) |
825
- | `id` | non-empty, no leading ASCII whitespace, no control byte except HTAB | empty, a leading space or tab (the receiver strips it, so the consumer's dedup key stops matching the journal), any C0 control byte or DEL (the consumer's header parser refuses the field: every delivery dies with `http: failed to parse header value`) |
826
- | `secret` | non-empty; and one secret per ENDPOINT — two targets that share a secret must point at the same url | empty — an empty secret signs with the empty key, which authenticates nothing. Two targets that share a secret while pointing at DIFFERENT urls are refused — the signed body carries no target id and no endpoint (the headers that do are not signed), so what one receiver verifies the other verifies too: measured, A's accepted bytes replayed at B under a forged event-id header and passed B's own HMAC check (`tools/measure-secret-domain.mjs`, 4/4 faces on the published v0.7.15 asset). One secret is one identity space; give each endpoint its own secret. The same secret on the SAME url stays legal — one consumer listed twice |
827
- | `timeout_ms` | whole milliseconds in `1..=3600000` (1 hour) | `0` (the poster would silently re-read it as 1 ms), anything above the 1-hour ceiling (a timeout no process will live to see — an attempt that cannot finish within an hour is stuck, not slow), a fraction, a negative number, a string, an integer beyond `u64` |
828
- | `dead_letter_after` | non-negative integer in `i64` | a fraction, a negative number, an integer outside `i64` (the count is compared against an `i64` refusal counter) |
829
-
830
- The `url` and `id` rules are not opinions: they call the CONSUMER's own
831
- parser — `http::Uri` and `http::HeaderValue::from_str`, the same functions
832
- ureq and tokio-tungstenite link (one `http` version in Cargo.lock). What the
833
- checker refuses, the consumer would have refused later, at delivery time,
834
- in its own words.
835
-
836
- Every number behind those rules was measured against a real receiver, and
837
- the measurement is reproducible from this repository:
838
-
839
- ```bash
840
- # 1. the published baseline (the release asset's sha256 equals the
841
- # Release-API digest) — the consumer's behaviour does not change
842
- # between releases, but the config refusals do
843
- gh release download v0.7.10 --repo auroraxo/reeflog -p "*linux*"
844
- tar xzf reeflog-linux-amd64.tar.gz
845
-
846
- # 2. the probe: one byte (or character) per delivery, against the binary
847
- node tools/probe-webhook-value-domain.mjs ./reeflog /tmp/whprobe
848
- node tools/make-webhook-value-domain-artifact.mjs /tmp/whprobe \
849
- research/webhook-value-domain-<date>.json
850
-
851
- # 3. the before/after: the same values against the published baseline
852
- # and against this tree's build
853
- node tools/measure-webhook-value-domain.mjs ./reeflog \
854
- "$(pwd)/target/release/reeflog" /tmp/before-after.json
855
- ```
856
-
857
- The raw sweeps land in `research/webhook-value-domain-<date>.json` and are
858
- pinned by `tests/webhook_value_measurement.rs`; the before/after counts
859
- (24 cases, 11 defects in the published baseline, all refused up front by
860
- this tree) are pinned by `tests/webhook_value_before_after.rs`.
861
-
862
- Every refusal names **the key and the offending value** and happens before
863
- any connection is attempted — a refusal with a report would look like a
864
- delivery pass. `timeout_ms: 5000` (default) and `dead_letter_after: 5`
865
- (default) are unaffected; `dead_letter_after: 0` keeps its documented
866
- meaning of "disable the valve".
867
-
868
- reeflog webhook push --db reeflog.db --config webhooks.json # one pass
869
- reeflog webhook status --db reeflog.db --config webhooks.json # cursors + failures
870
- reeflog webhook dlq list --db reeflog.db --config webhooks.json # quarantined events
871
- reeflog webhook dlq replay --db reeflog.db --config webhooks.json # hand them over
872
- reeflog start --db reeflog.db --webhook-config webhooks.json --webhook-every 5
873
- ```
874
-
875
- **The contract, in order of importance:**
876
-
877
- 1. **A delivery is recorded only when the endpoint answered 2xx.** A 403,
878
- a timeout or a rate limit is recorded as a *failed attempt* with the
879
- status that caused it. `webhook push` exits 3 while a target is blocked,
880
- exactly like `verify` and `hydrate`: silence is never health.
881
- 2. **At-least-once, in order.** Per target the deliveries follow the
882
- journal's `seq` and stop at the first failure, so a consumer sees events
883
- in chain order and never with a hole in front of it. It may see one
884
- twice; the stable id (`X-Reeflog-Event-Id: <target>:<kind>:<seq>`) is
885
- what a consumer dedups on for exactly-once.
886
- 3. **A payload the journal still owes is not delivered.** A row whose
887
- content the journal owes (`logs IS NULL AND payload_state IS NULL`) is a
888
- signature, not an event — since v0.7.23 the event queue HOLDS at it:
889
- rows before it are delivered, the target's cursor stays behind the owed
890
- row, no receipt is written, `push` exits 3 naming `hydrate`, and once the
891
- debt is paid the SAME target receives the row complete. Measured on the
892
- published v0.7.22 asset (`research/webhook-payload-hold-baseline-v0722-
893
- 2026-11-13.json`, verdict `webhook-payload-hold-defect-present`): the
894
- receiver got the owed row as a signed body with `logs: null`, `next_seq`
895
- moved 0 → 1, a receipt was written, and the second pass after `hydrate`
896
- delivered nothing — the finished event was permanently unreachable for
897
- that consumer while a fresh target read it. This is the same rule
898
- `tail` holds since v0.7.22, one channel over: the webhook body is
899
- SIGNED, so a receiver cannot tell a signature without content from a
900
- real event. A recorded outcome (`unavailable` / `invalid` / `no_metadata`)
901
- is never held — it is an answer, not a debt — and retractions are never
902
- held by this rule either.
903
- 4. **The payload is authenticated.** Every request carries
904
- `X-Reeflog-Signature: sha256=<hex>` = HMAC-SHA256(secret, body) over the
905
- exact bytes sent (canonical JSON — `serde_json`'s sorted keys, so the same
906
- event always produces the same signature). The receiver checks it with
907
- one line of its own language:
908
-
909
- ```python
910
- import hmac, hashlib
911
- want = "sha256=" + hmac.new(secret, body, hashlib.sha256).hexdigest()
912
- ```
913
- 5. **A retraction is delivered too**, on its own cursor, and it is delivered
914
- *even when the event queue is blocked behind a refusal* — a retraction is
915
- the one message whose lateness costs money.
916
- 6. **Receipts make the crash window safe.** The state (per-target cursor)
917
- and the receipts (which exact body was accepted, by sha256) live in the
918
- journal. A daemon that dies between "POST succeeded" and "cursor advanced"
919
- does not deliver the event twice on the next pass.
920
- 7. **A refusal stops being a verdict only when it stops repeating.** One
921
- 4xx can be an accident (mid-deploy, a config reload), so the refusal is
922
- counted. After `dead_letter_after` refused attempts on the same event
923
- (default 5, `0` disables it) the event moves to the **dead-letter queue**:
924
- it is quarantined with the exact bytes it was owed, and the target's
925
- cursor moves past it, so one consumer that will never accept one event
926
- does not stop its own mailbox forever while the events behind it are
927
- never delivered. A dead letter is **not** a delivery — no receipt is
928
- written, `push` reports it as `dead_lettered`, and `webhook status` shows
929
- it as `dead_letters` until it is replayed. Transient failures (5xx, 429,
930
- timeout) are never quarantined: an outage is not a verdict. Retractions
931
- are never quarantined either: a late retraction costs money, so its queue
932
- stays blocked rather than being papered over.
933
-
934
- ```bash
935
- reeflog webhook dlq list --db reeflog.db --config webhooks.json
936
- reeflog webhook dlq replay --db reeflog.db --config webhooks.json
937
- ```
938
-
939
- The replay sends the stored body byte for byte, so the signature over it
940
- is the same signature it would have had — a receiver deduping on
941
- `X-Reeflog-Event-Id` sees the same event it rejected (or missed), not a
942
- new one. A replay that is refused again stays in the queue and exits 3;
943
- nothing is ever dropped.
944
- 8. **A failure that ended stops being reported.** `webhook status` carries
945
- `last_error` — the reason the target's MOST RECENT delivery pass stopped.
946
- Since v0.7.24 that reason belongs to the pass that produced it: the next
947
- pass for that target clears it and re-sets it only if the pass stops
948
- again. Measured on the published v0.7.23 asset
949
- (`research/webhook-stale-error-published-v0723-2026-11-13.json`, verdict
950
- `stale-error-verdict-present`): the endpoint refused once (`http 500`,
951
- `push` exited 3), the next pass delivered everything and exited 0 — and
952
- `webhook status` still said `http 500`. A monitoring caller that polls
953
- the status alarmed forever on a failure that had ended, and an operator
954
- repaired the same endpoint twice. A failure that still stands is
955
- re-reported on every pass it still happens, so nothing is hidden: the
956
- rule moves the verdict forward with the passes, it does not swallow it.
957
-
958
- ### Measured, against an independent receiver
959
-
960
- `tools/e2e-webhook.mjs` (2026-10-29, live mainnet, Raydium AMM v4): a
961
- separate receiver process that reimplements the signature check with
962
- `node:crypto`, decides its own answers, and logs what it actually received —
963
- **14/14 checks passed** over 118 real events:
964
-
965
- | Check | Result |
966
- |---|---|
967
- | Fan-out to two independent targets | 118/118 delivered to **each**, in seq order, 118 distinct |
968
- | Signature verification at the receiver | 236 bodies, all valid (independent `node:crypto`) |
969
- | Delivered bodies carry content | 6 of 118 with real non-empty `logs` |
970
- | A **403** endpoint | 0 delivered, `blocked_seq: 1`, exit 3, cursor parked at 0, 0 receipts, attempt recorded as status 403 |
971
- | A body signed with the **wrong secret** | receiver rejects, 0 delivered, blocked |
972
- | A **429** endpoint | blocks the queue instead of dropping; recovery delivers 118 exactly once (118 accepted, 118 distinct) |
973
- | Caught-up queue | delivers nothing on a repeat pass |
974
- | Crash between POST and cursor (receipt without cursor) | 117 of 118 delivered, the owed event **not** re-sent, cursor catches up |
975
- | Retraction while the event queue is blocked | reorg delivered and verified, event queue still blocked, exit 3 |
976
- | Third implementation | Python `hmac` recomputes 476 bodies, **0 mismatches** against what the receiver accepted |
977
-
978
- Report: `research/webhook-e2e-report.json`.
979
-
980
- ### The daemon path, killed mid-run
981
-
982
- `tools/e2e-webhook-daemon.mjs` (2026-10-29, live mainnet): `reeflog start
983
- --webhook-config` ingests over a live WebSocket and pushes to the receiver on
984
- its own cadence. The daemon is **SIGKILLed** after 35 s — a mailbox that only
985
- survives a clean shutdown is not a mailbox. **7/7 checks passed**:
986
-
987
- | Check | Result |
988
- |---|---|
989
- | The daemon delivers what it ingested | 968 of 1.069 journal events delivered live (the rest were ingested after the last pass before the kill) |
990
- | Order and no repeats | 968 deliveries, 968 distinct, seqs 1..968 |
991
- | Signatures at the receiver | 968/968 valid (independent `node:crypto`) |
992
- | The cursor survives the SIGKILL | cursor 968 = last delivered seq, 968 receipts |
993
- | The follow-up resumes only the owed remainder | 101 delivered — exactly the 101 that were still owed, **0 re-sends** |
994
- | The consumer ends up holding the whole journal | 1.069 distinct of 1.069 journal events |
995
-
996
- Report: `research/webhook-daemon-e2e-report.json`.
997
-
998
- ### The dead-letter queue, measured against the same receiver
999
-
1000
- `tools/e2e-dlq.mjs` (2026-10-29, live mainnet, Raydium AMM v4): the receiver
1001
- permanently refuses ONE event (seq 1) and accepts everything behind it —
1002
- the exact case that used to stop that target's mailbox forever.
1003
- **13/13 checks passed**:
1004
-
1005
- | Check | Result |
1006
- |---|---|
1007
- | A 503 is retried, not quarantined | blocked, 0 dead letters, 0 pending — an outage is not a verdict |
1008
- | The refused event lands in the dead-letter queue | 1 letter, status 403, `attempts: 3` (= `dead_letter_after`), state `pending` |
1009
- | The push reports the quarantine | `dead_lettered: 1`, `blocked: false` — the queue is no longer stuck |
1010
- | The events BEHIND the refused one are delivered | 54 of 54, in seq order — what the old design lost |
1011
- | The refused event is never counted as delivered | 0 receipts for seq 1, cursor at the tip |
1012
- | The quarantined body is the canonical event body | same type/seq/signature/slot as the journal row |
1013
- | The quarantined body carries its own signature | `stuck:event:1`, HMAC over exactly those bytes |
1014
- | Replay after recovery | `replayed: 1`, `failed: 0` |
1015
- | The replayed event verifies at the receiver | 1 accepted, signature valid |
1016
- | The dead letter is marked replayed | `state: replayed`, `replayed_at` set |
1017
- | **The replay sends byte-identical bytes** | 287 = 287 bytes — a replay is the same event, not a new one |
1018
- | A second replay has nothing left to send | `replayed: 0` |
1019
- | A wrong secret fails the receiver's own check | 55 requests, 0 with a valid signature |
1020
-
1021
- Report: `research/dlq-e2e-report.json`.
1022
-
1023
- ### The harness can fail
1024
-
1025
- A green harness proves nothing if it cannot go red.
1026
- `tools/e2e-webhook-mutations.mjs` mutates the sender in four ways — sign the
1027
- wrong bytes, count a refusal as a delivery, drop the receipt guard, gate the
1028
- retraction behind the blocked queue — and requires the e2e harness to catch
1029
- each one with its named check. **4/4 caught.**
1030
-
1031
- `tools/e2e-dlq-mutations.mjs` does the same for the dead-letter harness —
1032
- **4/4 caught**: disable the queue (the refused event never lands and the
1033
- queue dies), count a quarantined event as a delivery, reclassify a 5xx as a
1034
- verdict, and send different bytes on replay.
1035
-
1036
- The third mutation earned its keep: the first version of the refusal counter
1037
- selected on a hard-coded `400..500` status range while the classifier
1038
- decides "refusal" independently, so a mutation that reclassified 5xx as a
1039
- refusal produced a dead letter the counter never counted — the harness went
1040
- green over a real defect. The counter now follows the classification (a
1041
- recorded non-2xx status IS a refusal, by construction of where attempts are
1042
- recorded), and the mutation turns the harness red as it must.
1043
-
1044
- ### The dead-letter report must match the queue
1045
-
1046
- The dead-letter queue is the one place where "empty" means "a consumer is not
1047
- holding events it was promised". So the report about that queue has to be
1048
- right, and it was not. Measured 2026-11-09 on the **published v0.7.6
1049
- binary** (the sha256 of whose release archive equals the digest the release
1050
- API publishes), against a journal holding planted pending letters and a
1051
- config with two targets:
1052
-
1053
- | command | v0.7.6 answered | the truth was |
1054
- | --- | --- | --- |
1055
- | `dlq list --target bogus` | exit 0, `{"targets": [], "pending_total": 0}` | 3 pending letters |
1056
- | `dlq replay --target bogus` | exit 0, `{"replayed": 0, "failed": 0}` | 3 pending letters |
1057
- | `dlq list --limit 2` | `pending: 2`, `pending_total: 2` | 7 pending letters |
1058
- | `dlq list` (target removed from config) | `pending_total: 0` | 1 pending letter |
1059
-
1060
- Four separate lies. A typo in a target id was indistinguishable from a clean
1061
- queue. `pending` counted **the rows the listing showed**, not the letters that
1062
- are pending — so `webhook status` said `dead_letters: 7` for the same queue
1063
- that `dlq list` called 2. And a letter whose target had left the config was
1064
- invisible: it can never be replayed (no url, no secret) and no command ever
1065
- said so, which breaks the product's promise that nothing is ever lost.
1066
-
1067
- The rule now:
1068
-
1069
- * `--target <id>` must name a target the config knows — an unknown id is a
1070
- **refusal** that names the flag, the value and the known ids, in both
1071
- `list` and `replay`;
1072
- * `pending` and `pending_total` count the letters **pending in the journal**,
1073
- not the rows a `--limit` happened to show; a truncated listing says
1074
- `truncated: true`;
1075
- * a pending letter whose target is not in the config is **reported as an
1076
- orphan** (`in_config: false`), counted in the total, and keeps `dlq replay`
1077
- from exiting 0 — a letter nobody can replay is a fact the operator needs,
1078
- not a number to round to zero.
1079
-
1080
- Measured on both trees by `tools/measure-dlq-target-domain.mjs`
1081
- (`research/dlq-target-domain-2026-11-09.json`): the published v0.7.6 asset
1082
- violates **5 of 8** cases; this tree violates **0 of 8**, with the three
1083
- guards — a known target is still listed with its letters, a listing without a
1084
- filter still reports every config target, an empty queue still reads healthy
1085
- — unchanged on both trees.
1086
-
1087
- The fix first reached only the branch, so the release that carried it is
1088
- itself measured: v0.7.7 (linux asset, archive sha256 equal to the digest the
1089
- release API publishes) violates **0 of 8** against the same baseline
1090
- (`research/dlq-target-domain-published-v0.7.7-2026-11-09.json`, pinned by
1091
- `tests/dlq_target_published_v077.rs`), and the same two cases were driven by
1092
- hand through the empty-app install path (`npm install
1093
- github:auroraxo/reeflog#v0.7.7` → `bootstrap()` → `reeflog 0.7.7`): an
1094
- unknown `--target` is a refusal naming the flag, the value and the known
1095
- targets (exit 1), and `dlq list --limit 2` over a journal holding three
1096
- letters reports `pending: 3, listed: 2, truncated: true` — the queue, not the
1097
- page.
1098
-
1099
- The ladder is `tests/dlq_target_domain.rs` (10 tests) and
1100
- `tools/mutation-ladder-dlq-target-domain.sh` (**5/5 mutations caught**): the
1101
- unknown target is accepted again, `pending` counts the rows shown again, an
1102
- orphaned letter is dropped from the report again, a replay with an orphaned
1103
- letter reads clean again, and the refusal no longer names the offending
1104
- value.
1105
-
1106
- ### The webhook URL domain is the consumer's domain, not a prefix
1107
-
1108
- Measured 2026-11-09 against the **published v0.7.7 binary** (the sha256 of
1109
- whose release archive equals the digest published by GitHub Releases):
1110
- `load_targets` judged the target URL with a string prefix test
1111
- (`url.starts_with("http://") || url.starts_with("https://")`) — the FORM of
1112
- a URL, not its meaning. Two faces, both measured:
1113
-
1114
- | Case | What v0.7.7 did | What the consumer does |
1115
- |---|---|---|
1116
- | `"url": "HTTP://127.0.0.1:8794/hook"` | **REFUSED** (`"url" must be http:// or https://`) | `ureq` (the poster's transport) lowercases the scheme: answers **HTTP 200** to `HTTP://` and `HtTp://` from a live receiver |
1117
- | `"url": "http://"` | **ACCEPTED** (exit 0, report printed, queue configured) | every delivery dies in transport with `http: empty string` |
1118
- | `"url": "http:///x"` | **ACCEPTED** | every delivery dies with `http: invalid format` |
1119
- | `"url": "https://"` | **ACCEPTED** | every delivery dies with `http: empty string` |
1120
-
1121
- The flag parser had already closed this exact defect for the very same
1122
- consumer (`--rpc`, v0.7.6): scheme checked case-insensitively, host required,
1123
- refusal naming the offending value. The config loader, one layer up, still
1124
- used the prefix test.
1125
-
1126
- The fix is ONE implementation for both layers (`check_url_domain` in
1127
- `src/flags.rs`), shared by the flag parser and `load_targets`: scheme
1128
- compared case-insensitively, authority required, and the refusal names the
1129
- key, the offending value, the expected domain and an example before any
1130
- delivery is attempted.
1131
-
1132
- Measured on both trees by `tools/measure-webhook-url-domain.mjs`
1133
- (`research/webhook-url-domain-2026-11-09.json`, pinned by
1134
- `tests/webhook_url_measurement.rs`): the published v0.7.7 asset violates
1135
- **5 of 8** cases; this tree violates **0 of 8**, with the three guards
1136
- (lowercase `http://`, lowercase `https://`, non-http scheme refused)
1137
- unchanged on both trees.
1138
-
1139
- The fix first reached only the branch, so the release that carried it is
1140
- itself measured: v0.7.8 (linux asset, archive sha256 equal to the digest
1141
- the release API publishes) violates **0 of 8** against the same baseline
1142
- (`research/webhook-url-domain-published-v0.7.8-2026-11-09.json`, pinned by
1143
- `tests/webhook_url_published_v078.rs`, falsified five ways — the count, the
1144
- verdict, a dropped case id, a source line pointing at the branch, an
1145
- injected host path).
1146
-
1147
- The ladder is `tests/webhook_url_domain.rs` (12 tests) and
1148
- `tools/mutation-ladder-webhook-url.sh` (**5/5 mutations caught**): the host
1149
- requirement dropped, the scheme check made case-sensitive again, the scheme
1150
- check dropped entirely, the value no longer named, and `webhook.rs` reverted
1151
- to the prefix test. The end-to-end case drives an uppercase-scheme target
1152
- against a local receiver: the URL loads **and delivers** (`delivered: 1`,
1153
- `blocked: false`, push exit 0), while the no-host URL is refused before any
1154
- delivery runs.
1155
-
1156
- ### The webhook config values are judged by the consumer that consumes them
1157
-
1158
- Measured 2026-11-10 against the **published v0.7.8 binary** (the sha256 of
1159
- whose release archive equals the digest published by GitHub Releases), one
1160
- byte or character per delivery, against a receiver that records what
1161
- actually arrives. The webhook config accepted values the consumer cannot
1162
- carry:
1163
-
1164
- * a target `id` with a C0 control byte — accepted (exit 0, report), then
1165
- **every** delivery died with `http: failed to parse header value`: the id
1166
- becomes the `X-Reeflog-Event-Id` field value, and ureq builds that field
1167
- with `http::HeaderValue::from_str`. **32 of the 33** control bytes (all but
1168
- HTAB) were fatal;
1169
- * a target `id` with a leading space — accepted and delivered, but the
1170
- receiver **strips** it: the journal stored ` t1` while the consumer saw
1171
- `t1:event:1`, so the consumer's deduplication key — the field the README
1172
- tells consumers to dedup on — stopped matching the sender's record;
1173
- * a `url` with a raw space, `<`, `>` or a backtick in the path, or one of
1174
- `" < > \ ^ \` { | }` in the host — accepted, then every delivery died with
1175
- `http: invalid uri character`;
1176
- * a `url` with raw non-ASCII in the path — accepted, sent raw, and the
1177
- receiver answered **HTTP 400**;
1178
- * a `url` with a non-numeric port (`:abc`, `:99999`, a bare `:`) — accepted,
1179
- and the consumer **silently dialled 80/443** instead of the configured
1180
- target.
1181
-
1182
- The rules now call the consumer's own parser — `http::Uri` and
1183
- `http::HeaderValue::from_str`, the functions ureq and tokio-tungstenite
1184
- link (one `http` version in `Cargo.lock`) — so the checker cannot drift from
1185
- the code that delivers. The `secret` stays a key: it is an HMAC key and
1186
- never reaches the wire, so only emptiness is out of its domain (measured: a
1187
- secret with a newline, a control byte or non-ASCII bytes delivers, and the
1188
- receiver's own HMAC over the received body verifies every time).
1189
-
1190
- Measured on both trees by `tools/measure-webhook-value-domain.mjs`
1191
- (`research/webhook-value-domain-before-after-2026-11-09.json`, pinned by
1192
- `tests/webhook_value_before_after.rs`): the published v0.7.8 asset accepts
1193
- **11 of 24** sampled values and then fails every delivery with them; this
1194
- tree refuses all 11 up front and still delivers the 9 that always worked.
1195
-
1196
- The raw sweeps are in `research/webhook-value-domain-2026-11-09.json`
1197
- (pinned by `tests/webhook_value_measurement.rs`) and are reproducible from
1198
- this repository — `tools/probe-webhook-value-domain.mjs` is the probe that
1199
- produced them. The ladder is `tests/webhook_value_domain.rs` (22 tests) and
1200
- `tools/mutation-ladder-webhook-value.sh` (**6/6 mutations caught**): the
1201
- leading-whitespace rule dropped, the header validator judging a constant
1202
- instead of the operator's id, the internal-space rule widened into an
1203
- over-refusal, the port-0 rule dropped, the non-ASCII rule dropped, the empty
1204
- secret allowed.
1205
-
1206
- ### The refusal voice must be one line of reeflog's own
1207
-
1208
- Measured 2026-11-10 against the **published v0.7.9 asset** with
1209
- `tools/probe-error-voice.mjs` (`research/error-voice-*.json`). Every refusal
1210
- in the seven hostile cases was *correct* — exit 1, nothing on stdout — and
1211
- the voice was still forgeable, because a refusal names the offending value
1212
- and named it **raw**:
1213
-
1214
- * a value carrying a newline split the refusal into **two** lines, the
1215
- second one starting with `reeflog: error: ` exactly like a real message.
1216
- The operator's own input — or a config file from anywhere — wrote a line
1217
- into the log stream that no log reader can distinguish from reeflog's:
1218
-
1219
- ```
1220
- reeflog: error: flag --seconds expects a non-negative whole number, got `5
1221
- reeflog: error: forged` — not a number (e.g. --seconds 10)
1222
- ```
1223
-
1224
- * ESC, CR, NUL and DEL reached stderr **raw** — into the terminal (ESC
1225
- sequences redraw what the operator sees) and into every log file.
1226
-
1227
- 5 of 7 cases forged a line, 14 control bytes crossed the stream. The fix is
1228
- one choke point: `die()` escapes the message, so every refusal path gets it
1229
- (`src/main.rs`). Control bytes become their escape (`\n`, `\r`, `\x1b`,
1230
- `\x00`, `\x7f`); **HTAB stays literal** — it cannot end a line, so it
1231
- cannot forge a message, and it is inside the id domain (an HTAB id
1232
- delivers byte-identical, measured 2026-11-09).
1233
-
1234
- The verdict never changed: all 8 cases are still refused, and a plain
1235
- refusal is byte-identical to before. Measured by
1236
- `tools/measure-error-voice.mjs`
1237
- (`research/error-voice-before-after-2026-11-10.json`, pinned by
1238
- `tests/error_voice_measurement.rs`): the published v0.7.9 asset forges **5**
1239
- lines and leaks **7** raw control bytes over 8 cases; this tree forges **0**
1240
- and leaks **0**, refusing the same 8. The ladder is `tests/error_voice.rs`
1241
- (7 tests) and `tools/mutation-ladder-error-voice.sh` (**5/5 mutations
1242
- caught**): escaping dropped, escaping narrowed to newlines only, ESC left
1243
- raw, HTAB escaped too (the over-refusal direction), the value no longer
1244
- named.
1245
-
1246
- ### One journal, one program
1247
-
1248
- Measured 2026-11-10 against the **published v0.7.10 asset** (the release
1249
- archive's sha256 equals the digest the Release API publishes): the rule "a
1250
- program recorded in the journal is the journal's identity" existed — in
1251
- exactly ONE command of four. The other three disagreed with it, each in its
1252
- own direction:
1253
-
1254
- | command (journal watches `<A>`, caller passes `<B>`) | v0.7.10 answered |
1255
- | --- | --- |
1256
- | `init --program <B>` | exit 0, `initialized … for program B` — the mailbox **silently changed identity** |
1257
- | `verify --program <B>` | exit 0 with a full report — **about A**; the flag was silently ignored |
1258
- | `repair --program <B>` | the same silent ignore, on the command that **writes** to the journal |
1259
- | `start --program <B>` | exit 1, `journal watches A, refusing --program B` — the rule defended itself here |
1260
-
1261
- An operator who re-ran `init` with the wrong id walked away with a journal
1262
- that reported `program: <B>` while holding A's events. An operator who
1263
- asked `verify --program <B>` got a verdict about A, with nothing to show
1264
- that the two differ — on the command whose whole job is to tell the truth
1265
- about a journal. The rule was real; it was just local to `start`.
1266
-
1267
- The rule now:
1268
-
1269
- * a program recorded in the journal **is** the journal's identity;
1270
- * a command given a different program is **refused**, naming the recorded
1271
- program, the given program and the command — before anything is scanned,
1272
- rewritten or opened for writing;
1273
- * a command given the same program, or none, is served from the recorded
1274
- one; `init` with the same program stays idempotent.
1275
-
1276
- Measured on both trees by `tools/measure-program-identity.mjs`
1277
- (`research/program-identity-2026-11-10.json`, pinned by
1278
- `tests/program_identity_measurement.rs`): the published v0.7.10 asset
1279
- violates **3 of 6** cases and changed the journal's identity **1** time;
1280
- this tree violates **0 of 6**, identity changes **0**. The journal's
1281
- recorded program is read back from the journal itself, never from the
1282
- report of the command under test — the report is the thing that lied. The
1283
- ladder is `tests/program_identity_domain.rs` (7 tests: the three defect
1284
- faces are red on the pre-fix tree, guards green on both) and
1285
- `tools/mutation-ladder-program-identity.sh` (**5/5 mutations caught**,
1286
- tree restored byte-identically).
1287
-
1288
- **And again on the published v0.7.11 asset**, so the refusal is what a
1289
- stranger actually installs (`research/program-identity-published-v0.7.11-2026-11-10.json`,
1290
- pinned by `tests/program_identity_published_v0711.rs`): the published
1291
- v0.7.10 asset violates **3 of 6** and repoints the journal **1** time, the
1292
- published v0.7.11 asset violates **0 of 6** and repoints it **0** times —
1293
- both archives' sha256 equal the digests the Release API publishes
1294
- (`2f6eded2…` / `f75898ff…`). The pin is falsified two ways
1295
- (`tools/falsify-program-identity-pin-v0711.sh`: a wrong count and an
1296
- injected machine path both turn it red, the restored artifact is
1297
- byte-identical and green again). Installed from an empty directory
1298
- (`npm install github:auroraxo/reeflog#v0.7.11`), the refusal speaks
1299
- naming both programs and the command:
1300
-
1301
- ```
1302
- reeflog: error: journal …/j.db watches 11111111111111111111111111111111,
1303
- refusing init --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc —
1304
- one journal, one program; the recorded program is the journal's identity,
1305
- pass --db <other-journal> to work on a different one
1306
- ```
1307
-
1308
- `init`, `verify` and `repair` all exit 1 on a disagreement;
1309
- `init` with the recorded program stays idempotent (exit 0).
1310
-
1311
- ### The dlq replay report describes the queue, not the page
1312
-
1313
- Measured 2026-11-10 on the **published v0.7.11 binary** (archive sha256
1314
- `f75898ff9121df1bd280399401893268e161ec95e1af38b0ccabbd656e588ac3` equal
1315
- to the GitHub Releases digest), against a receiver that answers 200 to
1316
- everything, on a journal holding 7 pending dead letters:
1317
-
1318
- ```
1319
- reeflog webhook dlq replay --db reeflog.db --config wh.json --limit 2
65
+ reeflog init --db reeflog.db --program whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc
66
+ reeflog start --db reeflog.db --seconds 60 # ingest over WS
67
+ reeflog seal --db reeflog.db # promote confirmed -> finalized
68
+ reeflog tail --db reeflog.db --cursor ./c.txt # resume from a cursor
69
+ reeflog verify --db reeflog.db # find holes
70
+ reeflog status --db reeflog.db
1320
71
  ```
1321
72
 
1322
- exited **0** with `{"replayed": 2, "failed": 0, "targets": [{"target": "alpha", "replayed": 2, "failed": 0}], "unreachable_orphans": 0}` —
1323
- while the journal still held **5 pending letters**.
1324
-
1325
- Nothing in the report named them, no exit code carried them. The report
1326
- described the page and the command called that the job — the exact class
1327
- v0.7.7 closed for `dlq list` ("`pending` counted the rows the listing
1328
- showed, not the letters pending"): the listing got the fix, the command an
1329
- operator automates did not. A cron job trusting exit 0 ("the command cannot
1330
- report clean success while events remain un-deliverable in the mailbox" —
1331
- the rule this very function documents) believed the queue drained while 5
1332
- owed events sat un-attempted.
73
+ From Node, the client mirrors the CLI's exit-code contract (0 delivered,
74
+ 3 held/incomplete, 1 refusal) instead of making you re-derive it:
1333
75
 
1334
- The rule now:
1335
-
1336
- * `dlq replay` reports `pending_before` (the queue at the start of the pass),
1337
- `remaining_pending` (what the journal still holds for the targets this pass
1338
- was responsible for), and `truncated: true` per target;
1339
- * a pass that leaves letters it never reached pending exits **3** (the same
1340
- exit-3-means-incomplete contract `tail`/`verify`/`push` use), naming the
1341
- unreached count and the repair (`run again, or raise --limit`);
1342
- * delivery itself is unchanged: exactly `limit` letters per target are
1343
- attempted, byte-identical, marked replayed;
1344
- * a full drain, and an empty queue, still exit 0.
76
+ ```js
77
+ import reeflog, { bootstrap } from 'reeflog-client';
1345
78
 
1346
- Measured on both trees by `tools/measure-dlq-replay-page.mjs`
1347
- (`research/dlq-replay-page-2026-11-10.json`): the published v0.7.11 asset
1348
- violates **5 of 5** cases; this tree violates **0 of 5**, with the guards —
1349
- a full drain still exits clean, an empty queue still reads healthy, delivery
1350
- itself is unchanged, and `--target` still scopes the pass — unchanged on both
1351
- trees.
79
+ const { bin } = await bootstrap(); // verified prebuilt binary; bin is its path
80
+ const box = reeflog({ db: 'reeflog.db', bin });
1352
81
 
1353
- **And again on the published v0.7.12 asset**, so the rule is what a stranger
1354
- actually installs (`research/dlq-replay-page-published-v0.7.12-2026-11-10.json`,
1355
- pinned by `tests/dlq_replay_page_published_v0712.rs`): the published v0.7.11
1356
- asset violates **5 of 5**, the published v0.7.12 asset violates **0 of 5** —
1357
- both archives' sha256 equal the digests the Release API publishes
1358
- (`f75898ff…` / `ab28aac2…`). An undrained pass on the published asset exits
1359
- **3** and says so:
82
+ // `init` is a CLI-only first-touch step; raw() is the documented escape hatch:
83
+ const { exitCode } = await box.raw(['init', '--db', 'reeflog.db',
84
+ '--program', 'whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc']);
1360
85
 
1361
- ```text
1362
- reeflog: 5 pending dead letter(s) were not reached by this pass — the queue
1363
- still holds them; run `reeflog webhook dlq replay` again (or raise --limit)
1364
- to drain it
86
+ // (after `reeflog start` has ingested — the daemon, not the client, talks to the chain)
87
+ const { events, held, cursor } = await box.tail({ cursor: './c.txt', limit: 100 });
1365
88
  ```
1366
89
 
1367
- The pin is falsified two ways (a tampered count and an injected machine path
1368
- both turn it red, the restored artifact is byte-identical and green again).
1369
-
1370
- The ladder is `tests/dlq_replay_page_domain.rs` (8 tests) and
1371
- `tools/mutation-ladder-dlq-replay-page.sh` (**5/5 mutations caught**): the
1372
- truncated replay exits 0 again, the remaining queue is dropped from the
1373
- report again, `pending_before` is dropped again, `truncated` is dropped
1374
- again, and the refusal voice stops naming the unreached count.
1375
-
1376
- The measurement is pinned by `tests/dlq_replay_page_measurement.rs` (5 tests,
1377
- falsified two ways: count tampered → red, host path injected → red).
1378
-
1379
- ### Limits
1380
-
1381
- No TLS client authentication and no per-target payload filtering yet; the
1382
- secret is a shared HMAC key, not a certificate. A target that stays refused
1383
- used to hold its own queue forever; since the dead-letter queue it holds it
1384
- for `dead_letter_after` refused attempts, then the event is quarantined and
1385
- the queue moves on — `webhook status` shows `dead_letters`, `webhook dlq
1386
- list` shows what is in them, and `webhook dlq replay` hands them over. Other
1387
- targets always kept moving. Nothing here is an outside user yet: the
1388
- receiver above is my own test harness, not a third party. The first consumer
1389
- that is not reeflog lives in its own repository —
1390
- [auroraxo/whirlpool-watcher](https://github.com/auroraxo/whirlpool-watcher),
1391
- which reads this mailbox over `tail` and the signed webhook, decodes Whirlpool
1392
- `Traded` events with a layout it verified against live mainnet itself, and
1393
- keeps a fill ledger without spending an RPC call (see the qualification list).
1394
-
1395
- A **second** consumer in a different repository now reads the envelope's
1396
- instruction bytes: [auroraxo/sememe](https://github.com/auroraxo/sememe)
1397
- harvests from this mailbox instead of paying `getSignaturesForAddress` /
1398
- `getBlock` / `getTransaction`, with the zero-RPC claim *enforced* (its
1399
- `fetch` is replaced by a recording rejector, and the run fails if any call
1400
- was attempted). Measured 2026-10-31 on a fresh 45-second window (456 events,
1401
- 452 envelopes held): 156 payloads from 137 transactions with **0 network
1402
- attempts**; against independently fetched chain transactions, 10/10 sampled
1403
- payload-bearing transactions **byte-identical** and 10/10 sampled
1404
- no-target transactions corroborated, 0 different, 0 unavailable. Its own
1405
- report of the first run was wrong (it labelled v1-transaction rejections as
1406
- rate limiting) and says so — the retraction and the reader contract (what
1407
- was read, what was written, which counters reconciled) are in
1408
- [issue #1](https://github.com/auroraxo/reeflog/issues/1).
1409
-
1410
- ## Three kinds of gap
1411
-
1412
- They need different tools, and conflating them is how a "we never lose events" claim turns out to be false:
1413
-
1414
- - **Tail gap** — events after the newest signature the journal holds (crash, disconnect, restart). Fixed by walking back from the chain tip with `before` until the boundary **slot**. Bounded and cheap. reeflog closes this automatically on `start`.
1415
- - **Rolling-window hole** — events missing from the middle of the recent history the daemon saw. Repaired by `start` every 30s over a rolling window (default 1800 slots) so rows the WebSocket dropped do not age out unrescued.
1416
- - **Archive hole** — events missing from history deeper than one tip-walk budget reaches (measured 2026-10-30: 40 pages of 1000 signatures = 40,000 signatures covers only ~17,490 slots / ~2.4h on Raydium AMM v4). A tip-started walk burns its budget crossing history the journal never claimed, reports `truncated: true` and exits 3. `reeflog verify --anchor-slot` and `reeflog repair --anchor-slot` anchor the walk at a finalized block's last signature at the window's ceiling, making 24h-deep (or deeper) holes reachable and verifiable with zero tip-crossing overhead.
1417
-
1418
- A walk that is cut short is reported as truncated, the gap stays **open**, and the command exits non-zero. A partially filled gap is never presented as filled, and a scan that read nothing cannot report `missing_count: 0` as if it had checked.
1419
-
1420
- ## The cursor file is read strictly
1421
-
1422
- The cursor file is state the consumer owns, and it never passes through the
1423
- flag parser — so the numeric-flag gate (v0.7.2, v0.7.3) never saw it.
1424
- Measured 2026-11-08 against the published v0.7.3 binary with 20 planted rows,
1425
- one cursor file per case (`research/cursor-state-2026-11-08.json`):
1426
-
1427
- | What the existing cursor file held | v0.7.3 did | Now |
1428
- |---|---|---|
1429
- | `abc` | exit 0, **all 20 rows replayed**, file overwritten | refused, 0 rows, file untouched |
1430
- | `9223372036854775808` (outside i64) | exit 0, all 20 rows replayed | refused, 0 rows |
1431
- | `-1` | exit 0, all 20 rows replayed | refused, 0 rows |
1432
- | empty (what a truncate-then-crash leaves) | exit 0, all 20 rows replayed | refused, 0 rows |
1433
- | `15.0` | exit 0, all 20 rows replayed | refused, 0 rows |
1434
- | **missing** (a first run) | exit 0, 20 rows | unchanged — that is the documented start |
1435
- | `0` (a deliberate rewind) | exit 0, 20 rows | unchanged |
1436
- | `15` (resume) | exit 0, 5 rows | unchanged |
1437
-
1438
- `read_cursor` was `.ok().and_then(parse).unwrap_or(0)`: every way an
1439
- existing cursor file can fail to carry a cursor collapsed into "start from
1440
- the beginning", delivered in full, with an exit code that says the run was
1441
- fine. The consumer's central promise — *resumes exactly where it stopped* —
1442
- is not a promise a silent restart from 0 can keep. The rule now: a missing
1443
- file is a first run; an existing file that does not hold a non-negative
1444
- whole number inside `i64` is a refusal that names the file and its content,
1445
- before the journal is opened, and the file is left untouched. `reorg tail`
1446
- reads cursors the same way. Ladder: `tests/cursor_state.rs` (7 defect cases
1447
- RED on v0.7.3, 6 guards green on both trees), artifact pinned by
1448
- `tests/cursor_state_measurement.rs`.
1449
-
1450
- ## A cursor knows where it came from
1451
-
1452
- A strict read still left the cursor anonymous: one bare number, written by
1453
- whichever command last ran, with no record of **which journal** or **which
1454
- channel** it belonged to. reeflog keeps two cursors per journal — `tail`
1455
- stores an event `seq`, `reorg tail` stores a reorg `id` — and each reader
1456
- accepted any number from any source. Measured 2026-11-10 against the
1457
- published v0.7.12 binary, 20 events + 3 reorgs per journal, one crossing per
1458
- case (`research/cursor-provenance-2026-11-10.json`):
1459
-
1460
- | The crossing | v0.7.12 did | Now |
1461
- |---|---|---|
1462
- | event cursor `20` handed to `reorg tail` | exit 0, `0 reorg(s)` — **3 reorgs silently lost** | refused, 0 rows, file untouched |
1463
- | reorg cursor `3` handed to `tail` | exit 0, **17 delivered events replayed** | refused, 0 rows |
1464
- | cursor copied to another journal | exit 0, `0 events after cursor 20` — a fresh mailbox read as drained | refused, 0 rows |
1465
-
1466
- The rule: a cursor reeflog **wrote** now carries its provenance — one line,
1467
- `{"v":1,"channel":"events","journal":"3b07e3569b86c5e2","value":20}` — and is
1468
- honoured only by the same channel of the same journal. Any other use is a
1469
- refusal that names the file, both sides of the crossing, and the repair; it
1470
- delivers nothing, exits non-zero, and leaves the file byte-identical so the
1471
- operator can still see what it said. A **bare** hand-written number stays
1472
- legal — `0` to restart, `15` to resume, an ahead-of-tip value to wait: it is
1473
- the documented deliberate rewind, makes no claim about its origin, and a
1474
- reader that refuses the operator's own numbers is worse than the defect.
1475
- The journal's id is seeded once at `init` and stored in the journal itself;
1476
- the summary line still reports the plain number (`cursor now 20`), and the
1477
- Node client's `cursorValue` reads the position out of either form.
1478
- Ladder: `tests/cursor_provenance.rs` (4 crossing cases RED on the pre-fix
1479
- tree, 7 guards green on both), mutations caught by
1480
- `tools/mutation-ladder-cursor-provenance.sh` (7/7), both trees measured by
1481
- `tools/measure-cursor-provenance.mjs`.
1482
-
1483
- ## A failed cursor write never destroys the cursor
1484
-
1485
- Reading the cursor strictly did not help if the **write** of the cursor
1486
- could destroy it. `write_cursor` was `std::fs::write`: open with `O_TRUNC`,
1487
- then write. When that write failed — `ENOSPC`, and this build host hit a
1488
- genuinely full disk twice in the two days before the fix — the cursor file
1489
- was left **zero bytes** *after* the rows had already been printed on stdout.
1490
- Every later run then refused with the reader's own message about an empty
1491
- file being what a crash between truncate and write leaves behind: the reader
1492
- named the state correctly, and the **writer** had produced it. The consumer
1493
- was wedged at exit 1, and the one documented repair (delete the file) means
1494
- rewinding delivery to the very beginning.
1495
-
1496
- Measured 2026-11-10 against the published v0.7.13 binary (archive sha256
1497
- `4a0355f9536e30217b3b0ab2cb1510df1ab1583eafe0e0e8fa0c1cc577399eb6`, equal
1498
- to the Release-API digest), with an `ENOSPC` fault injected only on the
1499
- cursor write (`research/cursor-write-durability-2026-11-10.json`):
1500
-
1501
- | The failed write | v0.7.13 did | Now |
1502
- |---|---|---|
1503
- | `tail` with a hand-written `0` cursor fails mid-run | exit 1, cursor file **0 bytes** — the old position gone | exit 1, cursor file still `0`, refusal names file + errno + the redeliver promise |
1504
- | same face on `reorg tail` | exit 1, truncated | exit 1, unchanged |
1505
- | first batch, no cursor file yet | exit 1, an **empty file appears** — the wedge | no file created at all |
1506
- | run again after the failure | refused forever (`empty file … crash between truncate and write`) | **redelivers the same rows** and advances the cursor |
1507
-
1508
- The rule: the cursor is staged in a sibling temp file, fsynced, and renamed
1509
- over the cursor path — the file always holds the old full cursor or the new
1510
- full cursor, never a hole. A failed stage refuses naming the file, the
1511
- errno, and the contract that makes the refusal safe: *the cursor file is
1512
- UNCHANGED … the rows printed above were delivered and will REDLIVER on the
1513
- next run (delivery is at-least-once), so simply run again*. That is honest
1514
- for the same reason the reader is: the delivery already happened on stdout,
1515
- so at-least-once redelivery is the truth, and a redelivery the consumer has
1516
- seen before is strictly better than a silent rewind to zero.
1517
-
1518
- Ladder: `tests/cursor_write_durability.rs` (4 defect cases RED on the
1519
- pre-fix tree, 4 guards green on both), mutations caught by
1520
- `tools/mutation-ladder-cursor-write.sh` (**7/7**), both trees measured by
1521
- `tools/measure-cursor-write-durability.mjs`; the artifact's verdict is
1522
- pinned by `tests/cursor_write_durability_measurement.rs`, falsified three
1523
- ways (`tools/falsify-cursor-write-pin.sh`).
1524
-
1525
- **And again on the published v0.7.14 asset**, so the rule is what a stranger
1526
- actually installs (`research/cursor-write-durability-published-v0.7.14-2026-11-10.json`,
1527
- pinned by `tests/cursor_write_durability_published_v0714.rs`): the published
1528
- v0.7.13 asset violates **4 of 4** defect cases, the published v0.7.14 asset
1529
- violates **0 of 4** — both archives' sha256 equal the digests the Release
1530
- API publishes (`4a0355f9…` / `9b6c3034…`). The pin is falsified three ways
1531
- (`tools/falsify-cursor-write-published-v0714.sh`: a falsified count, an
1532
- injected machine path and a fixed seat repointed at the build instead of the
1533
- release each turn it red; the restored artifact is byte-identical and green
1534
- again).
1535
-
1536
- ## A webhook target belongs to its endpoint
1537
-
1538
- A delivery cursor is a promise to ONE consumer. Until v0.7.15 the journal
1539
- keyed that promise by the target id alone — measured 2026-11-11 against the
1540
- PUBLISHED v0.7.14 asset (`tools/measure-endpoint-binding.mjs`,
1541
- `research/endpoint-binding-published-v0.7.14-2026-11-11.json`): target `t1`
1542
- delivered 3 events to receiver A; the SAME id repointed to receiver B
1543
- (200-OK receiver, same secret) delivered **0**, `webhook push` exited **0**
1544
- reporting `delivered: 0, blocked: false, next_seq: 3`, and `webhook status`
1545
- printed `delivered: 3` next to B's url while `pending_events: 0`. A consumer
1546
- that moves — a new port after a restart, a second environment behind the
1547
- same id — starves behind a green report, and the old consumer's receipts
1548
- silence every event the new one is owed. This is the same class v0.7.13
1549
- closed for cursor *files*: a position that does not say where it applies
1550
- reads as "already delivered" somewhere it never was.
1551
-
1552
- The rule now: the state row and every delivery receipt carry the endpoint
1553
- they were written for, and a config whose url differs from the recorded one
1554
- is **refused before a single byte moves** — exit 1, naming the target id,
1555
- BOTH endpoints and the repair (give the new endpoint its own target id), on
1556
- `webhook push`, `webhook status` and `webhook dlq replay` alike. The journal
1557
- is untouched: the old endpoint's cursors survive the refused pass exactly.
1558
- A state row written before this rule (no endpoint recorded) is **adopted**
1559
- by the url it meets first — same url, same cursors, no re-delivery — so an
1560
- upgrading journal keeps working without ceremony.
1561
-
1562
- Measured after the fix (`research/endpoint-binding-fixed-2026-11-11.json`,
1563
- pinned by `tests/endpoint_binding_measurement.rs`, defect faces also pinned
1564
- as CLI tests in `tests/endpoint_binding_domain.rs`): the published v0.7.14
1565
- asset starves a repointed consumer (A: 3, B: 0, exit 0); this tree refuses
1566
- the same pass with exit 1 naming both endpoints and leaves B at 0 — the
1567
- refusal sends nothing, and nothing inherited was owed to B.
1568
-
1569
- ### One secret is one consumer, not two
1570
-
1571
- Measured 2026-11-11 against the **published v0.7.15 release asset**
1572
- (sha256 `8f941253…`, the Release-API digest) with
1573
- `tools/measure-secret-domain.mjs` — two VERIFYING live receivers, each
1574
- answering 200 only when its own documented rule `HMAC(secret, body)`
1575
- passed, else 401. Two targets (`ta`, `tb`) with DIFFERENT urls sharing ONE
1576
- non-empty secret — a config the table above used to accept, because the
1577
- secret is a key and never reaches the wire. Four faces, all violated
1578
- (`research/secret-domain-published-v0.7.15-2026-11-11.json`, verdict
1579
- `SIGNATURE-TARGET-AGNOSTIC`):
1580
-
1581
- * the pair receiver A accepted verifies 3/3 — and it is exactly the pair
1582
- receiver B computes, because B holds the same key: the signature attests
1583
- "someone with this secret signed this body", never "signed it FOR me";
1584
- * A's accepted bytes replayed at B under a **forged** `tb:` event-id header
1585
- answered **200** from B's own rule, and B recorded the forged id as its
1586
- own — the delivery-id header that names the consumer is transport
1587
- metadata outside the signature;
1588
- * the same replay in the other direction (B's body at A) also answered 200:
1589
- one secret = *** identity space, both ways;
1590
- * a body with a forged `seq: 999` header under a valid signature answered
1591
- 200 — the seq header lies independently of the signature. (The body
1592
- carries its own `seq`; a receiver that reads `body.seq` instead of the
1593
- header is immune to this face — that limit stays with the receiver.)
1594
-
1595
- The **guard** (`tools/measure-secret-domain-guard.mjs`, same published
1596
- asset, distinct secrets) is 4/4 green: foreign pairs are refused 401 by
1597
- the receivers' own rules, and the legitimate delivery still gets 200 — so
1598
- the defect is the shared secret across endpoints, not the wire format.
90
+ The typed client also carries `status()`, `verify()`, `repair()`, `hydrate()`,
91
+ and `reorgs()`; an incomplete verify/repair pass throws instead of returning
92
+ `complete: false`, and a `tail` held at an owed row comes back with `held`
93
+ named, cursor not advanced past it.
1599
94
 
1600
- The rule now: a config whose targets share a secret while pointing at
1601
- **different** urls is refused before a byte moves — exit 1, one line,
1602
- naming both target ids, both endpoints and the repair (`fix: give each
1603
- endpoint its own secret, or use the same url on both rows`). The refusal
1604
- never echoes the secret: a key that appears in a refusal lands in logs,
1605
- terminals and CI output. The same secret on the same url stays legal —
1606
- one consumer listed twice is nothing a signature can betray.
1607
-
1608
- Measured after the fix (`research/secret-domain-fixed-2026-11-11.json`,
1609
- pinned by `tests/secret_domain_measurement.rs`, defect faces also pinned
1610
- as CLI tests in `tests/secret_domain.rs`): the published v0.7.15 asset
1611
- violates 4/4 faces, this tree refuses the same config up front
1612
- (`REFUSED-UP-FRONT`, 0 faces measurable — the door is closed, not the
1613
- measurement skipped). Mutation ladder
1614
- `tools/mutation-ladder-secret-domain.sh` catches 9/9, tree byte-identical
1615
- afterwards.
1616
-
1617
- ### The tag's own client asked for the PREVIOUS release
1618
-
1619
- Measured 2026-11-11 on the **published v0.7.16 tag tree**
1620
- (`research/client-pin-lag-published-v0716-2026-11-11.json`, verdict
1621
- `defect-present-on-baseline`): the v0.7.16 release commit moved Cargo.toml,
1622
- Cargo.lock and the install pins above — and forgot `package.json`, which
1623
- still read `"version": "0.7.15"`. `bootstrap()` derives the release tag it
1624
- requests from the package manifest (that coupling is
1625
- `test/bootstrap.test.mjs` B8), so a stranger who followed this README
1626
- (`npm install github:auroraxo/reeflog#v0.7.16`) installed a client that
1627
- asked the release API for **v0.7.15** and downloaded the **v0.7.15**
1628
- archive — the binary that still carries the shared-secret defect
1629
- v0.7.16 fixed. The tag the README pinned and the binary a stranger got
1630
- were two different releases; nothing on CI could see it, because the
1631
- manifest was the lying side.
1632
-
1633
- The measurement runs each tree's own `index.js` against a stand-in release
1634
- API and records the request it actually makes (not what it should make):
1635
- the published v0.7.16 tree requests `v0.7.15`
1636
- (`servesThePinnedRelease: false`), this tree requests the release it names
1637
- (`servesThePinnedRelease: true`); the tool fails if the baseline shows no
1638
- defect or the fixed tree still does. Gate: `test/bootstrap.test.mjs` B10
1639
- ties `package.json` to `Cargo.toml` (semver shape + equality) — green here,
1640
- red on the published tag tree. Mutation ladder
1641
- `tools/mutation-ladder-client-pin-lag.sh` catches 5/5 (lag, lead, bad
1642
- shape, crate-bumped-manifest-forgotten, pre-release), tree byte-identical.
1643
- The artifact is pinned by 5 tests, falsified by a tampered number and by a
1644
- host path.
1645
-
1646
- ## Repeated use
95
+ ## What a day with reeflog looks like
1647
96
 
1648
97
  ```bash
1649
98
  reeflog init --db reeflog.db --program <pubkey>
@@ -1661,258 +110,120 @@ reeflog webhook push --db reeflog.db --config webhooks.json # deliver to outsi
1661
110
  reeflog status --db reeflog.db
1662
111
  ```
1663
112
 
1664
- `tools/e2e-repair.mjs` runs detect → repair → prove against live mainnet on a
1665
- journal with planted holes and fails if any step misreports; it passed on
1666
- 2026-10-28 with 333 planted holes: verify saw 333, repair filled 333, the
1667
- second verify returned `missing_count: 0, complete: true`, and a deliberately
1668
- truncated scan exited 3.
1669
-
1670
- `tools/e2e-archive.mjs` proves archive-anchored gap fill on live mainnet
1671
- at 24h depth (~216,000 slots back, well past a tip-walk's 17.5k-slot budget):
1672
- tip-started verify exits 3 refusing to claim completeness, anchored verify
1673
- sees all 39 planted holes, anchored repair fills all 39, second verify
1674
- returns `missing_count: 0, complete: true`, and `--anchor-slot` without value
1675
- anchors at the journal's ceiling. Report: `research/archive-e2e-report.json`.
1676
-
1677
- The raw report of the confirmation kill test is committed at
1678
- `research/killtest-full-after-fix-2026-10-28.json` (compact form: the
1679
- per-signature loss samples are truncated, every number and every check verdict
1680
- is the harness's own).
1681
-
1682
- ## Free and paid
1683
-
1684
- - Free, MIT: daemon, journal, CLI, the Node client (below), webhook fan-out, dead-letter queue, archive backfill. No event or cursor limits.
1685
- - Paid later: managed always-on feeds, replicated journals with delivery objectives, and an enterprise mailbox with access control and a delivery audit trail. Events are never sold individually.
113
+ Three kinds of gap need different tools, and reeflog keeps them apart
114
+ instead of calling everything "reconnected":
1686
115
 
1687
- ## The Node client: the exit-code contract, as a library
116
+ - **Tail gap** (crash/disconnect at the tip): closed automatically on `start`.
117
+ - **Rolling-window hole** (dropped rows in recent history): repaired on a
118
+ schedule by `start`, verifiable by `verify` / `repair`.
119
+ - **Archive hole** (deeper than one tip-walk reaches): `--anchor-slot`
120
+ anchors the walk at a finalized block, proven to 24 h depth live.
1688
121
 
1689
- The CLI is a contract before it is a convenience: exit 0 delivers and
1690
- advances the cursor, exit 3 (on `tail`) means the delivery **held** at a row
1691
- whose payload or account envelope is still owed — rows before it are
1692
- delivered, nothing at or after it is consumed — and exit 1 is a refusal. A
1693
- consumer that maps those three outcomes wrongly either replays rows forever
1694
- or drops the owed row without ever seeing it. Until now every Node consumer
1695
- had to derive that by hand.
122
+ A scan that was cut short reports `truncated: true`, leaves the gap open,
123
+ and exits non-zero. A partially filled gap is never presented as filled.
1696
124
 
1697
- `index.js` (npm package `reeflog-client`, MIT, zero dependencies, Node ≥ 18)
1698
- ships that mapping as one file. It never opens the journal — every call runs
1699
- the `reeflog` binary, so there is exactly one implementation of the reader
1700
- contract, and a library's idea of the schema cannot drift from the daemon's.
125
+ Reorgs reach the consumer too:
1701
126
 
1702
127
  ```bash
1703
- npm install reeflog-client # the registry; this version, no Rust toolchain
1704
- npm install github:auroraxo/reeflog#v0.7.24 # the same client from the tagged release
1705
- ```
1706
-
1707
- The client needs the `reeflog` binary. `bootstrap()` gets it **without a
1708
- Rust toolchain** — it downloads the prebuilt archive the release already
1709
- ships for your platform (linux-amd64, macos-arm64), verifies its bytes
1710
- against the sha256 the GitHub release API stores for that asset, and only
1711
- then installs it to `~/.reeflog`:
1712
-
1713
- ```js
1714
- import reeflog, { bootstrap } from 'reeflog-client';
1715
-
1716
- const { bin, digest, binarySha256 } = await bootstrap();
1717
- // digest = sha256 of the ARCHIVE (what the release API stores)
1718
- // binarySha256 = sha256 of `bin` itself — check this against the file
1719
- const box = reeflog({ db: 'reeflog.db', bin });
128
+ reeflog reorg check --db reeflog.db --limit 200 # detect + record
129
+ reeflog reorg tail --db reeflog.db --cursor ./reorgs.txt
1720
130
  ```
1721
131
 
1722
- - **Refuses instead of trusting.** A download whose sha256 does not match
1723
- the release digest is refused and nothing is written — no binary, no
1724
- sidecar. A release asset with no digest is refused too.
1725
- - **A cache hit is verified, not assumed.** The sidecar (`install.json`)
1726
- records the extracted binary's own `binarySha256`; a second call re-hashes
1727
- the file on disk and only reports `cached: true` when it matches. Before
1728
- v0.7.1 the cache path compared only version + asset name, so a replaced or
1729
- truncated binary came back as `cached: true` under a digest that named the
1730
- ARCHIVE, not the bytes on disk (measured: claimed `f6b36c53…`, actual
1731
- `6b9fd8b7…`). A sidecar without `binarySha256` — every install made before
1732
- v0.7.1 — is re-fetched once to upgrade it, then caches again. The check
1733
- defends against a stale or corrupted cache being reported as verified; an
1734
- attacker who can write your install directory can simply re-run
1735
- `bootstrap()` afterwards, so it is not a defence against a hostile local
1736
- user. Live measurement: `research/bootstrap-cache-2026-11-07.json`.
1737
- - **A platform without a published archive is refused by name** (the
1738
- refusal lists what exists and gives the `cargo install` fallback), rather
1739
- than fetching a file that is not there.
1740
- - **The default tag comes from the package version**, never a hard-coded
1741
- string, so the client can never fetch an older release than the package
1742
- it ships in.
1743
- - **A stalled or looping download gets a verdict, not silence.** Every
1744
- request `bootstrap()` makes carries a 30 s inactivity budget (tune with
1745
- `{ timeoutMs }`): no response headers in time, or no bytes between two
1746
- chunks for that long, destroys the socket and rejects with a named refusal
1747
- — an install must never hang without an answer. Redirects are followed at
1748
- most 5 hops; a loop is refused by name. Before v0.7.17 a server that
1749
- accepted the connection and never answered left `await bootstrap()`
1750
- hanging forever, and a self-referential 302 re-requested itself 11,727
1751
- times in 8 s (measured on the published v0.7.17 client:
1752
- `research/bootstrap-hang-2026-11-11.json`).
1753
- - **An answer that is not a release gets a verdict in the product's voice.**
1754
- The release API can answer `200` with something that is not a release
1755
- object: a captive portal's login page (HTML), a corporate proxy, a
1756
- rate limiter's JSON. Before v0.7.19 the HTML case threw a raw
1757
- `SyntaxError` out of `bootstrap()` — a crash no `ReeflogError` catch ever
1758
- saw — and the rate-limit case was refused as `release v0.7.18 has no
1759
- asset named …`, which sends the stranger to a release page that is fine.
1760
- Now both refuse with the endpoint's OWN words: the URL that answered, the
1761
- fact that it answered HTML, and the API's message when it sent one. A
1762
- release object that genuinely lacks this platform's asset keeps its
1763
- honest `has no asset named` refusal (measured on the published v0.7.18
1764
- client: `research/captive-portal-2026-11-11.json`).
1765
- - **Trust is the same anchor as `cargo install --git`**: this repository
1766
- over TLS. It protects against a corrupted or substituted download, not
1767
- against whoever can push tags and releases here. If you want to be
1768
- independent of that, compare `digest` against the asset's published
1769
- sha256 yourself.
1770
- - Measured end to end against the published release —
1771
- `research/bootstrap-2026-11-07.json` (install + cached re-run + real
1772
- rows delivered through the installed binary) and
1773
- `research/bootstrap-cache-2026-11-07.json` (fresh install, verified cache
1774
- hit, tampered binary, pre-v0.7.1 sidecar upgrade, other-platform asset).
1775
-
1776
- ```js
1777
- import reeflog from 'reeflog-client';
1778
-
1779
- const box = reeflog({ db: 'reeflog.db' }); // bin on PATH, or { bin: '/path/to/reeflog' }
1780
- const s = await box.status(); // parsed health: events, envelope debt, gaps
1781
-
1782
- const { events, held, cursorValue } = await box.tail({
1783
- limit: 500,
1784
- consumer: 'my-service', // separate default cursor per consumer
1785
- withAccounts: true, // hold rather than deliver an unreadable row
1786
- });
1787
- if (held) {
1788
- // exit 3: `held.seq` and `held.signature` name the owed row; the rows
1789
- // above it are already delivered and the cursor already advanced past
1790
- // them. Pay the debt (`reeflog hydrate --accounts`) and call again — the
1791
- // held row and everything behind it arrive next, nothing was lost.
1792
- }
132
+ And delivery can leave the machine entirely — signed webhooks with a
133
+ dead-letter queue:
1793
134
 
1794
- const { retractions } = await box.reorgs({ consumer: 'my-service' }); // own cursor channel
1795
- await box.verify({ pages: 400 }); // incomplete scan THROWS (exit 3): no clean-looking result
1796
- await box.hydrate({ accounts: true });
135
+ ```bash
136
+ reeflog webhook push --db reeflog.db --config webhooks.json # one pass
137
+ reeflog webhook status --db reeflog.db --config webhooks.json # cursors + failures
138
+ reeflog webhook dlq list --db reeflog.db --config webhooks.json # quarantined events
139
+ reeflog webhook dlq replay --db reeflog.db --config webhooks.json # hand them over
140
+ reeflog start --db reeflog.db --webhook-config webhooks.json --webhook-every 5
1797
141
  ```
1798
142
 
1799
- The rules the client enforces instead of leaving to the caller:
1800
-
1801
- - **exit 0 vs exit 3 are different answers.** `tail` returns
1802
- `{ events, held, cursorValue }`; `held: null` is the only shape that means
1803
- "the batch is fully delivered". `verify`/`repair`/`hydrate` throw on exit 3,
1804
- because a pass that proves nothing must never arrive as a resolved value.
1805
- - **exit 1 keeps the binary's words.** The thrown `ReeflogError` carries the
1806
- CLI's own refusal (`no journal at … — run \`reeflog init …\``), plus
1807
- `exitCode`, `stderr` and the `args`, so a caller can act on it rather than
1808
- on a rephrased guess.
1809
- - **Only the documented codes are verdicts.** The typed API accepts 0
1810
- (delivered), 1 (refused) and 3 (held/incomplete). Every OTHER close is a
1811
- refusal carrying the REAL code and the binary's last words: the
1812
- journal-write failure's `2`, an old binary's usage `64` (client/binary
1813
- version skew), a panic's `101`, and a kill by signal (the process died, so
1814
- there is no exit code at all — the refusal names the signal instead of
1815
- inventing one). Before v0.7.20 the client treated "not 1 and not 3" as
1816
- success, so a run that died AFTER printing its rows resolved as a
1817
- delivered batch (`held: null` = "fully delivered"), an old binary's usage
1818
- error resolved as a clean empty batch — and on the JSON commands it
1819
- crashed out of `JSON.parse` with a raw `SyntaxError` no `ReeflogError`
1820
- catch ever saw — and a SIGKILL'd run was dressed up as a forged `exit 1`
1821
- refusal naming no signal. Measured on the published v0.7.19 client:
1822
- `research/client-exit-voice-2026-11-12.json` (6/6 defect faces on the
1823
- baseline, 6/6 refused afterwards). `raw()` stays the escape hatch: there,
1824
- any code comes back as `{ exitCode }`.
1825
- - **Cursors are the binary's.** The client only chooses a default path per
1826
- db + channel + consumer label, so two consumers of one journal never share
1827
- a cursor by accident; pass `cursor` to name it yourself.
1828
- - **A missing binary says how to install one** — naming `bootstrap()` (no
1829
- Rust toolchain) and the `cargo install` fallback — instead of crashing
1830
- with `ENOENT`.
1831
-
1832
- The contract is tested against the real binary and a real journal in
1833
- `test/client.test.mjs` (`npm test`, 8 tests): delivery order, no replay
1834
- across a resumed cursor, the hold at an owed row (delivered before, not
1835
- consumed at/after, delivered once the debt is paid), separate cursor
1836
- channels, refusals with their fix line intact, and the incomplete-verify
1837
- throw. CI runs them on every push.
143
+ ## Measured, not asserted
1838
144
 
1839
- Webhook fan-out ships in this tree: `reeflog webhook push` / `status`, signed
1840
- bodies, per-target durable cursors, in-order at-least-once with receipts —
1841
- plus the dead-letter queue (`reeflog webhook dlq list` / `replay`) for a
1842
- target that refuses one event instead of holding its queue forever. Archive
1843
- backfill (`verify --anchor-slot` / `repair --anchor-slot`) ships as well,
1844
- anchored at finalized blocks for holes deeper than the tip-walk window.
145
+ What a stranger's empty directory did with the published binary (full
146
+ detail and every raw number in
147
+ [docs/measured-history.md](https://github.com/auroraxo/reeflog/blob/main/docs/measured-history.md)):
1845
148
 
1846
- ## Evidence and limits
149
+ | Claim | Evidence |
150
+ |---|---|
151
+ | Live ingest | 3122 events in 75 s on mainnet, 0 disconnects (2026-10-27) |
152
+ | Coverage | 224/226 chain signatures in the claimed window = 99.12 % |
153
+ | Crash survival | 600 s kill test (SIGKILL every 60 s): the first run **falsified** the claim; the fixed version survived the same run |
154
+ | Gap repair | 333 planted holes: verify saw 333, repair filled 333, second verify complete (live mainnet) |
155
+ | Archive depth | 39 holes at ~24 h depth filled and verified anchored |
156
+ | Retractions | proven against a **real fork of a real chain**, 47/47 checks |
157
+ | Refusals | every documented CLI line runs against the binary in CI (`tests/documented_surface.rs`); unknown flags are refused, never ignored |
158
+
159
+ ## Conditions and limits — the honest part
160
+
161
+ - **No third-party user yet.** Two out-of-tree consumers exist
162
+ ([whirlpool-watcher](https://github.com/auroraxo/whirlpool-watcher),
163
+ [sememe](https://github.com/auroraxo/sememe)). Adoption is
164
+ instrumented (`tools/measure-adoption.mjs`), not assumed.
165
+ - **Finality is the boundary of the promise.** reeflog seals events to the
166
+ chain's own finalized state and reports reorgs; it cannot make
167
+ confirmed-but-reverted history look trustworthy — it marks and delivers
168
+ the retraction instead.
169
+ - **Archive availability is the RPC's, not reeflog's.** Deep gap-fill works
170
+ while the endpoint serves `getSignaturesForAddress` history; a public
171
+ endpoint is rate-limited, so provider independence has to be real rather
172
+ than assumed.
173
+ - **No pre-event account state.** The envelope carries logs, balance
174
+ movements, instructions, the account list with owners, and fetch-time
175
+ bytes. The bytes *as of the event* are not in it, because no public-RPC
176
+ data source can serve them — measured, two free routes are dead
177
+ (`research/pre-event-bytes-probe-2026-11-02.json`).
178
+ - **One transport.** Public WebSocket `logsSubscribe` only; Yellowstone as a
179
+ second transport does not ship.
180
+ - **A consumer must read the exit code.** `tail` exits 3 when a row's
181
+ payload is still owed and holds the cursor there; treating every
182
+ non-zero as "retry everything" replays forever. `exit 1` is a refusal.
183
+ - **The client is zero-dependency since v0.7.25.** Until then the npm
184
+ manifest named *itself* as a dependency on github:, so every
185
+ `npm install reeflog-client` also cloned a second, older copy of the
186
+ client over git+ssh — and failed outright on a host without `git`
187
+ (`npm error code ENOENT / syscall spawn git`). Measured on the
188
+ published 0.7.24 bytes; fixed and re-measured in
189
+ [`research/npm-self-dependency-fixed-2026-11-13.json`](research/npm-self-dependency-fixed-2026-11-13.json).
190
+ Older tags keep the defect: they are immutable.
191
+
192
+ ## Deep dives
193
+
194
+ - [docs/measured-history.md](https://github.com/auroraxo/reeflog/blob/main/docs/measured-history.md) — the complete
195
+ README measurement and contract history through v0.7.25, unbroken.
196
+ - [`research/`](research/) — every committed measurement artifact behind the
197
+ numbers above (raw JSON reports, pinned by tests).
198
+ - [`tests/`](tests/) and [`test/`](test/) — the contract tests: the CLI
199
+ surface the README documents is run against the real binary, and the
200
+ Node client's exit-code mapping is pinned offline.
201
+ - [Releases](https://github.com/auroraxo/reeflog/releases) — per-platform
202
+ archives with sha256; the release workflow refuses to publish a tag the
203
+ binary disagrees with.
204
+ - [Issues](https://github.com/auroraxo/reeflog/issues) — including issue
205
+ #1, where a consumer retracted one of this project's own claims.
1847
206
 
1848
- Checked on 2026-10-23: Solana client errors already name the failure modes, including closed WebSockets, missing subscription IDs, and corrupt iterators. Yellowstone is a strong live transport but explicitly ephemeral. The closest general-chain project, Chainhook, is archived and does not cover Solana. One developer rebuilt reliable replay into Postgres as a four-star personal pipeline.
207
+ ## Free and paid
1849
208
 
1850
- reeflog does not recover account bytes from before an event, and the account
1851
- envelope is balance MOVEMENTS an event caused — not the state of the accounts
1852
- involved. `tail --with-accounts` does not skip past a row whose envelope has
1853
- not been fetched yet, and since v0.7.22 the default tail holds the same way
1854
- for a row whose payload is still owed: the cursor stays, tail exits 3 until
1855
- `hydrate` (or `hydrate --accounts`) pays the debt, so an owed row can never
1856
- be lost to the consumer. Rows the chain itself made unfillable are the only
1857
- ones skipped, and each skip is reported. That is a different problem.
1858
- Public WebSocket endpoints are also rate-limited, so provider independence
1859
- has to be real rather than assumed.
209
+ Free, MIT: daemon, journal, CLI, Node client, webhook fan-out, dead-letter
210
+ queue, archive backfill. No event or cursor limits. Paid later (not today):
211
+ managed always-on feeds and replicated journals. Events are never sold
212
+ individually.
1860
213
 
1861
- Ownership is the partial exception: `hydrate --owners` fills an `account_owners`
1862
- map (owning program per address), and changed token accounts name their token
1863
- program for free. Two limits remain and are stated rather than smoothed over:
1864
- owners are read **at fetch time**, not at event time (an account may have
1865
- changed owner since), and account **bytes / pre-event state** are still not in
1866
- the envelope at all — that stays open, see *Next qualification* item 4.
214
+ ## Development
1867
215
 
1868
- The account BYTES are now in the envelope too (`hydrate --account-data`),
1869
- which closes one half of that sentence and not the other: the bytes say what
1870
- an account held **at fetch time**, not what it held when the event happened.
1871
- 16–18 % of the addresses a transaction enumerates no longer exist by the
1872
- time the pass runs (measured on two live windows), and a hot token
1873
- account's balance moves between two reads. Historical state — the bytes AS
1874
- OF the event — remains open, and no envelope field claims otherwise.
1875
- Measured 2026-11-02 (`tools/probe-pre-event-bytes.mjs`,
1876
- `research/pre-event-bytes-probe-2026-11-02.json`): both free routes to
1877
- pre-event bytes are dead — a past-slot `getAccountInfo` parameter answers
1878
- anchored at the current context slot (requested 464542126, got 464542734),
1879
- and predecessor transactions carry instruction bytes but no account-state
1880
- payload (0 across 6 scans) — so the boundary is "no data source on this RPC
1881
- class", not "a second fetch" and not "the chain's history". What history
1882
- does prove, measured on the same window: 7/60 touched accounts had no
1883
- touch after the event, for which fetch-time bytes ARE the post-event bytes
1884
- (witness: lamports 4/4, token mint/owner/amount 2/2; wrong-side guard on
1885
- moved accounts 0/3; 3 certified accounts already gone at fetch time are
1886
- counted, never witnessed). That certification is checked after the fetch,
1887
- not before: a certification can expire mid-run (an earlier
1888
- classify-then-fetch run showed exactly one such disagreement,
1889
- root-caused) — and it is measured, not shipped: no envelope field claims
1890
- it, and it covers the post-event side only.
216
+ ```bash
217
+ cargo build --release # Rust daemon + CLI
218
+ cargo test --release # CLI/journal contract tests
219
+ npm install && npm test # Node client contract tests (offline)
220
+ node tools/measure-documented-surface.mjs # README lines vs. the binary
221
+ ```
1891
222
 
1892
- Still open: Yellowstone as a second transport, and a real consumer other
1893
- than my own two repositories (the Whirlpool watcher and sememe). Reorg handling marks a
1894
- retracted event, delivers the retraction, runs the check inside the daemon,
1895
- and joins retractions against the fill watcher's ledger — and it has now been
1896
- proven against a REAL fork of a real chain, not only against a planted
1897
- signature. What remains unproven is the check's reach on mainnet: it still
1898
- looks only at sealed rows near the tip, and no mainnet fork has been observed
1899
- end to end.
223
+ The `tools/` directory holds the harnesses behind every number above
224
+ (`killtest.mjs`, `e2e-*.mjs`, `measure-*.mjs`); their reports are committed
225
+ under `research/` and pinned by the measurement tests.
1900
226
 
1901
- ## Next qualification
227
+ ## License
1902
228
 
1903
- 1. ~~Put my own RFQ fill polling and Whirlpool watcher on `reeflog tail`.~~ **User zero exists**: [auroraxo/whirlpool-watcher](https://github.com/auroraxo/whirlpool-watcher) (2026-10-31) is a real out-of-tree consumer — separate repository, no reeflog code linked, the `Traded` decode verified from live mainnet independently. Measured against a direct-RPC poller on the same window: 7/7 checks, including `window_superset` (0 in-window trades missing), `decoded_amounts_match` (two independently written decoders agree) and `chain_witness` (every sampled fill's amounts appear as real token-balance deltas); the watcher made **0 RPC calls** for 138 fills where the poller spent **85 calls** for 19 trade-bearing transactions. Its mutation probe goes red as required. What remains out of scope: this consumer is still the mailbox owner's own (user zero, not user one), and a third party adopting it is what would close this item.
1904
- **Correction, measured 2026-10-31**: the second half of this item as originally written ("my own RFQ fill polling has not been moved over yet") named work that does not exist. A scan of all 28 repositories of mine on this machine plus the home directory for an RFQ fill poller found none — no such script, service or file exists. The premise behind that sentence was the falsified "21 own key-load sites" claim from 24.10, the same category of error: a sentence that repeated a plausible-sounding inventory without checking it against the disk. The honest remainder of this item is the one that was always the real gate: **a third party using reeflog**, not the migration of a poller nobody ever wrote.
1905
- **A second out-of-tree consumer landed the same day** (2026-10-31): [auroraxo/sememe](https://github.com/auroraxo/sememe) harvests the envelope's `instructions[]` instead of paying RPC (0 enforced attempts; 10/10 sampled transactions byte-identical against independently fetched chain data, 10/10 no-target corroborated, mutation red at the named check), its reader contract and one retracted claim of its own in [issue #1](https://github.com/auroraxo/reeflog/issues/1). Two consumers, two different byte surfaces (logs/fills and instruction bytes), both in my own repositories — still no third party.
1906
- **The install blocker is gone (2026-11-01, v0.1.0)**: before this release a third party could not adopt reeflog at all — there was no tag, no release, no install path, and the binary could not even say which version it was. `v0.1.0` is now a real release (`cargo install --git https://github.com/auroraxo/reeflog --tag v0.1.0`, per-platform archives attached to the [release](https://github.com/auroraxo/reeflog/releases/tag/v0.1.0)), the binary reports its own version, and the release workflow refuses to publish a tag whose version the binary disagrees with. Proven from a stranger's empty directory: installed from the tag, then journaled 535 live mainnet events with 0 gaps. That removed the *install* blocker; the *adoption* blocker (an unaffiliated third party) is untouched and remains the real remainder of this item.
1907
- **Distribution opened, adoption instrumented (2026-11-02)**: the first earned placement is submitted as [StockpileLabs/awesome-solana-oss#93](https://github.com/StockpileLabs/awesome-solana-oss/pull/93) (Data section, per that list's CONTRIBUTING; the list merges about half of what outsiders send — 24 merged vs 22 closed at submission time, so a merge is not assumed). And "no third party" is no longer a sentence in a README but a measured verdict: `node tools/measure-adoption.mjs` classifies external use from non-owner stars/forks/issues/PRs, non-GitHub referrers and the live state of the placement PR. Baseline committed in `research/adoption-2026-11-02.json`: 0 non-owner stars, 0 forks, 0 external issues/PRs, 0 external referrers, 14-day views 0/0 — verdict `not-distributed-not-consumed` (the 356 clones in the same window are our own CI checkouts, explicitly not an adoption signal). Controls: the probe flips to `user-one-evidence` on a repo with real outsiders (100 non-owner forks, outsider issues), and prints `NO VERDICT` with exit 3 when a data source is partial — a merge alone will read as `distributed-not-yet-consumed`, never as adoption.
1908
- 2. Yellowstone source as a second transport.
1909
- 3. Observe a real **mainnet** fork end to end: the detector is now proven
1910
- against a real fork of a real chain (`tools/e2e-fork.mjs`, 47/47 checks,
1911
- 2026-10-29) and against a planted signature on live mainnet — but no
1912
- mainnet fork has been caught while watching. The daemon's loop makes that
1913
- plausible to catch; only an observed one closes it.
1914
- 4. ~~An account envelope says what an event *did* to balances… The next envelope would carry the account list a transaction touched — addresses, owners, writable flags — so a consumer can decide which accounts it cares about without a second RPC round trip.~~ **Landed 2026-10-31**: `accounts[]` carries every account a transaction touched with `signer`, `writable` and `source`, derived from the message header and the two loaded groups (or stated by the response where it says so), with `accounts_flags_source` recording which. Falsified live against `jsonParsed`'s authoritative flags: 346/346 keys over 8 transactions including lookup tables, 0 mismatches; the loaded path is pinned offline by a real transaction fixture.
1915
- **Ownership landed 2026-11-01**: `hydrate --owners` resolves the owning program of every address the envelope enumerates (batched `getMultipleAccounts`, 12 calls for 1,111 addresses in the live e2e), and changed token accounts name their token program for free inside the transaction response. Falsified live: 412/412 stored owners agree with an independent re-fetch, 57/57 addresses the chain no longer has are recorded in an explicit `missing` list with **no** owner entry (never guessed), and the probe pins the authority three ways (stated vs decoded from the account's own bytes vs the map): 12/12.
1916
- **Account bytes landed 2026-11-02**: `hydrate --account-data` stores what each enumerated account HELD — the bytes, not just the owner — batched (11 calls for 964 distinct addresses in the live e2e), capped at 16 KiB per address with the real size and a verified truncation flag, program code flagged instead of stored, and absence recorded as `missing`. Premise measured before building (`tools/probe-account-data.mjs`, 428 distinct addresses over 30 live transactions): p50 165 B, p95 4483 B, 5 addresses over the cap, and a 1,048,588-byte slab — which is why the cap exists and why program binaries are not journalled as event data. Falsified live: 258/258 stored sizes equal the chain's byte length; 232 of 258 re-reads byte-identical, 17 live-moving, 4 in the mutable region of a classic token account, 5 unattributed (no immutability claimed — reported, not waved through); the mint/owner decoded from the stored bytes matched the transaction's own token metadata 12/12; 82/82 program accounts flagged byte-free with correct sizes; 47/47 recorded-missing addresses confirmed absent on an independent read; a second run stored nothing and cost 0 calls.
1917
- **What remains open is the half that is genuinely different**: account bytes AS OF the event (historical state), not bytes at fetch time. That is a different problem, and it was last written here as "needs the chain's history, not a second fetch" — **measured 2026-11-02, that sentence was wrong in its remedy**: the history has no account bytes to serve (6 predecessor transactions, 0 account-state payloads), and a past-slot parameter on a public RPC answers with current state, not the requested slot. Pre-event bytes need a data source this RPC class does not offer; the two dead routes and the measured boundary are in *Evidence and limits* (`tools/probe-pre-event-bytes.mjs`). The same probe found the one thing history does prove — for 7/60 touched accounts with no later touch, fetch-time bytes ARE the post-event bytes (witness 4/4 lamports, 2/2 token fields, wrong-side guard 0/3; 3 certified accounts already gone stay counted but unwitnessed) — measured, deliberately not shipped as an envelope field, and it certifies the post-event side only.
1918
- **Released as v0.2.0 (2026-11-02)**: this owners + account-bytes envelope is the release. The historical-state half above remains the open item it was the day before.
229
+ MIT — see [LICENSE](LICENSE).