sememe 0.4.24 → 0.4.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,21 @@
2
2
 
3
3
  A local meaning layer for Solana bytes. `sememe pull <program>` fetches and verifies a program interface; every process on the machine can then decode instructions, accounts, and events offline.
4
4
 
5
- **Status: pull/store plus an offline decoder for instructions, Anchor event self-CPIs and account state, all runnable (October 2026), verified against nine real mainnet programs (Whirlpool, Jupiter AMM v6, Kamino Lend, OpenBook v2, Marinade, Meteora Vault, Meteora DAMM v1, Meteora DAMM v2/cp-amm and pump.fun), with union decode across a program's on-chain IDL and its adopted first-party published copy (`sememe adopt` + `sememe decode --union`, measured 915/996 on real pump.fun traffic) **and the same arbitration for account bytes (`sememe account --union`, 2026-11-08: measured on 12 real mainnet accounts, byte-exact on 6 of 12 where the single-receipt path managed 0 of 12)**, plus six programs that publish no IDL at all — the two token programs (SPL Token, Token-2022), the two native programs (System, ComputeBudget), Raydium AMM v4 and the SPL Associated Token Account, all six with wire tables DERIVED from pinned upstream Rust — and the ecosystem-wide measured coverage is now 88.5% (18,895/21,350 payloads across 6 finalized blocks, 2026-11-12; 96.4% on the curated watchlist, 62.4% outside it; byte-exact re-encode on 93.4% of decoded), with all 1,410 ATA payloads and all 1,214 Lighthouse payloads decoded byte-for-byte (the Lighthouse copy adopted from the program's own published IDL, hash-pinned, and labelled `adopted` apart from the 17,681 `pulled` decodes) — and one own repository reading it for both checking and producing recipes (`solana-tx-remix`), no external users.** The 20-program coverage qualification does not show demand or general ecosystem coverage. The fingerprint lockfile ships (`sememe lock` + `sememe verify`, 2026-11-05); catalog import and daemon are still proposals.
5
+ **Status: v0.4.25 — published on the npm registry AND as a git tag. Pull/store plus an offline decoder for instructions, Anchor event self-CPIs and account state, verified against real mainnet programs; union decode across a program's on-chain IDL and its adopted first-party published copy (`sememe adopt` + `sememe decode --union`); the fingerprint lockfile ships (`sememe lock` + `sememe verify`); the decode path installs zero runtime dependencies. The ecosystem-wide measured coverage is 88.8% (18,967/21,350 payloads across 6 finalized blocks, 2026-11-13; 96.4% on the curated watchlist, 63.8% outside it; byte-exact re-encode on 93.1% of decoded). The honest limits ride with every number: zero external users so far — publish is not consumption — one machine, no audit.**
6
+
7
+ Everything below the fold is what you run. The full measurement and qualification history — every number, artifact, defect found and fixed, and every honest limit — was moved verbatim to [docs/measured-history.md](https://github.com/auroraxo/sememe/blob/main/docs/measured-history.md) on 2026-11-13; nothing was deleted.
8
+
9
+ ## The idea
10
+
11
+ "What do these bytes mean?" is currently either a paid cloud feature or a decoder rebuilt inside every repository. Vendored IDL files are snapshots without provenance, a fingerprint, or a freshness check.
12
+
13
+ sememe is a local, content-addressed schema store with a verification contract:
14
+
15
+ - **pull:** fetch an interface from an on-chain IDL, a curated catalog, or a local file.
16
+ - **validate:** fingerprint the canonical form, check discriminator uniqueness, cross-check the chain where possible, and tolerate Anchor IDL format changes.
17
+ - **resolve (now implemented for instruction data):** decode and encode locally from the stored IDL, with strict errors instead of guesses. A program without a trustworthy source stays explicitly unverifiable.
18
+ - **store:** keep records under `~/.sememe/`; `sememe.lock` pins exact fingerprints per project (**ships** — see *Working now: pin the answers*).
19
+ - **resolve:** decode and encode locally, with strict errors instead of guesses. A program without a trustworthy source stays explicitly unverifiable.
6
20
 
7
21
  ## Quickstart: the copy-paste 1,000-transaction check (2026-11-04)
8
22
 
@@ -39,100 +53,10 @@ node tools/stranger-decode.mjs \
39
53
 
40
54
  Measured on that sample (2026-11-04): 1,000 checked transactions, 6,337 payloads, 6,337 decoded, 0 decode errors, 0 disagreements, 0 no-interface, 100% byte-exact across 15 programs, 0 network attempts. `--absent error` (the default) refuses the witnessed short forms on purpose; `verified` applies only forms a chain witness covers, `report` names the absent fields without evidence — the same switch the product CLI exposes.
41
55
 
42
- ## Ecosystem coverage measured (2026-11-11): the denominator the watchlist never gave
43
-
44
- The 100% figure above is real for what it measured: **contiguous mainnet traffic filtered to the 22 programs this repo curates**. The same committed artifact (`research/traffic-coverage-2026-11-04.json`) also reports, separately and honestly, 320 distinct programs / 36,027 instructions / 22,622 transactions OUTSIDE that list — counted, bytes not stored, explicitly "NOT in the coverage denominator". Nobody had ever measured that outside traffic.
45
-
46
- `tools/ecosystem-coverage.mjs` measures it. Capture takes 6 contiguous finalized blocks (2,436 transactions, 22,265 instructions, **192 distinct programs**), ranks programs by observed traffic volume, measures the watchlist plus the top 24 outside programs (46 programs, 21,350 payloads), pulls each interface through the product's own acquisition path and installs it through the product's own write path (`installIdl`), then decodes through the product's own entry points. Store-contents never influence which programs are measured.
47
-
48
- Result (`research/ecosystem-coverage-2026-11-11.json`, pinned by `test/ecosystem-coverage.test.mjs`, 7 tests, falsification proven: a hand-doctored artifact turns 3 of them red):
49
-
50
- | denominator | payloads | decoded | coverage |
51
- |---|---|---|---|
52
- | curated watchlist (the claim's own) | 16,382 | 15,797 | **96.4%** |
53
- | outside the watchlist (never measured) | 4,968 | 3,098 | **62.4%** |
54
- | all measured traffic | 21,350 | 18,895 | **88.5%** |
55
-
56
- **pfeeU's 459 refusals in this window are ONE instruction the on-chain interface never learned, and the card's own candidate for closing them was byte-identical to what the store holds (measured 2026-11-12, `tools/measure-pfee-second-source.mjs` -> `research/pfee-second-source-2026-11-12.json`, pinned by `test/pfee-second-source.test.mjs`, 10 tests).** Every refusal is the same 57-byte face, unknown instruction discriminator `9aed8a5ca202a2bb`, and both halves of the planned closing step measured wrong: the MIT candidate the card named (`needsmorergb/charlie-protocol-v1`, `_idl_pfeeUxB6.json@eb852415`) canonicalizes to the SAME fingerprint as the stored on-chain pull (`9c1ef6ed…`, 29 instructions, same discriminator per name — adopting it would add a receipt and close nothing), and a live pull at slot 468428470 (capture: 468262003) returned that same stale copy, so the lag is live on chain, not stale in the store. The name→bytes binding is computed here, not trusted from any repo: Anchor's own rule `sha256("global:get_fees_with_quote_mint")[0..8]` produces exactly `9aed8a5ca202a2bb`, sanity-checked against `get_fees`, whose discriminator both copies already store. The source that carries the instruction is first-party — `pump-fun/pump-public-docs@639b87f`, `idl/pump_fees.json` (31 instructions) — and adopting it through the product's one write path into a throwaway store copy decodes **464/464, 0 refusals, 464/464 byte-exact** on the same committed bytes, with all 5 baseline names kept and the pull still answering its own 5 (`adoptedAnswered: 459, pulledAnswered: 5`, receipt `slot: null`). Same license face as pump: the repo ships no LICENSE file, so the bytes stay OUT of this repository and the npm package — `committedStoreModified: false`, and the headline above stays 88.5% on the shipped store.
57
-
58
- **Two acquisition faces were closed since that gap was first measured, and the report names the evidence class of every decode.** The SPL Associated Token Account (`ATokenGPvbd…`, 1,410 payloads, v0.4.18) got a wire table derived from pinned upstream Rust. Lighthouse (`L2TExMFK…`, 1,214 payloads — the same program the 2026-11-04 acquisition-gap sweep already found publish-nothing at 42 payloads) turned out to be the case where the program's OWN repository publishes an IDL: `sememe adopt-lighthouse` adopts that first-party copy (`Jac0xb/lighthouse@4c579479`, MIT) through the product's own write path, hash-checked against a vendored pin BEFORE anything is written (falsification ladder `test/lighthouse-acquisition.test.mjs`, 9 tests, 7 red on the tree without the copy; measurement artifact `research/lighthouse-acquisition-2026-11-11.json`). 2,624 payloads moved from *no interface at all* to *decoded*, and coverage of measured traffic moved 75.6% → 82.2% → 87.9% → **88.5%** (outside the watchlist 7.1% → 35.4% → 59.9% → **62.4%**, the last step from the stored-short-discriminator fix below).
59
-
60
- An adoption and an on-chain pull are **different evidence classes**, and the headline keeps them apart: `decodedVia` reads `pulled: 17,681, adopted: 1,214`. The pulled number is slot-bound by the chain; the adopted number is hash-bound to a reviewed copy whose origin stays **declared** — nothing checks that the commit is signed or that the bytes describe the deployed program; the evidence for THAT is the byte-exact re-encode on 1,214/1,214 live payloads. The watchlist view carries **zero** adopted decodes: the curated 96.4% still rests entirely on on-chain receipts.
61
-
62
- **pump.fun's 442 refusals in this window are source lag, and a second source closes every one of them (measured 2026-11-12, `tools/measure-pump-second-source.mjs` -> `research/pump-second-source-2026-11-12.json`, pinned by `test/pump-second-source.test.mjs`, 7 tests).** Inside the watchlist pump.fun decoded only 400/842 (47.5%) because this capture store holds the on-chain IDL account alone (47 instructions @slot 468262003) while the deployed program emits discriminators it never learned (`e1f7501ed` x251, `20f6bf340` x98, `1c92de772` x84, plus 9 more payloads). Adopting the program's OWN published copy (`pump-fun/pump-public-docs@639b87f`, `idl/pump.json`, 56 instructions) beside the pull — through the product's one write path, into a throwaway copy of the store — decodes **842/842, 0 refusals**, the same committed bytes. The names of all 400 baseline decodes are unchanged (0 renames) and 151 of them are re-answered by the adopted copy where it explains the documented TradeEvent tail byte-exactly (23 go from trailing-bytes to full byte-exact). One license face the pin refuses to look away from: pump-public-docs ships **no LICENSE file**, so its bytes are NOT vendored into this MIT repository or the npm package — the committed artifact carries the sha256 fingerprint and the store gains no receipt (`committedStoreModified: false`); the measurement reads the canonical object that already sits in a committed test fixture and reproduces it at run time. That is why the ecosystem headline above stays at 88.5% on the shipped store: the published copy is not in it. For a consumer who adopts it themselves, the watchlist face moves from 96.4% (582 refusals) to 442 of those refusals closed — the remaining 140 belong to other programs.
63
-
64
- The remaining gap stays **acquisition-led**: 1,013 of the 2,455 uncovered measured payloads still have no interface at either pullable address (`FLASH…` 134, `Diab…` 102, …; the queue's former head `MNFSTqtC9…` 307 is now measured closable by the program's own fuzz wire table — above) and 1,135 are **source lag on interfaces that ARE acquirable**: pfeeU… refuses 459 of 464 payloads against its freshly pulled anchor IDL — now measured to a single instruction the account does not list even at slot 468428470, closed by the first-party published copy as above; HADRo… refuses 72 of 195 since 2026-11-12 — its total-refusal face was NOT source lag but a decoder bug (below). Inside the watchlist the same lag shows up: pump.fun — the program the union path decoded 915/996 for — decodes 400 of 842 payloads (47.5%) from the single on-chain source in this window; the union's second source is what carried the old number, and this measurement pulls only on-chain copies. 915 payloads from 152 tail programs were counted, not measured; the report names every one of them so the denominator cannot quietly shrink.
65
-
66
- **Manifest's 307 no-interface payloads — the acquisition queue's head — are closed by the program's own generated wire table, and the defect face is the upstream's own client copy (measured 2026-11-12, `tools/measure-manifest-second-source.mjs` -> `research/manifest-second-source-2026-11-12.json`, pinned by `test/manifest-second-source.test.mjs`, 10 tests).** `MNFSTqtC9…` publishes no IDL account at either address (the 11-11 queue names it first: 307 payloads, acquirable:false). The program's own repository — `Bonasa-Tech/manifest`, the annotated successor of `CKS-Systems/manifest`, which this repo's 11-04 parser-upstream research already cited — ships two copies at the SAME commit `d4e908b3`, and only ONE of them is a wire table: `fuzz/crucible/idls/manifest.json` (15 shank instructions, zero duplicate discriminants). Adopting it through the product's one write path into a throwaway store copy decodes **307/307, 0 refusals, 307/307 byte-exact** on the same committed bytes, names per discriminant Deposit 46 / Withdraw 46 / Swap 3 / BatchUpdate 201 / GlobalWithdraw 1 / SwapV2 10. The client copy (`client/idl/manifest.json`, 16 instructions) lists **"Swap" and "SwapV2" on discriminator 04** — the product refuses every one of the 307 payloads by name (`duplicate instruction discriminator 04 (Swap / SwapV2)`), measured against the upstream's own bytes, not a synthetic fixture: an adopt-and-measure area that only ever showed the sunny path would have trusted a table that matches no wire. The pin refuses either copy if it drifts (canonical-fingerprint gates on BOTH files, plus a gate that the claimed duplicate is still present in the client copy and absent from the wire table). License face, stated: the repo is **GPL-3.0** at the pin — its bytes are NOT vendored into this MIT repository or the npm package, the committed store gains no receipt (`committedStoreModified: false`), and the ecosystem headline stays 88.5% on the shipped store. For a consumer who adopts the pinned copy themselves, the queue's largest entry closes.
67
-
68
- What this changes about the thesis, stated plainly: sememe's decode path is strong on interfaces it holds (93.4% byte-exact of decoded across all 46 programs), and the honest ecosystem number moved from ~75% to 88.5% — because two named acquisition gaps were CLOSED with pinned, byte-exact artifacts and one decoder bug was fixed against measurement, not because the denominator was softened: the watchlist number stayed 96.4% and the outside denominator stayed 4,968 payloads. The product's remaining value step is still acquisition breadth — the queue's head is now `MNFSTqtC9…` (307) — not decoder polish. "no-interface" remains the named, counted, ranked queue (`acquisitionQueue`), which is where the next cards come from.
69
-
70
- **HADRo's 195 refusals were NOT source lag — stored discriminators are stored at ANY length (fixed and measured 2026-11-12, `tools/measure-hadro-short-discriminator.mjs` -> `research/hadro-short-discriminator-2026-11-12.json`, pinned by `test/hadro-short-discriminator.test.mjs` 7 tests + `test/hadro-short-discriminator-measurement.test.mjs` 5 tests).** The ecosystem measurement had labelled HADRo (`HADRo…VDQ8`) a "total source-lag face" at 0/195. It was wrong, and the PULS V two-tree measurement (`git archive c304e80` for the pre-fix tree, a byte-identical probe in both) proved why: the program's published IDL stores **38 one-byte instruction discriminators** — a real, chain-accepted wire table (unique values 0x00..0x26) — and `withDerivedDiscriminators` treated only 8-byte arrays as "stored", so every stored byte was OVERWRITTEN with Anchor's 8-byte sighash, against that module's own header rule ("stored discriminators are never changed"). The replaced table matches no payload: pre-fix 0/195 decoded; post-fix **123/195 decode through the product path** (51 full byte-exact, 72 `setCurve` described-exact with reported trailing bytes), and the remaining 72 (`setRiskCurve`) stay **honestly refused and NAMED** — that IDL declares a vector length the wire bytes do not carry. The same `length === 8` assumption in `src/union.mjs`'s rename-evidence map was fixed the same way (stored means any non-empty byte array). The fix is surgical and the measurement pins that: pump.fun (400/842, 249 byte-exact), pfeeU (5/464) and Eternal (227/242 byte-exact) verdicts are BYTE-IDENTICAL between the trees; Eternal's 76 stored 1-byte EVENT discriminators are now preserved too (the event dispatch path reads 8-byte heads, so that preservation changes no decode in this window — stated, not smoothed). Honest limits ride in the artifact; the unit face flips `false -> true` on both instruction and event preservation, and the store face `0/38 -> 38/38`.
71
-
72
- The stranger-facing truth this pins down: **the 1,000-transaction decode is one command, offline, and fails loudly.** What it does not prove is that an outside person has run it — and that is now **measured, not asserted** (`tools/measure-foreign-run.mjs` -> `research/foreign-run-2026-11-05.json`): verdict `not-distributed-not-consumed`, i.e. zero non-owner signals on every channel that was read — 0 non-owner stars/forks/issues/PRs on this repo, 0 non-owner comments on the two outbound threads (`step-finance-oss/step-ingestooor#1` and `jup-ag/rfq-webhook-toolkit#96`, both open), 0 foreign-authored threads containing this project's slug. The first external run therefore remains the open gate. The measurement has limits and names them: the broad mention sweep covers 92 of 93 results (GitHub's search index hides one result in every ordering — measured, not assumed), a registry `404` for the bare name `sememe` on npm/crates.io/PyPI meant **no install channel was published yet** at that time, never "nobody used it" — npm published the name on 2026-11-12 (crates.io/PyPI still have no `sememe`; both are aliasing checks, not consumption evidence), and the code-index absence is only claimed for the indexed part of GitHub (the sweep's self-reference control passes). Reproduce with `npm test` (includes `test/stranger-decode.test.mjs`, which runs this tool against both fixtures and falsifies it three ways: tampered payload, missing store, and a mutation of the pass/fail logic). The funnel a stranger walks before that first run is measured separately (2026-11-05, `tools/measure-stranger-funnel.mjs` -> `research/stranger-funnel-2026-11-05.json`): a fresh GitHub clone ran both quickstart commands verbatim (1,089 and 6,337 payloads, all decoded byte-exactly, exit 0), the published tag `v0.3.0` installed into an empty app with no npm account, and its entry decoded 1,119/1,119 real payloads byte-exactly from `node_modules/sememe` — the harness (tools/test/research) does not travel into the install. The quickstart block itself is pinned by `test/quickstart-funnel.test.mjs`, which caught and fixed one defect on 2026-11-05: the documented output line was the pre-2026-11-04 schema (missing `rpcEncoding`, `chainParsedInstructions`, `missingBytes`) — a stranger comparing her terminal to the README saw two different products. The README now documents the line the tool actually prints, and a hand-edited documented line goes red. Re-measured 2026-11-13 after the registry channel opened (`research/foreign-run-2026-11-12.json`): the registry check reads `npm: found` for the bare name while the verdict stays `not-distributed-not-consumed` — publish is not consumption, and the first external run remains the open gate.
73
-
74
- ## Your own transactions: raw RPC responses (the shape you already hold)
75
-
76
- The two commands above decode sememe's fixtures. A repository, archive, dashboard or indexer does not hold sememe fixtures — it holds **raw RPC responses**, overwhelmingly with the default `encoding: "base64"`, where the transaction is its wire bytes and only the chain's own parser can read it. sememe now reads that shape directly (legacy, v0 and v1 transactions, outer and inner instructions) and **verifies each transaction's ed25519 signatures before decoding it**, so a damaged or forged byte is refused by name (`wire-signature-invalid`) instead of decoded wrongly:
77
-
78
- ```bash
79
- node tools/stranger-decode.mjs \
80
- --store test/fixtures/traffic-sample-store-2026-11-04 \
81
- --rpc-response your-getBlock-or-getTransaction-response.json \
82
- --absent verified
83
- ```
84
-
85
- Accepts a `getBlock` response (`result.transactions`), a `getTransaction` response (`result.transaction`), or an archive document carrying a `transactions` array — each entry in the `["<base64 wire bytes>", "base64"]` shape the RPC returns. Exit codes: 0 pass, 1 a decode disagreement, an unreadable/damaged transaction or a decode error, 2 usage/IO. A transaction whose bytes cannot be parsed or verified is counted under `transactionsUnreadable` with its reason in `unreadableReasons` and fails the run — never silently skipped.
86
-
87
- Measured 2026-11-04 on `test/fixtures/foreign-archive-wire-4blocks.json` (real mainnet, contiguous slots 464588080–464588083, 1,947 transactions in their raw base64 wire shape): the first 1,000 transactions carry 6,355 payloads across 115 programs, 4,646 decoded byte-exactly (100%), 0 decode errors, 0 disagreements, 0 unreadable, 0 network attempts. The 1,709 payloads the store has no interface for are reported by program, not dropped. The wire parser itself was validated against the chain's own `encoding:"json"` parser for all 648 transactions of slot 464588080: per-payload, byte-for-byte identical rows (3,528 payload rows, 0 mismatches), pinned in `test/rpc-response-ingest.test.mjs`.
88
-
89
- **Honest limits:** the store answers only for the programs it carries (no-interface payloads are counted, not decoded); signature verification covers the static signers the message declares; v1 transactions are parsed per the upstream `MessageV1` layout pinned on 2026-11-04 and verified against the json encoding of the same block; and no third party has run this on their own archive yet — that remains the gate this command exists to make runnable.
90
-
91
- ### The shape a foreign repository's own test fixtures hold (`encoding:"json"`)
92
-
93
- The paragraph above measured what an RPC *returns by default*. But foreign repositories that keep transaction fixtures do not keep that shape: they keep what the chain's parser already read — `encoding:"json"` — and some hold a **flattened dialect** where `instructions` and `accounts` sit at the file level (37 of the 156 fixtures in [step-finance-oss/step-ingestooor](https://github.com/step-finance-oss/step-ingestooor)). Measured on that entire 156-fixture set and the fixture of [CryptGodSon/Solana-Explorer](https://github.com/CryptGodSon/Solana-Explorer) (both MIT): instruction `data` as base58, every instruction carrying `programId` spelled out, address-table keys expanded, and **1,193 instructions already `parsed` by the chain with no bytes left in the document at all**.
94
-
95
- Before this, `--rpc-response` read only the base64 shape: pointed at a foreign repository's own fixture it exited 2 (`not-an-rpc-response`). The integration could not read a single artifact a foreign repository actually holds — because the one shape it accepted was the shape sememe's own harvester produces. The same command now reads both shapes, plus the flattened dialect:
96
-
97
- ```bash
98
- node tools/stranger-decode.mjs \
99
- --store test/fixtures/traffic-sample-store-2026-11-04 \
100
- --rpc-response their-own-getTransaction-fixture.json
101
- ```
102
-
103
- Or sweep a whole directory tree of fixtures with one command:
104
-
105
- ```bash
106
- node tools/stranger-decode.mjs \
107
- --store test/fixtures/traffic-sample-store-2026-11-04 \
108
- --fixtures-dir path/to/foreign/crates/parsers/tests \
109
- --out report.json
110
- ```
111
-
112
- Measured 2026-11-04 across all 156 fixtures of `step-finance-oss/step-ingestooor` (commit 001cf39, MIT; research artifact `research/step-ingestooor-full-sweep-2026-11-04.json`): **156 files (119 standard getTransaction, 37 flattened dialect), 806 payload rows, 493 decoded byte-exactly (100%), 0 unreadable transactions, 0 disagreements, 0 network attempts**. The 306 payloads the store has no interface for are quantified by program across 70 gap programs (top: Raydium Concentrated Liquidity `L2TExM…` 42, Jupiter DCA `DCA265…` 20, Jito Tip `j1o2qR…` 14, Flash Trade `FLASH6…` 12). Seven payloads refuse with named reasons (3 stale interface, 4 unwitnessed short forms). Pinned in `test/step-ingestooor-full-sweep.test.mjs` (5 tests, red on pre-tree, four mutations each individually red).
113
-
114
- Measured 2026-11-04 on the SAME 156 fixtures after closing the measured acquisition gap (`research/acquisition-gap-2026-11-04.json`: 34 of the 70 gap programs publish an on-chain IDL sememe had never pulled): the 34 IDLs were pulled in two batched probes (`tools/batch-pull.mjs`, slots 465308534/465308538, same write path as `sememe pull`) into `test/fixtures/traffic-sample-store-2026-11-04-pulled/`, and the 13 of them that also have a first-party published copy in the foreign repo were adopted next to the pull (union decode arbitrates by bytes, naming-convention differences `closeDca`/`close_dca` and one discriminator-preserving rename decode instead of misreading as interface conflicts — `src/union.mjs`, pinned with 6 new tests and 2 mutations in `test/union.test.mjs`). Result (`research/step-ingestooor-full-sweep-pulled-2026-11-04.json`, pinned in `test/batch-pull.test.mjs`): **noInterface 306 → 129, decoded 493 → 661 (byte-exact 100%, 0 disagreements), decodeErrors 7 → 16 — every remaining refusal named** (deployed-newer discriminators the published IDL does not list, stale IDL layouts against successfully executed transactions, unwitnessed short forms). The pull alone did NOT decode the 177 payloads: 152 decode with the on-chain IDL, 25 refuse for named reasons, 16 more decode only through the union with the published copies.
115
-
116
- Measured on the seven committed fixtures (six from step-ingestooor, one from Solana-Explorer, verbatim under `test/fixtures/foreign-repo-json-2026-11-04/` with both MIT licenses): 7 transactions, 160 payload rows of which 56 carry bytes — 48 decoded byte-exactly (100%), 8 named `noInterface` (programs outside the store), and **104 chain-parsed rows counted as `missingBytes`, never fed to the decoder as empty payloads**: a chain-parsed instruction has no bytes to decode, and turning that missing document field into a decode refusal would be a lie about what was measured. Zero disagreements, zero decode errors, zero network attempts. Pinned in `test/foreign-repo-json-ingest.test.mjs` (red against the pre-fix tree; four mutations of the new code each individually red).
117
-
118
- **Honest limits of this path:** the json encoding carries **no message bytes, so no signature can be verified against them** — the ed25519 self-verification of the base64 path cannot apply, and a structurally-valid value mutation inside an argument passes decode+re-encode by design (only the wire path's signature check covers value-level tampering). The encoding is detected per transaction; an unresolvable `programIdIndex` is refused by name rather than attributed to nothing. And the same gate as before: no third party has run this against their own repository — the command exists so they can.
119
-
120
- ## The idea
121
-
122
- "What do these bytes mean?" is currently either a paid cloud feature or a decoder rebuilt inside every repository. Vendored IDL files are snapshots without provenance, a fingerprint, or a freshness check.
123
-
124
- sememe is a local, content-addressed schema store with a verification contract:
125
-
126
- - **pull:** fetch an interface from an on-chain IDL, a curated catalog, or a local file.
127
- - **validate:** fingerprint the canonical form, check discriminator uniqueness, cross-check the chain where possible, and tolerate Anchor IDL format changes.
128
- - **resolve (now implemented for instruction data):** decode and encode locally from the stored IDL, with strict errors instead of guesses. A program without a trustworthy source stays explicitly unverifiable.
129
- - **store:** keep records under `~/.sememe/`; `sememe.lock` pins exact fingerprints per project (**ships** — see *Working now: pin the answers*).
130
- - **resolve:** decode and encode locally, with strict errors instead of guesses. A program without a trustworthy source stays explicitly unverifiable.
131
-
132
56
  ## Install (as a consumer, not a contributor)
133
57
 
134
58
  ```bash
135
- npm install github:auroraxo/sememe#v0.4.24 # tagged release; no npm account needed
59
+ npm install github:auroraxo/sememe#v0.4.25 # tagged release; no npm account needed
136
60
  ```
137
61
 
138
62
  ```js
@@ -151,14 +75,14 @@ The **library** refuses the same way: importing `sememe/pull` or `sememe/gap` wi
151
75
  **From an install (not a clone), the CLI is `npx sememe …`** — npm exec resolves the package's own `bin` from `node_modules`; the bare command `sememe` is not on your `PATH` after a plain `npm install`, and `node bin/sememe.mjs` works only inside a clone of the repository (both measured 2026-11-05 from a real tarball install). The consumer journey from the install, verbatim:
152
76
 
153
77
  ```bash
154
- npm install github:auroraxo/sememe#v0.4.24 @solana/web3.js # acquisition needs the peer
78
+ npm install github:auroraxo/sememe#v0.4.25 @solana/web3.js # acquisition needs the peer
155
79
  npx sememe pull whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc --store ./store
156
80
  npx sememe decode whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc 2b04ed0b1ac91e62a40d0100000000000100000000000000af331ba8327fbb35b1c4feff00000000010001010000000601 --store ./store
157
81
  ```
158
82
 
159
83
  `./store` is the local content-addressed store the error `no sememe store at ./store: run "sememe pull" first` tells you to create — the `pull` above is that creation step; the vendored programs (System, ComputeBudget, SPL Token, Token-2022, Raydium AMM v4, SPL Associated Token Account) need no store at all: `npx sememe decode-native 11111111111111111111111111111111 <payload>` answers offline from the pinned upstream tables. If you ever point the decoder at a store directory that does not exist yet, the refusal names the fix; `npx sememe` is the form of that fix that runs where you are. Measured 2026-11-05 from the packed tarball of this tree: install (zero deps) → `npx sememe pull` (live mainnet) → `npx sememe decode` (byte-exact, live-pulled object byte-identical to the fixture store object `0b886e37…660f`).
160
84
 
161
- Publishing to the npm registry changed on 2026-11-12: the `sememe` name is now live on registry.npmjs.org (first shipped as 0.4.21; the measured registry install of `sememe@0.4.22` into an empty app decoded the Whirlpool fixture payload `swap_v2`, `roundTripByteExact: true`). `npm install sememe` and `npm install github:auroraxo/sememe#v0.4.24` deliver the same product files from the same code: the files allowlist already ships none of this repository's tests, fixtures or qualification harness (pinned by `test/package.test.mjs`), so the git tag remains the release artifact — `test/package.test.mjs` (the contract above), `tools/e2e-package-consumer.mjs` (a real install into an empty directory) and `test/install-surface.test.mjs` (the shipped README's install commands run verbatim from the packed tarball) keep both channels honest.
85
+ Publishing to the npm registry changed on 2026-11-12: the `sememe` name is now live on registry.npmjs.org (first shipped as 0.4.21; the measured registry install of `sememe@0.4.22` into an empty app decoded the Whirlpool fixture payload `swap_v2`, `roundTripByteExact: true`). `npm install sememe` and `npm install github:auroraxo/sememe#v0.4.25` deliver the same product files from the same code: the files allowlist already ships none of this repository's tests, fixtures or qualification harness (pinned by `test/package.test.mjs`), so the git tag remains the release artifact — `test/package.test.mjs` (the contract above), `tools/e2e-package-consumer.mjs` (a real install into an empty directory) and `test/install-surface.test.mjs` (the shipped README's install commands run verbatim from the packed tarball) keep both channels honest.
162
86
 
163
87
  ## Working now: pin the answers (`sememe lock` + `sememe verify`, 2026-11-05)
164
88
 
@@ -182,2660 +106,19 @@ npx sememe verify --store ./store --lock ./sememe.lock
182
106
 
183
107
  `lock` snapshots every program's newest verified receipt into one canonical-JSON document (fingerprint + source + slot + receipt name, one commit-able file); every receipt is re-checked through the store reader first, so a store the reader refuses produces NO lock, never a lock over damage. `verify` re-checks the store against the pin and refuses drift by name — `drifted` (the store now answers with a different fingerprint, both quoted), `missing` (a pinned program has no receipt), `unverified` (an object fails the re-hash), `relabelled` (a DECLARED origin changed), `unbound` (the store declares an origin this lock does not pin, so the gate cannot say whether that label changed — the lock predates origin pinning; the repair is `sememe lock`). An **extra** store program — one the lock does not pin, because the lock pins minimums, not store equality — is **reported, never a violation for being there**; but if its receipt is DAMAGED, that is a violation like any other, marked `scope: "extra"` so the repair is not mistaken for pin drift (the pin is fine; the store's receipt for an unpinned program has to be re-pulled). `--mode error` (verify's default) exits 1 on any violation, so the CI gate is `npx sememe verify` after checkout.
184
108
 
185
- **Measured 2026-11-07 (`verify` must not call clean a store the reader refuses):** a store holding Whirlpool (locked) and Jupiter pulled *afterwards*, so Jupiter is an extra, with Jupiter's object rewritten so it no longer re-hashed to its receipt — the store reader refuses that program, and `sememe decode` on it exits 1 with "object and receipt disagree, refusing to decode". On the published `v0.4.10` install: `sememe lock` exited **1** (no lock written) while `sememe verify` exited **0** with **violations 0** — a green CI gate over a program the reader refuses. The cause was structural, not a typo: `writeLock` re-checked **every** receipt, `checkLock` re-checked only the **pinned** ones, so the shared helper was shared but only ever called on one side. Both paths now re-check the extras through the same helper, and only their FAILURES are reported. See [Working now: `verify` agrees with `lock`](README.md#working-now-verify-agrees-with-lock-about-a-store-whose-extra-program-is-damaged-2026-11-07) and [research/extra-damage-2026-11-07-before.json](research/extra-damage-2026-11-07-before.json) (the published install: 3 contract violations) / [research/extra-damage-2026-11-07.json](research/extra-damage-2026-11-07.json) (this tree: 0).
109
+ **Measured 2026-11-07 (`verify` must not call clean a store the reader refuses):** a store holding Whirlpool (locked) and Jupiter pulled *afterwards*, so Jupiter is an extra, with Jupiter's object rewritten so it no longer re-hashed to its receipt — the store reader refuses that program, and `sememe decode` on it exits 1 with "object and receipt disagree, refusing to decode". On the published `v0.4.10` install: `sememe lock` exited **1** (no lock written) while `sememe verify` exited **0** with **violations 0** — a green CI gate over a program the reader refuses. The cause was structural, not a typo: `writeLock` re-checked **every** receipt, `checkLock` re-checked only the **pinned** ones, so the shared helper was shared but only ever called on one side. Both paths now re-check the extras through the same helper, and only their FAILURES are reported. See [Working now: `verify` agrees with `lock`](https://github.com/auroraxo/sememe/blob/main/docs/measured-history.md#working-now-verify-agrees-with-lock-about-a-store-whose-extra-program-is-damaged-2026-11-07) and [research/extra-damage-2026-11-07-before.json](research/extra-damage-2026-11-07-before.json) (the published install: 3 contract violations) / [research/extra-damage-2026-11-07.json](research/extra-damage-2026-11-07.json) (this tree: 0).
186
110
 
187
111
  What the lock does NOT do: it is not a freshness check against the chain (the pin says "this is what I answered", not "this is what the program is now"); it does not cover vendored programs (System/ComputeBudget/SPL Token/Token-2022/Raydium AMM v4/SPL Associated Token Account have no receipts, they are pinned from upstream sources instead); and it cannot stop you from re-pulling — it makes every re-pull that changes an answer visible.
188
112
 
189
- ## Working now: `verify` agrees with `lock` about a store whose EXTRA program is damaged (2026-11-07)
190
-
191
- A CI gate is only worth its exit code if it refuses what the product itself
192
- refuses. Measured on the **published `v0.4.10` install** (its tag installed
193
- into an empty app, no clone), on a store holding two programs:
194
-
195
- - Whirlpool, pinned by a committed `sememe.lock`;
196
- - Jupiter, pulled **afterwards** — so it is an **extra**: the lock pins
197
- minimums, not store equality, and an extra's mere presence is not a
198
- violation;
199
- - Jupiter's `objects/<fingerprint>.json` rewritten so it no longer re-hashes
200
- to the fingerprint its receipt claims.
201
-
202
- The store reader refuses that program: `sememe decode <jupiter> <payload>`
203
- exits 1 with `object and receipt disagree, refusing to decode`.
204
-
205
- | command | answer | exit |
206
- |---|---|---|
207
- | `sememe decode <jupiter> …` | refused, `object and receipt disagree` | 1 |
208
- | `sememe lock` | refused, no lock written | **1** |
209
- | `sememe verify --mode error` | **`violations: 0`** | **0** |
210
- | `sememe verify --mode warn` | `violations: 0`, `extra: [<jupiter>]` | 0 |
211
-
212
- The writer refused and the checker passed — the same "one store, two
213
- verdicts" class this repo killed for the *newest-receipt* path earlier the
214
- same day, one level down. The cause was structural rather than a typo:
215
- `writeLock` re-checks **every** receipt of **every** program, `checkLock`
216
- re-checked only the **pinned** programs, and the shared helper
217
- (`verifyEveryReceipt`) was therefore only ever called from one side. The
218
- README's promise — "lock and verify cannot disagree about what a clean store
219
- is" — held for pinned programs and silently stopped holding for extras.
220
-
221
- - **The fix:** `checkLock` runs the extras through the **same helper** and
222
- reports only their FAILURES: a damaged extra becomes an `unverified` entry
223
- with `scope: "extra"`, counted as a violation, so `--mode error` exits 1.
224
- What is *not* done: an extra is never held against the lock's origin pin —
225
- an unpinned program has no pin to bind to, and demanding one would turn
226
- "someone pulled a program" into a CI failure.
227
- - **Why `scope` is in the report and in the refusal:** the two failures have
228
- different repairs. A *pinned* name means the pin no longer describes the
229
- store (`sememe lock`); an `extra` name means the lock is fine and the
230
- **store** holds a damaged receipt for a program the lock never pinned
231
- (re-pull that program, or drop the receipt). One list, two repairs — the
232
- refusal names them separately:
233
- `unverified: <jupiter> (extra, receipt <name>)`.
234
- - **The repair is measured, not asserted:** S4 in the tool clears the
235
- violation end to end — drop the damaged receipt+object, re-pull, then
236
- `verify --mode error` exits 0 again. A violation nobody can clear is not a
237
- gate. (The one-step re-pull was measured to *collide*: `adopt` refuses with
238
- `digest collision or corrupted local object`, because the stale bytes still
239
- occupy the fingerprint's path — so the repair is remove-then-pull.)
240
- - **Two guards, green on the pre-fix tree:** an **intact** extra is still
241
- never a violation, and a damaged **pinned** program is still named. The fix
242
- moved neither.
243
-
244
- The measurement ships as a tool and two committed artifacts, following the
245
- repo's "the tool produces the report" rule:
246
-
247
- ```bash
248
- node tools/measure-extra-damage.mjs # this tree → research/extra-damage-2026-11-07.json
249
- node tools/measure-extra-damage.mjs --bin <installed bin> --tag v0.4.10 --out before.json
250
- ```
251
-
252
- [research/extra-damage-2026-11-07-before.json](research/extra-damage-2026-11-07-before.json)
253
- is the published `v0.4.10` install: **3 contract violations** (error mode
254
- exited 0, warn mode counted 0, no scope marker), with both controls green;
255
- [research/extra-damage-2026-11-07.json](research/extra-damage-2026-11-07.json)
256
- is this tree: **0 violations**.
257
- `test/extra-damage-measurement.test.mjs` pins both artifacts and fails if a
258
- committed report stops being what the tool derives (plus a portability check:
259
- neither artifact may carry a host path — the declared-origin artifact had to
260
- be fixed for exactly that on 2026-11-07 13:5x); the 6-test ladder plus three
261
- mutations live in [test/extra-program-damage.test.mjs](test/extra-program-damage.test.mjs).
262
-
263
- **Measured cost of the change** (`verify` on a 60-program store, median of 6
264
- runs after a warm-up): the case the fix adds work to — 1 pinned, 59 extras,
265
- damaged — **236 ms → 637 ms**; the normal committed-lock case where the lock
266
- covers the store (0 extras) — **1123 ms → 1064 ms**, i.e. unchanged within
267
- noise. Extras now cost what pinned programs already cost: one re-hash each.
268
-
269
- **Stated limits:** the lock still does not cover vendored programs or check
270
- freshness against the chain; `scope: "extra"` tells you *where* the damage
271
- is, not *who* wrote it; and the reported extra is an unpinned program, so a
272
- store can still gain a program the lock has never seen (by design — that is
273
- the minimums rule). No third party has run this; there are 0 external users,
274
- 0 stars (`npm publish` was credentials-blocked until 2026-11-12, when the
275
- registry name shipped — see the Install section). The published
276
- `v0.4.10` tag keeps this defect (a tag is immutable); `v0.4.11` carries
277
- the fix.
278
-
279
- ## Working now: a crashed write must not brick the store (2026-11-07)
280
-
281
- Every receipt and object is installed through a temporary file
282
- `<name>.tmp-<pid>-<random>` in the SAME `receipts/` directory and removed in
283
- a `finally`. A crash between the write and the cleanup (SIGKILL, OOM, power
284
- loss) leaves that file behind **forever** — and until now every reader
285
- treated EVERY file in `receipts/` as a receipt. Measured on the published
286
- `v0.4.10` install (the published v0.4.10 tag, installed into an empty
287
- app), one leftover `.tmp-4711-9f2c1a` file broke every surface at once:
288
-
289
- ```
290
- sememe decode <program> <payload> -> EXIT 1 "store receipt <tmp file> is unreadable: Unexpected token"
291
- sememe decode --union <program> ... -> EXIT 1 same refusal
292
- sememe lock -> EXIT 1 "Cannot destructure property 'idl' ... as it is null"
293
- sememe verify --mode error -> EXIT 1 "unverified: <program>" <- the worst one
294
- ```
295
-
296
- The last line is the damage: the gate went **red over an intact program**
297
- and named no repair — the user is told to re-pull a store that is perfectly
298
- healthy. One crashed write permanently bricked a working store. The same
299
- root cause made foreign files phantom programs: `.DS_Store` showed up as
300
- `extra`, and `my-file.json` became a phantom program `my` that failed the
301
- gate over a file that was never a receipt.
302
-
303
- The fix is a single classifier, `src/receipt-name.mjs`, used by every reader
304
- (`store-read`, `lock`), with the rule kept narrow on purpose:
305
-
306
- - `receipt` — a well-formed name (`<program>-<slot>-<source>-<64 hex>.json`):
307
- read and verified exactly as before.
308
- - `malformed` — receipt-shaped (it carries a slot segment) but not
309
- well-formed: this may be a renamed or corrupted receipt, so it stays in the
310
- verified set and is **refused by name** — ignoring it would be a new green
311
- lie, letting a receipt vanish from the store's answers in silence.
312
- - `debris` — sememe's own `.tmp-<pid>-<hex>` leftover: safe to ignore,
313
- because `store-write.mjs` installs by **hard-linking** the temporary to its
314
- final name first. Died before the link and only the temp exists (the write
315
- was never committed); died after the link and the temp is a byte-identical
316
- copy of the committed receipt. Either way it carries no information the
317
- final name does not.
318
- - `foreign` — anything else (`.DS_Store`, a README): not a program, never
319
- read as one.
320
-
321
- Debris and foreign files are ignored **but never silently**: `verify`'s
322
- report gains a `stray` field naming each file and its kind, so a user can see
323
- and delete them. Measured afterwards: decode, union-decode, lock and
324
- `verify --mode error` all answer green over a store with crash debris, while
325
- a damaged receipt is still refused and named (the guard did not move).
326
-
327
- Ladder: `test/receipt-hygiene.test.mjs` (13 tests — G1-G3 green on the
328
- pre-fix tree, T1-T6 red there, 4 mutations caught) and
329
- `test/receipt-hygiene-measurement.test.mjs` (4 tests pinning the artifacts).
330
- Measurements: [research/receipt-hygiene-2026-11-07-before.json](research/receipt-hygiene-2026-11-07-before.json)
331
- (published `v0.4.10`: 6 contract violations) /
332
- [research/receipt-hygiene-2026-11-07.json](research/receipt-hygiene-2026-11-07.json)
333
- (this tree: 0). Honest limits: 0 external users, no audit, the grammar check
334
- against 256 real receipt names found 0 mismatches but no store outside this
335
- host has been observed, and `v0.4.10` keeps the defect (tags are immutable);
336
- `v0.4.11` carries the fix.
337
-
338
- ## Working now: the CLI refuses what it does not understand (2026-11-07)
339
-
340
- A command line you typed wrong must not turn into a *different* command line that happens to run. Measured on the tree before this (v0.4.6, `641b895`): the parser looked flags up **by name anywhere in the argument list** and dropped what it did not find, so an unknown flag was not an error — it was an invisible edit to your request:
341
-
342
- - `sememe lock --store ./store --o out.lock` exited **0** and wrote the lock to `./sememe.lock`, not to `out.lock`. You asked for a file at one path, got a file at another, and the success exit said otherwise.
343
- - `sememe decode <program> <payload> --stro ./store` ignored the typo'd store and read the **default** `~/.sememe`, failing with `no sememe store at /home/…/.sememe` — a message about a directory you never named.
344
- - `sememe verify … --mod warn` dropped `--mod`, so verify ran in its default `error` mode and exited 1 on a drift you had asked it to only warn about.
345
- - `sememe native-provenance --store ./store` read the flag's **value** as the program id: `./store is not a native program sememe pins`.
346
- - `sememe decode --store ./store <program> <payload>` (flag first) made `--store` the program id and `./store` the payload.
347
- - `sememe pull <program> --stor ./store` dropped the typo and **pulled into the default store** — a write you never asked for, in a directory you never named.
348
-
349
- Every command now declares its surface — the flags it accepts and how many positional arguments it consumes — and that surface is checked **before any work happens**: no store is read, no file is written, no RPC is contacted. A refusal names the closest supported flag and the full list, and it exits 2 (the usage channel) rather than the exit 1 reserved for work that actually failed:
350
-
351
- ```bash
352
- node bin/sememe.mjs lock --store ./store --o out.lock
353
- sememe lock: unknown flag "--o" — did you mean "--out"? supported flags: --store, --out
354
- node bin/sememe.mjs decode whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc 2b04ed0b… --store a --store b
355
- sememe decode: flag "--store" given more than once — pass it once
356
- ```
357
-
358
- What is refused, by name: a flag this command does not accept (including a flag belonging to another command, and a flag that is actually a subcommand), a flag given twice — the first value won and the second was silently discarded — a flag with no value, and more positional arguments than the command consumes. What is *not* refused: too few positional arguments, which each command already refuses in its own words (`decode needs a program id`, `pull needs a program id`), so those messages are unchanged.
359
-
360
- The rule is the one the repository already applies to stores, receipts and locks: an unrecognized input is refused by name, never tolerated. `src/cli-surface.mjs` is that table — data, not scattered `if`s — so a flag cannot be added to a command without appearing in one place, and `test/flag-surface.test.mjs` (29 tests) pins the refusals and the repair text, while `test/flag-tag-journey.test.mjs` pins the SAME properties on the bytes a stranger installs — the flag level of the pinned tag, which no instrument measured until 2026-11-07 (measured then on the published `v0.4.7` install: for a flag with no close match the supported list printed **twice in one line**, and a command that takes no flags printed `accepts: (none) supported flags: (none)`; the fix was already on main and the pin pointed behind it — the same class as every pinned-tag defect above, now covered by a journey instead of by luck). Honest limit: this is a behavioural change to argument handling on a package with no external users yet, so nothing downstream has been shown to depend on the old silent acceptance. It shipped the same day it was measured and `v0.4.8` carries both the strict surface and the once-only message: `v0.4.6` still accepts the silent typos above and `v0.4.7` still prints the list twice (tags are immutable), so a pin behind either keeps the old behaviour.
361
-
362
- ## Working now: pull a program's on-chain IDL
363
-
364
- ```bash
365
- npm ci
366
- npm test
367
- node bin/sememe.mjs pull whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc --store /tmp/sememe-demo
368
- # optional: --rpc https://your-mainnet-rpc.example
369
- ```
370
-
371
- The CLI fetches the canonical Program Metadata and legacy Anchor IDL accounts at a finalized RPC slot, checks account owner/program binding and JSON instruction shape, prefers canonical when both exist, and stores normalized JSON by SHA-256 under `<store>/objects/`. A receipt under `<store>/receipts/` records program ID, source account address, slot, fingerprint and instruction count. Repeating the pull of identical content does not rewrite the stored object; corrupt files are refused rather than overwritten. A public Whirlpool pull succeeded on 27.10. at slot 462039125 with 66 named instructions and fingerprint `sha256:0b886e375b112206a846a061f076f7db02a4b01b25cd593cd03caa490d86660f` (own run; no outside use).
372
-
373
- ### Before you pull: ask why an interface is missing (`sememe gap`)
374
-
375
- When a consumer hits `no stored IDL for program X`, running `sememe pull` blindly is the wrong reflex: many programs on Solana publish no IDL at all, so a pull will simply fail repeatedly and leave the consumer guessing whether the RPC failed, the address is wrong, or the program was closed. `sememe gap` probes both on-chain IDL addresses (Program Metadata and legacy `anchor:idl`) and returns the exact diagnosis, with the fix that actually applies:
376
-
377
- ```bash
378
- node bin/sememe.mjs gap DCA265Vj8a9CEuX1eb1LWRnDT7uK6q1xMipnNyatn23M
379
- ```
380
-
381
- ```json
382
- {
383
- "program": "DCA265Vj8a9CEuX1eb1LWRnDT7uK6q1xMipnNyatn23M",
384
- "verdict": "pullable",
385
- "source": "anchor",
386
- "instructions": 12,
387
- "next": "sememe pull DCA265Vj8a9CEuX1eb1LWRnDT7uK6q1xMipnNyatn23M"
388
- }
389
- ```
390
-
391
- Pointed at a program that publishes nothing at either address (for example `L2TExMFKdjpN9kozasaurPirfHy9P8sbXoAN1qA3S95`, the largest single gap in the 156-fixture set, 42 payloads, Lighthouse):
392
-
393
- ```json
394
- {
395
- "program": "L2TExMFKdjpN9kozasaurPirfHy9P8sbXoAN1qA3S95",
396
- "verdict": "no-interface-published",
397
- "next": "this program publishes no on-chain interface at either address — pulling it can never succeed. sememe ships its first-party published copy (Jac0xb/lighthouse, MIT, hash-pinned): use sememe adopt-lighthouse --store <dir>, then sememe decode --union."
398
- }
399
- ```
400
-
401
- The five possible verdicts, all mutually exclusive:
402
- - `pullable`: an on-chain interface exists that your local store does not have yet -> `sememe pull` is the fix.
403
- - `no-interface-published`: neither address answers -> pulling it forever will never help. Use `sememe adopt` with a first-party published copy, or a pinned upstream derivation.
404
- - `present-undecodable`: an account exists at one of the addresses, but its schema cannot be decoded (malformed, or declares another program).
405
- - `vendored`: one of the programs sememe answers via pinned upstream wire tables (System, ComputeBudget, SPL Token, Token-2022, Raydium AMM v4, SPL Associated Token Account) — there is no on-chain IDL to store.
406
- - `unknown`: the RPC failed. **An RPC failure is never recorded as an absence** (the crates.io 403 lesson: a client-side failure that reads as missing data). The CLI exits 3 (`no-verdict`) in this case so scripted callers know the chain's state is unmeasured.
407
-
408
- Measured 2026-11-04 over all 70 gap programs of the foreign 156-fixture set (`research/acquisition-gap-2026-11-04.json`): **34 programs publish an on-chain IDL (177 payloads)** — never-pulled gaps, not derivation work — and **36 publish nothing (129 payloads)**.
409
-
410
- #### Closing a measured gap in one command (`tools/batch-pull.mjs`)
411
-
412
- When the gap report says 34 programs are pullable, pulling them one CLI process each is 34 RPC round-trips at 34 different slots. The batch tool probes them in batched `getMultipleAccounts` calls (25 programs = 50 keys per request; the public endpoint rejects larger batches) and stores each IDL through the SAME write path the single-program CLI uses:
413
-
414
- ```bash
415
- node tools/batch-pull.mjs --gap research/acquisition-gap-2026-11-04.json \
416
- --store my-store \
417
- --out research/batch-pull-2026-11-04.json
418
- ```
419
-
420
- The artifact records, per program: verdict (`stored` / `undecodable` / `unknown`), source account, slot, fingerprint, instruction count, and — honestly — whether the stored IDL carries its own discriminators or needs Anchor's derivation rule (measured: 12 of the 34 carry stored discriminators, 22 need the rule). An RPC failure is `unknown` and exits 3 (`no-verdict`), never an absence; a hand-edited gap report is refused before any probe runs (its rows must sum to its own headline).
421
-
422
- **What a pull does NOT promise:** the pulled IDL describes the program AS PUBLISHED, not as deployed. Measured on the 156-fixture sweep: after pulling all 34, 152 of the 177 payloads decode byte-exactly, but 25 refuse with named reasons — 19 discriminators the deployed program emitted that the published IDL does not list, 6 stale argument layouts — in transactions that SUCCEEDED on chain. Where a first-party published copy exists (the foreign repo ships 13 of them), adopting it next to the pull and decoding through the union closes most of those; see the union section below.
423
-
424
- **Limit:** This is acquisition and content-addressed storage, not an independent audit of the on-chain ABI, offline resolver, freshness guarantee or authority check. Fetch responses are trusted to the configured RPC; first-party repo parity has not been established. `resolve`, `decode-tx`, `diff` and the optional local daemon remain proposals — `lock` no longer does (it ships, see below). (Account and event decoding are NOT proposals: they ship — `decodeStoredAccount` for account state and Anchor `emit_cpi!` event self-CPIs are both on the entry; this sentence said otherwise until 2026-11-05, and the contradiction was measured and pinned by `test/release-surface.test.mjs`.) A later Rust core/TypeScript SDK may replace this Node-based prototype; neither exists yet.
425
-
426
- ## Working now: decode instruction data offline from the stored IDL
113
+ ## Where the numbers come from
427
114
 
428
- ```bash
429
- node bin/sememe.mjs decode whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc \
430
- 2b04ed0b1ac91e62... --store /tmp/sememe-demo # payload: base58 or hex
431
- ```
432
-
433
- **Base58 and hex are the same payload (fixed 2026-11-06).** The CLI has always
434
- accepted "base58 or hex" for a payload, but the base58 path decoded to a byte
435
- array that was not a `Buffer`, and three consumers need `Buffer` methods. So
436
- the SAME bytes behaved differently depending on how they were written:
437
-
438
- - `sememe decode --union <program> <base58>` failed on EVERY base58 payload
439
- with `unionDecode needs the payload bytes as a Buffer` (the union path
440
- compared the re-encoded bytes with the input), while the same payload as
441
- hex decoded fine — so the README's own union walkthrough, which uses a
442
- base58 payload, could not run;
443
- - an `f64`/`f32` argument died with `slice.readDoubleLE is not a function`
444
- (measured on OpenBook v2 `stubOracleCreate`);
445
- - a `bytes` argument came out as a comma-separated byte list (`1,2,3,255`)
446
- instead of base64 (`AQID/w==`) — silent corruption, not an error.
447
-
448
- All three were measured against the PUBLISHED v0.4.4 install (a pinned
449
- install of tag `v0.4.4`), not against a working tree.
450
- v0.4.5 fixes the root cause (`decodeBase58` returns a `Buffer`; a `Buffer` is
451
- still a `Uint8Array`, so every existing caller keeps working) and adds a
452
- second guard at each consumption point, so the shape of the input no longer
453
- decides whether it decodes. `test/base58-payload-bytes.test.mjs` pins the
454
- equality: for every one of those three cases, the base58 form and the hex form
455
- must produce the same instruction, the same fields and the same verdict.
456
-
457
- The install-level half of that measurement is `tools/journey-pinned-install.sh`,
458
- and it takes the tag it installs from the README — the same pin the front page
459
- hands out — instead of naming one itself. That was a measured defect, not a
460
- preference: the script used to install `v0.4.5` by name while the README pinned
461
- a newer tag, so it was measuring an install nobody received (the same class as
462
- every pinned-tag defect above, one level down, in the instrument). It now
463
- refuses to run when the README pins no tag or more than one, and
464
- `test/base58-journey.test.mjs` (G1a) keeps it honest: the script may contain
465
- no install pin of its own, and a README without one must fail the run with
466
- exit 2.
467
-
468
- `src/codec.mjs` reads the byte layout from the stored IDL itself — no generated code and no third-party codec library — and covers primitives, `option`, `vec`, `array`, `defined` struct/enum types, and both unit, tuple and named enum variants. It is deliberately strict: unknown discriminators, unknown variants, invalid `option` tags, invalid bool bytes, out-of-range integers, unknown types and unknown arguments are errors rather than silent coercions, and hostile field names can never pollute prototypes. Decoded values are JSON-safe: 64/128-bit integers become decimal strings, pubkeys become base58, and re-encoding the decoded value must reproduce the input bytes byte-for-byte (`roundTripByteExact`).
469
-
470
- **Bytes the IDL does not describe are reported, not guessed and not dropped.** A payload can carry more bytes than the stored IDL explains; sememe decodes the described prefix and returns the remainder under `trailing: {count, hex}`, with `describedRoundTripByteExact` covering the part the IDL does describe. `--trailing error` (or `{trailing: 'error'}`) restores whole-payload refusal when a caller wants it.
471
-
472
- This default was changed by a measurement, not by preference. Two real mainnet `swap_v2` payloads are 50 bytes where the stored IDL accounts for 43; the previous behaviour refused them outright. Checks against both references said they were valid:
473
-
474
- - Both transactions **succeeded on-chain** (`meta.err = null`, slot 462101502), and the 43-byte form of the same instruction also succeeds (slot 462101506), so the deployed program accepts both lengths. Anchor deserialises instruction args with Borsh `deserialize` over a slice, which reads what the struct needs without requiring the slice to be exhausted.
475
- - **Orca's own `@orca-so/whirlpools-client` decoder accepts both payloads and returns the same argument values** sememe returns.
476
-
477
- So refusing them rejected 2 of 50 real, successful mainnet payloads — a decoder that fails on valid traffic cannot replace the first-party path it is meant to replace. `trailing.count` still makes unexplained bytes impossible to overlook: the CLI prints a note to stderr, `tools/crosscheck.mjs` reports them under `unexplained`, and `trailing.count === 0` remains a usable completeness assertion. All 2 unexplained tails observed so far are zero bytes (padding); a non-zero tail would mean the IDL is missing a real field, which the test suite asserts separately.
478
-
479
- The fidelity claim was tested against two independent references rather than assumed:
480
-
481
- - **Real mainnet bytes:** 50 harvested Whirlpool instruction payloads (8 distinct instructions: `swap`, `swap_v2`, `collect_fees`, `collect_fees_v2`, `decrease_liquidity`, `decrease_liquidity_v2`, `close_position_with_token_extensions`, `update_fees_and_rewards`) decode from the stored IDL with byte-exact re-encoding of the described part, pinned in `test/fixtures/mainnet-whirlpool.json` so the suite keeps re-proving it. Every payload in that fixture was re-verified byte-for-byte against `getTransaction` on mainnet (50/50 matched, 0 mismatches), so the fixture is not a transcription of trusted input.
482
- - **External oracle cross-check:** `tools/crosscheck.mjs` compares sememe against `@orca-so/whirlpools-client` for **all 66 Whirlpool instructions** (oracle-generated bytes → sememe decode → sememe re-encode) plus the same 50 real payloads, checking both byte identity and field-by-field values. Latest run: **66/66 instructions and 50/50 payloads with 0 byte mismatches, 0 value mismatches, 0 errors.** The oracle is used as a test oracle only under the Orca License; nothing from it is vendored into this repository, and it is an optional dependency of the tool.
483
-
484
- **Limits of this evidence:** it covers one Anchor program's instruction data on mainnet traffic. Account layouts were untested here and are covered separately below (three programs, 39 real accounts); `repr(C)`/`packed` account types are not decoded yet, and byte-exact agreement with a reference codec shows layout fidelity — not that anyone outside this machine uses sememe. The 50-payload fixture sample was originally the evidence limit; that limit is now closed at scale (see "Verified against an external oracle at scale" below). Run `npm test` and `SEMEME_ORACLE_PATH=<dir>/node_modules node tools/crosscheck.mjs --store <store>` to reproduce. The mainnet-payload tests no longer depend on a live store: the IDL snapshot in `test/fixtures/whirlpool-idl.json` is re-fingerprinted against its receipt on every run, so they execute on a fresh checkout and in CI instead of skipping.
485
-
486
- ## Compared against my own existing Whirlpool decoder
487
-
488
- `tools/decoder-parity.mjs` runs sememe's stored-IDL codec and `solana-tx-remix`'s own first-party Whirlpool decoder path (`src/args.mjs`) over the same 50 real mainnet payloads and compares them field by field. That decoder is the one this repository is supposed to make replaceable, so it is the honest baseline. `disagreed` counts only fields both paths decoded to different values; fields the first-party path declines to decode are counted separately as `firstPartyRefuses`, because a missing value is not a disagreement.
489
-
490
- Measured on 27.10. (reproduce: `node tools/decoder-parity.mjs --store <store> --tx-remix <dir> --generated <dir>`):
491
-
492
- - **226 of 226 fields both paths decode agree, 0 value disagreements** (50 payloads).
493
- - **0 fields only one side has** — no silent drift in the other direction either.
494
- - **36 field instances the first-party path refuses to decode** (it stops at `option` and defined-type arguments and emits `__optionRemaining`/`__unsupportedDecoder` instead of a value); sememe decodes them, e.g. `remaining_accounts_info: {"__option":"None"}` vs `{"__optionRemaining":"AA=="}`. These are the `option`/`defined`-typed arguments of 17 of 66 Whirlpool instructions (`swap_v2`, `decrease_liquidity_v2`, `open_position`, `lock_position`, …). The count is instances across payloads, not distinct fields.
495
- - The first-party path finds schemas for 65 of 66 instructions (19 have no generated schema file at all).
496
-
497
- So the first comparison is exact where both paths overlap, and sememe additionally decodes what that path declines to decode. That clears this repository's own stated gate for the next step.
498
-
499
- ## One own decoder path is migrated
500
-
501
- `solana-tx-remix` now reads the stored sememe artifact for BOTH halves of its single-program Whirlpool path: the verifier decodes arguments from it, and the emitter produces recipes from it (`solana-tx-remix` commits, 28.10). The earlier status — verifier only, emitter still on generated codecs — is superseded. The rest of the paragraph above still holds: other decoder paths, 1,000 transactions, an outside user — none of that has happened.
502
-
503
- What was actually measured, on that repository's own flagship recipe `out/remix-6754RLWd.ts` (7 Whirlpool instructions, one real mainnet transaction):
504
-
505
- - **The recipe verifies on a fresh clone.** It was `SKIP` in CI for want of an external codec directory; it now passes, all 18 negative controls still failing as declared.
506
- - **The two decoders cross-check each other**: 7 instructions, 24 fields compared, 17 exact agreements, 7 fields the generated path refuses, **0 disagreements**, and 7/7 instruction names agreeing.
507
- - **The bytes now name themselves.** Relabelling `SWAP_V2` as `SWAP` (bytes untouched) was `PASS` for the old verifier and is `FAIL` for the new one — the old one looked the layout up *by* the label, so a false label found a false layout.
508
- - **The artifact is verified, not trusted**: `sha256(canonicalJson(idl) + '\n')` must equal the receipt's fingerprint, re-checked before every decode. Tampering with the stored object is caught and reported as a failure rather than silently falling back to the other decoder — the first version of the verifier did exactly that (found the damage, then printed `PASS`), which was caught by tampering with the store on purpose. That tampering is now performed on purpose by the consumer's own acceptance run, every push: it copies the intact store, edits one argument type so the object still parses as valid JSON, and requires the recipe to fail. Re-disabling the fix makes that check fail loudly, which is how it was shown to be a real control rather than a passing line.
509
-
510
- Two enabling changes were needed on this side, both landed and tested:
511
-
512
- - **The decode core has no npm dependencies.** `src/base58.mjs` replaces the only third-party import in the decode path (`@solana/web3.js`, used solely to base58 32 raw bytes), and `src/canonical.mjs` holds `canonicalJson` so a consumer can re-check a fingerprint without pulling in the acquisition path. A consumer repository with no `package.json` and no `node_modules` can import the decoder directly — which is what makes the migration above possible without vendoring a copy that could drift. `test/deps.test.mjs` keeps that property: the decode core must import nothing outside `node:` builtins and its own files, and everything else in `src/` must stay within an explicit allowlist. Parity with `@solana/web3.js` is checked on 202 vectors.
513
- - **`canonicalJson` moved out of the pull path** and is still re-exported from `src/pull.mjs`, so nothing that imports it from there changed.
514
-
515
- Honest limits: `solana-tx-remix` is my own repository, so this is one consumer I wrote, not an outside user. It is one program (Whirlpool) and one recipe (7 instructions, one transaction), not 1,000 transactions. Nothing here shows ecosystem coverage or demand.
516
-
517
- ## Both halves of that consumer now read the artifact
518
-
519
- The first migration covered the VERIFIER: it re-derives Whirlpool arguments from the pinned IDL. The EMITTER — the code that produces a recipe in the first place — was still reading Codama-generated codec modules that are not in that repository, so a fresh clone exited with "Whirlpool codec directory not found" before it could name a single argument. It now reads the same artifact (`src/sememe-idl-source.mjs` in `solana-tx-remix`), which changed three things, all measured:
520
-
521
- - **The tool can produce a recipe on a fresh clone.** The flagship `out/remix-6754RLWd.ts` was re-emitted from the stored IDL. What the verifier reads — `ACCOUNTS`, `LOOKUP_TABLE_ADDRESSES`, `INSTRUCTION_BYTES`, `INNER_BYTES` — is byte-identical to the previous recipe, and the `.json` is unchanged: the transaction did not move, only the description of it got more accurate.
522
- - **The stored IDL names more instructions.** It carries all 57 discriminators the generated directory carries, with identical names, and 9 it does not have at all (`close_position_with_token_extensions`, `initialize_position_bundle_with_metadata`, `set_reward_authority_by_super_authority`, …). Those instructions could not be named even with the codecs on disk.
523
- - **The argument names are the program's own.** `IX6_SWAP_V2_SQRT_PRICE_LIMIT` rather than `IX6_SWAP_V2_SQRTPRICELIMIT`, and `remaining_accounts_info` reads `{"__option":"None"}` rather than the raw `{"__optionRemaining":"AA=="}` the generated path declined to interpret.
524
-
525
- The `absent`/`broken` distinction is kept on this side too: an absent store falls back to the generated codecs (or exits naming both sources when neither exists), while a store that is present and does not verify is a hard stop — a damaged artifact must not silently become another source's names.
526
-
527
- ## Verified against an external oracle at scale (1,000-transaction scale)
528
-
529
- The earlier evidence compared 50 harvested payloads. That was a small sample, and "50 payloads, not 1,000 transactions" was the explicitly stated limit of the qualification.
530
-
531
- Tools added on 2026-10-28:
532
- - `tools/harvest.mjs`: reproducible bulk harvest over real mainnet blocks. It queries `getSignaturesForAddress` to identify slots that carry the program, fetches full finalized blocks via `getBlock`, and extracts every instruction of the target program (top-level and CPI). Instructions in the JSON encoding resolve through `programIdIndex` against the transaction's static and loaded address tables, and inner instructions are parsed from `meta.innerInstructions` (two real extraction bugs caught and covered by `test/payloads.test.mjs`). The harvest pins the transactions it actually read, so any caller can re-derive the payloads with `--from-signatures`.
533
- - `tools/tx-diff.mjs`: cross-decoder diff across all harvested transactions against `@orca-so/whirlpools-client` (Orca License, used only as a test oracle, nothing vendored).
534
-
535
- For every payload, the diff answers three separate questions:
536
- 1. **Naming:** does the oracle's encoder reproduce the bytes sememe names the instruction with? (The oracle's per-instruction decoders do not validate discriminators, so the honest naming oracle is the encoder round-trip: the one instruction whose encoder reproduces the bytes is the true name.)
537
- 2. **Values:** field-by-field equality across every argument the IDL declares, with known representation differences (camelCase, hoisted structs, unit enums) handled explicitly.
538
- 3. **Bytes:** re-encoding the decoded values reproduces the IDL-described prefix byte-for-byte. Trailing bytes the IDL does not describe are tracked separately under `unexplained`.
539
-
540
- **Measured on real mainnet traffic (run 2026-10-28):**
541
- - **40,013 transactions** scanned across **148 finalized blocks** (slots 462358216..462358364), with **3,309 pinned signatures** collected by paging `getSignaturesForAddress` backwards until enough slots were covered.
542
- - **1,029 transactions** contain Whirlpool payloads, yielding **1,119 instruction payloads** (the vast majority are CPIs inside router and aggregator calls — Whirlpool top-level invocations are rare on mainnet).
543
- - **1,119/1,119 payloads decoded** by sememe (`sememeErrors: 0`).
544
- - **1,119/1,119 naming agreements** (`namingDisagrees: 0`).
545
- - **1,119/1,119 value agreements** across every decoded argument (`valueDisagrees: 0`).
546
- - **1,119/1,119 byte-exact round-trips** of the described prefix (`byteMismatch: 0`).
547
- - **33 unexplained trailing-byte payloads** (2.95%), all `swap_v2`. The earlier 50-payload sample measured 4.0% (2 of 50), so this phenomenon is confirmed present at over 1,000-payload scale while varying somewhat between samples.
548
- - Instructions observed at scale: `swap_v2` (910), `swap` (162), `increase_liquidity_v2` (15), `reposition_liquidity_v2` (12), and single instances of `open_position`, `open_position_with_token_extensions`, `close_position_with_token_extensions`, `collect_fees_v2`, `decrease_liquidity_v2`, `update_fees_and_rewards`.
549
-
550
- **Falsification:** the diff tool is proven to fail when defects are introduced:
551
- 1. Flipping a byte in instruction data: `sememeErrors: 1`, exit 1.
552
- 2. Identical input to both decoders: exit 0 (no false positives).
553
- 3. IDL tamper (`swap_v2` `amount_specified_is_input` bool → u8): `valueDisagrees: 34`, exit 1 (`sememe: "1"` vs `oracle: "true"`), while `byteMismatch: 0` remains exact because both types are 1 byte.
554
- 4. Swapping `swap` and `swap_v2` discriminators in the IDL: `namingDisagrees: 34`, `sememeErrors: 10`, exit 1.
555
-
556
- **Reproducibility:** the committed fixture pins its own payload set. `tools/lib/pinned.mjs` resolves the signature list from the fixture's transaction records first (the payload set itself), falls back to the declared `signatures` array, and throws when a document pins nothing rather than resolving to an empty list. This distinction is measured, not assumed: only **298 of the 1,029** payload-bearing transactions appear in the `getSignaturesForAddress` page, because a transaction that reaches Whirlpool only through a CPI does not list Whirlpool in its own account keys. Pinning the page instead of the payloads would therefore re-derive a different, smaller sample while still reporting success. Two defects on this path were found and fixed on 2026-10-28 (the fixture carried no `signatures` array at all, so the documented command harvested zero and crashed; and the harvester pinned the page), covered by `test/pinned.test.mjs`.
557
-
558
- Because the public RPC rate-limits, a re-derivation can also come back **partial** — 37 of 40 transactions fetched — and still write a file and exit 0. `--verify <fixture>` closes that gap: it compares the re-derivation against the fixture and exits non-zero unless every pinned transaction returned byte-identical payloads (`tools/lib/rederive-verify.mjs`, 5 tests). Measured against live mainnet: a 15-transaction subset re-derived completely (15/15, 17 payloads, byte-identical, exit 0); the same command on a tampered payload exited 1 with `payload 0 dataHex differs`.
559
-
560
- **Remaining limits:** one Anchor program (Whirlpool); events and (at that point) accounts were still untested; other own decoder paths remain to be migrated; no outside user has adopted sememe yet. But the "50 payloads, not 1,000 transactions" limit is closed: agreement holds across 1,119 real payloads from 1,029 transactions sampled from 40,013 mainnet transactions, with zero value or byte discrepancies.
561
-
562
- ## Second real program: Jupiter AMM v6 (and two defects it exposed)
563
-
564
- Testing against only one program hid a whole class of defects: Whirlpool's enums are simple and its transactions rarely use Anchor event CPIs. On 2026-10-29, sememe pulled Jupiter AMM v6 (`JUP6LkbZbjS1jKKwapdHNy74zcZ3tLUZoi5QNyVTaV4`) from mainnet (slot 462867554, 19 instructions, 6 events, fingerprint `sha256:37fa69c115f5883314374e01c4aa8f242778f7aebccd75f5064d6afb0277afee`), harvested **18,193 transactions** across **48 finalized blocks** (slots 462867935..462867982), and decoded every payload referencing the program (313 transactions, **773 payloads**).
565
-
566
- Initial run against the existing codec: **only 35 of 773 payloads decoded**. Two real defects were exposed and fixed:
567
-
568
- 1. **Enum minSize lower bound was inverted:** `minSize` used the *largest* enum variant as the lower bound for an element in a vector. Jupiter's `Swap` enum has 204 variants; variant 47 (`WhirlpoolSwapV2`) needs 85 bytes, so the hostile-length pre-check required at least 85 bytes per element for every `route` instruction. Real payloads carry smaller variants (e.g. 1 byte for unit variants), so **342 payloads were rejected** with `"vector length 1 needs 85 byte(s) but 25 remain"` — even though the payloads themselves decode byte-exact. Fixed: `minSize` takes the *smallest* variant as the lower bound, and options bound at 1 byte (the `None` tag).
569
- 2. **Anchor `emit_cpi!` event payloads arrive like instructions:** Anchor programs emit events through a self-CPI with instruction data `EVENT_IX_TAG_LE (8) ++ event_discriminator (8) ++ borsh(fields)`. In a transaction (top-level or CPI) these look identical to instructions, but the discriminator table did not know `EVENT_IX_TAG` (`0x1d9acb512ea545e4`, hex `e445a52e51cb9a1d`), so **396 payloads were rejected** with `"unknown instruction discriminator e445a52e51cb9a1d"`. Added `decodeEventData`, `encodeEventData`, `eventIndex`, and transparent dispatch in `decodeData`/`decodePayload`: instructions match first, event self-CPIs match second.
570
-
571
- **Measured after the fix (`tools/coverage.mjs`):**
572
- - **773/773 payloads decoded** (377 instructions + 396 Anchor event self-CPIs).
573
- - **773/773 round-trip byte-exact** (0 codec errors, 0 unknown discriminators, 0 trailing bytes).
574
- - Instructions observed: `route_v2` (153), `route` (102), `shared_accounts_route_v2` (39), `set_token_ledger` (30), `route_with_token_ledger` (26), `shared_accounts_route` (18), `create_idempotent_associated_token_account` (5), `shared_accounts_route_with_token_ledger` (4).
575
- - Events observed: `SwapEvent` (205), `SwapsEvent` (188), `FeeEvent` (3).
576
- - **Naming cross-check:** compared against the independent bitquery IDL in `solana-tx-remix/idl/multi/jupiter-amm/amm.json`: **768/768 shared discriminators agree on the name** with 0 name disagreements; the remaining 5 payloads are `create_idempotent_associated_token_account`, which is on-chain but absent from the older bitquery copy.
577
-
578
- Covered by `test/jupiter-v6.test.mjs` (9 tests) using real pinned mainnet fixtures. Whirlpool qualification re-run: 1,119/1,119 byte-exact, 33 trailing, 0 regressions. Runs: `npm test` (58 tests pass).
579
-
580
- ## Third real program, family-distant: Kamino Lend (and a gap in the coverage measure)
581
-
582
- Whirlpool and Jupiter AMM v6 are both swap venues from the same family. On 2026-10-29 sememe pulled Kamino Lend (`KLend2g3cP87fffoy8q1mQqGKjrxjC8boSyAYavgmjD`, a lending market) from mainnet (slot 462900393, legacy `anchor:idl` account, fingerprint `sha256:8a9a5d124ac3a2b50ef096f1fff56557367ee9cb8d08a17bde8a61dd373da4b5`), harvested **142,981 transactions** across **324 finalized blocks** (slots 462905866..462909439) and extracted **1,432 payloads** from 400 payload-bearing transactions.
583
-
584
- **The finding: this program's on-chain IDL stores no discriminators at all.** 66 instructions, 5 events, 0 discriminators. A decoder indexes by discriminator, so the index built from the stored IDL is empty and **0 of 1,432 real payloads decoded** — every one refused with "unknown instruction discriminator".
585
-
586
- That also exposes a gap in the 20-program coverage measure, which classified a program as *usable* whenever its IDL named at least one instruction. Kamino Lend was counted usable (it was one of the eleven legacy-Anchor pulls) although its schema could not decode a single byte. "Names instructions" is not "can decode".
587
-
588
- **Fix:** `src/discriminators.mjs` fills the gaps with Anchor's own rule — `sha256("global:" + snake_case(name))[0..8]` for instructions, `sha256("event:" + name)[0..8]` for events. Stored discriminators are never overwritten, and a stored value that disagrees with the derivation is reported as a conflict rather than silently replaced.
589
-
590
- **The rule is verified, not assumed:** against IDLs that *do* store their discriminators, the derivation reproduces **Whirlpool 72/72** and **Jupiter AMM v6 24/25**. The single exception is instructive: Jupiter's `create_idempotent_associated_token_account` stores `5368c096…`, while the name-derived value is `bdbcecfa…` — and 5 real mainnet payloads carry `5368c096…` and round-trip byte-exact, so the stored value is the true one. A name is not proof of a wire discriminator, which is exactly why the derivation only fills gaps.
591
-
592
- **Measured after the fix (`tools/coverage.mjs`):**
593
- - **1,432/1,432 payloads decoded and round-trip byte-exact** (0 codec errors, 0 unknown discriminators, 0 trailing bytes).
594
- - **Naming cross-check** against Kamino's own published IDL (`@kamino-finance/klend-sdk`): **1,432/1,432 named by both sources, 0 disagreements**.
595
- - Instructions observed: `refreshReserve` (520), `refreshReservesBatch` (290), `refreshObligation` (268), `depositReserveLiquidityAndObligationCollateralV2` (54), `depositReserveLiquidity` (51), `borrowObligationLiquidityV2` (47), `withdrawObligationCollateralAndRedeemReserveCollateralV2` (45), `repayObligationLiquidityV2` (43), `flashBorrowReserveLiquidity` (36), `flashRepayReserveLiquidity` (36), `redeemReserveCollateral` (33), `initObligation` (5), `initObligationFarmsForReserve` (3), `initUserMetadata` (1).
596
-
597
- Covered by `test/klend-derived.test.mjs` (6 tests) using real pinned mainnet fixtures; `test/discriminators.test.mjs` (7 tests) pins the rule and the Jupiter exception. `decodePayload` now reports `discriminatorSource` (`stored` / `derived` / `stored+derived`) and any derivation conflicts, so a consumer can tell a name-derived match from a stored one. Whirlpool and Jupiter re-runs: unchanged, 0 regressions.
598
-
599
- **A second, unrelated defect fixed along the way:** `tools/harvest.mjs` computed the slot range with `Math.min(...slots)`. A 142,981-transaction harvest collects far more slots than the call stack accepts, so the run died with `RangeError: Maximum call stack size exceeded` *after* all RPC work and before the file was written. The range is now reduced in a loop (`tools/lib/slot-range.mjs`).
600
-
601
- **Remaining limits at that point:** three programs, still no outside user; accounts were still undecoded (now closed below); the derivation is Anchor-specific (a non-Anchor program needs its own scheme); and the coverage measure's "usable" definition should be tightened to require a decodable schema, not just named instructions.
602
-
603
- ## Fourth program: OpenBook v2, checked against anchor-ts's own sighash
604
-
605
- Kamino Lend left one weak spot: the derivation rule was verified against programs that *store* discriminators (Whirlpool, Jupiter) and then *applied* to a program that stores none (Kamino), whose own published SDK happens to hardcode them. A reader could still say: you checked your rule against yourself.
606
-
607
- OpenBook v2 (`opnb2LAfJYbRMAHHvqjCwQxanZn7ReEHp1k81EohpZb`, a central limit order book — a third program family after the swap AMMs and the lending market) closes that. Its on-chain IDL was pulled on 2026-10-29 at slot 462921401 (legacy `anchor:idl` account, fingerprint `sha256:d12a2e16314b418a173cc6da755746396e2b15caf9e28ccb4913aa335050913f`): **29 instructions, 8 events, 0 discriminators** — the same premise as Kamino Lend.
608
-
609
- **The oracle is an independent implementation of Anchor's sighash:** `@coral-xyz/anchor` (anchor-ts 0.29.0) computes `sha256("global:" + snake_case(name))[0..8]` at runtime from the same IDL, in code this repository shares nothing with. Fixture `test/fixtures/openbook-anchor-ts-discriminators.json` records its 29 instruction + 8 event discriminators; the derived values match **29/29 and 8/8, byte for byte** (`test/openbook-v2.test.mjs`). Events were also reproduced by the event rule, 8/8.
610
-
611
- **Measured on real traffic (`tools/coverage.mjs`):** a harvest of **224,894 transactions** across 900+ finalized blocks (slots 462476178..462929711) yielded **600 payload-bearing transactions with 600 payloads**:
612
-
613
- - **600/600 decoded and round-trip byte-exact**, 0 codec errors, 0 unknown discriminators, 0 trailing bytes.
614
- - **Without derivation: 0/600** — the discriminator index built from the published bytes alone is empty, exactly as for Kamino Lend.
615
- - **Naming cross-check** against the independently published IDL (`openbook-dex/openbook-v2`, `idl/openbook_v2.json`, fetched from GitHub): **600/600 named by both, 0 disagreements**; instruction sets identical (29/29 both sides).
616
- - Instructions observed: `placeTakeOrder` (562), `cancelOrder` (10), `consumeEvents` (7), `placeOrders` (7), `settleFunds` (4), `consumeGivenEvents` (3), `cancelAllOrders` (3), `deposit` (2), `cancelAllAndPlaceOrders` (2).
617
-
618
- Covered by `test/openbook-v2.test.mjs` (6 tests) and pinned fixtures (`openbook-v2-idl.json`, `openbook-anchor-ts-discriminators.json`, `openbook-v2-published-idl.json`, `harvest-openbook-v2-2026-10-29.json`). Suite: **82/82**. Disabling the fill in `src/discriminators.mjs` fails the new tests (falsification), restoring it passes.
619
-
620
- **Remaining limits after four programs:** still no outside user; both derivation checks (Kamino SDK, anchor-ts) are Anchor-specific — a non-Anchor program needs its own scheme; and the sample remains family-clustered. OpenBook's on-chain and published IDLs are also both legacy shapes, so the naming cross-check shares the Anchor rule on both sides; the byte-exact round-trip against 600 real payloads is what cannot be satisfied by a wrong discriminator.
621
-
622
- ## Fifth program: Marinade liquid staking (a staking program, and a published SDK that stores nothing either)
115
+ Every claim above is pinned by a test and backed by a committed artifact. If you want to know **why** a number is what it is, and what was measured wrong before it was measured right:
623
116
 
624
- Marinade (`MarBmsSgKXdrN1egZf5sqe1TMai9K1rChYNDJgjq7aD`, liquid staking — a staking program, not a swap AMM, lending market or order book) is one of the five programs in the strict coverage table that can only decode because the tool reconstructs the missing discriminators. Its on-chain IDL was pulled on 2026-10-29 at slot 462985426 (legacy `anchor:idl` account, fingerprint `sha256:61ccc0c1a5995dd4aaaa0ec0108e88343642f8da4f74603c67472c8ae42ea14e`): **29 instructions, 26 events, 0 discriminators** — the same premise as Kamino Lend and OpenBook v2, now on a fourth program family.
117
+ - [docs/measured-history.md](https://github.com/auroraxo/sememe/blob/main/docs/measured-history.md) — the unabridged record: per-program verification (9+ programs, real mainnet payloads), the ecosystem coverage measurement and its corrections, acquisition-gap closings (ATA, Lighthouse, Manifest, Archer, DiabLokxis, pfeeU, pump.fun, HADRo), the dialect gate and raw-LE engine, short-form and struct-absent rules with chain witnesses, and the registry-channel measurements. Sections are named by date.
118
+ - `research/` — the JSON artifacts the README used to quote inline; the history file links them by path.
119
+ - `npm test` — 970 tests: the quickstart output line, the documented commands against the pinned tag, the install channels, the journey from a bare `node_modules` install, and the measurement artifacts themselves are all pinned. A hand-edited number in this README goes red.
625
120
 
626
- **Two independent confirmations, one per open question:**
627
-
628
- - **Is the rule right?** anchor-ts (0.29.0) computes Anchor's sighash at runtime from the same IDL in code this repo shares nothing with. Fixture `test/fixtures/marinade-anchor-ts-discriminators.json` records its 29 instruction + 26 event discriminators; the derived values match **29/29 and 26/26, byte for byte** — this time including the events (26 of them, four times the OpenBook event set).
629
- - **Is the interface the one the program ships?** the naming cross-check uses `marinade_finance.json` from `marinade-finance/marinade-ts-sdk`, an artifact fetched from GitHub that this machine's store never sees. **29/29 instruction names identical, 0 only-on-one-side**, and the coverage run reports **623/623 payloads named by both sources, 0 disagreements**.
630
-
631
- **Measured on real traffic:** a harvest of **236,290 mainnet transactions** across 560 finalized blocks (slots 462439010..462984538) yielded **600 payload-bearing transactions with 623 payloads**:
632
-
633
- - **623/623 decoded and round-trip byte-exact**, 0 codec errors, 0 unknown discriminators, 0 trailing bytes.
634
- - **Without derivation: 0/623** — the discriminator index built from the published bytes alone is empty, exactly as for Kamino Lend and OpenBook v2.
635
- - Instructions observed (14 distinct): `liquidUnstake` (164), `deposit` (150), `mergeStakes` (136), `updateActive` (125), `setValidatorScore` (24), `updateDeactivated` (6), `stakeReserve` (5), `partialUnstake` (5), `claim` (3), `configLp` (1), `withdrawStakeAccount` (1), `removeLiquidity` (1), `removeValidator` (1), `addValidator` (1).
636
-
637
- Covered by `test/marinade.test.mjs` (6 tests) and pinned fixtures (`marinade-liquid-staking-idl.json`, `marinade-anchor-ts-discriminators.json`, `marinade-finance-published-idl.json`, `harvest-marinade-liquid-staking-2026-10-29.json`); measurement in [research/coverage-marinade-liquid-staking-2026-10-29.json](research/coverage-marinade-liquid-staking-2026-10-29.json). Suite: **114/114**. Falsification: flipping one byte of the derived discriminator fails 2 of the 6 new tests (the byte-exact round-trip and the anchor-ts comparison), restoring it passes.
638
-
639
- **Remaining limits after five programs:** still no outside user; no Marinade *account* decoding yet (the account branch covers Whirlpool/OpenBook/Kamino); the derivation is still Anchor-specific; and this harvest recorded **0 event payloads** — Marinade's 26 events were verified as discriminators against anchor-ts but none appeared in the sampled traffic, so the event path on this program is confirmed by the oracle, not by real bytes.
640
-
641
- ## Sixth program: Meteora Vault (a yield vault, and a measured interface drift)
642
-
643
- Meteora Vault (`24Uqj9JCLxUeoC3hGfh5W3s9FM9uCHDS2SG3LYwBpyTi` — strategy rotation over token vaults, a fifth program family: not a swap AMM, lending market, order book or staking program) is one of the five programs in the strict coverage table that can only decode because the tool reconstructs the missing discriminators. Its on-chain IDL was pulled on 2026-10-29 at slot 463051650 (legacy `anchor:idl` account, fingerprint `sha256:945b541c0a3f2d81d049016ec21c8afd49329de78d76453c99dd42dbc41a5e94`): **14 instructions, 8 events, 0 discriminators** — same premise, fifth family.
644
-
645
- **The interface drift finding (new):** the independent published copy (`idl.ts` shipped in MeteoraAg/vault-sdk) is version **0.9.5**, while the on-chain account is **0.7.2**. The inventories are not identical: the on-chain copy carries `collectDust` and `withdraw2`, which the published copy dropped; the published copy carries 7 instructions the on-chain account never heard of (`claimRewards`, `getUnlockedAmount`, `initializeIdleVault`, `migrateVault`, `transferAdmin`, `transferFeeVault`, `updateLockedProfitDegradation`). Every on-chain **event** name exists in the published copy. This is pinned as a measured fact in `test/meteora-vault.test.mjs` so a future pull that changes the drift fails loudly — "same interface" is not asserted where it is not true.
646
-
647
- **Two independent confirmations:**
648
-
649
- - **Is the rule right?** anchor-ts (0.29.0) computes Anchor's sighash at runtime from the same IDL in code this repo shares nothing with. Fixture `test/fixtures/meteora-vault-anchor-ts-discriminators.json` records its 14 instruction + 8 event discriminators; the derived values match **14/14 and 8/8, byte for byte**.
650
- - **Does real traffic only use names both copies know?** **1,290/1,290 payloads named by both sources, 0 disagreements**, and the two instructions actually observed (`withdraw`, `deposit`) exist in both interface copies — the drift above is inventory drift, not a name under shipped bytes.
651
-
652
- **Measured on real traffic:** a harvest of **325,087 mainnet transactions** across 428 finalized blocks (slots 463049519..463051719) yielded **603 payload-bearing transactions with 1,290 payloads**:
653
-
654
- - **1,290/1,290 decoded and round-trip byte-exact**, 0 codec errors, 0 unknown discriminators, 0 trailing bytes.
655
- - **Without derivation: 0/1,290** — the discriminator index built from the published bytes alone is empty.
656
- - Instructions observed: `withdraw` (654), `deposit` (636). Honest limit: vault traffic is two high-volume flows, not a tour of all 14 instructions; **0 event payloads**, so the event path on this program is confirmed by the oracle (8/8 against anchor-ts), not by real bytes — same limit as Marinade.
657
-
658
- Covered by `test/meteora-vault.test.mjs` (7 tests) and pinned fixtures (`meteora-vault-idl.json`, `meteora-vault-anchor-ts-discriminators.json`, `meteora-vault-published-idl.json`, `harvest-meteora-vault-2026-10-29.json`); measurement in [research/coverage-meteora-vault-2026-10-29.json](research/coverage-meteora-vault-2026-10-29.json). Falsification: disabling the derivation fill fails 2 of the 7 new tests (byte-exact round-trip and the anchor-ts comparison), restoring it passes.
659
-
660
- **Remaining limits after six programs:** still no outside user; the derivation is still Anchor-specific; **one of the five derivation-only programs remains individually unproven against real payloads (Meteora DAMM v1)** — Kamino Lend, OpenBook v2, Marinade and Meteora Vault now each are; no event bytes observed on this program.
661
-
662
- ## Seventh program: Meteora DAMM v1 (last derivation-only program, and a case-style finding)
663
-
664
- Meteora DAMM v1 (`Eo7WjKq67rjJQSZxS6z3YkapzY3eMj6Xy8X5EQVn5UaB` — Meteora's constant-product/stable AMM, a sixth program family) is the fifth and last program in the strict coverage table that can only decode because the tool reconstructs the missing discriminators. Its on-chain IDL was pulled on 2026-10-29 at slot 463068495 (legacy `anchor:idl` account, fingerprint `sha256:dd60d3d367986a1d2ad574df3ea5855c5a9801fdaa6fdea57b898deafb95fdba`): **26 instructions, 18 events, 0 discriminators** — same premise as Kamino Lend, OpenBook v2, Marinade and Meteora Vault.
665
-
666
- **Two new findings:**
667
-
668
- - **Case-style difference between interface copies is not a disagreement.** The on-chain copy names instructions in camelCase (`claimFee`, `removeBalanceLiquidity`), the published `MeteoraAg/damm-v1-sdk` copy in snake_case (`claim_fee`, `remove_balance_liquidity`). Anchor's rule hashes `snake_case(name)`, so the two copies produce identical wire bytes while differing in spelling convention. `tools/coverage.mjs` now counts these apart: `caseOnlyDifference` (cosmetic) versus `nameDisagreement` (survives snake_casing, therefore a real conflict). Before this fix the DAMM run reported 13 false "name disagreements" that were 13× `claimFee`/`claim_fee`. Measured now: **643/643 named by both copies, 0 real disagreements, 13 case-only differences.**
669
- - **The published copy stores its own discriminators, and the rule reproduces them.** Unlike the on-chain account, the published `idl.json` carries discriminator arrays written by the program's own IDL generator: **30/30 instructions and 19/19 events match the Anchor rule byte for byte**, computed from names this repository never touched. That is confirmation of the rule from a second source that *stores* rather than *derives*.
670
-
671
- **Measured on real traffic:** a harvest of **152,097 mainnet transactions** across finalized blocks (slots 463065616..463068621) yielded **602 payload-bearing transactions with 643 payloads**:
672
-
673
- - **643/643 decoded and round-trip byte-exact**, 0 codec errors, 0 unknown discriminators, 0 trailing bytes.
674
- - **Without derivation: 0/643** — the discriminator index built from the published bytes alone is empty.
675
- - **Two independent oracles agree:** anchor-ts (0.29.0) computes the sighash at runtime — **26/26 instructions + 18/18 events byte-exact**; the published sdk's own stored discriminators — **30/30 + 19/19 byte-exact**.
676
- - Instructions observed: `swap` (630), `claimFee` (13). Honest limit: two flows in the traffic, not a tour of all 26 instructions; **0 event payloads**, so the event path on this program is confirmed by both oracles (18/18 each), not by real bytes — same limit as Marinade and Meteora Vault.
677
- - **Inventory drift pinned:** the published copy (0.5.7) carries 4 instructions the on-chain 0.5.2 account does not (`claim_protocol_fee2`, `close_operator_account`, `create_operator_account`, `zap_protocol_fee`); every on-chain instruction exists in the published copy after snake_casing.
678
-
679
- Covered by `test/meteora-damm-v1.test.mjs` (7 tests) and pinned fixtures (`meteora-damm-v1-idl.json`, `meteora-damm-v1-anchor-ts-discriminators.json`, `meteora-damm-v1-published-idl.json`, `harvest-meteora-damm-v1-2026-10-29.json`); measurement in [research/coverage-meteora-damm-v1-2026-10-29.json](research/coverage-meteora-damm-v1-2026-10-29.json). Suite: **128/128**. Falsification: disabling the derivation fill fails 2 of the 7 new tests (byte-exact round-trip and the anchor-ts comparison), restoring it passes.
680
-
681
- **Remaining limits after seven programs:** all five derivation-only programs are now individually proven against real mainnet payloads (**Kamino Lend 1,432/1,432, OpenBook v2 600/600, Marinade 623/623, Meteora Vault 1,290/1,290, Meteora DAMM v1 643/643**, each byte-exact with 0 decodable without derivation); still no outside user; the derivation is still Anchor-specific; no event bytes observed on the last three programs.
682
-
683
- ## Accounts: state, not just calls (three programs, 39 real accounts)
684
-
685
- Every proof above ends instructions/events and says the same remaining limit: *accounts are still undecoded*. That limit is now closed for the same three programs.
686
-
687
- `src/account.mjs` decodes an account's bytes against the same stored IDL:
688
-
689
- ```bash
690
- node bin/sememe.mjs account whirLbMiicVdio4qvUfM5KAg6Ct8VwpYzGff3uctyCc \
691
- '<base64 from getMultipleAccounts>' --store /tmp/sememe-demo
692
- ```
693
-
694
- The 8-byte Anchor account discriminator picks the account type (no `--name` needed), the struct is read from the IDL — inline in `accounts` for legacy IDLs, via `types` for newer ones — and the decoded value must re-encode to the account's own bytes.
695
-
696
- **The account discriminator is derived the same way instructions are, and it is verified by an independent implementation.** Fixture `test/fixtures/account-discriminators-anchor-ts.json` records `BorshAccountsCoder.accountDiscriminator` computed by anchor-ts (0.29.0) for **all 29 account names across 4 IDLs**; sememe's `sha256("account:" + name)[0..8]` matches **29/29, byte for byte**. Whirlpool stores its account discriminators (12/12, so stored = derived, 0 conflicts); OpenBook v2 and Kamino Lend store none, so all 6 + 10 are derived — without derivation **0 of their accounts identify at all** (falsification test in `test/account.test.mjs`).
697
-
698
- **Measured on real mainnet accounts** (`tools/account-harvest.mjs` → owner-filtered `getMultipleAccounts`, pinned as fixtures; `tools/accounts.mjs` decodes them from a live store):
699
-
700
- | program | accounts | identified | decoded | re-encoded byte-exact | trailing bytes |
701
- |---|---|---|---|---|---|
702
- | Whirlpool | 27 | 27 | 27 | 27 | 0 |
703
- | OpenBook v2 | 4 | 4 | 4 | 4 | 0 |
704
- | Kamino Lend | 8 | 8 | 8 | 8 | 0 |
705
- | **total** | **39** | **39** | **39** | **39** | **0** |
706
-
707
- Types covered include `Whirlpool`, `Oracle`, `TickArray`, `DynamicTickArray` (Whirlpool), `Market`, `OpenOrdersAccount`, `BookSide`, `EventHeap` (OpenBook v2, including its 90 KB book-side accounts), and `Obligation`, `Reserve`, `LendingMarket`, `UserMetadata` (Kamino Lend).
708
-
709
- **Values are checked against anchor-ts, not just round-tripped.** Round-tripping through sememe's own encoder would prove only that decode and encode agree with each other. Instead, anchor-ts decoded every pinned account and the hash of its normalised value is pinned in `test/fixtures/account-values-golden-2026-10-29.json`; the suite recomputes the hash from sememe's own decode. **39/39 agree**, and the live cross-check with the independently published OpenBook IDL (`tools/accounts.mjs --cross-check`) reports 4/4 compared with 0 disagreements.
710
-
711
- **What the byte-exactness gate actually proves is itself falsified:** dropping the struct's first field, or flipping a discriminator byte, must not produce "decoded + trailing 0 + byte-exact" — both are asserted to fail (`test/account.test.mjs`), and foreign program bytes must identify as nothing.
712
-
713
- Run: `npm test` (the account commit brought the suite to 108; **114** after the fifth program, **128** after the seventh), plus `node tools/accounts.mjs --accounts test/fixtures/accounts-<program>-2026-10-29.json --store <store> --program <id>` to re-measure from a live pull.
714
-
715
- **Remaining limits:** still no outside user; **zero trailing bytes were observed**, so the report path for account bytes the IDL does not describe is implemented but unexercised by real data; `repr(C)`/`packed` (zero-copy) accounts are not decoded; account *discriminators* are derived for three Anchor programs — a non-Anchor program's account framing still needs its own scheme, exactly as for instructions; and 39 accounts from traffic of three programs is a convenience sample, not an inventory of what accounts exist on chain.
716
-
717
- ## Eighth and ninth programs: Meteora DAMM v2 and pump.fun (and a tuple-struct codec fix)
718
-
719
- On 2026-10-30 `solana-tx-remix` carried three CPI labels that read `(unknown)` because
720
- the consumer assumed the programs published no on-chain IDL: **Meteora DAMM v2**
721
- (`cpamdpZCGKUy5JxQXB4dcpGPiikHawvSWAd6mEn1sGG`, `cp-amm`) and **pump.fun**
722
- (`6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P`). One sememe pull per program
723
- falsified the assumption:
724
-
725
- | program | on-chain source | slot | instructions | events | stored discriminators |
726
- |---|---|---|---|---|---|
727
- | cp-amm | `anchor:idl` @ `F1hSvut...` | 463372654 | 38 | 23 | 38/38 ix, 23/23 ev (100%) |
728
- | pump.fun | `anchor:idl` @ `AYgC53t...` | 463372656 | 47 | 28 | 47/47 ix, 28/28 ev (100%) |
729
-
730
- Every stored value matches Anchor's derivation rule byte for byte (38/38, 47/47, 23/23,
731
- 28/28), so no gaps had to be filled. Both programs are from the frozen 20-ID set
732
- (indexes 16 and 11) where they were previously classified "usable"; this measurement
733
- proves they actually decode real mainnet bytes.
734
-
735
- **Measured on real traffic:** two fresh mainnet harvests (2026-10-30, slots
736
- 463380807..463380978) collected **2,890 payloads across 1,661 payload-bearing
737
- transactions** (cp-amm: 1,894 payloads / 809 tx from 22,272 transactions; pump.fun:
738
- 996 payloads / 852 tx from 4,459 transactions):
739
-
740
- - **cp-amm decodes every payload:** **1,894/1,894 decoded (100%)**, 0 codec errors, 0
741
- unknown discriminators; **1,888/1,894 round-trip byte-exact**, 6 payloads with
742
- trailing bytes (all `swap`, 2 with 8 zero bytes, 4 with 1 zero byte) — the deployed
743
- swap accepts bytes its own interface file does not declare. That residual is now
744
- measured rather than noted: see *Three wire forms, one interface* below.
745
- - **pump.fun with the on-chain IDL alone:** **669/996 decoded (67.2%)**, 327 refused
746
- (326 unknown instruction discriminators across five instructions, 1 unknown event
747
- discriminator); **581 byte-exact**, 88 with trailing bytes.
748
- - **The published first-party IDL names the 326 missing instructions:**
749
- `buy_exact_quote_in_v3` (107), `buy_v3` (2), `multi_hop_curve_swap` (1), `sell_v3`
750
- (194), `sweep_creator_fee` (22) — all absent from the on-chain account, all present
751
- in `pump-fun/pump-public-docs` `idl/pump.json`. The on-chain account is simply stale.
752
- - **Neither source alone describes the deployed program:** with the published copy,
753
- **880/996 decode** (782 instructions + 98 events); the remaining 116 refusals come
754
- from `buy` carrying a `partial_fill` argument the deployed program does not always
755
- send. With the **union** of both, **914/996 decode (91.8%)**, 82 refused.
756
- - **Refusal does not predict on-chain failure:** of 116 refused payloads, **32 were
757
- part of transactions that SUCCEEDED on-chain** (and 84 failed); of 880 decoded
758
- payloads, **715 were from transactions that FAILED on-chain** (slippage, etc.). A
759
- decoder refusal is evidence about the IDL, not about whether the program executed.
760
- - **TradeEvent has an unexplained tail:** every one of the 87 TradeEvents decoded
761
- under the published IDL carries exactly **8 trailing zero bytes** (`count: 8, hex:
762
- "0000000000000000"`) no declared field explains.
763
-
764
- **Codec defect found and fixed:** Anchor describes a Rust tuple struct (`pub struct
765
- OptionBool(pub bool)`) as `{"name":"OptionBool","type":{"fields":["bool"],"kind":"struct"}}`
766
- — bare types, no `{name, type}` pairs. sememe's struct branch demanded a `name` on
767
- every field and threw `OptionBool: malformed struct field` on **18 payloads** under
768
- the on-chain IDL and **145 payloads** under the published IDL before the fix. The fix
769
- adds tuple-struct support in both decode and encode, matching Anchor's own
770
- representation (`{"0": value}`). Falsification: `test/tuple-struct.test.mjs` fails
771
- against the pre-fix tree (1 pass / 3 fail; the 1 is the fixture-shape guard), passes
772
- 4/4 post-fix. After the fix, on-chain decodes 651 → 669; published decodes 851 → 880.
773
-
774
- Covered by `test/tuple-struct.test.mjs` (4 tests) and `test/traffic-coverage.test.mjs`
775
- (10 tests); fixtures `test/fixtures/cp-amm-idl.json`, `pump-idl.json`,
776
- `pump-published-idl.json`, `harvest-cp-amm-2026-10-30.json`,
777
- `harvest-pump-2026-10-30.json`. The numbers above are reproduced, not written:
778
- `node tools/traffic-coverage.mjs` emits
779
- [research/traffic-coverage-2026-10-30.json](research/traffic-coverage-2026-10-30.json),
780
- and a test fails if the committed file differs from what the tool computes
781
- (measured: one hand-edited number fails it, restoring passes). Suite: **154/154**
782
- (up from 140).
783
-
784
- ## Working now: union decode — two verified sources, one payload, disagreement reported
785
-
786
- The pump.fun measurement above produced three numbers for the same 996 payloads:
787
- 669 (on-chain copy alone), 880 (published copy alone), 914 (a hand-merged union
788
- inside a measurement tool). The union was the only view that described the deployed
789
- program, and it did not exist outside that tool. It does now:
790
-
791
- ```bash
792
- sememe adopt 6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P idl/pump.json --store <dir> --origin pump-fun/pump-public-docs
793
- sememe decode --union 6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P <payload> --store <dir>
794
- ```
795
-
796
- - **`adopt` stores a first-party published copy** content-addressed exactly like a
797
- pull: canonical bytes, sha-256 fingerprint, write-once object, receipt. The
798
- receipt says `source: published`, `slot: null` and carries the `origin` the
799
- caller declared. That distinction is deliberate: `pull` gets its authority from
800
- the chain, `adopt` is **provenance of bytes, not proof the bytes describe the
801
- deployed program** — and the decode result must keep saying which source
802
- answered.
803
- - **The adopted copy can never become the default source.** It sorts before every
804
- real slot in the single-source reader's filename ordering, so plain
805
- `sememe decode` reads exactly what it read before this feature existed — the
806
- pre-existing 154 tests pass unchanged, and a test pins the selection
807
- explicitly (falsified by mutation M4: letting the adopted receipt win turns it
808
- red).
809
- - **Both sources are verified by the same reader** (`src/store-read.mjs`): receipt
810
- program binding, object presence, re-hash against the receipt, address
811
- agreement. A tampered adopted object is refused **by name in the union path
812
- too** — not silently dropped in favour of the surviving source (falsified by
813
- mutation M5).
814
-
815
- **The arbitration is decided by bytes, not by preference**, strongest evidence first:
816
-
817
- 1. both sources decode the same name and args → `agree`;
818
- 2. both decode, one re-encodes the **whole** payload and the other does not → the
819
- full-round-trip source wins (`byte_evidence_full_roundtrip`);
820
- 3. neither explains everything, one leaves **strictly fewer** unexplained bytes →
821
- that source wins (`byte_evidence_fewer_unexplained_bytes`);
822
- 4. evidence does not decide → the on-chain primary stands and the drift is
823
- **reported with the differing field paths** — never absorbed;
824
- 5. one source refuses → the other answers, but only with byte evidence behind it
825
- (`primary_refused` / `secondary_refused`);
826
- 6. **two different names for the same bytes → refusal**, not a coin flip
827
- (`name_conflict`); the same for a secondary source that names bytes it cannot
828
- reproduce (`union_fallback_refused`).
829
-
830
- **Measured, product path, the same 996 pump.fun payloads**
831
- (`node tools/union-coverage.mjs` →
832
- [research/union-coverage-2026-10-30.json](research/union-coverage-2026-10-30.json),
833
- recomputed in CI — a committed-report mismatch fails the suite):
834
-
835
- | verdict | payloads | what it means |
836
- |---|---|---|
837
- | `agree` | 546 | both sources decode identical arguments |
838
- | `fallback_secondary` | 246 | 245 payloads of the five stale on-chain instructions + the `SweepBondingCurveFeeEvent` payload the hand-merged union refused |
839
- | `primary_only` | 35 | the published copy's layout is taller than the bytes sent (truncated), the on-chain copy decodes them |
840
- | `layout_drift` | 88 | both decode, arguments differ — reported with field paths, winner chosen by byte evidence |
841
- | refused by both | 81 | the on-chain copy lacks the instruction entirely AND the published copy cannot decode these bytes |
842
- | **decoded** | **915/996 (91.9%)** | 915 re-encode their described prefix byte-exactly; **828 re-encode the WHOLE payload** |
843
-
844
- The **+1 over the hand-merged 914** is a real difference, not noise: that merge
845
- added *instructions* only, so the published-only event `SweepBondingCurveFeeEvent`
846
- (`742b4dbd117a482b`) was refused; the product union carries both sources whole.
847
-
848
- The 88 drifts are two real, measured disagreements — the reason the union exists:
849
-
850
- - **CreateEvent (1 payload):** the published copy declares `depth: u8`, the
851
- on-chain copy does not. On-chain leaves 1 byte unexplained; the published copy
852
- re-encodes the **whole** payload, so byte evidence picks it, and the winning
853
- decode carries `depth`.
854
- - **TradeEvent (87 payloads):** the published copy declares an additional
855
- `creator_fee_unclaimed: u64`. On-chain leaves 16 bytes unexplained, published
856
- leaves 8 (the known 8-byte zero tail neither explains) — strictly-fewer-
857
- unexplained-bytes picks the published copy, and the tail stays visible under
858
- `drift.trailingBytes` instead of being laundered into a clean decode.
859
-
860
- **Falsification:** `bash tools/union-mutations.sh` breaks each rule on purpose —
861
- M1 name conflict, M2 fallback without byte evidence, M3a/M3b the two
862
- byte-evidence arbitration rules, M4 adopted-as-default, M5 tamper check — and
863
- requires exactly the named test to go red in every case, control run green. The
864
- measurements are pinned twice: `test/union-coverage.test.mjs` deep-equals the
865
- committed report against what the tool computes right now, and asserts each
866
- headline count with a message saying what a regression would mean. Suite:
867
- **169/169** (up from 154).
868
-
869
- **Limits, stated:** one program, one harvest window, both sources are mine (on-chain
870
- pull + first-party published copy), no outside user. Account decoding still reads a
871
- single receipt — union arbitration is not wired for accounts. An adopted receipt's
872
- `origin` is a declared label, not a verified origin — it is now *marked* as declared
873
- everywhere it is shown and *bound* so it cannot be rewritten silently (see
874
- "Working now: a declared origin is declared, and cannot be rewritten silently"),
875
- but its truth is still not checked.
876
-
877
- ## Working now: one store, one answer — multiple copies of one program (2026-11-07)
878
-
879
- A store may hold **more than one adopted copy of the same program** (a docs
880
- pair, or a pull plus a copy adopted again under a different origin). Measured
881
- on the PUBLISHED `v0.4.8` release (installed from its tag into an empty app,
882
- no clone) — with two adopted copies of pump.fun in one store:
883
-
884
- | command | answer | exit |
885
- |---|---|---|
886
- | `sememe decode <payload>` | `sell_v3`, fingerprint `a35349f3…` | 0 |
887
- | `sememe decode --union <payload>` | **REFUSED** `unknown instruction discriminator 1c92de7726c469d5 (47 known, …)` | **1** |
888
- | `sememe lock` | pinned `a35349f3…` over a store whose **second copy was damaged** | **0** |
889
- | `sememe verify` | `violations: 0`, `verified: [pump.fun]` | **0** |
890
-
891
- One store, two verdicts depending on the flag you passed — and a green CI gate
892
- over damage. Both holes are named, in `src/store-read.mjs` and `src/lock.mjs`:
893
-
894
- - **The adopted copies were sorted by LABEL, not by content.** `findStoredIdls`
895
- sorted adopted receipts by `label.localeCompare(label)` — and the label
896
- embeds the **origin the caller typed** (`published (adopted, sdk-copy)`),
897
- while the plain reader (`findStoredIdl`) and `src/lock.mjs` sorted by
898
- **receipt filename**, which ends in the fingerprint. Two sorts, two winners,
899
- no disagreement reported: the union path read the copy the origin spelling
900
- pointed at, the plain path and the lock read the copy the content pointed
901
- at. The rule is now one ordering everywhere — receipt filename — and when a
902
- store holds no on-chain receipt and several adopted copies, they form the
903
- arbitrated pair instead of one of them being consulted and the rest dropped.
904
- Whatever the pair does not consult is returned as `additionalSources`, so a
905
- store holding three copies never reads as a two-source store.
906
- - **The lock re-checked only the newest receipt per program.** README already
907
- promised `every receipt is re-checked through the store reader first, so a
908
- store the reader refuses produces NO lock, never a lock over damage` — but
909
- `writeLock` only re-verified `names[names.length - 1]`, so a damaged
910
- **second** copy (the very copy union decode reads as its secondary) was
911
- skipped in silence. `writeLock` and `checkLock` now share ONE helper
912
- (`verifyEveryReceipt`), so lock and verify cannot disagree about what a
913
- clean store is: a damaged copy refuses the lock **and** is named `unverified`
914
- at verify time, and `--mode error` exits 1.
915
-
916
- The measurement ships as a tool and two committed artifacts, following the
917
- repo's `tool produces the report` rule:
918
-
919
- ```bash
920
- node tools/measure-multi-copy-store.mjs # this tree → research/multi-copy-store-2026-11-07.json
921
- node tools/measure-multi-copy-store.mjs --bin <installed bin> --tag v0.4.8 --out before.json
922
- ```
923
-
924
- [research/multi-copy-store-2026-11-07-before.json](research/multi-copy-store-2026-11-07-before.json)
925
- is the published `v0.4.8` install: **7 contract violations** (union refused,
926
- two different copies named, lock written over damage, verify green over
927
- damage); [research/multi-copy-store-2026-11-07.json](research/multi-copy-store-2026-11-07.json)
928
- is this tree: **0 violations**, and the single-copy control stays green in both.
929
- `test/multi-copy-store-measurement.test.mjs` pins both artifacts and fails if
930
- the committed report stops being what the tool derives; the 11-test ladder in
931
- [ test/multi-copy-store.test.mjs ](test/multi-copy-store.test.mjs) pins the
932
- behaviour with three mutations that must each turn a test red (label sort →
933
- M1 red, newest-only lock → M3 red, dropped second source → M2 red), and
934
- `test/multi-copy-store-tag-journey.test.mjs` packs the **pinned tag** and runs
935
- the same measurement against the bytes a stranger installs — red against
936
- `v0.4.8` by design, green from `v0.4.9` on.
937
-
938
- **Stated limit:** `origin` is still a declared label, not a verified origin —
939
- this fix makes the label *not decide which copy answers*, which is the part
940
- that was silently changing decode results; it does not verify the label's
941
- truth. The labelling of a declared origin is now done (a later fix marks it
942
- declared and binds it); what remains open is verifying the label's truth,
943
- and adopting published copies for a second program.
944
-
945
- ## Working now: one ordering for receipts — slot order is not digit order (2026-11-07)
946
-
947
- The receipt filename is `<program>-<slot>-<source>-<fingerprint>.json`, and the
948
- slot inside it is a decimal **number stored as a string**. Sorting receipt
949
- names with a plain string comparison therefore orders by **digits, not by
950
- value**: `…-99999999-…` sorts *after* `…-123456789-…`, so the OLDER receipt
951
- wins "last position" — and `findStoredIdl` takes the last position as
952
- "newest". Measured on the PUBLISHED `v0.4.9` install (installed from its git
953
- tag into an empty app, no clone) with a store holding one program's receipts at slots
954
- 99999999 (older) and 123456789 (newer):
955
-
956
- | command | answers | exit |
957
- |---|---|---|
958
- | `sememe decode <payload>` | **refused** — it read the OLDER copy, which does not know `sell_v3` | **1** |
959
- | `sememe decode --union <payload>` | `sell_v3`, slot `123456789` (the NEWER copy) | 0 |
960
- | `sememe lock` | pinned slot `99999999` — a *different* receipt than the union decode read | 0 |
961
- | `sememe verify` | clean over that lock | 0 |
962
-
963
- One store, three surfaces, two different receipts — and a lock pinning a
964
- receipt the decoders did not read. The union path has sorted on-chain receipts
965
- by **numeric slot** since it was written (`onChain.sort((a, b) => a.slot -
966
- b.slot)`), while the plain reader, `receiptNames()` and `src/lock.mjs` sorted
967
- by **filename**. The v0.4.9 fix collapsed label-order and filename-order into
968
- "ONE ordering everywhere (receipt filename)" — true for labels, but filename
969
- order is digit order, and digit order is not slot order. Same defect shape,
970
- one dimension over: two sorts, two winners.
971
-
972
- **The rule now is one comparator** (`src/receipt-order.mjs`), shared by the
973
- plain reader, the union path and both lock paths: sort by **slot
974
- numerically** (adopted copies carry filename segment `0`, so they still sort
975
- before every real slot — adoption never switches the default source), with the
976
- rest of the filename as byte-wise tiebreak so equal slots stay deterministic.
977
-
978
- The measurement ships as a tool and two committed artifacts, following the
979
- repo's `tool produces the report` rule:
980
-
981
- ```bash
982
- node tools/measure-slot-order.mjs # this tree → research/slot-order-2026-11-07.json
983
- node tools/measure-slot-order.mjs --bin <installed bin> --tag v0.4.9 --out before.json
984
- ```
985
-
986
- [research/slot-order-2026-11-07-before.json](research/slot-order-2026-11-07-before.json)
987
- is the published `v0.4.9` install: **5 contract violations** (plain decode
988
- refused while union answered, lock pinned a different receipt);
989
- [research/slot-order-2026-11-07.json](research/slot-order-2026-11-07.json) is
990
- this tree: **0 violations**, and the equal-length control — what a fresh
991
- `pull` writes today — stays green on both sides.
992
- [research/slot-order-2026-11-07-v0.4.10.json](research/slot-order-2026-11-07-v0.4.10.json)
993
- is the **published `v0.4.10` tag** installed into an empty app: **0
994
- violations** — the fix is in the bytes a stranger receives, not only in this
995
- tree.
996
- `test/slot-order-measurement.test.mjs` pins all three artifacts and fails if a
997
- committed report stops being what the tool derives; the ladder in
998
- [test/slot-order-store.test.mjs](test/slot-order-store.test.mjs) pins the
999
- behaviour (O1/O2/O3 red on the pre-fix tree, O4 control green there, two
1000
- mutations that must each turn a test red), and **O6 pins a SECOND program** —
1001
- whirlpool with two adopted copies, plain/union/lock agreeing and a damaged
1002
- second copy refusing the lock — closing the "adopted copies for more than one
1003
- program" item for the lock contract.
1004
-
1005
- **Reachability, stated honestly rather than sold:** every live slot measured
1006
- on this host is 9 digits (44,661 values in `research/` + fixtures, min
1007
- 462039125, max 465713413), so two receipts of one program written by `pull`
1008
- today have equal-length slots and both orderings agree — the defect is latent
1009
- for a fresh pull. It becomes LIVE for any store holding receipts whose slot
1010
- strings differ in length: a receipt carried over from another era or cluster,
1011
- a hand-assembled store, or mainnet slots crossing 1,000,000,000. The contract
1012
- ("newest stored IDL") has to hold in that store too.
1013
-
1014
- ## Working now: a declared origin is declared, and cannot be rewritten silently (2026-11-07)
1015
-
1016
- `sememe adopt` writes a receipt whose `origin` is whatever the caller typed.
1017
- Every reader surface then presented that string as where the bytes came from:
1018
- `"origin": "solana-labs-anchor-deadbeef-does-not-exist.json"`, reader label
1019
- `published (adopted, solana-labs-...)`, union view `primary.label = ...`.
1020
- Nothing checked that the named repository exists, that the file is there, or
1021
- that the bytes came from it at all.
1022
-
1023
- Measured on the published `v0.4.10` tag:
1024
-
1025
- 1. **A caller-typed string was shown as provenance without qualification.**
1026
- The CLI said `"origin": "..."` and reader labels said
1027
- `published (adopted, ...)`.
1028
- 2. **The origin was covered by NOTHING.** The fingerprint covers the
1029
- object (the canonical IDL bytes); the receipt was covered by nothing.
1030
- Rewriting `origin` in a receipt to `attacker/rewritten-origin` and running
1031
- `sememe verify --mode warn` reported **violations: 0, verified: ["6EF8rrec..."]**;
1032
- `sememe verify --mode error` exited **0**. The CI gate was green over a
1033
- rewritten provenance label.
1034
- 3. **Adopting the SAME bytes under two declared origins was a false collision.**
1035
- The second adopt failed with `"digest collision or corrupted local object"`
1036
- because both wrote the same receipt filename with different content.
1037
-
1038
- The fix:
1039
-
1040
- - **Every reader surface marks the origin as DECLARED** (`originStatus: "declared"`,
1041
- reader label `published (adopted, declared origin: ...)`), and the CLI prints
1042
- the advisory note that sememe did not check that the repo, commit or file
1043
- exists.
1044
- - **The receipt carries its own digests** (`originDigest` binds the declared
1045
- origin; `receiptDigest` binds the whole receipt body). A receipt edited
1046
- afterwards is refused on every read with a named reason; `sememe lock` pins
1047
- the declared origin alongside the fingerprint, and `sememe verify` names a
1048
- rewritten origin as `relabelled`, exiting 1 in error mode.
1049
- - **An existing user's committed lock (which predates origin pinning) does
1050
- not silently certify the label.** `sememe verify` reports an adopted copy
1051
- whose lock pins no origin as `unbound`, exiting 1 with the one-command
1052
- repair (`sememe lock`) named in the refusal. The pre-fix gate passed over
1053
- the rewrite (measured: exit 0, `verified: [program]`), certifying a
1054
- provenance label it never pinned.
1055
- - **Two declared origins for one set of bytes are two receipts.** The second
1056
- adopt writes `<prog>-0-<source>-<originKey>-<fingerprint>.json`, where
1057
- `originKey` is a short digest of the declared origin. Both receipts keep
1058
- the same fingerprint, so which of them sorts last cannot change a decoded
1059
- byte — only which declared label is shown first.
1060
-
1061
- [research/declared-origin-2026-11-07-before.json](research/declared-origin-2026-11-07-before.json)
1062
- is the published `v0.4.10` tag: **12 contract violations**;
1063
- [research/declared-origin-2026-11-07.json](research/declared-origin-2026-11-07.json)
1064
- is this tree: **0 violations**, and the control — an untouched store — stays
1065
- green on both sides.
1066
- `test/declared-origin-measurement.test.mjs` pins both artifacts;
1067
- [test/declared-origin.test.mjs](test/declared-origin.test.mjs) pins the
1068
- behaviour (P1/P2/P3/P3b/P4/P4b red on the pre-fix tree, P5 control green there).
1069
- [research/legacy-unbound-origin-2026-11-07.json](research/legacy-unbound-origin-2026-11-07.json)
1070
- records the legacy case: pre-fix exit 0 (the defect), post-fix exit 1
1071
- (`unbound`, with the one-command repair named), the control (lock first ->
1072
- `relabelled`), and the false-positive guard (a pull-only store declares no
1073
- origin and stays clean). `test/legacy-unbound-measurement.test.mjs` pins that
1074
- artifact; [test/legacy-origin-binding.test.mjs](test/legacy-origin-binding.test.mjs)
1075
- pins the behaviour.
1076
-
1077
- **Stated limit:** an adopted copy's origin is still not verified by this fix.
1078
- What the fix does is stop presenting a typed string as a checked fact, and
1079
- stop letting the string change without the CI gate noticing. Verification of
1080
- provenance remains first-party adoption with a recorded commit and the
1081
- on-chain pull, which needs no label at all.
1082
-
1083
- ## Working now: account union — two verified sources, one account (2026-11-08)
1084
-
1085
- Union decode covered instructions and events since v0.4.0. **Accounts never
1086
- had the second half**: `sememe account` read exactly ONE receipt, so a store
1087
- holding two verified copies of a program had two opinions about an account's
1088
- layout and only one of them was ever consulted — the item this README carried
1089
- as open ("the same arbitration for account decoding (accounts still read a
1090
- single receipt)").
1091
-
1092
- ```bash
1093
- sememe account --union 6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P <base64> --store <dir>
1094
- ```
1095
-
1096
- The pairing, the verdicts and the arbitration rules are the instruction
1097
- union's: primary = newest on-chain receipt, secondary = newest adopted copy,
1098
- `agree` / `layout_drift` / `fallback_secondary` / `primary_only` /
1099
- `single_source`, and a **name conflict is refused, never arbitrated** (two
1100
- different account names for the same 8 bytes is an interface conflict, not a
1101
- decision to make).
1102
-
1103
- **The measured case** (`research/account-union-2026-11-08.json`, 12 real
1104
- mainnet pump.fun accounts harvested 2026-11-08 with `tools/account-harvest.mjs`;
1105
- the two committed copies disagree about `BondingCurve` — 13 fields on-chain,
1106
- 19 in the published copy):
1107
-
1108
- | | single receipt (before) | union (`--union`) |
1109
- |---|---|---|
1110
- | byte-exact | **0 / 12** | **6 / 12** |
1111
- | bytes left unexplained | **12 / 12** | 6 / 12 |
1112
- | verdict | one source, no verdict | `layout_drift` 6, `agree` 6 |
1113
-
1114
- The 6 recovered accounts (5 `BondingCurve`, 1 `Global`) are the ones where
1115
- the published copy reproduces every byte and the on-chain copy leaves 41 (and
1116
- 1) unexplained — byte evidence picks the published copy and records why. The
1117
- 6 `UserVolumeAccumulator` accounts are `agree`: both copies say the same
1118
- thing, and **both** leave 31 bytes unexplained — agreement is reported, not
1119
- papered over as coverage.
1120
-
1121
- The rules that make this an arbitration rather than a decoder that always
1122
- answers (`test/account-union.test.mjs`, 14 tests, 2 mutations verified red on
1123
- either side of their fix):
1124
-
1125
- - **A fallback needs byte evidence.** A source that names an account but cannot
1126
- reproduce it is refused. The honest no-evidence state is reachable and is
1127
- pinned with a real codec property, not a mock: an `f32` field holding a
1128
- *signaling* NaN decodes to `null` and re-encodes to a different bit pattern —
1129
- the decode succeeds, consumes the whole account, and reproduces nothing.
1130
- - **When the evidence ties, the primary stands.** Two layouts can serialise to
1131
- the same bytes and decode to different values (`u32` vs `i32` over
1132
- `0xffffffff`): both are `full_roundtrip`, so the on-chain pull is the answer
1133
- and the drift is reported — never resolved by preference.
1134
- - **The plain path is untouched.** `sememe account` still reads one receipt and
1135
- carries no `view`; the union is a flag, not a behaviour change.
1136
-
1137
- **Stated limits:** a byte-exact round trip proves the bytes are *reproduced*,
1138
- not that the field names mean what the program intends; the published copy is
1139
- adopted content whose origin stays declared, never verified.
1140
-
1141
- **Does this generalize beyond pump.fun?** Measured 2026-11-08 across all 6
1142
- programs with committed account fixtures (74 real mainnet accounts,
1143
- `research/account-union-cross-2026-11-08.json`, `tools/measure-account-union-cross.mjs`,
1144
- pinned in `test/account-union-cross-measurement.test.mjs`):
1145
-
1146
- | program | two sources? | plain byte-exact | union byte-exact | recovery | verdict |
1147
- |---|---|---|---|---|---|
1148
- | **pump.fun** | yes (12 accts) | 0 / 12 | **6 / 12** | **+6** | `layout_drift` 6, `agree` 6 |
1149
- | **meteora-vault** | yes (12 accts) | 0 / 12 | 0 / 12 | 0 | `primary_only` 12 (copy renames `Vault` -> `vault`) |
1150
- | **meteora-damm-v1** | yes (11 accts) | 0 / 11 | 0 / 11 | 0 | `layout_drift` 11 (both leave 77/512 bytes unexplained) |
1151
- | **openbook-v2** | yes (4 accts) | 4 / 4 | 4 / 4 | 0 | `agree` 4 (both copies already byte-exact) |
1152
- | **whirlpool** | control (27 accts) | 27 / 27 | 27 / 27 | 0 | `single_source` 27 (plain answer untouched) |
1153
- | **kamino-lend** | control (8 accts) | 8 / 8 | 8 / 8 | 0 | `single_source` 8 (plain answer untouched) |
1154
-
1155
- The honest finding: **arbitration recovered a byte-exact answer on exactly 1 of
1156
- the 4 two-source programs.** On the other three, the second copy agreed (openbook-v2),
1157
- disagreed without either side being byte-exact (meteora-damm-v1), or could not
1158
- even identify the account because the copy renames it (meteora-vault: Anchor
1159
- derives the discriminator from the exact name, so `vault` derives `ded54f7c…`
1160
- and can never match the deployed bytes which carry `Vault`'s `d308e82b…` — the
1161
- reader now diagnoses this mechanism explicitly instead of reporting "no such
1162
- account"). The four regression invariants (union never answers worse than
1163
- plain, never refuses what plain answered, never changes single-source
1164
- verdicts, never names a different account type) hold at 0 failures across all
1165
- 74 accounts.
1166
-
1167
- ### The stale layout behind the rename, measured (2026-11-09)
1168
-
1169
- The `meteora-vault` rename is not only a spelling difference. Measured on the
1170
- same 12 accounts (`research/stale-layout-2026-11-09.json`,
1171
- `tools/measure-stale-layout.mjs`, pinned in
1172
- `test/stale-layout-measurement.test.mjs`):
1173
-
1174
- - the **live on-chain IDL is frozen at v0.7.2** and declares
1175
- `strategies: publicKey[30]` — a 30th strategy slot that the deployed program
1176
- never filled;
1177
- - the **published copy** declares `strategies: publicKey[29]` followed by
1178
- `version: u8`, `totalLpSupply: u64` and `padding0: [u8; 23]` — the same
1179
- 960 bytes, read three more fields;
1180
- - **offline proof, 12 / 12 accounts**: the 32 bytes the old layout calls
1181
- `strategies[29]` decompose *exactly* as `version || totalLpSupply || padding0`;
1182
- `version == 1` in 12 / 12 and `padding0` is all zero in 12 / 12;
1183
- - **live oracle on mainnet, 2026-11-09**: the phantom 30th strategy account
1184
- exists in **0 / 12** vaults, while the 29 real strategy accounts exist in
1185
- **348 / 348**; the live `totalLpSupply` equals the live LP mint supply in
1186
- **9 / 12** vaults (the other 3 grew after their harvest, which is why the
1187
- frozen bytes no longer match the chain).
1188
-
1189
- So the published copy describes the deployed program and the frozen on-chain
1190
- IDL does not. **The verdict still does not move.** The account union refuses
1191
- the renamed copy (Anchor derives the discriminator from the exact name, so
1192
- `vault` can never match the deployed bytes), and `primary_only` stands. The
1193
- layout is reported as `view.declaredLayout` — a *report*, not evidence: it
1194
- says "the answer you got is contested, field by field", and it is attached
1195
- only when a rename was actually diagnosed, never invented.
1196
-
1197
- The rules this report obeys (`test/account-union-stale-layout.test.mjs`, 8
1198
- tests, 2 mutations verified red):
1199
-
1200
- - **the report never arbitrates.** `primary_only` stays `primary_only`; a
1201
- copy that explains more bytes does not become the winner.
1202
- - **naming style is not a layout difference.** Fields are paired by
1203
- normalised spelling, so `totalAmount` vs `total_amount` is not reported as
1204
- drift; a field only one copy declares, or one both declare with a different
1205
- type, is.
1206
- - **no rename, no report.** The field exists only when a rename was really
1207
- diagnosed.
1208
-
1209
- **Stated limits:** the on-chain IDL being stale is a claim about the
1210
- *program's* published artifact, not about the deployed program — the
1211
- decomposition above is measured from the harvested bytes and confirmed against
1212
- the chain, but the upstream Meteora IDL is not this repository's to fix. The
1213
- staleness is reported, never repaired here.
1214
-
1215
-
1216
- ## Free and paid
121
+ ## Free and paid
1217
122
 
1218
123
  - Free, MIT: all acquisition channels, normalization, fingerprint store, checks, decode and encode APIs, CLI, daemon, lock files, and fully offline resolution. No account or quota.
1219
124
  - Paid later: managed team registries, curated coverage packs for programs without a public interface, and enterprise approval and audit workflows. A decode call is never sold.
1220
-
1221
- ## Evidence and limits
1222
-
1223
- Checked on 2026-10-25: about 99 first-party decoder or discriminator sites and four provenance-free vendored IDLs exist across my own repositories — an **estimate from notes, not a counted figure**. A computed scan on 2026-10-27 ([research/decoder-sites-2026-10-27.json](research/decoder-sites-2026-10-27.json), `node tools/decoder-sites.mjs`) found **42 (file, class) sites across 35 files in 5 own repositories** (plus 7 own fork checkouts holding 407 sites that are someone else's code, and 394 in third-party checkouts, which are not mine to replace). The two figures use different counting units, so do not treat 99 and 42 as measuring the same thing; until one unit is fixed and re-counted, the honest statement is that the number of my own decoder sites is *somewhere in that range and not yet pinned down*. Jupiter maintains its own parser, and a static public IDL catalog has users, while earlier registry and deserializer attempts stalled. Helius sells parsed events and streams as a paid product.
1224
-
1225
- The largest risk is coverage. If on-chain acquisition works for fewer than half of the programs I actually touch, sememe is only an import-and-lock tool and the venture depends on paid curation. A first selected sample now tests this **before any product implementation**: on 2026-10-27, a batch mainnet RPC query at finalized slot 462035811 found an IDL with a nonempty instruction list at one of two known account addresses for **15/20 programs (75%)**; 5/20 had neither account, 0 RPC errors. Excluding the two deliberately non-Anchor SPL negative controls: **15/18 (83.3%)**. Four succeeded via Program Metadata, eleven via legacy Anchor. This is a family-clustered convenience sample concentrated in Raydium/Meteora, **not** a representative estimate of all Solana programs or even a systematic inventory of my own production workloads, nor evidence that third parties need sememe. 'Usable' here means decodable JSON with at least one named instruction and correct program binding where declared; it does not prove full ABI fidelity or transaction decoding. **Correction 2026-10-29:** it also does not prove that a decoder can *use* the schema — Kamino Lend, one of the eleven legacy-Anchor successes, stores no discriminators at all (0 of 71 entries), so its schema decoded 0 of 1,432 real payloads until Anchor's rule filled the gaps.
1226
-
1227
- The strict figure **has now been measured** over the same frozen 20 IDs and the same RPC shape (commit `dcb0d3`, [research/coverage-strict-2026-10-29.json](research/coverage-strict-2026-10-29.json)):
1228
-
1229
- | definition | count | share of 20 |
1230
- |---|---|---|
1231
- | IDL present with a named instruction (the old 'usable') | 15/20 | 75% |
1232
- | **decodable from published bytes alone** (the program stored its own discriminators) | **10/20** | **50%** |
1233
- | decodable after Anchor's rule fills a missing discriminator | 15/20 | 75% |
1234
-
1235
- The middle row is the honest headline: **half** of the sample's usable programs publish an interface that decodes as-is; the other five (Kamino Lend, OpenBook v2, Marinade, Meteora DAMM v1, Meteora Vault) publish names and layouts but no wire discriminators, and only become decodable because the tool can reconstruct them from the name. That reconstruction is verified per program, but it is the tool's inference, not the program's published interface — and it is Anchor-specific. **All five are now individually proven against real mainnet payloads** (Kamino Lend 1,432/1,432, OpenBook v2 600/600, Marinade 623/623, Meteora Vault 1,290/1,290, Meteora DAMM v1 643/643, each byte-exact with 0 decodable without derivation). See [frozen 20 IDs](programs-20.json), [raw classification with slot and account addresses](coverage-result.json), and [selection/chronology limitations](research/selection-method.md). The 20 IDs were committed before the full batch run, but the detailed selection-method note came later; one exploratory Whirlpool RPC check preceded the freeze. Do not call the whole study blind or fully preregistered.
1236
-
1237
- ## The stale copies in my own repositories, measured
1238
-
1239
- The claim this repository opens with — "vendored IDL files are snapshots without provenance, a fingerprint, or a freshness check" — was an assertion from notes. `tools/vendored-drift.mjs` turns it into a number for the IDL copies checked into my own repositories, by pulling the on-chain artifact for each program the copy names and comparing the two.
1240
-
1241
- Measured on 2026-10-30 (`research/vendored-drift-tx-remix-2026-10-30.json`):
1242
-
1243
- | vendored copy | instructions | on-chain | missing from the vendored copy |
1244
- |---|---|---|---|
1245
- | `solana-tx-remix/idl/multi/jupiter-amm/amm.json` | 17 | 19 | `create_idempotent_associated_token_account`, `withdraw_token_account_excess_lamports` |
1246
- | `solana-tx-remix/idl/multi/dflow/dflow.json` | 17 | 18 | `withdraw_fees` |
1247
- | `solana-tx-remix/idl/multi/jupiter-order_engine/order_engine.json` | 1 | 1 | — |
1248
- | `solana-tx-remix/idl/multi/jupiter-rfq_v2/rfq_v2.json` | 1 | 1 | — |
1249
- | `aquifer`, `bisonfi`, `solfi`, `spl/token-2022` | 2 / 3 / 1 / 44 | — | no on-chain Anchor IDL exists for these programs |
1250
-
1251
- **Every vendored copy that HAS an on-chain IDL is strictly stale.** That is the whole claim, measured rather than asserted — and the staleness is not academic: `create_idempotent_associated_token_account` appears in **real mainnet traffic**, so a decoder built on the vendored copy emitted `(unknown)` for a successful instruction, and `(unknown)` is the one state every label check deliberately skips.
1252
-
1253
- The four programs with no on-chain IDL are not a gap in the store: they are native or shank-origin programs that publish no Anchor IDL account at all. For those the checked-in file is the only description available, and it is kept.
1254
-
1255
- The tool distinguishes three defects, because they are not the same defect: a **stale** copy (has an on-chain IDL, and it differs), an **unbound** copy (declares no program address, so it binds nothing — the `bisonfi.json` failure mode `solana-tx-remix` already documents), and a copy for a program that has no on-chain IDL at all.
1256
-
1257
- ### The measurement tool misclassified 396 payloads as "unknown to both" (measured 2026-10-30)
1258
-
1259
- The traffic column of that report split payloads the vendored copy cannot name into three buckets: named by the on-chain copy as an instruction, an Anchor `emit_cpi!` event self-CPI, or unknown to both copies. The published numbers were **773 payloads, 5 unnamed by the vendored copy, 0 event self-CPIs, 396 unknown to both**.
1260
-
1261
- That was wrong, and it was wrong in the direction that looks like a coverage finding. Anchor event payloads are `EVENT_IX_TAG (8) ++ event_discriminator (8) ++ borsh(body)`, so the discriminator that identifies the event sits at bytes **8..16** — the tool looked up bytes **0..8**, which is the *tag*, in the event table. The tag is never an event discriminator, so the lookup always missed: all 396 tagged Jupiter payloads were published as "unknown to both copies" while the event bucket stayed empty, even though this repository's own decoder names every one of them (README above: *773/773 payloads decoded — 377 instructions + 396 Anchor event self-CPIs*). The measurement and the decoder contradicted each other, and the report was the wrong one.
1262
-
1263
- The fix looks up bytes 8..16 when the payload starts with the event tag, and the tag constant is now imported from the decoder (`EVENT_IX_TAG_HEX` in `src/codec.mjs`) instead of being written out a second time. Regenerated against the same committed harvest:
1264
-
1265
- | | payloads | unnamed by vendored | event self-CPIs | unknown to both |
1266
- |---|---|---|---|---|
1267
- | before | 773 | 5 | 0 | 396 |
1268
- | after | 773 | 5 | **396** | **0** |
1269
-
1270
- The finding that mattered is unchanged: the vendored Jupiter copy still cannot name `create_idempotent_associated_token_account`, which appears in **5 real mainnet payloads**. What changed is that the other 396 payloads are no longer charged to a defect that does not exist — a decoder that names them was always in the room.
1271
-
1272
- The falsification is committed: `test/vendored-drift.test.mjs` pins the classification against the real committed harvest (396 event self-CPIs, 0 unknown to both, 5 unnamed by the vendored copy) and against a synthetic three-payload case (one tagged event, one shared instruction, one genuinely unknown). Against the pre-fix tool the synthetic test fails (`eventSelfCpiPayloads` 0 instead of 1) and the real-data assertions fail with it.
1273
-
1274
- ## A second consumer reads the artifact for every program it names
1275
-
1276
- `solana-tx-remix` migrated its single-program Whirlpool path first. Its multi-program path kept reading the vendored copies — and the measurement above is what that cost: `(unknown)` on a real Jupiter instruction, in a transaction that was otherwise described in full.
1277
-
1278
- Both of that consumer's entry points now resolve labels through one resolver (`buildStoredResolver` in its `src/dispatch.mjs`) that layers the stored artifacts over the vendored copies. The overlay is additive per program, so the vendored copy still describes the programs the store does not cover.
1279
-
1280
- Two things it does not paper over, both measured:
1281
-
1282
- - A **stored** discriminator wins over a **derived** one, and the disagreement is reported. Jupiter AMM v6 stores `5368c096` for `create_idempotent_associated_token_account` where Anchor's rule derives `bdbcecfa`; five real payloads carry `5368c096` and re-encode byte-exactly. The stored value is the one that survives contact with the chain.
1283
- - Every reader of the store re-hashes the object against its receipt. The consumer's multi-program loader did not at first, and its acceptance harness caught the difference: a tampered Jupiter object (valid JSON, one instruction renamed) was **accepted** by the multi path while the single path refused the identical damage. One store with two readers must verify with one strength.
1284
-
1285
- ### The producer's own reader was the weak one (measured 2026-10-30)
1286
-
1287
- The sentence above was true of `solana-tx-remix` and false of `sememe`. A probe against `sememe`'s own reader (`src/decode.mjs`, `findStoredIdl`) built a store from the committed Whirlpool snapshot, renamed one instruction in the object — still valid JSON, still a decodable IDL — and decoded a **real mainnet `swap_v2` payload** against it:
1288
-
1289
- ```
1290
- INTACT decode -> swap_v2 | fingerprint sha256:0b886e3...
1291
- TAMPERED decode-> totally_honest_name | roundTripByteExact true
1292
- ```
1293
-
1294
- The reader took the receipt's fingerprint as a label, printed it as the provenance of bytes it had never checked, and reported the tampered name as a decode result. `roundTripByteExact: true` is the sting: the damage was *only* in the name, so every byte-level check passed. The consumer refused the identical damage; the library that produced the store accepted it.
1295
-
1296
- Four checks now stand between a receipt and a decode (`src/decode.mjs`, `findStoredIdl`):
1297
-
1298
- 1. the receipt's `program` field must be the program asked for (a receipt swapped in from another program's file is refused by name);
1299
- 2. the receipt's fingerprint must be a well-formed sha256;
1300
- 3. the object must exist, parse as JSON, and carry an instruction list;
1301
- 4. `sha256(canonicalJson(idl) + '\n')` must equal the claimed fingerprint — **the check that was missing**, plus a separate refusal when the object's own `address` names another program.
1302
-
1303
- Every one of the five is pinned by a test that asserts the refusal *and its reason*, in `test/store-integrity.test.mjs`. Each was falsified by mutation: removing the re-hash check, the program-binding check, the missing-object message, the address check or the unparsable-JSON message turns exactly one of them red, and restoring it turns it green. The pre-fix behaviour is reproducible from the same test file by deleting the re-hash block.
1304
-
1305
- The CLI now refuses too: `sememe decode` on a tampered store exits 1 with `object and receipt disagree, refusing to decode`, naming both hashes. Before the fix the same command printed the tampered instruction name and exited 0.
1306
-
1307
- The falsification is a committed control: `solana-tx-remix`'s `neg/idl-name-stored-only.ts` relabels the newly-named instruction to a name the stale vendored copy carries, and the verifier answers `bytes resolve to create_idempotent_associated_token_account ... recipe claims create_token_account` — exit 1.
1308
-
1309
- ## Native programs: the two every transaction carries, pinned from the runtime (2026-10-31)
1310
-
1311
- System and ComputeBudget have no IDL account to pull — their interface lives in the Solana runtime itself. Sememe decoded neither; its first consumer (`solana-tx-remix`) carried a hand-transcribed copy of both instruction tables instead, which is exactly the per-repository hand-build this repository exists to replace. The cost of that copy was not hypothetical, it was measured: over a real mainnet harvest, **the hand table could name 0 of 63 on-chain `CreateAccountWithSeed` payloads**, because it reads a 4-byte string length header while the chain carries a u64 (bincode fixint) length word — every later field sat 4 bytes off, so the payload never matched the declared shape.
1312
-
1313
- `src/native.mjs` now decodes both programs from a pinned interface, with the provenance chain written into the file: the upstream enum (anza-xyz/solana-sdk, pinned by commit), the runtime's acceptance rule (anza-xyz/agave: borsh `try_from_slice_unchecked`, where an unclaimed tag is `InvalidInstructionData`), and the corroboration (`@solana/web3.js`'s own layouts, read as the same u64 length word). Three facts the pins settle and a hand copy is free to get wrong:
1314
-
1315
- - **Tag 0 of ComputeBudget is wire-rejected.** The runtime refuses the reserved `Unused` variant; `@solana/web3.js@1.98.4` still *encodes* legacy `requestUnits` as tag 0 + (u32, u32). This decoder refuses tag 0 with `wire-rejected` instead of copying the SDK's opinion.
1316
- - **System has 14 variants, not 12.** `UpgradeNonceAccount` (12) and `CreateAccountAllowPrefund` (13) exist upstream; a 12-entry table answers `null` on those bytes.
1317
- - **The string header is u64.** web3.js writes it as u32 length + u32 zero padding; read as one word, that *is* the bincode fixint length. The defect was dropping the padding word, not a layout disagreement.
1318
-
1319
- Measured against real mainnet traffic (1,828 System payloads + 1,348 ComputeBudget payloads, harvested 2026-10-31, [reports](research/native-coverage-system-2026-10-31.json)):
1320
-
1321
- | decoder | System named | ComputeBudget named |
1322
- |---|---|---|
1323
- | hand-written table (the one being replaced) | 1,765 / 1,828 | 1,348 / 1,348 |
1324
- | sememe pinned table | **1,828 / 1,828, all byte-exact re-encodes** | **1,348 / 1,348 (1,321 byte-exact)** |
1325
-
1326
- The 27 ComputeBudget payloads that are not byte-exact all carry trailing bytes after the declared argument. Chain adjudication (9 fetched directly, byte-identical to the harvest) shows the bytes are real and the runtime tolerates the remainder — every one sits in a transaction that failed at a *later* instruction. They decode with `trailing: { count: n }` reported, and are refused under `--trailing error`, the same contract the IDL decoder uses.
1327
-
1328
- Falsification: the new test file (`test/native.test.mjs`, 10 tests) was run against the old hand-written table, adapted to the same API surface with the table itself verbatim: **7/10 fail, 3/10 pass, and the three that pass are the designed ones** — the false-positive guard (a System `Transfer`, the one case the old table decoded correctly), the dependency-free import check, and the report test. Against the pre-module tree the file fails outright. A committed coverage report is re-derived from the committed fixture on every test run, so hand-editing the numbers goes red.
1329
-
1330
- CLI: `sememe decode-native <program> <payload>` decodes with the pinned interface; `sememe native-provenance [program]` prints the pinned sources, the encoding rules and the measured traffic.
1331
-
1332
- ### The token tables are derived too (2026-11-02)
1333
-
1334
- The same argument applies to the two token programs, and the stake is higher:
1335
- **SPL Token and Token-2022 publish no IDL account at either pullable address**,
1336
- yet they carry 43% of real watched mainnet traffic. Their interface is the
1337
- upstream Rust interface crate, and their wire rule lives in hand-written
1338
- `pack()`/`unpack()` pairs rather than in a Borsh enum — so the derivation reads
1339
- upstream's own text:
1340
-
1341
- - `vendor/spl-token/<commit>/…` and `vendor/spl-token-2022/<commit>/…` carry the pinned
1342
- interface files (Apache-2.0, LICENSE included) of `solana-program/token@737788ff` and
1343
- `solana-program/token-2022@5251dcc0`; their sha256 is the pin.
1344
- - `src/token-interface.mjs` derives each variant's wire shape from the upstream TEXT: the
1345
- tag is the integer upstream `pack()` writes and `unpack()` matches (explicit tags, so
1346
- `WithdrawExcessLamports` is 38 and `UnwrapLamports` 45 — the declaration index would
1347
- say 25 and 26), the arg sequence is the pack arm's byte-write sequence, and the
1348
- `COption` encoding is upstream's own `pack_pubkey_option`/`pack_u64_option` (1 tag byte
1349
- + payload, not a bincode option wrapper). It refuses on any surprise: a tag in only one
1350
- of the two arms, an upstream byte write no pinned rule covers, a field upstream does not
1351
- declare, a variant upstream declares that this table does not name. The variant list
1352
- itself is derived from upstream's enum — so a new upstream instruction fails loudly
1353
- instead of decoding as "unknown".
1354
- - `tools/token-interface.mjs` writes `interface/token-<program>.json` (28 variants for SPL
1355
- Token, 48 for Token-2022); `--check` fails when the committed artifact differs from a
1356
- fresh derivation.
1357
- - `src/token.mjs` RE-DERIVES at load and refuses to decode unless the committed artifact
1358
- equals the derivation byte for byte — the same two-check contract as the native tables.
1359
- It ships as `sememe/token`; the CLI is `sememe decode-token <program> <payload>` and
1360
- `sememe token-provenance [program]`.
1361
-
1362
- Measured on the unbiased traffic sample (four contiguous finalized blocks, 12,572 watched
1363
- payloads): the two token programs' 5,402 payloads went from **0 decoded** (no interface
1364
- acquirable at all) to **5,401 decoded**, one refusal (a `spl-token-metadata` fallback
1365
- instruction — a different upstream crate, refused by name rather than force-fitted) and
1366
- 422 payloads with explained trailing bytes. Overall coverage on that sample moved from
1367
- **55.5% to 98.5%**; the denominator never moved, the product gained a decode path.
1368
- (A later step — adopting Jupiter Lend's published Liquidity IDL, see below — moved
1369
- it to 98.6%; the Raydium AMM v4 derivation, also below, took no-interface to 0. The
1370
- token-metadata fallback, one step further, closed the last token refusal.)
1371
-
1372
- Falsification (`test/token-interface.test.mjs`, 8 tests, each a mutation in a temp copy of
1373
- the repo): swap two args in the artifact → load refused; edit the vendored Rust → load
1374
- refused with both sha256s named; delete `vendor/spl-token-2022` → load refused. The
1375
- measured-wire test pins the real numbers (a 1,000,000-amount Transfer, a TransferChecked
1376
- amount-then-decimals pair, a SetAuthority byte that decodes to `MintTokens`) and pins the
1377
- refusal reasons for an unknown tag, a truncated payload and an invalid option tag.
1378
-
1379
- Limits: Token-2022's 15 extension dispatch tags are named at the top level only — their
1380
- sub-instruction bytes report as trailing, because the extension crates are separate
1381
- interfaces this table does not vendor. Upstream's `Batch` (tag 255) unpacks but
1382
- the processor refuses it — the table records the wire shape, not a runtime guarantee.
1383
- (The `spl-token-metadata` fallback crate IS vendored now — see the next section.)
1384
-
1385
- Limits: Linux-mainnet measured only; the strict-20 coverage study does not include the native programs (they were never part of the sample, so no coverage number moved); the runtime acceptance rule is checked against the agave source at one point in time, not against a live validator; no outside user yet.
1386
-
1387
- ### The rows are derived, not typed (2026-10-31, later the same day)
1388
-
1389
- Even a pinned, measured table is still a table someone typed — and this repository's whole
1390
- premise is that a copy that can drift is the defect. The native tables were the last
1391
- hand-transcribed rows in sememe. They are now DERIVED:
1392
-
1393
- - `vendor/anza-solana-sdk/<commit>/…` carries the two pinned interface files from upstream
1394
- (Apache-2.0, LICENSE included), fetched at the pinned commit; their sha256 is the pin.
1395
- - `src/native-interface.mjs` parses each pinned `pub enum` — declaration order is the wire
1396
- selector, struct-variant field order is the wire order, the tuple type is the arg type —
1397
- and refuses on any surprise (wrong variant count, unknown type, duplicate name, an
1398
- undeclared unit variant). It is dependency-free like the rest of the decode core.
1399
- - `tools/native-interface.mjs` writes the derived tables to `interface/native-<program>.json`;
1400
- `--check` fails when the committed artifact differs from a fresh derivation.
1401
- - `src/native.mjs` RE-DERIVES at load and refuses to decode unless the committed artifact
1402
- equals the derivation byte for byte. Two callers, one derivation: an edited artifact and
1403
- a swapped vendored file both fail loudly at import, before a single byte is decoded.
1404
-
1405
- Falsification (`test/native-interface.test.mjs`, 7 tests, each a mutation in a temp copy of
1406
- the repo): swap two rows in the artifact → load refused; change `SetComputeUnitLimit(u32)`
1407
- to `(u64)` in the vendored Rust → load refused with both sha256s named; delete `vendor/` →
1408
- load refused. Removing the load-time equality check turns the first two into silent passes.
1409
- A separate pin asserts the three argument names a hand copy actually got wrong upstream —
1410
- `micro_lamports`, `from_seed`, `from_owner` (the in-flight camelCase transcription had all
1411
- three).
1412
-
1413
- The consumer's copy is gone too: `solana-tx-remix/src/native.mjs` is now a thin facade over
1414
- this module (its own hand table removed), and its acceptance suite gained check 7, which
1415
- re-measures every defect the hand table had against committed traffic — 63/63 seeded creates
1416
- named and byte-exact (hand table: 0/63), variants 12/13 nameable, tag 0 refused, upstream
1417
- argument names — plus a run of `native-interface.mjs --check`.
1418
-
1419
- ## Harvesting from the local mailbox instead of RPC (2026-10-31)
1420
-
1421
- `tools/harvest-reeflog.mjs` writes the same document `tools/harvest.mjs` writes, but reads the instruction bytes from a local [reeflog](https://github.com/auroraxo/reeflog) mailbox (its account envelope carries `instructions[]` verbatim — program ids resolved against the canonical account list, data base58 as served) instead of paying `getSignaturesForAddress`/`getBlock`/`getTransaction`. The source is in the document (`rpc: none (local reeflog journal)`), and the row accounting keeps the debts the network harvester does not have: rows without an envelope and envelopes without instructions are counted separately, never dropped, and a journal with zero enriched rows exits non-zero instead of writing a zero-payload success document.
1422
-
1423
- **Zero RPC is enforced, not asserted:** `tools/lib/network-guard.mjs` replaces `globalThis.fetch` with a function that records the attempt and rejects; the harvester imports it *first* (pin: `test/harvest-reeflog.test.mjs` fails if the import order changes), and the run exits 1 if `guard.attempts` is ever non-empty.
1424
-
1425
- **Live proof** (`tools/e2e-reeflog-harvest.mjs`, report: [research/reeflog-harvest-e2e-report.json](research/reeflog-harvest-e2e-report.json)): a fresh 45-second mainnet window for Whirlpool — 456 journal events, 452 account envelopes held — harvested for **156 payloads across 137 transactions with 0 network attempts**. Two directions checked against independently fetched chain transactions: **10/10 sampled payload-bearing transactions byte-identical** (`JSON.stringify` of the payload records, same order, same key order), and **10/10 sampled no-target transactions corroborated** (the chain shows no Whirlpool instruction either — a real contradiction there would mean the extractor drops bytes, and it is a named check). 0 different, 0 unavailable.
1426
-
1427
- The equivalence is pinned by mutation, twice: swapping `dataBase58`/`dataHex` in `payloadsFromEnvelope` turns 2 unit tests red, and turns the live run red at `mailbox_bytes_equal_chain_bytes` (8 of 10 sampled different). A third defect this harness caught before any of it: the first version of the e2e requested `maxSupportedTransactionVersion: 0`, which rejects every v1 transaction with rpc error -32015 — the retry loop misreported those as rate limiting (2/12 sampled), while `harvest.mjs` has requested version 1 all along. The harness now reports permanent client errors as what they are.
1428
-
1429
- **Limits:** the mailbox is my own reeflog daemon, not a third-party producer; the sample is 10 + 10 transactions from one 45-second window; the envelope is fetched by reeflog's own `hydrate --accounts` pass (the cost moved to the mailbox, it did not disappear).
1430
-
1431
- ## Tenth program: the Jupiter RFQ stack, and the first live "the wire changed shape" firing (2026-11-01)
1432
-
1433
- The suite already said what would count as a serious finding: in `test/decode.test.mjs` the unexplained-bytes guard reads *"a non-zero tail would mean the IDL is missing a real field, which is a different and more serious finding than padding."* Until now every observed tail was zero padding. **The Jupiter RFQ stack now fires that exact case — and the first measurement of it was wrong in a way worth recording.**
1434
-
1435
- **`rfq_v2` (`fd3nMFYTQjX1yr5ER8u7tPdHJB7qt8RpDpNtLQX2Br5`) — the path is whole.** 40 payloads across slots 463374166–463994162 (21,179 pinned transactions, 0 missing): **40/40 named `fill_exact_in`, 40/40 byte-exact round-trip, 0 payloads with a trailing byte.**
1436
-
1437
- **`order_engine` (`2En5Y11SEAGLNmEezTuRUCwTyzyNReHaMbSnS5gjGsL1`) — the live wire has two forms, and only one is described anywhere.**
1438
-
1439
- | wire form | payloads | described bytes | trailing | full round-trip |
1440
- |---|---|---|---|---|
1441
- | 32 bytes | 82 | 32/32 | 0 | 82/82 byte-exact |
1442
- | 40 bytes | 59 | 32/32 (the described prefix) | **8, undeclared** | 0/59 |
1443
-
1444
- Both forms are `fill`, both are accepted by the same deployed program, and they **coexisted** in the overlapping window (slots 463951326–463985613). After the last 32-byte payload (slot 463985613) the wire carries **only** the 40-byte form — 45 consecutive payloads.
1445
-
1446
- The 8 extra bytes are structured, not padding: **a constant 4-byte tag `0x06942069` (little-endian, present in 59/59) plus 4 varying bytes that are pairwise distinct across all 59 payloads**. The described 32-byte prefix re-encodes byte-exact on every single one, so this is not a decoder bug on the three arguments the sources do describe.
1447
-
1448
- Why the gap is real and not stale-pull noise:
1449
-
1450
- - The program's **own on-chain IDL account** (`GJwPa6DAFzYKNddAgxwzKzuDgnE4NeaWkKPN9FFqkmkb`, owner = the program itself) declares `fill` with **three** arguments (`input_amount: u64`, `output_amount: u64`, `expire_at: i64` = 24 bytes). A fresh `sememe pull` at slot 464073733 returns **the same fingerprint** (`sha256:f13a31e6…`) as the receipt pinned at slot 463089384 — the description did not change.
1451
- - The **first-party source** ([jup-ag/rfq-webhook-toolkit](https://github.com/jup-ag/rfq-webhook-toolkit) @ `3aa4fce`) declares the same three arguments in `programs/order-engine/src/lib.rs`, and `git log --all -S` finds no history with a fourth.
1452
- - The **vendored copy** in `solana-tx-remix/idl/multi/jupiter-order_engine/order_engine.json` has the identical single instruction and the identical discriminator `[168,96,183,163,92,10,40,160]`.
1453
- - The bytes are **accepted**: all 141 payload-bearing transactions succeeded on chain (`txErr: null`).
1454
- - Hypotheses tested and **rejected**: tail equals a declared argument (BE/LE), equals `expire_at` (raw or ×1000), equals the slot (raw or ×1000), equals a window of the transaction signature, or is a price-like ratio of `output_amount`/`input_amount` (spread ~10×). The varying 4 bytes are client-emitted and not derivable from the payload, the signature, the slot, or the declared args.
1455
- - **The rejection is now reproducible, not remembered.** `tools/tail-hypotheses.mjs` re-derives every count from the committed fixture ([research/rfq-order-engine-tail-2026-11-01.json](research/rfq-order-engine-tail-2026-11-01.json), human-readable table in [research/rfq-order-engine-tail-2026-11-01.md](research/rfq-order-engine-tail-2026-11-01.md)). New and stronger this round: the **entire affine class** `tail ≡ A·base + B (mod 2³²)` is refuted for base ∈ {slot, expire_at, input_amount, output_amount} — A and B are derived from the first two payloads with an invertible delta, never fitted to the verdict. One refutation covers the constant offset, every seconds→milliseconds→microseconds rescaling, and every scaling of a declared argument. The SHA-256-prefix and CRC-32 checksum interpretations are refuted per-payload as well. Two measured bands constrain what remains: the varying u32 (LE) never uses the u32 space — its top byte is only `0xAC`/`0xAD` across all 59 payloads (16 high bits confined to [0xACAB, 0xADCC], ~0.004% of the space) while the low 16 bits are fully distinct; and the varying value does not correlate with slot, amounts, or expire_at (|r| ≤ 0.17). The field stays **unidentified**: whatever it is, it is not a single word carried in this instruction, and the strongest surviving hypothesis is a value derived from state outside the instruction bytes (the fixture does not pin the surrounding transaction), or a field of an interface revision whose description was never published.
1456
-
1457
- **The correction, stated plainly.** A first, narrower 30-block harvest saw only the 40-byte form (32 payloads, all with undeclared bytes) and I concluded *"every live fill carries 8 undeclared bytes"*. A 140-block scan **falsified that**: the 32-byte form is real and ran alongside the 40-byte form. The committed fixture is the wide scan, and every number above is derived from it. The difference matters for the product — the finding is not "the decoder misses a field", it is **"the deployed program's wire changed shape and no interface description followed"**. A decoder built only from the published interfaces now refuses 100% of the recent live traffic on this program.
1458
-
1459
- **Both remaining hypotheses were refuted on chain (2026-11-01), and the finding's shape changed.** The 14:00 study left two explanations standing: a value derived from state outside the instruction bytes, or a field of an interface revision that was never published. Two read-only probes (public RPC, every fetch counted in the reports) closed them:
1460
-
1461
- - **No published revision.** A fresh pull of *both* published IDL surfaces — the canonical `anchor:idl` account and the program-owned metadata account — reproduces the receipt fingerprint `sha256:f13a31e6…` (measured at slot 464182250). Nothing on chain declares a 4th argument, so "a revision nobody published" cannot explain bytes that are on the wire **now**, in every transaction.
1462
- - **Nothing outside the instruction carries it.** For 6 fills spread across the slot range: **0** matches of the varying u32 (or the assembled `tag<<32|word` u64) in the transaction's balance surface (pre/post lamport and token balances, raw and `mod 2³²`), **0** in any sibling instruction's bytes (outer + inner), **0** in every touched account's data (`getMultipleAccounts` over the transaction's own account list), and **0** in the program's own state accounts (`getProgramAccounts`: 1 account, 1078 bytes — the compressed IDL itself; the program is stateless).
1463
- - **The attribution that decides it:** every one of the **59** long-form fills was attributed from the chain to counterparty `MfDuWeqS…`, and every one of the **82** short-form fills to a *different* counterparty, `7rhxnLV8C7…` — **141/141 resolved, zero overlap** ([research/order-engine-counterparty-2026-11-01.json](research/order-engine-counterparty-2026-11-01.json)). The 8 undeclared bytes are **one counterparty's own extension of a shared protocol instruction**, not a protocol change and not protocol state.
1464
-
1465
- **Who that counterparty is, measured (2026-11-02).** "Counterparty = the `maker` account" was an inference from the instruction's account list; two premises behind it were still unchecked, and both now have chain measurements ([research/order-engine-maker-identity-2026-11-02.json](research/order-engine-maker-identity-2026-11-02.json), `tools/order-engine-maker-identity.mjs`, all 141 fixture transactions re-fetched, payloads byte-identical to the fixture 141/141):
1466
-
1467
- - **The makers are key-holding clients.** Both maker addresses are on-curve (an exact ed25519 decompression test — an off-curve address *cannot* hold a key, which would have made "client" wrong), system-owned, carry 0 bytes of data, and **sign** their transaction (141/141). The takers sign too (141/141). The maker never pays the fee: the fee payer is the taker in 110 transactions and the undeclared co-signer in 31 (107 distinct fee payers — not one service wallet).
1468
- - **The account list carries more than the IDL declares.** The IDL declares **11** accounts for `fill`; the wire carries **12** (52 transactions) or **13** (89 transactions) — never 11. The extra accounts are constant in role: one variable slot (the order_engine program itself, or an off-curve derived address) plus `pAEmG7…`, which appears in **141/141** transactions and **signs every one of them** — a co-signing party the IDL does not mention at all. An account list is a carrier of meaning on Solana, and this channel was unmeasured until now.
1469
- - **The limit that travels with the verdict:** three parties sign each fill (taker, maker, co-signer), so *which* of them constructed the 8-byte extension is **not** proven by the correlation with `MfDuWeqS…` — the builder among the three signers is explicitly not claimed. The 4 varying bytes still have no identified meaning: sha256 of the referenced account, sha256 of the signature, the slot, both amounts, `expire_at` and the account's address boundaries were all tested against the 59 payloads and refuted (0/59 hits).
1470
-
1471
- So the honest headline is narrower than "the wire changed shape": the deployed program accepts a counterparty-appended tail that no interface source describes. `trailing.count` remains the only place this surfaces for a decoder built from published interfaces, and it must not be normalized away.
1472
-
1473
- **What this changes in the product.** "Unexplained bytes" used to mean padding, and permissive `trailing: 'report'` was a courtesy. It is now a **live-drift signal**: `trailing.count` is the only place this surfaces, and it must not be normalized away. Two behaviours are pinned as measured, not as desired — permissive mode keeps **serving** both forms (the chain accepted them; refusing would be a decoder defect, not a safety property), and strict mode **refuses only** the 40-byte form instead of pretending a field exists that no source declares.
1474
-
1475
- The committed fixtures (`test/fixtures/harvest-{rfq-v2,order-engine}-2026-11-01.json` plus the two on-chain IDL receipts) make this re-measurable: `test/rfq-stack.test.mjs` (8 tests) re-derives every number above from the fixtures. If a future interface description accounts for the extra field, `trailing` drops to 0 and the test goes **red**, so the number gets re-measured instead of going quietly stale. The counter-test pins the fixture itself: editing the committed IDL to declare a 4th `u64` argument fails the suite, because "the interface now covers 40 bytes" is a **new claim that needs new evidence** (a re-pull of the on-chain account), not a fixture edit.
1476
-
1477
- **Limits:** 141 payloads from one program id in one slot range; the extra field's *name and type* are still unknown — what is measured is that 8 structured bytes exist, are stable per payload, and are undeclared by every available interface source, and that the varying word is drawn from a narrow band that no interpretation recorded so far produces. The upstream SDK decodes `fill` with 3 args, so if a 4th field exists it is one the SDK also does not name.
1478
-
1479
- ### Reported to the first party (2026-11-02): jup-ag/rfq-webhook-toolkit#96
1480
-
1481
- Both wire findings above were delivered to the first-party repository as [issue #96](https://github.com/jup-ag/rfq-webhook-toolkit/issues/96): the undeclared co-signer account (present in 141/141 fills, signs all of them, pays the fee in 31/141) and the 40-byte form with its constant `0x06942069` tail tag. The issue carries one-click reproductions for each claim (fill tx `3iTpfkc7…` for the account list, `3cQfqyqaD…` for the payload), the fixture and report paths, and the limits stated as limits: which of the three signers builds the 8 bytes is not proven, the 4 varying bytes stay unidentified, and the 19 refuted interpretations remain named. The earlier, separate stale-IDL address finding is tracked as [#95](https://github.com/jup-ag/rfq-webhook-toolkit/issues/95).
1482
-
1483
- This is delivery, not adoption: no response exists yet, and neither issue makes sememe's first outside user — that threshold stays "a third party imports the artifact and 1,000 transactions decode without disagreement."
1484
-
1485
- ## Eleventh program, third wire form: cp-amm's `swap` accepts three shapes (2026-11-02)
1486
-
1487
- The RFQ finding above is a program whose wire changed shape with **no** interface following. cp-amm is the opposite shape of problem, and it resolves the residual that the 30.10. traffic-coverage note left open: that note said the deployed `swap` *"reads further than its own interface file"*. **That sentence was wrong in an important way, and the measurement says so.**
1488
-
1489
- **The interface describes exactly one shape.** The program's own on-chain IDL account (`anchor:idl` @ `F1hSvutXSc9iAvB9bQPmjNDwiN2tCXzazTEEvDgNCBMa`) declares `swap` with one argument, `SwapParameters { amount_in: u64, minimum_amount_out: u64 }` — 8-byte discriminator + 16 bytes = **24 bytes, the only length it can describe**. The upstream source agrees ([MeteoraAg/damm-v2](https://github.com/MeteoraAg/damm-v2) @ `5398b5c`, `programs/cp-amm/src/instructions/swap/ix_swap.rs`): `pub struct SwapParameters { pub amount_in: u64, pub minimum_amount_out: u64 }`.
1490
-
1491
- **The chain carries three shapes.**
1492
-
1493
- | wire form | bytes | described by the IDL | trailing | accepted on-chain |
1494
- |---|---|---|---|---|
1495
- | 24 B | 477 (30.10.) / 284 (02.11.) | 24/24 | 0 | yes |
1496
- | 25 B | 4 / 3 | 24/24 (the described prefix) | **1 zero byte, undeclared** | yes, 7/7 |
1497
- | 32 B | 2 / 0 | 24/24 (the described prefix) | **8 zero bytes, undeclared** | yes, 2/2 |
1498
-
1499
- Measured over two independent mainnet windows — the committed harvest of 30.10. (slots 463380931..463380978, 48 blocks, 1,894 payloads) and a **fresh harvest of 02.11.** (slots 464398786..464398809, 24 blocks, 5,633 transactions fetched, 0 missing, 1,001 payloads) — so the wide form is not a one-window artifact: it appears in both, three days apart, and **still appears today**.
1500
-
1501
- **The extra bytes are not read, they are tolerated — and the program's own source proves it.** The deployed program does not use Anchor's dispatcher; it has a hand-written fast entrypoint (`programs/cp-amm/src/entrypoint.rs`) that does:
1502
-
1503
- ```rust
1504
- let swap_parameters = unwrap_or_return!(
1505
- SwapParameters::deserialize(
1506
- &mut &instruction_data[crate::instruction::Swap::DISCRIMINATOR.len()..]
1507
- )
1508
- );
1509
- ```
1510
-
1511
- `deserialize` here is borsh's **slice** reader: it reads the two `u64` it needs and leaves the rest of the buffer alone. Anchor's own `deserialize` has the same property, which is why sememe's permissive `trailing: 'report'` mode exists at all. So the deployed program does **not** parse bytes past its own struct — it accepts a payload with bytes appended. The 30.10. wording ("reads further than its own interface file") overstated it, and the correction matters for the product: **an over-long payload is not evidence of a missing field**, and a decoder must not invent one. A missing field would carry information; these bytes carry none.
1512
-
1513
- **The 25-byte form is a `swap2` body under the `swap` discriminator.** Its body is **byte-identical** to what the interface's *other* instruction would encode: `swap2` takes `SwapParameters2 { amount_0: u64, amount_1: u64, swap_mode: u8 }`, and re-encoding the same two amounts through `swap2` with `swap_mode = 0` reproduces the 25-byte body exactly — verified on every 25-byte payload, in both windows. The 32-byte form is the same idea with 8 bytes of zero padding (the trailing bytes are **zero in 9/9** wide payloads — padding, not a field). So a client that switched from `swap` to `swap2` and kept sending under the old discriminator produces exactly these shapes.
1514
-
1515
- **Where they come from, measured on chain** ([research/cp-amm-wire-forms-2026-11-02.json](research/cp-amm-wire-forms-2026-11-02.json), `tools/cp-amm-wire-forms.mjs`, every fetch counted, 0 failures):
1516
-
1517
- - **9 wide payloads, 12 IDL-exact controls** — a control group, so "the wide form comes from somewhere else" is a contrast and not a sample of one.
1518
- - **8 of 9 wide payloads are top-level** (8 of 12 controls likewise) — the wide form is **not** a CPI-only artifact; the same mix appears in both groups. One wide payload is a CPI, invoked by `4X3dGCFh63SyE2bNZdGUV7Rj5ukK1XgrxesoxyKQNiRN`.
1519
- - **7 distinct fee payers among 9 wide payloads** — not one service wallet or one bot; and **zero overlap** with the controls' fee payers.
1520
- - **9/9 wide payloads were accepted on-chain** (`txErr: null`), against 8 of 12 controls that failed on-chain — which is the repo's own rule from the RFQ section: *decode refusal is evidence about the IDL, not about whether the program executed*, applied in the other direction here.
1521
- - The wide payloads carry the **declared 14 accounts** (top-level) or 18 (the CPI case) — the account list is not the carrier.
1522
-
1523
- **The interface has not moved.** A fresh `sememe pull` of the program's on-chain IDL today (slot 464399566) returns fingerprint `sha256:e80f49b1…` — **byte-identical** to the fingerprint of the fixture pulled on 30.10. So "the published interface is stale" is refuted by a measurement, not assumed; the description has not changed in three days while the chain kept sending all three shapes.
1524
-
1525
- **What this changes for sememe.** The residual is now a named, reproducible *tolerance* finding instead of an open question:
1526
-
1527
- - The permissive mode is **right** for this case, and now for a measured reason: the program accepts these payloads, so refusing them would be a decoder defect. `trailing.count > 0` stays the place where the fact surfaces.
1528
- - The **strict** mode refuses only the wide forms — and refuses them *as undeclared*, not by inventing a field for the extra bytes. The test pins exactly that: the 25-byte body must equal a `swap2` body, and every trailing byte must be zero.
1529
- - A consumer who wants the narrower meaning can read the 25-byte form as `swap2`/`swap_mode = 0` — that is a **client-side interpretation** the interface does not license, and sememe does not perform it silently.
1530
-
1531
- **Falsification.** `test/cp-amm-wire-forms.test.mjs` (6 tests) re-derives every number above from the two committed fixtures and the committed IDL, and pins the observed wire-length set as exactly `{24, 25, 32}` / `{24, 25}` so a new shape goes red instead of widening the finding silently. The committed report is cross-checked against the fixtures (no hand-edited numbers). Mutation ladder, each mutation red on its own check: editing the report's wide count (1 fail), making one trailing byte non-zero (2 fail), marking a wide payload as failed on-chain (1 fail), introducing a new wire length 26 (3 fail) — restored tree green 6/6. Suite 244 → **250/250**.
1532
-
1533
- **Limits.** Two windows and one program id; 9 wide payloads total, so the *frequency* is not established (1.24% of `swap` payloads in 30.10., 1.05% in 02.11. — small samples on both sides). The 32-byte form did not appear in the fresh window, so it may be rarer now or slot-bound; that is stated, not resolved. The claim that the extra bytes are *tolerated* rests on the upstream source of the deployed program at one commit (`5398b5c`) — the deployed **binary** was not disassembled, and the on-chain IDL is not proof of what the binary does. No third party is involved; this is a measurement about a program's wire, not an adoption claim.
1534
-
1535
- ## Coverage measured on unbiased traffic (2026-11-02)
1536
-
1537
- Every qualification above measured programs that were CHOSEN — usually programs
1538
- already in the store. The sample and the store therefore shared a selection
1539
- bias, and a favorable fraction measured against them is not a coverage number.
1540
- This section measures the other way round.
1541
-
1542
- **The sample.** `tools/traffic-sample.mjs` takes the newest finalized mainnet
1543
- blocks (four contiguous slots, 2,214 transactions) and inventories EVERY
1544
- instruction in them, without asking the store anything. Programs outside the
1545
- watched list are counted but their bytes are not stored (fixture size limit), so
1546
- the denominator cannot silently shrink. The fixture is committed
1547
- (`test/fixtures/traffic-sample-2026-11-02.json`, captured 2026-11-02T15:50Z),
1548
- together with a store built for exactly this sample
1549
- (`test/fixtures/traffic-sample-store/`: ten on-chain pulls plus four adopted
1550
- first-party published copies).
1551
-
1552
- **The measurement.** `tools/traffic-coverage-report.mjs` runs every payload
1553
- through the product's own entry points — `decodeUnionPayload`, `decodePayload`,
1554
- and the native, token and Raydium decoders — and asks one question per payload:
1555
- could sememe have NAMED it? The four outcomes are kept apart on purpose.
1556
-
1557
- **First run, same day, same sample (kept for the record):** decoded 6,983,
1558
- refused 171, no-interface 5,418 — 55.5% coverage, with 99.7% of the gap on SPL
1559
- Token + Token-2022, which publish no IDL account at either pullable address.
1560
- That measurement named the fix, and the fix shipped the same day: the two token
1561
- wire tables are now DERIVED from the pinned upstream Rust crates
1562
- (`solana-program/token@737788ff`, `solana-program/token-2022@5251dcc0`, both
1563
- Apache-2.0, both vendored with license + sha256 pin) — the System/ComputeBudget
1564
- pattern, one level up: the wire rule lives in the crates' hand-written pack()
1565
- and unpack() pairs, so the derivation reads the pack arm's byte-write sequence
1566
- cross-checked against the unpack arm's tag set, and refuses any upstream write
1567
- it cannot pin. The decoder ships as `sememe/token` (artifact + load-time
1568
- derivation check, same two-check contract as the native tables).
1569
-
1570
- | outcome (current) | count | meaning |
1571
- | --- | --- | --- |
1572
- | decoded | 12,401 | an available interface named it |
1573
- | refused | 171 | an interface exists and the bytes were refused (source lag in the pump programs' wire forms) |
1574
- | no-interface | **0** | no payload in this sample lacks an interface |
1575
-
1576
- **98.6% coverage** (12,401/12,572 watched payloads), with 11,708/12,401 = 94.4%
1577
- of the decoded payloads re-encoding byte-exactly. What the 700 non-byte-exact
1578
- decodes are, measured: 419 legacy `GetAccountDataSize` payloads carry two extra
1579
- bytes the legacy program itself ignores (upstream's own comment says so), and
1580
- three Token-2022 extension dispatches carry their sub-instruction bytes, which
1581
- this table reports as trailing instead of inventing a field for them. The
1582
- trailing count is part of the result, not a failure hidden by it.
1583
-
1584
- **What the number means now.** The acquisition gap this measurement existed to
1585
- expose is **empty for this sample**: every watched payload now sits on a program
1586
- that has an interface sememe can decode against, so `acquisitionQueue` carries
1587
- no entry. "Refused" (171) is source lag in interfaces that exist — PumpSwap's
1588
- undeclared discriminator form and pump.fun's unexplained event tail are the bulk
1589
- of it, both already reported to the first parties. The denominator never moved:
1590
- same sample, same product entry points, one new decode path per gap.
1591
-
1592
- Empty is a property of these four blocks, not of the ecosystem. The coverage
1593
- illusion the card names is **measured, not removed**: one window, one RPC, no
1594
- outside user. A different window can put a program back in the queue, and the
1595
- queue is derived from the rows, so that would show up as a row first.
1596
-
1597
- **The union path is a coverage lever, not a decode nicety.** Adopting the
1598
- first-party published copy `pump-fun/pump-public-docs` `idl/pump_amm.json` moved
1599
- PumpSwap from 82.0% to 96.4% (487/505 decoded) in one step: its on-chain IDL is
1600
- stale, and the published copy covers 9 of the 10 discriminators seen in the
1601
- window. The tenth (`e445a52e51cb9a1d`, 226 payloads) is declared by neither
1602
- copy.
1603
-
1604
- **An adopted copy may now stand alone (2026-11-03).** The second half of the
1605
- acquisition gap was **Jupiter Liquidity** (`jupeiUmn…`, 7 payloads): no IDL at
1606
- either on-chain surface — measured at capture time, re-measured against live
1607
- mainnet at slot 464744781 (both `null`), so the only verified interface is a
1608
- published copy. Adopting the first-party
1609
- [`jup-ag/jupiter-lend`](https://github.com/jup-ag/jupiter-lend)
1610
- `target/idl/liquidity.json` at commit `0ab370a0` (26 instructions,
1611
- `sha256:8ed1e22661ab…`) into the committed fixture store closed it — **7/7
1612
- payloads decoded byte-exact through `decodeUnionPayload`**, and Jupiter
1613
- Liquidity left the acquisition queue.
1614
-
1615
- Adopting it alone was not enough, because of a defect in the reader:
1616
- `decodeUnionPayload` required a *primary* source, and a store with no on-chain
1617
- receipt for the program has none — so the union path refused with
1618
- `unionDecode needs a primary source` and reported "no interface" about a store
1619
- that held exactly one, while plain `decodePayload` on the same bytes answered
1620
- 7/7. Two entry points, two answers, and the coverage measurement used the wrong
1621
- one. **The rule now: an adopted copy may stand alone when there is no on-chain
1622
- pull to displace — it may still never displace one** (the on-chain receipt stays
1623
- primary whenever it exists, pinned by the existing M4 test). The verdict carries
1624
- `view.adoptedOnly: true` plus `idl.source` / `idl.slot: null`, so a consumer can
1625
- see that no chain slot answered. Refusals still refuse: a payload no stored
1626
- source explains is refused **by name**, with the adopted-only provenance
1627
- visible rather than laundered into a decode.
1628
-
1629
- Covered by `test/jupiter-liquidity-acquisition.test.mjs` (5 tests: the
1630
- capture-time verdict still says `acquirable: false`; the store still holds no
1631
- on-chain receipt; all 7 byte-exact with `adoptedOnly: true`; **delete the
1632
- adopted receipt and the gap comes back by name** — the number moved because an
1633
- interface was added, not a counter; the committed report and queue reflect the
1634
- closure) plus the adopted-only union tests in `test/union.test.mjs`.
1635
-
1636
- **The last gap closed by derivation: Raydium AMM v4 (2026-11-03).** The final
1637
- 9 payloads sat on a program that publishes no interface at all — no IDL account
1638
- either pullable address, measured at capture time and re-measured live. Unlike
1639
- Jupiter Liquidity there was no first-party copy to adopt: its wire is not an
1640
- Anchor 8-byte discriminator but a hand-written `pack()`/`unpack()` pair over a
1641
- **sparse u8 tag set** (19 declared variants, 12 reachable on the wire, 7 refused
1642
- by upstream with `unimplemented!()`). So the interface was derived from the
1643
- program's own source, exactly as the token tables were derived from their
1644
- crates: `vendor/raydium-amm@91739d8d` (Apache-2.0, license + sha256 pin on both
1645
- files), `src/raydium-interface.mjs` for the rules, `interface/raydium-<program>.json`
1646
- as the reviewable artifact, `src/raydium.mjs` for the two-check load and decode.
1647
-
1648
- The derivation reads upstream's own text rather than transcribing it: the wire
1649
- tag is the literal integer the `pack()` arm pushes, cross-checked against the
1650
- `unpack()` arm that binds that same integer; the args are the arm's byte-write
1651
- sequence, cross-checked against the reads by count and width; a trailing
1652
- optional group carries upstream's own `rest.len() >= N` threshold (**there is no
1653
- option byte**, so a C-style read would misparse every real payload); a branch
1654
- carries upstream's own discriminator values from the `AmmParams` enum. Because
1655
- the seven refused variants have no `pack()` arm to read a number from, their
1656
- tags come from their declaration position — which is checked against all twelve
1657
- reachable variants first, so an upstream reorder fails the derivation instead
1658
- of handing a refused tag a wrong name.
1659
-
1660
- Measured on the same sample, same entry points: **9/9 payloads decode and
1661
- re-encode byte-exact** (`SwapBaseIn` ×2, `SwapBaseInV2` ×7), `no-interface`
1662
- **9 → 0**, and `acquisitionQueue` is empty. The decoder ships as `sememe/raydium`
1663
- with `sememe decode-raydium` / `sememe raydium-provenance`, and `sememe decode`
1664
- on this program now points at it instead of telling the user to pull a receipt
1665
- that can never exist.
1666
-
1667
- Falsification (`test/raydium-interface.test.mjs`, 12 tests, mutations in temp
1668
- copies): a tampered artifact, a tampered `instruction.rs`, a tampered `state.rs`
1669
- and a missing `vendor/` all refuse the load; removing the load-time equality
1670
- check turns all four into silent passes (measured: 9 of 12 tests go red).
1671
- Refusals are pinned by reason — unknown tag, each refused variant, truncated
1672
- u64/pubkey/fees, unmatched discriminator, wildcard rejection — and the optional
1673
- group round-trips in both directions with half a group refused as
1674
- `partial-optional` rather than guessed.
1675
-
1676
- **The largest gap closed by derivation: SPL Associated Token Account
1677
- (2026-11-11).** The ecosystem measurement (`tools/ecosystem-coverage.mjs`, the
1678
- next section) put ATokenGPvbd… first in its acquisition queue: **1,410 payloads
1679
- in 6 finalized mainnet blocks, no IDL account at either pullable address** —
1680
- the single biggest acquisition gap the unbiased denominator exposed. Its
1681
- interface is upstream Rust again, and this time the wire rule is borsh's own:
1682
- `solana-program/associated-token-account@fbf5379` (Apache-2.0, LICENSE + sha256
1683
- pin on all three vendored files) declares `AssociatedTokenAccountInstruction`
1684
- as a borsh UNIT enum — Create, CreateIdempotent, RecoverNested — so the payload
1685
- is exactly the one-byte variant index and no argument bytes follow. The wire
1686
- bytes come from upstream's own builders (`data: vec![2], // …::RecoverNested`
1687
- and the `instruction` argument of `build_associated_token_account_instruction`),
1688
- cross-checked against the declaration index the borsh derive would use — both
1689
- must agree or the derivation refuses. The RUNTIME verdicts are read from the
1690
- pin's processor, not assumed: `input.is_empty()` is EXECUTED as `Create` (so
1691
- the 4 empty payloads the measurement saw are a named Create via the processor's
1692
- own fallback — honest about not being byte-exact, since the canonical encoding
1693
- is `[0]`), everything else is judged by `try_from_slice`, and an unjudged input
1694
- is `InvalidInstructionData` — this decoder refuses it under the same name.
1695
-
1696
- Measured on the committed 6-block sample, product entry points only:
1697
- **1,410/1,410 decode (1,406 byte-exact), `no-interface` 1,410 → 0**, ecosystem
1698
- coverage **75.6% → 82.2%** (watchlist 96.4% unchanged, outside-watchlist
1699
- **7.1% → 35.4%**), and `acquisitionQueue`'s head moved from ATA to pump L2
1700
- (L2TExMFK…, 1,214). A re-run of the 1,000-transaction foreign-archive wire
1701
- sweep decodes **5,032** (was 4,646) with zero disagreements. The decoder ships
1702
- as `sememe/ata` with `sememe decode-ata` / `sememe ata-provenance` (the empty
1703
- payload is real input here: `sememe decode-ata ATokenGPvbd… --empty`), and
1704
- `sememe decode` on this program points at it instead of telling the user to
1705
- pull a receipt that can never exist.
1706
-
1707
- ## The #1 remaining acquisition gap decodes: Lighthouse adopted from first-party (2026-11-11)
1708
-
1709
- Closing the SPL Associated Token Account left `L2TExMFKdjpN9kozasaurPirfHy9P8sbXoAN1qA3S95`
1710
- standing at the top of the acquisition queue: **1,214 payloads across the 6
1711
- committed blocks**, no on-chain IDL account at either address. But unlike ATA
1712
- (which publishes no IDL anywhere and needed a Rust wire table), Lighthouse is
1713
- the program whose **own repository publishes an IDL** (`Jac0xb/lighthouse`,
1714
- commit `4c579479c98635e419b1b167f08be02a71604a71`, MIT; `programs/lighthouse/lighthouse.json`,
1715
- 18 instructions in the [shank](https://github.com/metaplex-foundation/shank) dialect).
1716
-
1717
- `sememe adopt-lighthouse` ships that verified first-party copy and adopts it
1718
- through the product's one write path (`src/adopt.mjs`), with the vendored bytes
1719
- hash-checked against their pin BEFORE anything is written:
1720
-
1721
- ```bash
1722
- sememe adopt-lighthouse --store my-store
1723
- sememe decode L2TExMFKdjpN9kozasaurPirfHy9P8sbXoAN1qA3S95 0500000c6a0a0f0000000004 --store my-store --union
1724
- # -> AssertAccountInfo (Lamports >= 252340748), byte-exact re-encode, origin declared:
1725
- # github.com/Jac0xb/lighthouse@4c579479:programs/lighthouse/lighthouse.json
1726
- ```
1727
-
1728
- **Measured on the committed 6-block sample (1,214 payloads):**
1729
- - **1,214 / 1,214 decoded (100%)**
1730
- - **1,214 / 1,214 byte-exact re-encode (100%)**
1731
- - 0 disagreements, 0 refusals
1732
- - 4 instruction faces active: `AssertAccountInfo` (342), `AssertTokenAccount` (386), `AssertSysvarClock` (337), `AssertAccountData` (149)
1733
- - `no-interface` on this program: **1,214 → 0**
1734
- - Ecosystem coverage of measured traffic moves **82.2% → 87.9% → 88.5%** (outside the watchlist: **35.4% → 59.9% → 62.4%**; total outside-watchlist decoded: **2,975 → 3,098 / 4,968**)
1735
- - `acquisitionQueue`'s new head is `MNFSTqtC93rEfYHB6hF82sKdZpUDFWkViLByLd1k1Ms` (307 payloads)
1736
-
1737
- **The evidence class stays honest.** The headline split in `research/ecosystem-coverage-2026-11-11.json`
1738
- names every decode: `pulled: 17,681, adopted: 1,214`. The 1,214 Lighthouse decodes
1739
- are hash-bound to a reviewed copy whose origin stays **declared** (the commit is
1740
- not checked for a signature, and nothing proves the bytes describe the deployed
1741
- program; the byte evidence for that is the 100% byte-exact re-encode on live
1742
- traffic). The curated watchlist carries zero adopted decodes: its 96.4% still
1743
- rests entirely on slot-bound on-chain receipts.
1744
-
1745
- Falsification (`test/lighthouse-acquisition.test.mjs`, 9 tests):
1746
- - **L1..L5:** a tampered vendor file, a missing file, a file declaring another address, and a tampered shank discriminant table all refuse the load by name before any write is attempted;
1747
- - **L6:** adopting the verified copy into an empty store makes all 1,214 payloads decode byte-exactly, while the store before the adoption refuses every one as `no stored IDL`;
1748
- - **G1/G2:** the 1,214 denominator is pinned, and adopt's product refusal for a foreign address is guarded.
1749
-
1750
- `sememe gap` on this program points directly at `sememe adopt-lighthouse` instead
1751
- of telling the stranger to search for a file the package already carries.
1752
-
1753
- Falsification (`test/ata-interface.test.mjs`, 8 tests): a tampered artifact, a
1754
- tampered vendored source and a missing `vendor/` all refuse the load; the wire
1755
- and runtime shape is pinned against upstream's own text (tags, unit enum, empty
1756
- fallback, `InvalidInstructionData`); every measured payload must decode, and
1757
- the 4 empty ones must come out as Create — named, not refused. Guards: ATA
1758
- stays OUT of the token table, and a real SPL Token payload keeps decoding
1759
- through its own table.
1760
-
1761
- **Limits.** One 6-block window, one RPC, one pin. The table describes the
1762
- program AT THE PIN: a tagged upstream release that changes the enum or the
1763
- processor refuses the load loudly instead of decoding stale. No account list
1764
- and no PDA derivation is claimed — the table covers instruction data only. The
1765
- empty-input verdicts carry `roundTripByteExact: false` BY DESIGN: upstream
1766
- never writes a zero-length payload, the canonical encoding is `[0]`, and the
1767
- 4 empty wire forms are named through the processor's own empty-input branch.
1768
-
1769
- **Limits.** One four-block window (~20 seconds of mainnet), one RPC. The
1770
- acquisability check ran at capture time and reflects that moment. The token
1771
- tables are derived from upstream's interface crates at one pin each; a tagged
1772
- upstream release that changes a pack() arm refuses the load loudly instead of
1773
- decoding stale. Token-2022 extension sub-instructions (the 15 extension
1774
- dispatch tags) are named at the top level only; their sub-instruction bytes
1775
- report as trailing. The spl-token-metadata fallback interface (8-byte
1776
- discriminators, own crate) is vendored and derived like the others — see below.
1777
- Traffic outside the watched list (181 further programs, 4,842 instructions) is
1778
- counted in the report but excluded from the denominator, because their bytes are
1779
- not stored. A coverage number over one window is a window, not the ecosystem —
1780
- this one can only be compared with another measurement of the same kind. The
1781
- Raydium table is derived at one pin of one repository; a tagged upstream change
1782
- to a `pack()` arm refuses the load loudly rather than decoding stale. Reproduce:
1783
- `npm test` (the report is pinned twice — as a deep-equal against
1784
- what the tool computes, and as explicit assertions), or
1785
- `node tools/traffic-coverage-report.mjs`.
1786
-
1787
- ## Short wire forms: 171 of 172 refusals were chain-accepted (2026-11-03)
1788
-
1789
- The 172-payload refusal column above is 172 separate facts about an
1790
- *interface*, not about the chain. This section asked which side was right, one
1791
- shape at a time.
1792
-
1793
- **The method.** `tools/short-form-wire-forms.mjs` groups the refused payloads by
1794
- `(program, discriminator, wire length)` and, for each group, pulls a FINALIZED
1795
- mainnet transaction witness with `maxSupportedTransactionVersion: 1`, resolves
1796
- versioned account keys through their address lookup tables, reads the
1797
- instruction data as base58 (that is how `encoding:"json"` carries it — the
1798
- first version of this tool read it as base64 and compared nonsense), and
1799
- requires three independent things before it will say the chain executed the
1800
- payload: the transaction's `err` is `null`, the witness bytes are byte-IDENTICAL
1801
- to the fixture payload, and the program's own `Program log: Instruction: ...`
1802
- line matches the name the decode will report. A control group of already-decoded
1803
- shapes runs alongside, so a witness fetch that silently failed would show up as
1804
- a wrong answer rather than as evidence. Report:
1805
- [`research/short-form-wire-forms-2026-11-03.json`](research/short-form-wire-forms-2026-11-03.json).
1806
-
1807
- **The result.** The five refused shapes are:
1808
-
1809
- | program | discriminator | wire bytes | payloads | instruction | field the wire does not carry |
1810
- | --- | --- | --- | --- | --- | --- |
1811
- | Pump.fun bonding curve | `e1f7501ed5b38488` | 24 | 152 | `buy_exact_quote_in_v3` | `partial_fill` |
1812
- | PumpSwap | `c62e1552b4d9e870` | 24 | 17 | `buy_exact_quote_in` | `track_volume` |
1813
- | Pump.fun bonding curve | `07051dc4f5176550` | 24 | 1 | `buy_v3` | `partial_fill` |
1814
- | PumpSwap | `66063d1201daebea` | 24 | 1 | `buy` | `track_volume` |
1815
- | Token-2022 | `d2e11ea258b84d8d` | 131 | 1 | not named | — |
1816
-
1817
- **171 of 172 payloads were executed by the chain** with bytes identical to the
1818
- fixture. **0 were genuine chain-rejected refusals.** The stored interfaces
1819
- over-describe the wire: they declare one `OptionBool` more than the deployed
1820
- program reads, and a payload that ends where that field would start is refused
1821
- as truncated. The published first-party IDL knows all four discriminators; it is
1822
- the trailing option tag it expects that the 24-byte form does not carry. The
1823
- fifth shape (Token-2022, tag 210) is not a short form and stays refused.
1824
-
1825
- **Why this is not simply "decode whatever ends on a field boundary."**
1826
- Byte-exact re-encoding — the evidence that makes trailing bytes trustworthy —
1827
- does NOT carry over. A payload truncated by a capture bug also ends on a field
1828
- boundary and also re-encodes byte-exactly to the shortened bytes. Offline, on
1829
- bytes alone, "the program accepts a shorter form" and "this payload was cut
1830
- short" are indistinguishable. So `src/codec.mjs` keeps the default strict, and
1831
- the opt-in comes in two flavours:
1832
-
1833
- | `--absent` | behaviour | risk |
1834
- | --- | --- | --- |
1835
- | `error` (default) | refuse, as before | none — and the 171 stay unnamed |
1836
- | `report` | decode any payload ending on a field boundary, name the absent fields | names truncated payloads too |
1837
- | `verified` | decode only shapes with a chain witness | evidence can go stale |
1838
-
1839
- `verified` is the product form of the measurement: the witnesses travel with
1840
- the decoder as `src/short-forms.mjs`, keyed on program + discriminator + exact
1841
- length, and a decode whose absent set differs from the measured one is REFUSED
1842
- rather than decoded and labelled unverified — stale evidence is not spent on a
1843
- shape nobody measured. A 16-byte truncation of the 24-byte form, which ends on a
1844
- field boundary and re-encodes exactly, is accepted under `report` and refused
1845
- under `verified`; that asymmetry is the whole point, and it is a test
1846
- (`test/short-form-decoding.test.mjs`).
1847
-
1848
- **Coverage, same sample, same entry points, one opt-in** (after the token-metadata
1849
- fallback landed, which closed the last strict refusal — see below):
1850
-
1851
- | outcome | strict | `--absent verified` |
1852
- | --- | --- | --- |
1853
- | decoded | 12,401 | 12,572 (+171) |
1854
- | refused | 171 | **0** |
1855
- | no-interface | 0 | 0 |
1856
- | coverage | 98.6% | **100%** |
1857
- | byte-exact | 11,708 | 11,879 (+171, every short form re-encodes exactly) |
1858
-
1859
- The denominator did not move; the strict measurement is unchanged and still
1860
- committed, because a gain that only exists when you relax the decoder is a
1861
- finding to measure, not a number to replace. Reproduce:
1862
- `node tools/traffic-coverage-report.mjs --absent=verified`
1863
- ([`research/short-form-coverage-2026-11-03.json`](research/short-form-coverage-2026-11-03.json)).
1864
-
1865
- **What is still open.** The strict-verified gap is closed for this sample: after the
1866
- token-metadata fallback below landed, the last strict refusal decodes, so
1867
- `--absent=verified` reports 0 refusals on this window. The acquisition queue is empty
1868
- *for this sample* (the Raydium closure above removed the last entry, and a different
1869
- window can put a program back); and the short-form table covers one four-block window —
1870
- a longer window may well find a short form nobody has witnessed, which must refuse
1871
- under `verified` until a witness exists.
1872
-
1873
- ## The last refusal was the program's own fallback: Token-2022 token-metadata (2026-11-03)
1874
-
1875
- One payload in the unbiased sample refused every path sememe had: a Token-2022
1876
- instruction whose first byte is `0xd2`, 112 bytes long. It was not source lag and not
1877
- an acquisition gap — it was a decode path sememe did not have.
1878
-
1879
- **What the program does.** Token-2022's processor tries the ONE-BYTE token selector
1880
- first (`decode_instruction_type(input)`); when that byte is not a token variant it
1881
- falls through to `TokenMetadataInstruction::unpack(input)` and, if that fails too, to
1882
- `TokenGroupInstruction::unpack`. The token-metadata instructions therefore carry
1883
- **EIGHT-BYTE discriminators** (`sha256("spl_token_metadata_interface:<action>")[..8]`)
1884
- and never match the token table's 1-byte tags. sememe's refusal was a true statement
1885
- about the token table and a false one about the program.
1886
-
1887
- **What landed.** `vendor/spl-token-metadata-interface/7038969e` (Apache-2.0, byte-
1888
- identical to crates.io `spl-token-metadata-interface` 1.0.1 — the version Token-2022's
1889
- own `Cargo.lock` pins; the directory name is the crate's upstream git sha) plus two
1890
- modules and one artifact, honouring the same two-check contract as every other table:
1891
-
1892
- - `src/token-metadata-interface.mjs` derives the table FROM the vendored source:
1893
- the discriminator from upstream's own `#[discriminator_hash_input("...")]` string,
1894
- the argument order from each struct's Borsh field declaration, and the wire rules
1895
- (string = u32 LE length + bytes, enum = 1-byte variant index, option = 1 tag byte,
1896
- `MaybeNull<Address>` = 32 transparent bytes where all-zero is `None`).
1897
- It refuses on anything it cannot pin: an unknown field type, a variant upstream
1898
- declares that the table does not name, a discriminator collision.
1899
- - The **discriminator rule is anchored by upstream's own literal**: `state.rs`
1900
- hardcodes the `TokenMetadata` account discriminator `[112,132,90,90,11,88,157,87]`,
1901
- and upstream's own test derives it from `sha256("<NAMESPACE>:token_metadata")`. The
1902
- derivation recomputes it and throws if it differs — so the hash rule cannot drift
1903
- silently while the five instruction discriminators ride along with it.
1904
- - `src/token-metadata.mjs` loads it with the two-check contract (fresh derivation must
1905
- byte-equal the committed artifact) and decodes; `src/token.mjs` reaches it exactly
1906
- where upstream does — the 1-byte selector is consulted FIRST, and only a payload
1907
- whose first byte is not a token variant gets there. Legacy SPL Token has no such
1908
- branch and still refuses.
1909
- - `interface/token-metadata.json` + `tools/token-metadata-interface.mjs [--check]`.
1910
-
1911
- **Strictness, verified rather than assumed.** Upstream's `unpack` calls
1912
- `try_from_slice`, which REFUSES a payload with unexplained trailing bytes — and it does
1913
- so in the upstream crate itself: a trailing byte is a `BorshIoError`. sememe mirrors
1914
- that (trailing bytes are a refusal, not a reported count), so the decoder never claims
1915
- the chain accepts bytes the chain rejects.
1916
-
1917
- **The falsifier is the program, not a second reading of the source.**
1918
- `tools/token-metadata-oracle.mjs` builds a Rust program
1919
- (`tools/token-metadata-oracle/`) against the exact pinned crates —
1920
- `spl-token-metadata-interface 1.0.1`, `spl-discriminator 0.5.2`, `solana-nullable 1.3.0`,
1921
- `borsh 1.8.1` — and asks them: it packs every variant, unpacks the observed mainnet
1922
- payloads, and reports which inputs it refuses
1923
- ([`research/token-metadata-oracle-2026-11-03.json`](research/token-metadata-oracle-2026-11-03.json)).
1924
- `test/token-metadata.test.mjs` then compares sememe's table against that output:
1925
- all five discriminators (produced by upstream's own `SplDiscriminate` derive macro),
1926
- all ten re-encodes, and all nine refusal reasons agree.
1927
-
1928
- **Measured, same sample, same product entry points:** Token-2022 goes from 1,597/1,598
1929
- (the one refusal was this payload) to **1,598/1,598**; strict coverage 12,400 →
1930
- **12,401 decoded** / 172 → **171 refused**; `--absent=verified` 12,571 →
1931
- **12,572 / 0 refusals = 100%**. The denominator never moved.
1932
-
1933
- **Tests: 10 new** (`test/token-metadata.test.mjs`, 306 → 316). The mutation ladder:
1934
- tampered artifact → load refused; tampered vendored `instruction.rs` → load refused;
1935
- deleted `vendor/spl-token-metadata-interface` → load refused; a hand-mutated
1936
- `RemoveKey` field order (bool moved behind the string) is caught by the artifact check
1937
- AND by the oracle comparison. Refusals stay named: unknown 8-byte discriminator,
1938
- truncated discriminator, forged Borsh string length, trailing byte, invalid bool byte,
1939
- undeclared `Field` tag.
1940
-
1941
- **Known limits, kept explicit.** This is the instruction path only — the `TokenMetadata`
1942
- account (the TLV account those instructions mutate) is a different structure and is not
1943
- decoded. The `TokenGroup` fallback (`spl-token-group-interface`) is the third branch in
1944
- that same processor and is still not vendored, so a group instruction would refuse by
1945
- name; it has no payload in this window, so this is a stated boundary rather than a
1946
- measured gap.
1947
-
1948
- ## The interface's other half: the account list, checked against the wire (2026-11-03)
1949
-
1950
- Everything before this section verifies BYTES. An instruction is bytes AND an
1951
- account list, and the account list is where a real first-party program diverged
1952
- from its own interface: jup-ag's `order_engine` declared 11 accounts for `fill`
1953
- while every observed fill carried 12 or 13 — and the extra account SIGNS every
1954
- fill (reported as [jup-ag/rfq-webhook-toolkit#96](https://github.com/jup-ag/rfq-webhook-toolkit/issues/96)).
1955
- A consumer that builds its signer/authority reasoning from the interface alone is
1956
- blind to an account that signs.
1957
-
1958
- So the interface sememe stores now answers that question too, offline, from the
1959
- same receipt-verified store:
1960
-
1961
- - `src/account-list.mjs` — `checkAccountList(instruction, observedAccounts)`:
1962
- compares an instruction's declared account list with the list one transaction
1963
- passed. Verdicts: `missing-declared-accounts`, `declared-signer-not-signing`,
1964
- `fixed-address-mismatch`, `undeclared-signing-account`, and the informational
1965
- `remaining-accounts` (extras that do NOT sign — Anchor's legitimate
1966
- `remaining_accounts`).
1967
- - `src/conformance.mjs` — `checkTransaction(tx, store)`: every instruction of an
1968
- RPC `json`-shaped transaction (top level and CPI) checked against the stored
1969
- interface. The signer flags are derived from `message.header`'s counts —
1970
- `accountKeys` in that encoding carries addresses only — and a shape this file
1971
- cannot read is REFUSED (`ConformanceRefusal`), never reported as conforming.
1972
- - `sememe conform <tx.json> --store <dir>`: the CLI entry; exit 1 on any
1973
- disagreement.
1974
- - `tools/account-list-conformance.mjs`: the live measurement (program's own
1975
- signature page → `getTransaction` → every instruction checked).
1976
-
1977
- Four rules came out of the measurement, each from a case the first version got
1978
- wrong, each pinned by a test that fails without its rule (19/20 on the stripped
1979
- tree, 21/21 with it):
1980
-
1981
- 1. **The optional flag has two spellings.** OpenBook v2's legacy IDL marks
1982
- `openOrdersAdmin` `isOptional: true, isSigner: true`; live traffic passes it
1983
- unsigned in 40/40 of one sample because the admin is not needed. Reading only
1984
- `optional` (the new spec) turned the account being ABSENT into 40 violations.
1985
- 2. **A declared signer that does not sign is a violation at depth 0 only.**
1986
- Whirlpool `swap_v2` invoked via CPI by the Titan router: the inner
1987
- instruction's `token_authority` is signed for inside the CPI stack
1988
- (`invoke_signed` PDA seeds), which never appears in the top-level
1989
- `numRequiredSignatures`. At depth > 0 the signer flag is not observable from
1990
- the header, so such a position is a named note (`cpi-signer-not-observable`),
1991
- not a violation. At depth 0 the header IS the authority and the check stands.
1992
- 3. **Anchor's absent-optional placeholder is the program's own id.** Comparing
1993
- that position's signer flag manufactures a violation out of the convention.
1994
- 4. **A declared-OPTIONAL signer arriving unsigned is absent, not violating**
1995
- (the same account being absent, reached through a different shape).
1996
-
1997
- **The measurement** ([research/account-list-conformance-2026-11-03.json](research/account-list-conformance-2026-11-03.json),
1998
- 2026-11-03T21:26Z, 170 RPC calls, 0 failures, 236 instructions from 137
1999
- transactions of 6 programs, every row committed so the summaries re-derive from
2000
- them):
2001
-
2002
- | program | instructions | verdicts |
2003
- |---|---|---|
2004
- | Whirlpool | 57 | 40 conforms, 17 remaining-accounts, 0 violations |
2005
- | Jupiter v6 | 22 | 13 unknown-instruction, 2 conforms, 7 undeclared-signing-account |
2006
- | OpenBook v2 | 43 | 36 conforms, 7 remaining-accounts, 0 violations |
2007
- | PumpSwap | 56 | 25 remaining-accounts, 31 unknown-instruction |
2008
- | pump.fun | 18 | 7 remaining-accounts, 10 unknown-instruction, 1 conforms |
2009
- | jup-ag order\_engine | 40 | **40 undeclared-signing-account** |
2010
-
2011
- The contrast is the finding. Jupiter's signing extras sit inside routes of 14–43
2012
- extra accounts and belong to 6 distinct caller wallets per 7 fills — the router
2013
- passing the user's authority through, a legitimate `remaining_accounts` shape
2014
- that happens to sign. order\_engine's signing extra is ONE CONSTANT address
2015
- (`pAEmG7…`) in 40/40 fills, always at the LAST position, signing every fill and
2016
- paying the fee in 31/141 of the November-1 sample: a hidden authority the
2017
- published interface does not name. Same verdict kind, opposite meanings — which
2018
- is why the report commits the rows, not just the counts.
2019
-
2020
- The `unknown-instruction` rows (54 total) are honest, not noise: pump.fun and
2021
- PumpSwap's IDLs store all their discriminators (88/88 and 65/65 agree with
2022
- Anchor's derivation), but the sampled fills carry NEWER instruction revisions
2023
- than the stored pull (the `*_v2`/`*_v3` families measured in the short-form
2024
- section above). A stale pull shows up here as unknown-instruction instead of a
2025
- wrong decode — that is the check doing its job.
2026
-
2027
- **Known limits:** the signer observation is the transaction-level view — a CPI
2028
- can sign for its PDAs and the header cannot see it (rule 2 handles the
2029
- consequence, not the visibility); a program that legitimately accepts several
2030
- account-list shapes produces one verdict per transaction, and only repeated
2031
- agreement/disagreement across a sample means anything; writability disagreements
2032
- are reported per position but never decide a verdict (the runtime enforces from
2033
- the transaction, not the IDL); a failed transaction can have been rejected FOR
2034
- one of these reasons, and `executed` is carried so a reader can tell.
2035
-
2036
- ## A filter that selects nothing is a refusal, not a green CI gate (2026-11-09)
2037
-
2038
- `sememe conform` is a CI gate: exit 1 on any disagreement, exit 0 when the
2039
- checked instructions agree with the store. `--only <program-id>` narrows the
2040
- check to one program. Measured on the published `v0.4.12` install, a filter
2041
- that selected **zero** instructions still exited **0**:
2042
-
2043
- ```
2044
- sememe conform tx.json --store ./store --only 6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P
2045
- -> { "rows": [], "summary": { "instructions": 0, "executed": { "disagreements": 0 } } }
2046
- -> exit 0
2047
- ```
2048
-
2049
- The program id above is a real one — it is just not invoked by that
2050
- transaction. A caller who mistyped the id they meant to gate on was told the
2051
- transaction conformed: a green gate over a check that never ran. The second
2052
- face of the same defect: `--only not-a-program-id` produced the identical
2053
- empty green, and the value was never validated as a program id at all — it
2054
- could not be one.
2055
-
2056
- This is the class this repository has already measured twice (`sememe verify`
2057
- green over a store the reader refuses, 2026-11-07; `conform` refusing an
2058
- unreadable transaction SHAPE but not an empty selection). The rule is the one
2059
- this file already states for an unknown shape: an empty check is not a clean
2060
- bill of health.
2061
-
2062
- **What changed.** Two refusals, both before the store is read:
2063
-
2064
- - a filter that is not a canonical 32-byte base58 program id is refused by
2065
- name (`filter-is-not-a-program-id`) — the check is dependency-free
2066
- (`src/program-id.mjs` round-trips the base58 itself), because the decode
2067
- core is imported by consumers with no npm dependencies at all and a CLI
2068
- refusal must not be the thing that makes them install a peer;
2069
- - a filter that selects none of the transaction's instructions is refused with
2070
- `filter-matched-no-instruction`, and the refusal **names the programs the
2071
- transaction actually invokes**, so a typo is visible instead of green.
2072
-
2073
- The unfiltered path is untouched, and a filter that does match still runs.
2074
-
2075
- **The measurement ships as an instrument, not a claim.**
2076
- `tools/measure-conform-filter.mjs` derives
2077
- [research/conform-filter-domain-2026-11-09.json](research/conform-filter-domain-2026-11-09.json)
2078
- by running the SAME probe file in the pre-fix tree (`git archive 58d3228`) and
2079
- in this tree: on the pre-fix tree both defect cases exited 0 over 0
2080
- instructions; in this tree both are refused with the reason named above, and
2081
- the unfiltered run's exit code and instruction count are unchanged. The tool
2082
- fails if the baseline ever shows no defect, and
2083
- `test/conformance-filter-measurement.test.mjs` fails if the committed
2084
- artifact stops being what the tool derives. The ladder
2085
- `test/conformance-filter-domain.test.mjs` is 6 red on the pre-fix tree and
2086
- 11 green here; `tools/mutation-ladder-conform-filter.sh` catches 5/5
2087
- mutations (the empty-selection guard, the program-id check, the reason name,
2088
- the invoked-program detail, and the filter actually applying).
2089
-
2090
- ## A lock that pins nothing is a refusal, not a green CI gate (2026-11-09)
2091
-
2092
- `sememe lock` + `sememe verify --mode error` is the CI gate the section above
2093
- (*pin the answers*) tells a stranger to run after checkout. Two faces of ONE
2094
- defect made that gate able to pass without checking anything:
2095
-
2096
- ```bash
2097
- npx sememe lock --store ./store --out ./sememe.lock # store: receipts/ exists, no receipt in it
2098
- -> { "lockVersion": 1, "programs": {} }
2099
- -> exit 0
2100
-
2101
- npx sememe verify --store ./store --lock ./empty.lock --mode error # empty.lock pins 0 programs
2102
- -> { "pinned": 0, "violations": 0, "extra": [ ...18 real programs... ] }
2103
- -> exit 0
2104
- ```
2105
-
2106
- The first command wrote a lock pinning **zero** answers: no fingerprint, no
2107
- receipt, no origin — nothing a drifted store could ever contradict. The second
2108
- command then certified a **real 18-program store** as clean against it: every
2109
- program appeared as an `extra`, extras are never violations for being there
2110
- (that rule is correct and stayed), so the report even *looked* informative
2111
- while pinning nothing. `--mode warn` did the same. This is the same class this
2112
- repository has measured three times: `verify` green over a store the reader
2113
- refuses (2026-11-07), `conform --only` green over a filter selecting nothing
2114
- (fixed in v0.4.13), and now `lock`/`verify` green over a lock pinning nothing.
2115
-
2116
- **What changed.** Two refusals, both named `lock-pins-nothing`:
2117
-
2118
- - `sememe lock` over a store with no receipt **refuses and writes no file** —
2119
- the repair (a `sememe pull` first) is named in the refusal. A lock over a
2120
- store whose `receipts/` directory does not exist is still refused as
2121
- `no sememe store`: the two refusals have different repairs and must not
2122
- collapse into one message;
2123
- - `sememe verify` against a lock whose `programs` map is empty **refuses in
2124
- both modes** — the defect is a property of the DOCUMENT, not of the store,
2125
- so `--mode warn` (which exists to downgrade *store* violations) does not
2126
- certify it either. The check runs after the `lockVersion` check, so an
2127
- unsupported version is still reported as a version problem.
2128
-
2129
- The ordinary journey is untouched: locking a real store writes a lock pinning
2130
- every program, and verifying it against an intact store still exits 0.
2131
-
2132
- **The measurement ships as an instrument, not a claim.**
2133
- `tools/measure-lock-nothing-pinned.mjs` derives
2134
- [research/lock-nothing-pinned-2026-11-09.json](research/lock-nothing-pinned-2026-11-09.json)
2135
- by running the SAME probe file in the pre-fix tree (`git archive f22dbf5`) and
2136
- in this tree: on the pre-fix tree **all three defect faces are green** (lock
2137
- exit 0 writing a 0-program file; verify exit 0 with `pinned: 0` in both modes);
2138
- in this tree all three are refused with `lock-pins-nothing`, and the ordinary
2139
- lock-then-verify journey exits 0 with 18 programs pinned in BOTH trees. The
2140
- tool fails if the baseline ever shows no defect, and
2141
- `test/lock-nothing-pinned-measurement.test.mjs` fails if the committed artifact
2142
- stops being what the tool derives. The ladder
2143
- `test/lock-nothing-pinned.test.mjs` is 5 red on the pre-fix tree and 12 green
2144
- here; `tools/mutation-ladder-lock-nothing-pinned.sh` catches 5/5 mutations
2145
- (the writer's guard, the reader's guard, the reason name, the repair sentence,
2146
- and the empty check running after the version check).
2147
-
2148
- The delivery side is measured separately, because "main fixed, the pin behind
2149
- it" is this repository's most-repeated defect: `tools/measure-lock-nothing-pinned-tag.mjs`
2150
- installs PUBLISHED TAGS into an empty directory and runs the same cases against
2151
- the bytes a stranger receives — see
2152
- [research/lock-nothing-pinned-tag-2026-11-09.json](research/lock-nothing-pinned-tag-2026-11-09.json).
2153
-
2154
- ## A store the reader refuses cannot pass the conformance gate (2026-11-10)
2155
-
2156
- `sememe conform <tx.json> --store <dir>` is a CI gate: exit 1 on any
2157
- disagreement, exit 0 when the checked instructions agree with the store. The
2158
- gate read the store through **one line that swallowed every failure**:
2159
-
2160
- ```js
2161
- const sources = await findStoredIdls(storeDir, instruction.programId);
2162
- idl = sources.primary?.idl ?? sources.secondary?.idl ?? null;
2163
- // ... inside: } catch { idl = null; }
2164
- ```
2165
-
2166
- `findStoredIdls` is the ONE store reader that refuses damage by name
2167
- (`src/store-read.mjs`: a receipt whose object no longer re-hashes to its
2168
- fingerprint, a receipt bound to another program, a missing object, a
2169
- malformed receipt, a rewritten origin). `sememe decode` on such a store exits
2170
- 1 with the reader's own message. `sememe conform` on the **same store** turned
2171
- that refusal into a verdict and exited 0.
2172
-
2173
- Three faces of ONE defect, measured on the committed fixture transaction
2174
- ([test/fixtures/order-engine-conformance-tx.json](test/fixtures/order-engine-conformance-tx.json),
2175
- which invokes `2En5Y11SEAGLNmEezTuRUCwTyzyNReHaMbSnS5gjGsL1`):
2176
-
2177
- ```bash
2178
- # the store the fixture reports a REAL disagreement against:
2179
- npx sememe conform tx.json --store ./store --only 2En5Y11…
2180
- -> { "byVerdict": { "undeclared-signing-account": 1 } } # exit 1 (correct)
2181
-
2182
- # one receipt's fingerprint tampered (the reader already refuses it on decode):
2183
- npx sememe conform tx.json --store ./store-damaged --only 2En5Y11…
2184
- -> { "byVerdict": { "no-stored-interface": 1 } } # exit 0 (green)
2185
-
2186
- # the --store directory does not exist, or has objects/ but no receipts/:
2187
- npx sememe conform tx.json --store ./nowhere
2188
- -> 12 rows, all "no-stored-interface", disagreements 0 # exit 0 (green)
2189
- ```
2190
-
2191
- The tampered-receipt face is the one a real user hits, and it is worse than a
2192
- false green: the transaction's actual disagreement **disappeared**. The gate
2193
- reported "12 instructions, 0 disagreements" for a transaction that
2194
- demonstrably disagrees, because the program it could no longer read was
2195
- reported as a program the store never held. Damage was indistinguishable from
2196
- an acquisition gap — so the gap verdict, which is honest and useful, was
2197
- carrying a lie.
2198
-
2199
- This is the same class this repository has measured four times before:
2200
- `verify` green over a store the reader refuses (2026-11-07), `conform --only`
2201
- green over a filter selecting nothing (v0.4.13), `lock`/`verify` green over a
2202
- lock pinning nothing (v0.4.14) — and structurally the same as the
2203
- `receipt-hygiene` finding: one reader's refusal being dropped by a caller.
2204
-
2205
- **What changed.** The store reader's two *absence* errors now carry a machine
2206
- readable `code` (`no-sememe-store`, `no-stored-idl`) so a caller can tell
2207
- them apart **without parsing prose** — their messages are byte-identical to
2208
- before, because `bin/sememe.mjs`, this README and several tests quote them.
2209
- `checkTransaction` then distinguishes exactly three cases:
2210
-
2211
- - **the store holds no receipt for this program** (`no-stored-idl`) — an
2212
- honest acquisition gap, unchanged: the row keeps its
2213
- `no-stored-interface` verdict and the gate keeps running;
2214
- - **the store does not exist** (`no-sememe-store`) — refused as
2215
- `no-sememe-store`;
2216
- - **anything else the reader refuses** — refused as `store-reader-refused`.
2217
-
2218
- Both refusals are `ConformanceRefusal` and **lead with the program id**,
2219
- followed by the reader's own message and the repair, so the CLI exits 1 with
2220
- `conformance refused: <reason>` instead of printing a clean report.
2221
-
2222
- **The limits of this fix, in the same breath.** A refusal means the gate could
2223
- not answer — it is not a statement that the transaction disagrees. The gate
2224
- now fails CLOSED on stores it cannot read; it does not decide anything about
2225
- the transaction in that case. A program the store genuinely does not hold
2226
- still produces `no-stored-interface` (exit 0 when nothing else disagrees),
2227
- because that is the acquisition gap the coverage numbers depend on: the fix
2228
- had to separate the two without collapsing either. `--only` filtering still
2229
- applies before the store is read, so a filter that matches no instruction is
2230
- refused first (`filter-matched-no-instruction`), and this fix does not change
2231
- the `unknown-instruction` / `unknown-discriminator` verdicts.
2232
-
2233
- **The measurement ships as an instrument, not a claim.**
2234
- `tools/measure-conform-store-damage.mjs` derives
2235
- [research/conform-store-damage-2026-11-10.json](research/conform-store-damage-2026-11-10.json)
2236
- by running the SAME probe file in the pre-fix tree (`git archive 3d386b2`) and
2237
- in this tree: on the pre-fix tree **all three defect faces are green** (exit 0
2238
- over a store the reader refused); in this tree all three are refused with the
2239
- named reasons, and both controls — the honest acquisition gap and the real
2240
- disagreement on an intact store — are byte-identical between the trees. The
2241
- tool also counts the measured HARM directly (`hiddenDisagreement`): the
2242
- pre-fix intact store showed `undeclared-signing-account` and the pre-fix
2243
- damaged store showed nothing. The tool fails if the baseline ever shows no
2244
- defect, and `test/conform-store-damage-measurement.test.mjs` fails if the
2245
- committed artifact stops being what the tool derives. The ladder
2246
- `test/conform-store-damage.test.mjs` is 7 red on the pre-fix tree and 11 green
2247
- here; `tools/mutation-ladder-conform-store-damage.sh` catches 5/5 mutations.
2248
-
2249
- The delivery side is measured separately, because "main fixed, the pin behind
2250
- it" is this repository's most-repeated defect:
2251
- `tools/measure-conform-store-damage-tag.mjs` installs PUBLISHED TAGS into an
2252
- empty directory and runs the same cases against the bytes a stranger receives.
2253
-
2254
- ## An instruction the gate cannot resolve must refuse, not vanish (2026-11-11)
2255
-
2256
- `sememe conform <tx.json> --store <dir>` resolves each instruction's program
2257
- one of two ways: `instruction.programId` directly, or `instruction.programIdIndex`
2258
- by INDEX into the canonical key list. An index past the end of the list
2259
- resolves to nothing — and the gate's row loop simply stepped over it:
2260
-
2261
- ```js
2262
- for (const instruction of instructions) {
2263
- if (!instruction.programId) continue; // no row, no count, no refusal
2264
- ```
2265
-
2266
- The instruction disappeared from `rows`, from `summary.instructions`, and from
2267
- every verdict count. The report it produced was over a **subset of the
2268
- transaction it was handed** — while describing itself as a check of that
2269
- transaction. The file's own header already states the rule ("an unknown shape
2270
- must never be reported as a clean bill of health") and the code already
2271
- refuses when **NO** instruction resolves
2272
- (`transaction-has-no-resolvable-program-id`); only the middle case — some
2273
- resolve, some do not — was silent.
2274
-
2275
- **The worst face, measured on the order-engine fixture** (committed fixture,
2276
- no network): the fill instruction is the ONE instruction with a real
2277
- disagreement (an undeclared signing account). Pushing its `programIdIndex`
2278
- past the end of `accountKeys`:
2279
-
2280
- ```
2281
- intact transaction: rows 12, byVerdict {no-stored-interface: 11, undeclared-signing-account: 1}, exit 1
2282
- broken fill index: rows 11, byVerdict {no-stored-interface: 11}, exit 0
2283
- ```
2284
-
2285
- The disagreement was not just unchecked — it was **hidden**, behind a report
2286
- that looked complete over 11 of 12 instructions. Same class as the store the
2287
- reader refuses (2026-11-10), one level over: not the store, the transaction.
2288
-
2289
- **What changed.** One refusal, `instruction-program-id-unresolvable`, raised
2290
- **before** the store is read and **before** the filter selection:
2291
-
2292
- - some resolve, some do not -> refused, naming each offending instruction by
2293
- position (top-level or `CPI of top-level #n`) and by the raw
2294
- `programIdIndex` value it carried, plus the repair;
2295
- - ALL unresolvable keeps its **existing** reason
2296
- (`transaction-has-no-resolvable-program-id`) — that refusal predates this
2297
- fix and must not be rewritten (it is pinned as guard G2);
2298
- - a filter that selects nothing still fires its own reason first for callers
2299
- whose argument is the problem — the filter check stays ahead of both;
2300
- - CPI instructions (`meta.innerInstructions`) are covered too: an unresolved
2301
- inner instruction is part of the transaction and cannot silently shrink the
2302
- report.
2303
-
2304
- The unfiltered and `--only` paths now agree: a transaction with a broken
2305
- shape refuses on both, and the CLI exits 1 with the reason on stderr and no
2306
- decision-carrying report (the decision is the refusal).
2307
-
2308
- **The measurement ships as an instrument, not a claim.**
2309
- `tools/measure-conform-unresolvable.mjs` derives
2310
- [research/conform-unresolvable-2026-11-11.json](research/conform-unresolvable-2026-11-11.json)
2311
- by running the SAME probe in the pre-fix tree (`git archive 94aa00b` = the
2312
- published v0.4.15) and in this tree: on the pre-fix tree the defect case exits
2313
- 0 over the dropped instruction AND the transaction's real disagreement is
2314
- absent from the report (`hiddenDisagreement: true`); in this tree all three
2315
- defect cases refuse with the named reason, the intact control still exits 1
2316
- with its disagreement, and the all-dropped control keeps its original reason
2317
- in **both** trees. The tool exits non-zero if the baseline ever shows no
2318
- defect, and
2319
- `test/conform-unresolvable-measurement.test.mjs` fails if the committed
2320
- artifact stops being what the tool derives. The ladder
2321
- `test/conform-unresolvable-program-id.test.mjs` is **6 red on the pre-fix
2322
- tree and 16 green here** (6 defect cases U1–U6, 4 guards G1–G4 that pass on
2323
- both trees); `tools/mutation-ladder-conform-unresolvable.sh` catches **6/6**
2324
- mutations (the refusal itself, the reason name, the refusal order vs the
2325
- filter, the detail naming positions, the all-dropped reason staying put, and
2326
- CPI coverage).
2327
-
2328
- **Limits.** The refusal is about a shape no reader can interpret — it does
2329
- not decide what the *right* program id is, only that the gate must not guess.
2330
- A transaction whose `programIdIndex` legitimately points at a lookup-table
2331
- loaded address is resolved through `meta.loadedAddresses` first (unchanged);
2332
- an instruction that omits both `programId` and `programIdIndex` lands in the
2333
- same refusal (`(none)` in the detail). One transaction fixture, committed —
2334
- no live RPC shape was sampled for this pulse.
2335
-
2336
- The delivery side is measured separately, because "main fixed, the pin behind
2337
- it" is this repository's most-repeated defect:
2338
- `tools/measure-conform-unresolvable-tag.mjs` installs PUBLISHED TAGS into an
2339
- empty directory and runs the same defect cases against the bytes a stranger
2340
- receives — see
2341
- [research/conform-unresolvable-tag-2026-11-11.json](research/conform-unresolvable-tag-2026-11-11.json).
2342
- v0.4.15 is shown to carry the defect on all three dropped shapes (and exits 0
2343
- with the disagreement hidden on the worst face); v0.4.16 refuses all three
2344
- with `instruction-program-id-unresolvable`, with both controls unchanged.
2345
- `test/conform-unresolvable-tag-measurement.test.mjs` fails if those roles ever
2346
- blur.
2347
-
2348
- ## An account index the gate cannot resolve is the same defect one level over (2026-11-11)
2349
-
2350
- One pulse ago the program reference was fixed; the ACCOUNT reference still
2351
- had the identical shape. `sememe conform` resolves each instruction's
2352
- accounts by index into the canonical key list, and in the published v0.4.16
2353
- an index that did not resolve — past the end, negative, or a non-integer
2354
- entry such as the `jsonParsed` OBJECT shape this file's own header says it
2355
- does not model — silently became `{address: null, signer: null,
2356
- writable: null}`. `checkAccountList` files a null signer under
2357
- `unknownSigner`: a note, never a violation. The report looked complete.
2358
-
2359
- **The worst face, measured on the order-engine fixture** (committed fixture,
2360
- no network): the fill instruction carries the transaction's ONE real
2361
- disagreement — a signing account the interface does not declare, at position
2362
- 11. Replacing exactly that index with an out-of-range value:
2363
-
2364
- ```
2365
- intact transaction: verdict undeclared-signing-account, EXIT 1
2366
- broken extra account index: verdict remaining-accounts, EXIT 0
2367
- whole transaction in jsonParsed object shape: remaining-accounts, EXIT 0
2368
- ```
2369
-
2370
- The disagreement was not just unchecked — it was **hidden**, behind a green
2371
- gate. This is the class this repository keeps finding one level down: not
2372
- the program reference, the account reference.
2373
-
2374
- **What changed.** One refusal, `instruction-account-index-unresolvable`,
2375
- raised **before** the store is read and **before** the filter selection —
2376
- the same ordering discipline as the program-id sibling:
2377
-
2378
- - any instruction whose accounts list carries an index the key list cannot
2379
- resolve is refused, naming each offending instruction (top-level `#n`, or
2380
- `CPI of top-level #n at position p`), the account POSITION, and the RAW
2381
- value it carried, plus the repair;
2382
- - CPI instructions are covered — an inner instruction is part of the
2383
- transaction;
2384
- - the `jsonParsed` object entries are refused by name instead of folding
2385
- into null rows — the header's own "this file handles the `json` shape"
2386
- is now enforced, not just stated;
2387
- - the older refusals did not move: an unresolvable PROGRAM index keeps
2388
- `instruction-program-id-unresolvable` even beside broken accounts (guard
2389
- G4), an EMPTY accounts list stays legal (guard G2 — emptiness is not
2390
- unresolvability), and indices into the loaded (ALT) range stay resolvable
2391
- (guard G3).
2392
-
2393
- **The measurement ships as an instrument, not a claim.**
2394
- `tools/measure-conform-unresolvable-accounts.mjs` derives
2395
- [research/conform-unresolvable-accounts-2026-11-11.json](research/conform-unresolvable-accounts-2026-11-11.json)
2396
- by running the SAME probe in the pre-fix tree (`git archive 7b30c68` = the
2397
- published v0.4.16) and in this tree: on the pre-fix tree **three of the four
2398
- drop shapes exit 0** (`droppedExtraSigner`, `droppedFillAll`,
2399
- `parsedObjectAccounts`) AND the fill's real disagreement is gone from the
2400
- report (`hiddenDisagreement: true`); in this tree all four refuse with the
2401
- named reason, and both controls — the intact disagreement (exit 1 in BOTH
2402
- trees) and the legal empty-accounts shape — are unchanged. The tool exits
2403
- non-zero if the baseline ever shows no defect, and
2404
- `test/conform-unresolvable-accounts-measurement.test.mjs` re-derives the
2405
- artifact and fails if the committed numbers stop being reproducible. The
2406
- ladder `test/conform-unresolvable-accounts.test.mjs` is **6 red on the
2407
- pre-fix tree and 10 green here** (A1–A6 defect, G1–G4 guards that pass on
2408
- both trees); `tools/mutation-ladder-conform-unresolvable-accounts.sh` catches
2409
- **6/6** mutations (the refusal itself, the reason name, the detail's
2410
- positions and raw values, CPI coverage, the non-integer check, the
2411
- unfiltered path).
2412
-
2413
- One ladder error, own-fault, caught by the ladder's own gates: the first
2414
- M3/M5/M6 were VACUOUS — M3's replacement string matched the program-id
2415
- refusal first (wrong block mutated, suite stayed green), M5's
2416
- `typeof index !== 'number'` still refused objects (the mutation did not
2417
- remove the behaviour under test), and M6's `[] && x` is truthy in bash-side
2418
- JS. All three were rewritten to bite, and the vacuous first run is why the
2419
- ladder requires an ANCHORED mutation, not just a plausible one.
2420
-
2421
- **Limits.** The refusal is about a shape no reader can interpret — it does
2422
- not decide what the *right* account is. The check covers indices at the
2423
- `transactionInstructions` seam; a transaction that resolves every index
2424
- still passes through `checkAccountList` unchanged, so honest
2425
- `unknownSigner` notes (a genuinely mangled encoding downstream of this
2426
- seam) keep their existing meaning. One transaction fixture, committed — no
2427
- live RPC shape was sampled for this pulse.
2428
-
2429
- ## The largest publish-nothing gap decodes in its own dialect: shank (2026-11-04)
2430
-
2431
- The acquisition-gap classification left one program standing at the top of the
2432
- list: `L2TExMFKdjpN9kozasaurPirfHy9P8sbXoAN1qA3S95` (Lighthouse, 42 payloads in
2433
- the 156-fixture foreign sweep) publishes **no IDL account at either address**.
2434
- `sememe pull` can never fetch it — so the biggest single remaining gap was not
2435
- an acquisition question but a dialect question: its interface exists as a
2436
- file in the upstream repository (`Jac0xb/lighthouse`, MIT), generated by
2437
- [shank](https://github.com/metaplex-foundation/shank), and it is NOT an Anchor
2438
- IDL.
2439
-
2440
- **What was actually different, measured rather than assumed.** Shank IDLs
2441
- change two things sememe had hard-coded:
2442
-
2443
- 1. **Dispatch is a 1-byte variant selector, not an 8-byte sighash.** Each
2444
- instruction carries `discriminant: { type: "u8", value: N }` — Lighthouse's
2445
- 18 instructions are discriminants 0..17 (`MemoryWrite=0` …
2446
- `AssertBubblegumTreeConfigAccount=17`). sememe's `withDerivedDiscriminators`
2447
- ran Anchor's `sha256("global:<name>")` over these and filled all 18 with
2448
- sighashes that match no real wire — a table that would REFUSE every
2449
- Lighthouse payload while looking fully populated. The rule now: a stored
2450
- shank discriminant is copied verbatim, counted as a stored value (not an
2451
- Anchor derivation), and never checked against the sighash rule; the two
2452
- schemes are independent.
2453
- 2. **The lighthouse family encodes lengths as LEB128, not Borsh's u32.**
2454
- `CompactU64`, `CompactBytes` and the six `*Assertions` vector types are
2455
- referenced by the IDL but never defined in it — their wire form is pinned
2456
- upstream in `lighthouse_common::CompactU64`/`LEB128Vec` and mirrored by the
2457
- program's own generated JS client (`clients/js/src/hooked.ts`). sememe now
2458
- hooks exactly those names, **only** inside a shank-dialect IDL, and only
2459
- when the IDL does not define the name itself: an Anchor IDL that mentions
2460
- `CompactU64` still refuses it (`unknown defined type`), and a name the IDL
2461
- defines wins over the hook. Nothing is inferred from a value's shape.
2462
-
2463
- **What landed.** `src/shank.mjs` (dialect detection, discriminant extraction,
2464
- the LEB128 hook table) wired into `src/codec.mjs` and
2465
- `src/discriminators.mjs`; the first-party IDL adopted into the store through
2466
- `sememe adopt` (`test/fixtures/shank-lighthouse-2026-11-04/lighthouse.json`,
2467
- receipt fingerprint `sha256:461e28fd…`, origin
2468
- `Jac0xb-lighthouse-4c579479c986`), so the decode goes through the product's
2469
- own store path like every other program.
2470
-
2471
- **Measured, same sweep, same instrument**
2472
- ([`research/step-ingestooor-full-sweep-shank-2026-11-04.json`](research/step-ingestooor-full-sweep-shank-2026-11-04.json),
2473
- 2026-11-04T15:44Z, 156 fixtures, 806 payloads, 0 network attempts): decoded
2474
- 661 → **703**, acquisition gap 129 → **87**, byte-exact 654 → **696**, rate
2475
- **100%**, disagreements **0**, decode errors unchanged at 16. Exactly one
2476
- program row moved — Lighthouse 0 → **42/42 decoded**, both of its observed
2477
- instructions (`AssertAccountInfoMulti` 21, `AssertTokenAccountMulti` 21)
2478
- round-tripping byte-exactly; the refusal reasons of every other program are
2479
- byte-identical to the previous run.
2480
-
2481
- **Tests: 11 new** (9 in `test/shank-discriminants.test.mjs`, 2 in
2482
- `test/shank-sweep-report.test.mjs` pinning the committed report against the
2483
- byProgram sum and the store receipt hash). The ladder, run against the
2484
- pre-fix tree `4174bb3`: **7/9 RED** — T1–T5 and T8 fail exactly because the
2485
- shank support is missing (T8's error on the old tree is the refusal this work
2486
- fixes: `unknown instruction discriminator 0605020053bb5016 (18 known, e.g.
2487
- 7f92fad78d80cfee…)` — i.e. the old table holds 18 invented Anchor sighashes
2488
- and none of the real 1-byte selectors); T6 and T9 are the false-positive
2489
- guards and pass on **both** trees (Anchor 8-byte sighashes fill and decode
2490
- unchanged; an Anchor IDL mentioning a shank-named type still refuses).
2491
-
2492
- **Known limits, kept explicit.** This is one program from one upstream
2493
- repository — shank is a tool family, and the dialect was pinned from
2494
- Lighthouse's own generated client, not from every shank-generated IDL that
2495
- exists; a shank IDL whose hooked type is defined in its own `types` follows the
2496
- IDL, and one with a different hooked set is unknown until pinned the same way.
2497
- The two observation dialects (json/flat fixtures) carry no message bytes, so
2498
- the ed25519 verification path still does not apply to this payload family.
2499
- Lighthouse publishes no on-chain interface: the store copy is only as honest
2500
- as its upstream source (first-party, fingerprinted, origin-labelled) — if
2501
- upstream changes the wire, sememe will refuse by name, not decode wrongly.
2502
-
2503
- ## A dialect it does not speak is a NAMED refusal, never a crash (2026-11-12)
2504
-
2505
- The acquisition queue's fourth face looked like a data problem and turned out
2506
- to be a product defect. `DiabLokxisGR8P4Qqp2QL6PmBhhVsdzPstCE4WoLxc5z` — 102
2507
- payloads in the committed ecosystem sample — has a first-party interface
2508
- ([cryptodiablo/diabloarb](https://github.com/cryptodiablo/diabloarb),
2509
- `sdk/diabloarb-executor.idl.json` @ `f5005ac4`), and that file says in its own
2510
- description what it is: *"Raw little-endian byte layout (NOT Anchor/Borsh:
2511
- list lengths are u8 counters in the header)"*. It dispatches on a 1-byte
2512
- `tag` with fields under `data`, its `types` is a dict keyed by name
2513
- (`{Hop, Base, Route}`), and it declares its program under `program_id`.
2514
-
2515
- Feeding it to sememe before this section's fix produced three measured defect
2516
- faces, none of them a refusal:
2517
-
2518
- 1. **The codec CRASHED.** `context()` iterated `idl.types ?? []` — a dict is
2519
- not iterable — so all 102 payloads died with the unnamed
2520
- `TypeError: object is not iterable (cannot read property
2521
- Symbol(Symbol.iterator))`. A crash voice cannot be counted, grouped or
2522
- gated on: the coverage artifact recorded 102 `noInterface` and never saw
2523
- that the codec was broken one layer under.
2524
- 2. **The derivation LIED.** `withDerivedDiscriminators` filled all four
2525
- tag-dispatch instructions with Anchor sighashes (`filled {instructions: 4,
2526
- events: 0}`, zero conflicts) — an authoritative-looking table matching no
2527
- wire byte, the stored-discriminator defect one dialect up.
2528
- 3. **adopt RE-LABELLED.** The file declares its program under `program_id`;
2529
- adopt and the store reader checked `idl.address` only, so adopting the
2530
- diabloarb interface under WSOL's program id **succeeded** without a word.
2531
-
2532
- **What landed.** `src/dialect.mjs` defines the named refusal ONCE
2533
- (`unsupported IDL dialect: …`) and every boundary the defect entered through
2534
- checks it: the codec's `context()` (before any iteration), the derivation
2535
- (no table over a dialect it does not speak), `sememe adopt` (write path) and
2536
- `store-read` (read path, so a store that already contains such an object
2537
- refuses on every read). The declared-program check now reads `address`,
2538
- `metadata.address` **and** `program_id`, and the refusal names the field that
2539
- carried the declaration. Post-fix, the honest product answer for an
2540
- unadoptable interface is `no stored interface` — 102 payloads in a state you
2541
- can COUNT, group and gate on, never a TypeError.
2542
-
2543
- **The gate is surgical, and measured as such**
2544
- ([`research/dialect-named-refusal-2026-11-12.json`](research/dialect-named-refusal-2026-11-12.json),
2545
- two-tree falsification against the pre-fix commit `aa6901b`): 24
2546
- falsification counts, all zero. The prefix tree SHOWS every face — crash
2547
- 102/102, fill 4/0, adopt true under both ids, read boundary open; this tree
2548
- lands on the named voices. The supported-dialect controls pass unchanged: a
2549
- full anchor IDL, a shank IDL, and — deliberately — a legacy **name-only**
2550
- anchor IDL (Kamino's, no selectors): such IDLs have no wire claims, the
2551
- conform gate reads their account lists for layout checking, and the decode
2552
- path already refused their payloads by name (`unknown instruction
2553
- discriminator`) before this gate existed. Calling that shape "unsupported"
2554
- would have changed attested behavior without a single measured wire failure.
2555
- All 22 committed store objects stay readable. The upstream bytes carry no
2556
- license at the pin, so they are **not vendored**: the measurement takes
2557
- `--source` (the file fetched at the pin) and verifies the pinned canonical
2558
- fingerprint `sha256:d6d4e8aa…` before anything runs.
2559
-
2560
- **Tests: 19 new** (9 in `test/dialect-named-refusal.test.mjs`, 10 in
2561
- `test/dialect-named-refusal-measurement.test.mjs` — the artifact-vs-tool pin
2562
- re-derives the whole two-tree measurement and is network-skipped by name).
2563
- Suite 893 → **912: 908 pass / 4 named network skips / 0 fail**. Released as
2564
- v0.4.21: product code changed, so the pin moved.
2565
-
2566
- **Known limits, kept explicit.** Naming the refusal is not support: none of
2567
- the 102 DiabLokxis payloads decode yet. A raw little-endian dialect with
2568
- header-counted vectors is a real second dialect family and the follow-up
2569
- work is to decode it — pinned per program from its first-party source, the
2570
- way shank was. What this fix guarantees is that until that happens, the
2571
- interface is refused by name at every boundary and never crashes the codec
2572
- or pretends to be decodable.
2573
-
2574
- ## The product SPEAKS the dialect it named: the raw-LE engine is wired (2026-11-12)
2575
-
2576
- PULS X named the refusal ("naming a dialect is not speaking it") and PULS Y
2577
- built the speaker — an engine for exactly the grammar the diabloarb file
2578
- DECLARES (`encoding: "little-endian; u8/u16/u32/u64 unsigned, i64 signed; no
2579
- padding; no length prefixes except the u8 counters named in each header"`):
2580
- u8-tag dispatch, dict `types` with declared fixed sizes, counter-prefixed
2581
- vectors (`Base[n_bases]`, `u8[n_hops]`), one byte-decided optional trailing
2582
- field. 102/102 committed payloads decoded byte-exactly, and the product
2583
- boundary stayed untouched — deliberately: capability first, the wiring with
2584
- the pins it rewrites.
2585
-
2586
- **This pulse is the wiring** (the card's own declared next step):
2587
-
2588
- - `src/codec.mjs` routes every decode/encode of a file OUTSIDE the Anchor
2589
- shape but INSIDE the engine grammar through `src/rawle.mjs`, before the
2590
- shape gate. The result carries the codec's own fields (`discriminator` is
2591
- the dispatch prefix in hex; `dialect: 'raw-le'`), and every engine refusal
2592
- is translated into `CodecError` — one refusal family at the product
2593
- boundary, never a `RawLeError` leak, never a crash.
2594
- - `src/dialect.mjs` splits the question "can anything read this file?" from
2595
- "can the Anchor codec read it?": `unreadableDialectReason`/`assertReadableDialect`
2596
- pass an engine-readable file and keep the exact named refusal (code
2597
- `unsupported-idl-dialect`) for everything else; `idlDialect` reports
2598
- `'raw-le'` for a wired dialect.
2599
- - The store gates (`src/adopt.mjs` write path, `src/store-read.mjs` read
2600
- path) use the readable-dialect guard: a raw-LE object adopts under its
2601
- own program id, reads back, and the CLI loop `sememe adopt` → `sememe
2602
- decode` works end-to-end offline (measured 2/2). Mislabelling under a
2603
- foreign program id stays refused via `program_id` — the wiring opens the
2604
- dialect gate, not the re-labelling gate.
2605
- - The derivation (`src/discriminators.mjs`) passes the file WITHOUT filling
2606
- any Anchor sighash table (the PULS X fill defect stays dead — measured
2607
- filled {instructions: 0, events: 0}); the union (`src/union.mjs`)
2608
- decodes through the engine like through any source.
2609
-
2610
- Measured two-tree, byte-identical probe
2611
- ([`research/rawle-product-2026-11-12.json`](research/rawle-product-2026-11-12.json),
2612
- pinned in `test/rawle-product-measurement.test.mjs` with a full
2613
- gh-gated re-derivation): prefix tree = the PULS Y engine commit `05e6643`
2614
- (engine present, product refuses — 0/102, store loop closed), this tree =
2615
- 102/102 decoded AND 102/102 byte-exact re-encoded at `decodeInstructionData`,
2616
- 5/5 product-level refusals as named `CodecError` faces (unknown tag,
2617
- truncated payload, trailing bytes, unknown instruction on encode, missing
2618
- argument on encode), derivation passes with NO table, adopt/read/union open,
2619
- CLI store loop 2/2, Whirlpool control unchanged, a file that defeats BOTH
2620
- engines keeps the exact PULS X named refusal at codec AND union. 24
2621
- falsification counts, all zero. The PULS X measurement
2622
- ([`research/dialect-named-refusal-2026-11-12.json`](research/dialect-named-refusal-2026-11-12.json))
2623
- was re-derived in its invariant form: its delta now reads crash → decodes
2624
- through the engine, fill {4,0} → {0,0} — the no-crash, no-fill, no-mislabel
2625
- invariants pinned on every payload. The PULS Y artifact stays the frozen
2626
- pre-wiring baseline; its tool now exits 3 NO VERDICT in a wired tree BY
2627
- DESIGN (stated in the tool header).
2628
-
2629
- **Tests: 1 new pin file** (`test/rawle-product-measurement.test.mjs`, 12
2630
- tests) plus the bent faces in `test/dialect-named-refusal.test.mjs` (D1b),
2631
- `test/rawle-decoder.test.mjs` (G1) and
2632
- `test/dialect-named-refusal-measurement.test.mjs`. Suite 912 → **933: 929
2633
- pass / 4 named network skips / 0 fail**. Product code changed; the pin is
2634
- the next release.
2635
-
2636
- ## The wiring SHIPS in v0.4.22 — and the ecosystem number it moves is +102 for the user who adopts, not the shipped store (2026-11-12)
2637
-
2638
- This release carries the raw-LE wiring above to the install channel: the
2639
- pinned tag now contains `src/rawle.mjs` and every wired boundary, so
2640
- the v0.4.22 install degrades nobody (Whirlpool,
2641
- shank, name-only anchor controls pin unchanged) and answers the diabloarb
2642
- dialect where v0.4.21 refused it by name.
2643
-
2644
- The wiring also prompted the card's declared coverage re-measurement, and
2645
- the honest answer is TWO-HALVED (`tools/measure-diablo-adoption-coverage.mjs`
2646
- → `research/diablo-adoption-coverage-2026-11-12.json`, pinned by
2647
- `test/diablo-adoption-coverage.test.mjs`, 9 tests, gh-gated re-derivation
2648
- included):
2649
-
2650
- - **At the shipped store the headline does NOT move.** Re-running the
2651
- committed 2026-11-11 fixture through the coverage tool on this tree
2652
- reproduces the committed artifact exactly: **88.5%, DiabLokxis 0 decoded
2653
- / 102 no-interface.** The wiring changed the codec, not the store, and
2654
- the diabloarb copy cannot ship inside the store: `cryptodiablo/diabloarb`
2655
- carries **no license** at the pin `f5005ac4` — live re-checked inside the
2656
- pin test (GitHub license API 404 at the pin AND at HEAD, repo field
2657
- null), not narrated. The bytes stay out of the repository and the
2658
- package; `committedStoreModified: false` (readdir + per-file hash digest
2659
- before/after).
2660
- - **For the user who adopts, the 102 are real.** ONE `adopt` of the
2661
- hash-gated first-party copy (canonical fingerprint
2662
- `d6d4e8aa…` verified before anything runs) through the product's own
2663
- write path, into a throwaway copy of the committed store, and the SAME
2664
- report tool answers **102/102 decoded, 0 refused, 102 byte-exact**:
2665
- headline **88.5% → 89.0%** of measured traffic, outside the watchlist
2666
- **62.4% → 64.4%**. Every one of the 102 rides the **adopted** evidence
2667
- column (`decodedVia.adopted +102`; `pulled` moves by exactly zero) — the
2668
- same class split the pfeeU, manifest and archer measurements keep, so an
2669
- adoption can never silently inflate the on-chain-pull number.
2670
-
2671
- So "102 jetzt zählbar" was half true: countable **yes** — as adoption-path
2672
- coverage, with a name and a byte-exact check — and **not** as a shipped
2673
- headline. The acquisition gap at that program is a LICENSE gap, not a
2674
- decoder gap: the one thing between the user and those 102 payloads is a
2675
- licensed first-party copy, which is exactly what `sememe adopt` exists to
2676
- carry while origin stays declared.
2677
-
2678
- Suite after this release: **954 tests / 950 pass / 4 named network skips /
2679
- 0 fail** (945 → 954: the 9 new pin tests). CI green on `main`
2680
- (43287274405) and on the published tag v0.4.22 (43287274209); the release
2681
- surface re-measured against that tag is clean (newest = latest = pinned,
2682
- 0 defects — `research/release-surface-2026-11-12b.json`).
2683
-
2684
- ## v0.4.23 / v0.4.24: struct-internal absent fields decode under the evidence gate — and the registry stays truthful (2026-11-13)
2685
-
2686
- The absent-field policy (`absent: "report"`) only fired at TOP-LEVEL argument
2687
- boundaries, so a payload that ends on the boundary of the LAST FIELD INSIDE a
2688
- struct argument was refused under every policy. Measured on mainnet: Archer's
2689
- legacy Swap form (tag `0f`, 19 bytes) omits `SwapParams.builderFeePpm`
2690
- INSIDE `swapParams` and is chain-executed — witness `5Kpdai4oRG…`, slot
2691
- 468261978, `err: null`, instruction bytes byte-identical to the payload
2692
- (`research/archer-legacy-swap-witness-2026-11-12.json`; the pinned IDL's own
2693
- `SwapParams` docs describe the shorter form as supported and read as a 0 fee,
2694
- with the program's own `success` log line). One of the 83 payloads in
2695
- `research/archer-second-source-2026-11-12.json` hit exactly this and v0.4.22
2696
- refused it with `truncated input: need 4 byte(s) at offset 19`.
2697
-
2698
- This release decodes it and re-encodes it byte-exactly: inside a named struct,
2699
- under the same decode-only evidence gate (`ctx.__absentPolicy`), a boundary
2700
- ending mid-struct makes the remaining fields ABSENT, named by their dotted path
2701
- (`swapParams.builderFeePpm`), never invented; cuts INSIDE a field and boundary
2702
- cuts inside vec/array elements still refuse; the strict default policy is
2703
- untouched; `encodeInstructionData` accepts the dotted absent names (trailing
2704
- run per struct). `verifiedShortForm` now matches short (sub-8-byte) shank
2705
- discriminators by prefix — the rule the codec's instruction index already used
2706
- — so the 1-byte Archer tag reaches the chain-verified short-form table. Pinned
2707
- by `test/struct-absent-short-form.test.mjs` (10 tests: the falsification
2708
- ladder is 6 red on the v0.4.22 tree, green here).
2709
-
2710
- The registry channel is live and carries this release with the tag: the
2711
- foreign-run sweep measured `npm: found` for the bare name with the verdict
2712
- still `not-distributed-not-consumed`
2713
- (`research/foreign-run-2026-11-12.json`) — publish is not consumption, and
2714
- the first external run stays the open gate.
2715
-
2716
- Install — the registry and the tag ship the same product files (`npm install
2717
- sememe` resolves to 0.4.23, the one tag this README pins above):
2718
-
2719
- ```bash
2720
- npm install sememe # registry: 0.4.23, zero runtime dependencies
2721
- ```
2722
-
2723
- Limits, stated plainly: the Archer interface still does not ship inside the
2724
- store (no on-chain IDL; the Apache-2.0 first-party copy stays adopt-only and
2725
- hash-gated — unchanged); enum- and vec-body boundary cuts still refuse until a
2726
- chain witness names them (the rule stays evidence-gated; no speculative wire
2727
- form is accepted); tags <= v0.4.22 keep the truncation refusal (a tag is
2728
- immutable); one owner account, zero external users, no audit.
2729
-
2730
- Suite for this release: **964 tests / 960 pass / 4 named network skips /
2731
- 0 fail** (954 → 964: the 10 new pin tests). CI green on `main`
2732
- (43332583023); the release surface re-measured after publishing is CLEAN —
2733
- newest = latest = pinned, 0 defects (`research/release-surface-2026-11-13.json`),
2734
- and the registry carries the release as `latest` (measured after propagation).
2735
-
2736
- v0.4.23 (published 2026-11-13 00:49 UTC) carried this release to both channels
2737
- but its README held TWO install pins — the front page plus the historical
2738
- v0.4.22 section line — and the journey contract
2739
- (`test/base58-journey.test.mjs` and five siblings) requires exactly one:
2740
- tag CI 43340061132 RED with `README must pin exactly one tag, found v0.4.23,
2741
- v0.4.22` (394b37b is the fix). The v0.4.23 tag and its npm tarball are
2742
- published and immutable, so they keep the two-pin README; **v0.4.24 is the
2743
- corrected release** — same product bytes as v0.4.23 (no src/ change since
2744
- 3fa9627), one pin, journey suite green (27/27), registry `latest` = 0.4.24.
2745
-
2746
- ## The remaining 35 publish-nothing programs, classified against foreign sources (2026-11-04)
2747
-
2748
- With Lighthouse decoded under shank, exactly **35 programs (87 payloads)** remained
2749
- in the `no-interface-published` set. Rather than polishing the coverage headline,
2750
- `tools/classify-publish-nothing.mjs` measured where an interface for each one
2751
- could come from, scanning the foreign repository's own committed source
2752
- (`step-finance-oss/step-ingestooor @001cf39`, MIT; committed artifact:
2753
- `research/publish-nothing-classification-2026-11-04.json`, pinned in
2754
- `test/classify-publish-nothing.test.mjs`):
2755
-
2756
- | Category | Programs | Payloads | Nature |
2757
- |---|---|---|---|
2758
- | `first-party-foreign-idl` | **7** | **22** | First-party Anchor/Shank IDL in foreign repo (`crates/idl-gens/idls/`) declaring the program address |
2759
- | `dedicated-foreign-parser` | **9** | **32** | Custom hand-written Rust parser under `crates/parsers/src/` (Solend: 10, Saber: 5, Stake Pool: 4, etc.) — no published IDL exists |
2760
- | `core-runtime-program` | **3** | **5** | Core Solana runtime / sysvar programs (`Stake111...`, `Ed25519SigVerify...`, `noopb9bk...`) — no IDL by protocol design |
2761
- | `unrepresented-source-gap` | **16** | **28** | Appears in test fixture traces; neither an IDL nor a dedicated parser in the foreign repo |
2762
- | **Total** | **35** | **87** | Partition verified without remainder (pinned in `test/classify-publish-nothing.test.mjs`) |
2763
-
2764
- ### Seven first-party copies adopted, sweep re-measured
2765
-
2766
- The 7 first-party foreign IDLs were swept into the store via
2767
- `tools/adopt-foreign-first-party.mjs` through `sememe adopt` (`test/fixtures/traffic-sample-store-2026-11-04-pulled/`,
2768
- receipts source `first-party`, origin naming repo, commit and file; artifact:
2769
- `research/adopt-foreign-first-party-2026-11-04.json`, pinned in `test/adopt-foreign-first-party.test.mjs`):
2770
- Lulo (`FL3X2p...`), NX Lending (`NXFiKi...`), Solayer Restake (`sSo1iU...`),
2771
- Solayer AVS (`endoLN...`), Reward Options (`RopsJ7...`), Titan (`T1TANp...`),
2772
- Step Staking (`Stk5NC...`).
2773
-
2774
- Re-measurement of the full 156-fixture foreign set with identical flags
2775
- (`--absent error`, `research/step-ingestooor-full-sweep-classified-2026-11-04.json`,
2776
- pinned in `test/classified-sweep-report.test.mjs`):
2777
-
2778
- - **decoded:** 703 → **724** (+21 payloads decoded)
2779
- - **gap (`noInterface`):** 87 → **65** (-22 payloads; 21 decoded, 1 refused by name)
2780
- - **byteExactFull:** 696 → **716** (+20 full byte-exact)
2781
- - **byteExactDescribed:** 7 → **8** (+1 Titan swap described exact)
2782
- - **byteExactRate:** 100% (every decoded payload re-encodes byte-for-byte)
2783
- - **decodeErrors:** 16 → **17** — the single non-decoded payload is **REFUSED BY NAME**:
2784
- `T1TANpTeScyeqVzzgNViGDNrkQ6qHz9KrSBS4aNXvGT: Venue: unknown variant index 40` (an honest
2785
- refusal on an unrepresented enum variant, not a silent mis-parse).
2786
- - **disagreements:** 0 (still completely diff-free)
2787
- - Exactly the 7 adopted programs moved in `byProgram`; the remaining 28 programs
2788
- sum to exactly 65 payloads.
2789
-
2790
- The true acquisition gap for this sweep is now **65 payloads** across 28 programs —
2791
- all of which have NO published IDL (9 dedicated custom parsers, 3 core runtime,
2792
- 16 unrepresented trace programs).
2793
-
2794
- ### The 16 unrepresented trace programs, measured against external sources (2026-11-04)
2795
-
2796
- The 16 programs (28 payloads) were labeled `unrepresented-source-gap` because they
2797
- appeared in the foreign repository ONLY inside `crates/parsers/tests/**` transaction
2798
- traces (local premise check: 16/16 verified trace-only via `git grep HEAD`). Rather
2799
- than assuming they have no interface anywhere, `tools/measure-unrepresented-external.mjs`
2800
- scanned GitHub's code index for the literal program address, with a positive control
2801
- (`bitquery/solana-idl-lib solfi/solfi.json` declares `SoLFiHG9Tfgt...` -> 1 index match,
2802
- file readable and declares program -> control resolved; artifact:
2803
- `research/unrepresented-external-sweep-2026-11-04.json`, pinned in
2804
- `test/unrepresented-external.test.mjs`):
2805
-
2806
- - **4 programs (9 payloads)** have a published external interface whose file explicitly
2807
- **DECLARES** the program (`metadata.address` match, content-verified):
2808
- - SolFi (`SoLFiHG9TfgtdUXUjWAxi3LtvYuFyDLVhBWxdMZxyCe`, 4 payloads): `bitquery/solana-idl-lib solfi/solfi.json`
2809
- - ZeroFi (`ZERor4xhbUycZ6gb9ntrhqscUcZmAbQDjEAtCf4hbZY`, 1 payload): `bitquery/solana-idl-lib zerofi/zerofi.json`
2810
- - HumidiFi (`9H6tua7jkLhdm3w8BvgpTn5LZNU7g4ZynDmCiNN3q6Rp`, 1 payload): `bitquery/solana-idl-lib humidifi/humidifi.json`
2811
- - Manifest Wrapper (`wMNFSTkir3HgyZTsB7uqu3i7FA73grFCptPXgrZjksL`, 3 payloads): `Bonasa-Tech/manifest client/idl/wrapper.json`
2812
- - **12 programs (19 payloads)** have external references (code or docs) but no published IDL file.
2813
- - **0 unknown** (positive control resolved, no partial-sweep claims).
2814
-
2815
- ### External adoption preview (falsification against fixture bytes)
2816
-
2817
- Does a declared external interface actually decode the foreign repo's fixture bytes?
2818
- The claim was previewed via `tools/preview-external-adoption.mjs` on a throwaway store
2819
- copy (artifact: `research/external-adoption-preview-2026-11-04.json`, pinned in
2820
- `test/external-adoption-preview.test.mjs`):
2821
-
2822
- - **decoded:** 724 → **732** (+8 payloads decoded: SolFi 4/4, Manifest Wrapper 3/3, ZeroFi 1/1)
2823
- - **gap (`noInterface`):** 65 → **56** (-9 payloads resolved)
2824
- - **decodeErrors:** 17 → **18** — HumidiFi's single payload is **REFUSED BY NAME**:
2825
- `9H6tua7jkLhdm3w8BvgpTn5LZNU7g4ZynDmCiNN3q6Rp: unknown instruction discriminator f6dc6...`
2826
- (an honest refusal, not a silent mis-parse).
2827
- - **disagreements:** 0 (still completely diff-free).
2828
- - **byteExactRate:** 100% (all 8 decoded payloads reproduce byte-for-byte).
2829
- - **isolation:** exactly the 4 adopted programs moved in `byProgram`; 0 unadopted rows shifted.
2830
- - **license gate:** `bitquery/solana-idl-lib` has license `null` (no LICENSE file),
2831
- `Bonasa-Tech/manifest` is `GPL-3.0`. Neither may be copied into this MIT repository or
2832
- the published npm package. The preview therefore ran on a throwaway work-dir;
2833
- the committed store remains untouched (`committedStoreModified: false`, 76 receipts).
2834
-
2835
- ## Next qualification
2836
-
2837
- 1. Reproduce the classification with `npm ci && npm test && node tools/measure.mjs programs-20.json another-run.json` (or set `SOLANA_RPC_URL` to another mainnet RPC). The script checks canonical Program Metadata and legacy Anchor IDL accounts in one finalized batch, separates RPC failures from missing accounts, and checks PDA derivation against independently pinned fixtures from `auroraxo/idl-drift`. It is **not** a pull/store/decode implementation. An incomplete RPC run is not zero coverage; absences at these two addresses do not mean no interface exists elsewhere.
2838
- 2. **Whirlpool pull + content fingerprint has landed, the instruction decoder decodes real mainnet bytes, one own consumer reads the stored artifact for BOTH checking and producing, and the decoder is verified against an independent oracle across 1,000+ transactions** (see sections above): `solana-tx-remix` uses the pinned IDL for its single-program Whirlpool path (verifier + emitter), and `tools/tx-diff.mjs` verifies 1,119 real mainnet payloads from 1,029 transactions (sampled across 40,013 mainnet transactions in 148 finalized blocks) with 0 naming disagreements, 0 value disagreements, 0 byte mismatches, and 33 unexplained trailing-byte payloads (2.95%). Still to do, in order: migrate the remaining own decoder paths; then seek an outside user. The selected-sample threshold does not validate other program families. The `41 paths` figure above is an unverified note estimate; the computed scan in [research/decoder-sites-2026-10-27.json](research/decoder-sites-2026-10-27.json) reports **42 (file, class) decoder sites across 35 files in 5 own repositories**, of which 36 sit in `solana-tx-remix` + `sememe` — the counting unit matters and the two numbers are not interchangeable.
2839
- 3. **Account decoding has landed** (see "Accounts: state, not just calls"): 39 real mainnet accounts across three programs, 39/39 re-encoded byte-exact, values checked against anchor-ts, 29/29 account discriminators matching anchor-ts's own computation. Still open: `repr(C)`/`packed` accounts, non-Anchor account framing, and the report path for account bytes the IDL does not describe (no real data has exercised it yet — 0 trailing bytes observed).
2840
- 4. **Union decode has landed** (see "Working now: union decode"): `sememe adopt` puts a first-party published copy in the store, `sememe decode --union` answers from both verified sources with byte-evidence arbitration and a reported drift ledger — measured 915/996 decoded on real pump.fun traffic (the on-chain account alone decodes 669, the published copy alone 880). The declared-origin qualification has landed too (see "Working now: a declared origin is declared, and cannot be rewritten silently"): reader surfaces mark it as declared, the receipt carries its own digests, `sememe verify` names rewritten origins as relabelled, and the false collision on identical bytes is gone. **The account half has landed too (2026-11-08, see "Working now: account union"): `sememe account --union` arbitrates the same two sources over account bytes — measured on 12 real mainnet pump.fun accounts, the single-receipt path was byte-exact on 0 of 12 and the union path on 6 of 12, with the drift named per account.** A cross-program re-measurement over all 6 programs with account fixtures (74 accounts, `research/account-union-cross-2026-11-08.json`) put the honest bound on that number: recovery happened on 1 of 4 two-source programs, the other three agreed, disagreed without deciding, or could not name the account because the copy renames it — with 0 failures on the four regression invariants. Still open: verifying an adopted copy's provenance against its upstream source.
2841
- 5. Test a less family-clustered, independently selected sample and seek an outside user actually importing an interface before claiming this is a venture. A favorable fraction or self-run tests alone are not adoption.