cow-backtester 0.9.0__tar.gz

cow_backtester-0.9.0/CHANGELOG.md

@@ -0,0 +1,136 @@
+# Changelog
+
+## 0.9.0 — 2026-08-18
+
+**`--reward-ev`: CIP-85 v2 consistency economics.** Capture ratios and ranks
+measure competitiveness; solver income on most CoW chains is the consistency
+pool. This release computes the actual v2 metric
+(`Σ executed orders: your_best_fair_surplus / Σ all_solvers_surplus`) from
+the same competition records `--compete` already fetches:
+
+- the challenger's counterfactual metric, per-order surpluses inserted into
+  the historical denominators (conservatively — field terms keep their
+  historical values, so the reported share is a floor);
+- every FIELD solver's real historical metric — a consistency leaderboard
+  for the window (also useful for spotting pool-dilution patterns);
+- with `--self-address`, your actual historical metric side by side with
+  the replayed one;
+- the win floor, reported explicitly: v2 pays ZERO on a chain where the
+  solver won nothing in the period, so the estimate never hides it;
+- `--consistency-budget <COW>` converts the share into a COW/week estimate.
+
+Honesty labels throughout: field surpluses come from record amounts (net of
+protocol fees) while a replayed challenger's are gross (a few bps flattering,
+labeled `basis`); challenger fairness is assumed (the field uses CoW's own
+`filteredOut` flags); success_rate is a flag, not a silent multiplier.
+
+
+## 0.8.0 — 2026-08-17
+
+**`--compete`: rank your solver against the historical field.** The v2
+competition endpoint serves per-auction records by auction id (probed:
+retention ≥ 2 months) carrying every submitted solution — solver, score,
+ranking, CoW's own fairness-filtering outcome (`filteredOut`), per-solution
+clearing prices, reference scores, and the auction's start/deadline blocks.
+
+- `--compete` fetches each scored auction's record and inserts the
+  challenger's surplus into the fairness-surviving score list: per-auction
+  `field_rank` (rank, field size, gap-to-winner bps, winner solver), and a
+  per-solver summary — rank-1 %, top-3 %, median rank/gap, and a rivals
+  table (who beat you, how often, by how much). Honesty label everywhere:
+  `rank_basis: surplus_vs_score_proxy` — historical scores include protocol
+  fees, challenger surplus does not, so the rank is a floor.
+- `--archive-dir DIR` persists every fetched record as
+  `DIR/<chain>/<auction_id>.json.gz` (idempotent) — a local competition
+  dataset that outlives the API's retention window and feeds future
+  fairness-simulation / reward-EV work.
+- `--self-address 0x…` adds shadow-vs-actual: when your historical
+  solverAddress appears in a record, the row carries its actual score,
+  ranking, and fairness outcome next to the replayed one.
+- Rows gain `auction_start_block` / `auction_deadline_block` — the auction
+  CUT context (what bidders saw), complementing `settlement_block`.
+
+## 0.7.2 — 2026-08-17
+
+Scoring-fidelity batch from an independent line-level audit. Every item
+below changes reported numbers or their labels — see README caveats.
+
+- **Coverage-adjusted capture is the new headline.** Solver errors now keep
+  the historical winner's surplus in the denominator (previously an errored
+  auction vanished from the ratio — a solver could look better by failing
+  hard auctions). The old ratio is kept as `capture_conditional_pct`
+  ("when it answered, how competitive was it"), and a new
+  `lost_to_errors_wei` reports winner surplus forfeited to errors/timeouts.
+- **Exact best-combination optimizer.** The greedy highest-first selection
+  of internally compatible solutions undercounted challengers (A=10 on two
+  pairs beats B+C=6+6 split — greedy picked 10, optimum is 12). Now solved
+  exactly (branch-and-bound; greedy fallback above 20 candidates, labeled
+  via `combiner`).
+- **U256 range enforcement.** `to_u256` accepted arbitrarily large Python
+  ints; values above 2^256-1 are now rejected as invalid.
+- **Scoring-basis labels.** Every row carries `baseline_quality`
+  (`exact_uniform` / `wrapper_lower_bound` / `mixed`); a same-basis
+  `capture_exact_basis_pct` (direct settlements only) is reported alongside
+  the mixed-basis proxy, and the HTML cards are relabeled accordingly
+  ("Winner fee take" → "Known direct-path fee wedge").
+- **Per-transaction wrapper attribution.** UID-overlap proof is now enforced
+  per settlement transaction, not per auction group (one attributed tx can
+  no longer vouch for an unrelated one; drops count as `tx_uid_mismatch`).
+- **A/B on byte-identical bodies.** One shared `deadline` per auction is
+  computed before the solver loop (was regenerated per request); rows carry
+  `replay_deadline`.
+- **Readiness semantics.** `{"solutions": []}` counts as a healthy answer
+  (schema-legitimate abstention) with bid coverage reported separately;
+  "valid" wording clarified to auctions with ≥1 valid solution; READY now
+  requires `--min-evidence` attempted auctions (default 10) — below that the
+  best verdict is REVIEW. Readiness is now rendered in the HTML report.
+- **`settlement_block`** replaces the row field `block` (kept one release as
+  a deprecated alias): it is the settlement's block, NOT the auction cut
+  block — README fork guidance corrected to match.
+- Docs: `--max-auctions` keeps the **newest** auctions (behavior since
+  0.7.1; help/README said oldest).
+
+## 0.7.1 — 2026-08-16
+
+- Wrapper-routed settlements (solver router contracts, ~43% of mainnet /
+  ~40% of Base settlements) are attributed via the v2 by-tx-hash endpoint,
+  proven by UID overlap against the S3 auction body, and scored on a
+  clearly-labeled delivered basis (`entry: "wrapper"`). Previously excluded
+  silently.
+- Submitter identification prefers the competition `solverAddress` over the
+  relay EOA. Coverage disclosure in the scorecard, `--readiness`, and a
+  `_meta` line in `--json-out`. `--max-auctions` keeps the newest auctions.
+
+## 0.7.0 — 2026-08-12
+
+- `--readiness`: a one-screen pre-production readiness check for a solver
+  endpoint. Replays recent auctions against the endpoint and reports answer
+  rate, latency (p50/p95/max vs the solve budget), solution validity, and
+  surplus captured against the on-chain winners, with a READY / REVIEW /
+  NOT READY verdict and a copy-paste command to reproduce the exact run.
+  A focused alternative to the full field scorecard for anyone evaluating a
+  solver before staging or shadow. Emitted in the `--json-out`/`--html-out`
+  summary under `readiness`.
+
+## 0.6.0 — 2026-08-07
+
+First public release.
+
+- Replays archived CoW auctions (the public S3 instance bucket) against any
+  solver's `/solve` endpoint, offline.
+- Reconstructs each auction's winning set from on-chain settlement calldata
+  and Trade events; groups multiple winning transactions per auction (CIP-67);
+  optional cross-check against the v2 competition API (`--verify-api`).
+- Scores both sides on the same basis: before-fee surplus over signed limits
+  at uniform clearing prices, converted at the auction's reference prices.
+- Validates solver responses the way the settlement layer would: per-order
+  fill accounting, fill-or-kill exactness, fee-adjusted limit feasibility,
+  strict U256 parsing; infeasible solutions are excluded and tallied.
+- A/B mode: repeatable `--solver-url`, rotating call order, head-to-head
+  panel, per-pair breakdown, solve-latency stats.
+- Ten chains, content cache, concurrent fetching, streamed JSONL rows,
+  single-file HTML report, watch mode.
+- 37 offline tests against pinned fixtures; CI runs them plus lint and a
+  packaging check on Python 3.10 and 3.12.
+
+Pre-release development history is summarized in `docs/DESIGN_NOTES.md`.

cow_backtester-0.9.0/CONTRIBUTING.md

@@ -0,0 +1,25 @@
+# Contributing
+
+Corrections are the most valuable contribution — especially anywhere this
+tool's accounting diverges from what the protocol actually does. Every scoring
+rule cites its source (GPv2 contracts or `cowprotocol/services`); if you can
+show a rule is wrong, please open an issue with the primary-source reference.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python3 -m pytest          # offline suite — no network, must always pass
+python3 -m ruff check .    # lint — must be clean
+python3 -m cow_backtester.scorer --selftest && python3 -m cow_backtester --selftest  # network tests
+```
+
+Ground rules:
+- The offline suite stays offline. New scoring behavior needs a fixture-based
+  test, and exploit-class regressions (duplicate trades, fee padding, negative
+  numerics, malformed shapes) must keep passing.
+- No silent drops: anything excluded from scoring gets a tallied reason that
+  reaches the scorecard.
+- No new runtime dependencies without strong cause (`eth_abi` is the only one).
+- The pinned reference values (auction 8339027: event 2385773, surplus 1471,
+  fee 805) are load-bearing; never adjust them to make a test pass.

cow_backtester-0.9.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 kaisersolver
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cow_backtester-0.9.0/MANIFEST.in

@@ -0,0 +1,5 @@
+graft fixtures
+graft tests
+include mock_solver.py
+include CHANGELOG.md CONTRIBUTING.md LICENSE requirements.txt
+global-exclude __pycache__ *.pyc

cow_backtester-0.9.0/PKG-INFO

@@ -0,0 +1,411 @@
+Metadata-Version: 2.4
+Name: cow-backtester
+Version: 0.9.0
+Summary: Offline backtester, A/B harness, and counterfactual scorecard for CoW Protocol solvers
+Author: kaisersolver
+License-Expression: MIT
+Project-URL: Repository, https://github.com/kaisersolver/cow-backtester
+Project-URL: Issues, https://github.com/kaisersolver/cow-backtester/issues
+Project-URL: Changelog, https://github.com/kaisersolver/cow-backtester/blob/main/CHANGELOG.md
+Keywords: cow-protocol,solver,backtesting,mev,defi
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: eth_abi<6,>=5
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Dynamic: license-file
+
+# cow-backtester
+
+Offline backtester, A/B harness, and counterfactual scorecard for CoW Protocol
+solvers.
+
+Point it at your own solver's `/solve` endpoint and it replays recent CoW
+auctions against it, then scores each of your solutions against the set of
+solutions that actually won on-chain. Point it at two endpoints and it runs
+a head-to-head A/B on identical auctions, so a routing or config change can
+be judged before it goes to production.
+
+Nothing touches production: no shadow mode, no staging deployment, no keys.
+
+## Why this exists
+
+Today a solver can only be evaluated *live*: shadow mode consumes the
+production auction stream, and the local playground runs against a chain fork.
+Neither lets you take a fixed set of recent auctions, run your solver against
+them offline, and ask "would I have out-surplused the winners, and by how
+much?" the way you'd backtest a trading strategy. This tool does.
+
+## Quick start
+
+Python 3.10+.
+
+```bash
+pip install .          # installs the `cow-backtester` command (one dependency: eth_abi)
+
+# 1. Baseline — what the field actually captured (no solver needed)
+cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc>
+
+# 2. Counterfactual — replay through your solver
+cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc> \
+        --solver-url http://localhost:8080 --json-out results.jsonl
+
+# 3. A/B — two solvers, same auctions, head-to-head
+cow-backtester --chain base --blocks 5000 --rpc-url <your-rpc> \
+        --solver-url http://localhost:8080 --solver-name baseline \
+        --solver-url http://localhost:8081 --solver-name candidate \
+        --html-out ab.html
+```
+
+From a source checkout without installing, `python3 -m cow_backtester ...`
+works identically. Your endpoint only needs the standard CoW solver-engine
+API (`POST /solve`).
+
+## Readiness check
+
+If you are bringing up a new solver and want a fast "is this endpoint healthy
+enough to face production auctions?" read, `--readiness` prints a one-screen
+report instead of the full field scorecard:
+
+```bash
+cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc> \
+        --solver-url http://localhost:8080 --solver-name mine --readiness
+```
+
+It replays recent auctions against your endpoint and reports the four things
+that gate a pre-prod solver — does it answer, is it fast enough, are its
+solutions valid, and are they competitive with the on-chain winners — as
+pass/warn checks with a `READY` / `REVIEW` / `NOT READY` verdict:
+
+```
+====================================================================
+  READINESS — mine   [REVIEW]
+  base · prod · blocks 49506127..49510000
+====================================================================
+  [PASS] reached auctions         50 auctions attempted
+  [WARN] no transport errors      2/50 errored
+  [PASS] answers reliably         96% returned a solution
+  [PASS] inside the deadline      0 past deadline
+  [PASS] latency headroom         p95 420 ms of a 15000 ms budget
+  [PASS] solutions are valid      98% passed limit/fee checks
+  [PASS] competitive vs winners   61% of winner surplus captured
+```
+
+It prints the exact `--from-block/--to-block` command to reproduce the run,
+and the same data lands in `--json-out`/`--html-out` under `readiness`. Works
+on any of the supported chains, so you can readiness-check an endpoint for a
+chain you are not yet onboarded on. It is a signal, not a settlement
+guarantee — pair it with a self-hosted shadow run before going to production.
+
+## Consistency economics (`--reward-ev`)
+
+On most CoW chains the money is not in winning — it is in the CIP-85
+consistency pool, which pays
+`success_rate × Σ executed orders ( your best fair bid's surplus / everyone's )`.
+`--reward-ev` computes exactly that metric from the competition records:
+your solver's counterfactual metric and pool share, every field solver's
+real historical metric (a consistency leaderboard for the window), and,
+with `--consistency-budget <COW>`, a COW/week estimate. The win floor is
+reported explicitly — v2 pays zero on a chain where a solver won nothing in
+the period — and every number carries its basis label (field surpluses are
+net of protocol fees, a replayed challenger's are gross; challenger
+fairness is assumed while the field uses CoW's own `filteredOut` flags).
+
+## Rank against the historical field (`--compete`)
+
+Capture ratio tells you how much surplus you generate; `--compete` tells you
+**where you would have ranked**. For every scored auction it fetches the
+historical competition record (every submitted solution with its score,
+CoW's own fairness-filtering outcome, and the winner) and inserts your
+solver's result into the fairness-surviving score list:
+
+```bash
+cow-backtester --chain base --blocks 2000 --rpc-url <your-rpc> \
+        --solver-url http://localhost:8080 --solver-name mine --compete \
+        --archive-dir ./competition-data
+```
+
+The scorecard gains a field-rank line (rank-1 %, top-3 %, median rank,
+median gap to the winner in bps) and a rivals table — which solvers beat
+you, how often, and by how much. Per-auction `field_rank` lands in the JSON
+rows. Two honesty notes: historical scores include protocol fees while your
+replayed surplus does not, so the reported rank is a **floor** (labeled
+`surplus_vs_score_proxy`); and records exist only for auctions that had a
+winner. `--archive-dir` keeps every fetched record on disk — the API serves
+roughly two months of history, so an archive you build today is a dataset
+you keep. `--self-address` marks your historical solverAddress so rows show
+shadow-vs-actual side by side.
+
+### Try it without a solver
+
+The bundled mock solver lets you see the full counterfactual/A/B output in
+about a minute, before wiring up your own engine:
+
+```bash
+MODE=limit  python3 mock_solver.py 8901 &   # fills exactly at the limit
+MODE=better python3 mock_solver.py 8902 &   # fills at 1.5x the limit
+
+cow-backtester --chain base --blocks 1500 --rpc-url <your-rpc> \
+        --solver-url http://127.0.0.1:8901 --solver-name at-limit \
+        --solver-url http://127.0.0.1:8902 --solver-name better --html-out demo.html
+kill %1 %2
+```
+
+You get the per-solver panels, the head-to-head table, and the pair
+breakdown. The `implausible_surplus` guard usually fires on the "better"
+mock: claiming 1.5x the limit on real order flow is exactly what the
+validator exists to flag.
+
+## How it works (all public data)
+
+1. Input: the exact `/solve` bodies CoW sent solvers, from the public S3
+   instance bucket (`solver-instances.s3.amazonaws.com/<env>/<chain>/auction/<id>.json`).
+   The archived file *is* a valid `/solve` request body.
+2. Winner reconstruction: from the on-chain settlements. (The v1
+   `/solver_competition` endpoints were removed in the competition-data
+   migration; v2 exists but is retention-bounded. Reconstructing from chain
+   data is trustless and independent of API retention — and `--verify-api`
+   cross-checks it against v2 where available.) Settlements are grouped by
+   auction: since combinatorial auctions (CIP-67), one auction can settle
+   through several winning transactions, so the baseline is the combined
+   winning set.
+3. Replay and score: each auction body is POSTed to each solver; responses
+   are validated (below) and scored with the same function, the same
+   limits, and the same price convention as the winners. Your valid solutions
+   are combined the CIP-67 way (best-first, disjoint directed token pairs).
+
+## The comparison basis (read this before quoting numbers)
+
+Both sides are scored on **before-fee surplus over the signed order limits, at
+uniform clearing prices**, converted to the chain's native token at the
+auction's own reference prices.
+
+* Signed limits: CoW's accounting scores against the order's signed amounts
+  (`fullSellAmount`/`fullBuyAmount`), not the remaining/fee-adjusted amounts.
+* Uniform prices: on-chain settlements carry per-trade post-fee "custom"
+  prices; the first occurrence of each token in the price vector is the uniform
+  (pre-fee) price, which is what autopilot itself resolves. Scoring the winner
+  post-fee but a challenger pre-fee would bias every comparison.
+* Surplus token: buy token for sell orders, sell token for buy orders,
+  converted at that token's `referencePrice` (matches official accounting).
+* What the number is: user surplus + protocol fees + network fee. CoW's
+  official ranking score is user surplus + protocol fees only, so absolute
+  levels here overstate the official score by the network fee (on the pinned
+  reference trade, a $2.4 fill where the network fee dominates: 1471 vs an
+  official 666 atoms; the gap shrinks as trades grow). When two solutions carry
+  very different fees this basis can order them differently than CoW would;
+  gas-heavy solutions are flattered. It is the only basis we found that can
+  be computed symmetrically offline for both sides.
+
+## Response validation (your solver can't accidentally cheat)
+
+A solution is scored only if **every** fulfillment trade is feasible:
+
+* known order uid (case-insensitive) and strict U256 numerics (decimal or
+  0x-hex; negatives and malformed values rejected);
+* prices present for both tokens;
+* at most one fulfillment per order (GPv2 accumulates fills and the driver
+  rejects duplicate trades, so N copies of a trade can't score N times the
+  surplus);
+* no over-fill (`executed + fee` vs `sellAmount` for sell orders, `executed`
+  vs `buyAmount` for buy orders; exact for fill-or-kill);
+* on-chain feasibility at the fee-adjusted terms: the net delivery must still
+  cover the gross-scaled limit, which is what GPv2 enforces, so a padded `fee`
+  cannot manufacture surplus.
+
+Violations mark the whole **solution INVALID** (tallied with a reason), never a
+silent zero. Solutions scoring >10× the winning set (or >0.001 native when the
+winning set scored 0) are flagged `implausible_surplus`. Malformed responses
+are tallied, never crash a run. **Prices are claimed, not simulated** — the
+tool checks feasibility; it does not execute routes.
+
+## What a run prints
+
+* COVERAGE: settlements found, auctions formed/scored, every skip with its
+  reason, unscanned block spans, auction ages, cache stats, `--verify-api` results.
+* FIELD SURPLUS: winner surplus by USD size bucket, plus the winners' fee take.
+* WINNING SUBMITTERS: who is actually winning (label them with `--solver-map`).
+* COUNTERFACTUAL (per solver): returned / valid / positive / beat the
+  winning set, surplus sums, capture ratio, solve latency p50+p95, invalid
+  reasons, error classes.
+* HEAD TO HEAD (exactly two solvers): side-by-side metrics, per-auction
+  win counts, and the surplus delta.
+* TOP PAIRS: winner surplus by token pair with each solver's surplus
+  beside it: *where* you win and lose, not just by how much.
+
+A real single-solver run (Arbitrum, eight auctions, trimmed):
+
+```
+====================================================================
+  COUNTERFACTUAL — kaisersolver  (8 replayed)
+====================================================================
+  returned a solution : 7/8
+  valid solutions     : 7/8
+  positive surplus    : 7/8
+  beat the winning set: 0/8
+  our surplus (sum)   : 0.001643 ETH
+  winners (sum)       : 0.001748 ETH  (replayed auctions only)
+  capture ratio       : 94.0% of the winning set
+  solve latency       : p50 1174 ms / p95 2471 ms
+
+====================================================================
+  TOP PAIRS BY WINNER SURPLUS (top 2)
+====================================================================
+                        pair  trades         winner   kaisersolver
+                   WETH->MOR       7       0.001722       0.001643
+                 USDC->USD₮0       1       0.000026       0.000000
+```
+
+Eight auctions is a small sample, but the shape is the point: this solver is
+consistently a few percent behind one competitor on one pair, and the table
+says which pair. That took minutes to learn here; it took weeks from logs.
+
+`--json-out` streams one row per auction (block, timestamp, age, winner txs +
+submitters + surplus/fees, per-solver validation detail and latency,
+`expired_orders_pct`, flags). `--html-out` writes a single self-contained HTML
+report: no external assets, light/dark aware, fine to attach to a PR or post.
+
+## Flags
+
+| flag | meaning |
+|---|---|
+| `--chain` | `mainnet`, `arbitrum-one`, `base`, `xdai`, `polygon`, `bnb`, `avalanche`, `linea`, `ink`, `plasma` |
+| `--env` | `prod` (default) or `staging` |
+| `--blocks N` | scan the most recent N blocks |
+| `--from-block/--to-block` | absolute, reproducible window |
+| `--max-auctions N` | cap auctions scored, newest first (default 25; 0 = all) |
+| `--rpc-url` | your RPC (recommended — public defaults rot and rate-limit) |
+| `--solver-url` / `--solver-name` | repeatable; two of them = A/B |
+| `--solve-timeout S` | deadline advertised to the solver (HTTP waits S+5) |
+| `--workers N` | concurrent RPC/S3 fetches (default 8) |
+| `--cache-dir` / `--no-cache` | content cache (default `.cowbt-cache`) |
+| `--json-out` / `--html-out` | machine-readable rows / HTML report |
+| `--solver-map FILE` | JSON `{address: name}` to label winning submitters |
+| `--verify-api` | cross-check winner txs against the v2 competition API |
+| `--compete` | rank the challenger against the historical fairness-surviving field (rank / gap / rivals) |
+| `--archive-dir DIR` | persist competition records as `DIR/<chain>/<id>.json.gz` (local dataset) |
+| `--self-address 0x…` | your historical solverAddress → shadow-vs-actual comparison |
+| `--reward-ev` | CIP-85 v2 consistency economics: counterfactual metric, field leaderboard, COW estimate (implies `--compete`) |
+| `--consistency-budget N` | the chain's weekly consistency pool in COW, to convert share → COW/week |
+| `--min-evidence N` | attempted-auction floor before `--readiness` may say READY (default 10) |
+| `--clamp-validto` | extend expired `validTo` so engines that filter them still solve |
+| `--max-age-hours H` | warn when replayed auctions are older than this |
+| `--watch N` | continuous mode: rescan every N seconds from the last block |
+| `--quiet` | suppress progress (stderr); the scorecard still prints to stdout |
+| `--version` | print version |
+
+Progress goes to **stderr**, the scorecard to **stdout** (`2>/dev/null` gives
+clean results; `--json-out`/`--html-out` are unaffected). Exit codes: **0**
+success, **1** runtime error, **2** usage error, **130** interrupted mid-run
+(a partial scorecard is printed when auctions had already been processed;
+stopping `--watch` during its idle sleep is a clean stop and exits 0).
+
+## Caching
+
+Only **immutable** facts are cached: archived auction bodies, settlements below
+the reorg margin, and block timestamps. Nothing derived from live liquidity is
+ever cached, so a warm re-run is faster (3.8x on a measured six-auction,
+two-solver run) without changing a single number. Delete `.cowbt-cache` any time, or pass `--no-cache`.
+
+## Limitations
+
+* **Replay uses live liquidity.** Your solver quotes against *current* chain
+  state, not the historical block. The counterfactual is **indicative** — it
+  answers "how does my solver handle this real order flow", not "the exact
+  outcome at that block". Ages are printed and a warning fires beyond
+  `--max-age-hours` (default 6h). Prefer recent, short windows. Each row
+  carries `settlement_block` — the block the winning settlement landed in,
+  which is *later* than the auction cut block the bidders actually saw. A
+  fork pinned there is an approximation, not a faithful auction replay (it
+  can even include the settlement itself); true historical-fork replay needs
+  the auction cut block, which this tool does not yet reconstruct.
+* **The S3 bucket retains roughly one month** of auctions (measured Aug 2026).
+  This is a recent-window backtester, not an archive.
+* **Entrypoint coverage.** Direct `settle()` calls are attributed by the
+  auction id appended to their calldata. Wrapper-routed settlements (solver
+  router contracts — measured Aug 2026 at **~43% of mainnet, ~40% of Base,
+  ~2% of Arbitrum** settlements, and *larger* than direct ones at the median,
+  so they are not a random slice) are attributed via the v2
+  `solver_competition/by_tx_hash` endpoint and **proven by uid overlap with
+  the S3 auction body** before scoring; they score on a DELIVERED basis
+  (Trade events vs signed limits), which understates the direct-path
+  before-fee basis by the settlement's fee wedge (typically a few bps) —
+  rows carry `entry: "wrapper"` so the bases are distinguishable. Settlements
+  the endpoint cannot resolve are counted under `wrapper_unattributed` and
+  disclosed in the coverage block, `--readiness`, and the `_meta` JSON line.
+* **"Beat the winning set" is necessary, not sufficient.** Real winner
+  selection also applies fairness filters, and bids score net of gas; the tool
+  also takes your solutions at face value while the driver merges and simulates.
+* **Orders keep their historical `validTo`** (~1%/day expire; more near the
+  retention edge). Engines that filter expired orders look age-degraded — see
+  `expired_orders_pct` per row, or pass `--clamp-validto`.
+* **JIT orders.** The winner baseline credits surplus-capturing JIT trades
+  (owners in `surplusCapturingJitOrderOwners`), per official accounting; the
+  challenger side never credits JIT (a response's JIT order has no verifiable
+  owner). On auctions where winner surplus comes from CoW-AMM-style JIT, the
+  comparison is conservative *against* the challenger, never in its favor.
+* **Bring your own RPC.** Public defaults rot and rate-limit `eth_getLogs`; the
+  tool splits and retries failing ranges and always **reports** any span it
+  could not scan rather than under-counting silently. `eth_chainId` is checked
+  against `--chain` on startup.
+
+## Provenance of the reconstruction
+
+* Auction id: CoW's driver appends it to `settle()` calldata; autopilot reads
+  back exactly the **last 8 bytes** (`META_DATA_LEN = 8` in
+  `cowprotocol/services`). The tool applies the same rule, only when the
+  calldata prefix re-encodes canonically, cross-checked by requiring **at least
+  one** settled order uid to appear in the fetched body (`body_uid_mismatch`
+  guard — this also catches staging/prod contamination, which shares the
+  settlement contract).
+* Trade events are filtered by **emitting address + topic** and must align 1:1
+  with calldata trades, else the settlement is skipped (`trade_event_mismatch`).
+* Fill-or-kill: GPv2 ignores the calldata `executedAmount` and uses the signed
+  amount — the tool applies the same substitution.
+* Response-schema authority: `crates/solvers-dto` / the solver-engine OpenAPI
+  in `cowprotocol/services`.
+* `--verify-api` compares the reconstructed winning-tx set per auction against
+  the v2 competition endpoint (match / mismatch / unavailable counters).
+
+## Tests
+
+```bash
+pip install -e ".[dev]"                      # pytest + ruff (one-time)
+python3 -m pytest                            # OFFLINE suite (fixtures; no network)
+python3 -m cow_backtester.scorer --unittest  # offline core checks
+python3 -m cow_backtester.scorer --selftest  # network: winner path on the reference settlement
+python3 -m cow_backtester --selftest         # network: counterfactual == winner baseline +
+                                             # every exploit class as regression cases
+```
+
+The pinned reference (Arbitrum auction 8339027): decode integrity is exact
+against the GPv2 `Trade` event (2385773 atoms), before-fee surplus 1471 atoms,
+fee take 805. `mock_solver.py` (modes: empty / limit / better / hex / garbage /
+invalid) exercises the wire path including validation and implausibility
+guards. CI runs the offline suite on Python 3.10 and 3.12.
+
+## Files
+
+| file | role |
+|---|---|
+| `cow_backtester/scorer.py` | settlement decode: auction id + winner surplus |
+| `cow_backtester/backtest.py` | enumerate, group, replay, validate, scorecard |
+| `cow_backtester/cache.py` | immutable-fact content cache |
+| `cow_backtester/report.py` | single-file HTML report |
+| `mock_solver.py` | test/demo solver (six modes) |
+| `fixtures/` | pinned reference data for the offline tests |
+| `tests/` | offline pytest suite, no network |
+| `docs/DESIGN_NOTES.md` | design decisions and validation history |
+
+MIT licensed. Contributions and corrections welcome — especially from CoW core
+devs on anything where this tool's accounting diverges from the protocol's.
+</content>