robustrep 0.1.1__tar.gz → 0.2.0__tar.gz

{robustrep-0.1.1 → robustrep-0.2.0}/CHANGELOG.md

@@ -4,6 +4,115 @@ All notable changes to robustrep are recorded here. The format follows
 [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project is pre-1.0,
 so a `0.x` bump may change published numbers without changing the API.
 
+## [0.2.0] — 2026-10-12
+
+Adaptive `eth_getLogs` fetching. On 2026-10-12 mainnet.base.org began answering
+every `eth_getLogs` with HTTP 429 (`-32011 request limit reached`), even for a
+100-block window on a first request. The remaining free endpoints each cap the
+window differently (publicnode serves roughly the last 5,000 blocks, nodies
+50 blocks per call but deep history, drpc about 100), so fixed 2,000-block
+chunks could not back-fill the roughly 1.2M blocks since the last report. `fetch`
+now learns each endpoint's window at runtime and records which host served
+what. Scoring is unchanged.
+
+### Added
+
+- **`LogFetcher` (`robustrep/sources/log_fetcher.py`).** Sits between
+  `sync_feedback` and `RpcClient`: given a block range it returns all logs or
+  raises `RpcError`. It learns each endpoint's window cap (shrinks by at least
+  half on a range-too-large error, in one step to the cap when the server
+  names one), marks an endpoint dead for the run when it rejects the minimum
+  window (default 50 blocks), rotates past dead endpoints, and bounds invalid
+  rounds so it always terminates. `LogFetcher.from_config(rpc, cfg)` is the
+  construction path shared by the CLI and the live tests.
+- **`sync_log` table and `Store.sync_sources()`.** One row per synced
+  sub-window of a chunk (`from_block, to_block, host, window, n_logs, fetched_at`), written by
+  delete-intersecting-then-insert so reruns are idempotent;
+  `sync_sources()` aggregates per host.
+- **`rpc_sources` report provenance** (last in `PROVENANCE_KEYS`). `report.md`
+  gets a `Log sources:` line and a `Source recording starts at block ...`
+  line under Provenance (`Log sources: not recorded ...` for data fetched
+  before this version), and an always-present Limitations bullet, `Log sources
+  not cross-checked.` RPC endpoint hostnames appear in the public report and
+  `scores.json`; do not use an endpoint whose hostname embeds an account id or
+  key.
+- **`Config` fields:** `rpc_window_hints` (tuple of `(host, cap)` pairs,
+  default empty), `rpc_min_window` (50) and `rpc_min_interval` (0.5 s between
+  call starts; this is the CLI default). The library default (`LogFetcher`'s
+  `min_interval`) is 0; `RpcClient` has no pacing.
+- **`RpcRangeTooLargeError(endpoint, numbers)`**, raised immediately (no
+  rotation or backoff) on provider range-cap errors. Its message is fixed
+  text; only parsed integers leave the module. `RpcClient` also gains
+  `current_url` and `rotate()`.
+- **`Store.failed_ranges()`**, a read-only view of the recorded failed
+  ranges, used by the live suite.
+
+### Changed
+
+- **Default `rpc_urls` is now four endpoints, in rotation order:**
+  `https://mainnet.base.org`, `https://base-rpc.publicnode.com`,
+  `https://base-pokt.nodies.app`, `https://base.drpc.org`. `--rpc-url`
+  (repeatable) still replaces the whole list. Report provenance does not
+  record this list; it records `rpc_sources`, the hosts that actually served
+  each block range.
+- **`sync_feedback(..., fetcher=None)`.** With a fetcher it records
+  `sync_log`; without one it behaves as before and records no sources. The
+  CLI always passes one.
+- **Live suite: an unusable endpoint is a labelled failure, never a skip.**
+  `rpc_available()` and the new `check_failed_ranges()` report
+  `RPC unavailable: rate limited` / `transport failure: ...` with `pytest.fail`
+  instead of `pytest.skip`. A run that cannot reach the chain has checked
+  nothing, and a skip would have hidden the mainnet.base.org change for weeks;
+  the owner prefers the email. The label (never server-supplied text) says
+  whether to look at the endpoint list or at the data. Genuine data failures
+  propagate unchanged. Both live tests now go through `LogFetcher.from_config`.
+- **`--chunk` smaller than the minimum window still works**; it caps the
+  window and is not raised to 50.
+
+### Fixed
+
+- **Live suite saw a throttled `eth_getLogs` as a bare checkpoint failure.**
+  `sync_feedback` records a chunk whose fetch failed in `failed_ranges` and
+  carries on rather than raising, so the availability guard never saw the 429
+  and the test failed on `last_block` with no hint why (weekly run of
+  2026-10-12). The test now reads the recorded errors through
+  `Store.failed_ranges()` and classifies them with the same markers; a
+  throttled sibling range can never hide a real one.
+- **JSON-RPC error bodies carried by HTTP 400 are now classified.**
+  base.drpc.org answers an over-cap `eth_getLogs` with HTTP 400 and a JSON-RPC
+  error (`ranges over 10000 blocks are not supported on free plan`).
+  `RpcClient` raised on the status before reading the body, so the range error
+  was never recognised and drpc's window was never learned; every rotation to
+  it re-sent a full-width request that failed. A 400 whose body is a JSON
+  object with an `error` now goes through the same classification as a 200
+  with an error (range too large, non-retryable, or retry and rotate). A 400
+  without such a body, and every 403, 429 and 5xx, is still an HTTP error as
+  before.
+- **Default CLI pacing is 0.5 s between `eth_getLogs` calls (was 0.2 s).** The
+  2026-10-12 back-fill showed nodies throttling at about 5 calls per second
+  sustained (104 HTTP 429s). `LogFetcher`'s library default stays 0.
+
+### Rollback
+
+To go back to 0.1.2 (no source recording, no automatic rotation), install the
+old release first: `pip install 'robustrep==0.1.2'`, then
+`robustrep fetch --db data/base.db --chunk 50 --rpc-url https://base-pokt.nodies.app`;
+the database is compatible in both directions (0.1.2 ignores `sync_log`;
+blocks fetched under 0.1.2 have no source rows).
+
+## [0.1.2] — 2026-10-03
+
+A dependency security release. No API, CLI or scoring changes.
+
+### Security — Fixed
+
+- **urllib3 floor raised to 2.8.** urllib3 2.6–2.7 can enter an infinite loop
+  on a chunked deflate stream (PYSEC-2026-4176) and buffer an unbounded
+  chunk-size line into memory (PYSEC-2026-4177). Both are reachable from
+  `fetch`, which reads responses from attacker-chosen evidence hosts, so a
+  hostile URI could hang or exhaust the fetcher. 2.8 also fixes HTTPS-proxy TLS
+  settings being ignored (PYSEC-2026-4175). The CI pin moves to 2.8.0 as well.
+
 ## [0.1.1] — 2026-09-15
 
 A security release: it closes every finding of an external security review of
@@ -114,5 +223,7 @@ no longer rescales a group by its own extremes.
 Initial release on PyPI: the scoring library, the Base ERC-8004 adapter, and the
 reproducible report.
 
+[0.2.0]: https://github.com/iamdongwang/robustrep/compare/v0.1.2...v0.2.0
+[0.1.2]: https://github.com/iamdongwang/robustrep/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/iamdongwang/robustrep/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/iamdongwang/robustrep/releases/tag/v0.1.0

{robustrep-0.1.1/robustrep.egg-info → robustrep-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: robustrep
-Version: 0.1.1
+Version: 0.2.0
 Summary: Robust, transport-agnostic reputation scoring for AI agents
 Author: Dong Wang
 License: Apache-2.0
@@ -25,7 +25,7 @@ Requires-Dist: numpy>=1.24
 Requires-Dist: eth-abi>=5.0
 Requires-Dist: eth-hash[pycryptodome]>=0.5
 Requires-Dist: requests>=2.31
-Requires-Dist: urllib3>=2.6
+Requires-Dist: urllib3>=2.8
 Requires-Dist: tabulate>=0.9
 Requires-Dist: matplotlib>=3.7
 Requires-Dist: typer>=0.12
@@ -68,6 +68,10 @@ result = score(records, Config())
     robustrep score  --db data/base.db --out scores.csv
     robustrep report --db data/base.db --out-dir reports
 
+The default endpoint list is rotated automatically and the `eth_getLogs` window adapts per
+endpoint; `--rpc-url` (repeatable) replaces the list and `--chunk` caps the window.
+RPC endpoint hostnames appear in the public report and `scores.json`; do not use an endpoint whose hostname embeds an account id or key.
+
 Rater profiles: Blockscout (free, no key) by default; Etherscan V2 requires a paid plan for Base.
 Pick a source explicitly with `fetch --profile-source {auto,blockscout,etherscan,none}`.
 

{robustrep-0.1.1 → robustrep-0.2.0}/README.md

@@ -29,6 +29,10 @@ result = score(records, Config())
     robustrep score  --db data/base.db --out scores.csv
     robustrep report --db data/base.db --out-dir reports
 
+The default endpoint list is rotated automatically and the `eth_getLogs` window adapts per
+endpoint; `--rpc-url` (repeatable) replaces the list and `--chunk` caps the window.
+RPC endpoint hostnames appear in the public report and `scores.json`; do not use an endpoint whose hostname embeds an account id or key.
+
 Rater profiles: Blockscout (free, no key) by default; Etherscan V2 requires a paid plan for Base.
 Pick a source explicitly with `fetch --profile-source {auto,blockscout,etherscan,none}`.
 

{robustrep-0.1.1 → robustrep-0.2.0}/constraints.txt

@@ -98,7 +98,7 @@ typer==0.27.2
 typing-inspection==0.4.4
 typing_extensions==4.16.0
 tzdata==2026.4
-urllib3==2.7.0
+urllib3==2.8.0
 zipp==4.1.0
 # -- hand-added (not emitted by `pip freeze` on the dev box) -----------------
 # pip itself: freeze omits it. CI installs this version explicitly before the

{robustrep-0.1.1 → robustrep-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "robustrep"
-version = "0.1.1"
+version = "0.2.0"
 description = "Robust, transport-agnostic reputation scoring for AI agents"
 readme = "README.md"
 license = { text = "Apache-2.0" }
@@ -31,7 +31,10 @@ dependencies = [
   # decompression bomb (`max_length` on urllib3's decoders) first exists in
   # 2.6.0. On 2.0-2.5 the same call decompresses the whole body before
   # truncating -- measured at a 455 MB peak against 0.8 MB on 2.7.
-  "urllib3>=2.6",
+  # >=2.8: 2.6-2.7 can loop forever on a chunked deflate stream and buffer an
+  # unbounded chunk-size line (PYSEC-2026-4176/4177), both reachable from an
+  # attacker-chosen evidence host.
+  "urllib3>=2.8",
   "tabulate>=0.9",
   "matplotlib>=3.7",
   "typer>=0.12",

{robustrep-0.1.1 → robustrep-0.2.0}/robustrep/cli.py

@@ -37,6 +37,7 @@ from .report.publish import (EVIDENCE_CAP_PER_URI_KEY, EVIDENCE_CAP_TOTAL_KEY, b
 from .report.sensitivity import sensitivity_table
 from .sources import base_erc8004 as base
 from .sources.evidence_batch import DEFAULT_MAX_TOTAL_LOOKUPS, classify_all
+from .sources.log_fetcher import LogFetcher
 from .sources.rater_profile import DEFAULT_RPS, default_client, enrich_raters, estimate_seconds
 from .sources.rpc import RpcClient, RpcError
 from .store import Store
@@ -125,7 +126,8 @@ def _rpc_guard():
 
 def _step_sync(store: Store, rpc: RpcClient, cfg: Config, to_block: Optional[int]) -> str:
     """Sync feedback/revocation/response events up to ``to_block`` (or head)."""
-    n = base.sync_feedback(store, rpc, chunk=cfg.chunk_blocks, end_block=to_block, confirmations=cfg.confirmations)
+    n = base.sync_feedback(store, rpc, chunk=cfg.chunk_blocks, end_block=to_block,
+                           confirmations=cfg.confirmations, fetcher=LogFetcher.from_config(rpc, cfg))
     return f"feedback rows added: {n}"
 
 
@@ -530,6 +532,8 @@ def report(
             # Same reason: provenance publishes the caps the fetch run
             # recorded in *this* store, not this build's constants.
             evidence_caps = evidence_caps_for_provenance(store)
+            # And which RPC endpoints served the logs, from this store's sync_log.
+            sync_sources = store.sync_sources()
         block = int(last_block) if last_block is not None else 0
         # Inside the guard: sensitivity_table re-scores the whole dataset once
         # per variant, so any ValueError scoring can raise, it can raise too.
@@ -541,7 +545,8 @@ def report(
     adv = scenario_table()
     provenance = build_provenance(mode, cfg, records, result,
                               n_lookup_budget_starved=n_lookup_budget_starved,
-                              evidence_caps=evidence_caps)
+                              evidence_caps=evidence_caps,
+                              sync_sources=sync_sources)
 
     block_dir = out_dir / str(block)
     # Guarded like the scoring above: `export_json` refuses an address-bearing

{robustrep-0.1.1 → robustrep-0.2.0}/robustrep/config.py

@@ -59,9 +59,25 @@ class Config:
     bootstrap_seed: int = 0
     ci_level: float = 0.95
     # data source
-    rpc_urls: tuple[str, ...] = ("https://mainnet.base.org",)
+    # Measured 2026-10-12: mainnet.base.org answers every eth_getLogs with 429;
+    # publicnode serves only ~the last 5000 blocks; nodies caps at 50 blocks per
+    # call but serves deep history; drpc caps at ~100. RpcClient rotates on
+    # failure and sticks, LogFetcher learns each host's window.
+    rpc_urls: tuple[str, ...] = (
+        "https://mainnet.base.org",
+        "https://base-rpc.publicnode.com",
+        "https://base-pokt.nodies.app",
+        "https://base.drpc.org",
+    )
+    # (scheme://hostname, max block window) pairs; a tuple so Config stays hashable.
+    rpc_window_hints: tuple[tuple[str, int], ...] = ()
+    rpc_min_window: int = 50
+    # Seconds between eth_getLogs call starts (CLI default; LogFetcher's own is 0).
+    # nodies throttled at ~5 calls/s sustained during the 2026-10-12 back-fill
+    # (104 HTTP 429s), so 0.2 s was too tight.
+    rpc_min_interval: float = 0.5
     chunk_blocks: int = 2000
-    user_agent: str = "robustrep/0.1 (+https://github.com/iamdongwang/robustrep)"
+    user_agent: str = "robustrep/0.2 (+https://github.com/iamdongwang/robustrep)"
     confirmations: int = DEFAULT_CONFIRMATIONS
 
     def __post_init__(self) -> None:
@@ -90,5 +106,11 @@ class Config:
             raise ValueError("chunk_blocks must be >= 1")
         if len(self.rpc_urls) < 1:
             raise ValueError("rpc_urls must have at least 1 entry")
+        if any(w < 1 for _, w in self.rpc_window_hints):
+            raise ValueError("rpc_window_hints values must be >= 1")
+        if self.rpc_min_window < 1:
+            raise ValueError("rpc_min_window must be >= 1")
+        if self.rpc_min_interval < 0:
+            raise ValueError("rpc_min_interval must be >= 0")
         if self.confirmations < 0:
             raise ValueError("confirmations must be >= 0")

{robustrep-0.1.1 → robustrep-0.2.0}/robustrep/report/provenance_keys.py

@@ -31,4 +31,5 @@ PROVENANCE_KEYS = (
     "evidence_max_lookups_per_uri",
     "evidence_max_total_lookups",
     "evidence_lookup_starved_uris",
+    "rpc_sources",
 )

{robustrep-0.1.1 → robustrep-0.2.0}/robustrep/report/publish.py

@@ -22,19 +22,20 @@ says about the run that made it is part of the artifact, not part of the CLI.
 """
 from __future__ import annotations
 
+import dataclasses
 import logging
 import os
 import shutil
 import tempfile
 from pathlib import Path
-from typing import List, Optional
+from typing import List, Optional, Union
 
 import pandas
 
 from ..config import Config
 from ..evidence import MAX_TX_LOOKUPS_PER_URI
 from ..sources.evidence_batch import DEFAULT_MAX_TOTAL_LOOKUPS
-from ..store import Store
+from ..store import Store, SyncSources
 from .export import export_json, versions
 from .figures import fig_evidence, fig_mean_vs_robust, fig_rank_shift, fig_sybil_clusters
 from .provenance_keys import PROVENANCE_KEYS
@@ -173,10 +174,22 @@ def evidence_caps_for_provenance(store: Optional[Store]) -> dict:
     return caps
 
 
+def _rpc_sources_for_provenance(sync_sources: Optional[SyncSources]) -> Union[str, dict]:
+    """`config["rpc_sources"]`: the store's per-host sync_log summary, or
+    `UNAVAILABLE` when there is no store or it recorded no source at all."""
+    if sync_sources is None or sync_sources.recorded_from_block is None:
+        return UNAVAILABLE
+    return {
+        "sources": [dataclasses.asdict(source) for source in sync_sources.sources],
+        "recorded_from_block": int(sync_sources.recorded_from_block),
+    }
+
+
 def build_provenance(mode: str, cfg: Config, records: pandas.DataFrame,
                 result: Optional[pandas.DataFrame] = None,
                 n_lookup_budget_starved: int = 0,
-                evidence_caps: Optional[dict] = None) -> dict:
+                evidence_caps: Optional[dict] = None,
+                sync_sources: Optional[SyncSources] = None) -> dict:
     caps = evidence_caps if evidence_caps is not None else evidence_caps_for_provenance(None)
     config = dict(
         bootstrap_n=cfg.bootstrap_n,
@@ -204,6 +217,10 @@ def build_provenance(mode: str, cfg: Config, records: pandas.DataFrame,
         # caller with no store to count from (e.g. `build_provenance` called
         # directly on an in-memory `records` frame, as several tests do).
         evidence_lookup_starved_uris=n_lookup_budget_starved,
+        # Which RPC endpoints served the logs, per Store.sync_sources. Last,
+        # and UNAVAILABLE (not an empty list) when this store never recorded
+        # a source: "not recorded" and "recorded, nothing" are different claims.
+        rpc_sources=_rpc_sources_for_provenance(sync_sources),
     )
     # PROVENANCE_KEYS is the published order, shared with `report.render` so the
     # two artifacts cannot drift (a test pins them equal). Ordering through it