cotterbot 0.1.0__tar.gz → 0.3.0__tar.gz

{cotterbot-0.1.0 → cotterbot-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cotterbot
-Version: 0.1.0
+Version: 0.3.0
 Summary: Compliance-testing framework for AI-controlled robot policies — pytest for robot policies.
 License-Expression: MIT
 License-File: LICENSE
@@ -9,11 +9,17 @@ Author-email: hyihan2007hong@gmail.com
 Requires-Python: >=3.11,<3.12
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
+Provides-Extra: falsify
+Provides-Extra: onnx
+Provides-Extra: stl
+Requires-Dist: cma (>=4.0,<5.0) ; extra == "falsify"
 Requires-Dist: gymnasium-robotics (>=1.3,<2.0)
 Requires-Dist: gymnasium[mujoco] (>=1.0,<2.0)
 Requires-Dist: mujoco (>=3.2,<4.0)
 Requires-Dist: numpy (>=1.26,<2.0)
+Requires-Dist: onnxruntime (>=1.17,<2.0) ; extra == "onnx"
 Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: rtamt (>=0.3,<0.4) ; extra == "stl"
 Requires-Dist: scipy (>=1.13,<2.0)
 Requires-Dist: stable-baselines3 (>=2.4,<3.0)
 Requires-Dist: torch (>=2.4,<3.0)
@@ -24,9 +30,11 @@ Description-Content-Type: text/markdown
 **Compliance testing for AI-controlled robot policies — pytest for robot
 policies.**
 
-Cotter loads a trained robot policy as a black box (observation → action),
-runs it through a battery of standardized tests in MuJoCo simulation, and
-produces structured pass/fail results with statistical guarantees. It is
+Cotter loads a trained robot policy as a black box (observation → action)
+— a Stable-Baselines3 `.zip`, a raw PyTorch `.pt`/`.pth` module, or an
+ONNX `.onnx` model — runs it through a battery of standardized tests in
+MuJoCo simulation, and produces structured pass/fail results with
+statistical guarantees. It is
 aimed at the emerging regulatory need (EU Machinery Regulation, ISO 10218)
 for evidence that a learned controller actually behaves — but the core is
 just honest, reproducible testing:
@@ -34,9 +42,10 @@ just honest, reproducible testing:
 | Category | Question | Method |
 |---|---|---|
 | **Performance** | Does it succeed at the task? | Wald's sequential probability ratio test (SPRT) — stops sampling as soon as the evidence is decisive |
-| **Safety** | Does it ever exceed a hard physical limit? | Per-timestep checks on joint velocities / actuator forces / contacts; a single violation anywhere fails, no averaging |
+| **Safety** | Does it ever exceed a hard physical limit? | Per-timestep checks on joint velocities / actuator forces / contacts; a single violation anywhere fails, no averaging. Includes an **ISO/TS 15066** power-and-force-limiting check (per-body-region contact-speed limits) |
 | **Regression** | Did the new version break behavior? | Matched pairs on a shared seed sequence, exact McNemar (binary) and Wilcoxon signed-rank (continuous) |
-| **Adversarial** | How bad is the worst case? | A PPO adversary trained to perturb the policy's *observations* within an L∞ budget, plus a guaranteed random-noise baseline |
+| **Adversarial** | How bad is the worst case? | A PPO adversary trained to perturb the policy's *observations* (sensor attack) or *actions* (actuator attack) within an L∞ budget, plus a guaranteed random-noise baseline |
+| **Specification** | Does it meet a declared behavioral requirement? | Signal Temporal Logic (STL) specs scored by *quantitative robustness* per rollout — e.g. `always (speed <= 1.5)`, `eventually (dist_to_goal <= 0.05)` |
 
 Everything runs on CPU (developed on Apple Silicon; no CUDA anywhere in
 the stack).
@@ -50,13 +59,27 @@ Requires Python 3.11. From PyPI (the distribution is published as
 pip install cotterbot
 ```
 
+To load **ONNX** policies, install the optional extra (pulls in `onnxruntime`):
+
+```sh
+pip install "cotterbot[onnx]"
+```
+
+> **Note:** on some systems the `cotter` binary lands outside your PATH (pip
+> will warn you with the exact directory). Either add that directory to PATH,
+> or invoke via `python -m cotter` instead:
+>
+> ```sh
+> python -m cotter run --policy policy.zip --config run.yaml
+> ```
+
 Or from source with [Poetry](https://python-poetry.org/):
 
 ```sh
 git clone https://github.com/yih0nk/cotter.git
 cd cotter
 poetry install
-poetry run pytest   # 213 tests, unit + real-MuJoCo integration
+poetry run pytest   # 435 tests, unit + real-MuJoCo integration
 ```
 
 ## Quickstart (CLI)
@@ -103,6 +126,224 @@ confidence interval (SPRT and adversarial results), and regression
 results carry an effect size — the discordant-pair odds ratio for
 McNemar, the rank-biserial correlation for Wilcoxon.
 
+#### Adversarial attack surface
+
+The adversarial category attacks either surface, set by
+`adversarial.attack`:
+
+- `observation` (default) — perturb what the victim *sees* (sensor noise,
+  calibration drift).
+- `action` — perturb what the victim *does* (actuator noise, control-channel
+  corruption), clipped back into the action space.
+- `both` — run both; the report carries distinct `action_*` entries.
+
+Each surface runs a guaranteed random baseline and, when `train: true`, a
+PPO adversary trained against the frozen victim.
+
+### Specifications (Signal Temporal Logic)
+
+Beyond the fixed categories, declare behavioral requirements in **Signal
+Temporal Logic** and Cotter scores each rollout by *quantitative
+robustness* — a signed margin (positive = satisfied, negative = violated,
+and by how much), not just pass/fail. Requires the `stl` extra
+(`pip install cotterbot[stl]`):
+
+```yaml
+stl:
+  n_episodes: 20
+  variables:
+    speed: cotter/tcp_speed                          # scalar info key
+    jv0:   {key: cotter/joint_velocities, index: 0}  # a vector component
+  threshold: 0.0            # min robustness required to pass (0 = the STL boundary)
+  specs:
+    - name: speed_cap
+      formula: "always (speed <= 1.5)"
+    - name: settles
+      formula: "eventually (speed <= 0.1)"
+```
+
+Each spec becomes a `specification`-category result reporting the worst
+(minimum) episode robustness. Signals are pulled from the per-step `info`
+dict by the `variables` map. STL operators (`always`, `eventually`,
+`until`, bounded `[a,b]`) are evaluated offline with
+[`rtamt`](https://github.com/nickovic/rtamt).
+
+### Falsification (active worst-case search)
+
+The categories above *sample* scenarios; falsification *searches* for the
+worst one. Point CMA-ES at an STL-robustness objective and it hunts the
+initial condition / disturbance / parameter vector that most violates a
+spec — returning the concrete counterexample. Requires the `falsify`
+extra (`pip install cotterbot[falsify]`); a Python-API feature, since the
+scenario→env mapping is code:
+
+```python
+import cotter
+
+# how a scenario vector configures the env (masses, frictions, disturbance...)
+def apply_scenario(env, params):
+    env.unwrapped.model.body_mass[1] = params[0]      # e.g. sweep a link mass
+
+objective = cotter.stl_scenario_objective(
+    policy, env_factory, apply_scenario,
+    formula="always (tcp_speed <= 1.5)",
+    variables={"tcp_speed": {"key": "cotter/tcp_speed"}},
+)
+result = cotter.falsify(objective, lower=[0.5], upper=[5.0], max_evaluations=200)
+if result.falsified:
+    print("counterexample scenario:", result.best_params)
+```
+
+`falsify` minimizes the objective and reports whether it dropped below the
+threshold (0 = a spec violation). Derivative-free and CPU-only — each
+evaluation is one rollout; 1-D spaces use a deterministic sweep, higher
+dimensions use CMA-ES.
+
+### Coverage sweeps (operating-envelope coverage)
+
+A green report shouldn't be able to hide untested regions. Coverage
+sweeps the scenario space with a low-discrepancy **Sobol** sequence and
+reports the metric *as a function of the scenario* — the failure fraction
+and worst region, not one aggregate number. Same `objective` as
+falsification (sweep it here, search it there), CPU-only, no extra:
+
+```python
+import cotter
+
+result = cotter.sweep(
+    objective,                       # e.g. cotter.stl_scenario_objective(...)
+    lower=[0.5, 0.0], upper=[5.0, 1.0],   # the operating envelope
+    n_scenarios=64,
+)
+print(result.summary())              # "covered 64 scenarios, 7 below threshold (11% failure region) ..."
+for o in result.outcomes:            # every (scenario, value, pass/fail) — a coverage map
+    ...
+```
+
+Coverage-of-operating-envelope reporting is something regulators ask for
+(UNECE-style ODD coverage) but no open policy-eval tool provides.
+
+For the common case — coverage across the **observation-disturbance**
+envelope — a `coverage` config section runs it as part of `cotter run`
+(no code), sweeping a constant observation offset in the L∞ box and
+reporting a `coverage`-category result:
+
+```yaml
+coverage:
+  epsilon: 0.1          # L∞ budget on the observation offset
+  n_scenarios: 32
+  n_episodes: 10
+  min_success_rate: 0.8  # a scenario fails below this
+```
+
+### ISO/TS 15066 power-and-force-limiting (PFL)
+
+Collaborative-robot safety comes down to a concrete numeric question: at
+its speed, would a contact exceed the human injury threshold for the body
+region it might hit? Cotter checks it directly. Declare an
+`iso_ts_15066` section:
+
+```yaml
+iso_ts_15066:
+  m_robot: 5.0                 # effective robot mass at the TCP (kg), incl. payload
+  speed_key: cotter/tcp_speed  # per-step info key holding TCP linear velocity
+  contact_type: transient      # transient (dynamic) or quasi_static (clamping)
+  regions: [chest, hand_finger, skull_forehead]   # omit for the full library
+```
+
+For each region, Cotter back-solves the maximum permissible contact speed
+from the ISO/TS 15066 collision model — `F = v·√(k·μ)`, reduced mass
+`1/μ = 1/m_robot + 1/m_human` — and flags any timestep whose TCP speed
+would produce a force over that region's limit, reporting the binding
+(most restrictive) region and the implied contact force.
+
+> The bundled per-region limits (`cotter.ISO_TS_15066_LIMITS`) are the
+> widely-published approximations and are **user-editable** — confirm them
+> against the purchased ISO/TS 15066 before relying on them for
+> certification.
+
+### Clause traceability
+
+Conformity assessment is clause-by-clause. Declare which checks evidence
+each regulatory clause and Cotter reports every clause as verified,
+failed, or **unverified** — the coverage-gap view an auditor asks for:
+
+```yaml
+traceability:
+  EHSR-1.3.7:
+    description: "moving parts of machinery"
+    checks: [hard_limits, iso_ts_15066_pfl]
+  AI-Act-15:
+    description: "robustness & accuracy"
+    checks: [learned_ppo, speed_cap]
+```
+
+A clause is *failed* if any of its checks failed, *verified* if it has a
+passing check and none failed, else *unverified* (no evidence). The result
+is a `traceability`-category entry that fails the report if any clause is
+failed or unverified. (The certified clause *thresholds* are the paid
+layer; this open mapping ties your checks to the clauses you declare.)
+
+### Reports (JSON + HTML + JUnit + Markdown)
+
+Every run prints a console summary and can write four artifacts:
+
+- **JSON** (`report:` in the config, or `--report`) — the machine-readable
+  evidence, one object per category with the full statistics.
+- **HTML** (`report_html:` in the config, or `--report-html`) — a single
+  self-contained page you can open in a browser, drop into a pull request,
+  or attach to a demo. No external assets, no JavaScript, adapts to the
+  viewer's light/dark theme.
+- **JUnit XML** (`report_junit:` in the config, or `--report-junit`) — each
+  category as a `<testcase>`, so a Cotter run shows up as native test
+  results in GitHub Actions, GitLab, Jenkins, CircleCI, etc.
+- **Markdown** (`report_md:` in the config, or `--report-md`) — a shareable
+  plain-text report to paste into a PR, an issue, or a wiki.
+
+```sh
+poetry run cotter run \
+    --policy artifacts/victim_ppo_inverted_pendulum.zip \
+    --config examples/inverted_pendulum.yaml \
+    --report report.json --report-html report.html
+```
+
+The HTML page is the *free* counterpart to the paid compliance layer: the
+open engine renders the pass/fail evidence into something shareable; the
+licensed layer renders it into a signed, regulator-ready technical file.
+
+### Reproducibility manifest
+
+Every report (schema v2+) carries a `manifest` recording the full
+provenance of the run — so a result can be reproduced and audited, not
+just read:
+
+```json
+"manifest": {
+  "cotter_version": "0.2.0",
+  "python_version": "3.11.9",
+  "platform": "Linux-6.5.0-x86_64",
+  "dependencies": { "gymnasium": "1.0.0", "mujoco": "3.2.3", "torch": "2.4.1", ... },
+  "env_id": "InvertedPendulum-v5",
+  "base_seed": 0,
+  "policy_path": "artifacts/victim.zip",
+  "policy_sha256": "sha256:1f3a…"
+}
+```
+
+The report also includes `content_sha256`, a digest over the report body
+(excluding the timestamp and the digest itself). It is a stable id and a
+tamper-evidence check.
+
+Verify a report's integrity with `cotter verify`:
+
+```sh
+cotter verify report.json                 # recompute + check content_sha256
+cotter verify report.json --policy p.zip  # also re-hash the policy vs the manifest
+```
+
+Exit 0 means verified, 1 means a check failed (the report or policy was
+modified), 2 means the report is missing or invalid.
+
 ### Comparing two policy versions
 
 `cotter compare` runs only the regression test between a baseline and a
@@ -115,6 +356,24 @@ poetry run cotter compare \
     --config examples/inverted_pendulum.yaml
 ```
 
+Like `run`, it accepts `--report` (JSON) and `--report-html` to write the
+comparison result to disk — handy for archiving the artifact or feeding a
+PR comment.
+
+### Diffing reports (safety budget)
+
+Where `compare` re-runs the regression test, `cotter diff` compares two
+existing **reports** and fails if any previously-passing check now fails —
+a safety-budget gate across the *whole* battery, not just regression:
+
+```sh
+cotter diff baseline_report.json candidate_report.json
+```
+
+Each check is classified regressed / improved / added / removed; exit 1
+means a check that used to pass now fails (0 otherwise, 2 on a bad
+report). Store a baseline report and diff every retrain against it.
+
 ### Parallel rollouts
 
 Safety, regression, and fixed-N adversarial evaluation parallelize across
@@ -162,7 +421,7 @@ from cotter import (
 #    magnitudes in every step's info dict.
 env = CotterWrapper(gym.make("InvertedPendulum-v5"))
 
-# 2. Load the policy under test (SB3 .zip or a raw torch .pt module).
+# 2. Load the policy under test (SB3 .zip, raw torch .pt, or .onnx).
 #    Observation/action spaces are validated and mismatches fail loudly.
 policy = load_policy("artifacts/victim_ppo_inverted_pendulum.zip", env, algo=PPO)
 
@@ -184,13 +443,14 @@ safe = evaluate_safety(rollouts.episode_infos, [
 
 adv = run_adversarial_test(policy, env, success, epsilon=0.07, n_episodes=20)
 
-# 4. Aggregate into a report (console summary + JSON artifact).
+# 4. Aggregate into a report (console summary + JSON + HTML artifacts).
 report = TestReport(policy_name="my_policy", env_id="InvertedPendulum-v5")
 report.add_sprt(perf)
 report.add_safety(safe)
 report.add_adversarial(adv)
 print(report.summary())
 report.to_json("report.json")
+report.to_html("report.html")   # self-contained, shareable page
 ```
 
 ## Demo
@@ -347,8 +607,28 @@ Trained adversaries are expensive and specific to what they attack, so
 adversarial section and the first run trains and stores the attacker;
 later runs on the same victim reuse it instead of retraining. The victim
 hash is taken over the policy artifact (or its parameter tensors), so a
-retrained victim never silently reuses a stale adversary. A hosted zoo
-of pretrained expert adversaries is the planned paid tier.
+retrained victim never silently reuses a stale adversary.
+
+### Pretrained adversaries (transfer attacks)
+
+`cotter.zoo.PretrainedZoo` is a different registry: keyed by **robot
+class** — `(robot_class, env_id, epsilon, attack)` — not by victim. An
+adversary trained per class transfers to *any* compatible victim on that
+env (adversarial policies exploit the environment, not victim-specific
+weights), so you can attack your own policy without training one:
+
+```yaml
+adversarial:
+  epsilon: 0.05
+  pretrained: franka-panda    # transfer-attack from the pretrained zoo
+```
+
+The run loads the class adversary, checks it matches the env, and reports
+a `pretrained_<class>_<surface>` category (a missing class or incompatible
+adversary is skipped, never fatal). Inspect the registry with
+`cotter pretrained list`. The open engine ships the mechanism
+(`register` / `transfer_attack`); a hosted library of GPU-trained experts
+per robot class is the paid tier.
 
 ## Compliance layer (paid tier)
 
@@ -380,7 +660,8 @@ cotter/
 │   ├── registry.py      # resolves gymnasium-robotics (and other) extension envs
 │   └── factory.py       # picklable env factory for parallel rollouts
 ├── report.py            # TestReport: console summary + JSON (results container only)
-├── zoo/                 # adversary registry keyed by (env, victim, epsilon)
+├── render.py            # self-contained HTML rendering of a TestReport (free tier)
+├── zoo/                 # adversary registries: victim-keyed cache + class-keyed pretrained
 ├── compliance/          # paid-tier regulatory document stub (license required)
 └── tests/
     ├── sprt.py          # Wald's sequential probability ratio test (+ CI)