cotterbot 0.1.0__tar.gz

cotterbot-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 yihan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cotterbot-0.1.0/PKG-INFO

@@ -0,0 +1,409 @@
+Metadata-Version: 2.4
+Name: cotterbot
+Version: 0.1.0
+Summary: Compliance-testing framework for AI-controlled robot policies — pytest for robot policies.
+License-Expression: MIT
+License-File: LICENSE
+Author: yihan
+Author-email: hyihan2007hong@gmail.com
+Requires-Python: >=3.11,<3.12
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Requires-Dist: gymnasium-robotics (>=1.3,<2.0)
+Requires-Dist: gymnasium[mujoco] (>=1.0,<2.0)
+Requires-Dist: mujoco (>=3.2,<4.0)
+Requires-Dist: numpy (>=1.26,<2.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: scipy (>=1.13,<2.0)
+Requires-Dist: stable-baselines3 (>=2.4,<3.0)
+Requires-Dist: torch (>=2.4,<3.0)
+Description-Content-Type: text/markdown
+
+# Cotter
+
+**Compliance testing for AI-controlled robot policies — pytest for robot
+policies.**
+
+Cotter loads a trained robot policy as a black box (observation → action),
+runs it through a battery of standardized tests in MuJoCo simulation, and
+produces structured pass/fail results with statistical guarantees. It is
+aimed at the emerging regulatory need (EU Machinery Regulation, ISO 10218)
+for evidence that a learned controller actually behaves — but the core is
+just honest, reproducible testing:
+
+| Category | Question | Method |
+|---|---|---|
+| **Performance** | Does it succeed at the task? | Wald's sequential probability ratio test (SPRT) — stops sampling as soon as the evidence is decisive |
+| **Safety** | Does it ever exceed a hard physical limit? | Per-timestep checks on joint velocities / actuator forces / contacts; a single violation anywhere fails, no averaging |
+| **Regression** | Did the new version break behavior? | Matched pairs on a shared seed sequence, exact McNemar (binary) and Wilcoxon signed-rank (continuous) |
+| **Adversarial** | How bad is the worst case? | A PPO adversary trained to perturb the policy's *observations* within an L∞ budget, plus a guaranteed random-noise baseline |
+
+Everything runs on CPU (developed on Apple Silicon; no CUDA anywhere in
+the stack).
+
+## Install
+
+Requires Python 3.11. From PyPI (the distribution is published as
+`cotterbot`; the import name, CLI, and project are all still `cotter`):
+
+```sh
+pip install cotterbot
+```
+
+Or from source with [Poetry](https://python-poetry.org/):
+
+```sh
+git clone https://github.com/yih0nk/cotter.git
+cd cotter
+poetry install
+poetry run pytest   # 213 tests, unit + real-MuJoCo integration
+```
+
+## Quickstart (CLI)
+
+Declare the test battery in YAML and point `cotter run` at a policy:
+
+```sh
+poetry run cotter run \
+    --policy artifacts/victim_ppo_inverted_pendulum.zip \
+    --config examples/inverted_pendulum.yaml
+```
+
+Exit code 0 means every declared category passed; 1 means at least one
+failed; 2 means a config/usage error. Real captured output from the
+command above (2026-07-07, seed 0 — trial-by-trial progress lines
+elided):
+
+```
+[cotter] loaded policy 'victim_ppo_inverted_pendulum' onto InvertedPendulum-v5
+[cotter] performance: SPRT p0=0.8 p1=0.95 n_max=50
+[cotter]   => PASS after 18 trials
+[cotter] safety: 4 limit(s) over 20 episodes
+[cotter]   worst |cotter/joint_velocities| = 0.6789 (limit 5.0)
+[cotter]   worst |cotter/actuator_forces| = 0.9062 (limit 2.5)
+[cotter]   worst |cotter/contact_count| = 0.0000 (limit 0.5)
+[cotter]   worst |cotter/contact_forces| = 0.0000 (limit 1.0)
+[cotter]   => PASS
+[cotter] regression: vs baseline .../victim_ppo_inverted_pendulum.zip on 30 paired seeds
+[cotter]   => McNemar NO_REGRESSION (p=1), Wilcoxon NO_REGRESSION (p=1)
+[cotter] adversarial: eps=0.07 over 20 episodes
+[cotter]   random baseline: 100% clean -> 100% perturbed
+[cotter]   ppo adversary: 100% clean -> 0% perturbed
+[cotter] JSON report written to .../artifacts/cli_report.json
+...
+OVERALL: FAIL (1 failing, 5 passing, 0 informational)
+```
+
+(The FAIL is the learned-adversary category doing its job; the config's
+regression section compares the victim against itself as a sanity check,
+hence p = 1.) The config file schema is documented in
+[`examples/inverted_pendulum.yaml`](examples/inverted_pendulum.yaml) and
+`cotter/config.py`. Every reported success rate carries a Clopper-Pearson
+confidence interval (SPRT and adversarial results), and regression
+results carry an effect size — the discordant-pair odds ratio for
+McNemar, the rank-biserial correlation for Wilcoxon.
+
+### Comparing two policy versions
+
+`cotter compare` runs only the regression test between a baseline and a
+candidate and exits 0 (no regression) or 1 (regression detected) — a
+drop-in CI gate:
+
+```sh
+poetry run cotter compare \
+    --baseline old_policy.zip --candidate new_policy.zip \
+    --config examples/inverted_pendulum.yaml
+```
+
+### Parallel rollouts
+
+Safety, regression, and fixed-N adversarial evaluation parallelize across
+worker processes via `AsyncVectorEnv`. Set `n_workers` in the relevant
+config section (default 1 = serial); SPRT stays serial by design because
+it must check its boundary after each trial. Parallel results are
+bit-identical to serial on the same seeds — each episode runs in an
+isolated env seeded from a disjoint per-worker slice.
+
+```yaml
+safety:
+  n_episodes: 40
+  n_workers: 4
+  limits: { cotter/joint_velocities: 5.0 }
+```
+
+### Managing the adversary zoo
+
+```sh
+poetry run cotter zoo list                 # cached adversaries (+ --env filter)
+poetry run cotter zoo prune                # drop entries with missing artifacts
+```
+
+### Backends
+
+`RunConfig.backend` (default `gymnasium`) selects the simulator backend
+through `BackendFactory.from_name`. `GymnasiumBackend` is the CPU MuJoCo
+default; `IsaacSimBackend` targets NVIDIA Isaac Sim and raises
+`BackendNotAvailableError` at construction when `omni.isaac.gym` is not
+installed (it is not exercised on the CPU dev machine).
+
+## Quickstart (Python API)
+
+```python
+import gymnasium as gym
+from stable_baselines3 import PPO
+from cotter import (
+    CotterWrapper, SafetyLimit, TestReport, load_policy, run_rollouts,
+    run_sprt, evaluate_safety, mcnemar_exact, run_adversarial_test,
+    rollout_one, make_seed_sequence, JOINT_VELOCITIES, ACTUATOR_FORCES,
+)
+
+# 1. Wrap any Gymnasium MuJoCo env; the wrapper exposes qvel,
+#    actuator_force, contact count, and per-body contact-force
+#    magnitudes in every step's info dict.
+env = CotterWrapper(gym.make("InvertedPendulum-v5"))
+
+# 2. Load the policy under test (SB3 .zip or a raw torch .pt module).
+#    Observation/action spaces are validated and mismatches fail loudly.
+policy = load_policy("artifacts/victim_ppo_inverted_pendulum.zip", env, algo=PPO)
+
+# 3. Define task success and run the categories you need.
+def success(total_reward, length, terminated, truncated, final_info):
+    return length >= 1000  # survived the full horizon
+
+seeds = make_seed_sequence(50, base_seed=0)
+perf = run_sprt(
+    lambda i: rollout_one(policy, env, seeds[i], success).success,
+    p0=0.80, p1=0.95, alpha=0.05, beta=0.05, n_max=50,
+)
+
+rollouts = run_rollouts(policy, env, 20, success, base_seed=1)
+safe = evaluate_safety(rollouts.episode_infos, [
+    SafetyLimit(JOINT_VELOCITIES, 5.0),
+    SafetyLimit(ACTUATOR_FORCES, 2.5),
+])
+
+adv = run_adversarial_test(policy, env, success, epsilon=0.07, n_episodes=20)
+
+# 4. Aggregate into a report (console summary + JSON artifact).
+report = TestReport(policy_name="my_policy", env_id="InvertedPendulum-v5")
+report.add_sprt(perf)
+report.add_safety(safe)
+report.add_adversarial(adv)
+print(report.summary())
+report.to_json("report.json")
+```
+
+## Demo
+
+```sh
+poetry run python examples/demo.py
+```
+
+Runs all four categories against a checked-in PPO policy and writes
+`artifacts/demo_report.json`. Takes ~40 s on Apple Silicon CPU (dominated
+by training the adversary; pass `--skip-adversary-training` to use only
+the random baseline). The victim can be retrained from scratch with
+`poetry run python scripts/train_victim.py` (~17 s, 100k timesteps,
+eval reward 1000.0 ± 0.0).
+
+### Demo environment choice
+
+The demo uses **InvertedPendulum-v5**, chosen deliberately for
+reliability over spectacle: it steps fast on CPU, PPO solves it in
+seconds (so the whole pipeline is verifiable end-to-end in one sitting),
+and its MuJoCo `data` exposes physically meaningful joint velocities,
+actuator forces, and contact counts for the safety checks. Nothing in
+Cotter is specific to this env — `CotterWrapper` works with any
+Gymnasium MuJoCo environment with Box spaces.
+
+### Real captured results
+
+Verbatim summary from an executed run (2026-07-07, seed 0, M5 CPU —
+full output in [RESULTS.md](RESULTS.md)):
+
+```
+==============================================================================
+COTTER TEST REPORT — policy 'victim_ppo' on InvertedPendulum-v5
+generated 2026-07-07T12:54:58+00:00
+==============================================================================
+[PASS] performance/sprt_success_rate
+       PASS: 18/18 successes (100.0%) after 18 sequential trials (H0 p<=0.8, H1 p>=0.95)
+[PASS] safety/hard_limits
+       PASS: no violations in 20020 timesteps across 20 trials
+[FAIL] regression/success_mcnemar
+       REGRESSION: baseline 1.000 vs candidate 0.000 over 30 paired trials (p=9.31e-10, mcnemar_exact_one_sided)
+[FAIL] regression/return_wilcoxon
+       REGRESSION: baseline 1000.000 vs candidate 94.900 over 30 paired trials (p=8.63e-07, wilcoxon_signed_rank_one_sided)
+[PASS] adversarial/random_baseline
+       PASS: success rate 100.0% clean -> 100.0% under random linf perturbation (eps=0.07, n=20, required >= 50%) [uniform random baseline]
+[FAIL] adversarial/learned_ppo
+       FAIL: success rate 100.0% clean -> 0.0% under ppo linf perturbation (eps=0.07, n=20, required >= 50%) [trained PPO adversary]
+------------------------------------------------------------------------------
+OVERALL: FAIL (3 failing, 3 passing, 0 informational)
+==============================================================================
+```
+
+The FAILs are the point of the demo: the regression category is fed a
+deliberately undertrained candidate and catches it (p ≈ 10⁻⁹ from just
+30 paired episodes), and the adversarial category shows the headline
+result — **at a perturbation budget where random sensor noise is
+completely harmless (100% success), a learned adversary drives the same
+policy to 0%.** Random-noise robustness testing alone would have
+certified this policy.
+
+## Locomotion demo (Ant-v5)
+
+A PPO victim trained on the MuJoCo quadruped (500k steps,
+`scripts/train_ant_victim.py`), tested with all four categories via
+`cotter run --policy artifacts/victim_ant.zip --config examples/ant.yaml`.
+Real captured report (2026-07-08, seed 0, M5 CPU):
+
+```
+COTTER TEST REPORT — policy 'victim_ant' on Ant-v5
+[PASS] performance/sprt_success_rate
+       PASS: 7/7 successes (100.0%, 95% CI [59.0%, 100.0%]) after 7 sequential trials (H0 p<=0.5, H1 p>=0.8)
+[PASS] safety/hard_limits
+       PASS: no violations in 12813 timesteps across 20 trials
+[PASS] regression/success_mcnemar
+       NO_REGRESSION: baseline 0.000 vs candidate 0.950 over 20 paired trials (p=1, discordant_odds_ratio=0, ...)
+[PASS] regression/x_position_wilcoxon
+       NO_REGRESSION: baseline -0.066 vs candidate 9.446 over 20 paired trials (p=1, rank_biserial_correlation=0.971, ...)
+[PASS] adversarial/random_baseline
+       PASS: success rate 100.0% clean -> 75.0% [50.9%, 91.3%] under random linf perturbation (eps=1.0, n=20, required >= 50%)
+[PASS] adversarial/learned_ppo
+       PASS: success rate 100.0% clean -> 95.0% [75.1%, 99.9%] under ppo linf perturbation (eps=1.0, n=20, required >= 50%)
+OVERALL: PASS (0 failing, 6 passing, 0 informational)
+```
+
+Locomotion needs the right success signal: Ant's per-step *survival
+bonus* means a policy that stands still scores high total reward, so raw
+return would rank a do-nothing policy above a walking one. The demo
+therefore defines success as **forward displacement** (`min_info` on
+`x_position`) and runs the regression on the same metric. That cleanly
+separates the trained victim (95% success, ~9.4 m forward) from the
+random-init baseline (0%, ~0 m). Here the learned adversary happened to
+underperform the random baseline at this budget (the PPO attacker is
+time-boxed); Cotter reports both honestly.
+
+## Real manipulation demo (HER + SAC)
+
+Cotter handles the `Dict` observation spaces used by
+[gymnasium-robotics](https://robotics.farama.org/) Fetch/manipulation
+envs, where success is a goal flag (`{type: info_flag, key: is_success}`)
+rather than a reward threshold.
+
+The intended target was **FetchPickAndPlace-v4** trained with HER+SAC
+(`scripts/train_fetch_pickplace.py`), but it did not converge on the M5
+CPU within the time budget (0% success through 100k steps). Per the
+documented fallback the demo uses the much easier **FetchReachDense-v4**,
+which the same HER+SAC recipe solves to 100% in about a minute. Real
+captured report from `cotter run --policy
+artifacts/victim_fetch_reach_hersac.zip --config
+examples/fetch_pickplace.yaml`:
+
+```
+COTTER TEST REPORT — policy 'victim_fetch_reach_hersac' on FetchReachDense-v4
+[PASS] performance/sprt_success_rate
+       PASS: 8/8 successes (100.0%, 95% CI [63.1%, 100.0%]) after 8 sequential trials (H0 p<=0.4, H1 p>=0.6)
+[PASS] safety/hard_limits
+       PASS: no violations in 1020 timesteps across 20 trials
+[PASS] regression/success_mcnemar
+       NO_REGRESSION: baseline 0.033 vs candidate 1.000 over 30 paired trials (p=1, discordant_odds_ratio=0, ...)
+[PASS] regression/return_wilcoxon
+       NO_REGRESSION: baseline -8.198 vs candidate -1.013 over 30 paired trials (p=1, rank_biserial_correlation=1, ...)
+[PASS] adversarial/random_baseline
+       PASS: success rate 100.0% clean -> 55.0% [31.5%, 76.9%] under random linf perturbation (eps=0.1, n=20)
+[FAIL] adversarial/learned_ppo
+       FAIL: success rate 100.0% clean -> 0.0% [0.0%, 16.8%] under ppo linf perturbation (eps=0.1, n=20, required >= 50%)
+OVERALL: FAIL (1 failing, 5 passing, 0 informational)
+```
+
+The safety limits target Fetch's joint velocities and contacts (its
+end-effector is mocap-controlled, so `actuator_forces` is empty and a
+limit on it passes trivially). **Adversarial perturbation works on Dict
+observations**: it perturbs only the sensed-state `observation` sub-key
+(leaving the goal keys untouched), and the headline result reappears on
+a manipulation env — the learned PPO adversary drives the victim to 0%
+at ε = 0.1 where random noise of the same budget only reaches 55%.
+
+Regression *detection* is a one-liner. Comparing the trained victim
+against its random-init counterpart flags the difference immediately:
+
+```
+$ cotter compare --baseline victim_fetch_reach_hersac.zip \
+      --candidate victim_fetch_reach_randominit.zip --config examples/fetch_pickplace.yaml
+[FAIL] regression/success_mcnemar
+       REGRESSION: baseline 1.000 vs candidate 0.033 over 30 paired trials (p=1.86e-09, discordant_odds_ratio=inf, ...)
+[FAIL] regression/return_wilcoxon
+       REGRESSION: baseline -1.013 vs candidate -8.198 over 30 paired trials (p=9.31e-10, rank_biserial_correlation=-1, ...)
+OVERALL: FAIL (2 failing, 0 passing, 0 informational)   # exit code 1
+```
+
+## Adversary zoo
+
+Trained adversaries are expensive and specific to what they attack, so
+`cotter.zoo.AdversaryZoo` caches them keyed by
+`(env_id, victim_hash, epsilon)`. Set `use_zoo: true` in a config's
+adversarial section and the first run trains and stores the attacker;
+later runs on the same victim reuse it instead of retraining. The victim
+hash is taken over the policy artifact (or its parameter tensors), so a
+retrained victim never silently reuses a stale adversary. A hosted zoo
+of pretrained expert adversaries is the planned paid tier.
+
+## Compliance layer (paid tier)
+
+The open-source engine produces the *evidence* — structured pass/fail
+results with statistical guarantees. Rendering that into regulator-ready
+documentation (EU Machinery Regulation 2027 technical files, ISO 10218
+conformity records) is the licensed commercial layer, stubbed here with
+a stable import path:
+
+```python
+from cotter.compliance import EUMachineryReg2027
+EUMachineryReg2027(report)   # raises LicenseRequiredError in the OSS build
+```
+
+## Architecture
+
+```
+cotter/
+├── cli.py               # `cotter run` / `compare` / `zoo` entrypoints
+├── config.py            # YAML config schema + loader
+├── pipeline.py          # executes the declared categories, aggregates a report
+├── backends.py          # simulator backend abstraction (gymnasium / isaac-sim)
+├── success.py           # declarative success criteria (min_length/return/info_flag)
+├── stats.py             # Clopper-Pearson exact binomial confidence intervals
+├── policy.py            # black-box loading (SB3 .zip / torch .pt) + space validation
+├── runner.py            # seeded rollouts (serial + parallel) -> EpisodeRecords
+├── envs/
+│   ├── wrapper.py       # instruments info dict with qvel / actuator_force / contacts
+│   ├── registry.py      # resolves gymnasium-robotics (and other) extension envs
+│   └── factory.py       # picklable env factory for parallel rollouts
+├── report.py            # TestReport: console summary + JSON (results container only)
+├── zoo/                 # adversary registry keyed by (env, victim, epsilon)
+├── compliance/          # paid-tier regulatory document stub (license required)
+└── tests/
+    ├── sprt.py          # Wald's sequential probability ratio test (+ CI)
+    ├── safety.py        # per-timestep hard limits, zero tolerance
+    ├── regression.py    # exact McNemar + Wilcoxon on matched pairs (+ effect sizes)
+    └── adversarial.py   # observation-perturbation attack (PPO or random, + CI)
+```
+
+Design notes:
+
+- **Space validation fails loudly.** Loading a policy checks its declared
+  and functional observation/action shapes against the env — the #1
+  real-world integration bug surfaces at load time, not mid-rollout.
+- **Shared seeds everywhere it matters.** Regression pairs and
+  clean-vs-attacked comparisons run on identical seed sequences, so
+  differences are attributable to the policy, not the physics draw.
+- **The adversarial floor never fails.** If PPO adversary training
+  errors, `get_adversary` falls back to the random baseline and says so
+  in the result — the category always produces a number.
+- **`report.py` is a results container**, not a compliance-document
+  generator; that is the paid `cotter.compliance` layer.
+- **Config-driven, seed-reproducible runs.** `cotter run --config
+  run.yaml` executes whichever categories the YAML declares; every
+  category derives its seeds from one `base_seed`, so enabling or
+  disabling a category never perturbs another's rollouts.
+