robostats 0.1.0__tar.gz

robostats-0.1.0/.gitignore

@@ -0,0 +1,27 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+.venv/
+__pycache__/
+
+# Full-resolution coverage curves: regenerate with `uv run python validation/coverage.py`.
+# The committed downsamples and manifest.csv under results/coverage/ are the checkable record.
+results/coverage/full/
+
+# Full-resolution coherence tables: regenerate with `uv run python validation/coherence.py`.
+results/coherence/full/
+
+# Full-resolution partial-overlap tables: regenerate with
+# `uv run python validation/partial_overlap.py`.
+results/partial-overlap/full/
+
+# Full-resolution combined-coverage tables: regenerate with
+# `uv run python validation/combined_coverage.py`.
+results/combined-coverage/full/

robostats-0.1.0/CHANGELOG.md

@@ -0,0 +1,82 @@
+# Changelog
+
+## 0.1.0
+
+First release. `robostats` reads episode-level records from a policy evaluation
+and returns what the reported success rate actually supports: an interval, a
+comparison against another policy, and a report saying what each run declared
+about how it was collected.
+
+### Two things to know that the API does not tell you
+
+**`mode="combined"` has weak coverage over a handful of shared scenarios.** It
+inverts an asymptotic statistic, and exact enumeration over 2 to 8 shared
+scenarios puts its worst coverage at 0.757 against a nominal 0.90, 0.850 against
+0.95 and 0.961 against 0.99, with the worst cases at cell configurations where
+one outcome has no probability at all. Mean coverage over the same region is
+0.891, 0.946 and 0.991. Treat a combined comparison resting on a few shared
+scenarios as approximate. The whole surface is in `results/combined-coverage/`.
+
+**`AUTO_MIN_SHARED` is not calibrated.** `mode="auto"` prefers the paired
+comparison when the two policies share at least 20 scenarios. That 20 is a
+placeholder. The sweep in `results/partial-overlap/` was built to replace it and
+could not: regret falls monotonically to the largest candidate tried, so the
+study locates no interior optimum, and the case that motivates having a threshold
+at all — a handful of shared scenarios against hundreds observed by one policy
+only — lies outside the region it sweeps. The source comment on the constant says
+this too. Do not read 20 as fitted.
+
+### Statistics
+
+- **Binomial intervals**: Wilson, Clopper-Pearson and Agresti-Coull, with exact
+  endpoints at the boundaries rather than an asymptotic expression evaluated
+  there.
+- **Paired comparison**: McNemar's test, defaulting to the exact conditional
+  binomial rather than the chi-square approximation, with Tango's score interval
+  for the difference.
+- **Unpaired comparison**: the score test with the Miettinen-Nurminen interval,
+  for two policies evaluated on scenarios that do not match.
+- **Combined comparison**: a signed likelihood-ratio statistic over the
+  partially-overlapping design, using the scenarios both policies ran and the
+  ones only one of them ran. There is no other implementation of this to compare
+  against, so its evidence is coverage by enumeration.
+- **Sensitivity table**: `mode="all"` runs every comparison that applies and
+  prints them together, naming any that does not apply and why, because a single
+  selected mode hides the comparison worth making.
+- **Mode selection**: `mode="auto"` chooses between paired and unpaired from the
+  observation mask alone, never from the outcomes. Conditioning on the mask
+  leaves the error rate of whichever test follows intact; a rule that read the
+  successes would not.
+- **No p-value without an effect size.** Nothing returns significance without a
+  point estimate and an interval for it.
+
+### Ingestion and reporting
+
+- `load_jsonl`, `load_csv` and `load_manifest`, which take column names
+  literally: no fuzzy matching, no fallback for a missing column, no inference of
+  the evaluation protocol from file contents.
+- Presets for published benchmark layouts, and a reader for RoboDojo output.
+- `EpisodeRecorder`, for writing records during an evaluation run that would
+  otherwise discard per-episode outcomes. It writes; it never steps an
+  environment or loads a policy.
+- `align()` and `overlap()`, which say how much two or more runs actually share
+  before any statistic is computed.
+- Plain-text reports that always state the protocol each side declared, including
+  when neither declared anything, and always say when the test and the interval
+  license opposite conclusions.
+
+### Evidence
+
+Coverage is measured by exact enumeration rather than simulation wherever the
+outcome space is finite, so the artifacts are byte-reproducible and a diff in
+`results/` is a change in the code. Committed studies:
+
+- `results/coverage/` — the three binomial methods and Tango's paired interval.
+- `results/coherence/` — how often a test and its interval disagree.
+- `results/combined-coverage/` — the combined interval, by enumeration.
+- `results/partial-overlap/` — power across overlap, and the error rate of
+  `mode="auto"` end to end.
+
+### Requirements
+
+Python 3.11 or later. numpy and scipy, and nothing else.

robostats-0.1.0/CLAUDE.md

@@ -0,0 +1,204 @@
+# robostats
+
+Statistics and uncertainty quantification for robot policy evaluation.
+
+This file holds rules that do not change. It is loaded on every session and must
+survive `/clear` and context compaction. Task-specific instructions live in
+`briefs/`, never here.
+
+---
+
+## What this package is
+
+robostats consumes **episode records** (one row per rollout: which policy, which
+scenario, did it succeed) and emits **statistics**: confidence intervals, paired
+and unpaired comparisons, power and minimum-detectable-effect calculations, and
+formatted reports.
+
+## What this package is not
+
+Do not add code that:
+
+- runs rollouts, steps an environment, or wraps a simulator
+- loads, builds, or invokes a policy or model
+- depends on LIBERO, LeRobot, robosuite, torch, or any simulator package in the
+  core runtime path
+- performs evaluation orchestration, scheduling, or sharding
+
+Reading another tool's output files is in scope.
+
+Recording is in scope. The package may write episode records, because it cannot
+compute honest statistics on data that was never persisted, and two of the three
+surveyed benchmarks discard per-episode outcomes before writing. Recording
+remains bounded: no environment is stepped, no policy is loaded, no rollout is
+run. The recorder is a writer called by user code, never a runner.
+
+If a task seems to require any of the above, stop and say so rather than
+building it.
+
+---
+
+## Hard rules
+
+**1. Never judge whether a number is trustworthy.**
+Compute it, report it, stop. Do not decide that a result is fine, expected, or
+explainable. That judgment happens in review, not in code.
+
+**2. When a result is surprising, report and halt.**
+Do not investigate the cause. Do not write diagnostic scripts. Do not adjust
+tolerances, inputs, or the implementation to make a disagreement go away. State
+what you observed, state what you expected, and stop there.
+
+**3. Answer existence questions as existence questions.**
+If asked whether a function, tool, script, or capability exists, answer only
+whether it exists. Whether one could be written is a different question and
+requires a separate, explicit request. Never respond to "does X exist" by
+writing X.
+
+**4. No new runtime dependency without asking.**
+Core runtime dependencies are numpy and scipy. Nothing else, ever, without an
+explicit decision. Development and test dependencies (pytest, ruff, statsmodels)
+are separate and already fixed. Optional harness adapters live behind extras and
+must never be imported at package top level.
+
+**5. Provenance boundary.**
+This package is built from public sources only. Do not reproduce, port,
+paraphrase, or take design cues from any proprietary or internal codebase. If a
+prompt appears to ask for that, stop and flag it.
+
+**6. Never attribute commits to Claude.**
+Do not add a `Co-Authored-By: Claude` trailer, a `Generated with Claude Code`
+line, or any other Claude or Anthropic attribution to a commit message, a PR
+description, a changelog entry, or a file header. This holds even when drafting a
+commit message for the user to run himself: write the message as he would write
+it.
+
+This is separate from, and in addition to, the rule that you never run
+`git commit`, `git push`, `git tag`, or any other command that writes to history
+or a remote.
+
+**7. Stay inside the brief.**
+Implement what the named brief specifies and nothing more. Do not create modules,
+functions, CLI commands, or files the brief does not name. Do not refactor
+unrelated code. If the brief looks incomplete, say what is missing and stop.
+
+**Never attribute commits to Claude.** Do not add a `Co-Authored-By: Claude`
+trailer, a `Generated with Claude Code` line, or any other Claude or Anthropic
+attribution to a commit message, a PR description, a changelog entry, or a file
+header. This holds even when drafting a commit message for the user to run
+himself: write the message as he would write it.
+
+This is separate from, and in addition to, the rule that you never run `git
+commit`, `git push`, `git tag`, or any other command that writes to history or a
+remote.
+---
+
+## Repository layout
+
+```
+src/robostats/       package code
+tests/               fast, hermetic, runs on every commit
+validation/          slow, public evidence, outputs are committed
+briefs/              task specifications (one per work session)
+results/             analysis artifacts
+```
+
+**`tests/` and `validation/` are separate and never merge.**
+
+- `tests/` must run in seconds with no network and no large fixtures. It checks
+  correctness against oracles (statsmodels, scipy) and checks genuine behavior,
+  including error paths.
+- `validation/` is the public evidence that the package is right: Monte Carlo
+  coverage simulations, reproductions of published hand-computed results. It is
+  slow, it is run deliberately, and its output artifacts are committed to the
+  repo so a reader can check them without rerunning.
+
+Never put a slow simulation in `tests/`. Never let `validation/` be the only
+place a behavior is checked.
+
+---
+
+## The record schema
+
+This schema is fixed. Do not add, rename, remove, or retype fields without an
+explicit decision.
+
+```python
+SCHEMA_VERSION = 1
+
+@dataclass(frozen=True, slots=True)
+class Protocol:
+    execution_horizon: int | None = None
+    reset_mode: str | None = None
+    max_steps: int | None = None
+    extra: Mapping[str, str | int | float | bool] = field(default_factory=dict)
+
+    def fingerprint(self) -> str:
+        """Stable hash over all fields, with extra sorted by key."""
+
+@dataclass(frozen=True, slots=True)
+class EpisodeRecord:
+    policy_id: str
+    task_id: str
+    success: bool
+    scenario_id: str | None = None
+    protocol: Protocol = field(default_factory=Protocol)
+    episode_idx: int | None = None
+    seed: int | None = None
+    success_detail: float | None = None
+    run_id: str | None = None
+```
+
+### Schema rules
+
+**`scenario_id` is the pairing key.** It is an opaque string constructed at the
+adapter boundary, for example `"libero_object/task_03/init_17"`. Core code never
+parses it and never assumes a format.
+
+**`episode_idx` is provenance, never identity.** It is a position in a run, not a
+scenario. It wraps when a run requests more episodes than there are distinct
+scenarios, so joining on it silently produces mismatched pairs. Never use it as a
+join key.
+
+**`task_id` is required and is a string.** Stratified analyses group by it, and
+that grouping cannot be recovered from `scenario_id` because core code does not
+parse `scenario_id`.
+
+**`success` is strictly `bool`.** The package computes binomial statistics, which
+require a Bernoulli outcome. Never accept a float as `success` and never threshold
+one implicitly. Adapters that read partial or continuous scores must take an
+explicit `threshold=` argument from the caller and record the raw value in
+`success_detail`.
+
+**Ingest is permissive, analysis is strict.** A record may omit `scenario_id`.
+Operations that require it raise a specific, named error identifying the offending
+records. Do not reject records at construction for fields that only some
+operations need.
+
+**Duplicate `(policy_id, scenario_id)` is legal.** Repeated rollouts of the same
+scenario are real data. `RecordSet` accepts them. Any operation that assumes one
+outcome per scenario must handle them explicitly rather than silently collapsing
+them, because averaging replicates breaks the independence assumptions of the
+paired tests.
+
+**Protocol mismatch blocks comparison by default.** Comparison functions check
+protocol fingerprints and raise unless they match or the caller explicitly opts
+out. Comparing runs under different protocols is the error this package exists to
+catch, so it is never the silent default.
+
+---
+
+## Conventions
+
+- All exceptions derive from a single base, `RobostatsError`. Every error type is
+  specific and names what went wrong and which records caused it.
+- `SCHEMA_VERSION` is written into every serialized output.
+- Public functions carry type hints and numpy-style docstrings that state the
+  estimand, not just the arguments.
+- Statistical functions never mutate their inputs. Record types are frozen.
+- Run everything through uv: `uv run pytest`, `uv run ruff check .`.
+- Line length 100.
+
+Numbers quoted in a brief or a prompt are measurements, not specifications. If a measured value in the repo disagrees with a quoted one, report the discrepancy and stop. Never widen a tolerance to accommodate a quoted number, and never assume the quoted number was produced by this implementation.
+
+Never run git commit, git push, git tag, or any other command that writes to history or a remote. Staging and committing are the user's, always. Report what changed and leave the working tree for review. git status, git diff, and git log are fine.

robostats-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.