claimeval 1.0.0__tar.gz

claimeval-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Harish Mashetty
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

claimeval-1.0.0/PKG-INFO

@@ -0,0 +1,198 @@
+Metadata-Version: 2.4
+Name: claimeval
+Version: 1.0.0
+Summary: Uncertainty for cost-accuracy claims about AI agents: test whether a leaderboard or paper claim is supported by its own data.
+Author-email: Harish Mashetty <harish.mashetty@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/claimeval/claimeval
+Project-URL: Documentation, https://github.com/claimeval/claimeval#readme
+Project-URL: Repository, https://github.com/claimeval/claimeval
+Project-URL: Issue Tracker, https://github.com/claimeval/claimeval/issues
+Keywords: evaluation,llm,agents,leaderboard,bootstrap,pareto,statistics
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy>=1.22
+Dynamic: license-file
+
+# claimeval
+
+[![PyPI version](https://img.shields.io/pypi/v/claimeval.svg)](https://pypi.org/project/claimeval/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+
+**Uncertainty quantification for cost–accuracy claims about AI agents.**
+
+Leaderboards, press releases, and papers declare:
+* *"Agent A is #1 on SWE-bench Verified"*
+* *"Model X outperforms Model Y"*
+* *"System A matches System B at a fifth of the cost"*
+* *"Agent A is Pareto-optimal on the cost–accuracy frontier"*
+
+Each of these is a statement about noisy sample estimates from a finite benchmark of tasks and runs. `claimeval` attaches a probability to each such claim, using a bootstrap over the correct unit of generalization, and reports whether it is statistically supported.
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+`claimeval` requires only **NumPy**:
+
+```bash
+pip install claimeval
+```
+
+Or install from source in editable mode:
+
+```bash
+git clone https://github.com/claimeval/claimeval.git
+cd claimeval
+pip install -e .
+```
+
+---
+
+## 🔍 Test a Claim in One Command
+
+Evaluate claims against a raw rollout log:
+
+```bash
+claimeval claims examples/runs.jsonl examples/claims.json --map examples/map.json
+```
+
+**Output:**
+```
+id        kind            subject                      P(claim)  verdict
+c1        top1            agent-E                         0.960  supported
+c2        pairwise        agent-D                         0.996  supported
+c3        cost_dominance  agent-D                         0.717  NOT supported
+c4        frontier        agent-A                         0.993  supported
+supported = P >= 0.9; resampling: task
+```
+
+### Claim Definitions (`claims.json`)
+
+Define claims in testable JSON schemas:
+
+```json
+[
+  {
+    "id": "c1",
+    "kind": "top1",
+    "subject": "agent-A",
+    "comparators": ["agent-B", "agent-C"]
+  },
+  {
+    "id": "c2",
+    "kind": "pairwise",
+    "subject": "agent-A",
+    "comparators": ["agent-B"]
+  },
+  {
+    "id": "c3",
+    "kind": "cost_dominance",
+    "subject": "agent-B",
+    "comparators": ["agent-D"]
+  },
+  {
+    "id": "c4",
+    "kind": "frontier",
+    "subject": "agent-A",
+    "comparators": ["agent-B", "agent-C", "agent-D"]
+  }
+]
+```
+
+Supported claim types:
+* `top1`: The subject strictly leads all comparators.
+* `topk`: The subject ranks within the top $k$ systems.
+* `pairwise`: The subject strictly outperforms named rivals.
+* `parity`: Performance difference is within a threshold ($|a_c - a_f| \le \delta$).
+* `cheaper_noninferior`: Subject is cheaper than rival and accuracy is within parity margin.
+* `frontier`: Subject belongs to the non-dominated Pareto frontier.
+* `cost_dominance`: Subject is both strictly cheaper and more accurate than rival.
+
+---
+
+## 🐍 Python API
+
+```python
+from claimeval import Rollout, RolloutSet, Claim, evaluate_claims
+
+# Ingest rollout records: (config, task_id, seed, success_bool, cost_dollars, group)
+rollouts = [
+    Rollout("agent-A", "task-1", seed=0, success=1.0, cost=0.15, group="repo-1"),
+    Rollout("agent-B", "task-1", seed=0, success=0.0, cost=0.08, group="repo-1"),
+    # ...
+]
+dataset = RolloutSet(rollouts)
+
+# Test a pairwise claim
+claims = [Claim("pairwise", subject="agent-A", comparators=["agent-B"])]
+results = evaluate_claims(dataset, claims, threshold=0.9, n_boot=2000)
+
+for r in results:
+    print(f"Claim {r['id']}: P = {r['probability']:.3f} -> {r['verdict']}")
+```
+
+### Modular Architecture & Subpackages
+
+`claimeval` is organized into clean, focused subpackages:
+
+* **`claimeval.core`**: Data models (`Rollout`, `RolloutSet`), stream/file ingestion (`load`, `inspect_path`), and benchmark loaders (`swebench`, `terminal_bench`, `tau2`).
+* **`claimeval.stats`**: Resampling estimators (`hierarchical_bootstrap`, `variance_decomposition`), Pareto geometry (`dominance`, `pareto_mask`, `frontier_band`, `hausdorff`), cluster methods (`wild_intervals`), BCa intervals, bias corrections (`split_sample_debias`), and cost diagnostics (`diagnose_cost_axis`).
+* **`claimeval.analysis`**: Claim evaluation engine (`Claim`, `evaluate_claims`), benchmark baselines, pairwise comparison suites, and formatted tabular reporting (`analyze`, `headline`).
+* **`claimeval.sim`**: Generative ground-truth simulations (`make_population`, `simulate`) and Monte Carlo estimator coverage studies (`coverage_study`).
+
+All common functions remain accessible directly from the top-level `import claimeval`.
+
+---
+
+## 📊 Auditing Public Benchmarks Directly
+
+`claimeval` provides built-in loaders for public agent benchmarks:
+
+```bash
+# SWE-bench Verified (from github.com/SWE-bench/experiments)
+claimeval claims evaluation/verified claims.json --from swebench
+
+# SWE-bench Verified cost leaderboard (mini-SWE-agent runs)
+claimeval claims evaluation/verified claims.json --from swebench-cost
+
+# Terminal-Bench 2.0 (Harbor evaluation trees)
+claimeval claims submissions/terminal-bench/2.0 claims.json --from terminal-bench
+
+# tau2-bench leaderboard trajectories
+claimeval claims tau2_results/ claims.json --from tau2
+```
+
+---
+
+## 🛠️ CLI Toolkit
+
+| Command | Purpose |
+|---|---|
+| `claimeval claims` | Evaluate a set of typed claims against a rollout dataset |
+| `claimeval analyze` | Compute empirical Pareto frontier with membership & dominance probabilities |
+| `claimeval inspect` | Automatically inspect schema and statistics of an unknown evaluation dump |
+| `claimeval seed-curve` | Simulate error of a $k$-run study vs. full data |
+| `claimeval coverage` | Validate estimator calibration against known ground truth |
+| `claimeval demo` | Run an end-to-end demonstration on synthetic benchmark data |
+
+---
+
+## 👤 Author & Contact
+
+**Harish Mashetty**  
+Independent Researcher  
+E-mail: [harish.mashetty@gmail.com](mailto:harish.mashetty@gmail.com)  
+Project Repository: [https://github.com/claimeval/claimeval](https://github.com/claimeval/claimeval)
+
+---
+
+## 📜 License
+
+MIT License. See [LICENSE](LICENSE) for details.

claimeval-1.0.0/README.md

@@ -0,0 +1,179 @@
+# claimeval
+
+[![PyPI version](https://img.shields.io/pypi/v/claimeval.svg)](https://pypi.org/project/claimeval/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+
+**Uncertainty quantification for cost–accuracy claims about AI agents.**
+
+Leaderboards, press releases, and papers declare:
+* *"Agent A is #1 on SWE-bench Verified"*
+* *"Model X outperforms Model Y"*
+* *"System A matches System B at a fifth of the cost"*
+* *"Agent A is Pareto-optimal on the cost–accuracy frontier"*
+
+Each of these is a statement about noisy sample estimates from a finite benchmark of tasks and runs. `claimeval` attaches a probability to each such claim, using a bootstrap over the correct unit of generalization, and reports whether it is statistically supported.
+
+---
+
+## 🚀 Quick Start
+
+### Installation
+
+`claimeval` requires only **NumPy**:
+
+```bash
+pip install claimeval
+```
+
+Or install from source in editable mode:
+
+```bash
+git clone https://github.com/claimeval/claimeval.git
+cd claimeval
+pip install -e .
+```
+
+---
+
+## 🔍 Test a Claim in One Command
+
+Evaluate claims against a raw rollout log:
+
+```bash
+claimeval claims examples/runs.jsonl examples/claims.json --map examples/map.json
+```
+
+**Output:**
+```
+id        kind            subject                      P(claim)  verdict
+c1        top1            agent-E                         0.960  supported
+c2        pairwise        agent-D                         0.996  supported
+c3        cost_dominance  agent-D                         0.717  NOT supported
+c4        frontier        agent-A                         0.993  supported
+supported = P >= 0.9; resampling: task
+```
+
+### Claim Definitions (`claims.json`)
+
+Define claims in testable JSON schemas:
+
+```json
+[
+  {
+    "id": "c1",
+    "kind": "top1",
+    "subject": "agent-A",
+    "comparators": ["agent-B", "agent-C"]
+  },
+  {
+    "id": "c2",
+    "kind": "pairwise",
+    "subject": "agent-A",
+    "comparators": ["agent-B"]
+  },
+  {
+    "id": "c3",
+    "kind": "cost_dominance",
+    "subject": "agent-B",
+    "comparators": ["agent-D"]
+  },
+  {
+    "id": "c4",
+    "kind": "frontier",
+    "subject": "agent-A",
+    "comparators": ["agent-B", "agent-C", "agent-D"]
+  }
+]
+```
+
+Supported claim types:
+* `top1`: The subject strictly leads all comparators.
+* `topk`: The subject ranks within the top $k$ systems.
+* `pairwise`: The subject strictly outperforms named rivals.
+* `parity`: Performance difference is within a threshold ($|a_c - a_f| \le \delta$).
+* `cheaper_noninferior`: Subject is cheaper than rival and accuracy is within parity margin.
+* `frontier`: Subject belongs to the non-dominated Pareto frontier.
+* `cost_dominance`: Subject is both strictly cheaper and more accurate than rival.
+
+---
+
+## 🐍 Python API
+
+```python
+from claimeval import Rollout, RolloutSet, Claim, evaluate_claims
+
+# Ingest rollout records: (config, task_id, seed, success_bool, cost_dollars, group)
+rollouts = [
+    Rollout("agent-A", "task-1", seed=0, success=1.0, cost=0.15, group="repo-1"),
+    Rollout("agent-B", "task-1", seed=0, success=0.0, cost=0.08, group="repo-1"),
+    # ...
+]
+dataset = RolloutSet(rollouts)
+
+# Test a pairwise claim
+claims = [Claim("pairwise", subject="agent-A", comparators=["agent-B"])]
+results = evaluate_claims(dataset, claims, threshold=0.9, n_boot=2000)
+
+for r in results:
+    print(f"Claim {r['id']}: P = {r['probability']:.3f} -> {r['verdict']}")
+```
+
+### Modular Architecture & Subpackages
+
+`claimeval` is organized into clean, focused subpackages:
+
+* **`claimeval.core`**: Data models (`Rollout`, `RolloutSet`), stream/file ingestion (`load`, `inspect_path`), and benchmark loaders (`swebench`, `terminal_bench`, `tau2`).
+* **`claimeval.stats`**: Resampling estimators (`hierarchical_bootstrap`, `variance_decomposition`), Pareto geometry (`dominance`, `pareto_mask`, `frontier_band`, `hausdorff`), cluster methods (`wild_intervals`), BCa intervals, bias corrections (`split_sample_debias`), and cost diagnostics (`diagnose_cost_axis`).
+* **`claimeval.analysis`**: Claim evaluation engine (`Claim`, `evaluate_claims`), benchmark baselines, pairwise comparison suites, and formatted tabular reporting (`analyze`, `headline`).
+* **`claimeval.sim`**: Generative ground-truth simulations (`make_population`, `simulate`) and Monte Carlo estimator coverage studies (`coverage_study`).
+
+All common functions remain accessible directly from the top-level `import claimeval`.
+
+---
+
+## 📊 Auditing Public Benchmarks Directly
+
+`claimeval` provides built-in loaders for public agent benchmarks:
+
+```bash
+# SWE-bench Verified (from github.com/SWE-bench/experiments)
+claimeval claims evaluation/verified claims.json --from swebench
+
+# SWE-bench Verified cost leaderboard (mini-SWE-agent runs)
+claimeval claims evaluation/verified claims.json --from swebench-cost
+
+# Terminal-Bench 2.0 (Harbor evaluation trees)
+claimeval claims submissions/terminal-bench/2.0 claims.json --from terminal-bench
+
+# tau2-bench leaderboard trajectories
+claimeval claims tau2_results/ claims.json --from tau2
+```
+
+---
+
+## 🛠️ CLI Toolkit
+
+| Command | Purpose |
+|---|---|
+| `claimeval claims` | Evaluate a set of typed claims against a rollout dataset |
+| `claimeval analyze` | Compute empirical Pareto frontier with membership & dominance probabilities |
+| `claimeval inspect` | Automatically inspect schema and statistics of an unknown evaluation dump |
+| `claimeval seed-curve` | Simulate error of a $k$-run study vs. full data |
+| `claimeval coverage` | Validate estimator calibration against known ground truth |
+| `claimeval demo` | Run an end-to-end demonstration on synthetic benchmark data |
+
+---
+
+## 👤 Author & Contact
+
+**Harish Mashetty**  
+Independent Researcher  
+E-mail: [harish.mashetty@gmail.com](mailto:harish.mashetty@gmail.com)  
+Project Repository: [https://github.com/claimeval/claimeval](https://github.com/claimeval/claimeval)
+
+---
+
+## 📜 License
+
+MIT License. See [LICENSE](LICENSE) for details.

claimeval-1.0.0/claimeval/__init__.py

@@ -0,0 +1,95 @@
+"""claimeval -- test the claims made about AI agents against the data behind them.
+
+A claim ("#1", "beats X", "as good at lower cost", "on the cost-accuracy
+frontier") gets the probability that it holds, from a bootstrap over the right
+unit (tasks, or clusters such as repositories), and a verdict. Input is a
+stream of (config, task, seed, success, cost[, group]) runs from any harness,
+or a public leaderboard release read by `claimeval.core.adapters`. Also: frontier
+membership and dominance probabilities, a Hausdorff frontier band, a
+winner's-curse correction, wild cluster bootstrap intervals, and a task-vs-seed
+variance decomposition.
+"""
+
+import sys
+
+# Subpackages
+from . import core
+from . import stats
+from . import analysis
+from . import sim
+
+# Core data structures and loaders
+from .core.schema import Rollout, RolloutSet
+from .core.ingest import FieldMap, guess_map, inspect_path, load
+
+# Statistical methods, frontier geometry & diagnostics
+from .stats.frontier import dominance, frontier_band, frontier_points, hausdorff, log_cost_range, pareto_mask
+from .stats.bootstrap import BootstrapResult, auto_mode, hierarchical_bootstrap, variance_decomposition
+from .stats.wild import wild_intervals
+from .stats.bias import seed_count_curve, split_sample_debias
+from .stats.cost_axis import diagnose_cost_axis, format_diagnosis
+
+# Analysis, claims & reporting
+from .analysis.claims import Claim, evaluate_claims
+from .analysis.report import analyze, format_report, headline
+
+# Simulation & coverage validation
+from .sim.simulate import make_population, population_truth, sample_dataset, simulate
+from .sim.coverage import coverage_study, format_coverage
+
+# Backward compatibility aliases for module paths
+sys.modules["claimeval.schema"] = core.schema
+sys.modules["claimeval.ingest"] = core.ingest
+sys.modules["claimeval.adapters"] = core.adapters
+sys.modules["claimeval.bootstrap"] = stats.bootstrap
+sys.modules["claimeval.wild"] = stats.wild
+sys.modules["claimeval.bca"] = stats.bca
+sys.modules["claimeval.calibrate"] = stats.calibrate
+sys.modules["claimeval.bias"] = stats.bias
+sys.modules["claimeval.cost_axis"] = stats.cost_axis
+sys.modules["claimeval.frontier"] = stats.frontier
+sys.modules["claimeval.claims"] = analysis.claims
+sys.modules["claimeval.compare"] = analysis.compare
+sys.modules["claimeval.baselines"] = analysis.baselines
+sys.modules["claimeval.report"] = analysis.report
+sys.modules["claimeval.simulate"] = sim.simulate
+sys.modules["claimeval.coverage"] = sim.coverage
+
+__version__ = "1.0.0"
+
+__all__ = [
+    "core",
+    "stats",
+    "analysis",
+    "sim",
+    "Claim",
+    "evaluate_claims",
+    "auto_mode",
+    "wild_intervals",
+    "Rollout",
+    "RolloutSet",
+    "FieldMap",
+    "guess_map",
+    "inspect_path",
+    "load",
+    "analyze",
+    "headline",
+    "format_report",
+    "hierarchical_bootstrap",
+    "BootstrapResult",
+    "variance_decomposition",
+    "pareto_mask",
+    "dominance",
+    "frontier_points",
+    "hausdorff",
+    "frontier_band",
+    "log_cost_range",
+    "split_sample_debias",
+    "seed_count_curve",
+    "simulate",
+    "make_population",
+    "sample_dataset",
+    "population_truth",
+    "coverage_study",
+    "format_coverage",
+]

claimeval-1.0.0/claimeval/__main__.py

@@ -0,0 +1,177 @@
+"""CLI: claimeval {demo,inspect,analyze,claims,seed-curve,coverage}"""
+
+from __future__ import annotations
+
+import argparse
+import glob
+import os
+import json
+import sys
+
+import numpy as np
+
+from claimeval.stats.bias import seed_count_curve
+from claimeval.core.ingest import FieldMap, guess_map, inspect_path, load
+from claimeval.analysis.report import analyze
+from claimeval.core.schema import RolloutSet
+from claimeval.sim.simulate import simulate
+
+
+def _load(args) -> RolloutSet:
+    src = getattr(args, "source", None)
+    if src:
+        from claimeval.core import adapters
+        if src == "tau2":
+            rs, skipped = adapters.tau2(sorted(glob.glob(os.path.join(args.path, "*.json")))
+                                        if os.path.isdir(args.path) else [args.path])
+        elif src == "swebench-cost":
+            rs, skipped = adapters.swebench(args.path, with_cost=True)
+        else:
+            rs, skipped = {"swebench": adapters.swebench,
+                           "terminal-bench": adapters.terminal_bench}[src](args.path)
+        for k, v in skipped.items():
+            print(f"[skipped] {k}: {v}", file=sys.stderr)
+        return rs
+    if args.map:
+        fmap = FieldMap.from_dict(json.loads(open(args.map).read()))
+    else:
+        info = inspect_path(args.path)
+        fmap = info["guess"]
+        print(f"[guessed mapping] {fmap}\n", file=sys.stderr)
+    rs, dropped = load(args.path, fmap)
+    if dropped:
+        print(f"[dropped records] {dropped}\n", file=sys.stderr)
+    return rs
+
+
+def _print_curve(rows: list[dict]) -> None:
+    print(f"\n{'k':>3}{'Jaccard vs full':>18}{'Hausdorff':>12}"
+          f"{'acc inflation':>16}{'cost inflation':>16}{'|front|':>9}")
+    print("-" * 74)
+    for r in rows:
+        print(
+            f"{r['k']:>3}{r['jaccard_vs_full']:>18.3f}{r['hausdorff_vs_full']:>12.4f}"
+            f"{r['acc_inflation']:>+16.4f}{r['cost_inflation']:>+16.4f}"
+            f"{r['frontier_size']:>9.1f}"
+        )
+    print("\nacc inflation = what a k-seed study claims for its own frontier winners,")
+    print("minus what the full data says about those same configs.")
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="claimeval")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    d = sub.add_parser("demo", help="end-to-end run on simulated data")
+    d.add_argument("--seeds", type=int, default=20)
+    d.add_argument("--tasks", type=int, default=50)
+    d.add_argument("--configs", type=int, default=9)
+    d.add_argument("--boot", type=int, default=2000)
+
+    i = sub.add_parser("inspect", help="discover the schema of a results dump")
+    i.add_argument("path")
+
+    a = sub.add_parser("analyze", help="audit frontier claims in a results dump")
+    a.add_argument("path")
+    a.add_argument("--map", help="JSON field mapping; omit to auto-guess")
+    a.add_argument("--from", dest="source", choices=["swebench", "swebench-cost", "terminal-bench", "tau2"],
+                   help="read a public leaderboard release directly (see claimeval.adapters)")
+
+    a.add_argument("--boot", type=int, default=2000)
+    a.add_argument("--threshold", type=float, default=0.90)
+
+    c = sub.add_parser("claims", help="test published claims (top1, pairwise, margin, "
+                                      "cost_dominance, frontier) against a results dump")
+    c.add_argument("path")
+    c.add_argument("claims", help="JSON list of {kind, subject, comparators, margin, id}")
+    c.add_argument("--map")
+    c.add_argument("--from", dest="source", choices=["swebench", "swebench-cost", "terminal-bench", "tau2"],
+                   help="read a public leaderboard release directly (see claimeval.adapters)")
+
+    c.add_argument("--boot", type=int, default=4000)
+    c.add_argument("--threshold", type=float, default=0.90)
+    c.add_argument("--mode", default=None,
+                   choices=["task", "cluster_t"],
+                   help="resampling unit; default: clusters if the data has them, else tasks")
+
+    s = sub.add_parser("seed-curve", help="error of a k-seed frontier vs the full data")
+    s.add_argument("path")
+    s.add_argument("--map")
+    s.add_argument("--ks", default="1,2,3,5,10")
+
+    cv = sub.add_parser("coverage", help="validate the estimator against known truth")
+    cv.add_argument("--reps", type=int, default=200)
+    cv.add_argument("--configs", type=int, default=9)
+    cv.add_argument("--tasks", type=int, default=50)
+    cv.add_argument("--seeds", type=int, default=5)
+    cv.add_argument("--boot", type=int, default=500)
+    cv.add_argument("--level", type=float, default=0.95)
+    cv.add_argument("--threshold", type=float, default=0.90)
+    cv.add_argument("--mode", default="task", choices=["task", "both", "seed"],
+                    help="bootstrap mode; 'both' double-counts within-cell noise")
+
+    args = ap.parse_args(argv)
+    rng = np.random.default_rng(0)
+
+    if args.cmd == "demo":
+        rolls, truth = simulate(args.configs, args.tasks, args.seeds, rng)
+        rs = RolloutSet(rolls)
+        report, head = analyze(rs, n_boot=args.boot, rng=rng)
+        print(report)
+        print("\nGround truth accuracy (simulator):")
+        for c, t in zip(truth["configs"], truth["accuracy"]):
+            print(f"  {c:<26}{t:.3f}")
+        _print_curve(seed_count_curve(rs, (1, 2, 3, 5, 10), n_repeats=30, rng=rng))
+        return 0
+
+    if args.cmd == "coverage":
+        from claimeval.sim.coverage import coverage_study, format_coverage
+
+        res = coverage_study(
+            n_reps=args.reps, n_configs=args.configs, n_tasks=args.tasks,
+            n_seeds=args.seeds, n_boot=args.boot, level=args.level,
+            threshold=args.threshold, boot_mode=args.mode,
+        )
+        print(format_coverage(res))
+        return 0
+
+    if args.cmd == "inspect":
+        info = inspect_path(args.path)
+        print(f"{info['n_records']} records scanned\n")
+        print(f"{'field':<46}{'count':>8}  samples")
+        print("-" * 78)
+        for k, n in info["keys"].most_common(60):
+            ex = ", ".join(str(v)[:18] for v in info["samples"].get(k, []))
+            print(f"{k[:46]:<46}{n:>8}  {ex[:24]}")
+        print(f"\nguessed mapping: {info['guess']}")
+        print("Verify this, then pass it with --map. Wrong success/cost fields")
+        print("produce a confident, meaningless frontier.")
+        return 0
+
+    rs = _load(args)
+    if args.cmd == "claims":
+        from claimeval.analysis.claims import Claim, evaluate_claims
+
+        spec = json.loads(open(args.claims).read())
+        cl = [Claim(x["kind"], x["subject"], x.get("comparators", []), x.get("margin", 0.0),
+                    x.get("id", ""), x.get("k", 1)) for x in spec]
+        res = evaluate_claims(rs, cl, n_boot=args.boot, mode=args.mode, rng=rng, thr=args.threshold)
+        print(f"{'id':<10}{'kind':<16}{'subject':<28}{'P(claim)':>9}  verdict")
+        for r in res:
+            v = "supported" if r["supported"] else (
+                "NOT supported" if r["holds_at_point_estimate"] else "contradicted by point estimate")
+            print(f"{r['id'][:9]:<10}{r['kind']:<16}{r['subject'][:27]:<28}{r['probability']:>9.3f}  {v}")
+        print(f"\nsupported = P >= {args.threshold}; resampling: {res[0]['mode'] if res else '-'}")
+        return 0
+    if args.cmd == "analyze":
+        report, _ = analyze(rs, n_boot=args.boot, threshold=args.threshold, rng=rng)
+        print(report)
+    else:
+        ks = tuple(int(x) for x in args.ks.split(","))
+        print(rs.describe())
+        _print_curve(seed_count_curve(rs, ks, n_repeats=30, rng=rng))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())