agentstats 0.3.0__tar.gz → 0.4.0__tar.gz

{agentstats-0.3.0 → agentstats-0.4.0}/CHANGELOG.md

@@ -1,5 +1,47 @@
 # Changelog
 
+## 0.4.0 (2026-10-30)
+
+**Statistics**
+- The sampled design no longer crashes on rules-only rows (no verdicts at all); it reports the rule checks like the
+  curated design does. Found by agent-platform's weekly monitoring job.
+- Few failures: when either outcome is rare (under 10 failures, or passes, across both arms) the fixed-cases test
+  falls back to Newcombe's hybrid score interval on the pooled counts. Before, no failures at all gave a zero-width
+  interval (+0.0 to +0.0), so `switch` could say SWITCH on 120 vs 120 passes and rule comparisons looked exact. In
+  simulation (40 cases x 3 trials, base 0.99 vs candidate 0.95) coverage of the true difference rose from about 92%
+  to 97%. Found by the Acme example's model switch, which now says NOT YET.
+
+**Example**
+- `examples/acme/`: "a week at Acme", two real agents (a tool-calling support agent with OpenTelemetry traces, an
+  Inspect AI coding agent) through the whole back half: smoke and nightly gates, behaviour rules on production traces,
+  an unavailable model reading as UNHEALTHY, the rubric judge, sampled human review, promote, and a model switch.
+  Replays from recorded runs with no API keys (`bash week.sh`). The README records what went wrong while building it.
+
+**Inputs**
+- Inspect AI eval logs (`.eval` and `.json`): read directly by every command, or `agentstats convert --format inspect`
+  with `--scorer`, `--pass-threshold`, `--errors {infra,fail}`, `--rules`, `--arm`. One row per sample and epoch;
+  sample errors are infrastructure failures by default; limits and N scores become `no_answer`; tool calls feed rule
+  checks; metadata becomes tags. Uses inspect-ai's reader when installed (no new dependency). A log where no sample
+  was scored (e.g. every sample errored) reads as infrastructure failures instead of failing on scorer choice.
+
+**Eval sets**
+- `agentstats promote find` / `promote accept`: failures (failed verdicts, or rule violations on rules-only production
+  rows) grouped by signature and deduplicated into candidate cases with provenance tags (source, found, label);
+  accepting appends approved candidates to the cases file and prints the provenance mix, warning when one model
+  labelled most of the eval set.
+
+**Judges**
+- `agentstats grade`: an LLM judge on binary rubric questions over any OpenAI-compatible endpoint (standard library,
+  own User-Agent: Together's edge returns 403 to Python's default one). Writes judge label rows (each repeat, for
+  consistency) and, with `--rows`, attempt rows whose checks are the per-question majority. Unparsed replies are `na`,
+  transport failures infrastructure errors. Warns when a question's name is negated (`no_...`) and its passing answer
+  is "no": on a real run such a double negative made the judge flip between yes and no on the same reply.
+- `agentstats review sample` / `review import`: sampled human review of graded attempts. The sample favours attempts
+  where the verdict and the behaviour rules disagree, then failures, then passes, spread over a tag (`--stratify-by`);
+  the sheet (CSV) is blind, with verdicts, strata and sampling weights in a separate key. Import writes label rows for
+  `agentstats judge` and reports agreement, leniency and strictness on the sample and reweighted to the population
+  (stratified bootstrap 95% intervals).
+
 ## 0.3.0 (2026-10-29)
 
 First release on PyPI.

{agentstats-0.3.0 → agentstats-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: agentstats
-Version: 0.3.0
+Version: 0.4.0
 Summary: Statistics for agent evals: from trajectories to claims you can defend (CIs chosen by how your eval set was built, pass^k, infra failures kept apart, CI gates, behaviour rules, judge checks, pre-registered model-switch decisions).
 Project-URL: Homepage, https://github.com/sidreddy88/agentstats
 Project-URL: Repository, https://github.com/sidreddy88/agentstats
@@ -28,14 +28,23 @@ claim the numbers support.
 pip install agentstats        # Python 3.11+
 agentstats summary jobs/run1 jobs/run2 jobs/run3
 agentstats compare --a jobs/baseline --b jobs/candidate
+agentstats review sample rows.jsonl --n 40 --stratify-by intent --out review/wk1   # blind sheet for a person
+agentstats grade items.jsonl --rubric rubric.json --model MODEL --base-url URL --repeats 3 --out labels.jsonl
 agentstats judge labels.jsonl
+agentstats promote find prod_rows.jsonl --source production --existing evals/cases.jsonl --out candidates.jsonl
 agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/
 agentstats convert-traces traces.json --format otlp --rules rules.json --out rows.jsonl   # or --format langfuse
+agentstats summary logs/my-task.eval                     # Inspect AI logs are read directly
 agentstats preregister --name my-switch --margin 0.03 --provenance independent --out decisions/my-switch
 agentstats switch --prereg decisions/my-switch/prereg.json --current current.jsonl --candidate cheap.jsonl \
     --out decisions/my-switch
 ```
 
+**See it on two real agents:** [examples/acme](https://github.com/sidreddy88/agentstats/tree/main/examples/acme)
+replays "a week at Acme" (a support agent and an Inspect coding agent) from recorded runs: a PR blocked at smoke, a
+nightly regression found by slice, a lenient LLM judge caught by sampled human review, production failures promoted
+into the eval set, and a model switch. `bash week.sh`, no API keys.
+
 ## Why
 
 Agents are stochastic, so a single eval run is a sample, not a measurement. On 100 SWE-bench Verified tasks, the
@@ -55,7 +64,8 @@ Three other things go wrong in practice:
 
 Other good tools exist (see [Related tools](#related-tools)). agentstats is built around agent runs:
 
-1. **Reads trajectories, not score tables.** Harbor job directories and ATIF trajectories directly: pass/fail,
+1. **Reads trajectories, not score tables.** Harbor job directories, ATIF trajectories, Inspect AI logs and
+   OpenTelemetry/Langfuse traces directly: pass/fail,
    infrastructure errors, tokens, cost, latency, steps and tool calls, plus deterministic rule checks on the
    tool calls (`called`, `not_called`, `before`, `args`).
 2. **Picks the statistics from how the eval set was built, and says what claim they support.**
@@ -293,6 +303,45 @@ calls). One trace is one attempt; `--rules` fills its checks. Pass/fail comes fr
 `trace_id`, `passed`, optional `case`, `run`, `infra_error`); without it the rows are **rules-only**: no pass rate, but
 the rule gates work, which is what production traces without labels need.
 
+**From Inspect AI logs** (`.eval` or `.json`; read directly by every command, or
+`agentstats convert --format inspect LOG --rules rules.json --out rows.jsonl`): one row per sample and epoch (case =
+sample id, run = epoch, arm = model or `--arm`). Pass/fail comes from the headline scorer or `--scorer`: C passes; I
+and P fail; N fails as `no_answer`; numbers pass at `--pass-threshold` (default 1.0). A sample that errored is an
+infrastructure failure by default (`--errors fail` counts it as a failure); a sample that hit a limit and didn't pass
+is `no_answer: limit_<type>`. Tool calls from the assistant messages fill `--rules` checks; tokens, time, steps and
+tool calls become metrics; task, model and scalar sample metadata become tags (so `slice_by = "intent"` works).
+`.eval` files are zstd-compressed: they're read with inspect-ai's own reader when it's installed, or the standard
+library on Python 3.14+; otherwise `inspect log convert --to json` first.
+
+**LLM judge on yes/no rubric questions** (`agentstats grade items.jsonl --rubric rubric.json --model MODEL
+--base-url URL --api-key-env VAR --repeats 3 --out labels.jsonl --rows rows.jsonl`): items are
+`{"item", "input", "output", ...}` (every field but ids and tags is shown to the judge, or `--fields`); the rubric is a
+list of `{"name", "question", "pass": "yes"|"no"}`, and an item passes when every question gets its passing answer.
+Any OpenAI-compatible endpoint, called with the standard library. A reply that isn't the JSON asked for is recorded
+as `na`, never a silent pass; transport errors are retried, then recorded as infrastructure errors. Repeats are judge
+labels (for `agentstats judge`'s flip rate), never extra attempts; `--rows` writes one attempt row per item with the
+per-question majority as checks, so a rubric question can be gated like a behaviour rule.
+
+**Sampled human review** (`agentstats review sample ROWS --n 40 --stratify-by intent --link URL --out DIR`, then
+`agentstats review import DIR/review.csv --key DIR/key.jsonl --rater NAME --out labels.jsonl`): picks graded attempts
+for a person to check, favouring those where the verdict and the behaviour rules disagree (passed with a rule
+violated, or failed with every rule satisfied), then failures, then passes, spread over a tag. The sheet is blind
+(no verdicts, no reason for the pick); the key keeps each item's verdict, stratum and sampling weight. Import appends
+label rows for `agentstats judge` and reports the grader's agreement, leniency and strictness both on the sample and
+**reweighted to the population** (over-sampling disagreements would otherwise make the grader look worse than it is),
+with stratified bootstrap intervals.
+
+**Failures become eval cases** (`agentstats promote find ROWS --source production --label-from none --existing
+evals/cases.jsonl --group-by intent --out candidates.jsonl`, then `agentstats promote accept candidates.jsonl --into
+evals/cases.jsonl`): failures from eval runs, or rule violations on rules-only production rows, are grouped by
+signature (the rules that broke, the no-answer reason, optionally a slice tag), so one failure seen thirty times is one
+signature with up to `--per-signature` candidate cases (the most frequent first). Cases already in the eval set are
+skipped. Each candidate is tagged `source:`, `found:` and `label:` (`human`, `none`, or `model:<name>`); a person sets
+`"approved": true` and `accept` appends those to the cases file (the one `agentstats sample` reads) and prints the eval
+set's provenance mix, warning when most labels came from one model: comparisons against that model are then tilted
+in its favour, because its own answers are the answer key. agentstats keeps only case ids and tags; the inputs and
+expected answers go in your runner's dataset.
+
 Judge labels (`agentstats judge`), one row per verdict:
 
 ```json
@@ -306,6 +355,9 @@ A synthetic judge-labels file is in `examples/judge_labels.jsonl`.
 ## Methods and sources
 
 - Error bars for evals, clustered standard errors: Miller 2024, [arXiv 2411.00640](https://arxiv.org/abs/2411.00640).
+- Few failures (fewer than 10 of either outcome across both arms): Newcombe's hybrid score interval for a difference
+  of proportions (method 10), Newcombe 1998, [Statistics in Medicine 17:873-890](https://doi.org/10.1002/(SICI)1097-0258(19980430)17:8%3C873::AID-SIM779%3E3.0.CO;2-I),
+  instead of a normal approximation that collapses to zero width when nothing fails.
 - Choosing the method by how the eval set was built, evals in CI tiers, judge alignment and consistency metrics:
   Yesha Shah's "Agentic Evals" series (see [Credits](#credits)).
 - pass@k: Chen et al. 2021, [arXiv 2107.03374](https://arxiv.org/abs/2107.03374). pass^k: τ-bench, Yao et al.,

{agentstats-0.3.0 → agentstats-0.4.0}/README.md

@@ -9,14 +9,23 @@ claim the numbers support.
 pip install agentstats        # Python 3.11+
 agentstats summary jobs/run1 jobs/run2 jobs/run3
 agentstats compare --a jobs/baseline --b jobs/candidate
+agentstats review sample rows.jsonl --n 40 --stratify-by intent --out review/wk1   # blind sheet for a person
+agentstats grade items.jsonl --rubric rubric.json --model MODEL --base-url URL --repeats 3 --out labels.jsonl
 agentstats judge labels.jsonl
+agentstats promote find prod_rows.jsonl --source production --existing evals/cases.jsonl --out candidates.jsonl
 agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/
 agentstats convert-traces traces.json --format otlp --rules rules.json --out rows.jsonl   # or --format langfuse
+agentstats summary logs/my-task.eval                     # Inspect AI logs are read directly
 agentstats preregister --name my-switch --margin 0.03 --provenance independent --out decisions/my-switch
 agentstats switch --prereg decisions/my-switch/prereg.json --current current.jsonl --candidate cheap.jsonl \
     --out decisions/my-switch
 ```
 
+**See it on two real agents:** [examples/acme](https://github.com/sidreddy88/agentstats/tree/main/examples/acme)
+replays "a week at Acme" (a support agent and an Inspect coding agent) from recorded runs: a PR blocked at smoke, a
+nightly regression found by slice, a lenient LLM judge caught by sampled human review, production failures promoted
+into the eval set, and a model switch. `bash week.sh`, no API keys.
+
 ## Why
 
 Agents are stochastic, so a single eval run is a sample, not a measurement. On 100 SWE-bench Verified tasks, the
@@ -36,7 +45,8 @@ Three other things go wrong in practice:
 
 Other good tools exist (see [Related tools](#related-tools)). agentstats is built around agent runs:
 
-1. **Reads trajectories, not score tables.** Harbor job directories and ATIF trajectories directly: pass/fail,
+1. **Reads trajectories, not score tables.** Harbor job directories, ATIF trajectories, Inspect AI logs and
+   OpenTelemetry/Langfuse traces directly: pass/fail,
    infrastructure errors, tokens, cost, latency, steps and tool calls, plus deterministic rule checks on the
    tool calls (`called`, `not_called`, `before`, `args`).
 2. **Picks the statistics from how the eval set was built, and says what claim they support.**
@@ -274,6 +284,45 @@ calls). One trace is one attempt; `--rules` fills its checks. Pass/fail comes fr
 `trace_id`, `passed`, optional `case`, `run`, `infra_error`); without it the rows are **rules-only**: no pass rate, but
 the rule gates work, which is what production traces without labels need.
 
+**From Inspect AI logs** (`.eval` or `.json`; read directly by every command, or
+`agentstats convert --format inspect LOG --rules rules.json --out rows.jsonl`): one row per sample and epoch (case =
+sample id, run = epoch, arm = model or `--arm`). Pass/fail comes from the headline scorer or `--scorer`: C passes; I
+and P fail; N fails as `no_answer`; numbers pass at `--pass-threshold` (default 1.0). A sample that errored is an
+infrastructure failure by default (`--errors fail` counts it as a failure); a sample that hit a limit and didn't pass
+is `no_answer: limit_<type>`. Tool calls from the assistant messages fill `--rules` checks; tokens, time, steps and
+tool calls become metrics; task, model and scalar sample metadata become tags (so `slice_by = "intent"` works).
+`.eval` files are zstd-compressed: they're read with inspect-ai's own reader when it's installed, or the standard
+library on Python 3.14+; otherwise `inspect log convert --to json` first.
+
+**LLM judge on yes/no rubric questions** (`agentstats grade items.jsonl --rubric rubric.json --model MODEL
+--base-url URL --api-key-env VAR --repeats 3 --out labels.jsonl --rows rows.jsonl`): items are
+`{"item", "input", "output", ...}` (every field but ids and tags is shown to the judge, or `--fields`); the rubric is a
+list of `{"name", "question", "pass": "yes"|"no"}`, and an item passes when every question gets its passing answer.
+Any OpenAI-compatible endpoint, called with the standard library. A reply that isn't the JSON asked for is recorded
+as `na`, never a silent pass; transport errors are retried, then recorded as infrastructure errors. Repeats are judge
+labels (for `agentstats judge`'s flip rate), never extra attempts; `--rows` writes one attempt row per item with the
+per-question majority as checks, so a rubric question can be gated like a behaviour rule.
+
+**Sampled human review** (`agentstats review sample ROWS --n 40 --stratify-by intent --link URL --out DIR`, then
+`agentstats review import DIR/review.csv --key DIR/key.jsonl --rater NAME --out labels.jsonl`): picks graded attempts
+for a person to check, favouring those where the verdict and the behaviour rules disagree (passed with a rule
+violated, or failed with every rule satisfied), then failures, then passes, spread over a tag. The sheet is blind
+(no verdicts, no reason for the pick); the key keeps each item's verdict, stratum and sampling weight. Import appends
+label rows for `agentstats judge` and reports the grader's agreement, leniency and strictness both on the sample and
+**reweighted to the population** (over-sampling disagreements would otherwise make the grader look worse than it is),
+with stratified bootstrap intervals.
+
+**Failures become eval cases** (`agentstats promote find ROWS --source production --label-from none --existing
+evals/cases.jsonl --group-by intent --out candidates.jsonl`, then `agentstats promote accept candidates.jsonl --into
+evals/cases.jsonl`): failures from eval runs, or rule violations on rules-only production rows, are grouped by
+signature (the rules that broke, the no-answer reason, optionally a slice tag), so one failure seen thirty times is one
+signature with up to `--per-signature` candidate cases (the most frequent first). Cases already in the eval set are
+skipped. Each candidate is tagged `source:`, `found:` and `label:` (`human`, `none`, or `model:<name>`); a person sets
+`"approved": true` and `accept` appends those to the cases file (the one `agentstats sample` reads) and prints the eval
+set's provenance mix, warning when most labels came from one model: comparisons against that model are then tilted
+in its favour, because its own answers are the answer key. agentstats keeps only case ids and tags; the inputs and
+expected answers go in your runner's dataset.
+
 Judge labels (`agentstats judge`), one row per verdict:
 
 ```json
@@ -287,6 +336,9 @@ A synthetic judge-labels file is in `examples/judge_labels.jsonl`.
 ## Methods and sources
 
 - Error bars for evals, clustered standard errors: Miller 2024, [arXiv 2411.00640](https://arxiv.org/abs/2411.00640).
+- Few failures (fewer than 10 of either outcome across both arms): Newcombe's hybrid score interval for a difference
+  of proportions (method 10), Newcombe 1998, [Statistics in Medicine 17:873-890](https://doi.org/10.1002/(SICI)1097-0258(19980430)17:8%3C873::AID-SIM779%3E3.0.CO;2-I),
+  instead of a normal approximation that collapses to zero width when nothing fails.
 - Choosing the method by how the eval set was built, evals in CI tiers, judge alignment and consistency metrics:
   Yesha Shah's "Agentic Evals" series (see [Credits](#credits)).
 - pass@k: Chen et al. 2021, [arXiv 2107.03374](https://arxiv.org/abs/2107.03374). pass^k: τ-bench, Yao et al.,

{agentstats-0.3.0 → agentstats-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "agentstats"
-version = "0.3.0"
+version = "0.4.0"
 description = "Statistics for agent evals: from trajectories to claims you can defend (CIs chosen by how your eval set was built, pass^k, infra failures kept apart, CI gates, behaviour rules, judge checks, pre-registered model-switch decisions)."
 readme = "README.md"
 license = "MIT"
@@ -36,3 +36,6 @@ target-version = "py311"
 
 [tool.ruff.lint]
 select = ["E", "F", "W", "I", "UP", "B"]
+
+[tool.hatch.build.targets.sdist]
+exclude = ["examples/acme"]   # recorded runs: on GitHub, not in the package

{agentstats-0.3.0 → agentstats-0.4.0}/src/agentstats/__init__.py

@@ -1,3 +1,3 @@
 """agentstats: statistics for agent evals."""
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"

{agentstats-0.3.0 → agentstats-0.4.0}/src/agentstats/analysis.py

@@ -126,8 +126,9 @@ def _headline(design: str, runs: dict[int, float], counts: dict[str, tuple[int,
         notes.append(f"Can't use a clustered interval: {why}. Falling back to the curated (run-level) design.")
         design = "curated"
     if design == "sampled":
-        if k == 1:
-            notes.append("One run: per-case outcomes are binary, so a Wilson interval is used.")
+        if k <= 1 or len(p) < 2:               # one run, or too few graded cases (none, for rules-only rows)
+            if k == 1:
+                notes.append("One run: per-case outcomes are binary, so a Wilson interval is used.")
             return wilson(total_c, total_n, conf), notes, alt
         if len(p) < 30:
             notes.append(f"Only {len(p)} cases: a per-case interval this small may be unreliable; add cases.")

{agentstats-0.3.0 → agentstats-0.4.0}/src/agentstats/cli.py

@@ -35,6 +35,73 @@ def _load_many(paths: list[str], arm: str | None) -> list[records.Attempt]:
     return out
 
 
+def _review(args) -> int:
+    from agentstats import review
+    if args.review_cmd == "sample":
+        atts = _load_many(args.rows, None)
+        sheet, key, summary = review.sample(atts, args.n, stratify_by=args.stratify_by, seed=args.seed,
+                                            link=args.link)
+        sheet_path, key_path = review.write(args.out, sheet, key, summary)
+        print(f"wrote {sheet_path} ({len(sheet)} items: {summary['sampled']}) and {key_path}. "
+              "Give the reviewer only review.csv.")
+        return 0
+    labels, report = review.import_labels(args.sheet, args.key, rater=args.rater)
+    out = Path(args.out)
+    with out.open("a") as f:
+        f.writelines(json.dumps(r) + "\n" for r in labels)
+    print(json.dumps(report, indent=2) if args.json else review.render(report))
+    print(f"appended {len(labels)} label rows to {out}; run `agentstats judge {out}` for the full history")
+    return 0
+
+
+def _promote(args) -> int:
+    from agentstats import promote
+    if args.promote_cmd == "find":
+        existing = set(sample.load_cases(args.existing)) if args.existing else set()
+        cands, report = promote.find(_load_many(args.rows, None), source=args.source, label_from=args.label_from,
+                                     existing=existing, group_by=args.group_by, per_signature=args.per_signature,
+                                     found=args.found)
+        Path(args.out).write_text("".join(json.dumps(c) + "\n" for c in cands))
+        print(promote.render_report(report))
+        print(f"\nwrote {len(cands)} candidates to {args.out}. Set \"approved\": true on the ones to keep, then "
+              f"`agentstats promote accept {args.out} --into <cases.jsonl>`.")
+        return 0
+    added, prov = promote.accept(args.candidates, args.into)
+    print(f"added {added} cases to {args.into} ({prov['cases']} in total). Add their inputs and expected answers "
+          "to your runner's dataset.")
+    print("labels: " + ", ".join(f"{k} {v}" for k, v in sorted(prov["labels"].items())))
+    print("sources: " + ", ".join(f"{k} {v}" for k, v in sorted(prov["sources"].items())))
+    for w in prov["warnings"]:
+        print("WARNING: " + w)
+    return 0
+
+
+def _grade(args, chat=None) -> int:
+    from agentstats import judge_run
+    rubric, items = judge_run.load_rubric(args.rubric), judge_run.load_items(args.items)
+    for w in judge_run.rubric_warnings(rubric):
+        print("WARNING: " + w)
+    if chat is None:
+        key = os.environ.get(args.api_key_env)
+        if not key:
+            print(f"set {args.api_key_env} (or choose another variable with --api-key-env)")
+            return 2
+        chat = judge_run.http_chat(args.base_url, args.model, key,
+                                   temperature=None if args.temperature < 0 else args.temperature)
+    fields = [f.strip() for f in args.fields.split(",")] if args.fields else None
+    labels, rows, counts = judge_run.run(items, rubric, chat, repeats=args.repeats, workers=args.workers,
+                                         fields=fields)
+    Path(args.out).write_text("".join(json.dumps(r) + "\n" for r in labels))
+    if args.rows:
+        Path(args.rows).write_text("".join(json.dumps(r) + "\n" for r in rows))
+    print(f"{counts['calls']} judge calls: {counts['judged']} judged, {counts['unparsed']} unparsed replies, "
+          f"{counts['errors']} errors. Wrote {args.out}" + (f" and {args.rows}" if args.rows else "") + ".")
+    if args.repeats > 1:
+        print(f"Judge consistency: `agentstats judge {args.out}`; add human labels to the same file to check "
+              "agreement.")
+    return 0 if counts["judged"] else 3
+
+
 def main(argv: list[str] | None = None) -> int:
     ap = argparse.ArgumentParser(prog="agentstats", description="Statistics for agent evals.")
     sub = ap.add_subparsers(dest="cmd", required=True)
@@ -60,9 +127,16 @@ def main(argv: list[str] | None = None) -> int:
     k.add_argument("--rules", required=True, help="rules JSON (see agentstats.checks)")
     k.add_argument("--out", required=True, help="output JSONL with checks filled in")
 
-    v = sub.add_parser("convert", help="Harbor job dir(s) to agentstats JSONL")
-    v.add_argument("jobs", nargs="+")
-    v.add_argument("--arm")
+    v = sub.add_parser("convert", help="Harbor job dir(s) or Inspect AI logs to agentstats JSONL")
+    v.add_argument("jobs", nargs="+",
+                   help="Harbor job directories, or Inspect logs (.eval / .json) with --format inspect")
+    v.add_argument("--format", choices=["harbor", "inspect"], default="harbor")
+    v.add_argument("--scorer", help="Inspect: which scorer gives pass/fail (default: the log's headline scorer)")
+    v.add_argument("--pass-threshold", type=float, default=1.0, help="Inspect: numeric scores pass at or above this")
+    v.add_argument("--errors", choices=["infra", "fail"], default="infra",
+                   help="Inspect: a sample that errored is an infrastructure failure (default) or a failed attempt")
+    v.add_argument("--rules", help="Inspect: rules JSON; fills each row's checks from the sample's tool calls")
+    v.add_argument("--arm", help="arm name (Inspect default: the log's model)")
     v.add_argument("--out", required=True)
 
     tr = sub.add_parser("convert-traces", help="OpenTelemetry (OTLP JSON) or Langfuse traces to agentstats JSONL")
@@ -108,10 +182,59 @@ def main(argv: list[str] | None = None) -> int:
     ic.add_argument("--dir", default=".")
     ic.add_argument("--force", action="store_true")
 
+    rv = sub.add_parser("review", help="sampled human review of graded attempts: is the grader right?")
+    rsub = rv.add_subparsers(dest="review_cmd", required=True)
+    rs = rsub.add_parser("sample", help="pick attempts for a person to review (blind sheet + key)")
+    rs.add_argument("rows", nargs="+", help="attempt rows: JSONL, Inspect logs or Harbor jobs (one per run)")
+    rs.add_argument("--n", type=int, default=40, help="how many to review (default 40)")
+    rs.add_argument("--stratify-by", help="tag prefix to spread the sample over, e.g. intent for intent:<name>")
+    rs.add_argument("--seed", type=int, default=0)
+    rs.add_argument("--link", help="link template for each item, e.g. 'https://traces/{case}?run={run}' ({arm} too)")
+    rs.add_argument("--out", required=True, help="a new directory: review.csv (for the reviewer), key.jsonl")
+    ri = rsub.add_parser("import", help="read a filled review sheet: label rows for `judge`, plus agreement")
+    ri.add_argument("sheet", help="the filled review.csv")
+    ri.add_argument("--key", required=True, help="key.jsonl written next to the sheet")
+    ri.add_argument("--rater", default="human", help="the reviewer's name (labels get rater human:<name>)")
+    ri.add_argument("--out", required=True, help="label rows (JSONL) for `agentstats judge`; appended if it exists")
+    ri.add_argument("--json", action="store_true")
+    pm = sub.add_parser("promote", help="failures become candidate eval cases, deduplicated, with provenance")
+    psub = pm.add_subparsers(dest="promote_cmd", required=True)
+    pf = psub.add_parser("find", help="group failures by signature and propose new cases")
+    pf.add_argument("rows", nargs="+", help="attempt rows: eval runs or rules-only production rows")
+    pf.add_argument("--source", required=True, help="where the failures came from, e.g. production or nightly")
+    pf.add_argument("--label-from", default="none",
+                    help="who will write the expected answer: none (default), human, or model:<name>")
+    pf.add_argument("--existing", help="the eval set's cases file (JSONL of {case, tags}); its cases are skipped")
+    pf.add_argument("--group-by", help="tag prefix that splits signatures, e.g. intent")
+    pf.add_argument("--per-signature", type=int, default=3, help="cases proposed per signature (default 3)")
+    pf.add_argument("--found", help="date tag (default: today, UTC)")
+    pf.add_argument("--out", required=True, help="candidates JSONL; set \"approved\": true on the ones to keep")
+    pa = psub.add_parser("accept", help="append approved candidates to the eval set; print its provenance mix")
+    pa.add_argument("candidates")
+    pa.add_argument("--into", required=True, help="the eval set's cases file (created if missing)")
+    gr = sub.add_parser("grade", help="LLM judge on yes/no rubric questions (any OpenAI-compatible endpoint)")
+    gr.add_argument("items", help="JSONL: {item, input, output, ...}; case/run/arm/tags are carried into --rows")
+    gr.add_argument("--rubric", required=True, help='JSON list: [{"name", "question", "pass": "yes"|"no"}]')
+    gr.add_argument("--model", required=True)
+    gr.add_argument("--base-url", default="https://api.openai.com/v1", help="OpenAI-compatible API base URL")
+    gr.add_argument("--api-key-env", default="OPENAI_API_KEY", help="environment variable holding the API key")
+    gr.add_argument("--temperature", type=float, default=0.0, help="omit with a negative value for models that "
+                    "reject one")
+    gr.add_argument("--repeats", type=int, default=1, help="judge each item this many times (for consistency)")
+    gr.add_argument("--workers", type=int, default=4)
+    gr.add_argument("--fields", help="comma-separated item fields to show the judge (default: all but ids/tags)")
+    gr.add_argument("--out", required=True, help="judge label rows (JSONL) for `agentstats judge`")
+    gr.add_argument("--rows", help="also write attempt rows whose checks are the rubric answers")
     switch_cli.add_parsers(sub)
     args = ap.parse_args(argv)
     if args.cmd in ("preregister", "switch"):
         return switch_cli.run(args)
+    if args.cmd == "review":
+        return _review(args)
+    if args.cmd == "promote":
+        return _promote(args)
+    if args.cmd == "grade":
+        return _grade(args)
     if args.cmd == "summary":
         res = analysis.summarize(_load_many(args.results, None), args.design, args.cluster_by, args.conf)
         print(json.dumps(res, indent=2, default=str) if args.json else report.summary(res))
@@ -181,6 +304,14 @@ def main(argv: list[str] | None = None) -> int:
         graded = sum(1 for a in atts if a.passed is not None)
         print(f"wrote {args.out}: {len(atts)} traces, {graded} with a verdict"
               + ("" if graded else " (rules-only)"))
+    elif args.cmd == "convert" and args.format == "inspect":
+        from agentstats import inspect_logs
+        rules = checks.load_rules(args.rules) if args.rules else None
+        atts = [a for p in args.jobs
+                for a in inspect_logs.load(p, scorer=args.scorer, pass_threshold=args.pass_threshold,
+                                           errors=args.errors, rules=rules, arm=args.arm)]
+        records.write_jsonl(atts, args.out)
+        print(f"wrote {args.out}: {len(atts)} attempts")
     elif args.cmd == "convert":
         records.write_jsonl(_load_many(args.jobs, args.arm), args.out)
         print(f"wrote {args.out}")

{agentstats-0.3.0 → agentstats-0.4.0}/src/agentstats/comparisons.py

@@ -15,7 +15,9 @@ Is B different from A? Each test returns an Interval for the difference B - A (p
     test from agent-platform (app/evals/gate_v2.py) plus that correction. Cases that pass (or fail) every trial on
     both sides contribute nothing; the cases that sometimes pass carry the signal. With one trial per arm it
     reduces to McNemar's test (normal form). Valid when trials are independent across cases: a run-wide shock (a
-    provider's bad hour) breaks that, which analysis.py checks for.
+    provider's bad hour) breaks that, which analysis.py checks for. With fewer than 10 of the rarer outcome (e.g.
+    no failures at all) the variance can't be estimated and would collapse to zero; the test then uses Newcombe's
+    interval on the pooled counts instead (conservative, never zero width).
 
 Smallest detectable difference: about (t_{1-a/2} + z_{power}) x SE; with a = 0.05 and power 0.8, ~2.8 x SE.
 """
@@ -26,7 +28,9 @@ from collections.abc import Sequence
 
 from scipy import stats
 
-from agentstats.intervals import Interval, clustered, mean, sd, t_interval
+from agentstats.intervals import Interval, clustered, mean, newcombe, sd, t_interval
+
+MIN_EVENTS = 10   # below this many of the rarer outcome, the normal approximation isn't trusted
 
 
 def welch(a_runs: Sequence[float], b_runs: Sequence[float], conf: float = 0.95) -> Interval:
@@ -59,10 +63,25 @@ def paired_clustered(diffs: Sequence[float], clusters: Sequence[str], conf: floa
 
 
 def fixed_cases(a_counts: dict[str, tuple[int, int]], b_counts: dict[str, tuple[int, int]], conf: float = 0.95,
-                correction: bool = True) -> Interval:
-    """a_counts / b_counts: case -> (trials, passes), on the same cases."""
+                correction: bool = True, min_events: int = MIN_EVENTS) -> Interval:
+    """a_counts / b_counts: case -> (trials, passes), on the same cases.
+
+    With fewer than `min_events` of the rarer outcome (failures, or passes) across both arms, the per-case variance
+    can't be estimated (with no failures at all it is exactly zero, and the interval would have zero width). Then
+    the interval is Newcombe's on the pooled counts: conservative here (attempts at different per-case rates vary
+    less than one binomial with the same mean, and pairing is ignored), never zero width."""
     cases = sorted(set(a_counts) & set(b_counts))
     n = len(cases)
+    a_n, a_s = sum(a_counts[c][0] for c in cases), sum(a_counts[c][1] for c in cases)
+    b_n, b_s = sum(b_counts[c][0] for c in cases), sum(b_counts[c][1] for c in cases)
+    rarer = min(a_s + b_s, (a_n - a_s) + (b_n - b_s))
+    if n and rarer < min_events:
+        iv = newcombe(a_s, a_n, b_s, b_n, conf)
+        iv.method, iv.n = "fixed_cases (few failures: newcombe)", n
+        iv.note = (f"only {rarer} {'failures' if a_s + b_s > (a_n - a_s) + (b_n - b_s) else 'passes'} across both "
+                   f"arms: too few to estimate per-case noise, so this is Newcombe's interval on pooled counts "
+                   "(conservative). More trials, or harder cases, would narrow it")
+        return iv
     t = v = 0.0
     for c in cases:
         ka, sa = a_counts[c]