agentstats 0.3.0__tar.gz

agentstats-0.3.0/.github/workflows/release.yml

@@ -0,0 +1,40 @@
+name: release
+
+# Publishes to PyPI when a version tag (v0.1.0, ...) is pushed. Uses PyPI trusted publishing:
+# no token is stored anywhere; PyPI trusts this workflow in this repo via the "pypi" environment.
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - name: Tag must match the package version
+        run: |
+          python -c "import tomllib, os; v = tomllib.load(open('pyproject.toml', 'rb'))['project']['version']; t = os.environ['GITHUB_REF_NAME'].removeprefix('v'); assert v == t, f'tag {t} != pyproject version {v}'"
+      - run: pip install build twine
+      - run: python -m build
+      - run: twine check dist/*
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

agentstats-0.3.0/.github/workflows/tests.yml

@@ -0,0 +1,30 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[dev]"
+      - run: pytest -q
+
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+      - run: pip install "ruff>=0.15"
+      - run: ruff check .

agentstats-0.3.0/.gitignore

@@ -0,0 +1,7 @@
+.venv/
+__pycache__/
+*.pyc
+.pytest_cache/
+.ruff_cache/
+dist/
+*.egg-info/

agentstats-0.3.0/CHANGELOG.md

@@ -0,0 +1,63 @@
+# Changelog
+
+## 0.3.0 (2026-10-29)
+
+First release on PyPI.
+
+**Model switches** (moved in from the stepdown package)
+- `agentstats preregister`: write the switch rule (margin, confidence, cost ratio, design, provenance, slices, input
+  hashes) to `prereg.json` + `prereg.md` before any result exists.
+- `agentstats switch`: SWITCH / KEEP / NOT YET for moving to a cheaper model (non-inferiority at a margin + cost
+  ratio, as-is and adapted steps, slices, provenance warning), with a decision record; with `--prereg` the rule comes
+  from the registration and is verified (committed, unchanged, before the results, inputs unchanged).
+- Example: `examples/switch-triage-haiku-5-5/` (a production decision).
+
+**Gates**
+- `no_answer` outcome: attempts with no usable answer (step limit, unparseable output) count as failures but are
+  reported apart in `summary` and the gate; new requirement `max_no_answer`. A gate without it still notes a jump.
+- Per-slice comparisons: `slice_by` in a tier prints candidate vs baseline per slice (fixed-cases, Bonferroni-adjusted
+  across slices, intervals clipped to the possible range); `max_slice_drop` fails on a proven slice drop and names
+  large unproven drops as leads.
+
+- Rule gates: `min_rule_rate` (each trajectory rule holds at least this often, interval-based) and
+  `no_rule_regression` (each rule vs the baseline, fixed-cases, Bonferroni across rules). Behaviour checks that need
+  no labels; the gate prints a per-rule table.
+
+**Inputs**
+- `agentstats convert-traces`: OpenTelemetry (OTLP JSON, GenAI conventions) and Langfuse trace exports to rows; tool
+  calls become rule checks; `--verdicts` joins pass/fail by trace id, otherwise rows are rules-only and the rule gates
+  (and `summary`'s rule table) still work, while pass-rate requirements report INCONCLUSIVE instead of UNHEALTHY.
+- Rows are validated on load with the file, line and field in the error (e.g. `'steps' must be a number, got list`).
+- Optional `schema_version` (1); newer versions are refused. Contract tests load rows written by harness-evolve and by
+  agent-platform's triage gate.
+
+**Statistics**
+- Calibration tests: Wilson coverage, fixed-cases coverage, the drop gate's false-alarm rate (12,000 simulations; the
+  6% bound fails if the small-sample correction is removed) and power, and the shock check's false-positive rate.
+
+## 0.2.0 (not published)
+
+**Gates for evals in CI**
+- `agentstats.toml` policy with tiers (smoke / PR / nightly / release); `agentstats gate` exits 0 pass, 1 fail,
+  2 inconclusive, 3 unhealthy and writes the GitHub job summary.
+- Two regression rules: `no_significant_drop` (fails only on a proven drop, for pull requests) and `no_regression`
+  (non-inferiority at a margin, for releases: an under-powered run is INCONCLUSIVE, with a runs estimate).
+- `agentstats sample`: a deterministic, stratified smoke sample (critical cases always in).
+- `agentstats init-ci`: a starter policy and smoke / nightly / release GitHub workflows.
+- `agentstats size`: simulate a gate rule on your own baseline (pass / fail / inconclusive rates for a range of true
+  drops) to choose trials and margins.
+
+**Statistics**
+- Fixed-cases test for hand-picked eval sets, now the default for `curated`: per-case pass rates vs the baseline's,
+  noise from repeated trials per case, with a small-sample correction.
+- Run-wide shock check (whole runs varying more than per-case randomness predicts): falls back to run-level statistics
+  with 3+ runs; with fewer, the gate asks for a rerun.
+- Runs are compared on the cases attempted in every run; an infrastructure failure no longer removes its case.
+- Pass-rate intervals clipped to the possible range.
+- No graded attempts (every attempt an infrastructure failure) is UNHEALTHY, not a crash.
+
+## 0.1.0
+
+First version: Harbor / ATIF / JSONL inputs; pass rate with intervals chosen by eval-set design (curated / sampled /
+clustered); pass@k and pass^k; infrastructure failures kept apart; efficiency with CIs; trajectory rule checks; judge
+checks (Fleiss / Cohen kappa, strictness, leniency, flip rate); `summary`, `compare`, `judge`, `check`, `convert`.

agentstats-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Siddharth Sukumar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agentstats-0.3.0/PKG-INFO

@@ -0,0 +1,384 @@
+Metadata-Version: 2.5
+Name: agentstats
+Version: 0.3.0
+Summary: Statistics for agent evals: from trajectories to claims you can defend (CIs chosen by how your eval set was built, pass^k, infra failures kept apart, CI gates, behaviour rules, judge checks, pre-registered model-switch decisions).
+Project-URL: Homepage, https://github.com/sidreddy88/agentstats
+Project-URL: Repository, https://github.com/sidreddy88/agentstats
+Project-URL: Changelog, https://github.com/sidreddy88/agentstats/blob/main/CHANGELOG.md
+Author: Siddharth Sukumar
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,ci,confidence-intervals,evals,llm,llm-as-a-judge,non-inferiority,statistics
+Requires-Python: >=3.11
+Requires-Dist: scipy>=1.11
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.15; extra == 'dev'
+Requires-Dist: statsmodels>=0.14; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# agentstats
+
+Statistics for agent evals. Point it at your agent's runs (Harbor jobs, ATIF trajectories or a JSONL file) and
+it tells you how good the agent is, how reliable it is, whether version B is really better than A, and whether a
+cheaper model is shown to be no worse. Every number comes with a confidence interval, and the report says which
+claim the numbers support.
+
+```bash
+pip install agentstats        # Python 3.11+
+agentstats summary jobs/run1 jobs/run2 jobs/run3
+agentstats compare --a jobs/baseline --b jobs/candidate
+agentstats judge labels.jsonl
+agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/
+agentstats convert-traces traces.json --format otlp --rules rules.json --out rows.jsonl   # or --format langfuse
+agentstats preregister --name my-switch --margin 0.03 --provenance independent --out decisions/my-switch
+agentstats switch --prereg decisions/my-switch/prereg.json --current current.jsonl --candidate cheap.jsonl \
+    --out decisions/my-switch
+```
+
+## Why
+
+Agents are stochastic, so a single eval run is a sample, not a measurement. On 100 SWE-bench Verified tasks, the
+same agent with the same model and setup (mini-swe-agent, DeepSeek-V4.1-Flash) scored **48%, then 40%**. Eight
+tasks flipped between identical runs. If you compare two versions on one run each, that noise is your result.
+
+Three other things go wrong in practice:
+
+- **Infrastructure failures get scored as agent failures.** A quota error or a billing outage looks like a bad
+  agent. agentstats keeps them out of pass rates, counts them, and warns if one arm had more.
+- **The wrong interval for how the eval set was built.** Most eval sets are hand-picked. Treating each case as an
+  independent sample of real traffic gives intervals that are too narrow, and a claim the data can't support.
+- **An unchecked LLM judge.** A lenient judge passes behaviour humans would reject, and that's how regressions
+  ship.
+
+## What it does differently
+
+Other good tools exist (see [Related tools](#related-tools)). agentstats is built around agent runs:
+
+1. **Reads trajectories, not score tables.** Harbor job directories and ATIF trajectories directly: pass/fail,
+   infrastructure errors, tokens, cost, latency, steps and tool calls, plus deterministic rule checks on the
+   tool calls (`called`, `not_called`, `before`, `args`).
+2. **Picks the statistics from how the eval set was built, and says what claim they support.**
+
+   | `--design` | Your eval set | Interval / comparison | Claim |
+   |---|---|---|---|
+   | `curated` (default) | hand-picked cases, most eval sets | fixed cases: noise from repeated trials of each case (falls back to Student's t / Welch's t over complete runs when runs vary more than that predicts) | "on this eval set" |
+   | `sampled` | drawn at random from real traffic | per-case t / paired per-case differences | "on the population the cases came from" |
+   | `clustered` | sampled, in groups that fail together | cluster-robust standard errors | "on the population the groups came from" |
+
+   For a hand-picked set the cases are fixed, so the only randomness is the agent's own: the fixed-cases test
+   measures it from repeated trials of each case, and cases that always (or never) pass contribute nothing. It's
+   the paired test from a production diagnosis gate, plus a small-sample correction that brought simulated false
+   alarms from 6.8% to 5.8% at a nominal 5% (the rest is the discreteness of pass/fail; a simulated threshold
+   would close it). It checks for run-wide shocks (whole runs varying more than per-case randomness predicts, as
+   when a provider has a bad hour) and then falls back to treating each complete run as one observation. With one
+   run it uses a Wilson interval or an exact McNemar test. Overlapping groups can't be modelled credibly, so `clustered` refuses them and explains why.
+3. **Reliability as its own result.** pass@k (capability) and pass^k (consistency) with CIs, and lists of
+   always-pass, never-pass and flaky cases.
+4. **Gates that can say "I can't tell".** Two regression rules: `no_significant_drop` fails only on a proven drop
+   (for pull requests); `no_regression` passes only when the change is shown to be no worse than a margin
+   (non-inferiority, for releases), so an under-powered run is INCONCLUSIVE, with an estimate of the runs that would
+   settle it, never PASS. A run-wide shock with too few runs to correct for it is also INCONCLUSIVE ("rerun").
+   Infrastructure failures (Harbor exceptions or an `infra_error` field) are kept out of pass rates and make a run
+   UNHEALTHY (exit 3), as eval-sprint also does.
+5. **Judge checks:** human agreement (Fleiss' κ), judge vs humans (accuracy, strictness, leniency, Cohen's κ,
+   each with a CI, on a held-out split if you mark one) and judge consistency over repeated runs (flip rate, mean
+   pairwise disagreement).
+6. **Model switches decided by a rule written down first.** `agentstats switch` answers SWITCH / KEEP / NOT YET for
+   moving to a cheaper model: non-inferiority at your margin plus a cost-ratio limit, the model as-is and with an
+   adapted harness, per-slice results, and a warning when the eval set came from the current model.
+   `agentstats preregister` fixes the rule before any result exists, and the decision only counts as pre-registered
+   if the registration was committed before the results and its inputs are unchanged. See
+   [Deciding a model switch](#deciding-a-model-switch-agentstats-preregister-and-agentstats-switch).
+
+## Example: did giving the fixing agent a diagnosis help?
+
+mini-swe-agent on 100 SWE-bench Verified tasks, once plain and once with a diagnosis from a separate diagnosis
+agent added to the task as a hint. One Harbor job each.
+
+```
+$ agentstats compare --a jobs-plain100/... --b jobs-hint100/... --a-name plain --b-name with-diagnosis
+
+NO DETECTABLE DIFFERENCE: with-diagnosis minus plain = +1.0 pts (95% CI -6.2 pts to +8.2 pts),
+on this eval set, from a single run per arm (indicative: run-to-run noise is unmeasured).
+Method: exact McNemar test + paired differences. 100 matched cases; plain 48.0%, with-diagnosis 49.0%.
+
+- McNemar (exact): 7 cases pass only with with-diagnosis, 6 only with plain; p = 1.
+- Smallest difference this setup can reliably detect (80% power): about +10.2 pts.
+- Cost per attempt: plain $0.0858, with-diagnosis $0.0765; difference $-0.0093 (95% CI -0.0182 to -0.0005).
+```
+
+The +1 point is noise: 13 tasks moved, in both directions, and this setup can't see anything smaller than about
+10 points. The cost difference is real (its CI excludes zero): the hint made the agent about 11% cheaper.
+
+The same plain setup, run twice:
+
+```
+$ agentstats summary jobs-plain100/... jobs-all500/...
+
+Pass rate: 44.0% (95% CI 41.2% to 46.8%), on this eval set.
+Method: fixed cases, noise from repeated trials per case. 100 cases, 2 run(s).
+- Not used: Student's t over complete runs would give 44.0% (95% CI 0.0% to 94.8%) (for hand-picked sets
+  with 3+ runs that show run-wide shocks).
+- Note: Whole runs vary 8.0x more than per-case randomness predicts (p = 0.005): a run-wide shock. With fewer
+  than 3 runs there's nothing safer to fall back to; treat this with care.
+
+| k | pass@k (capability)           | pass^k (consistency)          |
+| 1 | 44.0% (95% CI 34.5% to 53.5%) | 44.0% (95% CI 34.5% to 53.5%) |
+| 2 | 48.0% (95% CI 38.0% to 58.0%) | 40.0% (95% CI 30.0% to 50.0%) |
+Always pass: 40, never pass: 52, flaky: 8
+```
+
+48% and 40% is more than per-case randomness explains: something changed across the whole run between the two
+(they ran at different times, in jobs of different sizes). A narrow interval from these two runs would be false
+confidence, and the report says so. A third run lets it fall back to run-level statistics.
+
+## In production: an incident-triage agent's CI gate
+
+agentstats runs the regression gate for a production incident-triage agent (classifies alerts as real / noise /
+duplicate with a P0-P3 severity): 114 held-out cases, 2 trials per pull request, compared case by case with a 4-trial
+baseline using the fixed-cases test, `no_significant_drop` at a 90% interval.
+
+**Sized by simulation from the baseline's own per-case rates** (each case's true rate drawn from its baseline trials,
+then candidate and baseline simulated from it; `agentstats size` does the same for your baseline), against the gate it
+replaced (fail if more than 6% of 80 filtered cases miss, one run):
+
+| True drop | agentstats gate catches it | Old threshold gate |
+|---|---|---|
+| none (false alarm) | 5% | ~0% |
+| ~5 points | 64% | 48% |
+| ~8 points | 94% | 82% |
+| ~12 points | 100% | 97% |
+
+The old gate also had to drop 32 noisy cases to get under its threshold; the paired test keeps them, since each case is
+compared with its own baseline rate.
+
+**A skipped tool the labels rewarded.** The triage agent (then on Claude Haiku 4.5) skipped a required lookup tool
+and wrote an invented count instead. The eval labels had been built from its own answers, so the accuracy gate couldn't
+see it. With behaviour rules, Haiku 4.5 as the candidate against Haiku 5.5: accuracy +0.9 pts (CI -2.1 to +3.9), PASS;
+`min_rule_rate` FAIL (lookup called in 25.4% of attempts, CI 21.0% to 30.5%); `no_rule_regression` FAIL (-74.6 pts vs
+100%). On 61 real-incident traces exported from Langfuse and read with `convert-traces` (rules-only, no labels): the
+lookup was called in 52.5% (CI 40.2% to 64.5%). The rules now run in that agent's CI gate.
+
+**What it caught in its first week:**
+- **An environment offset.** Six CI trials of unchanged code averaged 89.9% against a baseline measured on a laptop
+  (93.2%): -3.3 points, 95% CI -6.1 to -0.5, spread over many cases, no run-wide shock. One pull request had already
+  failed for it. The baseline is now measured in CI; the first gate run against it: -0.7 points (CI -3.4 to +2.1), PASS.
+- **Outages, as UNHEALTHY instead of FAIL.** Once when the API credit ran out mid-run (69 of 228 attempts failed:
+  exit 3, the cause listed, the remaining attempts compared separately), once when a new major SDK release rejected a
+  request parameter on every call.
+- **A doubtful "same setup".** Two runs of the same coding agent on the same 100 tasks scored 48% and 40%: the shock
+  check flagged it (whole runs varying 8x more than per-case randomness predicts, p = 0.005), so the narrow interval
+  wasn't trusted (see the example above).
+
+
+
+## Deciding a model switch: `agentstats preregister` and `agentstats switch`
+
+A cheaper model comes out. "No significant difference" on one eval run isn't "no worse": a small run usually just has
+a wide interval. `agentstats switch` asks the question the other way round, as a non-inferiority test: is the cheaper
+model **shown** to lose no more than your margin, at no higher cost? It answers **SWITCH**, **KEEP** or **NOT YET**
+and writes a decision record (`decision.json` + `decision.md`) with the rule, the numbers and hashes of the inputs.
+
+```bash
+agentstats preregister --name my-switch --margin 0.03 --provenance independent \
+    --input labels=evals/labels.jsonl --slice-by template --out decisions/my-switch
+git add decisions/my-switch && git commit -m "pre-register my-switch"     # before running anything
+# ... run both models on the same cases, a few trials each, same session ...
+agentstats switch --prereg decisions/my-switch/prereg.json \
+    --current current.jsonl --candidate cheap.jsonl [--adapted cheap-adapted.jsonl] --out decisions/my-switch
+```
+
+| Interval for (candidate - current) | Decision |
+|---|---|
+| lower bound above -margin, and the cost ratio's upper bound at most `--max-cost-ratio` (default 1.0) | SWITCH |
+| upper bound below -margin (worse even at the optimistic end), or more expensive | KEEP |
+| anything else | NOT YET |
+
+- **Pre-registration is checked, not trusted.** With `--prereg` the rule comes from the file (contradicting flags are
+  an error), and the decision only counts as pre-registered if the registration is committed and unchanged, was
+  committed before every result file was last modified, and every declared `--input` still has its registered hash.
+- **Two steps:** the cheaper model as-is, then with an adapted harness (`--adapted`, e.g. rows from
+  `harness-evolve export`); the better decision wins.
+- **Slices** (`--slice-by`) are reported per slice; `--strict-slices` makes the decision no better than the worst one.
+- **Provenance:** `--provenance current-model` says the eval set was made or filtered with the current model, and the
+  record warns that the comparison favours it.
+- **Cost per accepted result** is reported for each arm. Exit codes: 0 SWITCH, 1 KEEP, 2 NOT YET, 3 UNHEALTHY.
+
+**In production:** my triage agent moved from Claude Haiku 4.5 to Haiku 5.5 on a pre-registered decision: +2.2 pts
+(95% CI -0.7 to +5.1) at 0.09x the cost per accepted result. An earlier comparison had said 10 points worse; the gap
+came from labels that repeated counts the old model made up. Rule, rows and record:
+[examples/switch-triage-haiku-5-5](https://github.com/sidreddy88/agentstats/tree/main/examples/switch-triage-haiku-5-5).
+
+These two commands started as a separate package, stepdown. They're a thin layer on agentstats' own tests, so they
+live here for now. If the model-switch side grows into its own system (suggesting candidate models, adapting the
+harness with harness-evolve, deciding as runs accumulate, shadow runs on live traffic), it may become a separate
+package again.
+
+## Evals in CI, in tiers
+
+`agentstats init-ci` writes a policy (`agentstats.toml`) and three GitHub workflows. You replace one step in
+each, running your agent, and agentstats decides:
+
+| Tier | When | What runs | Typical requirements |
+|---|---|---|---|
+| smoke | every pull request, minutes | a stratified sample (`agentstats sample`) + all rule checks | `rules`, `min_pass_rate` (a crash threshold) |
+| nightly | every night | the whole eval set, k runs per case | `no_regression` vs the baseline, `max_infra` |
+| release | before shipping | the whole set against the production baseline | `no_regression`, `max_cost_ratio`, `max_p95_latency_ratio`, `min_pass_hat_k` |
+
+**Behaviour rules as a regression gate (no labels needed).** `min_rule_rate` requires every trajectory rule (the
+`checks` in each row, e.g. `{"called_lookup_tool": true}`) to hold at least that often, judged on its interval: FAIL
+only when the whole interval is below, INCONCLUSIVE when it can't tell. `no_rule_regression` compares each rule with
+the baseline, case by case (fixed-cases test, Bonferroni across rules), and fails when a rule is proven to hold less
+often. These catch what an accuracy gate can't when the labels are wrong: a triage agent that skipped a required
+lookup tool half the time and invented the number passed every accuracy check, because the labels had been built
+from its own answers. The gate prints a per-rule table either way.
+
+Two more requirements, both from real misses: `max_no_answer` caps the share of attempts with no usable answer, and
+`max_slice_drop` (with `slice_by = "<tag prefix>"` in the tier) fails when any slice, e.g. one error template, is
+proven worse than the baseline by more than the limit. Slice intervals are Bonferroni-adjusted across slices, slices
+whose estimate is past the limit but not proven are named as leads, and with `slice_by` set the per-slice table is
+printed even without the requirement.
+
+`agentstats gate --policy agentstats.toml --tier nightly --candidate results/ --baseline baseline/` exits
+**0 pass, 1 fail, 2 inconclusive, 3 unhealthy** (too many infrastructure failures to judge the agent), and writes
+its table to the GitHub job summary.
+
+**Size a gate before trusting it:** `agentstats size --baseline baseline.jsonl --trials 2 --rule drop` simulates the
+rule on your baseline's own per-case rates and prints how often it would pass or fail for a range of true drops, so you
+can pick trials and margins (e.g. no change passes >= 90%, an 8-point drop fails >= 90%).
+
+Comparisons with the baseline are **non-inferiority tests**. "The interval includes zero" doesn't show that
+nothing regressed, so `no_regression = 0.02` passes only when the lower confidence bound of (candidate − baseline)
+is at least −2 points. An under-powered run is INCONCLUSIVE, with an estimate of how many runs would settle it,
+never PASS. Small smoke tiers say what they can't see: "with 20 cases and 1 run, this tier can only see drops
+larger than about 40 points: it catches breakage, not regressions."
+
+On the real runs above (with-diagnosis as the candidate, plain as the baseline):
+
+```
+## agentstats gate: nightly ⚠️ INCONCLUSIVE
+
+| requirement | measured | threshold | result | detail |
+| runs | 1 | >= 3 | ⚠️ INCONCLUSIVE | the nightly tier expects 3 complete runs |
+| max_infra | 0.0% | <= 5.0% | ✅ PASS | |
+| no_regression | +1.0 pts (CI -6.2 pts to +8.2 pts, mcnemar) | lower bound >= -2.0 pts | ⚠️ INCONCLUSIVE | about 6 runs per arm would settle it if the estimate holds |
+| max_cost_ratio | 0.89x (CI 0.80x to 0.99x) | <= 1.10x | ✅ PASS | |
+```
+
+## Input
+
+One row per attempt (`agentstats convert` writes this from Harbor jobs):
+
+```json
+{"arm": "baseline", "run": 1, "case": "django-11099", "passed": true, "infra_error": null,
+ "cost_usd": 0.12, "input_tokens": 41000, "output_tokens": 2100, "steps": 14, "tool_calls": 9,
+ "latency_s": 33.0, "tags": ["feature:search"], "checks": {"no_refund_before_lookup": true}}
+```
+
+A *run* is one full pass over the eval set. Several paths for one arm are several runs. Runs are compared on the
+cases attempted in every run.
+
+`no_answer` (optional, a reason such as `"step_limit"` or `"parse_error"`) marks an attempt where the agent produced
+no usable answer at all. It counts as a failure, like a wrong answer, but is reported apart: a jump in no-answer
+attempts usually means the harness or the agent loop broke, not that the agent's judgement got worse. Rows are
+validated on load (the error names the file, line and field), and an optional `schema_version` (currently 1) lets a
+newer format be refused instead of half-read. `harness-evolve export` and agent-platform's triage gate both write this
+format; `tests/test_hardening.py` loads samples from each.
+
+**From traces** (`agentstats convert-traces`): an OpenTelemetry OTLP JSON export (tool calls are spans with
+`gen_ai.operation.name = "execute_tool"`, name in `gen_ai.tool.name`, arguments in `gen_ai.tool.call.arguments`;
+tokens from `gen_ai.usage.*`) or a Langfuse export (traces with their observations; TOOL observations are tool
+calls). One trace is one attempt; `--rules` fills its checks. Pass/fail comes from `--verdicts` (JSONL with
+`trace_id`, `passed`, optional `case`, `run`, `infra_error`); without it the rows are **rules-only**: no pass rate, but
+the rule gates work, which is what production traces without labels need.
+
+Judge labels (`agentstats judge`), one row per verdict:
+
+```json
+{"item": "conv-07", "rater": "human:alice", "verdict": "pass", "split": "heldout"}
+{"item": "conv-07", "rater": "judge", "verdict": "fail", "run": 2}
+```
+
+Trajectory rules (`agentstats check JOB --rules rules.json --out results.jsonl`): see `examples/rules.json`.
+A synthetic judge-labels file is in `examples/judge_labels.jsonl`.
+
+## Methods and sources
+
+- Error bars for evals, clustered standard errors: Miller 2024, [arXiv 2411.00640](https://arxiv.org/abs/2411.00640).
+- Choosing the method by how the eval set was built, evals in CI tiers, judge alignment and consistency metrics:
+  Yesha Shah's "Agentic Evals" series (see [Credits](#credits)).
+- pass@k: Chen et al. 2021, [arXiv 2107.03374](https://arxiv.org/abs/2107.03374). pass^k: τ-bench, Yao et al.,
+  [arXiv 2406.12045](https://arxiv.org/abs/2406.12045). Randomness in agentic evals: Bjarnason et al. 2026,
+  [arXiv 2602.07150](https://arxiv.org/abs/2602.07150).
+- Judge self-inconsistency: Haldar & Hockenmaier 2025 (EMNLP Findings); Yagubyan 2026,
+  [arXiv 2606.13685](https://arxiv.org/abs/2606.13685).
+- Pre-registration (fix the analysis before seeing the data): Nosek et al. 2018, "The preregistration revolution",
+  [PNAS](https://doi.org/10.1073/pnas.1708274114). The switch rule is a standard non-inferiority test (lower interval
+  bound above -margin) on the same fixed-cases interval the gates use.
+- Cost per accepted result rather than cost per call: from Grep AI's "The Managed Agent Wars".
+
+Every statistic is tested against scipy or statsmodels (`tests/test_stats.py`, `tests/test_judge.py`). The only
+runtime dependency is scipy.
+
+## Credits
+
+The framing behind several features comes from Yesha Shah's **Agentic Evals** series:
+[Part 1, "A Software Engineer's Guide"](https://yeshas93.substack.com/p/agentic-evals-a-software-engineers)
+(2026-09-21); [Part 2, "Statistical Tools for Measuring Agent Performance"](https://yeshas93.substack.com/p/agentic-evals-statistical-tools-for)
+(2026-09-30); [Part 3, "Introduction to Autoraters"](https://substack.com/home/post/p-220996535) (2026-10-19):
+
+- choosing the statistics by how the eval set was built (benchmark vs generalised performance; curated, sampled and
+  correlated eval sets);
+- pass@k and pass^k for agents, and reporting intervals rather than single runs;
+- evals in CI in tiers (smoke, nightly, release), a stratified smoke sample, and rule-based checks on trajectories;
+- the judge checks: strictness, leniency, Cohen's and Fleiss' κ, a held-out split, verdict flip rate and mean pairwise
+  disagreement.
+
+The statistical methods themselves come from the papers listed under [Methods and sources](#methods-and-sources),
+which the series also cites.
+
+Original to agentstats:
+
+- the fixed-cases test with its small-sample correction (from a production diagnosis regression gate), and the
+  run-wide shock check;
+- the gate rules: `no_significant_drop` and non-inferiority, INCONCLUSIVE with a runs estimate, a rerun on a shock;
+- gate sizing by simulation on your own baseline (`agentstats size`);
+- **behaviour rules as a regression gate**: `no_rule_regression` compares each trajectory rule with the baseline, case
+  by case with intervals, and `min_rule_rate` sets an interval-based floor. They need no labels, which is what caught a
+  production bug an accuracy gate passed (see the production section);
+- the **no-answer outcome** (a broken loop or unparseable output reported apart from wrong answers) and **per-slice
+  gates** with Bonferroni-adjusted intervals and named leads;
+- **rules-only rows from traces**: OpenTelemetry (GenAI conventions) and Langfuse exports, so behaviour can be gated on
+  production traces that have no labels;
+- **model-switch decisions** (`agentstats switch`): non-inferiority plus cost per accepted result, as-is and adapted
+  steps, an eval-set provenance warning, and **checked pre-registration** (committed before the results, inputs
+  hashed), with a decision record;
+- **calibration tests** that fail if the statistics drift (e.g. the drop gate's false-alarm rate);
+- reading Harbor jobs and ATIF trajectories, and the production results above.
+
+UNHEALTHY on infrastructure failures and keeping them out of pass rates are also in eval-sprint.
+
+## Related tools
+
+- [evalstats](https://github.com/ianarawjo/evalstats) ([arXiv 2609.35815](https://arxiv.org/abs/2609.35815)):
+  judge-corrected tests (PPI) and methods tuned for small samples. Use it when your verdicts come from an LLM
+  judge with a small human-labelled set.
+- [evalci](https://github.com/Shreyaskc/evalci) ([arXiv 2607.04429](https://arxiv.org/abs/2607.04429)): CIs,
+  paired tests, power analysis and multiple-comparison correction for per-item tables (lm-eval, HELM).
+- [eval-sprint](https://github.com/renanpgonzalez/eval-sprint): an eval kit with deterministic and LLM judges, a
+  labelling UI, and a CI regression gate (exact McNemar on paired flips, errors kept out of pass rates, an
+  unhealthy-run exit code). Its judge calibration (blind expert labels, Horvitz-Thompson weights, Rogan-Gladen
+  correction) is more thorough than agentstats' judge check.
+
+## Limitations
+
+- Pass rates need pass/fail verdicts; numeric scores are used for efficiency metrics and judge consistency.
+- Power analysis is by simulation (`agentstats size`) for the two gate rules; no closed-form sample-size planner.
+- The judge check uses the judge's most common verdict per item; it does not correct downstream statistics for
+  judge error (evalstats does).
+- `agentstats switch` decides on fixed batches: run both models, then decide. Deciding as runs accumulate
+  (anytime-valid intervals) isn't built. The margin is your judgement; agentstats makes you state it and checks you
+  stated it first. Cost is whatever your rows record (my runs: token counts at list prices, no cache discounts).
+
+MIT licence.