whileai 0.56__tar.gz → 0.57__tar.gz

{whileai-0.56 → whileai-0.57}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: whileai
-Version: 0.56
+Version: 0.57
 Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
 Author: While
 License: Apache-2.0
@@ -8,7 +8,7 @@ Project-URL: Homepage, https://github.com/whilehq/whileai-sdk
 Project-URL: Repository, https://github.com/whilehq/whileai-sdk
 Project-URL: Issues, https://github.com/whilehq/whileai-sdk/issues
 Project-URL: Datasets, https://huggingface.co/datasets/zero-proof-ai/agent-simulations
-Keywords: rl,reinforcement-learning,grpo,synthetic-data,agents,tool-use,evaluation,simulation,llm
+Keywords: zp,zeroproof,while,evals,agent-evals,rl,reinforcement-learning,grpo,synthetic-data,agents,tool-use,evaluation,simulation,llm
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
@@ -41,7 +41,7 @@ The While Python SDK. One package, two importable modules:
 - `whileai`: the platform client. OTLP trace ingest and trace-dataset listing against the token gate.
 - `whileai.simulations`: post-training data for an agent. Give it the agent's traces, or its tools and system prompt; it simulates the situations, the people, and the world, plays the agent through multi-turn tool-calling conversations, and returns rows for your grader.
 
-**Renamed.** This SDK was `zeroproof` (ZeroProof is now While). `pip install zeroproof` still works: it installs `whileai`, and `import zeroproof` (or the older `zeroproof_simulations`) resolves to the same modules with a deprecation warning. `ZEROPROOF_*` environment variables and a saved `~/.zeroproof/credentials.json` are still read. Change the import when you can; new releases land under `whileai`.
+**Renamed.** This SDK was `zeroproof` (ZeroProof is now While). `pip install zeroproof` still works: it installs `whileai`, and `import zeroproof` (or the older `zeroproof_simulations`) resolves to the same modules with a deprecation warning. `ZEROPROOF_*` environment variables and a saved `~/.zeroproof/credentials.json` are still read. Change the import when you can; new releases land under `whileai`. zp, ZeroProof and While all name this one product: the package is `whileai`, the import is `whileai.simulations`, keys start with `zp_`. A machine with the old package still picks up `~/.zeroproof/credentials.json`; set `WHILEAI_HOME` to a fresh directory to isolate a new account from it.
 
 Releases of `whileai` before 0.3 were an unrelated encrypted agent-to-agent messaging client. That code was removed in 0.04; pin `whileai<0.3` if you still depend on it.
 
@@ -187,6 +187,34 @@ and runs in seconds, but the situations are less varied than a model writes,
 so it is for wiring up your agent and grader, not for a training set — for
 that, bring a model below.
 
+### Evals for the agent you already have
+
+Not training anything yet? The shortest path is an eval: wrap your agent
+as `agent(message) -> {steps, final_text}`, write the policy as a judge
+that reads the trajectory, run the asks `k` times each, and read pass@1
+with its interval. Offline first, then the hosted writer. The how-to is
+[docs/evals.md](docs/evals.md); the runnable version is
+[`recipes/02-measure/eval-your-agent`](recipes/02-measure/eval-your-agent),
+which ends at a CI gate, not a push.
+
+```python
+data = wai.simulate(
+    agent,
+    tools=TOOLS,
+    system_prompt=POLICY,
+    seeds=SEEDS,
+    simulator=False,
+    mode="rl",
+    repeats=4,
+    repeat_policy="fixed",
+)
+scored = wai.evaluate(data, judge)  # eval lineage: never the reward
+print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
+```
+
+`scored.warnings` is new: no rollout called a tool, a declared tool no
+rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -421,8 +449,8 @@ When the rows carry none of that metadata the export warns: the reward
 reduces to `conduct_grade`, a process reward, and a policy trained on it
 alone learns to call nothing (`recipes/03-select/prime-intellect-rl`). `wai.load_environment(spec)`
 builds the environment in a process that has `verifiers` (`pip install
-'whileai[rl]'`); `examples/coding-efficiency` is the same shape built by
-hand over an executable world with a hidden test suite.
+'whileai[rl]'`); the [tool-call-efficiency](https://huggingface.co/datasets/zero-proof-ai/tool-call-efficiency)
+dataset is the same shape built by hand over an executable world with a hidden test suite.
 
 Training notes, each with the chapter of rlhfbook.com behind it. Calibrate
 difficulty with 8 to 16 rollouts per task before exporting so the band is
@@ -626,6 +654,7 @@ network.
 | Simulate and grade | [`recipes/01-simulate/agent-behavior`](recipes/01-simulate/agent-behavior) | Start here if the platform is new to you. Runs a coding agent with bad habits against real tests, streams every turn to While as OTLP spans plus a judge verdict, and fills a dashboard with behaviour worth looking at. Needs a key and a model endpoint; stdlib only. |
 | Simulate and grade | [`recipes/01-simulate/verifiers`](recipes/01-simulate/verifiers) | Verifiable rewards: math (`MathEqual`), an answer-and-format gate (`All`), code run against hidden tests (`CodeExec`), and a JSON-schema check, each feeding `grade`/`optimize`. Offline. |
 | Measure | [`recipes/02-measure/pass-at-k`](recipes/02-measure/pass-at-k) | pass@1, pass^k and pass@k with their intervals for one agent, the per-ask histogram the mean hides, and what each number tells you to do next. Offline. |
+| Measure | [`recipes/02-measure/eval-your-agent`](recipes/02-measure/eval-your-agent) | Evals for the agent you already have: the callable wrapper, the policy as a judge that reads the trajectory, pass@1 with an interval and pass^k per policy branch, the coverage warnings that catch a hollow run, and a CI gate. Two scripted refund bots, one careful and one eager, so the eval visibly separates them. Offline, seconds. How-to: [docs/evals.md](docs/evals.md). |
 | Measure | [`recipes/02-measure/reward-hacking`](recipes/02-measure/reward-hacking) | Reward hacking caught before, during and after training: the within-ask scan, the judge probes, the trajectory flags, and the proxy-vs-target verdict on a scripted agent and two judges. Offline, seconds, no key. How-to: [docs/reward-hacking.md](docs/reward-hacking.md). |
 | Measure | [`recipes/02-measure/safety-evals`](recipes/02-measure/safety-evals) | Safety evals for a tool-using agent: prompt injection (direct, and planted in a tool result), data exfiltration, secret leakage, unauthorized writes, plus the benign controls that catch over-refusal. Trajectory markers as the judge, pass^k per attack class, the judge checked against hand labels, and a before/after that fails the fix which got safe by refusing. Offline, seconds. How-to: [docs/safety-evals.md](docs/safety-evals.md). |
 | Measure | [`recipes/02-measure/safety-evals-marketplace`](recipes/02-measure/safety-evals-marketplace) | The same safety eval for a marketplace agent: the injection is planted in user-generated reviews, the private data is per tenant (a competitor's buyer-intent list), one of the writes is a public post, and a flag needs a moderation ticket. Six trajectory markers, pass^k per attack class, the guarded before/after, and `live.py` to run the suite on a real model through Ollama with no key. Offline, seconds. |

whileai-0.56/whileai.egg-info/PKG-INFO → whileai-0.57/README.md

@@ -1,39 +1,3 @@
-Metadata-Version: 2.4
-Name: whileai
-Version: 0.56
-Summary: While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt.
-Author: While
-License: Apache-2.0
-Project-URL: Homepage, https://github.com/whilehq/whileai-sdk
-Project-URL: Repository, https://github.com/whilehq/whileai-sdk
-Project-URL: Issues, https://github.com/whilehq/whileai-sdk/issues
-Project-URL: Datasets, https://huggingface.co/datasets/zero-proof-ai/agent-simulations
-Keywords: rl,reinforcement-learning,grpo,synthetic-data,agents,tool-use,evaluation,simulation,llm
-Classifier: Development Status :: 3 - Alpha
-Classifier: Intended Audience :: Developers
-Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Classifier: Typing :: Typed
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: requests>=2.25.0
-Provides-Extra: dev
-Requires-Dist: pytest>=8.0; extra == "dev"
-Requires-Dist: pytest-cov>=5.0; extra == "dev"
-Requires-Dist: ruff>=0.12; extra == "dev"
-Requires-Dist: mypy>=1.16; extra == "dev"
-Requires-Dist: types-requests; extra == "dev"
-Provides-Extra: rl
-Requires-Dist: verifiers>=0.3; (python_version >= "3.11" and python_version < "3.14") and extra == "rl"
-Dynamic: license-file
-
 # whileai
 
 The While Python SDK. One package, two importable modules:
@@ -41,7 +5,7 @@ The While Python SDK. One package, two importable modules:
 - `whileai`: the platform client. OTLP trace ingest and trace-dataset listing against the token gate.
 - `whileai.simulations`: post-training data for an agent. Give it the agent's traces, or its tools and system prompt; it simulates the situations, the people, and the world, plays the agent through multi-turn tool-calling conversations, and returns rows for your grader.
 
-**Renamed.** This SDK was `zeroproof` (ZeroProof is now While). `pip install zeroproof` still works: it installs `whileai`, and `import zeroproof` (or the older `zeroproof_simulations`) resolves to the same modules with a deprecation warning. `ZEROPROOF_*` environment variables and a saved `~/.zeroproof/credentials.json` are still read. Change the import when you can; new releases land under `whileai`.
+**Renamed.** This SDK was `zeroproof` (ZeroProof is now While). `pip install zeroproof` still works: it installs `whileai`, and `import zeroproof` (or the older `zeroproof_simulations`) resolves to the same modules with a deprecation warning. `ZEROPROOF_*` environment variables and a saved `~/.zeroproof/credentials.json` are still read. Change the import when you can; new releases land under `whileai`. zp, ZeroProof and While all name this one product: the package is `whileai`, the import is `whileai.simulations`, keys start with `zp_`. A machine with the old package still picks up `~/.zeroproof/credentials.json`; set `WHILEAI_HOME` to a fresh directory to isolate a new account from it.
 
 Releases of `whileai` before 0.3 were an unrelated encrypted agent-to-agent messaging client. That code was removed in 0.04; pin `whileai<0.3` if you still depend on it.
 
@@ -187,6 +151,34 @@ and runs in seconds, but the situations are less varied than a model writes,
 so it is for wiring up your agent and grader, not for a training set — for
 that, bring a model below.
 
+### Evals for the agent you already have
+
+Not training anything yet? The shortest path is an eval: wrap your agent
+as `agent(message) -> {steps, final_text}`, write the policy as a judge
+that reads the trajectory, run the asks `k` times each, and read pass@1
+with its interval. Offline first, then the hosted writer. The how-to is
+[docs/evals.md](docs/evals.md); the runnable version is
+[`recipes/02-measure/eval-your-agent`](recipes/02-measure/eval-your-agent),
+which ends at a CI gate, not a push.
+
+```python
+data = wai.simulate(
+    agent,
+    tools=TOOLS,
+    system_prompt=POLICY,
+    seeds=SEEDS,
+    simulator=False,
+    mode="rl",
+    repeats=4,
+    repeat_policy="fixed",
+)
+scored = wai.evaluate(data, judge)  # eval lineage: never the reward
+print(wai.pass_at(scored.rows), *scored.warnings)  # a hollow run says so here
+```
+
+`scored.warnings` is new: no rollout called a tool, a declared tool no
+rollout touched, a marker that fired on no row. A 1.00 on a run like that is not a result; the note names the fix.
+
 ### Bring a model
 
 Bring your own model. Any OpenAI-compatible chat endpoint that returns tool
@@ -421,8 +413,8 @@ When the rows carry none of that metadata the export warns: the reward
 reduces to `conduct_grade`, a process reward, and a policy trained on it
 alone learns to call nothing (`recipes/03-select/prime-intellect-rl`). `wai.load_environment(spec)`
 builds the environment in a process that has `verifiers` (`pip install
-'whileai[rl]'`); `examples/coding-efficiency` is the same shape built by
-hand over an executable world with a hidden test suite.
+'whileai[rl]'`); the [tool-call-efficiency](https://huggingface.co/datasets/zero-proof-ai/tool-call-efficiency)
+dataset is the same shape built by hand over an executable world with a hidden test suite.
 
 Training notes, each with the chapter of rlhfbook.com behind it. Calibrate
 difficulty with 8 to 16 rollouts per task before exporting so the band is
@@ -626,6 +618,7 @@ network.
 | Simulate and grade | [`recipes/01-simulate/agent-behavior`](recipes/01-simulate/agent-behavior) | Start here if the platform is new to you. Runs a coding agent with bad habits against real tests, streams every turn to While as OTLP spans plus a judge verdict, and fills a dashboard with behaviour worth looking at. Needs a key and a model endpoint; stdlib only. |
 | Simulate and grade | [`recipes/01-simulate/verifiers`](recipes/01-simulate/verifiers) | Verifiable rewards: math (`MathEqual`), an answer-and-format gate (`All`), code run against hidden tests (`CodeExec`), and a JSON-schema check, each feeding `grade`/`optimize`. Offline. |
 | Measure | [`recipes/02-measure/pass-at-k`](recipes/02-measure/pass-at-k) | pass@1, pass^k and pass@k with their intervals for one agent, the per-ask histogram the mean hides, and what each number tells you to do next. Offline. |
+| Measure | [`recipes/02-measure/eval-your-agent`](recipes/02-measure/eval-your-agent) | Evals for the agent you already have: the callable wrapper, the policy as a judge that reads the trajectory, pass@1 with an interval and pass^k per policy branch, the coverage warnings that catch a hollow run, and a CI gate. Two scripted refund bots, one careful and one eager, so the eval visibly separates them. Offline, seconds. How-to: [docs/evals.md](docs/evals.md). |
 | Measure | [`recipes/02-measure/reward-hacking`](recipes/02-measure/reward-hacking) | Reward hacking caught before, during and after training: the within-ask scan, the judge probes, the trajectory flags, and the proxy-vs-target verdict on a scripted agent and two judges. Offline, seconds, no key. How-to: [docs/reward-hacking.md](docs/reward-hacking.md). |
 | Measure | [`recipes/02-measure/safety-evals`](recipes/02-measure/safety-evals) | Safety evals for a tool-using agent: prompt injection (direct, and planted in a tool result), data exfiltration, secret leakage, unauthorized writes, plus the benign controls that catch over-refusal. Trajectory markers as the judge, pass^k per attack class, the judge checked against hand labels, and a before/after that fails the fix which got safe by refusing. Offline, seconds. How-to: [docs/safety-evals.md](docs/safety-evals.md). |
 | Measure | [`recipes/02-measure/safety-evals-marketplace`](recipes/02-measure/safety-evals-marketplace) | The same safety eval for a marketplace agent: the injection is planted in user-generated reviews, the private data is per tenant (a competitor's buyer-intent list), one of the writes is a public post, and a flag needs a moderation ticket. Six trajectory markers, pass^k per attack class, the guarded before/after, and `live.py` to run the suite on a real model through Ollama with no key. Offline, seconds. |

{whileai-0.56 → whileai-0.57}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "whileai"
-version = "0.56"
+version = "0.57"
 description = "While Python SDK: trace ingestion and verification client, plus agent simulations grounded in your tools and system prompt."
 readme = "README.md"
 license = {text = "Apache-2.0"}
@@ -12,6 +12,7 @@ requires-python = ">=3.10"
 dependencies = ["requests>=2.25.0"]
 authors = [{name = "While"}]
 keywords = [
+    "zp", "zeroproof", "while", "evals", "agent-evals",
     "rl", "reinforcement-learning", "grpo", "synthetic-data", "agents",
     "tool-use", "evaluation", "simulation", "llm",
 ]

{whileai-0.56 → whileai-0.57}/whileai/simulations/__init__.py

@@ -102,6 +102,7 @@ from .schema import (
     validate,
 )
 from .score.agreement import judge_agreement
+from .score.audit import audit_grades, format_audit
 from .score.checklist import expected_outcome, outcome_check, privileged_context, task_checklist
 from .score.curriculum import curriculum, format_curriculum, retire_solved
 from .score.delta import delta_report, format_delta_report
@@ -115,6 +116,7 @@ from .score.grounding import (
 from .score.hack_scan import format_hack_scan, format_hack_scan_diff, hack_scan, hack_scan_diff
 from .score.hygiene import (
     HACK_THRESHOLD,
+    coverage_warnings,
     dedupe_groups,
     length_report,
     near_duplicate_prompts,
@@ -235,6 +237,7 @@ __all__ = [
     "attach_holdout",
     "attach_labels",
     "attach_rubric",
+    "audit_grades",
     "behavior_signature",
     "behavioral_markers",
     "build_dimensions",
@@ -273,6 +276,7 @@ __all__ = [
     "export_training",
     "filter_rl_rows",
     "flaw_rows",
+    "format_audit",
     "format_curriculum",
     "format_delta_report",
     "format_hack_monitor",

{whileai-0.56 → whileai-0.57}/whileai/simulations/data.py

@@ -460,6 +460,7 @@ class SimulationData:
                 source="grade",
                 concurrency=min(int(concurrency), 32),
                 version=version,
+                tools=sorted(str(t) for t in self.declared_tools),
                 scale=scale,
             )
             note = trust_after_grade(scored.rows, mode=trust)["note"]

{whileai-0.56 → whileai-0.57}/whileai/simulations/run/engine.py

@@ -2851,6 +2851,27 @@ class Run:
                 data.degraded.append("same_model")
             data.warnings.append(note)
             log.warning(note)
+        # A run whose rollouts never called a tool is hollow: the writer
+        # asked about things the world does not have, or the wrapper did
+        # not record steps. Grading it gives a number that means nothing.
+        rows = data.trajectories
+        if rows and data.declared_tools:
+            with_calls = sum(
+                1
+                for r in rows
+                if any(isinstance(s, dict) and s.get("tool") for s in (r.get("steps") or []))
+            )
+            if with_calls == 0:
+                note = (
+                    f"0 of {len(rows)} rollouts called a tool, so this run says nothing "
+                    "about tool use. Put the ids your world has (order numbers, account "
+                    "names) in the tool descriptions or in seeds=, and check the agent "
+                    "wrapper records its steps, before grading it."
+                )
+                if "no_tool_calls" not in data.degraded:
+                    data.degraded.append("no_tool_calls")
+                data.warnings.append(note)
+                log.warning(note)
         if c.out_path is not None and data.trajectories:
             data.save(str(c.out_path), meta=True)
         return data

whileai-0.57/whileai/simulations/score/audit.py

@@ -0,0 +1,296 @@
+"""Is the verifier failing answers that are right?
+
+A verifier is a rule: execution match, exact match, a regex on the final
+number. Rules fail correct answers for reasons that have nothing to do
+with correctness (rounding, ordering, a column named differently, an
+equivalent row set), and every such false negative halves the
+separation on its task: the policy is told that a right answer was
+wrong. On 570 SEC XBRL tasks, fixing the grader moved mean reward from
+0.20 to 0.48 with no training at all. The RLHF book treats verifiers as
+solved (ch. 14: "a scoring function that returns a positive reward when
+the answer is correct and 0 otherwise"); this module is our own
+measurement of how often that function is wrong.
+
+``audit_grades`` samples failed rows, asks a judge whether each reply is
+in fact correct given the reference, and reports the false-negative rate
+with a Wilson interval and the verifier reasons that dominate. The judge
+is a second opinion, not ground truth: the book's judge-prompt rules
+apply (ch. 7: length must not sway it, no position bias, temperature 0
+for stable ratings), and a rate near the threshold deserves a check of
+the judge itself on human labels (``judge_agreement``, ``judge_trust``).
+"""
+
+from __future__ import annotations
+
+import random
+import re
+from collections.abc import Callable, Sequence
+from typing import Any
+
+from .judging import run_judge
+from .stats import wilson_interval
+
+#: above this share of audited failures, the verifier is the problem to fix first
+FN_WARN = 0.10
+
+#: what the judge is asked when the row has no rubric of its own
+AUDIT_QUESTION = (
+    "The reference answer is the answer key for this task. Is the reply's final answer "
+    "correct, meaning it gives the same result as the reference? Differences in "
+    "formatting, rounding, ordering (unless the task asks for an order), column or "
+    "field naming, or an equivalent way of expressing the same result do not make it "
+    "wrong. A missing, different, or partial result does."
+)
+
+_DIGITS = re.compile(r"\d+(\.\d+)?")
+
+
+def _reason_key(reason: Any, verifier: str | None) -> str:
+    """The kind of failure a verifier reason names, without its numbers:
+    ``result differs: got 3 rows x 2 cols`` and ``result differs: got 10
+    rows x 1 col`` are one reason."""
+    text = str(reason or "").strip()
+    if verifier and text.lower().startswith(verifier.lower() + ":"):
+        text = text[len(verifier) + 1 :].strip()
+    head = re.split(r"[:(]", text, maxsplit=1)[0].strip()
+    head = _DIGITS.sub("#", head).lower()
+    return head or "(no reason)"
+
+
+def _binary(row: dict) -> int | None:
+    value = row.get("reward")
+    if value is None or isinstance(value, bool):
+        return None
+    try:
+        f = float(value)
+    except (TypeError, ValueError):
+        return None
+    if f in (0.0, 1.0):
+        return int(f)
+    return None
+
+
+def _audit_row(row: dict, question: str) -> dict:
+    """The row as the judge sees it: the reply and the reference, the
+    audit question as the rubric when the row has none, and the
+    verifier's own verdict alongside so a judge that reads it can say
+    why the rule was wrong."""
+    out = dict(row)
+    priv = dict(row.get("privileged") or {})
+    if not priv.get("rubric"):
+        priv["rubric"] = question
+    out["privileged"] = priv
+    out["audit"] = {
+        "question": question,
+        "verifier": row.get("judge_name"),
+        "verifier_reason": row.get("reason"),
+        "verifier_reward": row.get("reward"),
+    }
+    for key in ("reward", "reason", "judge_status", "judge_meta", "judge_name", "failure_class"):
+        out.pop(key, None)
+    return out
+
+
+def _rate(hits: int, n: int) -> tuple[float | None, tuple[float, float] | None]:
+    if not n:
+        return None, None
+    return round(hits / n, 4), wilson_interval(hits, n)
+
+
+def audit_grades(
+    rows: Sequence[dict],
+    *,
+    judge: Callable[[dict], Any],
+    sample: int = 40,
+    passes: int = 0,
+    question: str = AUDIT_QUESTION,
+    seed: int = 0,
+    concurrency: int = 4,
+    timeout: float = 120,
+    judge_name: str | None = None,
+) -> dict[str, Any]:
+    """Estimate the verifier's false-negative rate from a judged sample.
+
+    ``rows`` are graded by the verifier (``run_judge(rows, verifier)`` or
+    ``data.grade(judge=verifier)``): ``reward`` 0/1, ``reason`` from the
+    rule. ``sample`` failed rows (reward 0, judge ok) are drawn with
+    ``seed`` and each is put to ``judge`` (any judge in the ``run_judge``
+    contract: ``rubric_judge()``, ``grade_llm``, your own callable) with
+    the reference in place and ``question`` as the rubric when the row
+    carries none. A judge reward at or above 0.5 on a failed row is a
+    false negative. ``passes`` samples passed rows the same way for the
+    false-positive side.
+
+    Returns ``fn_rate`` with ``fn_ci95`` (Wilson), ``estimated_wrong_fails``
+    (the rate over every failed row), ``reasons`` (the verifier's failure
+    kinds in the sample, each with how many the judge overturned), a few
+    ``examples``, ``fp_rate`` when ``passes`` > 0, and ``warnings``. Above
+    ``FN_WARN`` the summary says to fix the verifier before training;
+    ``select_for_rl(audit=report)`` and ``optimize(audit=)`` carry the
+    same warning into the selection.
+    """
+    if sample < 1:
+        raise ValueError("sample is how many failed rows to put to the judge; at least 1")
+    graded = [r for r in rows if isinstance(r, dict) and _binary(r) is not None]
+    failed = [r for r in graded if _binary(r) == 0 and r.get("judge_status", "ok") == "ok"]
+    passed = [r for r in graded if _binary(r) == 1 and r.get("judge_status", "ok") == "ok"]
+    verifier = next((str(r.get("judge_name")) for r in graded if r.get("judge_name")), None)
+    rng = random.Random(seed)
+    take_f = failed if len(failed) <= sample else rng.sample(failed, sample)
+    take_p = (
+        [] if passes <= 0 else (passed if len(passed) <= passes else rng.sample(passed, passes))
+    )
+
+    def _judge(batch: list[dict]) -> list[dict]:
+        if not batch:
+            return []
+        audit_rows = [_audit_row(r, question) for r in batch]
+        scored = run_judge(
+            audit_rows,
+            judge,
+            judge_name=judge_name,
+            concurrency=concurrency,
+            timeout=timeout,
+            source="audit",
+        )
+        return list(scored.rows)
+
+    judged_f = _judge(list(take_f))
+    judged_p = _judge(list(take_p))
+    judge_label = next((str(r.get("judge_name")) for r in judged_f + judged_p), judge_name)
+
+    checked_f = [r for r in judged_f if r.get("judge_status") == "ok" and _num(r) is not None]
+    overturned = [r for r in checked_f if float(_num(r) or 0.0) >= 0.5]
+    checked_p = [r for r in judged_p if r.get("judge_status") == "ok" and _num(r) is not None]
+    fp_rows = [r for r in checked_p if float(_num(r) or 0.0) < 0.5]
+    errors = sum(1 for r in judged_f + judged_p if r.get("judge_status") != "ok")
+
+    fn_rate, fn_ci = _rate(len(overturned), len(checked_f))
+    fp_rate, fp_ci = _rate(len(fp_rows), len(checked_p))
+
+    reasons: dict[str, dict[str, int]] = {}
+    for r in checked_f:
+        key = _reason_key((r.get("audit") or {}).get("verifier_reason"), verifier)
+        slot = reasons.setdefault(key, {"n": 0, "fn": 0})
+        slot["n"] += 1
+        if float(_num(r) or 0.0) >= 0.5:
+            slot["fn"] += 1
+    reasons = dict(sorted(reasons.items(), key=lambda kv: (-kv[1]["fn"], -kv[1]["n"], kv[0])))
+
+    examples = [
+        {
+            "scenario_id": r.get("scenario_id"),
+            "rollout_index": r.get("rollout_index"),
+            "verifier_reason": (r.get("audit") or {}).get("verifier_reason"),
+            "judge_reason": r.get("reason"),
+        }
+        for r in overturned[:5]
+    ]
+    estimated = round(float(fn_rate) * len(failed)) if fn_rate is not None else None
+
+    warnings: list[str] = []
+    if fn_rate is None:
+        summary = (
+            f"no failed row could be audited ({len(failed)} failed, {errors} judge errors); "
+            "nothing to say about the verifier"
+        )
+        warnings.append(summary)
+    else:
+        lo, hi = fn_ci or (0.0, 0.0)
+        summary = (
+            f"the judge overturned {len(overturned)} of {len(checked_f)} audited failures: "
+            f"false-negative rate {fn_rate:.0%} (95% {lo:.0%}..{hi:.0%}), about {estimated} of "
+            f"{len(failed)} failed rows are right answers the verifier rejected"
+        )
+        if fn_rate > FN_WARN:
+            top = next(iter(reasons), None)
+            warnings.append(
+                f"VERIFIER: {summary}. Fix the verifier before training: each false negative "
+                "halves the separation on its task, and a grader fix has moved mean reward "
+                "0.20 -> 0.48 with no training"
+                + (f". The reason the judge overturns most: {top!r}" if top else "")
+                + "."
+            )
+        if len(checked_f) < 20:
+            warnings.append(
+                f"{len(checked_f)} audited failures is a small sample; the interval is wide. "
+                "Raise sample= for a rate worth acting on."
+            )
+        if fp_rate is not None and fp_rate > FN_WARN:
+            warnings.append(
+                f"the judge also disagreed with {len(fp_rows)} of {len(checked_p)} audited passes "
+                f"(false-positive rate {fp_rate:.0%}); the rule may be too loose as well"
+            )
+    if errors:
+        warnings.append(f"{errors} audit rows were not judged (judge error or timeout)")
+    if fn_rate is not None:
+        warnings.append(
+            "The judge is a second opinion, not ground truth; check it on human labels "
+            "(judge_agreement, judge_trust) before acting on a rate near the threshold."
+        )
+    return {
+        "n_rows": len(graded),
+        "n_failed": len(failed),
+        "n_passed": len(passed),
+        "n_sampled": len(take_f),
+        "n_checked": len(checked_f),
+        "false_negatives": len(overturned),
+        "fn_rate": fn_rate,
+        "fn_ci95": fn_ci,
+        "estimated_wrong_fails": estimated,
+        "n_passes_sampled": len(take_p),
+        "n_passes_checked": len(checked_p),
+        "false_positives": len(fp_rows),
+        "fp_rate": fp_rate,
+        "fp_ci95": fp_ci,
+        "judge_errors": errors,
+        "reasons": reasons,
+        "examples": examples,
+        "verifier": verifier,
+        "judge": judge_label,
+        "question": question,
+        "warnings": warnings,
+        "summary": summary,
+    }
+
+
+def _num(row: dict) -> float | None:
+    value = row.get("reward")
+    if value is None or isinstance(value, bool):
+        return None
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def audit_warning(audit: dict[str, Any] | None) -> str | None:
+    """The one line a selection report carries when an audit found the
+    verifier wrong too often; ``None`` otherwise."""
+    if not isinstance(audit, dict):
+        return None
+    rate = audit.get("fn_rate")
+    if rate is None or float(rate) <= FN_WARN:
+        return None
+    ci = audit.get("fn_ci95") or (0.0, 0.0)
+    return (
+        f"verifier false-negative rate {float(rate):.0%} (95% {ci[0]:.0%}..{ci[1]:.0%}, "
+        f"{audit.get('n_checked')} audited): right answers are being scored 0. Fix the "
+        "verifier before training on this selection (audit_grades)."
+    )
+
+
+def format_audit(report: dict[str, Any]) -> str:
+    """The block a person reads: the summary, then the reasons."""
+    lines = [str(report.get("summary") or "")]
+    for key, slot in (report.get("reasons") or {}).items():
+        lines.append(f"  {key}: {slot['fn']} of {slot['n']} overturned")
+    for ex in report.get("examples") or []:
+        lines.append(
+            f"  {ex.get('scenario_id')} r{ex.get('rollout_index')}: verifier said "
+            f"{ex.get('verifier_reason')!r}; judge: {ex.get('judge_reason')!r}"
+        )
+    return "\n".join(lines)
+
+
+__all__ = ["AUDIT_QUESTION", "FN_WARN", "audit_grades", "audit_warning", "format_audit"]

{whileai-0.56 → whileai-0.57}/whileai/simulations/score/hygiene.py

@@ -314,11 +314,81 @@ def hygiene_warnings(
     return out
 
 
+def _tool_name(tool: Any) -> str:
+    if isinstance(tool, str):
+        return tool
+    if not isinstance(tool, dict):
+        return ""
+    fn = tool.get("function")
+    if isinstance(fn, dict) and fn.get("name"):
+        return str(fn["name"])
+    return str(tool.get("name") or "")
+
+
+def coverage_warnings(
+    rows: Sequence[dict], *, tools: Sequence[dict] | Sequence[str] | None = None
+) -> list[str]:
+    """Plain-words notes on whether a graded run can mean anything.
+
+    A pass@1 of 1.00 over rows where the agent never called a tool, or a
+    marker that fired on no row, is the most expensive eval failure there
+    is, because it reads as a result. ``run_judge`` (so ``evaluate`` and
+    ``data.grade``) attaches these to ``ScoredData.warnings`` and logs
+    them once. ``tools=`` is the declared tool list (OpenAI or bare shape,
+    or just names); ``evaluate(data, judge)`` and ``data.grade`` read it
+    off the run. The tool checks need it, so an agent with no tools is
+    never called hollow for calling none. Each note names the fix.
+    """
+    row_list = [r for r in rows if isinstance(r, dict)]
+    out: list[str] = []
+    n = len(row_list)
+    if n == 0:
+        return out
+    fix = (
+        "Put the ids your world has (order numbers, account names) in the tool "
+        "descriptions or in seeds=, and check the agent wrapper records its steps."
+    )
+    names = [nm for nm in (_tool_name(t) for t in tools or []) if nm]
+    with_calls = sum(1 for r in row_list if tool_calls(r) > 0)
+    if names and with_calls == 0:
+        out.append(f"0 of {n} rows called a tool, so this score says nothing about tool use. {fix}")
+    elif names:
+        called: set[str] = set()
+        for r in row_list:
+            for s in r.get("steps") or r.get("tool_trace") or []:
+                if isinstance(s, dict) and s.get("tool"):
+                    called.add(str(s["tool"]))
+        never = [nm for nm in names if nm not in called]
+        if never:
+            out.append(
+                f"{len(never)} of {len(names)} declared tools were never called "
+                f"({', '.join(never)}); the policy branches behind them are untested. "
+                "Add a seed ask for each."
+            )
+    populated: dict[str, int] = {}
+    for r in row_list:
+        marks = r.get("markers")
+        if not isinstance(marks, dict):
+            continue
+        for name, value in marks.items():
+            populated.setdefault(str(name), 0)
+            if isinstance(value, (int, float)) and not isinstance(value, bool):
+                populated[str(name)] += 1
+    for name in sorted(populated):
+        if populated[name] == 0:
+            out.append(
+                f"marker {name!r} fired on 0 of {n} rows; its rate is not a measurement. "
+                "Add a seed ask that exercises it, or drop it from the report."
+            )
+    return out
+
+
 __all__ = [
     "DEFAULT_MAX_SPREAD",
     "HACK_THRESHOLD",
     "NEAR_DUP_JACCARD",
     "assistant_turns",
+    "coverage_warnings",
     "dedupe_groups",
     "drop_truncated",
     "hygiene_warnings",